How to configure Spegel (Embedded Registry Mirror) for Rancher-provisioned RKE2/k3s clusters

This document (000021681) is provided subject to the disclaimer at the end of this document.

Environment

Rancher v2.9+
A Rancher-provisioned K3s cluster with v1.29.12+k3s1, v1.30.8+k3s1, v1.31.4+k3s1 or above, or a Rancher-provisioned RKE2 cluster with v1.29.12+rke2r1,v1.30.8+rke2r1, v1.31.4+rke2r1 or above

Situation

This article details how to enable and monitor the Embedded Registry Mirror, provided by an embedded instance of Spegel, in Rancher-provisioned K3s and RKE2 clusters.

Resolution

Enable the Embedded Registry Mirror

To enable the Embedded Registry Mirror the `embedded-registry: true` option needs to be set on server nodes within the cluster. To enable this on a Rancher-provisioned K3s or RKE2 cluster, navigate to Cluster Management within the Rancher UI and select Edit YAML for the relevant cluster. Define `embedded-registry:true` within a machineSelectorConfig block for controplane nodes, as below. Optionally, set `supervisor-metrics: true` to enable querying of Spegel metrics:
```
    machineSelectorConfig:
      - config:
          embedded-registry: true
          supervisor-metrics: true
        machineLabelSelector:
          matchLabels:
            rke.cattle.io/control-plane-role: 'true'
```
Per, the K3s and RKE2 documentation, "Enabling mirroring for a registry allows a node to both pull images from that registry from other nodes, and share the registry's images with other nodes." Therefore, if a registry that you wish to use with the Embedded Registry Mirror is not already defined in the cluster's registry mirror configuration, you will need to add it. You can add the registry without any endpoints, per the following example for docker.io (Docker Hub):
```
    registries:
      mirrors:
        docker.io: {}
```
Click Save to update the cluster with these changes, enabling the Embedded Registry Mirror.

Query metrics for the Embedded Registry Mirror

If you set `supervisor-metrics: true` in step 1 above, you will be able to query the Embedded Registry Mirror (Spegel) metrics on each node within the cluster. Please note that the kubectl queries below will only work to the Kubernetes API endpoint on cluster server nodes directly, and will not work with the Rancher-proxied Kubernetes API endpoint for the cluster, nor via the built-in kubectl shell for the cluster within the Rancher UI.

Query the Spegel metrics for an RKE2 cluster node

The Spegel metrics will be exposed via the RKE2 supervisor metrics for each cluster node, on port 9345. The example below shows how to query the spegel metrics for a node from an RKE2 cluster server node:

$ export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
$ alias kubectl=/var/lib/rancher/rke2/bin/kubectl
$ kubectl get --server https://<node-ip>:9345 --raw /metrics | grep spegel
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 25
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 25
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 50
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 218

Query the Spegel metrics for a K3s cluster node

The Spegel metrics will be exposed via the K3s supervisor metrics for each cluster node, on port 6443. The example below shows how to query the Spegel metrics for a node from a K3s cluster server node:

kubectl get --server https://<node-ip>:6443 --raw /metrics | grep spegel
libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 1
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 1
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 9
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 9
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 18
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 92

Additional Information

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.