Using Private Registries for Custom and Imported Clusters

Air-gapped

SUSE Rancher 2.9.3

Rancher fails to import or create a custom cluster node while pulling images when the default registry (Docker Hub) or the configured Global Default Registry is not accessible to this specific cluster.

At the time of writing of this KB (current as of v2.9.3) one of the following workarounds may be useful to you:

• In RKE2 Custom Clusters you can add the contents usually added to the `registries.yaml` manifest in RKE2 cluster nodes (see this doc) to the `machineGlobalConfig` in the cluster manifest when creating the cluster per https://documentation.suse.com/cloudnative/rancher-manager/latest/en/cluster-deployment/configuration/rke2.html#_machineglobalconfig (with the minor correction of `registries` instead of `private-registry` per rancher/rancher-product-docs/pull/86).
• In K3s Custom Clusters you can add the contents usually added to the `registries.yaml` manifest in K3s cluster nodes (see this doc) to the `machineGlobalConfig` in the cluster manifest when creating the cluster per https://documentation.suse.com/cloudnative/rancher-manager/latest/en/cluster-deployment/configuration/k3s.html#_machineglobalconfig.
• You can also manually pull the needed images to each of the nodes and retag them so that the local container management finds them locally (for example, you can use `docker tag repo.url:Port/rancher/rancher-agent:v2.9.1 rancher/rancher-agent:v2.9.1` to retag a `rancher-agent` image pulled from your private repo so your node doesn't attempt to pull from Dockerhub).  Consult the `rancher-images.txt` file in your Rancher version's release notes for a list of all images and versions Rancher will need in general, or review the logs generated on the node itself when import or creating nodes for a cluster for specific images the node is failing to retrieve. These will include (but are not necessarily limited to): `rancher/rancher-agent:v2.x.x`, `rancher/shell:vx.x.x`, `rancher/fleet-agent:vx.x.x` (see `rancher-images.txt` for your necessary version numbers)
• Previously there has been a workaround of adding an `agentImageOverride` or `desiredAgentImage` in the cluster manifest `spec` to override the default, but there is currently an issue with this which can be tracked for resolution here: https://github.com/rancher/rancher/issues/47593

At this time, when importing a cluster or creating a custom cluster, Rancher images are pulled from the Global configured private registry (or the default of Docker Hub if one is not configured). If you are using a separate private registry for multiple air-gaped downstream clusters, you may find that this is not a viable solution.  

Related GitHub issues:

• [RFE] Cluster level private registry for imported Air-gapped setup #35192
• Need documentation on deploying and managing airgapped clusters #80
• [BUG] Agent image field not getting updated for imported clusters #47593