etcd snapshots showing 0B size in the Rancher UI

This document (000021447) is provided subject to the disclaimer at the end of this document.

Environment

Rancher managed RKE2 cluster

Situation

When viewing etcd snapshots in the Rancher UI for a downstream RKE2 cluster, the snapshots show a size of 0 B.

Resolution

The workaround is to modify the rke2 lease and make the lease-holder match the rke2-etcd lease. The following one-liner can be used for implementing this workaround:

kubectl get lease -n kube-system rke2-etcd -o jsonpath='{.spec.holderIdentity}' | xargs -I {} kubectl patch lease -n kube-system  --patch='{"spec":{"holderIdentity":"{}"}}' --type=merge rke2; kubectl get lease -n kube-system -o custom-columns='name:.metadata.name,holder:.spec.holderIdentity' | grep rke2

After this is done, if the rke2 or rke2-etcd ConfigMap objects have an annotation with a differing holderIdentity value, please modify that to match as well.

After the workaround is implemented, please wait some time for the snapshots to reconcile.

Ultimately, the solution will be to upgrade to the version where this bug is fixed.

Cause

One reason this can happen is a mismatch between the leaseholder for the rke2 and rke2-etcd leases There is also an annotation in the rke2 and rke2-etcd ConfigMap objects that should be checked.

kubectl get cm -n kube-system rke2 -o yaml | grep holderIdentity
kubectl get cm -n kube-system rke2-etcd -o yaml | grep holderIdentity
kubectl get lease -n kube-system rke2
kubectl get lease -n kube-system rke2-etcd

This is a known bug: https://github.com/rancher/rke2/issues/5866

Additional Information

If there is no lease-holder mismatch, some other potential issues that could cause this problem are mentioned in the following knowledge base articles:

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.