How to clean the orphaned cluster objects from the deleted cluster namespaces.

This document (000020788) is provided subject to the disclaimer at the end of this document.

Environment

Rancher 2.x

Situation

In some cases there may be orphaned cluster objects left behind after the in-proper deletion of a downstream cluster in Rancher. These orphaned objects could introduce a condition that causes the leader Rancher pod to enter a CrashLoop state.

Examples of errors from the Rancher pod logs.

[ERROR] failed to call leader func: namespaces "c-xxxxx" not found
fatal error: concurrent map read and map write

[ERROR] error syncing ‘c-xxxx/p-xxxx’: handler mgmt-project-rbac-remove: failed to remove finalizer on controller.cattle.io/mgmt-project-rbac-remove, requeuing
[ERROR] error syncing ‘c-xxxxx/p-xxxxx’: handler mgmt-project-rbac-remove: failed to remove finalizer on controller.cattle.io/mgmt-project-rbac-remove, requeuing
[ERROR] error syncing ‘c-xxxxx/p-xxxxx’: handler cluster-registration-token: clusters.management.cattle.io "c-xxxxx" not found, requeuing

Resolution

Find the objects under the deleted cluster namespaces and manually delete each objects. Make sure there are no such orphaned objects or namespaces left in the local cluster.

1. Set a kubeconfig for the Rancher (local) management cluster to be used with the following steps

2. Verify the Active downstream clusters

kubectl get clusters.management.cattle.io -o custom-columns="ID:.metadata.name,NAME:.spec.displayName,K8S_VERSION:.status.version.gitVersion,CREATED:.metadata.creationTimestamp,DELETED:.metadata.deletionTimestamp,LAST_READY:.status.conditions[?(@.type == 'Ready')].lastUpdateTime,READY:.status.conditions[?(@.type == 'Ready')].status" --sort-by=.metadata.creationTimestamp

3. Cross verify with the Rancher pod logs to get the deleted downstream cluster namespace and collect the details. Compare with the active list of clusters versus the cluster namespaces.

kubectl logs -n cattle-system -l app=rancher -c rancher

kubectl get ns -A |grep "c-"

4. If there is a cluster that is stuck deleting, this may not complete. In this case, the finalizer object can be removed from the cluster.management.cattle.io object. Please note the c-xxxxx needs to be replaced with the cluster ID that is stuck deleting.

kubectl patch clusters.management.cattle.io <c-xxxxx> -p '{"metadata":{"finalizers":[]}}' --type=merge

5. If there is a namespace for a cluster that no longer exists, get the orphaned object details under the deleted cluster namespace.

kubectl api-resources --verbs=list --namespaced -o name  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <c-xxxxx>

6. Do the cleanup of orphaned objects.

Create the cluster namespace which is deleted, ignore if the cluster namespace is present

kubectl create ns <c-xxxxx>

Check the objects detected (in step 5) if desired, each object should have a deletion timestamp if a finalizer is preventing the object from being deleted.

kubectl -n <c-xxxxx> get <resource type> <name of object> -o yaml

Remove the finalizer to unblock the deletion of the objects. The command needs to be run for each object.

kubectl -n <c-xxxxx> patch <resource type> <name of object> -p '{"metadata":{"finalizers":[]}}' --type=merge

Make sure there are no objects left in the namespace.

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <c-xxxxx>

Finally, delete the namespace.

kubectl delete ns <c-xxxxx>

Cause

It is important to delete downstream clusters in a process to allow Rancher to delete clusters and clean nodes that are in an Active state.

Downstream cluster deletion is ideally performed from the Rancher UI / API, where nodes are available and able to be gracefully removed. For example, where possible do not terminate nodes in the infrastructure before the deletion is completed.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.