Known issue deleting etcd nodes in RKE clusters Rancher v2.8.6 ≤ version < v2.9.0

• Rancher v2.8.6 ≤ version < v2.9.0
• RKE 1 cluster
  • RKE v1.5.11 ≥ version < v1.5.14-rc.1 

Due to a known bug in these versions of Rancher and RKE, after removing a node with at least the etcd role, it won't get removed from the etcd list, even if from the kubectl perspective, the node is not part of the cluster anymore.

The node won't be part of the cluster and won't appear using the Rancher UI, the Rancher API, or the kubectl command line.

However, when going into one of the other etcd nodes, and checking the etcd member list, the old etcd node will still appear:

docker exec etcd etcdctl member list

This inconsistent removal may lead you into unexpected scenarios, as the number of etcd nodes running in the cluster won't match the number you expect it to be.
Depending on the number of etcd nodes running, and the actions performed on the theoretically removed etcd nodes, you may even lose control over the cluster if it becomes unresponsive due to etcd quorum lost.

To workaround this issue, you need to:

1. Double-check the etcd member list, where the old node/s should still be listed: 
   

   docker exec etcd etcdctl member list

2. Manually remove the desired etcd node, even if this node was already deleted via kubectl or editing the configuration YAML file.

docker exec etcdctl member remove <etcd node ID>

This bug won't affect you if you run an earlier or more modern version of Rancher or RKE. Please check the Environment section.

Due to a known bug in these versions of Rancher and RKE, after removing a node with at least the etcd role, it won't get removed from the etcd list, even if the node is not part anymore of the cluster from the kubectl perspective.