[Rancher] Operational Advisory, 20220405: Rancher Kubernetes Distributions and Etcd 3.5 Updates

• Users running Rancher 2.6.4+ who have deployed Rancher on a single node as a Docker installation. Reminder: This installation method is not recommended for any production environment and is only recommended for development/sandbox testing. If you are running Rancher on a managed Kubernetes cluster, then you will have to refer to your Kubernetes service provider to determine if you are affected by this advisory.
• Users running Kubernetes 1.22+ and 1.23+ of any of the Rancher Kubernetes Distributions (RKE, RKE2, K3s) and using etcd as your datastore. The default datastore for RKE and RKE2 is etcd. This applies to standalone Kubernetes clusters as well as any downstream clusters provisioned by Rancher.

• Stop deploying into production any new Kubernetes clusters using Rancher Kubernetes distributions versions 1.22/1.23 until a proper fix is provided by the etcd maintainers and included into the affected distribution.
• Update your etcd configuration to enable the experimental-initial-corrupt-checkoption. This flag will be turned on by default in etcd v3.6, but does not by itself fix the problem; it can only detect and repair the issue if it does occur.
  Note: each distribution has its own recommendation on how to enable this option; see below for more details.
• Avoid terminating etcd unexpectedly (using kill –9, etc)
  • For RKE1 clusters, avoid stopping/killing the etcd containers adhoc without properly cordoning/draining nodes and taking backups
  • For RKE2 clusters,
    • Avoid sending SIGKILL to the etcd or rke2 process.
    • Avoid using the killall script (rke2-killall.sh) to stop RKE2 on servers hosting production workloads. The killall script is meant to clean up hosts prior to uninstallation or reconfiguration and should not be used as a substitute for properly cordoning/draining a node and stopping services.
  • For k3s clusters,
    • Avoid sending SIGKILL to the k3s process.
    • Avoid using the killall script (k3s-killall.sh) to stop K3s on servers hosting production workloads. The killall script is meant to clean up hosts prior to uninstallation or reconfiguration and should not be used as a substitute for properly cordoning/draining a node and stopping services.
• Ensure nodes are not under significant memory pressure that may cause the Linux kernel to terminate the etcd process.

Ensure that nodes are not terminated unexpectedly. Avoid force-terminating VMs, unexpected power loss, etc.

How do I enable the recommended flag in etccd?
 

For Users provisioning RKE/k3s/RKE2 clusters through Rancher
Provisioned RKE Clusters
If you are running 1.22 or 1.23, upgrade to the following respective versions to enable the recommended experimental-initial-corrupt-check flag in etcd.

• RKE 1.22 - v1.22.7-rancher1-2
• RKE 1.23 (Experimental) - v1.23.4-rancher1-2

1. From the “Cluster Management” page, click the vertical three-dots on the right-hand side for the cluster you want to edit.
2. From the menu, select “Edit YAML”.
3. Edit the spec.rkeConfig.machineGlobalConfig.etcd-arg section of the YAML to add in an etcd argument. Note: Your YAML may be slightly different from the example below.

• Click “Save” at the bottom. Rancher will update the configuration and restart the necessary services.

• If you already have an existing RKE1 cluster using an affected version, you can set experimental-initial-corrupt-check: true in extra_args for etcd.

1. Create or edit the config file at /etc/rancher/rke2/config.yaml.
2. Add the following line to the end of the file:

1. Save the config file and then run systemctl restart rke2-server to apply the change.

1. Create or edit the config file at /etc/rancher/k3s/config.yaml.
2. Add the following line to the end of the file:

1. Save the config file and then run systemctl restart k3s to apply the change.

• Rancher Support Advisories
• Rancher Support FAQs