What is the process performed during a Rancher v2.x upgrade and what is the impact to the managed downstream Kubernetes clusters?

Question

What is the process performed during a Rancher v2.x upgrade and what is the impact to the managed downstream Kubernetes clusters?

Pre-requisites

• Running Rancher v2.x HA deployed using Helm.

Answer

The bulk of the Rancher HA installation and upgrade are performed by using Helm. The core piece of the Rancher Helm Chart is the Rancher deployment. Please note the following characteristics of this Helm Chart:

• Deployment is set to a replica of 3. This means Kubernetes will attempt to run and maintain three rancher pods.
• Deployment is set to do a rolling update with a max surge of 25% and max unavailability of 25%. This means:
  1. During an upgrade, pods are updated in chunks, not all at once.
  2. During an update, no more than 4 pods will be running at once
  3. During an update, no fewer than 2 pods will be available at once
• Deployment has an anti-affinity for the node's hostname. This means Kubernetes will attempt to place each pod on a separate host. For three pods and three hosts, that means one pod on each host.

Rancher will also apply two other important manifests to the Rancher HA cluster as well as all managed clusters. These are described below:

cattle-cluster-agent deployment

• Deployment is set to a replica of 1
• Deployment is set to do a rolling update with a max surge of 25% and a max unavailability of 25%. See Rancher's deployment description above for the behavior of these settings.

cattle-node-agent daemonset

• Daemonset will deploy one agent per node
• Daemonset is set to a rolling update with max unavailable of 1 pod. That means during an update, one pod is updated at a time.

Given the information above on how the manifests are defined, below is the expected sequence of events during a Rancher upgrade:

Rancher HA cluster

1. A new rancher pod is created
2. An old rancher pod is terminated
3. A new second rancher pod is created
4. A second old rancher pod is terminated
5. A new third rancher pod is created
6. A third old rancher pod is terminated
7. The latest versions of the cattle-cluster-agent and cattle-node-agent manifests are updated and deployed on the cluster. These deployments are triggered in parallel and will result in a new cattle-cluster-agent and new cattle-node-agents running on the cluster.

Downstream clusters

Once Rancher is upgraded, Rancher will check each cluster it manages to make sure the cattle-cluster-agent and cattle-node-agents are up to date. If the cluster is not in a "Provisioning" state, meaning another cluster update is in progress, it will deploy the latest cattle-cluster-agent and cattle-node-agent manifests into the cluster. All managed clusters are updated in parallel and not sequentially.

Other workloads running in the cluster should not be impacted.

Further Reading

Kubernetes deployments - https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

Kubernetes daemonsets - https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/