Rancher Upgrade Checklist

This article details the high-level steps required when planning and performing a Rancher and/or Kubernetes upgrade.

Planning

The following are the high-level rules for planning a Rancher/Kubernetes/Docker upgrade.
 

NOTE: The versions contained in this document were current at the time of writing and are meant only as an example. 
 

• The support matrix gives a great idea of the versions that are certified by Rancher and work best together.
• The recommended order of upgrades is: Rancher, Kubernetes, and then Docker/OS while remaining within the support matrix
  • For example, do not upgrade Rancher to a newer version if the Kubernetes versions of your local and downstream clusters are not supported by the upgraded version of Rancher. Instead, upgrade Kubernetes to a version supported by both the current and upgraded versions of Rancher.
• All upgrades should be performed in a lab or non-prod environment first
• Please see the following recommendations when planning version upgrades:
  • Rancher:
    • Avoid skipping minor versions when upgrading.
      • For example: when upgrading from v2.4.x -> v2.6.x we encourage upgrading v2.4.x -> v2.5.x -> v2.6.x
    • Upgrade from the newest stable version of the current release to the newest stable version of the next release
      • When upgrading from v2.5.7 -> v2.6.x, we encourage you to first upgrade to 2.5.16 (latest) as an intermediary step. We also recommend upgrading directly to the latest Rancher release for the new version (i.e. 2.5.16 -> 2.6.latest, not 2.6.10)
    • Do NOT upgrade to pre-release or non-stable versions for production workloads as they are not yet fully tested and supported.
      • Pre-release versions can be identified by a -rc# following the Rancher version number (i.e. v2.7.5-rc6)
  • Kubernetes: Avoid skipping minor versions, as this can increase the chances of an issue due to accumulated changes, as per the upstream Kubernetes documentation
    • For example: when upgrading from v1.19.x -> v1.22.x we encourage upgrading v1.19.x -> v1.20.x -> v1.21.x -> v1.22.x
  • RKE: Perform one minor RKE version jump at a time
    • For example: when upgrading from v0.1.x -> v1.1.0 instead do v0.1.x -> v0.2.x -> v0.3.x -> v1.0.x -> v1.1.x
• Adding a grace period between upgrades is recommended to run workloads and confirm that the single upgrade did not cause any issues
  • For example: add 24 hours between upgrading Rancher, the local cluster, and downstream clusters
  • This aids in troubleshooting any unexpected issues as we can reference the changes made by a single upgrade instead of looking at multiple changes that might be compounding the issue.
• It is not required, but it is recommended to pause any application deployments using the Rancher API during an upgrade

Data collection

Before you start your upgrade, please collect the following pieces of information to best prepare yourself in case you need to open a support ticket.

• Scheduled change window:
• Current Rancher version (rancher/rancher image tag, or shown bottom left in the UI):
• Target Rancher version:
• Installation option (single install/HA):
• Current Kubernetes version of Rancher local cluster (use kubectl version):
• Current Docker version (use docker version):

Rancher Pre-Upgrade

• Check if the Rancher UI is accessible
• Check if all clusters in UI are in an Active state
• Check if all pods in kube-system and cattle-system namespaces are running (in both Rancher and downstream clusters)

  kubectl get pods -n kube-system
  kubectl get pods -n cattle-system

• Verify the datastore has scheduled snapshots configured, and these are working.
  • RKE: If Rancher is deployed on a Kubernetes cluster built with RKE, verify etcd snapshots are enabled and working, on etcd nodes you can confirm with the following:

    ls -l /opt/rke/etcd-snapshots
    docker logs etcd-rolling-snapshots

  • k3s: If Rancher is deployed on a k3s Kubernetes cluster, ensure scheduled backups are configured and working. Please see the k3s documentation pages for further information on this.
  • RKE2: If Rancher is deployed on a RKE2 Kubernetes cluster, ensure scheduled backups are configured and working. Please see the RKE2 documentation pages for further information on this.
• Create a one-time datastore snapshot, please see the following documentation for RKE, RKE2, and k3s , and the single node Docker install options for more information

Rancher Upgrade steps

• The Rancher upgrade process is detailed in the upgrade documentation for both HA, and single node using Docker.

Rancher Review/Verify

After the upgrade is completed, go through the following checklist to verify your environment is in working order.

• Check if the Rancher UI is accessible
  • You should be able to login into Rancher, view clusters, and browse to workloads
• Verify the Rancher version has changed in UI
  • After logging into Rancher, review the version in the bottom left corner of the page
• Check if all clusters in UI are in an Active state
• Check if all pods in kube-system and cattle-system are running (in both Rancher and downstream clusters)
• Check the cattle-cluster-agent and cattle-node-agent pods are running in all downstream clusters and running the latest version
  • The rollout of the updated agent versions can take some time if there are a lot of downstream clusters or nodes
• Create a one-time datastore snapshot, please see the following documentation for RKE,  RKE2, and k3s, and the single node Docker install options for more information

Rancher Rollback steps

The Rancher rollback process is detailed in the rollback documentation, please follow the relevant link for Rancher installed on a Kubernetes cluster, or Docker

Follow-up tasks (optional)

• Upgrade the Rancher management cluster, this is often a follow-up to the Rancher upgrade. Please see the RKE, RKE2 , and k3s upgrade documentation for the upgrade process, as mentioned it is best to avoid skipping minor versions
• Upgrade the downstream clusters, please see the documentation for more information. A snapshot of both the local and downstream clusters before the upgrade is recommended to provides the maximum amount of recoverability options in the event of a rollback
• Docker/OS upgrades, please our article on performing rolling changes to nodes

Below is an example plan for upgrading from Rancher 2.5.7 to 2.7.5 and upgrading the local and downstream clusters from 1.18 to 1.25
 

• Upgrade Rancher 2.5.7 to Rancher 2.5.16
• Test for a minimum of 24 hours, preferably longer

• Upgrade Kubernetes on the local cluster from 1.18 to 1.19
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on the local cluster from 1.19 to 1.20
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.18 to 1.19
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.19 to 1.20
• Test for a minimum of 24 hours, preferably longer

• Upgrade Rancher 2.5.16 to Rancher 2.6.latest (2.6.13 at time of writing)
• Test for a minimum of 24 hours, preferably longer

• Upgrade Kubernetes on the local cluster from 1.20 to 1.21
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on the local cluster from 1.21 to 1.22
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on the local cluster from 1.22 to 1.23
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on the local cluster from 1.23 to 1.24
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.20 to 1.21
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.21 to 1.22
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.22 to 1.23
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.23 to 1.24
• Test for a minimum of 24 hours, preferably longer

• Upgrade Rancher 2.6.latest (2.6.13 at time of writing) to Rancher 2.7.latest (2.7.5 at time of writing)
• Test for a minimum of 24 hours, preferably longer

• Upgrade Kubernetes on the local cluster from 1.24 to 1.25
• Test for a minimum of 24 hours, preferably longer
• Upgrade Kubernetes on all downstream clusters from 1.24 to 1.25
• Test for a minimum of 24 hours, preferably longer