'Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"' error when upgrading Rancher immediately after performing a local cluster Kubernetes upgrade

This document (000021315) is provided subject to the disclaimer at the end of this document.

Environment

A Rancher v2.x instance running in an RKE or RKE2 cluster

Situation

The Rancher upgrade task is launched immediately after upgrading the underlying Kubernetes cluster. The Rancher upgrade fails with the following error:

Resolution

Before running a Rancher upgrade, check that ingress-nginx Pods in the local cluster are running. If the Rancher upgrade task is executed in an automated fashion, a delay should be introduced between the cluster upgrade and the Rancher helm upgrade.

The commands below will enable you to validate that the ingress-nginx pods are running before performing the Rancher helm upgrade. The flag "timeout" ensures the command exits with an error on a timeout, rather than waiting indefinitely, e.g. --timeout=2m.

RKE1

kubectl rollout status daemonset -n ingress-nginx nginx-ingress-controller --timeout=x

kubectl wait --for=condition=Available -n ingress-nginx nginx-ingress-controller

RKE2

kubectl rollout status daemonset -n kube-system rke2-ingress-nginx-controller  --timeout=x

kubectl wait --for=condition=Available -n kube-system rke2-ingress-nginx-controller

Cause

The 'helm upgrade --install` task for Rancher is invoked before the ingress-nginx pods in the cluster are in a running state, following the cluster upgrade. As a result, the ingress-nginx validating webhook cannot be satisfied, resulting in errors when helm attempts to patch the Rancher ingress resource.

Additional Information

Rancher upgrade checklist: https://www.suse.com/es-es/support/kb/doc/?id=000020061
Rancher upgrade: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades#steps-to-upgrade-rancher
How to install or upgrade to a specific Rancher v2.x version: https://www.suse.com/es-es/support/kb/doc/?id=000020217

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.