Why does the kubelet certificate still show as expired after performing a cluster certificate rotation in an Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned cluster?

Question

Why is Kubelet certificate still indicating expired after performing a cluster certificate rotation?

Pre-requisite

• A Rancher Kubernetes Engine (RKE) or Rancher v2.x provisioned Kubernetes cluster

Answer

Before Rancher v2.3.3 and RKE v1.0.0, cluster provisioning did not supply the --tls-cert-file and --tls-private-key-file arguments to the Kubelet container. As a result, the kubelet automatically generates the kubelet.crt, and kubelet.key files under the /var/lib/kubelet/pki ​directory and the certificate is not rotated during the certificate rotation.

How to verify the Kubelet certificate

• openssl s_client -connect <NODE IP>:10250 | openssl x509 -text

• curl -vk https://<NODE IP>:10250

Resolution

You can rotate the kubelet certificate in RKE and Rancher provisioned clusters as follows:

How to rotate the kubelet certificate in Rancher v2.2.0 - v2.3.0 and RKE v0.2.0 - v0.3.2 provisioned clusters

For clusters provisioned and managed by Rancher prior to v2.3.3 or RKE prior to v1.0.0, you will need to manually delete the kubelet.crt and kubelet.key in /var/lib/kubelet/pki and restart the Kubelet container:

docker exec kubelet rm /var/lib/kubelet/pki/kubelet.crt
docker exec kubelet rm /var/lib/kubelet/pki/kubelet.key
docker restart kubelet

How to rotate the kubelet certificate in Rancher v2.3.2+ provisioned clusters

For Rancher provisioned clusters managed by Rancher v2.3.3 and above, you can set the generate_serving_certificate kubelet option to true in the cluster configuration YAML to rotate the kubelet certificate.

N.B. If hostname_override is configured for any nodes in the cluster, please read the requirements within the documentation here before proceeding.

1. For the affected cluster click 'Edit Cluster' from within the Rancher UI cluster view.

2. Click 'Edit as YAML'.

3. Set the generate_serving_certificate option to true for the kubelet, per the below:

   services:
     kubelet:
       generate_serving_certificate: true

4. Click 'Save' to intitate a cluster reconciliation and trigger rotation of the kubelet certificate.

How to rotate the kubelet certificate in RKE v1.0.0+ provisioned clusters

For clusters managed by RKE v1.0.0 and above, you can set the generate_serving_certificate kubelet option to true in the cluster configuration YAML and invoke rke up to rotate the kubelet certificate.

N.B. If hostname_override is configured for any nodes in the cluster, please read the requirements within the documentation here before proceeding.

1. Set the generate_serving_certificate option to true for the kubelet, within the cluster configuration YAML file, per the below:

   services:
     kubelet:
       generate_serving_certificate: true

2. Invoke rke up --config <cluster configuration yaml> to update the cluster configuration with the new kubelet option and trigger rotation of the kubelet certificate.

Further Reading

RKE Certificate Rotation Documentation. Rancher v2.x Certificate Rotation Documentation. Kubelet Service Certificate Requirements Documentation.