Recovering cluster.yml and cluster.rkestate files from kubeconfig - RKE clusters
This document (000021114) is provided subject to the disclaimer at the end of this document.
Environment
Important:
This script is specifically intended for use with clusters created using the RKE CLI.
Do not utilize it against any other clusters (e.g. a Rancher-created downstream cluster).
Situation
- cluster.yml: also recognized as the Cluster Configuration File, this file is referenced by RKE to determine what nodes will be in the cluster and how to deploy Kubernetes.
- cluster.rkestate: the Kubernetes Cluster State file, which contains the credentials for full access to the cluster.
Resolution
To recover these two files, it's possible to use the following script. Please note that you will need to fulfill these prerequisites:
- Access to the kubectl command line tool, with the kubeconfig file correctly configured to access the cluster.
- jq command-line JSON processor installed.
- yq command-line YAML, JSON, and XML processor installed.
For RKE binary versions < 1.4.19 , 1.5.10 or 1.6.0:
On RKE versions prior to 1.4.19, 1.5.10 or 1.6.10, the Kubernetes Cluster state file (cluster.rkestate) and cluster.yml files are stored as a configmap, under the kube-system namespace.
#!/bin/bash echo "Building cluster_recovery.yml..." echo "Working on Nodes..." echo 'nodes:' > cluster_recovery.yml kubectl -n kube-system get configmap full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .desiredState.rkeConfig.nodes | yq -P | sed 's/^/ /' | \ sed -e 's/internalAddress/internal_address/g' | \ sed -e 's/hostnameOverride/hostname_override/g' | \ sed -e 's/sshKeyPath/ssh_key_path/g' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on services..." echo 'services:' >> cluster_recovery.yml kubectl -n kube-system get configmap full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .desiredState.rkeConfig.services | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on network..." echo 'network:' >> cluster_recovery.yml kubectl -n kube-system get configmap full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .desiredState.rkeConfig.network | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on authentication..." echo 'authentication:' >> cluster_recovery.yml kubectl -n kube-system get configmap full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .desiredState.rkeConfig.authentication | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on systemImages..." echo 'system_images:' >> cluster_recovery.yml kubectl -n kube-system get configmap full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .desiredState.rkeConfig.systemImages | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Building cluster_recovery.rkestate..." kubectl -n kube-system get configmap full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r . > cluster_recovery.rkestate
For RKE binary versions 1.4.19+ , 1.5.10+ or 1.6.0+:
On RKE 1.4.19, 1.5.10 and 1.6.0 or higher, the Kubernetes Cluster state file (cluster.rkestate) and cluster.yml files are stored as a secret, under the kube-system namespace.
#!/bin/bash echo "Building cluster_recovery.yml..." echo "Working on Nodes..." echo 'nodes:' > cluster_recovery.yml kubectl -n kube-system get secret full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r .desiredState.rkeConfig.nodes | yq -P | sed 's/^/ /' | \ sed -e 's/internalAddress/internal_address/g' | \ sed -e 's/hostnameOverride/hostname_override/g' | \ sed -e 's/sshKeyPath/ssh_key_path/g' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on services..." echo 'services:' >> cluster_recovery.yml kubectl -n kube-system get secret full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r .desiredState.rkeConfig.services | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on network..." echo 'network:' >> cluster_recovery.yml kubectl -n kube-system get secret full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r .desiredState.rkeConfig.network | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on authentication..." echo 'authentication:' >> cluster_recovery.yml kubectl -n kube-system get secret full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r .desiredState.rkeConfig.authentication | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Working on systemImages..." echo 'system_images:' >> cluster_recovery.yml kubectl -n kube-system get secret full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r .desiredState.rkeConfig.systemImages | yq -P | sed 's/^/ /' >> cluster_recovery.yml echo "" >> cluster_recovery.yml echo "Building cluster_recovery.rkestate..." kubectl -n kube-system get secret full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | base64 -d | jq -r . > cluster_recovery.rkestate
Once the execution is completed, you will find two files in the folder of execution: cluster_recovery.yml and cluster_recovery.rkestate, which correspond to a copy of the Cluster Configuration File and the Kubernetes Cluster State file.
After the recovery is done, please back up these files in a secure location to avoid future loss.
Cause
This change has been implemented to mitigate the CVE-2023-32191. More information is available at https://github.com/rancher/rke/security/advisories/GHSA-6gr4-52w6-vmqx
Additional Information
https://gist.github.com/mattmattox/d32b3fea4820075c08c6cc2f6d736702
Disclaimer
This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.
- Document ID:000021114
- Creation Date: 26-Jun-2023
- Modified Date:13-Nov-2024
-
- SUSE Rancher
For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com