Inability to attach/detach vSphere CNS block volumes

This document (000021286) is provided subject to the disclaimer at the end of this document.

Environment

Rancher 2.6 / 2.7
RKE1/RKE2
Kubernetes v1.19+
vSphere 6.7 U3+ or vSphere 7.0+
Vsphere cloud provider:
- Vsphere CPI: rancher-vsphere-cpi:100.3.0+up1.2.1+
- Vsphere CSI: rancher-vsphere-csi:100.3.0+up2.5.1-rancher1+

Situation

Inability to detach/attach CNS block volumes:

Customers can create CNS block volumes in the RKE1/RKE2 cluster using the Vsphere CSI.

However, when scaling down a workload (deployment,statefulset), the block volume does not get detached automatically from the nodes. Scaling up the workload, the following error appears in the cluster events:

rpc error: code = Internal desc = queryVolume failed for volumeID: "5db7cc3c-62b9-427d-823b-87729fcef771" with err=ServerFaultCode: NoPermission

Resolution

This error indicates the user is missing the permission "Cns.Searchable" at the root vCenter level and Datastore level.

To grant the user account the Cns.Searchable permission in vSphere, see the following documentation:
https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials

Cause

The user account within vSphere must be granted the following permissions:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-C04F1605-D158-4B65-810F-6F5B109BCDEC.html

Additional Information

vSphere CNS Block volumes:

Cloud Native Storage (CNS) integrates vSphere and Kubernetes and offers capabilities to create and manage container volumes in vSphere environment. CNS consists of the two components, CNS component in vCenter Server and a vSphere volume driver in Kubernetes, called vSphere Container Storage Plug-in.

vSphere Cloud Provider Interface (CPI):

Is responsible for running all the platform-specific control loops that were previously run in core Kubernetes components like the KCM and the kubelet, but have been moved out-of-tree to allow cloud and infrastructure providers to implement integrations that can be developed, built, and released independent of Kubernetes core

vSphere Container Storage Interface (CSI):

It is a specification designed to enable persistent storage volume management on Container Orchestrators (COs) such as Kubernetes. The specification allows storage systems to integrate with containerized workloads running on Kubernetes. Using CSI, storage providers, such as VMware, can write and deploy plugins for storage systems in Kubernetes without a need to modify any core Kubernetes code.

CSI allows volume plugins to be installed on Kubernetes clusters as extensions. Once a CSI-compatible volume driver is deployed on a Kubernetes cluster, users can use the CSI to provision, attach, mount, and format the volumes exposed by the CSI driver.

The CSI driver for vSphere is csi.vsphere.vmware.com.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.