NeuVector fails when validating the Admission Control Webhook through UI

• Product: SUSE NeuVector v5.x with Rancher v2.x.
• Feature: Admission Control Webhook
• Target: Nodes, Pods, and Images from registries.
• Capability: Configuring the admission control webhook functionality in NeuVector.

The Admission Control feature helps manage the creation of resources in your cluster and customize them as you see fit. The Admission Control feature is turned off by default. Go to the Policy -> Admission Control page to enable it in the NeuVector console.

Once the Admission Control feature is enabled successfully, the following ValidatingWebhookConfiguration resource will be created automatically. The most important information in the ValidatingWebhookConfiguration resource for NeuVector is cluster resources. Currently, once a cluster resource such as Deployment NeuVector registered is created, the request will be sent from the orchestration platform apiserver to one of the NeuVector Controllers to determine if it should be allowed or denied based on the user-defined rules in NeuVector Policy -> Admission Control page.

To test the Kubernetes connection for the UI mode access, go to Policy -> Admission Control -> Advanced Settings. Note that the test admission webhook feature only shows up after the admission webhook is enabled.

Suppose you identify that the admission webhook works fine when creating a resource via YAML files and see a failure when testing via NeuVector UI. This type of failure is valid only with NeuVector installed with Rancher, since refers to Rancher Integration, where NeuVector is blocked from managing some functionalities. The Workaround will be shown in this KB.

 

You can apply the workaround by registering a ClusterRole & ClusterRoleBinding directly in your domain, where this role will allow permissive communication between the Rancher and NeuVector webhooks:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: neuvector-psa
rules:
- apiGroups:
  - management.cattle.io
  resources:
  - projects
  verbs:
  - updatepsa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: neuvector-psa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: neuvector-psa
subjects:
- kind: ServiceAccount
  name: neuvector
  namespace: cattle-neuvector-system

Please remember to adjust the YAML above according to the namespace where NeuVector has been installed (Example: neuvector, cattle-neuvector-system, etc).
 
To apply the roles, save the above snippet in a file and run:

kubectl apply -f <filename>.yaml

This workaround is justified because Rancher's webhook blocks NeuVector from adding the labels when the controller modifies the namespace. After all, NeuVector's service account does not have the right to the "updatepsa" verb. The Roles that will be added grant the right to "updatepsa".

To start individual troubleshooting during a failure, please collect the logs immediately after displaying the error. Disabling and enabling Admission control is also helpful for more accurate analysis.

In this case, you don't need to activate debug mode, but it is useful for collecting more complete data from your controllers.
 
In your controller logs, if error 403 is displayed referencing Rancher's webhook, this is the cause of the Admission Control Webhook problems:

ERRO|CTL|nvvalidatewebhookcfg.workSingleK8sNsLabels: update resource failed - err=kubernetes api: Failure 403 admission webhook "rancher.cattle.io.namespaces" denied the request: Unauthorized nsName=cattle-neuvector-system

This is also related to an issue documented on the Rancher page: https://github.com/rancher/rancher/issues/41191.

Reported to Engineering