Capturing CPU and memory profiles

If the controller or enforcer consumes more CPU or memory than expected, a profile capture can aid our review of the resource consumption.  In general, these are the steps to perform along with capturing profiles.

1. "kubectl top pod" results
2. "ps aux" results inside the pod, "kubectl exec ......... -- ps aux ..."
3. capture the memory/CPU profiles (using the instructions detailed below)
4. collect the pod logs
5. collect the support-bundle from Settings > Configuration > Support

Total Profile Capture (v4.0.2+)

Performance issues can be caused by various factors.  The steps below will capture all profiles from the NeuVector components to have the whole picture.

kubectl -n neuvector exec -it neuvector-manager-pod-name -- cli
#neuvector-svc-controller.neuvector> login
admin#neuvector-svc-controller.neuvector> show controller
admin#neuvector-svc-controller.neuvector> request controller <controller_id> profile -c all
admin#neuvector-svc-controller.neuvector> show enforcer
admin#neuvector-svc-controller.neuvector> request enforcer <enforcer_id> profile -c all
admin#neuvector-svc-controller.neuvector> exit

The default profiling duration is 30s.  So after 35 seconds, the profiles should be available to copy from each of the NeuVector controller and enforcer pods under the /var/neuvector/profile (v5.2.2+ /var/nv_debug/pofile) folder.

Controller profiles

• ctl.memory.prof
• ctl.gc.memory.prof
• ctl.cpu.prof
• ctl.goroutine.prof

Enforcer profiles

• enf.memory.prof
• enf.gc.memory.prof
• enf.cpu.prof
• enf.goroutine.prof

Copy the profile files from the controller and enforcer pods.

for i in `kubectl get pods -n neuvector | egrep "controller|enforcer" | awk '{print $1}'`;do kubectl -n neuvector cp $i:/var/neuvector/profile $i.profile; done

(v5.2.2+ /var/nv_debug/pofile) 
for i in `kubectl get pods -n neuvector | egrep "controller|enforcer" | awk '{print $1}'`;do kubectl -n neuvector cp $i:/var/nv_debug/profile $i.profile; done

Controller | Enforcer Memory Profile

In case only specific profile capture is needed, the below steps demonstrate that.  Repeat the request command for each controller or enforcer.

kubectl -n neuvector exec -it $Manager_Pod -- cli
#neuvector-svc-controller.neuvector> login
admin#neuvector-svc-controller.neuvector> show (controller|enforcer)
admin#neuvector-svc-controller.neuvector> request (controller|enforcer) ($Controller_id|$Enforcer_id) profile -c (cpu|memory) -d 30
admin#neuvector-svc-controller.neuvector> exit

Copy the profile files from pods.

kubectl -n neuvector cp ($Controller_pod|$Enforcer_pod):/var/neuvector/profile/(enf|ctl).(cpu|memory).prof ($Controller_pod|$Enforcer_pod).(cpu|memory).prof

(v5.2.2+ /var/nv_debug/pofile) 
kubectl -n neuvector cp ($Controller_pod|$Enforcer_pod):/var/nv_debug/profile/(enf|ctl).(cpu|memory).prof ($Controller_pod|$Enforcer_pod).(cpu|memory).prof

NOTE: The parenthesis surrounds OR options in the above command references.  For example (controller|enforcer) suggest either either using controller or enforcer in its place.