System Requirements wrap-up

We are often asked for guidance on how to do good capacity planning for the various NeuVector micro-services.

This document aims to summarise all existing documentation in this regard.

First, please ensure you have a STAGING environment where you can do tests; every PRODUCTION environment is different, and arranging our indications with what you find in your non-production environment is the best practice to follow.

The performance and sizing considerations are dependent on the following:

• Number of Nodes/Hosts in the Cluster, as each Node will report and sync with the Control Plane
• Number of workloads (Containers) and distinct applications (example Deployments) running on each Node
• Security functions used, including:
  • Registry image scanning and size of images scanned
  • Runtime scanning of Hosts and Containers for CVEs, Secrets, and CIS Benchmarks
  • Network inspection and number of Network Rules
  • Deep Packet Inspection (DPI) such as layer 7 protocol-based segmentation, Web Application Firewall (WAF) rules, and Data Loss Prevention (DLP) rules
  • For Network Protection, whether the mode is set to MONITOR (alert) versus PROTECT (inline blocking)
• Number of remote clusters in Multi-Cluster Federation

Micro-services deep-dive:
Manager -> The Manager is a stateless container that provides a web-UI (HTTPS only) console for users to manage the NeuVector security solution. Occasionally, slow load times may be noticed if large datasets are retrieved from the Controller, such as Vulnerability Data in the Security Risks menu or network connections in the Network Activity map.
Controller -> The Controller manages the NeuVector Enforcer container cluster. It also provides REST APIs for the management console.
The Controller is sensitive to the number of Nodes (with Enforcers) in the Cluster, as it manages events and status from each Enforcer and pushes policy changes down.
It, by points:

• Integrates with the kube-apiserver
• Uses consul to sync Controllers
• Manages all Enforcers (Registration, Policy/Rule propagation, Event and Data collecting)
• Manages Scan Jobs to available Scanners and receives scan results from Scanner(s)
• Sync multiple Cluster through Federation (REST API). Note that in addition to Syncing Rules and Status, the primary Controller may also be syncing scan results to each of the downstream remote Clusters, which could be a significant dataset

Enforcer -> The Enforcer performs all runtime Security Functions such as Scanning, Network Inspection, and Process and File Protections for each Pod/Container and the Host. One Enforcer should be deployed on each Node (Host), e.g. as a DaemonSet.
Scanner -> The Scanner is a Container that performs the Vulnerability and Compliance Scanning for Images, Containers and Nodes. It is typically deployed as a ReplicaSet and can be scaled up to as many parallel Scanners as desired to increase the scanning performance. The Controller assigns scanning Jobs to each available Scanner in a round-robin fashion until all scans are completed. The Scanner also contains the latest CVE database and is updated regularly by NeuVector.

Below is the description of the various possible and recommended Design Patterns:

1. SIMPLE DEPLOYMENT - Deployment where Kubernetes can place the NeuVector containers on any node, in this case, worker nodes, unless scheduling to the Master has also been enabled. In a public cloud-managed service like EKS, AKS, etc., NeuVector can only run on worker nodes, the most common deployment.
2. MASTER NODE CONTROL PLANE - NeuVector control plane containers and scanners are placed on the Master nodes through taints/tolerations or labels. Master nodes can be appropriately sized for control plane requirements. <- This mode is recommended only and exclusively for TEST/DEVELOPMENT environments.
3. DEDICATED CONTROL PLANE - Similar to 2, but nodes are selected to run the NeuVector control plane and scanners. These could be worker nodes and are appropriately sized. Note that one NeuVector node has workloads allowed to run on it as well, whereas the others don’t allow workloads.
4. MASTER CONTROL PLANE (NO ENFORCER) - NeuVector control plane and scanners run on Master or dedicated nodes, but no Enforcer is deployed since no workloads run on them. In this case, the system containers running on the master node are not monitored by NeuVector, which is similar to a public cloud case.

Going into more detail about the CPU/Memory sizing, the official documentation suggests 1 vCPU and 1 GB of Memory for each component, but If you look at the Chart's default values.yaml (lines 539-546), the suggested resources are double those indicated in the documentation.
A good idea might be to not set limits in the STAGING environment, monitor resource usage and use peaks as baselines to assign limits in higher environments, always paying attention that the memory assigned to the Scanner is at least 0.5 GB larger than the largest image it will have to process.

PS: The Chart will allow you to customise each component's sizing globally and individually.

Unfortunately, we don't have a scientific formula to help you find the correct sizing for your installation, but we can share the results of some tests we did.
We tested a Cluster with 100 Nodes and 1000 workloads, noting that in this case, it was necessary to add more resources than suggested above and dedicate 3 Nodes (1 per Controller).
In another test with several Nodes (between 500 and 1000), we used 3 Controller Nodes with 32GB of Memory and 8 CPU cores each.
The more the Cluster grows, the more it will be necessary to monitor the use of the Network, as the Enforcers (which will be 1 per Node) will update the security policies with each change made to the various Security Functions.

We noticed another increase in CPU/Memory/Network usage with a Multi-Cluster Federation with 10 remote Clusters.

References:

• https://open-docs.neuvector.com/basics/requirements#performance-and-scaling
• https://open-docs.neuvector.com/basics/requirements#throughput
• https://open-docs.neuvector.com/basics/requirements#latency
• https://open-docs.neuvector.com/deploying/production/NV_Onboarding_5.0.pdf
• https://open-docs.neuvector.com/deploying/production/details#enforcer
• https://open-docs.neuvector.com/policy/dlp#applying-dlpwaf-sensors-to-container-groups
• https://open-docs.neuvector.com/scanning/scanners#increase-scanner-scalability-with-multiple-scanners
• https://open-docs.neuvector.com/scanning/scanners#performance-planning
• https://open-docs.neuvector.com/troubleshooting/troubleshooting#ongoing-operation
• https://github.com/neuvector/neuvector-helm/blob/master/charts/core/values.yaml
• https://www.suse.com/support/kb/doc/?id=000020921