How can I validate network policies within a Kubernetes cluster?

• A Kubernetes cluster v1.15.2+
• kubectl access to the cluster
• Network Plugin (CNI) which supports Network Policies

Traffic flowing inside a kubernetes cluster is non-isolated by default so that each pod can communicate with every other pod. In some environments there is a need to ensure the proper isolation or restriction of each application in differing namespaces. Network Policies handle this microservice network segmentation and meet this need by defining the control to other entities' IP addresses (on OSI Layer 3) or network ports (on OSI Layer 4).

Access for Pod communication with other entities are identified with Network Policies and defined by other pods, the relevant namespaces, and IP CIDR Blocks. Pods and Namespaces are specified using a selector, like app=example .

Once the Network Policies are defined, how can a Kubernetes administrator test and validate them?

Illuminatio is an open-source project written in Python3 by Inovex. It can run standalone, and is also available as a docker container. It creates test-cases for both the network policies and their inverse rules, generates an illuminatio-runner daemonset, tests all the cases against the defined network policies, and reports back on success or failure for each rule and inverted-rule. Illuminatio can use the current kubectl config for cluster access while working in the shell session, or designate the config file with the optional --kubeconfig flag.

Basic Usage

Assuming the Kubernetes admin has some network policies to test, the tool is very easy to use. It has three verbs to choose from, "clean", "generate" and, "run". The generate verb will only generate the tests, while clean removes them and run performs the test. Most users will want to use illuminatio clean run to start fresh, run the generated tests and report on their success. The results are also written to a configmap.

The following are some common examples of Network Policies, and how Illuminatio can assist with validation. Examples are taken from this network policies recipes github repo, and applied to a kubernetes cluster, in the default namespace. Validation is performed with Illuminatio instead of a temporary pod.

Deny Traffic to an application

Save this file as web-deny-all.yaml and then apply the network policy with kubectl -f web-deny-all.yaml. Notice it is deploying to the default namespace. Prepare the pod for this example with a selector of app=web.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-deny-all
spec:
  podSelector:
    matchLabels:
      app: web
  ingress: []

kubectl run --generator=run-pod/v1 web --image=nginx --labels app=web --expose --port 80

Show the current network policies, then run cases for all of them. Illuminatio will deploy a deamonset and run all the test cases, any passing tests show "success" in the last column of the report. Note: success indicates the test was successful, even if testing a connection denial.

$ kubectl get netpol
NAME           POD-SELECTOR   AGE
web-deny-all   app=web        13m

$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Generated 1 cases in 0.0616 seconds

FROM             TO               PORT
default:app=web  default:app=web  -*  

Ensure that Pods of DaemonSet illuminatio-runner are ready

Finished running 1 tests in 7.1175 seconds
FROM             TO               PORT  RESULT   
default:app=web  default:app=web  -*    success

Limit Traffic to an application

Allow app=bookstore pods to communicate with only other app=bookstore pods.

kubectl run --generator=run-pod/v1 apiserver --image=nginx --labels app=bookstore,role=api --expose --port 80

Save the following as api-allow.yaml and issue kubectl apply -f api-allow.yaml.
Network policies are accumulative.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: api-allow
spec:
  podSelector:
    matchLabels:
      app: bookstore
      role: api
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: bookstore

$ kubectl apply -f api-allow.yaml 
networkpolicy.networking.k8s.io/api-allow created

$ kubectl get netpol
NAME           POD-SELECTOR             AGE
api-allow      app=bookstore,role=api   5s
web-deny-all   app=web                  18m

$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Generated 5 cases in 0.0594 seconds

FROM                                                             TO                              PORT
illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*  
illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*  
default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*  
default:app=web                                                  default:app=web                 -*  
default:app=bookstore                                            default:app=bookstore,role=api  *   

Ensure that Pods of DaemonSet illuminatio-runner are ready

Finished running 5 tests in 13.2368 seconds
FROM                                                             TO                              PORT  RESULT 

illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*    success
illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*    success
default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*    success
default:app=web                                                  default:app=web                 -*    success
default:app=bookstore                                            default:app=bookstore,role=api  *     success

Allow whitelisted traffic for app=web

This policy will whitelist the app=web pods from the first example, with a new web-allow-all.yaml file. This Network Policy also voids the first example, by allowing all traffic. Because the traffic connections are allowed, Illuminatio recognizes this and avoids generating the negative (inverted) test cases.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-all
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - {}

$ kubectl apply -f web-allow-all.yaml 
networkpolicy.networking.k8s.io/web-allow-all created

$ kubectl get netpol
NAME            POD-SELECTOR             AGE
api-allow       app=bookstore,role=api   20m
web-allow-all   app=web                  32s
web-deny-all    app=web                  39m

$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Not generating negative tests for host ClusterHost(namespace=default, podLabels={'app': 'web'})as all connecti
ons to it are allowed
Generated 5 cases in 0.0551 seconds

FROM                                                             TO                              PORT
illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*  
illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*  
default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*  
default:app=bookstore                                            default:app=bookstore,role=api  *   
*:*                                                              default:app=web                 *   

Ensure that Pods of DaemonSet illuminatio-runner are ready

Finished running 5 tests in 13.4065 seconds
FROM                                                             TO                              PORT  RESULT 

illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*    success
illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*    success
default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*    success
default:app=bookstore                                            default:app=bookstore,role=api  *     success
*:*                                                              default:app=web                 *     success

Limit access to a Namespace

This policy will deny all traffic from other namespaces, limiting to just the current namespace. In other words, the secondary namespace allows connections internally, denying any from the default namespace in previous examples. Note how Illuimnatio tests all network policies, cluster-wide in all namespaces.

kubectl create namespace secondary

kubectl run --generator=run-pod/v1 web --namespace secondary --image=nginx \
    --labels=app=web --expose --port 80

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: secondary
  name: deny-from-other-namespaces
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}

$ kubectl apply -f deny-from-other-namespaces.yaml
networkpolicy "deny-from-other-namespaces" created"

$ kubectl get netpol -n secondary
NAME                         POD-SELECTOR   AGE
deny-from-other-namespaces   <none>         7s

$ kubectl get netpol -n default
NAME            POD-SELECTOR             AGE
api-allow       app=bookstore,role=api   44m
web-allow-all   app=web                  24m
web-deny-all    app=web                  63m

$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Not generating negative tests for host ClusterHost(namespace=default, podLabels={'app': 'web'})as all connecti
ons to it are allowed
Generated 7 cases in 0.0621 seconds

FROM                                                             TO                              PORT
illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*  
default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*  
illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*  
illuminatio-inverted-secondary:*                                 secondary:*                     -*  
default:app=bookstore                                            default:app=bookstore,role=api  *   
*:*                                                              default:app=web                 *   
secondary:*                                                      secondary:*                     *   

Ensure that Pods of DaemonSet illuminatio-runner are ready

Finished running 7 tests in 13.2361 seconds
FROM                                                             TO                              PORT  RESULT 

illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*    success
default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*    success
illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*    success
illuminatio-inverted-secondary:*                                 secondary:*                     -*    success
default:app=bookstore                                            default:app=bookstore,role=api  *     success
*:*                                                              default:app=web                 *     success
secondary:*                                                      secondary:*                     *     success

Allow All Traffic from a certain Namespace

In this example, there are two namespaces, dev with purpose=testing and prod with purpose=production. The default namespace should allow connections from production but not dev. This is convenient for establishing policies along namespace boundaries. All previous network policies have been removed for this scenario.

kubectl run --generator=run-pod/v1 web --image=nginx \
    --labels=app=web --expose --port 80

kubectl create namespace dev
kubectl label namespace/dev purpose=testing

kubectl create namespace prod
kubectl label namespace/prod purpose=production

The contents of the web-allow-prod.yaml file.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-prod
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          purpose: production

$ kubectl apply -f web-allow-prod.yaml
networkpolicy "web-allow-prod" created

$ kubectl get netpol -A
NAMESPACE   NAME             POD-SELECTOR   AGE
default     web-allow-prod   app=web        44s

$ illuminatio clean run
Starting cleaning resources with policies ['on-request', 'always']
Finished cleanUp
Starting test generation and run.
Generated 2 cases in 0.0645 seconds

FROM                                       TO               PORT
illuminatio-inverted-purpose=production:*  default:app=web  -*  
purpose=production:*                       default:app=web  *   

Ensure that Pods of DaemonSet illuminatio-runner are ready

Finished running 2 tests in 7.1767 seconds
FROM                                       TO               PORT  RESULT   
illuminatio-inverted-purpose=production:*  default:app=web  -*    success
purpose=production:*                       default:app=web  *     success

To view the results of the test programmatically, check the configmap for the illuminatio namespace, before performing another "clean" operation.

$ kubectl get cm -n illuminatio
NAME                               DATA   AGE
illuminatio-cases-cfgmap           1      45s
illuminatio-runner-s87rw-results   2      40s
illuminatio-runner-z52gb-results   2      41s

$ kubectl get cm -n illuminatio illuminatio-runner-s87rw-results -o yaml
apiVersion: v1
data:
  results: |
    illuminatio-inverted-purposeproduction:illuminatio-dummy-nqtc7:
      10.43.168.221:
        '-80':
          nmap-state: filtered
          string: 'Test 10.43.168.221:-80 succeeded

            Couldn''t reach 10.43.168.221 on port 80. Expected target to not be reachable'
          success: true
    prod:illuminatio-dummy-tc5v9:
      10.43.168.221:
        '80':
          nmap-state: open
          string: 'Test 10.43.168.221:80 succeeded

            Could reach 10.43.168.221 on port 80. Expected target to be reachable'
          success: true
  runtimes: |
    overall: error
    tests:
      illuminatio-inverted-purposeproduction:illuminatio-dummy-nqtc7:
        10.43.168.221: 2.1185858249664307
      prod:illuminatio-dummy-tc5v9:
        10.43.168.221: 0.2853882312774658
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-23T21:47:53Z"
  labels:
    illuminatio-cleanup: always
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:results: {}
        f:runtimes: {}
      f:metadata:
        f:labels:
          .: {}
          f:illuminatio-cleanup: {}
    manager: Swagger-Codegen
    operation: Update
    time: "2020-11-23T21:47:53Z"
  name: illuminatio-runner-s87rw-results
  namespace: illuminatio
  resourceVersion: "906614"
  selfLink: /api/v1/namespaces/illuminatio/configmaps/illuminatio-runner-s87rw-results
  uid: 2c2f7434-d1ee-49c0-b77d-c11b7848f4da

Further Reading and Other Useful Links

• Securing Kubernetes Cluster Networking
• Example Network Policy Recipes, ahmetb/kubernetes-network-policy-recipes
• Inovex Illuminatio, Kubernetes Network Policy Validator
• Inovex/Illuminatio GitHub Project Page