SUSE Support

Here When You Need Us

How can I validate network policies within a Kubernetes cluster?

This document (000020041) is provided subject to the disclaimer at the end of this document.

Environment

  • A Kubernetes cluster v1.15.2+
  • kubectl access to the cluster
  • Network Plugin (CNI) which supports Network Policies

Situation

Traffic flowing inside a kubernetes cluster is non-isolated by default so that each pod can communicate with every other pod. In some environments there is a need to ensure the proper isolation or restriction of each application in differing namespaces. Network Policies handle this microservice network segmentation and meet this need by defining the control to other entities' IP addresses (on OSI Layer 3) or network ports (on OSI Layer 4).

Access for Pod communication with other entities are identified with Network Policies and defined by other pods, the relevant namespaces, and IP CIDR Blocks. Pods and Namespaces are specified using a selector, like app=example .

Once the Network Policies are defined, how can a Kubernetes administrator test and validate them?

    Resolution

    Illuminatio is an open-source project written in Python3 by Inovex. It can run standalone, and is also available as a docker container. It creates test-cases for both the network policies and their inverse rules, generates an illuminatio-runner daemonset, tests all the cases against the defined network policies, and reports back on success or failure for each rule and inverted-rule. Illuminatio can use the current kubectl config for cluster access while working in the shell session, or designate the config file with the optional --kubeconfig flag.

    Basic Usage

    Assuming the Kubernetes admin has some network policies to test, the tool is very easy to use. It has three verbs to choose from, "clean", "generate" and, "run". The generate verb will only generate the tests, while clean removes them and run performs the test. Most users will want to use illuminatio clean run to start fresh, run the generated tests and report on their success. The results are also written to a configmap.

    The following are some common examples of Network Policies, and how Illuminatio can assist with validation. Examples are taken from this network policies recipes github repo, and applied to a kubernetes cluster, in the default namespace. Validation is performed with Illuminatio instead of a temporary pod.

    Deny Traffic to an application

    Save this file as web-deny-all.yaml and then apply the network policy with kubectl -f web-deny-all.yaml. Notice it is deploying to the default namespace. Prepare the pod for this example with a selector of app=web.

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-deny-all
    spec:
      podSelector:
        matchLabels:
          app: web
      ingress: []
    kubectl run --generator=run-pod/v1 web --image=nginx --labels app=web --expose --port 80

    Show the current network policies, then run cases for all of them. Illuminatio will deploy a deamonset and run all the test cases, any passing tests show "success" in the last column of the report. Note: success indicates the test was successful, even if testing a connection denial.

    $ kubectl get netpol
    NAME           POD-SELECTOR   AGE
    web-deny-all   app=web        13m
    
    $ illuminatio clean run
    Starting cleaning resources with policies ['on-request', 'always']
    Finished cleanUp
    Starting test generation and run.
    Generated 1 cases in 0.0616 seconds
    
    FROM             TO               PORT
    default:app=web  default:app=web  -*  
    
    Ensure that Pods of DaemonSet illuminatio-runner are ready
    
    Finished running 1 tests in 7.1175 seconds
    FROM             TO               PORT  RESULT   
    default:app=web  default:app=web  -*    success
    
    Limit Traffic to an application

    Allow app=bookstore pods to communicate with only other app=bookstore pods.

    kubectl run --generator=run-pod/v1 apiserver --image=nginx --labels app=bookstore,role=api --expose --port 80

    Save the following as api-allow.yaml and issue kubectl apply -f api-allow.yaml.
    Network policies are accumulative.

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: api-allow
    spec:
      podSelector:
        matchLabels:
          app: bookstore
          role: api
      ingress:
      - from:
          - podSelector:
              matchLabels:
                app: bookstore
    $ kubectl apply -f api-allow.yaml 
    networkpolicy.networking.k8s.io/api-allow created
    
    $ kubectl get netpol
    NAME           POD-SELECTOR             AGE
    api-allow      app=bookstore,role=api   5s
    web-deny-all   app=web                  18m
    
    $ illuminatio clean run
    Starting cleaning resources with policies ['on-request', 'always']
    Finished cleanUp
    Starting test generation and run.
    Generated 5 cases in 0.0594 seconds
    
    FROM                                                             TO                              PORT
    illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*  
    illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*  
    default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*  
    default:app=web                                                  default:app=web                 -*  
    default:app=bookstore                                            default:app=bookstore,role=api  *   
    
    Ensure that Pods of DaemonSet illuminatio-runner are ready
    
    Finished running 5 tests in 13.2368 seconds
    FROM                                                             TO                              PORT  RESULT 
    
    illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*    success
    illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*    success
    default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*    success
    default:app=web                                                  default:app=web                 -*    success
    default:app=bookstore                                            default:app=bookstore,role=api  *     success
    
    Allow whitelisted traffic for app=web

    This policy will whitelist the app=web pods from the first example, with a new web-allow-all.yaml file. This Network Policy also voids the first example, by allowing all traffic. Because the traffic connections are allowed, Illuminatio recognizes this and avoids generating the negative (inverted) test cases.

     

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-all
      namespace: default
    spec:
      podSelector:
        matchLabels:
          app: web
      ingress:
      - {}
    
    $ kubectl apply -f web-allow-all.yaml 
    networkpolicy.networking.k8s.io/web-allow-all created
    
    $ kubectl get netpol
    NAME            POD-SELECTOR             AGE
    api-allow       app=bookstore,role=api   20m
    web-allow-all   app=web                  32s
    web-deny-all    app=web                  39m
    
    $ illuminatio clean run
    Starting cleaning resources with policies ['on-request', 'always']
    Finished cleanUp
    Starting test generation and run.
    Not generating negative tests for host ClusterHost(namespace=default, podLabels={'app': 'web'})as all connecti
    ons to it are allowed
    Generated 5 cases in 0.0551 seconds
    
    FROM                                                             TO                              PORT
    illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*  
    illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*  
    default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*  
    default:app=bookstore                                            default:app=bookstore,role=api  *   
    *:*                                                              default:app=web                 *   
    
    Ensure that Pods of DaemonSet illuminatio-runner are ready
    
    Finished running 5 tests in 13.4065 seconds
    FROM                                                             TO                              PORT  RESULT 
    
    illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*    success
    illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*    success
    default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*    success
    default:app=bookstore                                            default:app=bookstore,role=api  *     success
    *:*                                                              default:app=web                 *     success
    
    Limit access to a Namespace

    This policy will deny all traffic from other namespaces, limiting to just the current namespace. In other words, the secondary namespace allows connections internally, denying any from the default namespace in previous examples. Note how Illuimnatio tests all network policies, cluster-wide in all namespaces.

     

    kubectl create namespace secondary
    
    kubectl run --generator=run-pod/v1 web --namespace secondary --image=nginx \
        --labels=app=web --expose --port 80
    
    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      namespace: secondary
      name: deny-from-other-namespaces
    spec:
      podSelector:
        matchLabels:
      ingress:
      - from:
        - podSelector: {}
    $ kubectl apply -f deny-from-other-namespaces.yaml
    networkpolicy "deny-from-other-namespaces" created"
    
    $ kubectl get netpol -n secondary
    NAME                         POD-SELECTOR   AGE
    deny-from-other-namespaces   <none>         7s
    
    $ kubectl get netpol -n default
    NAME            POD-SELECTOR             AGE
    api-allow       app=bookstore,role=api   44m
    web-allow-all   app=web                  24m
    web-deny-all    app=web                  63m
    
    $ illuminatio clean run
    Starting cleaning resources with policies ['on-request', 'always']
    Finished cleanUp
    Starting test generation and run.
    Not generating negative tests for host ClusterHost(namespace=default, podLabels={'app': 'web'})as all connecti
    ons to it are allowed
    Generated 7 cases in 0.0621 seconds
    
    FROM                                                             TO                              PORT
    illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*  
    default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*  
    illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*  
    illuminatio-inverted-secondary:*                                 secondary:*                     -*  
    default:app=bookstore                                            default:app=bookstore,role=api  *   
    *:*                                                              default:app=web                 *   
    secondary:*                                                      secondary:*                     *   
    
    Ensure that Pods of DaemonSet illuminatio-runner are ready
    
    Finished running 7 tests in 13.2361 seconds
    FROM                                                             TO                              PORT  RESULT 
    
    illuminatio-inverted-default:illuminatio-inverted-app=bookstore  default:app=bookstore,role=api  -*    success
    default:illuminatio-inverted-app=bookstore                       default:app=bookstore,role=api  -*    success
    illuminatio-inverted-default:app=bookstore                       default:app=bookstore,role=api  -*    success
    illuminatio-inverted-secondary:*                                 secondary:*                     -*    success
    default:app=bookstore                                            default:app=bookstore,role=api  *     success
    *:*                                                              default:app=web                 *     success
    secondary:*                                                      secondary:*                     *     success
    
    Allow All Traffic from a certain Namespace

    In this example, there are two namespaces, dev with purpose=testing and prod with purpose=production. The default namespace should allow connections from production but not dev. This is convenient for establishing policies along namespace boundaries. All previous network policies have been removed for this scenario.

    kubectl run --generator=run-pod/v1 web --image=nginx \
        --labels=app=web --expose --port 80
    
    kubectl create namespace dev
    kubectl label namespace/dev purpose=testing
    
    kubectl create namespace prod
    kubectl label namespace/prod purpose=production

    The contents of the web-allow-prod.yaml file.

     

    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: web-allow-prod
    spec:
      podSelector:
        matchLabels:
          app: web
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              purpose: production
    $ kubectl apply -f web-allow-prod.yaml
    networkpolicy "web-allow-prod" created
    
    $ kubectl get netpol -A
    NAMESPACE   NAME             POD-SELECTOR   AGE
    default     web-allow-prod   app=web        44s
    
    $ illuminatio clean run
    Starting cleaning resources with policies ['on-request', 'always']
    Finished cleanUp
    Starting test generation and run.
    Generated 2 cases in 0.0645 seconds
    
    FROM                                       TO               PORT
    illuminatio-inverted-purpose=production:*  default:app=web  -*  
    purpose=production:*                       default:app=web  *   
    
    Ensure that Pods of DaemonSet illuminatio-runner are ready
    
    Finished running 2 tests in 7.1767 seconds
    FROM                                       TO               PORT  RESULT   
    illuminatio-inverted-purpose=production:*  default:app=web  -*    success
    purpose=production:*                       default:app=web  *     success
    

    To view the results of the test programmatically, check the configmap for the illuminatio namespace, before performing another "clean" operation.

    $ kubectl get cm -n illuminatio
    NAME                               DATA   AGE
    illuminatio-cases-cfgmap           1      45s
    illuminatio-runner-s87rw-results   2      40s
    illuminatio-runner-z52gb-results   2      41s
    
    $ kubectl get cm -n illuminatio illuminatio-runner-s87rw-results -o yaml
    apiVersion: v1
    data:
      results: |
        illuminatio-inverted-purposeproduction:illuminatio-dummy-nqtc7:
          10.43.168.221:
            '-80':
              nmap-state: filtered
              string: 'Test 10.43.168.221:-80 succeeded
    
                Couldn''t reach 10.43.168.221 on port 80. Expected target to not be reachable'
              success: true
        prod:illuminatio-dummy-tc5v9:
          10.43.168.221:
            '80':
              nmap-state: open
              string: 'Test 10.43.168.221:80 succeeded
    
                Could reach 10.43.168.221 on port 80. Expected target to be reachable'
              success: true
      runtimes: |
        overall: error
        tests:
          illuminatio-inverted-purposeproduction:illuminatio-dummy-nqtc7:
            10.43.168.221: 2.1185858249664307
          prod:illuminatio-dummy-tc5v9:
            10.43.168.221: 0.2853882312774658
    kind: ConfigMap
    metadata:
      creationTimestamp: "2020-11-23T21:47:53Z"
      labels:
        illuminatio-cleanup: always
      managedFields:
      - apiVersion: v1
        fieldsType: FieldsV1
        fieldsV1:
          f:data:
            .: {}
            f:results: {}
            f:runtimes: {}
          f:metadata:
            f:labels:
              .: {}
              f:illuminatio-cleanup: {}
        manager: Swagger-Codegen
        operation: Update
        time: "2020-11-23T21:47:53Z"
      name: illuminatio-runner-s87rw-results
      namespace: illuminatio
      resourceVersion: "906614"
      selfLink: /api/v1/namespaces/illuminatio/configmaps/illuminatio-runner-s87rw-results
      uid: 2c2f7434-d1ee-49c0-b77d-c11b7848f4da

    Additional Information

    Disclaimer

    This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

    • Document ID:000020041
    • Creation Date: 06-May-2021
    • Modified Date:04-Apr-2024
      • SUSE Rancher

    < Back to Support Search

    For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

    tick icon

    SUSE Support Forums

    Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

    tick icon

    Support Resources

    Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

    tick icon

    Open an Incident

    Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.