How Kubernetes Networking Works – The Basics

Kubernetes 101

Before I get into Kubernetes networking, here are the basic concepts to know about in Kubernetes. For those not familiar with Kubernetes, these are the objects and terms used in a Kubernetes deployment.

Kubernetes is a container orchestration and management tool for automating the deployment and monitoring of containers. Kubernetes is supported by all major container management platforms such as OpenShift, Docker EE, AWS EKS, Rancher and IBM Cloud. Here are some of the key things to know about Kubernetes:

• Master Node. This is the node which manages worker nodes in a cluster and controls the deployment of pods.
• Worker Node. These servers typically run the application containers (in Pods) and other Kubernetes components such as proxies.
• Service. A service functions as a proxy to replicated pods and service requests can be load balanced across pods.
• Pod. The basic object for deployments in Kubernetes. Each pod has its own IP address and can contain one or more containers.

Other important system components which Kubernetes deploys include the API Server, Kubelet, and etcd.

The Basics of Kubernetes Networking

In Kubernetes, every pod has its own routable IP address. Kubernetes networking – through the network plug-in that is required to install (e.g. Calico, Flannel, Weave…) takes care of routing all requests internally between hosts to the appropriate pod. External access is provided through a service, load balancer, or ingress controller, which Kubernetes routes to the appropriate pod.

Kubernetes networking uses iptables to control the network connections between pods (and between nodes), handling many of the networking and port forwarding rules. This way, clients do not need to keep track of IP addresses to connect to Kubernetes services. Also, port mapping is greatly simplified (and mostly eliminated) since each pod has its own IP address and its container can listen on its native port.

With all of this overlay networking being handled dynamically by Kubernetes, it is extremely difficult to monitor network traffic, much less secure it. Here is an example of how Kubernetes networking works.

The above diagram shows how a packet traverses between pods on different nodes. In this Kubernetes networking example the Calico CNI network plugin is used. Every network plugin has a different approach for how a Pod IP address is assigned (IPAM), how iptables rules and cross-node networking are configured, and how routing information is exchanged between the nodes.

1. Once the CNI network plugin receives a notification from Kubernetes that a container is deployed, it is responsible for assigning an IP address and configuring proper iptables and routing rules on the node.
2. Pod1 sends a packet to Pod2 either using Pod2’s IP or Pod2’s service IP as the destination. (Pod2’s IP is used in the diagram)
3. If the service IP is used, the kube-proxy performs load-balancing and DNAT, translates the destination IP to the remote Pod’s IP.
4. The routing table on the node determines where the packets should be routed.
   – If the destination is a local Pod on the same node, the packet is forwarded directly to the Pod’s interface.
   – Otherwise, the packet is forwarded to the proper interface depending on whether overlay networking or L3 routing mechanisms are employed by the network plugin.
   – In the above diagram, the packet is sent to the IPIP tunnel interface and encapsulated with an IPIP tunnel header.
5. When the packet reaches the destination node, the encapsulation is removed.
6. The routing table on the remote node routes the packets to the destination Pod, Pod2.

With all the routing, possible NAT, and encapsulation occurring and being managed by the network plug-in, it is extremely difficult to inspect and monitor network traffic for attacks and connection violations. Traditional network monitoring and firewall tools are blind to container traffic, and therefore can’t be used to provide security for Kubernetes networking.

For a complete guide to Kubernetes run-time security and how to secure the network, container, and Kubernetes systems, please see this article.