Kubernetes System Security – Protecting Against Kubelet Exploits

By Andson Tung

As critical as it is to protect application containers deployed by Kubernetes, it is just as critical to protect the Kubernetes system containers from attacks or from being used in an attack. In this post I’ll focus on one important Kubernetes security area – protecting the Kubelet, which manages the pods on a worker node. The recent Kubernetes exploit at Tesla where crypto mining software was installed on servers by exploiting a open Kubernetes console makes it critical to examine potential attack surfaces for all Kubernetes system containers.

The kubelet hack has been documented and discussed for some time. It is surprising that such access can be allowed for any unprivleged pod to execute commands on the kubelet and control the entire cluster.

As a general rule of thumb, all access to Kubernetes system containers such as the API server and kubelet should be restricted with process role based access controls (RBACs). This applies to external access as well as internal access. But it’s not always possible to catch every potential attack vector when using new technologies. While the Tesla exploit came from an external source, the Kubelet exploit can occur over internal communications.

For every container deployment, it is useful to have network visibility and protection to catch possible misconfigurations and exploits. For business critical applications, it’s a must-have. Below, I’ll show how NeuVector can be configured with simple rules to protect the kubelet.

First, create a Group ‘allhost’ in NeuVector that includes all hosts in the Kubernetes cluster, by defining the subnet for the cluster. Then create another Group ‘allnamespace’ that includes all namespaces. All these actions can be done through the console, API, or CLI.

I can then create a new Rule that restricts access from any namespace to any host, on the kubelet port 10250. This will make sure no compromised application pod can exploit the kubelet.

Testing this, try connecting to the kubelet from within any application pod, and running a command. If successful, this command could be any damaging exploit.

$kubectl exec -ti httpserver-pod-00-787d79b8f4-b6gcg -- curl -k -XPOST "https://10.1.5.83:10250/run/neuvector/httpserver-pod-00-787d79b8f4-xxxxx/httpserver-pod-00" -d "cmd=ls -la /"

You’ll see a response showing that NeuVector has blocked this connection.

$curl: (7) Failed to connect to 10.1.5.83 port 10250: Operation timed out

You can see in the console how NeuVector is able to detect and block such activity.

If it is preferred not to block such connections, the containers could be in Monitor mode instead of Protect, which will allow the connection through but generate a security alert.

The picture above shows the violations, one which was blocked, and one which was allowed but generated an alert.

Although this is a simple example of a default configuration which can be easily protected by proper access controls, it demonstrates the simple point that with any new technologies and deployments there can be new attack surfaces exposed by misconfiguration or compromise. Having a Kubernetes security solution like NeuVector helps to detect and prevent these occurrences.