4 questions you aren’t asking about Kubernetes security

Buying security products can be overwhelming. It can be tempting to purchase them the way you purchase insurance: They all feel the same, so you want to sort it out fast, get the broadest coverage for the cheapest price and get back to your life. It might seem low risk and cost efficient  to do it this way, but it’s not. To make matters worse, you won’t realize that’s “not the way” until there is an incident that wasn’t covered and costs X-times more than the insurance/security would have ever cost.

We talk to many new customers who turn to us when – during or after an incident – they realize their needs haven’t been met. This happens because at some point a sales person gave their own customer  the impression there would be  all this protection in place… until the incident that revealed they were not covered, and were not protected.  You may have first-hand experience with this exact situation. Infuriating, isn’t it?

If anyone tells you that you can get a bargain for a security product and get the most security coverage for a small cost, it means they are selling you insurance and not actual security.  Insurance protects after an incident occurs.  Security is about preventing an incident from occurring in the first place, and needs to be tailored to your unique environment and risk to be effective.

In this article, I’ll walk you through the steps of how to make sure that you’re not wasting time and resources and are asking the right questions at the right time. Especially – asking those questions you didn’t know you had.

1. Do we need observability or network inspection?

Teams operating  Kubernetes often use other products like a service mesh expecting that network  security and observability are bonus features of service discovery. This assumption  leads to believing  that ‘observability’ means ‘complete observability’. However, a service mesh will only be able to identify  what was  defined, and will not provide visibility to any  anomalous network traffic you should need  to see. One of NeuVector’s key differentiators is that our solution enables you to see all network activity – defined or undefined – which is kinda-sorta important if you need to be secure.

The ability to have complete  visibility into every layer of  your network is perhaps the most critical part of run-time container security.  Layer 7 inspection  allows you to inspect container network traffic and learn specifically how each application service communicates with other application services. With complete visibility into your network traffic, you can also  stop attacks before they reach an application or workload, and prevent data breaches by exploited applications that may send data out over the network.

Questions to ask:

• Are you confident you have visibility into all network traffic beyond what was defined?
• How do you differentiate normal vs anomalous traffic?
• Does a service mesh provide observability or prevent unknown network traffic?
• Are  system calls the best kill-chain point for network activity, or is the network the best kill-chain point for network activity?

When you get answers to these questions, you can begin to form a better picture of observability versus network inspection. They might sound similar at first, but the difference is in the details.

2.  What is it that you do NOT get with Kubernetes or service mesh?

Focusing too much on the features and functionality a tool provides may lead to overlooking what it does not provide. And what it does not provide, might be the exact thing you need.

Imagine you are seeing an increased load on services related to sensitive data stores, but you can’t pinpoint the cause. It doesn’t appear to be related to anything you’ve defined, and you start to consider whether someone or something may have access or is downloading data.

You would like to find out, but you can’t  see it because you have observability only for the network elements you defined. To solve this problem you need to have observability into things that haven’t been defined. Imagine if someone inexplicably had a password giving them access into your infrastructure.  They didn’t have to hack a firewall, or inject a SQL command into an application entry field.  Nope.  The perpetrator – possibly an internal actor – was able to shell into running containers and scan for open ports on other containers, maybe install a root kit, cloaks or a back-door to hide future activity.  How would you know they were there or what they may have changed?  How are you going to detect anomalous behavior if the behavior appears legitimate?   

Questions to ask:

• The networking within a pod is wide open. What are the potential negative consequences if one or all of the containers in that pod are compromised?
• If pod networking is wide open – what do we need to do about it?
• Have you identified what you’re going to protect?
• What about the things you haven’t identified? If you didn’t define them, you didn’t secure them. Should they be secured?

3. I already have a web application firewall. Why do I need container security?

Web Application Firewalls  cannot identify or block  east-west network traffic within your containerized environment, full stop. The way deployments and services are defined is like a road on a map. You can define where you want the traffic to go but you won’t be  able to prevent a bad actor from driving into a ditch or off-road or wherever they want to go. Your  application will follow the road because you defined the road. But a bad actor getting access inside the environment will try to go anywhere – defined or not, and that is something a WAF cannot prevent.

Questions to ask:

• How do I know that I have visibility into all traffic beyond what was specifically defined?
• How do I stop traffic that was not specifically defined?
• How can I validate that the layer 7 protocol between containers isn’t being used for a tunneling attack?
• Can I capture packets within my cluster to see what that traffic is carrying

4. What are security tool limitations in containerized environments?

It can be tricky to know the limitations of a particular tool you’ve invested in before you’re already using it in your unique environment.  You might not know you’re missing something until there’s a problem. For example, you know you need to be PCI compliant, and have always been able to meet Data Loss Prevention (DLP) requirements in legacy environments.  But now your environment is transitioning to new containerized microservices and the legacy DLP approach won’t pass an audit. How would  you know to  be looking for DLP for containers  if you didn’t know you were missing it? And how would you know you have a problem with your existing product if you didn’t know it wasn’t doing something you need it to do?

Consider a common feature to many security tools: “Identifying security drift”.  The requirement itself is so common that in a review of RFP’s and RFI’s received by NeuVector in 2020, “Identifying Security Drift” was a requirement in over 85% of those submissions.  Not a single RFI or RFP included “Prevent Security Drift” as a requirement. 0%.  Wouldn’t you rather prevent security drift vs just identifying security drift? Of course. Unfortunately, many vendors claim this ability, but in fact are only able to detect activity once the attacker is in the container. 

Either your security is based on reacting to a situation that has already made it to your kernel, or you are preventing a situation from getting to your kernel.  At NeuVector, we don’t need to focus  on identifying drift, because we are the only solution that can prevent it from happening. NeuVector stops anomalous network activity or processes that have not been seen previously, without intervention from you or your team. You won’t have to approve or reject anything.  NeuVector’s ability to learn the specific network and process behaviors of your unique environment are automatic and as fine-grained as possible using Layer 7 network policies and protocol verification.

Questions to ask of your current Kubernetes security provider:

• Can you perform network packet capture inside a containerized environment?
• Do you automatically create  network security policies at layer 7?
• Do you validate the layer 7 protocol to prevent protocol tunneling attacks?
• Are you able to order your network security rules?
• Do your network security rules  provide  for both allow and deny rules?
• Are you able to automatically generate network, process and file security policies that correspond exactly to the unique behavior of your service or application?

You may have noticed we’ve suggested asking about packet capture more than once. Packet capture is important because it’s related to true network visibility and the ability to allow or block traffic at the network level.  Kernel level sys-call filtering cannot block or capture network traffic. Moreover, by the time the kernel reports the network traffic occured, it’s too late.  Being able to inspect, validate, capture or even block network packets means you have total control of the first and most important element of the attack kill-chain: The Network. The ability to capture packets proves better observability and better security controls.

Conclusion

We’ve summarized a few of the questions critical to the security of containerized environments, but the reality is that every environment is unique, with its own risk profile.  It is not uncommon to find yourself playing whack-a-mole for every vulnerability or security setting trying to assure an environment is as secure as possible.  While many technologies are excellent for one thing – eBPF for tracing or Web Application Firewalls – does not mean they are the best choice for securing network elements in containerized environments.  The most important question above all others is: If  your environment becomes compromised, how will you find out and how long will it take before you find out? ¯_(ツ)_/¯

If you are evaluating your license with your current container security tool and you feel like you have everything protected that needs to be protected, maybe you’re not missing anything.

But before you decide to stay with what they offer, may we suggest that you take 15 minutes to install NeuVector and see what you may be missing? .