How to Protect Secrets in Containers Using DPI and DLP

Every cloud application and service utilizes a key (secret) to identify and authorize communications. Secrets are also used to authorize access to containerized applications which require a login. These credentials are widely used by public facing services as well as internal and external REST API’s everywhere. Examples include the AWS IAM access key, Google API access token, Twitter API key, LinkedIn API ID, Facebook access token, Flickr API access token, OAuth Client Secrets and the list goes on.

Even in a well configured service environment, some of these secrets will be able to get authorization to access sensitive data because the service requires it to perform its functions. Examples are the ability to read database records, create new files or even delete data or files. So it should be obvious that these secrets should be carefully managed and stored, because any leakage or misuse of them can cause damaging data breaches or other security issues. Here are some real incidents involving compromised incidents:

On September 2017, DXC’s AWS private keys were compromised from a public GitHub space, ‘Unknown persons’ spin up 244 VMs at cost of $64k in just couple of days. “Various secure variables (cryptographic keys that allowed access to DXC procured Amazon Web Services resources) were hardcoded into a piece of work being shared between multiple teams and with the project architect.” Then on September 27, a member of the technical team created a personal space on the public Github, and the code was loaded to this unsecured repository that allowed individuals as yet unknown to access and use it. “Over a period of four days, the private keys were used to start 244 AWS virtual machines. The cost incurred was $64,000 (£48,799).”

In 2018, hackers found that an Uber engineer mistakenly left the credentials in a GitHub repository. This secret can be used to access an Amazon web server which was owned by Uber. The hacker accessed the server and downloaded more than a dozen files which included a backup file which contained millions of Uber customer data records. Hackers were asking for six-figure payout. Since the breach was discovered, not only did Uber pay $148 million for the data breach settlement, but they also agreed to adopt a comprehensive security auditing practice. This extended into all aspects of the company’s operations, including the development team, DevOps, IT and security groups. Each function and process was required to be audited and adjusted for compliance. Uber’s business was impacted and their IPO was delayed.

Solutions for Secrets Management

Secrets management and security solutions to provide this are not new. Various solutions are available in the market. There are also types of open source tools available. For example,

• Open Source git-secrets from AWSLabs  scans commits and merges to prevent a developer from committing passwords and other sensitive information to a git repository.
• detect-secrets is a tool to detect and prevent secrets in code. It was built for preventing new secrets from entering the code base, detecting if such preventions are explicitly bypassed, and providing a checklist of secrets to roll, and migrate off to a more secure storage.
• HashiCorp Vault is one solution to secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.
• Kubernetes as the de facto container orchestration platform also has built-in capabilities of secrets management. Kubernetes Secrets let you store and manage sensitive information, such as passwords, OAuth tokens, and ssh keys. Storing confidential information in a Secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image.
• In Red Hat OpenShift, which is a popular enterprise-grade Kubernetes platform, secrets management functions are available.

Secrets Auditing Solutions

In general, today most secrets auditing security tools can help find secrets from source code repositories, container images or in CI/CD pipeline. Container platforms and orchestrators will help encrypt, distribute and manage secrets when services are deployed. Secrets management is part of the control plane and secrets are secured together within system containers/services by default. These are typically secure enough to serve standard secrets protection needs.

But for highly sensitive workloads, for example Uber’s customer database backend service, standard image scanning, data encryption, secured secret stores and secret distribution is not enough. For these use cases, additional “defense in depth” protection is needed.

Using NeuVector to Audit and Find Secrets

Here is an example of how NeuVector can help to find secrets in running pods to protect mission critical secrets in the runtime production environment:

• NeuVector is fully integrated with Kubernetes platforms, it will leverage the Kubernetes platform to manage and distribute the application secrets. So there’s no 3^rd party plugins or deployments needed.
• In Policy -> Group, define a “CUSTOM CHECK” for the web server application which will access the backend database and storage services. This web server is reading and writing sensitive data by design. Adding a custom check script which will scan the web server pods for any AWS secret access keys.
  
• As soon as the new custom check is saved, NeuVector will automatically scan the running containers in this group. It reports and takes action accordingly when an AWS access secret is detected.
  
• At same time, a security incident will be generated and logged, which will have all the detailed information to help fix the source of the violation.
  

NeuVector will scan for secrets at run time and at scale, even when services are scaling up or down. The NeuVector platform will automatically discover any changes and apply all security checks automatically. Response rules can also be applied to trigger post actions when finding a secrets in containers.

Note: This extension capability can be used to support scanning for any Regex pattern-based secrets, for example AWS, Facebook, LinkedIn, Twitter, Foursquare etc. Please contact NeuVector support if you are interested in various pattern examples.

Using NeuVector DPI/DLP to Inspect Network Transmission of Secrets

In addition to scanning for secrets, NeuVector can also apply unique container DPI/DLP (deep packet inspection/data leakage prevention) technology to monitor the secrets in use. Here is a detailed example of how to do this:

• Define a AWS secret key as a NeuVector DLP sensor.
  
• Apply this sensor to the services that which require AWS access. For example, this Nginx service will have access to AWS services and it will use AWS access keys.
  
• Just like that, the DLP rule will be applied and NeuVector will start monitoring the Nginx network communications. Whenever there’s an AWS secret being transferred in the network packets, NeuVector will trigger an alarm then take appropriate actions on it.
  

Note: the Regex patterns in these examples are for demo purposes only. Like similar IDS/IPS or scanning solutions, pattern based scanning may generate false positives. NeuVector recommends that users design the Regex patterns to achieve the best match of real key, in order to get best results. For example, use some digits from the actual secrets, or add keywords found in the keys etc., which can increase accuracy and efficiency.

NeuVector’s behavior based security protection is the foundation to protect any known or unknown security threats at runtime. In addition to those protections, these additional DPI/DLP capabilities provide the industry’s only “security in depth” protection for mission critical services.

When combined with the existing platform or 3^rd party secrets management security tools, secrets can be secured during the entire container lifecycle.

Increasing Cloud and Container Applications Require Secrets Protection

Since March of this year 2020, the coronavirus pandemic and “shelter in place” orders have caused a 30x growth of the daily Zoom meetings. With the popularity of Zoom and other remote access work applications growing, security practices are being constantly challenged. All these Cloud workloads and services require a strong “defense in depth” security platform to provide strong protections for production environments. And only NeuVector is able to inspect and block network transmissions that contain secrets which thieves are trying to steal.

Never let security issues slow down your business growth!