How to Protect Container Infrastructures Against the Malware “Doki”

By Fei Huang

Recently, security researchers from Intezer shared an interesting case study about malware targeting public Docker environments.

Security researchers have named this malware Doki. It is being downloaded and installed as a Linux backdoor. It is using the DynDNS service and a unique Domain Generation Algorithm (DGA) based on the Dogecoin cryptocurrency blockchain to find it’s controller in real time. Doki queries the Dogecoin API, uses SHA265 encryption and constructs a full URL address dynamically at runtime, which bypasses the tradition network security checks such as blacklist URL/IP lists. The report claims this new malware “Doki” hasn’t been detected by any of the 60 malware detection engines in VirusTotal since it was first analyzed on January 14, 2020. Some malware engines have since caught up; better late than never!

This does not come as a surprise in container security space. Containers and container infrastructures are the new modern “attack surface.” This new attack surface has already been exploited to launch more complex and faster attacks, such as the crypto mining attack at Tesla.  Traditional anti-malware solutions are signature based, which won’t be able to catch any unknown attacks. In addition, signature databases provide little protection against modern attacks, and require constant updating and monitoring.

Using this “Doki” malware as an example, let’s dive into some technical details to explain how the combination of good security hygiene and the NeuVector container security platform can address these security challenges efficiently and practically.

1. Doki starts from scanning the public or private network for a misconfigured Docker API port. Network port scanning scripts or tools will find the publicly accessible Docker ports and exploit them.
   » The CIS Benchmark for Docker scanning will help find misconfigured docker Daemons as early as possible. In addition, security rules of public cloud providers, such as AWS Security Groups, should be configured to block all unauthorized ingress with IP address and port rules to allow only trusted connections.
2. Once an open or weak Docker port has been found, attackers will call the Docker API “create” to download and launch a “normal” container which serves as the stepping stone to expand the attack. This container is typically downloaded from the public registry like DockerHub. It is a “clean” container image based on the Alpine operating system, with some popular network tools included, for example “curl” in this case. So none of the vulnerability scanning engines will complain about it.
   » The NeuVector run-time security system blocks all network and process activity in unauthorized containers which start running, even those which bypass the orchestrator and start running directly from the Docker run-time. Alerts will be triggered every time this rogue container attempts any activity.
3. Attackers have full control of this stepping stone container, and will be able to spawn other containers quickly to trigger some actual malicious behavior.
   » Malicious behavior in any container needs to be detected and blocked immediately. The NeuVector run-time security policies detect and block unauthorized processes, file access and network connections, and can be auto-learned and generated as ‘security as code’ to automate deployment.
4. The next step is to setup a command and control link. The ngrok service is used to provide secure tunnels. Attackers use unique URLs with very short lifetimes to download payloads quickly into the container’s filesystem (/tmpXXXXXX directory). It runs and quits very quickly to make it difficult for traditional firewalls and SIEM system to detect and respond in time. After a SIEM system collects the logs and by the time any alerts can be generated, this container is gone and there are no clues left in the runtime environment to investigate the attack.
   » A layer 7 container firewall is needed to be able to detect ngrok botnet connections. In-line blocking with deep packet inspection is the best way to target short lived dynamic malicious connections without impacting the other services of other legitimate containers normal network traffic.
5. The attacker container binds the host root file system and gains access to the host system. Then it modifies the cron utility to gain host execution capabilities. This is a typical container threat pattern: once a privileged container is being controlled, the malicious application will try to escape to the host.
   » A container security solution needs to be able to detect privilege escalations and container escapes. NeuVector can also monitor and protect the behavior on host systems by detecting privilege escalations and blocking suspicious and unauthorized processes.
6. A host cron job is created to download a malicious payload every minute. The malicious payload includes a network scanner and a downloader script. From this point on with the files dropped in, the attacker successfully controls a worker node and the container environment.
   » A true container security platform will need to monitor process and file system behavior as well. NeuVector detects and also will be able to block these malicious behaviors by monitoring processes and sensitive system folders, and alert the DevSecOps team in real time.
7. The network scanner uses a list of public Cloud IP ranges to identify the next victim. It also further scans the local network for ports associated with Redis, Docker, SSH, and HTTP, using a scanning tools like zmap, zgrap, and jq.e etc. Once it finds a hole, it uploads itself to another ngrok URL to keep spreading the malware.
   » Port scanners and other unauthorized processes can be immediately detected at the source container or running on the host. NeuVector is able to learn and whitelist all host processes and lock the host down so malicious processes will be blocked.
8. The downloader script then downloads and installs various binaries. The malware being downloaded can be run on the host or as a container, to be able to scaled up quickly. For example, well known crypto miners are usually run as containers within just minutes.
   » Container creation needs to be guarded and enforced. External egress network communications needs to be inspected by a true layer 7 container firewall. The production container environment should be locked down to prevent unauthorized containers as well as unauthorized behavior by any running containers.
9. As mentioned in the beginning, Doki queries the Dogecoin API, uses SHA265 encryption and constructs a full URL address dynamically at runtime, which bypasses the tradition network security checks such as blacklist URL/IP lists.
   » A typical L3/L4 firewall won’t be able to inspect traffic if it’s encrypted. NeuVector is able to address difficult issue and can block malicious URLs or bad behavior in network connections. The lightweight, container-native NeuVector container security solution solves these issues.

Preparing the Infrastructure to Prevent Advanced Attacks

The Doki malware is another example of why a cloud-native security solution which enables defense in depth is critical for modern cloud infrastructures. Zero-day attacks, advanced malware, and vulnerability exploits will all continue to evolve. In this case, it is critical to be able to detect and block the decentralized backdoor – dynamic Dogecoin connections, unauthorized process/file activity, and privilege escalations.

“Doki” is a complicated malware attack that leverages the modern Docker infrastructure to attack, mine the infrastructure and spread. It is decentralized and immutable, taking only a few minutes or hours to infect then scale up and across. In a production environment, run-time security enforcement is the most critical piece to protect your environment from these attacks. But it doesn’t mean a security solution needs to be more complicated to address this. With the right container native security platform, the protection will be deep and wide, and won’t require complex tooling. This is why we’ve built NeuVector to have all the benefits of cloud-native containers but with the automation and integration to secure the entire CI/CD pipeline all the way into production.