A cybersecurity incident can cause severe damage to the reputation of the organization and competitive disadvantage in the market, the imposition of penalties, and unwanted legal issues by end-users. On average, the cost of each data breach is USD 3.92 million as per this IBM report. The biggest challenges providing security in organizations are:
Lack of skills and training in security tools and practices
Lack of visibility and vulnerabilities
Continuous monitoring of the current state of security
In the recent survey by PaloAlto Networks, State of Cloud Security report, it was discovered that 94% of organizations use one or more cloud platforms and around 45% of their compute is on containers or CaaS. The dominance of containers is increasing and thus the security threats. The top issues identified as being a threat in these reports are:
Data exposure and malware
Weak or broken authentication
Incorrect or over-permission access
In this article, we will go through some of the best practices, we can implement to reduce the security risks in the containerized workloads.
Below are some tips:
- When creating the container image, please use a hardened base image sources from a well known trusted publisher.
- Pick those images which are published frequently with the latest security fixes and patches.
- Use signed and labeled images (sign with Notary or similar tools) and verify the authenticity of the image during the pull to stop man-in-the-middle attacks.
As much the sourcing of base image needs to be from trusted sources, the packages installed on top of the base also need to be from verified and trusted sources for the same reason.
What I mean by surface area is the number of packages and libraries installed in the image. Common sense is if the number of objects is less, the chances of having vulnerability is also reduced. Keep the image size minimal satisfying the application runtime requirements. Preferably, have only a single Application in one application container.
- Remove unnecessary tools and software like package managers (eg. yum, apt), network tools and clients, shells from the image, netcat (can be used to create reverse shell).
- Use the multi-stage Dockerfiles to remove software build components out of production images.
- Do not expose unnecessary network ports, sockets or run unwanted services (e.g. SSH daemon) in the container to reduce threats.
- Choose alpine images or scratch images or container optimized OS as compared to full-blown OS images for the base image.
All the secrets should be kept out of the image and Dockerfile. The secrets include SSL certificates, passwords, tokens, API keys, etc should be kept outside and should be securely mounted through the container orchestration engine or external secret manager. Tools like Hashicorp Vault, Cloud provided secret management services like AWS Secrets Manager, Kubernetes secrets, Docker secrets management, CyberArk, etc. can improve the security posture.
Often the enterprises have their own base images with proprietary software and libraries which they don’t want to distribute in public. Ensure the image is hosted on a secure and trusted registry to prevent unauthorized access. Use a TLS certificate with trusted Root CA, and implement strong authentication to prevent MITM attack.
This is the most common misconfiguration in the containerized workload. With principles of least privileges in mind, create an application user and use it to run the application process inside the container. Why not root? The reason is that a process running in a container is similar to the process running on the host operating system except for the fact that it has additional metadata to identify that it is part of a container. With UID and GID of root user in a container, you can access and modify the files written by root on the host machine.
Note – If you don’t define any USER in the Dockerfile, it generally means that the Container will be running with root user.
When designing CI/CD for the container build and delivery, include image scanning solution to identify vulnerabilities (CVEs) and do not deploy exploitable images without remediation. Tools like Clair, Synk, Anchore, AquaSec, Twistlock can be used. Some of the container registries like AWS ECR, Quay.io are equipped with scanning solutions – do use them.
AppArmor is a Linux security module to protect OS and its applications from security threats. Docker provides default profile to allow the program to a limited set of resources like network access, kernel capabilities, and file permissions, etc. It reduces the potential attack surface and provides a great in-depth defense.
Usually, the containers log everything on STDOUT, and these logs are lost once they are terminated, it is important to securely stream the logs to a centralized system for audit and future forensics. We also need to ensure that this logging system is secured and there is no data leakage from the logs.
Even if you deploy vulnerability scanning solutions based on repository data and take all necessary precautions, there is still a chance of being victimized. It is important to continuously monitor and log the application behavior to prevent and detect malicious activities.
There is no silver bullet solution with Cyber Security, a layered defence is the only viable defence. – ICIT Research
By implementing the above best practices, you can make it harder for the attacker to find ways to exploit your system. I am pointing out some tools and references that can be used to audit and secure the containers. Security is a vast topic, we haven’t covered Kubernetes specific controls in this article but stay tuned, we can have a follow-up article focussing on the Kubernetes security best practices.
To simplify the adoption of Security controls, I am suggesting few opensource and commercial offerings which can be used to discover the current state, to generate advisories for your workload.
- Hadolint Linter for Dockerfile – Use the linter to do static code analysis of the Dockerfile. The linter helps in implementing the best practices. It can be integrated with popular code editors and integration pipelines.
- OWASP Cheatsheet – OWASP is an open community which is quite popular among security experts. This cheat sheet is a good starting point to start with.
- OpenSCAP for Container – Security Content Automation Protocol (SCAP) is a multi-purpose framework of specifications that supports automated configuration, vulnerability and patch checking, technical control compliance activities, and security measurement. It implements NIST standards.
- Sysdig Falco – Falco can be used to implement runtime security. It uses efficient eBPF to intercept calls and traffic for realtime monitoring and forensics. As hackers continue to evolve, new vulnerabilities are discovered and often not picked up by static scanning tools. A solution with machine learning capability, continuous behavioral monitoring, and advanced AI/ML-based engines can’t be ignored from the list of essentials.
Commercial offerings from AquaSec, Twistlock, Sysdig, Synk, Qualys for Enterprise-grade security tools, and solutions.
- NIST Application Container Security
- CIS Benchmark for Docker
Do comment if you have any interesting security incident or a preventable hack involving Containers you want to share with the community.