DEV Community: Siranjeevi Dheenadhayalan

Secure your enterprise critical assets from secret sprawl

Siranjeevi Dheenadhayalan — Thu, 13 Jun 2024 22:02:23 +0000

Understand the risks of secret sprawl, embracing shift-left and strategies to secure secret leaks in the modern software development lifecycle.

Secret sprawl:

Enterprises often need help with the uncontrolled proliferation of secrets across their IT infrastructure. The unchecked proliferation is called a secret sprawl, and usually, secrets get scattered across server systems, repositories, configuration files, applications, and other storage locations. The risk from secret sprawl can compromise security and enable unauthorized access, thus making it essential for organizations to address the issue.

Risks:

High Blast radius: When sensitive secrets get dispersed across multiple locations, the attack surface for possible data breaches multiplies. When attackers gain access to the secret, it can lead to unauthorized access and breaches.

Insider threat: Even if the sprawled secret exists within the enterprise VPC, unauthorized employees can access sensitive assets violating least privilege access. This insider threat can lead to compromised security posture and the theft of sensitive data.

Application downtime: When a sprawled secret expires or reaches the end of life due to a set TTL, determining the side effects is often complex and time-consuming. The effort is high to estimate the number of applications using the hardcoded secret and the importance of those applications to business operations. Hence, rotation or expiration of such secrets without proper usage analysis can lead to application downtime, affecting users and other applications.

Lack of visibility: Since SecOps cannot monitor hardcoded secret usage, it isn't easy to track the entities accessing the secret - making audit and access control challenging.

Compliance: Regulatory compliance policies such as GDPR, PCI DSS, HIPAA, NIST, etc… require enterprises to safeguard secrets that can access sensitive user information. Exposure of these secrets internally within the organization or externally in public repositories can lead to hefty fines and distrust among users.

Prevent Secret Sprawl Proactively

Enterprises often adopt the following best practices to prevent sprawl from happening and maintain governance:

Centralized secrets management: As organizations adopt a multi-cloud approach, developers store secrets in native secret managers like AWS Secrets Manager, Azure Key Vault, GCP Secrets Manager, etc. While this is convenient for developers, it creates multiple secret store hotspots and complicates audit and visibility for security operations. Hence, a centralized secrets management experience is essential.

Dynamic secrets: When an application requests a secret, creating a just-in-time secret with role-based access policies can remove secrets with a long lifetime and thus prevent any sprawl. It is essential to understand the required secret TTL based on application needs.

Secret Scanners: Secret scanners can scan to detect sprawled secrets from popular hotspots such as repositories, container images, applications, server system files, etc. While scanning leaked secrets seems essential, the secret scanners can also help prevent future secret sprawls by preemptively guiding developers and security teams in the right direction.

Education and Training: Enterprises should train and create security awareness programs to educate developers and security operations about the importance of collaboration and the risks associated with sprawl. Educated users are more likely to adhere to best practices and contribute to preventing sprawl.

Shift-Left in Security

To proactively reduce secret sprawl, enterprises are moving from the operational utopia of siloed security and development teams to an integrated, developer-first security approach. The main reasons that drive the integrated approach are the following:

Agility and faster time to market: Enterprise products aim to stay relevant and competitive in a fast-paced software market. Thus, constant business requirement changes require rapid development and release processes often hindered by a siloed security team. Therefore, an integrated security team can aid rapid release without compromising security.

Early Issue Identification and Remediation: Teams can identify and address issues sooner by shifting tasks such as testing, security reviews, and code analysis to earlier stages of the SDLC. Doing so reduces the cost and effort required to fix issues discovered later in the development process and minimizes the impact on project timelines.

Risk Reduction: Independent security and development teams may need to pay more attention to address security vulnerabilities promptly. The integrated team can proactively identify and address security threats on time.

Embracing the trend emphasizes that security is no longer a dedicated security team's concern; instead, there is a shared responsibility with the development teams. The approach is often termed "shift-left," enterprises that adopt this paradigm constantly seek to integrate security practices early in the software development lifecycle (SDLC) to address vulnerabilities.

With a shift-left paradigm, the developers refrain from embedding secrets such as API tokens, passwords, and encryption keys in their source code. While this is ideal in theory, the SDLC has multiple checkpoints; thus, more than developer education is needed to help maintain compliance. So, how can enterprises preserve developer productivity while ensuring security best practices in program files? This uncertainty has led to the production of security collaboration products that can centrally manage and provide visibility into secrets embedded in source code.

Attack Vectors

As we discuss the importance of shit-left and security in source code, it is imperative to understand how modern development practices can introduce attack vectors that increase risk. Attack vectors refer to the various points within the development process where malicious actors can exploit vulnerabilities to gain unauthorized access. Here are some common attack vectors that attackers look for:

OSS/3rd party library usage: To save time, developers often use OSS/3rd party libraries to extend the application's functionality. When such libraries have a security vulnerability, it is easier for attackers to target products that use it to exploit the vulnerability.

Configuration files: For modularity and portability, developers often use configuration files to store API endpoints and secrets. IaC tools like Terraform and containerization tools like docker also contain configuration files. Secrets in configuration files can lead to unauthorized access, service impacts, and many other possible threats.

Log files: While logging is helpful for debugging, developers may accidentally log sensitive information to logs/console outputs/ error messages, leading to unauthorized access.

Comments: Developers may inadvertently include secrets as comments during debugging or documentation. While comments are not executed, they are still visible in the source code.

Developers manage their source code using version control software like GitHub, GitLab, and BitBucket. These repositories often become central repositories for sensitive information. Recognizing all potential attack vectors, evaluating each checkpoint in the Software Development Life Cycle (SDLC), and implementing strategies to ensure developers adhere to security best practices throughout the process are crucial.

How to close attack vectors for exposed secrets in Docker

Siranjeevi Dheenadhayalan — Thu, 04 Apr 2024 18:22:43 +0000

Cybernews recently reported 5,500 out of 10,000 public docker images contained 48,000+ sensitive secrets - a combination of harmless and potentially vulnerable API keys. This report illustrates why it's imperative that security and platform teams know the most common attack vectors for their Docker containers and understand how to close them.

This post will provide a brief checklist of the various attack vectors into your Docker containers specifically originating from exposed secrets.

Docker and exposed secrets

Let’s quickly examine the relationship between container runtime and registry. When we spin-up a container, an image is pulled from the registry via APIs and is deployed. This is visualized below:

Image Source

The high number of secrets from the Cybernews report is attributed to developers re-using packages from a registry containing sensitive secrets. Secrets are commonly found in the container image metadata - the environment variables and filesystem. Also, source code leakage could allow attackers to generate newer valid tokens that could provide unauthorized system access.

Attack surface:

An attack surface is a collection of all vulnerable points an attacker can use to enter the target system. Attackers skillfully exploit these vulnerable points in technology and human behavior to access sensitive assets.

We need to understand two Docker concepts as we continue this discussion:

Filesystem: In Docker, each layer can contain directory changes. The most commonly used filesystem, OverlayFS, enables Docker to overlay these layers to create a unified filesystem for a container.

Layers: Docker images are created in layers - i.e., each command on the DockerFile corresponds to a layer.

With that context, let’s understand and analyze how exposed secrets can affect these Docker image attack vectors.

Docker image layers

Secrets explicitly declared in the Dockerfile or build arguments can easily be accessed via the Docker image history command.



#terminal

docker image history <image>

This represents one of the simplest methods for an attacker to capitalize on a secret.

Filesystem

This Dockerfile demonstrates a scenario where sensitive data like SSH private key and secrets.txt are added to the container's filesystem and later removed.



#Dockerfile
FROM nginx:latest

# Copy in SSH private key, then delete it; this is INSECURE,
# the secret will still be in the image.
COPY id_rsa .
RUN rm -r id_rsa

ARG DB_USERNAME
ENV DB_USERNAME =$DB_USERNAME
ARG DB_PASSWORD
ENV DB_PASSWORD =$DB_PASSWORD
ARG API_KEY
ENV API_KEY =$API_KEY

# Expose secrets via a publicly accessible endpoint (insecure practice)
RUN echo "DB_USERNAME=$DB_USERNAME" > /usr/share/nginx/html/secrets.txt
RUN echo "DB_PASSWORD=$DB_PASSWORD" >> /usr/share/nginx/html/secrets.txt
RUN echo "API_KEY=$API_KEY" >> /usr/share/nginx/html/secrets.txt
RUN rm /usr/share/nginx/html/secrets.txt

CMD ["nginx", "-g", "daemon off;"]

Docker uses layer caching - hence, the secret is still available in one of the layers. An internal attacker can also extract individual layers of a Docker image, stored as tar files in registries, which enables them to uncover hidden secrets.

After creating a Dockerfile, developers mistakenly use build arguments to create an image. For the above dockerfile, the secrets are input as arguments.

terminal



docker build \
--build-arg DB_USERNAME=root \
--build-arg DB_PASSWORD=Xnkfnbgf \
--build-arg  API_KEY=PvL4FjrrSXyT7qr \
-t myapp:1.0 .

While convenient, it is not secure since the arguments also get embedded in the image. A simple docker history --no-trunc can expose the secret values. Developers should either use multi-stage builds or secret managers.

Environment variables

Apart from Docker image access, unauthorized access to the source code of the docker image can provide additional attack vectors. The .env files are primarily used to store secrets such as API tokens, database credentials, and other forms of secrets that an application needs. When attackers have access to secrets in the .env, they can make unauthorized accesses that the secret allows.

Dockerfile

DockerFile is a standard file that contains execution instructions to build an image while spinning up containers. Hard-coding secrets into DockerFile creates a significant attack surface. When attackers access the DockerFile, they can see hard-coded secrets, the base image, the list of dependencies, and critical file locations. Developers need to use appropriate secret managers to reference variables.

Docker-compose.yml 

Docker-compose defines networks, services, and storage volumes. When an attacker views the file, they can understand the application architecture and exploit or disrupt its operation.



#docker-compose.yml
services:
  web:
    image: my-web-app:latest
    ports:
      - "80:80"
    networks:
      - app-network
  db:
    image: postgres:latest
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: example_db_password
networks:
  app-network:
volumes:
  db-data:

In the above docker-compose.yml, the postgres database password is hardcoded. The password can easily be accessed with the docker exec command as shown below:



#terminal
docker exec -it <container id or name> /bin/bash
env

Apart from secrets, an attacker can also analyze the volume mappings and identify potential points of weakness. If they discover that the database volume (db-data) is also mounted to the host filesystem, they could exploit and perform a container breakout attack, gaining access to the underlying host system.

CI/CD config files

CI/CD configuration files such as .gitlab-ci.yml, Azure-pipelines.yml , Jenkinsfile,  etc., contain instructions for building, testing, and deploying applications. The logs generated in CI/CD pipeline can contain debugging and logging information. If a developer includes a logging statement that inadvertently prints a sensitive secret, it can lead to unauthorized exposure and compromise. Such secret leaks need to be detected so that developers can fix their source code.

Developers also tend to leave the registry login credentials in the CI CD configuration files. Consider the following gitlab-ci.yml.



# .gitlab-ci.yml

variables:
  DOCKER_IMAGE_TAG: latest
  DOCKER_REGISTRY_URL: registry.example.com
  DOCKER_IMAGE_NAME: my-docker-image
  DOCKER_REGISTRY_USER: adminuser <-- should use $CI_REGISTRY_USER  
  DOCKER_REGISTRY_PASSWORD: secretpassword <-- should use $CI_REGISTRY_PASSWORD 

# Jobs
build:
  stage: build
  image: image_name:stable
  script:
    - docker build -t $DOCKER_REGISTRY_URL/$DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG .
    - docker login -u $DOCKER_REGISTRY_USER -p $DOCKER_REGISTRY_PASSWORD $DOCKER_REGISTRY_URL
    - docker push $DOCKER_REGISTRY_URL/$DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG

In the configuration above, the developer makes a docker registry login using a hardcoded username and password, leading to unwarranted secret exposure. A good development practice is to integrate CI/CD environments with secret managers like Hashicorp Vault.

Detecting secrets

Older or unused Docker images can contain unpatched vulnerabilities or outdated dependencies, posing security risks to the enterprise. Regularly scanning and removing unused images helps mitigate these risks by reducing the attack surface and ensuring that only secure images are deployed.

Enterprises also need to be actively using secret scanners to detect secrets in docker images, whether they’re stored in Docker Hub, JFrog Artifactory, AWS ECR, or any other repository. There are numerous secret scanning tools and infrastructure as code scanning tools as well as secrets management platforms that provide a more high-scale, holistic answer to these secrets detection, management, encryption, and storage challenges.

I recommend that enterprise users buy a mature platform with plenty of successful case studies, rather than embark on an expensive, high-skill DIY project to detect and manage secrets.