A hands-on lab: Why running as root in Kubernetes containers is dangerous?

#kubernetes #security #cloud #docker

Introduction

In the world of Kubernetes security, one commonly heard recommendation is to run containers as non-root users. But what are the real security implications of running as root within containers? This best practice is often emphasized in Docker images and Kubernetes configurations. In Kubernetes manifests it can be done with the following code:

securityContext:
   runAsNonRoot: true # Verifies that the UID specified by runAsUser is not 0. Pod will not run if UID is set to 0. 
   runAsUser: 1000 # Sets the user ID for the processes running in the container to 1000

You can set only the runAsUser field without including runAsNonRoot. However, if you specify runAsNonRoot, you must also define runAsUser.

Why running as root in containers can be dangerous? ⚠️

There are several potential attack vectors, but the most apparent ones that come to mind are outlined below. Let’s explore these through experimentation.

Setting up the lab 🛠️

To illustrate the risks of running containers as root, we'll conduct a practical test using two similar containers:

An alpine:3.20.2 container running as root by default.
A custom alpine:3.20.2 container configured to run as a non-root user.

Here’s the Dockerfile for the non-root container:

# Use the Alpine image as the base
FROM alpine:3.20.2

# Add a non-root user named 'nonroot' with user ID 1000
RUN adduser -D -u 1000 nonroot

# Set the user to 'nonroot' for subsequent commands
USER nonroot

# Optional: Set a default command
CMD ["sh"]

Using Minikube 1.32.0 with Kubernetes version v1.28.3 for this local setup, I build the image and make it accessible to my Minikube cluster with:

eval $(minikube docker-env)
docker build . -t nonroot:1.0.0

Next, I deploy these containers in Kubernetes. For this demonstration, I use a hostPath volume to test privileges, though I strongly advise against using hostPath outside your homelab! The root pod definition is:

apiVersion: v1
kind: Pod
metadata:
  name: root
spec:
  containers:
  - name: alpine
    image: alpine:3.20.2
    command: ["/bin/sh", "-c"]
    args: ["while true; do sleep 100; done"]
    volumeMounts:
    - mountPath: /data
      name: host
  volumes:
  - name: host
    hostPath:
      path: /testDir
      type: Directory

And the nonroot pod definition is:

apiVersion: v1
kind: Pod
metadata:
  name: nonroot
spec:
  containers:
  - name: alpine
    image: nonroot:1.0.0
    command: ["/bin/sh", "-c"]
    args: ["while true; do sleep 100; done"]
    securityContext:
      runAsUser: 1000          # Ensure the container runs as non-root user with UID 1000
    volumeMounts:
    - mountPath: /data
      name: host
  volumes:
  - name: host
    hostPath:
      path: /testDir
      type: Directory

Testing potential attacks 🧪

Downloading malware 🦠

One common attack vector is downloading and executing malicious packages. I tested this by attempting to just fetch some data from https://dev.to.

Since curl was not initially installed in the containers, I tried to install it:

The installation succeeded in the root container but failed in the nonroot. Let's try to fetch the data.

It worked correctly. This indicates that running as a non-root user can effectively mitigate this attack vector. Of course, other measures, such as using a read-only filesystem, can further enhance security. I will cover that in a future article.

Access to host resources 🔒

I have a host directory mounted to the pod (again - please do not do that outside of your homelabs!). With this kind of access attacker can try to access the directory with the static pods manifests and try to run the malicious pods (downloading malicious images should be blocked by your cluster policies though). It can be done by adding a new manifest to static manifest directory, which is usally /etc/kubernetes/manifests on the kubernetes node.

Having this access attacker can try to:

perform Man-in-the-Middle attack by deploying the pod that intercepts network traffic within the cluster and capture sensitive information,
deploy a backdoor pod with reverse shell (you can find examples at https://www.revshells.com/) accepting connections from hacker's machine,
run a pod that transfers the data from the volumes containing sensitive data to an external entity,
expand the attack vector by reading secrets from ETCD store and penetrating the infrastructure further,
run crypto miner using your resources for financial gain.

Let's deploy a simulation of crypto miner using the following manifest:

apiVersion: v1
kind: Pod
metadata:
  name: crypto-miner
spec:
  containers:
  - name: miner-container
    image: busybox
    command: ["/bin/sh", "-c", "while true; do echo 'Mining in progress...'; sleep 5; done"]

And adding the static pod file from both pods:

As you can see the command on the non-root pod was denied. It run successfully on root pod and added the crypto miner manifest to the static pods directory. Was the pod created in a cluster?

Yes, it runs in the cluster and will be rescheduled in case of failures or restarts.

Another thing that attacker can do with access to hostPath on a node is to read /etc/passwd file on the host machine. This file does not contain the password in plain text, but it gives attacker a knowledge about users existing in a system. This knowledge combined with some other data sources and /etc/shadow may let the attacker exploit the system further.

Privilege Escalation 🚫

Can't you just change the non-root user to root and do exactly the same? Let's try it.

No, you can't. The attempt to switch from non-root to root failed, demonstrating that without sudo access, privilege escalation is not feasible. Therefore, if a non-root user isn’t in the sudoers list, the risk is mitigated.

How to prevent? 🛡️

To prevent security issues related to running containers as root, follow these best practices:

Use non-root users: Always define and use non-root users in your Docker container. 🧑‍💻
Leverage Kubernetes Security Context: Specify the user for container execution using Kubernetes security contexts. 🔐

Conclusion ✨

I hope this article has shed light on the importance of running containers as non-root users within Kubernetes.

I’d love to hear your thoughts and insights - drop a comment below or share your feedback! Stay tuned for more insights as I continue to explore and describe various aspects of Kubernetes security.

Stay secure!🔐