DEV Community

Cover image for Hardening Kubernetes: How to Block `kubectl debug node` from Breaking Your Cluster Matrix
Andriesh Rusnac
Andriesh Rusnac

Posted on

Hardening Kubernetes: How to Block `kubectl debug node` from Breaking Your Cluster Matrix

#kubernetes #security #devops #tutorial

You've done everything right. You're hardening your Kubernetes cluster, meticulously setting runAsNonRoot: true, and applying readOnlyRootFilesystem: true across all your workloads. Your pods are locked down tight.

And then, you run this one command:

kubectl debug node/my-node-name -it --image=ubuntu
Enter fullscreen mode Exit fullscreen mode

Boom. Within seconds, you have a shell with full root access to the host machine's root filesystem.

Wait... what? ## The Privilege Escalation Trapdoor

When you run kubectl debug node, Kubernetes spawns a highly privileged "debugger" pod. This pod bypasses standard pod-level restrictions by design: it hooks directly into the hostโ€™s PID, network, and IPC namespaces, and mounts the hostโ€™s root directory (/).

If an attacker compromises a service account or user credentials with sufficient permissions, your container isolation vanishes instantly.

Here is how to completely close this loophole and build a bulletproof environmentโ€”including how to test it locally in a containerized k0s lab.

3 Layers of Defense to Block Node Debugging

1. The Quick Fix: Namespace-Level Pod Security Admission (PSA)

kubectl debug node spins up its helper pod inside your active namespace. If that namespace enforces the native Kubernetes Baseline or Restricted security standards, the API server will flat-out reject the debugger pod.

To secure your working namespace (e.g., default):

kubectl label --overwrite ns default pod-security.kubernetes.io/enforce=restricted
Enter fullscreen mode Exit fullscreen mode

Why this works: The debug pod requires hostPID: true and hostPath mounts. The restricted profile strictly forbids these, causing the API server to kill the request before the pod is even born.

2. The Bulletproof Way: Cluster-Wide Automation

Relying on manual namespace labels leaves room for human error. To make restricted the unyielding default for every new namespace, you can configure a cluster-wide Admission Configuration.

Create an admission-config.yaml file:

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "restricted"
      enforce-version: "latest"
    exemptions:
      namespaces: ["kube-system", "kube-public", "k0s-system"] #
Enter fullscreen mode Exit fullscreen mode

System pods still need host access
If you are using k0s, you can pass this configuration file straight to your API server by adding it to your /etc/k0s/k0s.yaml:

spec:
  api:
    extraArgs:
      admission-control-config-file: /etc/k0s/admission-config.yaml
Enter fullscreen mode Exit fullscreen mode
  1. The Structural Fix: Tighten Your RBAC kubectl debug node relies on access to specific API subresources. To prevent non-admin users from utilizing it, audit your ClusterRoles and ensure they do not have access to:

  • nodes/proxy (required for node debugging)

  • pods/ephemeralcontainers (used for pod-level debugging)

How to Test This Locally (The k0s Docker Playground)
Want to see this in action without breaking a production cluster? You can spin up a lightweight k0s cluster right inside a local Docker container.

(Note: If you are running Docker Desktop on macOS/Windows, we must pass a flag to bypass strict host kernel checks).

Step 1: Fire up the container

docker run -d --name k0s-lab \
  --hostname k0s-lab \
  --privileged \
  -v /var/lib/k0s \
  -p 6443:6443 \
  docker.io/k0sproject/k0s:v1.36.1-k0s.0 \
  k0s controller --enable-worker --ignore-pre-flight-checks
Enter fullscreen mode Exit fullscreen mode

Step 2: Connect your local kubectl
Wait about 15 seconds for the API server to bootstrap, then grab the kubeconfig:

docker exec k0s-lab k0s kubeconfig admin > ~/.kube/k0s-lab.config
export KUBECONFIG=~/.kube/k0s-lab.config

# Check that it's alive!
kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

Step 3: Trigger the Trap
First, let's enforce our security profile on the default namespace:

kubectl label --overwrite ns default pod-security.kubernetes.io/enforce=restricted
Enter fullscreen mode Exit fullscreen mode

Now, try to break out of the container matrix using the debug command:

kubectl debug node/k0s-lab -it --image=ubuntu
Enter fullscreen mode Exit fullscreen mode

The Result:
Instead of a root prompt, the API server will instantly drop a massive rejection block:

Error from server (Forbidden): pods "node-debugger-..." is forbidden: violates PodSecurity "restricted:latest": hostNetwork: true, hostPID: true...
Enter fullscreen mode Exit fullscreen mode

Your playground is officially secure!

What About Real Emergencies? (The "Break-Glass" Solution)

If you lock down your cluster so tightly that node debugging is completely blocked, how do you fix a node when things actually go wrong?

The best practice is to design a dedicated Break-Glass Namespace:

  1. Create a namespace called break-glass.
  2. Explicitly label it as privileged: 
kubectl create ns break-glass
kubectl label ns break-glass pod-security.kubernetes.io/enforce=privileged
Enter fullscreen mode Exit fullscreen mode
  1. To debug a node, you must now explicitly invoke that namespace: 
kubectl debug node/k0s-lab -it --image=ubuntu -n break-glass
Enter fullscreen mode Exit fullscreen mode

This ensures your day-to-day environment blocks accidental slips, while keeping an audit-logged backdoor open only for intentional, emergency maintenance.

How are you handling host-level security in your clusters? Let's talk in the comments below! ๐Ÿ›ก๏ธ

Top comments (0)