Hardening Kubernetes Node Agents: Why I Ditched Root Containers in kubelet's CRI Runtime

The Problem: Kubernetes Node Agents Running as Root

When I first started deploying Kubernetes clusters at scale, I inherited a fleet of nodes where kubelet's container runtime was configured with wide-open Docker socket access. The symptom was subtle: one compromised workload pod managed to write directly to /var/lib/kubelet on the host, bypassing all namespace isolation. The root cause? Every container spawned by kubelet was running as UID 0, and the Docker daemon itself was executing with unrestricted host privileges.

This isn't theoretical. A single privilege escalation in any workload container becomes a full node compromise when the runtime runs as root.

Architectural Boundary: CRI, containerd, and the Security Model

Kubernetes abstracts container execution through the Container Runtime Interface (CRI). Most production deployments use containerd as the low-level runtime, but the security model depends entirely on how you configure user namespaces and seccomp profiles.

Here's the default failure mode in many clusters:

# /etc/containerd/config.toml (INSECURE DEFAULT)
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true
  # Missing: user namespace remapping
  # Missing: seccomp profile enforcement

Without explicit user namespace configuration, every container process maps directly to host UID 0. Even with PodSecurityPolicy or PodSecurity admission, the runtime itself is still running privileged.

Hardened Configuration: Non-Root Runtime with User Namespaces

The fix requires three layers:

1. Enable User Namespace Remapping in containerd

# /etc/containerd/config.toml
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

  # Force user namespace remapping
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.Linux]
    UIDMappings = [{container_id = 0, host_id = 100000, size = 65536}]
    GIDMappings = [{container_id = 0, host_id = 100000, size = 65536}]

This remaps container UID 0 to host UID 100000, breaking the direct root mapping. Restart containerd:

sudo systemctl restart containerd
sudo systemctl restart kubelet

2. Lock Down the kubelet Configuration

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serializeImagePulls: false
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock

# Enforce seccomp and AppArmor
seccompDefault: true

# Restrict kubelet API access
authentication:
  webhook:
    enabled: true
authorization:
  mode: Webhook

# Drop legacy docker socket mounting
enableCAdvisorJSONEndpoint: false

3. Enforce Pod-Level Security Contexts

Even with runtime hardening, each pod must explicitly drop privileges:

apiVersion: v1
kind: Pod
metadata:
  name: secure-workload
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10000
    fsGroup: 10000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL
    volumeMounts:
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}

Operational Trade-offs and Failure Boundaries

Build cache invalidation: Multi-stage Dockerfiles with BuildKit cache mounts become critical when you can't rely on layer caching across nodes with different UID mappings. I had to rewrite every Dockerfile to use --mount=type=cache,target=/go/pkg/mod to avoid re-downloading dependencies on every build.

Persistent volume permissions: When user namespaces are enabled, any pre-existing PersistentVolumes with root-owned files will fail permission checks. You need an init container to chown recursively:

initContainers:
- name: fix-permissions
  image: busybox:1.36
  command: ['sh', '-c', 'chown -R 10000:10000 /data']
  volumeMounts:
  - name: data
    mountPath: /data
  securityContext:
    runAsUser: 0  # Init container exception for migration

Socket mounting restrictions: If any workload legitimately needs Docker socket access (CI/CD builders, monitoring agents), you must use rootless Docker-in-Docker with sysbox or gVisor runtime classes instead.

The Unresolved Dilemma: Performance vs. Isolation

User namespaces add measurable overhead. In high-throughput network workloads, I observed a 3-7% latency increase due to UID/GID translation in the kernel. For batch processing jobs with heavy filesystem I/O, the overhead was negligible.

The real question: is the security boundary worth the operational complexity when you're already running workloads in separate network policies and resource quotas? Some teams solve this by running untrusted workloads in dedicated node pools with user namespaces enabled, while keeping trusted platform services on traditional privileged runtimes.

What's your threshold for accepting kernel-level isolation overhead in production Kubernetes nodes?