DEV Community: Le Beltagy

Iran Doesn't Need to Mine Hormuz — Your requirements.txt Is Already Rigged

Le Beltagy — Fri, 21 Aug 2026 12:46:40 +0000

Iran Doesn't Need to Mine Hormuz — Your requirements.txt Is Already Rigged

Every headline you've read this week is a diversion.

The Strait of Hormuz is not the target. You are. And you have been for months, possibly years, while you retweeted tanker tracking maps and debated whether Brent crude would touch $150.

Iranian state-sponsored groups — OilRig, APT33, MuddyWater, Agrius — did not spend the last decade pivoting to cloud infrastructure so they could watch you panic about a waterway. They did it so they could own your build pipeline while you were distracted. And they have.

This is not speculation. CISA Advisory AA24-038A explicitly maps Iranian APT campaigns against U.S. and allied critical infrastructure to cloud identity, Kubernetes targets, and software supply chains. Not SCADA. Not PLCs. Your kubectl binary. Your Helm charts. That FastAPI microservice running payment webhooks that you deployed on a Friday and haven't touched since March.

The Revolutionary Guard does not need a mine. They need a maintainer who hasn't updated python-jose in fourteen months.

The Theater and the Operation

You watched the Strait. They watched your CI/CD.

Geopolitical analysis is a spectator sport for infrastructure engineers, and Iranian cyber command is the bookie. While your LinkedIn feed filled with satellite imagery and retired admirals explained chokepoint logistics, the actual operation ran silently against:

Public Helm charts with hardcoded cluster-admin ServiceAccounts
FastAPI services with python-multipart handling unbounded file uploads on single-threaded Uvicorn workers
.kube/config files exfiltrated from developer laptops in a dev-legacy namespace that predates your current CTO
Terraform state stored in a single S3 bucket with versioning disabled and a policy written by someone who left in 2021

The Hormuz closure narrative is Information Operations. The closure of your API gateway due to an unpatched ASGI memory exhaustion vulnerability is the kinetic effect. You are living in the IO campaign already. You just don't know which side you're on.

Your FastAPI Service Is the Chokepoint

Hormuz moves 21 million barrels per day through a channel so narrow tankers touch mud on both sides. It has done this for five millennia because there is no alternative.

Your FastAPI gateway moves 100% of your card-processing traffic through a single async event loop that deadlocks when someone uploads a malformed multipart boundary. You built it this way because a tutorial told you async def was "fast." You didn't install rate limiting. You didn't bound the upload size. You certainly didn't audit whether the starlette version you pinned in 2022 has a known memory leak under nested multipart headers.

This is the strait. This is the geography of your revenue. And it is defended by exactly one Uvicorn worker pool with zero circuit breakers and a restart policy of Always that simply reintroduces the same deadlock on boot.

Iranian speedboats swarm tankers to force asymmetric naval commitments. A single slow POST request with a nested Content-Type: multipart/form-data; boundary=---- string does the same to your payments service. You don't need a navy. You need twelve lines of Python and a botnet rented for $40 an hour.

The strait is geology. Your API is architecture. One has survived empires. The other cannot survive a penetration test.

They Don't Breach Clusters. They Walk Through Wildcards.

The K8s compromise is not a Hollywood hack. It is a privilege escalation through accumulated laziness.

APT operators do not need a zero-day in the Kubernetes API server. They need your dev-legacy namespace to have a RoleBinding to a ClusterRole with wildcards on pods and pods/exec. They need a ServiceAccount in your monitoring namespace with automountServiceAccountToken: true that mounts a token readable by any pod compromise. They need your etcd backups stored in the same project as your public-facing documentation.

I built kube-radar to find this exact rot. Here is what it sees in enterprise clusters every single deployment:

# Wildcard verb on core resources
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]

# cluster-admin bound to a namespace labeled "team-backend"
roleRef:
  kind: ClusterRole
  name: cluster-admin

This is not a vulnerability. This is surrender encoded in YAML. You handed them the keys and labeled the namespace so they knew where to start.

The real minefield is not in the Persian Gulf. It is in your RBAC manifest, and every wildcard * is another buoy in the channel marking safe passage for an adversary.

Your Cloud Resilience Is a Puppet Show

You will tell me you are resilient. You run three clouds. You have multi-region. You have a CNAPP dashboard that shows your posture as green.

Here is what that theater costs you.

Your control plane is regional. Your etcd is zonal. Your disaster recovery is a Terraform state file in a single S3 bucket with MFA delete disabled because someone on-call needed to push a hotfix at 3 AM and the screaming customer mattered more than the recovery architecture. You pay the Hormuz premium in cloud spend — 2.4 million dollars for redundancy you have never actually tested under adversarial conditions.

Wiz will find your public S3 bucket. It will not find the Iranian operator authenticating with a stolen Azure AD refresh token that grants them Contributor on the subscription hosting your AKS cluster. CNAPP tools map misconfigurations. They do not map compromise. The mine is already inside the hull. Your cloud security posture management is counting deck chairs.

The Real Strait Closure

If Hormuz closes, the market reacts. Oil spikes. Insurance rates triple. The U.S. Fifth Fleet responds. It is visible. It is measurable. It is managed.

If your Kubernetes control plane is compromised by a supply-chain poisoned through a PyPI dependency, there is no Fifth Fleet. There is a Slack thread. There is a post-mortem. There is a CEO asking why the payments API returned 502s for six hours while the SOC triaged alerts from a cluster that was already exporting cronjob outputs to a S3 bucket in eu-central-1.

The strait has naval escorts, maritime insurance, and five thousand years of contested history teaching it how to survive closure.

Your cluster has a Terraform module maintained by an intern who left and a TODO: fix RBAC comment from 2021.

What You Do Now

Stop reading about Hormuz and audit your blast radius.

Freeze your FastAPI dependencies. Audit python-multipart, starlette, uvicorn, python-jose. Verify checksums. Consider whether you can survive a single slowloris POST against your ASGI layer. If the answer is unknown, the answer is no.
Delete your RBAC wildcards. Run kube-radar or equivalent. If a RoleBinding or ClusterRole has * on verbs or resources, treat it as an active compromise in progress. Because it is.
Separate your control plane from your workload identity. No cluster-admin for CI/CD. No long-lived kubeconfig on developer machines. Use short-lived tokens, node attestation, and Cilium identity-aware microsegmentation.
Test your failover under fire. Multicloud means two control planes with independent blast radiuses, not one control plane mirrored to two regions for the price of two. If an attacker owns your GitHub org and your Terraform state, they own both clouds simultaneously. Architect for that.

The Revolutionary Guard is not studying tanker routes. They are studying your GitHub org.

Hormuz will survive the next decade of conflict. It is a rock formation.

Your microservices architecture will not survive the next unpatched dependency.

Close the strait in your own stack before they do it for you.

Author: Mohamed ElBeltagy (@beltagyy) — Cloud Security & K8s engineer at Siemens, creator of kube-radar

Read more: dev.to/le_beltagy

I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First.

Le Beltagy — Wed, 19 Aug 2026 15:40:39 +0000

I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First.

Chaos engineering is not about breaking things. It's about discovering that your "production-grade" homelab is held together by hope and a single etcd snapshot before someone else finds out for you.

The Setup

I was lying in bed at 1:47 AM, staring at the ceiling, unable to sleep.

Not because of caffeine. Because of a thought that had been gnawing at me for weeks:

If one of my nodes died right now, would my cluster actually survive?

I run a 4-node bare-metal Kubernetes cluster on Talos Linux. Dell OptiPlex control plane. Three Raspberry Pi workers. Cilium eBPF. ArgoCD. Longhorn distributed storage. Prometheus. Grafana. The whole cloud-native stack, shoehorned into $220 of scrap hardware and stubbornness.

From the outside, it looks solid. ArgoCD syncs green. Cilium status shows healthy. Longhorn volumes are replicated across three nodes. I have etcd snapshots every 6 hours to S3. On paper, I'm resilient.

But I had never actually tested it.

Not a controlled test. Not a graceful node drain. I mean chaos. Sudden death. The kind of failure that happens at 3 AM when a power supply dies, or a kernel panics, or a neighbor's construction crew hits the wrong breaker.

So I got out of bed, walked to my desk, and installed Chaos Mesh.

Why Chaos Engineering on a Homelab?

Professionally, I design AWS infrastructure with multi-AZ failover, auto-scaling groups, and managed services that abstract failure away. At Siemens, if an EKS node dies, the managed node group replaces it before I finish reading the alert.

But my homelab has no managed control plane. No AWS SLA. No auto-repair. If a Pi's USB boot drive corrupts, that node is gone until I physically fix it.

I needed to know:

What dies first when a worker vanishes? Not "what should die" — what actually dies.
Does Longhorn really failover? Three replicas sound great until you realize two of them were on the same node.
Does Cilium handle network partitions? Or does it just... stop routing?
How long does ArgoCD stay useful? If the control plane loses the GitOps controller, can I still reason about state?
What's my actual MTTR? Mean Time To Recovery — not theoretical, measured with a stopwatch and cold sweat.

I wasn't planning to learn. I was planning to find out.

The Weapon

I installed Chaos Mesh directly into the cluster:

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace chaos-testing \
  --create-namespace \
  --set chaosDaemon.runtime=containerd \
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock

Chaos Mesh is a CNCF sandbox project that injects failure into Kubernetes. It can kill pods, stress CPU, corrupt networks, simulate disk failures, and even tamper with DNS. It runs as controllers inside your cluster — the failures are real, not simulated.

I spend 20 minutes reading the docs. Then I stop reading and start attacking.

Attack 1: Pod Chaos (Kill Random Pods Every 30 Seconds)

Target: All pods in the production namespace (where I run kube-radar, job-digest, MarketPulse API, and PostgreSQL)

Experiment:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: random-pod-killer
spec:
  action: pod-kill
  mode: all
  duration: "5m"
  selector:
    namespaces:
      - production
  scheduler:
    cron: "@every 30s"

What I expected: Pods restart. Kubernetes self-heals. ArgoCD shows sync. Business as usual.

What actually happened:

Time	Event
T+0s	PostgreSQL pod killed. StatefulSet recreates it in 8 seconds.
T+30s	MarketPulse API pod killed. Deployment recreates it in 4 seconds. No impact — I have 2 replicas.
T+60s	Prometheus pod killed. Data loss: the last 15 minutes of scrape data vanished because I use `emptyDir` for Prometheus storage.
T+90s	kube-radar pod killed. No impact — it's a CLI CronJob, not a long-running service.
T+120s	PostgreSQL killed again. This time, the new pod schedules to `rpi-03`, which already has a Longhorn replica. But the active replica was on `rpi-01`. Longhorn takes 34 seconds to failover and attach the volume to `rpi-03`. During those 34 seconds, MarketPulse API returns `500` because it can't reach the database.
T+5m	Experiment ends. I have 6 minutes of missing Prometheus data and a database failover slower than my patience.

Lesson 1: Kubernetes recreates pods fast. Stateful workloads don't. A StatefulSet restart plus volume re-attachment is an eternity for a dependent service.

Attack 2: Network Partition (Isolate One Worker from the Control Plane)

Target: rpi-02 — completely isolated from all other nodes

Experiment:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: partition-rpi-02
spec:
  action: partition
  mode: all
  duration: "3m"
  selector:
    nodes:
      - rpi-02
  direction: both
  target:
    selector:
      namespaces:
        - kube-system
    mode: all

What I expected: rpi-02 goes NotReady. Workloads on it get rescheduled. Cilium handles the network split gracefully.

What actually happened:

Time	Event
	T+0s
T+15s	Cilium on `rpi-02` starts logging `endpoint regeneration failure`. It's trying to sync BPF maps with the Cilium operator but can't reach it.
T+45s	The node finally transitions to `NotReady` because the kubelet's node lease can't be renewed. But the 45-second grace period means pods on `rpi-02` are still considered "running" by the scheduler.
T+60s	A Longhorn volume with replicas on `rpi-02` and `rpi-03` loses quorum. Longhorn marks the volume as `Degraded`. Read operations still work. Write operations hang.
T+90s	MarketPulse API (running on `rpi-01`) tries to write to PostgreSQL. The write hangs because the database volume is degraded. The Go HTTP server goroutine blocks. After 30 seconds, the client times out.
T+120s	Prometheus can't scrape metrics from pods on `rpi-02` because they're network-isolated. It marks them as `DOWN`. My Grafana dashboard looks like a massacre.
T+3m	Partition removed. Cilium takes 18 seconds to rebuild BPF maps. Longhorn takes 47 seconds to re-replicate the missing replica. The cluster recovers. But the 3-minute window had 2 database write timeouts and 1 failed API request.

Lesson 2: Network partition + StatefulSet + distributed storage = distributed deadlock. The node stays Ready long enough to break things but not long enough to migrate workloads cleanly.

Attack 3: Node Chaos (Kill a Worker Node Entirely)

Target: rpi-01 — simulate total node death (power loss)

Experiment:

# Physical simulation: pull the power cable on rpi-01
# (Yes, I literally unplugged it. This is why I test at 2 AM.)

What I expected: Node goes NotReady after 40 seconds. Pods reschedule. Longhorn promotes the other replica to primary. Service continues.

What actually happened:

Time	Event
T+0s	Power cable pulled. `rpi-01` is dead. No graceful shutdown. No disk sync.
T+5s	Talos on the control plane detects the node is unresponsive. But the node controller waits for the default `--node-monitor-grace-period=40s` before marking it NotReady.
T+30s	During those 30 seconds, kube-proxy (Cilium eBPF) still routes traffic to pods on `rpi-01` because the endpoint slices haven't been updated yet. Requests to MarketPulse API hit a dead pod IP and time out.
T+40s	Node finally marked `NotReady`. Pod eviction starts. The eviction timeout is another 5 minutes by default. I didn't change it. I'm an idiot.
T+45s	Longhorn detects the node is gone. It has replicas on `rpi-02` and `rpi-03`. It promotes the `rpi-02` replica to primary. But PostgreSQL was the active workload on `rpi-01` — the pod hasn't been rescheduled yet because eviction hasn't timed out.
T+2m	I manually delete the PostgreSQL pod on `rpi-01` (it's stuck in Terminating because the node is dead). The StatefulSet creates a new pod on `rpi-02`. Longhorn attaches the volume. PostgreSQL starts crash recovery because the previous instance died uncleanly.
T+3m30s	PostgreSQL finishes WAL replay. MarketPulse API reconnects. My SIEM shows 3.5 minutes of total database unavailability.
T+5m	Kubernetes finally evicts all remaining pods from `rpi-01` automatically. They're already rescheduled by now because I intervened manually. The automatic eviction did nothing useful.

Lesson 3: Default Kubernetes timeouts are designed for cloud nodes that recover. For bare metal, 40 seconds + 5 minutes is an eternity. You need to tune --node-monitor-grace-period and pod-eviction-timeout to match your hardware reality.

Attack 4: Stress Chaos (CPU Burn on the Control Plane)

Target: Dell OptiPlex control plane — 100% CPU for 2 minutes

Experiment:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: cpu-burn-control-plane
spec:
  duration: "2m"
  selector:
    nodes:
      - optiplex-control-plane
  stressors:
    cpu:
      workers: 4
      load: 100

What I expected: Control plane slows down. API latency increases. But the cluster stays functional.

What actually happened:

Time	Event
T+0s	CPU stress applied. 4 workers pinned to 100%.
T+15s	`kubectl get nodes` latency jumps from 200ms to 4.2 seconds.
T+30s	etcd starts logging `apply request took too long`. The WAL fsync is competing with stress-ng for disk I/O.
T+60s	Cilium agent on the control plane can't reach the API server fast enough for endpoint updates. It logs `list-watcher timed out`. Pods on other nodes start losing network identity.
T+75s	ArgoCD application controller crashes with a context deadline exceeded error. It was trying to list resources and gave up. When it restarts, it re-triggers a full sync of all 45 manifests.
T+90s	Longhorn manager on the control plane can't renew its leader election lease. It steps down. The new leader takes 12 seconds to take over. During those 12 seconds, volume attachment requests queue up.
T+2m	Stress removed. CPU returns to normal. But ArgoCD is still syncing. Longhorn is still re-electing. etcd has a 200MB backlog of WAL entries to compact. It takes 90 seconds for the cluster to feel "normal" again.

Lesson 4: The control plane is not infinitely resilient. On bare metal with no CPU limits, a single noisy process can cascade through etcd → API server → Cilium → ArgoCD → Longhorn. In the cloud, this doesn't happen because the control plane is managed and isolated. On bare metal, you are the SRE.

Attack 5: DNS Chaos (Corrupt CoreDNS Responses)

Target: CoreDNS in kube-system

Experiment:

apiVersion: chaos-mesh.org/v1alpha1
kind: DNSChaos
metadata:
  name: corrupt-dns
spec:
  action: error
  mode: all
  duration: "2m"
  selector:
    namespaces:
      - kube-system
  scope: inner
  target:
    mode: all
    selector:
      namespaces:
        - production

What I expected: DNS failures. Services can't resolve hostnames. Health checks fail.

What actually happened:

Time	Event
T+0s	DNS chaos applied. CoreDNS returns `NXDOMAIN` for 50% of queries.
T+5s	MarketPulse API tries to connect to `postgresql.production.svc.cluster.local`. Gets `NXDOMAIN`. Retries. Fails. Returns `500`.
T+10s	job-digest Python scraper tries to resolve `api.linkedin.com` through cluster DNS. Gets `NXDOMAIN`. The urllib3 retry logic waits 3 seconds, retries, waits 6 seconds, retries, gives up. The CronJob fails.
T+20s	Prometheus can't resolve scrape targets. All targets show `DOWN`.
T+30s	Cilium Hubble Relay can't resolve `hubble-peer.kube-system.svc.cluster.local`. It stops receiving flow data. My network observability goes blind.
T+60s	ArgoCD can't resolve `github.com` to check for manifest updates. Sync status shows `Unknown` instead of `Synced`.
T+2m	Chaos removed. DNS recovers in 5 seconds because CoreDNS caches are in-memory. But the failed CronJob doesn't automatically re-run. The `500` errors in MarketPulse triggered my alerting. My phone buzzed. I remembered it's 2:47 AM and my wife is sleeping. I silence the alert.

Lesson 5: DNS is not a luxury. When DNS breaks, everything breaks simultaneously because everything assumes DNS works. In a homelab, you don't have Route 53 health checks or multi-region DNS. You have one CoreDNS replica and a dream.

The Scorecard

Attack	Expected	Reality	MTTR
Pod Chaos	Full recovery in <10s	34s database failover + data loss	~45s
Network Partition	Graceful isolation	3m distributed deadlock	~3m
Node Death	Auto-reschedule in 40s	3.5m manual intervention needed	~3.5m
CPU Stress	Slowdown, no outage	90s cascading recovery	~90s
DNS Chaos	Service failures	Total cluster blindness	~5s (after removal)

Summary: My cluster survived everything. But "survived" doesn't mean "handled gracefully." Every attack revealed a gap between "Kubernetes works" and "Kubernetes works when things go wrong."

What I Fixed the Next Morning

1. Tuned Kubernetes failure detection

# Kubelet configuration
nodeStatusUpdateFrequency: 10s
nodeMonitorGracePeriod: 20s
podEvictionTimeout: 60s

On bare metal, nodes don't come back. Don't wait 5 minutes to accept reality.

2. Added a second CoreDNS replica with anti-affinity

replicas: 2
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            k8s-app: kube-dns
        topologyKey: kubernetes.io/hostname

One CoreDNS death should not blind the cluster.

3. Moved Prometheus to Longhorn-backed storage

spec:
  volumeClaimTemplates:
    - metadata:
        name: prometheus-data
      spec:
        storageClassName: longhorn
        resources:
          requests:
            storage: 20Gi

emptyDir for monitoring data is a mistake. Promote it to persistent storage.

4. Configured pod disruption budgets

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: postgres-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: postgresql

This prevents voluntary evictions from destabilizing critical workloads during node drains or chaos experiments.

5. Automated the chaos experiments

I created a Kubernetes CronJob that runs a subset of these experiments every Sunday at 3 AM:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: weekly-chaos
spec:
  schedule: "0 3 * * 0"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: chaos
              image: chaos-mesh/chaos-daemon:latest
              command: ["/bin/sh", "-c"]
              args:
                - |
                  kubectl apply -f /experiments/pod-chaos.yaml
                  sleep 300
                  kubectl delete -f /experiments/pod-chaos.yaml
          restartPolicy: OnFailure

If you're not breaking it on purpose, someone will break it by accident.

3 Lessons That Transfer to Production

1. Hope is not a strategy

Before this night, my disaster recovery plan was: "I have etcd snapshots and Longhorn replicas. I'll figure it out." After 5 attacks, I have specific MTTR numbers, specific failure modes, and a specific list of what to fix. In production, "I'll figure it out" is not an SRE principle.

2. Defaults are not your friends

Kubernetes defaults are tuned for cloud providers with managed control planes, fast node replacement, and redundant infrastructure. On bare metal — or even in a cost-optimized cloud environment — those defaults are dangerously optimistic. Tune them or suffer at 2 AM.

3. Observability is the only thing that separates chaos from panic

During the DNS attack, the only reason I knew what was happening was because I had Prometheus, Grafana, and Cilium Hubble. Without them, I would have been guessing. With them, I could trace the failure chain: DNS → service resolution → API timeout → 500 response. Observability doesn't prevent failure. It prevents blind failure.

TL;DR

I installed Chaos Mesh on my homelab cluster and attacked it 5 ways: random pod kills, network partitions, node death, CPU starvation, and DNS corruption.

The cluster survived everything. But gracefully? No. Database failover took 34 seconds. Network partition caused a 3-minute distributed deadlock. Node death needed manual intervention. CPU stress triggered cascading failures through etcd, Cilium, ArgoCD, and Longhorn. DNS chaos blinded the entire cluster in 30 seconds.

I fixed 5 things the next morning: tuned failure detection, added CoreDNS redundancy, moved Prometheus to persistent storage, configured PodDisruptionBudgets, and automated weekly chaos experiments.

If you run Kubernetes anywhere — cloud, bare metal, or a drawer full of Raspberry Pis — break it yourself before the universe breaks it for you.

# Start here
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace chaos-testing --create-namespace

What's your MTTR? Have you ever actually measured it? Drop it in the comments — or admit you don't know. Both are valid.

I Automated My Entire GitOps Security Stack. The First Thing It Blocked Was My Own Salary.

Le Beltagy — Wed, 12 Aug 2026 10:00:00 +0000

I Automated My Entire GitOps Security Stack. The First Thing It Blocked Was My Own Salary.

From "zero-trust hero" to "I can't deploy my own payday fix" — how I chained kube-radar, admission webhooks, and ArgoCD into an autonomous security pipeline, the regex that deemed me a threat, and why your guardrails become prison bars when you forget the escape hatch.

The Setup

It started with a single YAML file that should never have made it to production.

I was reviewing a pull request for VehicleMetrics at 10 PM on a Thursday. A junior contributor — bless their enthusiasm — had added a new ClusterRole for a debugging sidecar. It looked innocent enough:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vehiclemetrics-debug
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

Cluster-admin in a trench coat. One kubectl apply away from "we don't know who owns this cluster anymore."

I caught it because I was looking. But what about the PRs I don't review? What about the 2 AM "hotfix" branches that bypass CODEOWNERS because someone's pager is screaming? What about the Helm chart I copy-pasted from Stack Overflow that binds a service account to pods/exec — the permission that lets anyone kubectl exec into a running pod and dump environment variables?

I manage Kubernetes at Siemens professionally. I run bare-metal clusters in my closet obsessively. And I had built NEMESIS, my purple-team tool, to attack my own infrastructure.

But I had never built something to stop the attack before it started.

So I did what any engineer with too much caffeine and a weekend would do: I automated the entire security review pipeline. No human in the loop. If the code was malicious, sloppy, or just stupid, it would never touch the cluster.

The system worked perfectly.

Then Friday at 4:47 PM happened.

Why Not Just Use Branch Protection?

You're thinking: this is a git problem, not a Kubernetes problem.

Branch protection, CODEOWNERS, required reviews — I had all of it. Here's why it's not enough:

1. YAML is a liar

That wildcard ClusterRole? It passed yamllint. It passed helm lint. It passed a human reviewer who was looking at 14 files in a 3,000-line PR. The danger wasn't in the syntax. It was in the semantics.

2. Security is boring until it's catastrophic

Nobody wants to be the reviewer who blocks a PR for three hours debating whether pods/exec is necessary. So they approve it. I know because I've done it.

3. "LGTM" is not a security control

A thumbs-up emoji doesn't enforce least privilege. A required reviewer count doesn't understand RBAC. I wanted a system that understood Kubernetes security natively, not a social protocol that assumed everyone was careful.

The Stack I Built

I took three tools I already trusted and wired them into ArgoCD's deployment pipeline:

GitHub PR
    │
    ▼
gitops-validator (GitHub App)
    ├─ kube-radar scan → RBAC wildcard / overprivilege detection
    ├─ NEMESIS static analysis → container image CVE + misconfig
    └─ kyverno-lite webhook → policy enforcement (custom rules)
    │
    ▼
ArgoCD PreSync Job
    └─ admission-controller validates the rendered manifests
    │
    ▼
Cluster (only if all gates pass)

Tool 1: kube-radar (my own Go CLI)

I wrote this when I was learning Go. It parses Kubernetes RBAC resources and scores them by risk. Wildcards = instant block. pods/exec, secrets/*, clusterroles/* without namespace restriction = flag for human review.

I containerized it and turned it into a GitHub Actions job.

# .github/workflows/gitops-security.yml
jobs:
  rbac-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: kube-radar scan
        uses: docker://ghcr.io/le-beltagy/kube-radar:v0.3
        with:
          args: scan --path ./manifests --severity critical --fail-on-critical

Tool 2: NEMESIS static analysis

The same purple-team engine that lives in my cluster as a DaemonSet has a scan mode. It reads a container image reference, pulls it into an isolated namespace, and runs Trivy + kube-bench + custom checks. If the image contains a known CVE with CVSS > 7.0, the pipeline fails.

Tool 3: kyverno-lite (custom admission webhook)

I didn't need all of Kyverno's DSL. I needed four hard rules:

No container runs as root
No image uses latest tag
No RBAC rule has * on apiGroups, resources, AND verbs
Every deployment must have resources.requests set

So I wrote a lightweight admission webhook in Go — just 400 lines — using controller-runtime. It receives AdmissionReview requests from the Kubernetes API server and returns allowed: true/false.

// webhook.go — the RBAC gatekeeper
func validateRBAC(req *admissionv1.AdmissionRequest) bool {
    var role rbacv1.ClusterRole
    json.Unmarshal(req.Object.Raw, &role)

    for _, rule := range role.Rules {
        // Rule 3: The "Deadly Asterisk"
        if slices.Contains(rule.APIGroups, "*") &&
           slices.Contains(rule.Resources, "*") &&
           slices.Contains(rule.Verbs, "*") {
            return false // ❌ DENIED
        }
    }
    return true // ✅ ALLOWED
}

I packaged it as a ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: gitops-security-webhook
webhooks:
  - name: rbac.gitops.lebeltagy.io
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["rbac.authorization.k8s.io"]
        apiVersions: ["v1"]
        resources: ["clusterroles", "roles"]
    clientConfig:
      service:
        name: gitops-webhook
        namespace: security
        path: "/validate-rbac"
    failurePolicy: Fail
    admissionReviewVersions: ["v1"]
    sideEffects: None

failurePolicy: Fail. This is the critical line. If the webhook is down, nothing gets deployed. I wanted security over availability. I would regret this later.

The ArgoCD Integration

ArgoCD has a feature most people ignore: PreSync hooks. You can run a Kubernetes Job before any sync operation. If the Job fails, the sync aborts.

I created a PreSync Job that:

Renders the Helm chart
Runs kube-radar against the rendered manifests
Runs NEMESIS against the container images referenced in the manifests
Sends a Slack notification with the scan results

apiVersion: batch/v1
kind: Job
metadata:
  name: gitops-security-gate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: validator
          image: ghcr.io/le-beltagy/gitops-validator:v1.2
          env:
            - name: REPO_URL
              value: "https://github.com/le-beltagy/vehiclemetrics"
            - name: TARGET_REVISION
              value: "HEAD"
      restartPolicy: Never

If this Job exits with code 0, ArgoCD deploys. If it exits with code 1, the sync is blocked and the Application shows SyncFailed.

I tested it on a deliberately bad PR — that wildcard ClusterRole from the beginning. The pipeline caught it. The PR was blocked. The cluster stayed safe.

I felt like a god.

Friday, 4:47 PM

It was the last workday of the month. Payroll for the Dutch startup's pilot program — my first real SaaS revenue — needed to be invoiced through the VehicleMetrics billing service.

The billing service had a bug. A timezone issue: it calculated prorated usage in UTC but invoiced in CET, overcharging the customer by exactly one day. The customer noticed. I needed to ship a fix before 5 PM or the invoice would go out wrong.

I wrote the fix in 12 minutes. One line changed in a Python utility. Tests passed. I pushed, merged, and watched ArgoCD.

The PreSync Job started.

It failed.

[gitops-validator] ERROR: kube-radar detected CRITICAL violation
[gitops-validator] File: manifests/vehiclemetrics-billing-sa.yaml
[gitops-validator] Resource: ServiceAccount/vehiclemetrics-billing
[gitops-validator] Issue: Binds to ClusterRole with pods/exec permission
[gitops-validator] SEVERITY: Critical — automatic block enabled
[gitops-validator] Exit code: 1

Wait. What?

I hadn't changed anything in vehiclemetrics-billing-sa.yaml. That file had been in the repo for weeks. Why was it failing now?

I checked the git diff. The billing fix was a one-line Python change. No RBAC touched. But the PreSync Job scans the entire rendered manifest tree, not just the diff. And kube-radar had a new rule I had merged the night before — version v0.3 — that now flagged pods/exec as critical, not just a warning.

The rule change was good. The ServiceAccount was overprivileged. But I had shipped the new kube-radar rule on Thursday evening, forgotten about it, and now on Friday at 4:52 PM, my own security pipeline was treating my production manifests as a threat.

ArgoCD showed SyncFailed.

The billing fix was not deployed.

4:55 PM: The Panic Override

I had two choices:

Option A: Fix the RBAC properly — create a restricted Role with only the necessary permissions, update the ServiceAccount binding, run the tests, commit, push, wait for the pipeline.

Estimated time: 20 minutes. It was 4:55 PM. The invoice batch job ran at 5:00 PM.

Option B: Bypass the security pipeline and force the sync.

ArgoCD lets you do this. You can click "Sync" with "Prune" and "Replace" checked. You can skip the PreSync hook. I am an admin. I have the power.

I hovered over the button.

And I realized: if I bypassed my own security gate the first time it inconvenienced me, the entire system was theater. I had built an automated bouncer and was about to sneak in through the back door because I was wearing the right jacket.

So I didn't.

5:03 PM: The Real Fix (And The Bug I Actually Shipped)

I spent 8 minutes — invoice job be damned — writing a proper Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vehiclemetrics-billing
  namespace: vehiclemetrics-prod
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

Removed pods/exec. Removed the ClusterRole binding. Applied the principle of least privilege.

Committed. Pushed. Pipeline passed. ArgoCD synced at 5:02 PM.

The invoice job? It ran at 5:03 PM. With the correct timezone fix. The customer never knew how close they came to a bad bill — or how close I came to disabling my own security stack.

But here's the part that haunts me:

The original pods/exec permission wasn't even needed. It was a copy-paste from a debugging session three weeks ago. I had left it in the manifest because "it worked" and I was too lazy to scope it down.

My security pipeline didn't create a problem. It revealed a problem I had been ignoring. The thing that blocked my salary was the thing that was already wrong.

The Aftermath: 48 Hours of Paranoia

I spent the weekend reviewing every manifest in the repo. Here's what I found:

File	Issue	Risk
`debug-namespace/role.yaml`	`verbs: ["*"]` on ConfigMaps	Any pod can read secrets mounted as ConfigMaps
`monitoring/sa.yaml`	ServiceAccount bound to `cluster-admin`	Prometheus can read all secrets
`temp/backup-job.yaml`	Container runs as root	Privilege escalation vector
`ingress/traefik-rbac.yaml`	`apiGroups: ["*"]` on 3 resources	Over-scoped for ingress needs

Four critical issues. In my own repo. That I had written or approved.

Without the automated gate, they would have stayed there until someone exploited them — or until a compliance audit found them and I had to explain why a billing service could exec into pods.

What I Changed (The No-Escape-Hatch Problem)

The system was right to block me. But the system was also dangerous because it had no emergency override that didn't require me to become a liar.

Here's the architecture now:

GitHub PR
    │
    ▼
gitops-validator (GitHub App)
    ├─ kube-radar scan → CRITICAL = block PR
    ├─ NEMESIS scan → CRITICAL = block PR
    └─ kyverno-lite webhook → policy check
    │
    ▼
ArgoCD PreSync Job
    └─ Full validation rerun
    │
    ▼
ValidatingWebhookConfiguration (cluster gate)
    │
    ▼
Cluster

The fix: I added a @security-override label. If a PR is labeled with this, the pipeline still runs, still reports every violation, but emits a warning instead of a block. The label can only be applied by a GitHub Team called security-admins, which has exactly one member: me. And every override is logged to a dedicated Slack channel and a write-once S3 bucket.

I also changed failurePolicy: Fail to failurePolicy: Ignore on the admission webhook, with a twist: if the webhook is unreachable, ArgoCD flags the Application as Unknown and pauses automated syncs. Security is enforced when healthy. Availability is preserved when degraded.

webhooks:
  - name: rbac.gitops.lebeltagy.io
    # ...
    failurePolicy: Ignore  # Don't crash deploys if webhook is down

But the real fix wasn't technical. It was procedural:

I stopped treating "works" as the standard. "Least privilege" is the standard.

The Numbers

Metric	Before (manual review)	After (automated gates)
RBAC violations in prod	4 known, unknown unknowns	0 (all caught in CI)
CVEs deployed to cluster	~3 per month (after-the-fact scans)	0 (blocked in PreSync)
Time to review a PR	45 min avg (human)	3 min (automated) + human for exceptions
False positive rate	N/A	~5% (tunable via severity threshold)
Times I almost disabled my own salary	0	1

What I'd Do Differently

1. Don't ship new scanner rules on Thursday night

If you're changing what "critical" means, do it Monday morning when you have the week to deal with the blast radius. Not the day before you might need to deploy a hotfix.

2. Every guardrail needs a documented escape hatch

Not a secret backdoor. A visible, audited, tightly-scoped bypass. If your emergency procedure is "log in as admin and disable the thing," you don't have security. You have security theater with an intermission.

3. Scan the diff, not the world

The PreSync Job originally scanned the entire manifest tree. Now it scans only the Helm release diff: what changed, not what exists. Existing bad configs get flagged in a weekly full scan, not during hotfix deployments.

4. Your production manifests are dirtier than you think

I promise you. Go run kube-radar or Popeye or any RBAC scanner against your cluster right now. You'll find something embarrassing. The question isn't whether you have debt — it's whether you have a system that finds it before your attacker does.

Why I'm Keeping It

It's been three weeks since the Friday incident. The Dutch pilot expanded to a second customer. I haven't had a single 3 AM security scare. And when my latest contributor opened a PR with pods/exec in it, the pipeline blocked it before I even saw the notification.

I didn't have to be the bad cop. The code was.

Is automated GitOps security more work to maintain? Yes. I spend maybe an hour per week tuning rules and reviewing override logs.

Is it worth it? Last week, a penetration tester — hired by the second customer — spent two days trying to escalate privileges in the cluster. He found one over-scoped Role. It was in a staging namespace with no production data.

He wrote in his report: "The target environment exhibits unusually robust RBAC hygiene for an early-stage SaaS platform."

That sentence was worth every minute.

TL;DR — The "Don't Block Your Own Salary" Checklist

[ ] Run an RBAC scanner against your cluster today (kube-radar, Popeye, or rbac-audit)
[ ] Add a PreSync security gate to ArgoCD/Flux before your next deploy
[ ] Ship new scanner rules on Monday, not Thursday
[ ] Build an override mechanism that is audited, not secret
[ ] Change failurePolicy: Fail to Ignore if you don't have 24/7 webhook SREs
[ ] Scan the diff for deploy gates, scan the world for weekly audits
[ ] Remember: the pipeline that blocks you is the pipeline that saves you

Want the admission webhook code + ArgoCD PreSync manifests? Drop a comment — I'll open-source the gitops-validator repo if there's interest.

Ever been locked out of your own system by your own automation? Tell me your war story below. We can start a support group.

Tags: #kubernetes #security #gitops #devops #go #argocd #rbac #automation #platformengineering

I Replaced kube-proxy with eBPF in Production (And Why My Monitoring Went Blind for 6 Hours)

Le Beltagy — Thu, 06 Aug 2026 19:42:26 +0000

I Replaced kube-proxy with eBPF in Production (And Why My Monitoring Went Blind for 6 Hours)

From a "simple" Cilium upgrade to a 6-hour outage where my SIEM couldn't see a single packet — the real story of deleting kube-proxy, the eBPF program that saved me, and why the docs never warned me about the one metric that matters.

The Setup

It started with a cilium upgrade command I ran on a Tuesday evening.

My homelab cluster — the same 4-node bare-metal setup I wrote about last month (Dell OptiPlex + 3 Raspberry Pis, Talos Linux, Cilium, ArgoCD, Longhorn) — was running Cilium 1.15 in kube-proxy-replacement partial mode. Cilium was handling some traffic, but kube-proxy's iptables rules were still doing the heavy lifting for NodePort and ClusterIP services.

I'd read the Cilium 1.16 release notes. The eBPF kube-proxy replacement was now "production-ready for all environments." The Talos Linux docs had a single bullet point: "Set kubeProxyReplacement: true in Cilium values."

It looked harmless. It looked like flipping a switch.

So I did.

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=auto \
  --set k8sServicePort=6443

The Cilium pods rolled. The nodes stayed up. kubectl get pods -A showed everything running.

I went to bed.

At 2:47 AM, my phone buzzed. Not an alert from the cluster. An alert from my SIEM — or rather, the absence of data from it. The log ingestion rate for kubernetes-audit and network-flow had flatlined to zero.

The cluster was "healthy." The monitoring was dead.

This is not a tutorial. This is an autopsy of what happens when you delete the datapath that your security stack depends on, without knowing it.

Why Not Just Keep kube-proxy?

I run Kubernetes professionally at Siemens. I know what managed EKS gives you: kube-proxy in iptables mode, happily translating ClusterIP virtual IPs into pod endpoints, maintained by AWS, never thought about.

But I was hitting a wall.

1. iptables doesn't scale

In my homelab, 45 manifests and ~120 services meant ~3,500 iptables rules across all nodes. Every new service triggered a full iptables-restore that locked the kernel's netfilter table for hundreds of milliseconds. On a Raspberry Pi 4, that's a noticeable latency spike.

2. The gap between "cluster works" and "I understand the datapath"

kube-proxy is a black box. You run it, it works, you ignore it. But if you can't explain how a packet flows from Service:80 to Pod:8080 at the kernel level, do you really own your cluster?

3. eBPF gives you superpowers — if you know the cost

Cilium's eBPF programs can:

Replace conntrack with BPF map-based connection tracking
Provide socket-level load balancing (no NAT needed for pod-to-pod)
Deliver Hubble flow logs with L3-L7 visibility
Enforce network policies before the packet hits the host stack

But eBPF isn't magic. It's a different universe. And the tools that read the old universe don't work in the new one.

The Architecture (Before and After)

Before: kube-proxy + Cilium (Hybrid)

┌─────────────────────────────────────────────────────────────────────┐
│                         Worker Node                                  │
│                                                                     │
│  ┌─────────────┐    ┌──────────────────┐    ┌──────────────────┐   │
│  │   Pod A     │───►│   ClusterIP      │───►│   kube-proxy     │   │
│  │  (app)      │    │   Service        │    │   (iptables)     │   │
│  └─────────────┘    └──────────────────┘    └────────┬─────────┘   │
│                                                      │              │
│                                     ┌────────────────▼─────────┐   │
│                                     │   iptables NAT/REDIRECT   │   │
│                                     │   -A KUBE-SVC-XXX ...     │   │
│                                     └────────────────┬─────────┘   │
│                                                      │              │
│                                     ┌────────────────▼─────────┐   │
│                                     │   conntrack table         │   │
│                                     │   /proc/net/nf_conntrack  │   │
│                                     └────────────────┬─────────┘   │
│                                                      │              │
│                                     ┌────────────────▼─────────┐   │
│                                     │   Pod B (target)          │   │
│                                     └──────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘
                       ▲
                       │
    ┌──────────────────┼──────────────────┐
    │  SIEM reads      │  Falco reads     │  Security tools
    │  iptables rules  │  conntrack       │  depend on netfilter
    │  for flow logs   │  for connections │
    └──────────────────┘──────────────────┘

After: Cilium eBPF (kube-proxy-deleted)

┌─────────────────────────────────────────────────────────────────────┐
│                         Worker Node                                  │
│                                                                     │
│  ┌─────────────┐    ┌──────────────────┐    ┌──────────────────┐   │
│  │   Pod A     │───►│   ClusterIP      │───►│   Cilium Agent   │   │
│  │  (app)      │    │   Service        │    │   (eBPF)         │   │
│  └─────────────┘    └──────────────────┘    └────────┬─────────┘   │
│                                                      │              │
│                                     ┌────────────────▼─────────┐   │
│                                     │   eBPF Programs           │   │
│                                     │   ├─ bpf_sock (socket LB) │   │
│                                     │   ├─ bpf_lxc (pod egress) │   │
│                                     │   ├─ bpf_host (NodePort)  │   │
│                                     │   └─ bpf_network (ingress)│   │
│                                     └────────────────┬─────────┘   │
│                                                      │              │
│                                     ┌────────────────▼─────────┐   │
│                                     │   BPF Maps (not iptables) │   │
│                                     │   cilium_lb4_services_v2  │   │
│                                     │   cilium_ct4_global       │   │
│                                     └────────────────┬─────────┘   │
│                                                      │              │
│                                     ┌────────────────▼─────────┐   │
│                                     │   Pod B (target)          │   │
│                                     └──────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘
                       ▲
                       │
    ┌──────────────────┘
    │  SIEM: "WHERE ARE MY FLOW LOGS?!"
    │  Falco: "I CAN'T SEE CONNTRACK!"
    │  Legacy tools: broken by design

Design Decision

Component	Before (kube-proxy)	After (eBPF)	Impact
Service Load Balancing	iptables DNAT	eBPF socket-level redirect	Latency: -30%
Connection Tracking	conntrack kernel module	eBPF BPF map (`cilium_ct4_global`)	No `/proc/net/nf_conntrack`
NAT	iptables MASQUERADE	eBPF SNAT/DNAT	No `iptables -t nat -L` output
Network Policy Enforcement	iptables + ipsets	eBPF TC/XDP programs	Earlier drop, better performance
Flow Logs	Conntrack events → Falco	Hubble eBPF exporter	Different schema, different source
NodePort	iptables REDIRECT	eBPF `bpf_host`	Same port, different kernel path

The Failures (And What Each One Taught Me)

Failure 1: I Upgraded and Went to Bed

I ran the Cilium upgrade at 11 PM. The pods cycled green. I checked kubectl get svc and curled a NodePort service from my laptop. It worked.

What I didn't check: whether Falco was still emitting network connection events.

Falco's default rules include this:

# Falco rule: outbound network connection
- rule: Outbound Connection
  desc: Detect outbound connections from pods
  condition: >
    outbound and
    fd.name != "" and
    (fd.typechar = '4' or fd.typechar = '6')
  output: "Outbound connection from pod (command=%proc.cmdline connection=%fd.name)"
  priority: NOTICE

Falco detects these connections by reading /proc/<pid>/fd/ and correlating with conntrack state. When kube-proxy's iptables rules disappeared, Falco's fd.name resolution for ClusterIP destinations broke — the eBPF path doesn't leave the same /proc artifacts that Falco's syscall-based engine expects.

Result: Falco emitted zero network alerts for 7 hours. Not because there was no traffic. Because it couldn't see the traffic the same way.

Lesson 1: "Services work" is not "security monitoring works." Test both.

Failure 2: My SIEM Couldn't Parse Hubble

At 2:47 AM, my phone woke me up. Grafana showed a flat line for network_flow_events_per_minute.

I checked the SIEM (a custom Fluent Bit → Kafka → ClickHouse pipeline I'd built). The Fluent Bit pods were running. The Kafka topics existed. The ClickHouse tables were healthy.

But the kubernetes.flow_logs stream had zero rows since 23:12.

Here's why:

My SIEM consumed network flow logs from two sources:

# /etc/fluent-bit/inputs.conf
[INPUT]
    Name              tail
    Path              /var/log/audit/kube-apiserver-audit.log
    Parser            json
    Tag               k8s.audit

[INPUT]
    Name              tail
    Path              /var/log/conntrack/flows.json
    Parser            json
    Tag               k8s.network-flows

The second input — conntrack/flows.json — was populated by a small sidecar that read /proc/net/nf_conntrack every 30 seconds and emitted JSON.

With eBPF replacing conntrack entirely, /proc/net/nf_conntrack was empty on every node. Cilium uses its own BPF map for connection tracking, not the kernel's conntrack module.

So my sidecar emitted nothing. Fluent Bit had nothing to tail. Kafka got empty batches. ClickHouse had nothing to insert.

The SIEM was blind. Not broken — blind by design, because I had built it around netfilter assumptions that eBPF invalidated.

Lesson 2: eBPF isn't a drop-in replacement. It's a migration. Treat it like one.

Failure 3: I Panic-Downgraded and Made It Worse

At 3:30 AM, I decided to roll back.

helm rollback cilium 1

Cilium reverted to the previous version. kube-proxy wasn't re-deployed automatically (I'd deleted the DaemonSet during the upgrade to "save resources").

Now I had no kube-proxy and no eBPF replacement.

ClusterIP services stopped resolving entirely. DNS inside the cluster broke (CoreDNS is a ClusterIP service). My apps couldn't reach the API server. ArgoCD went into a crash loop.

I had to:

SSH into the Talos API (not SSH — talosctl)
Re-apply the kube-proxy manifest manually
Wait for iptables rules to populate
Restart CoreDNS pods
Fix ArgoCD's sync state

Total downtime: 94 minutes.

Lesson 3: Don't delete the old path until the new path is fully monitored. I had eBPF working, but I hadn't validated monitoring. Deleting kube-proxy killed my safety net.

The Fix (And What Actually Works)

Step 1: Enable Hubble (Not Optional)

I had installed Cilium with Hubble disabled to "save resources on the Pis." That was stupid. Hubble is the only source of flow visibility in an eBPF-only cluster.

# cilium-values.yaml
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns:query
      - drop
      - tcp
      - flow
      - icmp
      - http
  # CRITICAL: Enable Hubble export for SIEM consumption
  export:
    static:
      enabled: true
      file:
        enabled: true
        path: /var/log/hubble/flows.json
        format: JSON
      fieldMask: []

This writes flow logs to /var/log/hubble/flows.json on each node in a structured JSON format.

Step 2: Rewrite the SIEM Pipeline

I replaced the conntrack sidecar with a Hubble reader:

# fluent-bit-daemonset.yaml
volumeMounts:
  - name: hubble-logs
    mountPath: /var/log/hubble
    readOnly: true

volumes:
  - name: hubble-logs
    hostPath:
      path: /var/log/hubble
      type: DirectoryOrCreate

And updated the Fluent Bit input:

[INPUT]
    Name              tail
    Path              /var/log/hubble/flows.json
    Parser            json
    Tag               k8s.network-flows
    # Hubble JSON has nested objects; flatten them
    Mem_Buf_Limit     50MB

Step 3: Schema Mapping (The Annoying Part)

Conntrack JSON and Hubble JSON have completely different schemas.

Old (conntrack):

{
  "src_ip": "10.0.1.15",
  "dst_ip": "10.0.2.88",
  "src_port": 54321,
  "dst_port": 443,
  "protocol": "tcp",
  "state": "ESTABLISHED",
  "timestamp": "2026-08-05T23:12:00Z"
}

New (Hubble):

{
  "flow": {
    "time": "2026-08-06T03:45:00Z",
    "verdict": "FORWARDED",
    "ethernet": {"source": "...", "destination": "..."},
    "IP": {"source": "10.0.1.15", "destination": "10.0.2.88", "ipVersion": "IPv4"},
    "l4": {
      "TCP": {"source_port": 54321, "destination_port": 443}
    },
    "source": {"identity": 12345, "namespace": "frontend", "pod_name": "api-xxx"},
    "destination": {"identity": 67890, "namespace": "backend", "pod_name": "db-yyy"},
    "event_type": {"type": 4, "sub_type": 3},
    "is_reply": false
  }
}

Hubble gives you way more — pod identity, namespace, Kubernetes labels, L7 protocol details, policy verdict. But you have to rewrite your SIEM parsers.

I wrote a small Go transform (ironic, given how I learned it) to flatten Hubble JSON into the old ClickHouse schema, with extra columns for the new metadata:

// hubble-flatten/main.go
package main

import (
    "encoding/json"
    "fmt"
    "os"
    "time"
)

type HubbleFlow struct {
    Flow struct {
        Time     string `json:"time"`
        Verdict  string `json:"verdict"`
        IP       struct {
            Source      string `json:"source"`
            Destination string `json:"destination"`
        } `json:"IP"`
        L4 struct {
            TCP struct {
                SourcePort      uint32 `json:"source_port"`
                DestinationPort uint32 `json:"destination_port"`
            } `json:"TCP"`
        } `json:"l4"`
        Source      HubIdentity `json:"source"`
        Destination HubIdentity `json:"destination"`
    } `json:"flow"`
}

type HubIdentity struct {
    Identity  uint32 `json:"identity"`
    Namespace string `json:"namespace"`
    PodName   string `json:"pod_name"`
}

func main() {
    // Reads Hubble JSON from stdin, emits flat JSON to stdout
    decoder := json.NewDecoder(os.Stdin)
    encoder := json.NewEncoder(os.Stdout)

    for decoder.More() {
        var h HubbleFlow
        if err := decoder.Decode(&h); err != nil {
            continue
        }

        flat := map[string]interface{}{
            "timestamp":        parseTime(h.Flow.Time),
            "src_ip":           h.Flow.IP.Source,
            "dst_ip":           h.Flow.IP.Destination,
            "src_port":         h.Flow.L4.TCP.SourcePort,
            "dst_port":         h.Flow.L4.TCP.DestinationPort,
            "protocol":         "tcp",
            "verdict":          h.Flow.Verdict,
            "src_namespace":    h.Flow.Source.Namespace,
            "src_pod":          h.Flow.Source.PodName,
            "dst_namespace":    h.Flow.Destination.Namespace,
            "dst_pod":          h.Flow.Destination.PodName,
            "src_identity":     h.Flow.Source.Identity,
            "dst_identity":     h.Flow.Destination.Identity,
        }
        encoder.Encode(flat)
    }
}

func parseTime(s string) int64 {
    t, _ := time.Parse(time.RFC3339Nano, s)
    return t.Unix()
}

Step 4: Replace Falco Network Rules with Hubble + Cilium Policy

Falco's syscall-based network detection doesn't work well in eBPF-only clusters because:

eBPF socket load balancing happens before the syscall layer Falco hooks
ClusterIP connections never appear as "outbound to external IP" because they're redirected at the socket layer

Solution: Use Cilium Network Policies for enforcement, Hubble for observability, and keep Falco for what it's good at (file integrity, process execution, privilege escalation).

# Example: Instead of Falco detecting unauthorized DB connections,
# use Cilium NetworkPolicy to deny them at the kernel level
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: backend-access-control
  namespace: frontend
spec:
  endpointSelector:
    matchLabels:
      app: frontend-api
  egressDeny:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: backend
            app: postgres
      toPorts:
        - ports:
            - port: "5432"
          rules:
            http:
              - method: "GET"
  # But allow the legitimate API service in backend
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: backend
            app: backend-api

This enforces at eBPF TC layer — before the packet even leaves the pod's network namespace. Falco could only tell you after it happened.

The Real Numbers

Performance Impact

Metric	kube-proxy (iptables)	Cilium eBPF	Delta
Service Latency (p99)	2.8ms	1.1ms	-61%
NodePort Throughput	1.2 Gbps	3.8 Gbps	+217%
Connection Tracking Overhead	18% CPU (conntrack)	4% CPU (BPF)	-78%
New Service Propagation	30-60s (iptables sync)	<1s (BPF map update)	-98%
Memory (per node)	185MB (conntrack table)	42MB (BPF maps)	-77%

Observability Impact

Capability	Before (conntrack)	After (Hubble eBPF)
Pod-to-Pod visibility	IP only	Pod name, namespace, labels, identity
L7 protocol detection	No	HTTP method, path, headers (with L7 proxy)
Policy verdict	No	`FORWARDED`, `DROPPED`, `AUDIT` per packet
DNS visibility	No	Query name, response code, TTL
Encryption	No visibility	WireGuard vs cleartext flagged per flow
Legacy SIEM compatibility	✅ Native	❌ Requires Hubble export rewrite

The One Packet That Broke Everything

Here's the specific failure that cost me 6 hours. I found it in the Hubble flow logs after I got everything working.

At 23:12:47, my Fluent Bit-to-Kafka pod in the logging namespace tried to connect to the Kafka bootstrap broker at kafka.logging.svc.cluster.local:9092.

With kube-proxy, this resolved to 10.43.120.15:9092 via iptables DNAT. Falco saw the connection to 10.43.120.15:9092 and logged it.

With eBPF socket load balancing, Cilium's bpf_sock program intercepted the DNS resolution and redirected the connection inside the kernel socket layer directly to the Kafka pod's IP (10.0.2.88:9092) without ever creating a conntrack entry or hitting iptables.

Falco's syscall probes saw a connection to 10.0.2.88:9092 but couldn't resolve that IP back to a Kubernetes service (there's no iptables rule mapping it anymore). So Falco discarded the event as "unknown destination."

My SIEM's conntrack scraper saw nothing because conntrack was empty.

Hubble saw everything — source pod, destination pod, protocol, policy verdict — but I hadn't enabled Hubble export yet.

That single Kafka bootstrap connection was the canary. Every pod in the cluster was connecting to services via eBPF redirection, and my entire security stack was missing all of them.

Why I'm Never Going Back

Despite the 6-hour panic, I'm keeping eBPF. Here's why:

1. Performance is absurd

61% latency reduction on service-to-service calls. On Raspberry Pi 4 nodes, that's the difference between "usable" and "painful."

2. Observability is actually better — once you migrate

Hubble flow logs tell me not just that pod A talked to pod B, but which Kubernetes identities were involved, whether a network policy allowed or dropped it, and what DNS query initiated it. Falco + conntrack could never give me that.

3. Security enforcement is earlier

Cilium Network Policies drop packets at the TC (Traffic Control) layer inside the kernel, before they ever hit the pod's network stack. iptables policies happen later. Earlier drop = smaller blast radius.

4. No more iptables lock contention

Adding a new service no longer triggers a 300ms iptables-restore that freezes all network paths. BPF map updates are atomic and microsecond-fast.

What I'd Do Differently

Enable Hubble BEFORE deleting kube-proxy. Not after. Not "to save resources." It's not optional.
Run both datapaths in parallel for 48 hours. Cilium supports kubeProxyReplacement=disabled with partial eBPF. I should have tested monitoring against both paths before committing.
Validate SIEM parsers against Hubble JSON before the switch. Don't discover schema incompatibilities at 3 AM.
Don't delete the kube-proxy DaemonSet. Scale it to zero replicas instead. If eBPF breaks, you can scale it back up in 30 seconds instead of manually re-applying manifests.

TL;DR

Replaced kube-proxy with Cilium eBPF for performance
eBPF works great for traffic — breaks legacy security tools
Falco's syscall-based network detection fails with socket-level load balancing
SIEMs consuming conntrack data go blind — eBPF uses BPF maps, not netfilter
Enable Hubble export before the switch — it's your only visibility source
Hubble gives BETTER data (pod identity, policy verdict, L7) but requires parser rewrites
Performance gains are real: -61% latency, +217% throughput, -78% CPU overhead
Don't delete kube-proxy — scale to zero. Keep your escape hatch.

GitHub: beltagyy/homelab (Talos + Cilium configs will be public this week)

Author: Mohamed ElBeltagy (@beltagyy) — Cloud Security & K8s engineer at Siemens

Topic: #kubernetes #ebpf #cilium #security #devops #homelab #performance

Next article: I wrote a custom eBPF program to detect lateral movement in Kubernetes — and ran it against my own cluster. Want to see what it found?

Great article for security concepts

Le Beltagy — Sun, 02 Aug 2026 12:53:24 +0000

Le Beltagy

Jul 23

I Deployed a Purple Team Inside My Kubernetes Cluster. The Scanners Never Saw It Coming.

#kubernetes #security #go #purpleteam

5 min read

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s.

Le Beltagy — Sat, 01 Aug 2026 10:07:53 +0000

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s.

From zero Go experience to a merged PR in the most popular Kubernetes terminal UI — how I misread HTTP/1.1 upgrades, broke my own test suite, and learned that create and get are not the same verb when Kubernetes 1.31 changes the rules underneath you.

The Setup

It started with a single function I didn't write.

I'd been contributing to k9s for a week, trying to understand how a 58,000-star project structured its Go code. I was deep in internal/dao/port_forwarder.go, staring at a block of code that checked whether a user was authorized to open a port-forward on a pod. The check was simple: does this service account have create permission on pods/portforward?

I'd read the Kubernetes docs. Port-forward requires create. Everyone said so. The code worked. I moved on.

Then someone opened an issue: "Port-forward fails on k8s 1.31 with restricted RBAC."

Kubernetes 1.31 introduced the PortForwardWebsockets feature — a new code path that forwards ports over WebSockets instead of SPDY. And WebSockets, unlike SPDY, only require the get verb on the pods/portforward subresource.

That was the bug. The code was checking create. The new WebSocket path needed get. Users with get-only access couldn't port-forward in k9s, even though kubectl worked fine for them.

This is the story of how I tried to fix it, failed the first time, and learned more about Go, HTTP upgrades, and Kubernetes authorization in three days than in three weeks of tutorials.

This is not a tutorial. This is an autopsy of a two-line fix that took me forty-eight hours.

Why Not Just Use kubectl?

I use kubectl port-forward professionally. It works. It handles the WebSocket migration transparently. There's no bug report. No edge case. No problem.

But k9s is not kubectl. k9s is a terminal UI that wraps client-go and presents a unified interface for pod interaction. When k9s shows a pod and lets you press Shift-F to forward a port, it first checks if you're allowed to do it. And that check was wrong.

The issue wasn't academic. A user with this RBAC role:

rules:
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["get"]

could run kubectl port-forward successfully on Kubernetes 1.31+ (WebSocket path), but k9s would silently grey out the port-forward option, convinced they lacked permission.

That user was real. The issue had reproduction steps. A real cluster, a real role, a real workflow broken by a verb mismatch I would never have noticed without reading the client-go source.

The k9s Architecture

┌─────────────────────────────────────────────────────────────┐
│                         k9s TUI                              │
│                    (tview / terminal UI)                     │
│  User presses Shift-F on a pod                              │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│              internal/dao/port_forwarder.go                  │
│                                                              │
│  1. Check pod is Running ( readiness gate )                 │
│  2. Authorize: can user create pods/portforward?            │
│         ▲                                                    │
│         │ THIS WAS THE BUG                                   │
│  3. If yes → open port-forward session                      │
│     If no  → grey out option / show error                   │
└──────────────────────┬──────────────────────────────────────┘
                       │
              ┌────────┴────────┐
              │                 │
              ▼                 ▼
    ┌─────────────┐     ┌─────────────┐
    │ SPDY path   │     │ WebSocket   │
    │ (legacy)    │     │ (k8s 1.31+) │
    │ needs       │     │ needs       │
    │ CREATE verb │     │ GET verb    │
    └─────────────┘     └─────────────┘
              │                 │
              └───────┬─────────┘
                      ▼
            ┌─────────────────┐
            │  K8s API Server │
            │  /api/v1/.../   │
            │  portforward    │
            └─────────────────┘

The Decision I Missed

Path	Protocol	Required Verb	k9s Checked?
Legacy SPDY	SPDY/HTTP/1.1 upgrade	`create` on `pods/portforward`	✅ Yes
WebSocket	WebSocket (RFC 6455)	`get` on `pods/portforward`	❌ No

k9s only checked for create. It didn't know about the WebSocket path's get requirement. So get-only users were blocked even though they were authorized for the code path their cluster actually used.

The Failures (And What Each One Taught Me)

Failure 1: I Changed One Word and Called It a Day

My first fix was embarrassingly naive.

I found the function in port_forwarder.go. It built an SelfSubjectAccessReview with verb "create" and sent it to the API server. I changed it to "get". It compiled. I opened a draft PR.

Within an hour, a maintainer commented: "This breaks backward compatibility for clusters still on the SPDY path."

I had replaced one hardcoded verb with another. I didn't check SPDY. I didn't check both. I didn't even think about clusters running Kubernetes <1.31, or clusters where the WebSocket feature gate was disabled. I assumed "new = right."

Root cause: I treated a protocol negotiation problem like a string replacement problem.

The fix: The code needed to check BOTH verbs independently. If create is authorized, the user can port-forward via SPDY. If get is authorized, they can port-forward via WebSocket. If either is authorized, k9s should allow the operation. The API server and client-go handle which protocol actually gets used at connect time.

// What I wrote first (WRONG — only checks get)
if !utils.CheckPodPortFwd(a.factory.Client(), a.factory.Config(), path) {
    return errors.New("insufficient permission")
}

// What merged (RIGHT — checks create AND get independently)
if !utils.CheckPodPortFwd(a.factory.Client(), a.factory.Config(), path) {
    if !utils.CheckPodPortFwdGet(a.factory.Client(), a.factory.Config(), path) {
        return errors.New("insufficient permission")
    }
}

Actually, the final merged code was cleaner — it added a new CheckPodPortFwdGet helper and called both from the port-forward check. The key insight: create and get are not alternatives that replace each other. They're independent capabilities for independent protocol paths. Both can coexist. Neither can be dropped.

Lesson: When a platform supports multiple protocols, a fix for the new one cannot delete support for the old one. Test for coexistence, not replacement.

Failure 2: I Blew Up the Test Suite Because I Didn't Understand `t.Parallel()`

k9s has a real test suite. Not toy tests — table-driven tests with ~50 test cases across multiple scenarios. I added a test that covered the new WebSocket path. It passed locally.

Then CI failed.

The error was a data race in the mock client. I'd instantiated a shared mock RestClient outside the test loop to save setup code. Some tests ran with t.Parallel(). Two parallel tests mutated the same mock's response state simultaneously.

The race detector (go test -race) caught it and failed the build.

Root cause: I thought I was being efficient by sharing mock setup. I was actually creating a concurrency hazard in a test file I didn't fully understand.

The fix: I refactored each test case to build its own RestClient mock inside the test closure. No shared state. The test file I added ended up being 223 lines — almost all of it test cases for every combination of permissions:

Pod not running
No get permission, no create permission → blocked
create-only permission → allowed (legacy SPDY)
get-only permission → allowed (WebSocket path, this was the fix)
Both permissions → allowed

// Part of the 223 lines of tests I ended up writing
{
    name: "get-only-portforward-allowed",
    pod:  runningPod,
    authorized: map[string]bool{
        "selfsubjectaccessreviews": true,
        "pods":                   true,
        "portforwardget":         true,
    },
    want: true,
},

Lesson: Parallel tests are not free. If you don't own the test infrastructure, assume shared state is forbidden until proven otherwise. A data race in a test is a resume stain.

Failure 3: I Didn't Know What an HTTP/1.1 Upgrade Was

The real bug was deeper than a verb string. I needed to understand WHY get was sufficient for WebSockets but not for SPDY.

SPDY (the legacy protocol) is initiated by an HTTP POST request to the port-forward endpoint. HTTP POST maps to the create verb in Kubernetes authorization. So SPDY port-forwards require create.

WebSockets are initiated differently. They start with an HTTP GET request that includes an Upgrade: websocket header. The server responds with 101 Switching Protocols, and the connection becomes a WebSocket.

Because the initial request is GET, Kubernetes authorization maps it to the get verb on the pods/portforward subresource.

I didn't know any of this when I opened the issue. I spent an evening reading:

RFC 6455 (The WebSocket Protocol)
Kubernetes client-go source for StreamWithContext
The KEP for PortForwardWebsockets (KEP-4006)
k9s own issue history for how it handles client-go deprecation

Root cause: I thought port-forward was "one API call." It's actually a protocol negotiation dance between client-go, the API server, and the kubelet. The authorization model is protocol-dependent.

The fix: I didn't change the fix. The code was correct once I checked both verbs. But I did update the error message to indicate both acceptable verbs, so users who were blocked would know exactly which permissions they needed:

return fmt.Errorf("user is not authorized to create or get portforward %q", path)

Lesson: Before you fix a bug in a distributed system, understand the protocol it's built on. The symptom was a missing verb. The cause was a protocol upgrade. The fix that didn't understand the protocol would have broken the legacy path.

The PR

Stat	Value
Changed files	2
Lines added	240
Lines deleted	3
Files touched	`internal/dao/port_forwarder.go`, `internal/dao/port_forwarder_test.go`
Tests added	6 test cases covering every permission combination
Review rounds	2
Time from first draft to merge	48 hours

The diff is small. The logic is simple. The effort was in understanding why the logic needed to exist at all — and proving it with tests that would catch someone else making my first-draft mistake.

PR title: fix(dao): allow port-forward with 'get' verb on pods/portforward for K8s 1.31+ WebSocket path

It merged cleanly. No follow-up fixes needed.

What I Learned About Contributing to Big Projects

1. Start with the issue, not the code

I read the original issue (#4144) three times before I touched a file. The reporter included their Kubernetes version, their RBAC role, and the exact error message. Without that reproduction, I would have never understood the WebSocket migration context.

2. Read the test file before the source file

port_forwarder_test.go taught me more about how k9s handles authorization than port_forwarder.go. Tests are documentation that can't lie.

3. `go test -race` is not optional

The race detector caught my bug before a maintainer did. It turned an embarrassing review comment into a private CI failure. Run it locally. Always.

4. Two-line fixes need two-hundred-line tests

The production change was ~20 lines. The tests were 223 lines. That's the ratio in production Go. If your fix doesn't have a test that would have failed before the fix, your fix isn't done.

5. Backward compatibility is a constraint, not a suggestion

My first draft worked on Kubernetes 1.31. It would have broken k9s for half the user base still on 1.30 or older. A real fix supports both paths simultaneously. This is what separates "it works on my machine" from "it merged into k9s."

Why This Matters (And Why It Doesn't)

It matters because:

k9s has ~58,000 GitHub stars and thousands of daily users. My fix affects real people with real clusters.
I learned Go by reading client-go source and writing table-driven tests, not by building TodoMVC.
I now understand Kubernetes authorization, HTTP upgrades, and protocol negotiation at a level I didn't before.

It doesn't matter because:

It's a 20-line fix. Senior engineers at Google write diffs like this before coffee.
I didn't architect a new subsystem. I didn't refactor the codebase. I fixed a verb check.
The value isn't in the code. The value is in the proof that I can read an issue, understand the context, write a correct fix, survive review, and ship.

That's the signal I'm building.

What's Next?

I'm continuing to contribute to k9s. The codebase is complex enough to keep teaching me things — how tview renders terminal UIs, how k9s watchers avoid polling the API server, how the dao layer abstracts client-go operations.

I also contributed to Checkov today — adding missing GCP taggable resources for Cloud SQL and GKE clusters. Small PR. Two lines. Merged in 20 minutes. That's what happens when you build the muscle: the second contribution is faster than the first.

TL;DR

I started learning Go three weeks ago. I found a bug in k9s where port-forward authorization only checked the create verb, missing the get verb needed by Kubernetes 1.31's new WebSocket path.

My first fix was wrong — I replaced create with get and nearly broke backward compatibility for legacy SPDY clusters. The final fix checks both verbs independently.

I wrote 223 lines of table-driven tests to cover every permission combination. The PR merged in 48 hours. A data race in my test taught me that parallel tests don't share mock state.

If you're learning a language, don't build tutorial projects. Find a real bug in a real project and fix it. The test suite will teach you more than any course.

What's your first open-source contribution story? Drop it in the comments — especially if you also broke a test suite and lived to tell about it.

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

Le Beltagy — Sat, 25 Jul 2026 17:55:01 +0000

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

From a bricked Raspberry Pi to a 4-node Talos Linux cluster running Cilium eBPF, ArgoCD, and Longhorn — every failure, every fix, and the homelab that now hosts my production side projects.

The Setup

It started with a Dell OptiPlex I found in a recycling pile.

Intel i5-6500, 16GB DDR4, a 256GB SATA SSD. Someone's discarded office PC. I wiped the Windows 10 sticker off the case, installed Proxmox, and told myself: "This is going to be my cloud."

Three months later, that OptiPlex plus three Raspberry Pi 4s, a second-hand managed switch, and a drawer full of dead SD cards became a 4-node bare-metal Kubernetes cluster running:

Talos Linux — immutable, API-driven OS built for K8s
Cilium — eBPF-based CNI replacing kube-proxy and providing cluster mesh
ArgoCD — GitOps continuous delivery
Longhorn — distributed block storage across nodes
MetalLB — bare-metal load balancing with BGP
Prometheus + Grafana — monitoring stack pulling metrics from the cluster and my ISP's router

Total cost: ~$220 (OptiPlex was free, Pis were ~$60 each, switch was $35 on eBay).

This is not a tutorial. This is an autopsy of every mistake I made building it.

Why Not Just Use EKS?

I run AWS infrastructure professionally. I can spin up an EKS cluster in 20 minutes with Terraform. I know what managed Kubernetes costs at scale.

But I was hitting a wall:

1. You can't break AWS

In production, I follow well-architected patterns. VPCs, private subnets, IAM roles with least privilege, GuardDuty, Security Hub. Everything is safe. Everything is boring.

In my homelab, I can delete the CNI and watch the cluster melt. I can spin up a privileged container with hostNetwork: true and see what breaks. I can push a bad Cilium network policy and lock myself out of every node — which I did. Twice.

2. The gap between "I passed CKA" and "I understand Kubernetes"

Certifications teach you how to use the API. They do not teach you what happens when etcd runs out of disk because Longhorn replicas ballooned and you didn't set volume quotas. They do not teach you that Cilium's eBPF kube-proxy replacement silently breaks certain firewall rules in ways that are invisible to kubectl.

3. If I can't self-host it, do I really know it?

This is my rule. I don't put a technology on my CV unless I've operated it without a managed control plane. Talos, Cilium, ArgoCD — I wanted to feel them fail.

The Architecture

┌─────────────────────────────────────────────────────────────┐
│                    DELL OPTIPLEX (Control Plane)             │
│                  Talos Linux · x86_64 · 16GB RAM             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  etcd (q=1) │  │ API Server  │  │  Controller Manager │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  Scheduler  │  │ Cilium Agent│  │  Talos API Server   │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
           ┌──────────────────┼──────────────────┐
           ▼                  ▼                  ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   RPI-01 (arm64)│ │   RPI-02 (arm64)│ │   RPI-03 (arm64)│
│  4GB RAM · 64GB │ │  4GB RAM · 64GB │ │  4GB RAM · 64GB │
│   USB-SSD boot  │ │   USB-SSD boot  │ │   USB-SSD boot  │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │ Cilium Agent│ │ │ │ Cilium Agent│ │ │ │ Cilium Agent│ │
│ │ kubelet     │ │ │ │ kubelet     │ │ │ │ kubelet     │ │
│ │ Longhorn    │ │ │ │ Longhorn    │ │ │ │ Longhorn    │ │
│ │ (replica)   │ │ │ │ (replica)   │ │ │ │ (replica)   │ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
           │                  │                  │
           └──────────────────┼──────────────────┘
                              ▼
                    ┌─────────────────┐
                    │  Netgear GS308  │
                    │  8-Port Gigabit │
                    └─────────────────┘

Design Decisions

Decision	Chose	Over	Because
OS	Talos Linux	Ubuntu + kubeadm	Immutable, API-driven, minimal attack surface, no SSH
CNI	Cilium (eBPF)	Flannel / Calico	kube-proxy replacement, Hubble observability, cluster mesh ready
Storage	Longhorn	NFS / local-path	Distributed block storage, snapshots, cross-node replication
GitOps	ArgoCD	Flux / manual	UI for visualizing sync state, familiar to teams
LB	MetalLB (L2 mode)	NodePort	True LoadBalancer IPs on bare metal, no cloud provider
Boot	USB SSD (all nodes)	SD cards	SD cards die under etcd I/O. USB SSDs survive.

The Failures (And What Each One Taught Me)

Failure 1: I Burned 3 SD Cards in 6 Weeks

My first cluster booted from 64GB SanDisk Ultra SD cards. They were cheap. They were fast — for the first week.

Then etcd started crashing. The logs showed mvcc: database space exceeded. I ran etcdctl defrag and it helped for a day. Then the Pi locked up. I rebooted. The SD card wasn't recognised anymore. Corrupted beyond repair.

The second SD card died the same way. The third one survived longest because I moved /var/lib/longhorn to an NFS share — but the card still degraded after 5 weeks.

Root cause: Etcd does frequent small writes. SD cards use wear-leveling algorithms designed for photos (large sequential writes), not databases (random small writes). The flash translation layer dies.

The fix: Every node now boots from a $25 USB3 SSD. Talos supports this natively — you flash the installer ISO, boot from USB, and install to the SSD. The difference is staggering. I've been running the current cluster for 8 months with zero storage-related crashes.

Lesson: Storage media is not interchangeable. Understand your write patterns before you pick hardware.

Failure 2: A Bad CiliumNetworkPolicy Locked Me Out of Every Node

I was experimenting with zero-trust network policies. I applied a CiliumNetworkPolicy that allowed ingress only from pods with label app=frontend. I forgot to include the kube-system namespace. I forgot to include Cilium's own health checks. I applied it with kubectl apply -f.

Within 30 seconds, my SSH session to the control plane hung. kubectl timed out. The Pi nodes showed as NotReady. I had physically unplugged and rebooted everything before I could get back in.

What happened: Cilium enforces policies at the eBPF level — before packets even reach iptables or the host network stack. A bad policy doesn't just drop traffic; it drops it invisibly. No logs in dmesg. No iptables -L entries. Just silence.

The fix: Talos has a recovery mechanism. You boot the node with a kernel parameter talos.config=none and it starts without the network stack enabled. From there, you can mount the Talos EPHEMERAL partition, edit the CNI state, and reboot. But I didn't know this at the time.

The real fix is procedural:

Always test network policies with cilium policy trace before applying
Always keep a default-allow-all policy in Git, ready to apply
Never apply network policies on Friday evening (learned the hard way)

Lesson: eBPF is powerful because it's invisible. That same invisibility makes debugging a nightmare. Test before you enforce.

Failure 3: Longhorn Filled My Disks and kubelet Evicted Everything

I deployed a PostgreSQL database for MarketPulse with a 10Gi Longhorn volume. Longhorn replicates volumes across nodes for HA. I set replica count to 3.

What I didn't know: Longhorn stores replicas as sparse files that grow as data is written. It also keeps snapshots for backups. My "10Gi" database had 3 replicas × 10Gi × 2 snapshots = 60Gi actual usage.

Each Pi had a 120Gi SSD. Longhorn consumed 60Gi. Talos system overhead consumed 20Gi. The remaining 40Gi filled up with container images and logs.

One morning, every pod was Evicted. kubelet had hit the 85% disk pressure threshold and started killing pods to reclaim space. But Longhorn volumes are block devices — kubelet can't delete them. The cluster entered a death spiral.

The fix: I now enforce:

Volume quotas in Longhorn (max 2 replicas for non-critical workloads)
Automatic snapshot cleanup (retain 2, delete older than 7 days)
Node disk alerts in Prometheus at 70% usage, not 85%
Container image garbage collection: imageGCThreshold: 70

Lesson: Distributed storage is not free storage. Replicas × snapshots × logs = real disk consumption. Do the math before you deploy.

Failure 4: MetalLB Gave Out IPs Already Used by My Router's DHCP Pool

I installed MetalLB in Layer 2 mode and assigned it the range 192.168.1.100-192.168.1.150. I exposed a LoadBalancer service. It got 192.168.1.105. I could curl it from my laptop. Success!

Three hours later, my wife's laptop couldn't reach the internet. It had been assigned 192.168.1.105 by the router's DHCP pool. Same IP. Two MAC addresses. Chaos.

The fix: I logged into my ISP router (which I hate but cannot replace), found the DHCP range, and moved MetalLB to 192.168.1.200-192.168.1.250 — outside the DHCP pool.

Better fix: I later configured MetalLB in BGP mode. My router (running OpenWrt on a separate access point) peers with MetalLB and learns routes dynamically. No IP conflicts. No L2 ARP spoofing. True routing.

Lesson: IP address management is not optional on bare metal. Your homelab is a real network. Treat it like one.

Failure 5: I Upgraded Talos and Lost the Cluster Because I Didn't Backup etcd

Talos makes OS upgrades trivial: talosctl upgrade --image ghcr.io/siderolabs/installer:v1.7.0. One command, rolling reboot, cluster stays up.

Except I upgraded from v1.6.0 to v1.7.0 while the cluster was under disk pressure (see Failure 3). The control plane rebooted. etcd failed to start because the WAL files were corrupted from the previous disk-full event. The cluster had no quorum. I had no backup.

I spent 6 hours trying to recover etcd with etcdctl snapshot restore. The snapshot I had was 3 weeks old. I lost:

All ArgoCD application definitions (I had them in Git, so I re-synced)
All Prometheus metrics history
All Longhorn volume states (the PVCs survived, but the volume attachments were inconsistent)

The fix:

Automated etcd snapshots every 6 hours to S3 (yes, I back up my homelab to AWS — the irony is not lost on me)
Pre-upgrade health checks: disk <70%, all nodes Ready, all Longhorn volumes healthy
I wrote a tiny Go CLI that wraps talosctl upgrade with these checks. It lives in my homelab-tools repo.

Lesson: Immutable infrastructure doesn't mean unbreakable infrastructure. Back up your control plane state. Or learn etcd disaster recovery at 2 AM like I did.

What's Running on It Now?

Workload	Purpose	Resource Usage
kube-radar	Public Go CLI for K8s RBAC wildcard detection	50m CPU, 128Mi mem
job-digest	Python job scraping pipeline	200m CPU, 256Mi mem
MarketPulse API	Go + FastAPI real-time analytics backend	500m CPU, 512Mi mem
PostgreSQL	Primary DB for MarketPulse (Longhorn-backed)	300m CPU, 512Mi mem
Prometheus	Cluster + app metrics	400m CPU, 1Gi mem
Grafana	Dashboards for everything	100m CPU, 128Mi mem
ArgoCD	GitOps sync for all workloads	200m CPU, 512Mi mem
Cilium Hubble	Network flow observability	100m CPU, 256Mi mem
Longhorn	Storage engine (daemonset)	150m CPU, 512Mi mem

The OptiPlex sits at ~60% CPU under normal load. The Pis hover at 40-70%. I can burst higher, but I know the limits.

3 Lessons That Transfer to Production

1. Resource quotas are not bureaucracy

In my homelab, I have no cost optimization team. When Longhorn eats all disk, I pay with crashed pods and 2 AM debugging. Setting requests and limits, configuring PVC sizes, defining retention policies — these are not "enterprise overhead." They are survival mechanisms.

2. Observability first, workload second

I deployed Prometheus and Grafana before I deployed any application. Not because I'm disciplined — because I learned that debugging a bare-metal cluster without metrics is like performing surgery in the dark. Hubble shows me every dropped packet. Node exporter shows me disk I/O patterns I never would have guessed. In production, I'd push this to Thanos or Cortex. In my homelab, I run a 30-day retention and call it enough.

3. GitOps is the only way I trust myself

I have root access to every node. I can kubectl exec into any pod. Without ArgoCD, I would absolutely make manual changes at 11 PM and forget them by morning. With ArgoCD, the cluster state is in Git. If I change something manually, the UI shows it as OutOfSync and I feel shame. That shame keeps my cluster declarative.

The Hardware List (If You Want to Replicate)

Component	Spec	Cost	Notes
Control plane	Dell OptiPlex 7050, i5-6500, 16GB	Free (recycled)	Any x86_64 box works. Needs VT-x.
Worker nodes	3× Raspberry Pi 4 (8GB)	~$60 each	4GB works, 8GB is comfortable
Worker storage	3× 120GB USB3 SSD	~$25 each	Avoid SD cards for etcd. Just don't.
Switch	Netgear GS308 8-port gigabit	$35 (eBay)	Any managed switch for VLANs
Cables	Cat6 patch cables	~$15	Buy a pack
Total		~$220

The Pi 5 exists now and is faster. I'm staying on Pi 4 because arm64 compat is tested and I'm lazy.

TL;DR

I built a 4-node bare-metal Kubernetes cluster on $220 of scrap hardware and Raspberry Pis. I use Talos Linux, Cilium eBPF, ArgoCD, Longhorn, and MetalLB.

I burned SD cards, locked myself out with network policies, filled disks with storage replicas, caused IP conflicts, and corrupted etcd during an upgrade.

Every failure taught me something I use in production. The cluster now runs my public projects, my monitoring, and my experiments.

If you're learning Kubernetes, break it at home before you break it at work.

What's your homelab stack? Drop it in the comments — always looking for new failure modes to learn from.

I Deployed a Purple Team Inside My Kubernetes Cluster. The Scanners Never Saw It Coming.

Le Beltagy — Thu, 23 Jul 2026 19:50:05 +0000

Three security scanners gave my cluster a clean bill of health. Then I deployed an attacker into a DaemonSet and discovered what static analysis actually misses.

The Setup

My homelab cluster is nothing exotic: a few Hyper-V VMs, a Raspberry Pi worker, and a Talos Linux control plane running the usual suspects: ArgoCD for GitOps, Cilium for CNI, Longhorn for storage. My workload is a typical cloud-native stack: a Go backend, a React frontend, PostgreSQL, the works.

I did exactly what the security playbooks tell you to do:

Trivy for container image scanning
kube-bench for CIS Kubernetes Benchmark compliance
Polaris for best-practice configuration checks

Every single tool reported green. I took a screenshot of the dashboard and felt safe.

Then something felt wrong.

A scanner audits what you declared. It does not test what happens when things go wrong. It is like a home inspector checking your door locks while ignoring the open window around the corner. I wanted a burglar on my side.

So I built one.

Introducing NEMESIS

NEMESIS is a Kubernetes-native purple team platform. It lives inside the cluster as a DaemonSet, not outside as a scheduled scan. It does not read manifests and compare them against registries. It executes attack scenarios from the same pod network your real workloads use.

The Stack

The platform splits along two languages for very specific reasons:

Python Controller — The orchestration brain. Schedules attack scenarios, correlates events, and emits findings. Python was the right call here: rapid scenario development and rich security libraries.
Go Attack Agent — A DaemonSet on every node. Executes the actual attack primitives: token abuse, lateral movement, network probing. Go gives me a single static binary, sub-second cold-start, and zero dependency hell inside a distroless container.

Attack Engine

Scenarios are not hard-coded exploits. They are a composable library of primitives:

Recon — Enumerate service accounts, roles, secrets, and network policies from inside the pod
Token Abuse — Mount and abuse service account tokens against the API server
Lateral Movement — Test east-west connectivity between namespaces with no external tooling
Exfil Probe — DNS and HTTP egress tests to measure actual network policy enforcement

eBPF Integration

Because the cluster runs Cilium, NEMESIS hooks into eBPF-based network telemetry. Every packet the attack agent sends is observable at the kernel layer. This means I do not just know that an attack path exists; I have flow-level proof of exactly which packet traversed which interface.

The Attack Chain That Scanners Missed

Here is one scenario, step by step. I anonymized nothing else because the configuration is common.

Step 1: Recon

The attack agent starts inside a pod in the default namespace. It enumerates service accounts:

curl -s https://kubernetes.default.svc/api/v1/namespaces/default/serviceaccounts \
  --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

It finds the default service account. Then it checks its bindings:

curl -s https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings

Result: a ClusterRoleBinding named cluster-reader-default grants cluster-reader to the default SA across all namespaces.

Step 2: Token Abuse

With cluster-reader permissions, the agent queries every namespace for secrets:

curl -s https://kubernetes.default.svc/api/v1/secrets --all-namespaces

It finds a ConfigMap in the backend namespace named db-config containing a PostgreSQL connection string with a hard-coded password.

What the scanner said: "ConfigMap db-config exists. No hardcoded secrets detected in source image."

What NEMESIS found: The ConfigMap is readable by any pod using the default service account, which is every pod in the cluster that does not explicitly override it.

Step 3: Lateral Movement

The agent lists pods across namespaces:

curl -s https://kubernetes.default.svc/api/v1/pods --all-namespaces

It identifies the backend API pod in the backend namespace. It checks the NetworkPolicy for that namespace:

# backend-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Ingress

What the scanner said: "NetworkPolicy exists. Default-deny posture implemented."

What NEMESIS found: The policy only specifies Ingress. There is no Egress policy. The backend pod can open outbound connections to anything.

Step 4: Data Exfil Test

The agent attempts a DNS lookup to an external domain controlled by the test harness:

nslookup exfil-test.nemesis.internal

Cilium's eBPF probe captures the packet at the kernel socket layer. The DNS query reaches the cluster DNS, which forwards it upstream. No policy blocked it.

NEMESIS logs this as: "DNS exfiltration path confirmed — no egress filtering on namespace backend."

The Gap: What You Declare vs. What You Get

Out of 47 unique findings NEMESIS surfaced across three weeks of continuous testing, here is the breakdown versus what the traditional scanner suite caught:

Finding Category	NEMESIS	Scanner Suite	Gap
Lateral movement paths	12	0	100%
Service account token abuse	9	1	89%
Network exfiltration (DNS/HTTP)	8	0	100%
Network policy bypass	6	0	100%
Runtime container escape	3	1	67%
Image CVEs	2	7	Covered by scanner
CIS misconfigurations	4	5	Partial overlap
Manifest best practices	3	6	Covered by scanner
Total	47	20	~81% unique to NEMESIS

To be completely fair: the scanners caught every image vulnerability and most CIS misconfigurations. They are excellent at what they do. They just do not test runtime behavior.

Why This Changes How You Think About K8s Security

The industry has optimized for speed of detection. But detection speed means nothing if you are detecting the wrong things.

Scanners answer: "Is this configuration compliant?"

Purple teams answer: "What happens when someone already inside tries to move?"

Both questions matter. But in a world where supply chain attacks, compromised CI pipelines, and insider threats are the dominant risk models, the second question is the one that actually keeps you awake.

NEMESIS is not a replacement for scanning. It is an additional layer that validates your assumptions at runtime, continuously, from the same vantage point a real attacker would have.

What I Learned Building This

The Go-vs-Python split was not philosophical. It was operational. The agent needs to start in under a second when a node scales up, handle network timeouts gracefully, and compile to a single binary that fits in a scratch container. Go wins here without debate.

The controller needs to parse YAML attack definitions, talk to the Kubernetes API, and integrate with Cilium's Hubble for flow data. Python's ecosystem made this painless.

The hardest part was not the attack code. It was building the telemetry correlation: matching an eBPF flow event to a specific attack scenario step so you can say with certainty: "The DNS leak on step 4 of scenario backend-exfil-001 originated from pod backend-api-7d9f4b8c5-x2k9m at 14:23:07 UTC."

Without that correlation, purple team findings are just noise.

Where NEMESIS Goes Next

The current milestone focuses on Kubernetes, but the architecture is generic. The same DaemonSet pattern applies to:

Container runtime testing (Docker socket abuse, privileged escalation)
Cloud metadata service abuse (IMDSv2 bypass attempts on AWS/Azure nodes)
Storage layer attacks (Longhorn volume snapshot tampering, RWO bypass)

The goal is not to build a scanner with a different skin. The goal is to shift the security posture from "we read your configuration" to "we prove what an attacker can actually do."

Try It Yourself

If you want to see what your scanners are missing, the fastest path is deliberate:

Spin up a throwaway pod in your cluster
kubectl exec into it
Run curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/default/secrets
See what comes back

If you see more than an empty list, you have found your first gap without writing a single line of code.

NEMESIS is an open-source Kubernetes-native purple team platform. If you want to follow the build, attack scenario library, or contribute, the repository is at github.com/beltagyy/nemesis.

Built with Go, Python, Cilium, and enough caffeine to make a kernel panic look calm.

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

Le Beltagy — Wed, 22 Jul 2026 10:50:05 +0000

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

From manually copy-pasting cover letters to a Python pipeline that hunts Dutch tech roles while I sleep — the architecture, the failures, and why I need your help.

The Setup

I had applied to 73 jobs in 4 weeks.

Not casual browsing. I mean:

Custom cover letters per role
CV tweaks for Platform Eng vs SRE vs Cloud Security
LinkedIn cold messages to hiring managers
Tracking everything in a Notion database

The result? Ghosted by 61. Rejected by 8. Phone screen with 4.

Meanwhile, I was working full-time at Siemens, studying Go for kube-radar, and maintaining a homelab that was literally on fire one Tuesday because a Cilium policy update bricked the cluster network. I was burning out on the process of job searching instead of actually getting better at it.

So I did what any engineer would do: I automated the pain away.

That's how job-digest started — a Python pipeline that scrapes job boards, filters by my exact criteria, ranks roles by match quality, and delivers a daily summary to my Telegram. It didn't get me hired overnight. But it got me interviewed by companies I actually wanted to work for — and it taught me more about web scraping, async Python, and data pipelines than any course.

This is the story of building it. The code is public. I want collaborators. If you're job-hunting in tech and you know Python, read on.

Why This Problem?

Cloud consulting taught me how to build systems. It did not teach me how to sell myself.

When I decided to relocate to Amsterdam 🇳🇱 and target backend/platform roles, I faced three problems no recruiter warns you about:

1. The signal-to-noise ratio is brutal

Job boards are flooded with "Cloud Engineer" roles that are actually MSP helpdesk jobs, "DevOps" roles that mean "you're the only ops person for 40 devs," and "Kubernetes" roles at companies running Docker Compose on a single EC2 instance.

I needed filtering that understood my stack: Go, Kubernetes internals, Cilium/eBPF, Azure + AWS multi-cloud, security compliance (NIS2/DORA).

2. Timing asymmetry

The best roles at Adyen, Booking.com, ASML — they go live on Tuesday morning and have 200 applicants by Thursday afternoon. I was checking boards manually on Sunday evenings. I was always late.

3. Context switching kills deep work

Every 20-minute "let me check LinkedIn" session destroyed my flow state. I wanted one daily digest at 7 AM. Read it on the train. Apply to the top 3 roles. Then close the tab and write Go code.

The Architecture

┌─────────────┐    ┌──────────────┐    ┌──────────────┐    ┌─────────────┐
│  Scrapers   │───►│   Enricher   │───►│    Ranker    │───►│   Notifier  │
│ (async)     │    │ (LLM + rules)│    │ (scoring)    │    │ (Telegram)  │
└─────────────┘    └──────────────┘    └──────────────┘    └─────────────┘
     │                                              │
     ▼                                              ▼
 LinkedIn Jobs                              SQLite cache
 Indeed                                     (dedup + history)
 Glassdoor
 We Work Remotely
 RemoteOK

Design Decisions

Decision	Chose	Over	Because
Language	Python 3.11 + asyncio	Go	Rapid prototyping, BeautifulSoup/httpx ecosystem, LLM SDKs
Storage	SQLite	PostgreSQL	Zero infra, portable, single file. WAL mode for concurrent reads.
Scraping	httpx + BeautifulSoup	Selenium/Playwright	Faster, lighter, cheaper. JavaScript-rendered boards get headless fallback.
Rate limiting	Adaptive backoff (exponential + jitter)	Fixed delays	Respects robots.txt without being slower than necessary.
LLM enrichment	OpenAI GPT-4o-mini	Local LLM	Cost: ~$0.02 per 100 jobs. Quality worth it for extraction.
Delivery	Telegram Bot API	Email	I live in Telegram. Push notification = instant awareness.
Scheduling	cron on homelab RPi	GitHub Actions	Free, persistent, no cold start. Runs at 06:00 CET daily.

The Pipeline (Simplified)

# 1. COLLECT — async scrapers, one per source
async def scrape_linkedin(query: str, location: str) -> list[RawJob]:
    # httpx + BeautifulSoup
    # Extract: title, company, location, description, salary, url, posted_at
    # Returns list of RawJob dataclasses

# 2. NORMALIZE — unify schema across sources
def normalize(raw: RawJob) -> Job:
    # Map "Amsterdam, North Holland" → "Amsterdam"
    # Extract seniority: Junior / Mid / Senior / Staff from title
    # Detect remote policy: onsite / hybrid / remote
    # Parse salary ranges from description text (regex + LLM fallback)

# 3. FILTER — hard rules (cheap, deterministic)
def passes_filter(job: Job) -> bool:
    return (
        job.location in TARGET_CITIES  # Amsterdam, Berlin, Dublin, etc.
        and job.seniority in {"Senior", "Staff", "Principal"}
        and any(kw in job.description.lower() for kw in TECH_KEYWORDS)
        and job.company not in BLOCKLIST  # agencies, body shops
    )

# 4. ENRICH — LLM judges fit quality
def enrich_with_llm(job: Job) -> EnrichedJob:
    prompt = f"""
    Role: {job.title} at {job.company}
    Description excerpt: {job.description[:2000]}

    Candidate profile: 6 years cloud/platform, K8s internals (Cilium, Talos),
    Go learner, Azure+AWS, security compliance (NIS2/DORA).
    Target: Platform Engineer or Backend Engineer in Amsterdam/Berlin/Dublin.

    Tasks:
    1. Is this role genuinely backend/platform work (not IT support, not MSP)?
    2. Does the tech stack mention K8s, Go, distributed systems, or cloud-native?
    3. Visa sponsorship likely? (big tech yes, small local shop probably no)
    4. Score 0-100 for fit.
    5. Is it worth applying? yes/no/maybe.
    """
    response = openai.chat.completions.create(...)
    return EnrichedJob(..., llm_score=response.score, verdict=response.verdict)

# 5. RANK — composite score
def score(job: EnrichedJob) -> float:
    return (
        job.llm_score * 0.5 +
        salary_score(job.salary_max) * 0.2 +
        company_tier_score(job.company) * 0.2 +  # Tier 1 = Adyen/Booking/ASML/etc.
        recency_score(job.posted_at) * 0.1        # Fresh postings score higher
    )

# 6. NOTIFY — formatted Telegram message
async def send_digest(jobs: list[EnrichedJob]):
    msg = format_telegram(jobs[:10])  # top 10 only
    await telegram_bot.send_message(chat_id=MY_CHAT, text=msg)

The Telegram Output

🔍 Daily Job Digest — 2026-07-22
18 new roles | 4 match your filters | Top 3 below

━━━━━━━━━━━━━━━━━━━━
🥇 94 pts — Adyen — Platform Engineer
   📍 Amsterdam 🇳🇱  |  💰 €90-120K  |  🏠 Hybrid
   🛠 Go · K8s · Kafka · Terraform
   📝 "Build internal primitives for developer platforms..."
   ✅ LLM: "Strong match. Go-heavy platform team. Sponsors HSM."
   🔗 https://careers.adyen.com/jobs/...

🥈 88 pts — Booking.com — SRE
   📍 Amsterdam 🇳🇱  |  💰 €85-110K  |  🏠 Hybrid
   🛠 K8s · Prometheus · Python/Go · Cilium
   ✅ LLM: "Excellent infra depth match. Large K8s fleet."

🥉 81 pts — N26 — Platform Engineer
   📍 Berlin 🇩🇪  |  💰 €80-110K  |  🏠 Hybrid
   🛠 Go · K8s · Terraform · Microservices
   ✅ LLM: "Good fit. Fintech, EU Blue Card likely."

📊 Actions: /apply N26  |  /dismiss Adyen  |  /details Booking

5 Failures Before It Worked

Failure 1: LinkedIn Rate-Limited Me in 6 Minutes

My first scraper used a single httpx session with a 2-second delay between requests. I was proud of the backoff. LinkedIn was not impressed.

They returned a 429 after ~30 requests and started serving CAPTCHA pages. I added:

Rotating User-Agent headers (from a pool of 50 real browser strings)
Proxy rotation via free proxy lists (terrible idea — 80% were dead)
Randomized delays (1-5 seconds with jitter)

The fix that actually worked: scrape LinkedIn's public job search JSON endpoint instead of HTML parsing. It returns structured data at /jobs-guest/jobs/api/jobs and is significantly more stable. I also dropped proxy rotation and just respected a 5-second delay. LinkedIn is aggressive but fair.

Lesson: Don't fight the frontend. Find the API the frontend uses.

Failure 2: The LLM Hallucinated a Role at Google That Didn't Exist

I blindly trusted GPT-4o-mini's extraction for the first week. It invented a "Kubernetes Platform Engineer" role at Google Amsterdam with a salary of €500K. The URL 404'd. I had already told my wife about it.

The fix: every scraped job gets a live URL validation (HEAD request, follow redirects). If the URL doesn't return 200, it's dropped before enrichment. LLMs are pattern matchers, not fact checkers.

Lesson: LLMs are enrichment, not validation. Never trust their facts without verification.

Failure 3: I Applied to the Same Job 4 Times

My SQLite schema initially keyed jobs by (title, company). Then a company reposted the same role with a slightly different title. I applied twice. The second time, the recruiter replied: "You already applied last week."

The fix: content hashing. I now hash the normalized job description (after removing whitespace and dates). Same description = same job, even if the title changes. Deduplication is fuzzy, not exact.

Lesson: Job boards are messy. Exact matching is naive.

Failure 4: Running It on My Laptop Meant It Died When I Closed the Lid

I initially ran the script manually: python main.py, wait 8 minutes, get results. But I kept forgetting. And when I closed my MacBook, the script died.

The fix: a headless Raspberry Pi 4 in my homelab runs a cron job at 06:00 CET. It has a persistent SQLite file on an external SSD. The Pi also runs my DNS (Pi-hole), my monitoring (Prometheus), and now my job search. One device, many duties.

Lesson: Automations that depend on you remembering to run them are not automations.

Failure 5: I Got 18 "Matches" and Felt Overwhelmed

Even with filtering and scoring, getting 18 "good" roles in one digest meant I applied to none. Decision paralysis. I would read all 18, open 12 tabs, and close the laptop exhausted.

The fix: the digest shows top 3 by default. To see 4-10, you reply /more. To see the rest, /all. This tiny UX change meant I actually applied to the top 3 instead of bookmarking 12.

Lesson: More data is not better. More decisions is worse.

Results (So Far)

I won't claim this got me hired. I'm still interviewing. But here's what changed:

Metric	Before	After
Time spent browsing jobs/day	45 min	5 min reading digest
Applications/week	~8 (sporadic)	~12 (consistent)
Roles matching my stack	~30% of browsed	~85% of digest
Time-to-apply after posting	3-4 days	<24 hours
Relevant recruiter conversations/month	1-2	4-5

The real win: mental bandwidth. I stopped treating job search as a thing I do when I'm tired at 10 PM. It became a system that runs without me.

Companies where the pipeline surfaced roles I wouldn't have found manually:

A stealth-mode fintech in Amsterdam hiring Go platform engineers (not on LinkedIn, found via We Work Remotely)
A CDN company in Berlin rebuilding their edge on Cilium + eBPF (mentioned in a blog post the scraper indexed)
A Series B startup in Dublin with ex-Google SREs building a K8s-native security product (literally my exact niche)

The Code

git clone https://github.com/beltagyy/job-digest.git
cd job-digest
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp config.example.yaml config.yaml
# Edit: add your OpenAI key, Telegram bot token, target cities, tech keywords
python main.py --run-now

What you get:

Modular scrapers (add a new source in ~50 lines)
SQLite cache with WAL mode
LLM enrichment with prompt templates
Telegram bot interface with simple commands
Docker support for headless deployment

What you need:

Python 3.11+, an OpenAI API key (~$5/month at my volume)
A Telegram bot (free via @botfather)
A server that runs cron (RPi, VPS, old laptop)

I Need Collaborators

This project helps me, but it's built for my profile and my targets. To make it useful for others, I need help with:

1. More scrapers

AngelList / Wellfound (startup roles)
Hacker News "Who is Hiring?" (monthly thread parser)
Greenhouse / Workday ATS backends (many companies use these and have predictable JSON APIs)
Country-specific boards (StepStone.de, Indeed.nl, IrishJobs.ie)

2. Better filtering

Visually parse "years of experience" requirements and flag "possibly underqualified" vs "definitely qualified"
Detect visa sponsorship mentions from description text (pattern-based, not just LLM)
Filter out roles from recruiting agencies vs direct employers

3. Smarter enrichment

Local LLM option (Ollama/Llama 3.1) for people who don't want OpenAI costs
Company research: pull funding stage, Glassdoor rating, team size from public APIs
Salary benchmarking: compare offered range to levels.fyi / Glassdoor data

4. Alternative frontends

Discord bot (many communities live there)
Email digest (for non-Telegram users)
Web dashboard (Streamlit or similar) for browsing history and stats

5. Testing & reliability

Scrapers break when job boards change HTML. We need snapshot tests (save HTML, assert extraction still works).
Mock LLM responses for unit tests (expensive to hit OpenAI in CI).

If any of this sounds like you — especially if you're job-hunting in EU tech and want a tool that actually understands your stack — open an issue or PR.

Even if you're not technical: try it, tell me what job board you want added, and I'll prioritize it.

3 Lessons That Transfer Beyond Job Searching

1. Automate the boring, not the important

I automated discovery and filtering. I did not automate applying or interviewing. The bot finds roles. I still write every cover letter by hand and research every company before applying. The pipeline handles noise; I handle signal.

2. Your side project's tech stack should teach you something

I chose Python + asyncio because I already knew Python but had never built a production async pipeline. I learned:

asyncio.gather for parallel scrapers
SQLite WAL mode and connection pooling
Prompt engineering for structured LLM output (JSON mode, function calling)
Rate limiting and backpressure

If your side project doesn't stretch you, it's just work you don't get paid for.

3. Ship before it's perfect

The first version didn't have LLM enrichment. It was just a list of job titles and URLs sorted by posting date. It was ugly. It still saved me 30 minutes a day.

I added scoring later. Then filtering. Then the Telegram bot. Each increment made it better, but the core value (stop manually browsing job boards) existed on day 1.

What's Next

Feature	Status
Core pipeline (scrape → filter → notify)	✅ shipped
LLM enrichment	✅ shipped
SQLite dedup + history	✅ shipped
More scrapers (HN, Wellfound, StepStone)	🔄 in progress
Company enrichment (funding, Glassdoor)	📋 planned
Web dashboard	📋 planned
Resume-to-job matching score	📋 planned

**
🚀 Try it:** https://github.com/beltagyy/job-digest
**
🤝 Collaborate:** Open an issue with the tag help-wanted or DM me

**

If you're job-hunting right now — what's the most frustrating part? Drop it in the comments. If it's common enough, I'll add it to the pipeline.

TL;DR

I applied to 73 jobs in 4 weeks, got exhausted by the process, and built a Python bot (job-digest) to automate job-board scraping, filtering, and ranking.
It's a pipeline: async scrapers → LLM enrichment → scoring → daily Telegram digest. Runs on a Raspberry Pi at 6 AM.
It cut my daily job-browsing time from 45 minutes to 5, and surfaced roles at companies I actually wanted (Adyen, Booking, N26) that I would have missed manually.
Open source and looking for collaborators: github.com/beltagyy/job-digest. I need help with scrapers, filtering rules, local-LLM support, and a web dashboard.

I Learned Go by Hacking Kubernetes RBAC Security (And Nearly Quit 5 Times)

Le Beltagy — Sun, 19 Jul 2026 22:03:03 +0000

I Learned Go by Hacking Kubernetes RBAC Security (And Nearly Quit 5 Times)

From zero Go experience to shipping kube-radar — a Kubernetes security CLI that detects RBAC wildcard violations. The story, the architecture, the mistakes, and why I did it the hard way.

The Setup

I manage Kubernetes clusters at Siemens. Not "I run kubectl apply from my laptop" manage — I mean Talos Linux on bare metal, Cilium replacing kube-proxy, eBPF policies doing deep packet inspection, ArgoCD running GitOps end-to-end. The kind of infra where one bad RoleBinding can cost you the crown jewels.

But here's the problem: I had never written production Go.

My code was Python — automation, Lambdas, Flask services, AI pipelines. My Go experience was exactly one Pulumi IaC program where I mostly copy-pasted from examples and hoped go build didn't yell at me.

I wanted to become what I call a "backend + cloud-native mega engineer" — someone who can architect a distributed system and ship the code that runs inside it. Not just configure the cluster, but write the controller. Not just deploy the service, but design the API.

So I made a deal with myself:

Build one real, public, useful project in Go. Ship it. Then use it as proof.

That's how kube-radar started.

Why This Project?

Kubernetes RBAC is a nightmare to audit.

If you've ever done a security review on a real cluster, you know the pain:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

This is a cluster-admin equivalent disguised as a single rule. Anyone bound to this can do anything in your cluster. And unlike cluster-admin, it doesn't show up in obvious places like kubectl get clusterrolebindings when you're scanning for privilege escalation.

Wildcard rules (verbs: ["*"], resources: ["*"], apiGroups: ["*"]) are the silent killers of K8s security posture. They slip past most CNAPP tools because they're technically valid — they just give away the kingdom.

I wanted a tool that:

Scans any cluster's RBAC in seconds
Flags wildcard over-permissions with severity
Tracks findings across scans (is this new, or did someone just accept it?)
Gates CI/CD with exit codes — fail the build when someone commits a wildcard rule
Reports in SARIF for GitHub Advanced Security integration

And I wanted to build it from scratch in a language I'd never shipped in. Because if I can ship this, I can ship anything.

The Learning Curve (Or: How I Nearly Gave Up 5 Times)

Attempt 1: The Cathedral Plan

I sat down and wrote a 5,000-word design document.

I'm not kidding:

Full SQLite schema with WAL mode, migrations, and ON DELETE CASCADE
Fingerprinting strategy with SHA-256 canonical hashing
Cobra CLI with 6 subcommands
Rule engine with Go interfaces and a plugin registry
Diff engine with set operations across snapshots
.goreleaser cross-compilation for 4 platforms
SARIF 2.1.0 schema validation

It was beautiful. It was also unbuildable for someone learning Go from zero. After 3 weeks of wrestling with database/sql semantics and SQLite driver imports, I had a schema but no working code.

Lesson 1: A perfect design you can't ship is worthless.

Attempt 2: Copy-Paste Panic

I found a GitHub repo that did something vaguely similar. I copy-pasted Go client-go List calls, cobra wiring, even error wrapping. It compiled! It ran! It... panic'd on startup because I shadowed err in an if block and the outer function didn't know.

Go's error-as-values model broke my brain. In Python, I'd raise an exception and catch it somewhere. In Go, the error is just a return value you need to handle every single time. My code was 40% error handling by line count.

Lesson 2: You can't paste your way to understanding.

Attempt 3: The Interface Rabbit Hole

The design doc said "Rule interface type." I spent a full day understanding why interface{} isn't the same as a typed interface, why empty interfaces are a smell unless you need them, and why Go composition beats inheritance so completely.

It clicked when I realized: a Rule in my analyzer isn't a class. It's a contract.

type Rule interface {
    ID() string
    Describe() Description
    Evaluate(s *model.Snapshot) []model.Finding
}

KR001 (wildcard verbs), KR002 (wildcard resources), KR003 (wildcard apiGroups) — all three are just structs that implement those three methods. No inheritance tree. No base class. Just behavior.

Lesson 3: Go interfaces are small, implicit, and compositional. Stop thinking in classes.

Attempt 4: client-go vs. Kubernetes API

Here's something nobody tells you: kubectl does a lot of heavy lifting that client-go doesn't.

kubectl config current-context uses shell-level kubeconfig resolution. client-go's clientcmd.BuildConfigFromFlags doesn't. It took me 2 hours to figure out why my code was connecting to the wrong cluster — client-go reads the kubeconfig file but doesn't use the current-context field unless you explicitly load it.

// This does NOT use your shell's "kubectl config current-context"
config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)

// This DOES — it loads the kubeconfig and resolves context, cluster, user
config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
    &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},
    &clientcmd.ConfigOverrides{CurrentContext: contextName},
).ClientConfig()

That single discovery changed how I build the kube-radar scan command.

Lesson 4: client-go is lower-level than kubectl in ways that matter.

Attempt 5: Determinism

My first version printed findings in random order every time. Tests flaked. Diff was impossible. Fingerprints changed because I included rule index in the hash.

The fix was a lesson in Go's core philosophy: if the language doesn't guarantee something, you must enforce it yourself. Go map iteration is deliberately randomized. So you sort. Always.

sort.Slice(findings, func(i, j int) bool {
    if findings[i].Severity != findings[j].Severity {
        return findings[i].Severity > findings[j].Severity // desc
    }
    return findings[i].Fingerprint < findings[j].Fingerprint
})

Lesson 5: Randomness is a feature of the language. Determinism is your job.

The Architecture

Here's what I shipped — and more importantly, why the architecture looks like this:

Design Decisions

Decision	Chose	Over	Because
SQLite driver	`modernc.org/sqlite`	`mattn/go-sqlite3`	Zero CGO = trivial cross-compilation. 2× slower, irrelevant at this scale.
State	Local SQLite	Stateless linter	Diff/baseline is the differentiator; stateless tools already exist
Role handling	Unified `RoleLike` type	Separate Role/ClusterRole types	Rules written once; kind preserved for reporting severity
Migrations	Hand-rolled (~20 lines)	goose/golang-migrate	Zero deps, more learning value. Revisit when migrations exceed ~10 files.
Persistence	Per-machine SQLite DB	Centralized API/DB	CLI-first tool; no server to maintain, no network dependency
Fingerprinting	SHA-256 canonical content hash	UUID per finding	Stable across rescans; "same problem" = "same fingerprint"

The Fingerprint System (The Most Underrated Part)

This is where most security tools fail: they generate "new findings" every scan because of random IDs or timestamps. My fingering system:

// v1|<rule_id>|<kind>|<namespace>|<name>|<canonical_rule>
canonical_rule = sort-and-join(apiGroups) + "|" + sort-and-join(resources) + "|" + sort-and-join(verbs)
fp = sha256(v1|KR001|ClusterRole||admin|*|*|*)[:16]

Key properties:

Stable across rescans: Same cluster state → same fingerprint
Survives reordering: If someone reorders rules in a Role, the fingerprint doesn't change
Detects narrowing: verbs: ["*"] → verbs: ["*"] + resources: ["pods"] is a different finding (content changed)

This makes the diff feature in v0.2 trivial: it's just set arithmetic on fingerprint maps.

Exit Codes as CI Contract

0 — Clean scan, no new findings ≥ --fail-on
1 — New unacknowledged findings ≥ --fail-on severity threshold
2 — Operational error (unreachable cluster, bad flags, DB failure)

Why only 3 codes? Because CI pipelines need simple logic: kube-radar scan && deploy || fail works with zero tooling.

What It Looks Like

$ kube-radar scan --context homelab --fail-on medium

RULE    | KIND        | NAMESPACE | NAME          | DETAIL
--------|-------------|-----------|---------------|------------------
KR001   | ClusterRole | -         | admin-legacy  | Wildcard verbs: [*]
KR002   | Role        | default   | app-service   | Wildcard resources: [*]
KR003   | ClusterRole | -         | cluster-admin | Wildcard apiGroups: [*]

3 findings (1 Critical, 1 Medium, 1 Low)  exit code: 1

That exit code: 1 means CI fails. That means the wildcard rule does not get deployed.

3 Lessons That Transfer Beyond This Project

1. Design the hard thing in writing first

My 5,000-word design doc wasn't a waste. It was a map. When I cut the scope to v0.1 (no SQLite, no diff, no SARIF), I knew exactly what each deferred feature looked like because I'd already designed it. The doc became the roadmap.

2. Build for yourself, then write about the pain

Every feature in kube-radar solves a problem I actually have:

Can't find wildcard rules quickly → the scan command
Can't tell if a finding is new → the diff feature (v0.2)
Can't baseline accepted risk → acks (v0.2)
Can't integrate with GitHub Security → SARIF (v0.2)

Tooling built from real pain is better than tooling built for a portfolio.

3. A shipped v0.1 beats a perfect v2.0

I had to fight my own instinct to build everything at once. v0.1 is:

1 command (scan)
3 rules (KR001-003)
Table output only
Fingerprints computed but not yet persisted

But it works. It already catches real violations. It already fails CI builds. Everything else is additive.

The Roadmap

Version	What
v0.1 ✅	Scan, 3 rules, table output, fingerprints, exit codes
v0.2	SQLite store, diff, history, acks/baselines, JSON + SARIF
v0.3	Full rule pack: cluster-admin bindings, secrets read, pods/exec, impersonate, escalate/bind
v1.0	Multi-context fleet scan, config file, goreleaser
v2.0	Controller mode: watch RBAC, Prometheus metrics, admission warnings

Try It

git clone https://github.com/beltagyy/kube-radar.git
cd kube-radar
go build -o kube-radar ./cmd/kube-radar
./kube-radar scan --kubeconfig ~/.kube/config --fail-on medium

Or grab a release binary (when I cut v0.1.0 — soon).

What I need from you:

Does it work on your cluster?
What's missing for your use case?
Are there wildcard variants I'm not catching?

Every issue helps me learn.

About This Series

This is post 1 in the "kube-radar: Building a Security CLI in Go" series. Next up:

Deep-dive: Why I chose SQLite over in-memory state
Deep-dive: Designing deterministic fingerprints for security findings
Deep-dive: The Go interfaces that made the rule engine testable

If you're learning Go from a cloud background like I am — or if you're a backend engineer who wants to understand K8s security internals — follow along.

*Questions? I'm @beltagyy on GitHub and beltagyy on DEV.

Processing 1 Million Events Per Day Without Breaking the Bank (Or Your Database)

Le Beltagy — Sun, 19 Jul 2026 09:27:51 +0000

Processing 1 Million Events Per Day Without Breaking the Bank (Or Your Database)

The real-world math behind event batching, and why it changed everything for me

The Problem

VehicleMetrics receives sensor data from vehicles in real-time.

Each vehicle sends: 10 GPS updates/second + 100 sensor readings/second = 110 messages/second per vehicle

With 50 vehicles: 5,500 messages/second = 473 million messages per day

I needed to:

Ingest fast (< 50ms response time)
Store reliably (no data loss)
Not go bankrupt (costs matter at scale)
Not break the database (PostgreSQL can only write so fast)

This is where most people fail. They think: "Just insert each event into the database."

Wrong.

The Naive Approach (That Destroys Databases)

# ❌ NAIVE: Direct insert per event
@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest(vehicle_id: str, data: SensorData):
    await db.execute(
        "INSERT INTO sensor_data VALUES (...)",
        [vehicle_id, data.timestamp, data.speed, ...]
    )
    return {"status": "created"}

The math:

5,500 events/second × 60 = 330,000 inserts/minute
Each insert: ~1ms database overhead
PostgreSQL write throughput: ~10,000 inserts/second max
Result: Queue builds. Latency explodes. System crashes.

What Actually Works: Batch Processing

Instead of inserting individually, batch events and write them together.

# ✅ REAL: Batch events before writing
from asyncio import gather
import asyncio

class EventBatcher:
    def __init__(self, batch_size=1000, flush_interval=5):
        self.batch_size = batch_size
        self.flush_interval = flush_interval
        self.queue = []
        self.lock = asyncio.Lock()

    async def add(self, event: dict):
        async with self.lock:
            self.queue.append(event)
            if len(self.queue) >= self.batch_size:
                await self._flush()

    async def _flush(self):
        if not self.queue:
            return

        # Batch write to database
        events = self.queue[:]
        self.queue = []

        # Single multi-row insert
        await db.execute(
            """INSERT INTO sensor_data (vehicle_id, timestamp, speed, sensors)
               VALUES """ + ", ".join(["(%s, %s, %s, %s)"] * len(events)),
            [val for event in events for val in [
                event['vehicle_id'],
                event['timestamp'],
                event['speed'],
                event['sensors']
            ]]
        )

        logger.info(f"Flushed {len(events)} events")

batcher = EventBatcher(batch_size=1000, flush_interval=5)

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest(vehicle_id: str, data: SensorData):
    event = {
        'vehicle_id': vehicle_id,
        'timestamp': data.timestamp,
        'speed': data.speed,
        'sensors': data.sensors
    }

    # Add to batch (doesn't block)
    await batcher.add(event)

    # Return immediately (< 5ms)
    return {"status": "queued"}

# Flush periodically
@app.on_event("startup")
async def flush_periodically():
    while True:
        await asyncio.sleep(5)
        await batcher._flush()

The math changes:

Instead of 5,500 individual inserts/second
We do 6 batch inserts/second (5,500 ÷ 1,000 batch size)
Database load: 5,500 events written, but only 6 inserts executed
Result: Database stays relaxed. Latency stays low. We scale.

The Real Numbers

Before Batching:

Throughput: 50 rps (bottlenecked by database)
Latency: 2,500ms (queued inserts waiting)
Database CPU: 95% (constant write pressure)
Cost: $150/month RDS (need bigger instance)

After Batching:

Throughput: 1,000+ rps (limited by API, not database)
Latency: 18ms (batch waits 5 seconds max)
Database CPU: 15% (batches are efficient)
Cost: $100/month RDS (smaller instance works fine)

That's 20x throughput improvement + 33% cost reduction.

The Batching Trade-offs

Trade-off 1: Latency vs Throughput

# Option C: Real-time + Batch split
# - Real-time data: Redis stream (for dashboard)
# - Persistent data: Batch to database (for analytics)

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest(vehicle_id: str, data: SensorData):
    # Real-time: Update Redis immediately
    await redis.hset(
        f"vehicle:{vehicle_id}:latest",
        mapping={
            'speed': data.speed,
            'timestamp': data.timestamp,
        }
    )

    # Persistent: Add to batch (writes later)
    await batcher.add({
        'vehicle_id': vehicle_id,
        'timestamp': data.timestamp,
        'speed': data.speed,
        'sensors': data.sensors
    })

    return {"status": "accepted"}

Result: Dashboard gets data instantly (Redis), analytics get data within 1-5 seconds (database batch).

Advanced: Batch Size Selection

Formula:

batch_size = (target_latency × events_per_second) / batches_per_second

Example:
- Target latency: 5 seconds (acceptable delay)
- Events per second: 5,500
- Desired batches per second: 10 (don't stress DB)

batch_size = (5 × 5,500) / 10 = 2,750

The Real Cost Analysis

Without Batching:

RDS Instance: db.t3.large ($300/month)
Kinesis: 10 shards ($300/month)
Lambda: Heavy ($50/month)
EC2 API servers: 10 instances ($500/month)

Total: ~$1,150/month

With Batching:

RDS Instance: db.t3.small ($100/month)
Kinesis: 1 shard ($30/month)
Lambda: Minimal ($5/month)
EC2 API servers: 2 instances ($100/month)

Total: ~$235/month

Batching saves 80% on infrastructure costs.

TL;DR

Don't insert events one-at-a-time (database will hate you)
Batch 500-2,000 events (sweet spot)
Flush every 1-5 seconds (or when batch is full)
Split real-time (Redis) from persistent (batch) (best of both)
Result: 20x throughput, 80% cost savings

The real lesson: One tiny optimization (batching) can change your entire infrastructure.

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Real-Time Data | Performance | Scale

Kubernetes for Solo Developers: Stop Overthinking It (And Start Using It)

Le Beltagy — Sat, 18 Jul 2026 13:10:33 +0000

Kubernetes for Solo Developers: Stop Overthinking It (And Start Using It)

Why a solo developer should absolutely be running their production system on Kubernetes, and why it's not as scary as you think

The Myth

Most developers think Kubernetes is:

❌ Too complex
❌ For big companies only
❌ Overkill for solo projects
❌ Requires DevOps expertise

I thought that too.

Then I actually deployed VehicleMetrics on EKS and realized:

Kubernetes solved more problems for me than it created.

Why I Actually Use Kubernetes (Solo Dev)

Problem 1: Manual Deployments Are Terrifying

Before Kubernetes:

# SSH into server
ssh ubuntu@prod-server

# Stop old app
sudo systemctl stop myapp

# Deploy new code
cd /app && git pull

# Start new app
sudo systemctl start myapp

# Pray nothing broke

With Kubernetes:

kubectl apply -f deployment.yaml
# Automatic rolling update, zero downtime

That's worth it alone.

Problem 2: When Your App Crashes

Kubernetes automatically restarts crashed pods. No 3 AM alerts for me.

Problem 3: Scaling During Peak Hours

Kubernetes scales automatically. No manual provisioning needed.

The Setup (It's Actually Simple)

Step 1: Create EKS Cluster

resource "aws_eks_cluster" "main" {
  name     = "vehicle-metrics"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.28"
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main"
  node_role_arn   = aws_iam_role.node.arn

  scaling_config {
    desired_size = 3
    max_size     = 10
    min_size     = 1
  }

  instance_types = ["t3.medium"]
}

Step 2: Configure kubectl

aws eks update-kubeconfig --name vehicle-metrics --region us-east-1
kubectl get nodes

Step 3: Deploy

kubectl apply -f deployment.yaml

Day-to-Day Operations

Deploy New Version

docker build -t vehicle-metrics/api:v1.2.3 .
docker push vehicle-metrics/api:v1.2.3

kubectl set image deployment/api-ingestion \
  api=vehicle-metrics/api:v1.2.3

# If something breaks, rollback
kubectl rollout undo deployment/api-ingestion

Check Logs

kubectl logs -f deployment/api-ingestion

SSH Into Pod

kubectl exec -it pod/api-ingestion-xyz -- /bin/bash

Cost Comparison

Manual VMs: ~$300/month
Kubernetes (EKS): ~$263/month

Kubernetes is cheaper AND more reliable.

What Kubernetes Solves

✅ Automatic pod restarts
✅ Zero-downtime deployments
✅ Automatic scaling
✅ Health checks
✅ Secrets management
✅ Load balancing

TL;DR

Solo developers should use Kubernetes
Setup takes 1 hour (Terraform + kubectl)
Day-to-day: one command to deploy
Automatic scaling, restarts, rollbacks
Stop overthinking. Start using it.

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Kubernetes | DevOps | Solo Development