DEV Community: Le Beltagy

Great article for security concepts

Le Beltagy — Sun, 02 Aug 2026 12:53:24 +0000

Le Beltagy

Jul 23

I Deployed a Purple Team Inside My Kubernetes Cluster. The Scanners Never Saw It Coming.

#kubernetes #security #go #purpleteam

5 min read

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s.

Le Beltagy — Sat, 01 Aug 2026 10:07:53 +0000

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s.

From zero Go experience to a merged PR in the most popular Kubernetes terminal UI — how I misread HTTP/1.1 upgrades, broke my own test suite, and learned that create and get are not the same verb when Kubernetes 1.31 changes the rules underneath you.

The Setup

It started with a single function I didn't write.

I'd been contributing to k9s for a week, trying to understand how a 58,000-star project structured its Go code. I was deep in internal/dao/port_forwarder.go, staring at a block of code that checked whether a user was authorized to open a port-forward on a pod. The check was simple: does this service account have create permission on pods/portforward?

I'd read the Kubernetes docs. Port-forward requires create. Everyone said so. The code worked. I moved on.

Then someone opened an issue: "Port-forward fails on k8s 1.31 with restricted RBAC."

Kubernetes 1.31 introduced the PortForwardWebsockets feature — a new code path that forwards ports over WebSockets instead of SPDY. And WebSockets, unlike SPDY, only require the get verb on the pods/portforward subresource.

That was the bug. The code was checking create. The new WebSocket path needed get. Users with get-only access couldn't port-forward in k9s, even though kubectl worked fine for them.

This is the story of how I tried to fix it, failed the first time, and learned more about Go, HTTP upgrades, and Kubernetes authorization in three days than in three weeks of tutorials.

This is not a tutorial. This is an autopsy of a two-line fix that took me forty-eight hours.

Why Not Just Use kubectl?

I use kubectl port-forward professionally. It works. It handles the WebSocket migration transparently. There's no bug report. No edge case. No problem.

But k9s is not kubectl. k9s is a terminal UI that wraps client-go and presents a unified interface for pod interaction. When k9s shows a pod and lets you press Shift-F to forward a port, it first checks if you're allowed to do it. And that check was wrong.

The issue wasn't academic. A user with this RBAC role:

rules:
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["get"]

could run kubectl port-forward successfully on Kubernetes 1.31+ (WebSocket path), but k9s would silently grey out the port-forward option, convinced they lacked permission.

That user was real. The issue had reproduction steps. A real cluster, a real role, a real workflow broken by a verb mismatch I would never have noticed without reading the client-go source.

The k9s Architecture

┌─────────────────────────────────────────────────────────────┐
│                         k9s TUI                              │
│                    (tview / terminal UI)                     │
│  User presses Shift-F on a pod                              │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│              internal/dao/port_forwarder.go                  │
│                                                              │
│  1. Check pod is Running ( readiness gate )                 │
│  2. Authorize: can user create pods/portforward?            │
│         ▲                                                    │
│         │ THIS WAS THE BUG                                   │
│  3. If yes → open port-forward session                      │
│     If no  → grey out option / show error                   │
└──────────────────────┬──────────────────────────────────────┘
                       │
              ┌────────┴────────┐
              │                 │
              ▼                 ▼
    ┌─────────────┐     ┌─────────────┐
    │ SPDY path   │     │ WebSocket   │
    │ (legacy)    │     │ (k8s 1.31+) │
    │ needs       │     │ needs       │
    │ CREATE verb │     │ GET verb    │
    └─────────────┘     └─────────────┘
              │                 │
              └───────┬─────────┘
                      ▼
            ┌─────────────────┐
            │  K8s API Server │
            │  /api/v1/.../   │
            │  portforward    │
            └─────────────────┘

The Decision I Missed

Path	Protocol	Required Verb	k9s Checked?
Legacy SPDY	SPDY/HTTP/1.1 upgrade	`create` on `pods/portforward`	✅ Yes
WebSocket	WebSocket (RFC 6455)	`get` on `pods/portforward`	❌ No

k9s only checked for create. It didn't know about the WebSocket path's get requirement. So get-only users were blocked even though they were authorized for the code path their cluster actually used.

The Failures (And What Each One Taught Me)

Failure 1: I Changed One Word and Called It a Day

My first fix was embarrassingly naive.

I found the function in port_forwarder.go. It built an SelfSubjectAccessReview with verb "create" and sent it to the API server. I changed it to "get". It compiled. I opened a draft PR.

Within an hour, a maintainer commented: "This breaks backward compatibility for clusters still on the SPDY path."

I had replaced one hardcoded verb with another. I didn't check SPDY. I didn't check both. I didn't even think about clusters running Kubernetes <1.31, or clusters where the WebSocket feature gate was disabled. I assumed "new = right."

Root cause: I treated a protocol negotiation problem like a string replacement problem.

The fix: The code needed to check BOTH verbs independently. If create is authorized, the user can port-forward via SPDY. If get is authorized, they can port-forward via WebSocket. If either is authorized, k9s should allow the operation. The API server and client-go handle which protocol actually gets used at connect time.

// What I wrote first (WRONG — only checks get)
if !utils.CheckPodPortFwd(a.factory.Client(), a.factory.Config(), path) {
    return errors.New("insufficient permission")
}

// What merged (RIGHT — checks create AND get independently)
if !utils.CheckPodPortFwd(a.factory.Client(), a.factory.Config(), path) {
    if !utils.CheckPodPortFwdGet(a.factory.Client(), a.factory.Config(), path) {
        return errors.New("insufficient permission")
    }
}

Actually, the final merged code was cleaner — it added a new CheckPodPortFwdGet helper and called both from the port-forward check. The key insight: create and get are not alternatives that replace each other. They're independent capabilities for independent protocol paths. Both can coexist. Neither can be dropped.

Lesson: When a platform supports multiple protocols, a fix for the new one cannot delete support for the old one. Test for coexistence, not replacement.

Failure 2: I Blew Up the Test Suite Because I Didn't Understand `t.Parallel()`

k9s has a real test suite. Not toy tests — table-driven tests with ~50 test cases across multiple scenarios. I added a test that covered the new WebSocket path. It passed locally.

Then CI failed.

The error was a data race in the mock client. I'd instantiated a shared mock RestClient outside the test loop to save setup code. Some tests ran with t.Parallel(). Two parallel tests mutated the same mock's response state simultaneously.

The race detector (go test -race) caught it and failed the build.

Root cause: I thought I was being efficient by sharing mock setup. I was actually creating a concurrency hazard in a test file I didn't fully understand.

The fix: I refactored each test case to build its own RestClient mock inside the test closure. No shared state. The test file I added ended up being 223 lines — almost all of it test cases for every combination of permissions:

Pod not running
No get permission, no create permission → blocked
create-only permission → allowed (legacy SPDY)
get-only permission → allowed (WebSocket path, this was the fix)
Both permissions → allowed

// Part of the 223 lines of tests I ended up writing
{
    name: "get-only-portforward-allowed",
    pod:  runningPod,
    authorized: map[string]bool{
        "selfsubjectaccessreviews": true,
        "pods":                   true,
        "portforwardget":         true,
    },
    want: true,
},

Lesson: Parallel tests are not free. If you don't own the test infrastructure, assume shared state is forbidden until proven otherwise. A data race in a test is a resume stain.

Failure 3: I Didn't Know What an HTTP/1.1 Upgrade Was

The real bug was deeper than a verb string. I needed to understand WHY get was sufficient for WebSockets but not for SPDY.

SPDY (the legacy protocol) is initiated by an HTTP POST request to the port-forward endpoint. HTTP POST maps to the create verb in Kubernetes authorization. So SPDY port-forwards require create.

WebSockets are initiated differently. They start with an HTTP GET request that includes an Upgrade: websocket header. The server responds with 101 Switching Protocols, and the connection becomes a WebSocket.

Because the initial request is GET, Kubernetes authorization maps it to the get verb on the pods/portforward subresource.

I didn't know any of this when I opened the issue. I spent an evening reading:

RFC 6455 (The WebSocket Protocol)
Kubernetes client-go source for StreamWithContext
The KEP for PortForwardWebsockets (KEP-4006)
k9s own issue history for how it handles client-go deprecation

Root cause: I thought port-forward was "one API call." It's actually a protocol negotiation dance between client-go, the API server, and the kubelet. The authorization model is protocol-dependent.

The fix: I didn't change the fix. The code was correct once I checked both verbs. But I did update the error message to indicate both acceptable verbs, so users who were blocked would know exactly which permissions they needed:

return fmt.Errorf("user is not authorized to create or get portforward %q", path)

Lesson: Before you fix a bug in a distributed system, understand the protocol it's built on. The symptom was a missing verb. The cause was a protocol upgrade. The fix that didn't understand the protocol would have broken the legacy path.

The PR

Stat	Value
Changed files	2
Lines added	240
Lines deleted	3
Files touched	`internal/dao/port_forwarder.go`, `internal/dao/port_forwarder_test.go`
Tests added	6 test cases covering every permission combination
Review rounds	2
Time from first draft to merge	48 hours

The diff is small. The logic is simple. The effort was in understanding why the logic needed to exist at all — and proving it with tests that would catch someone else making my first-draft mistake.

PR title: fix(dao): allow port-forward with 'get' verb on pods/portforward for K8s 1.31+ WebSocket path

It merged cleanly. No follow-up fixes needed.

What I Learned About Contributing to Big Projects

1. Start with the issue, not the code

I read the original issue (#4144) three times before I touched a file. The reporter included their Kubernetes version, their RBAC role, and the exact error message. Without that reproduction, I would have never understood the WebSocket migration context.

2. Read the test file before the source file

port_forwarder_test.go taught me more about how k9s handles authorization than port_forwarder.go. Tests are documentation that can't lie.

3. `go test -race` is not optional

The race detector caught my bug before a maintainer did. It turned an embarrassing review comment into a private CI failure. Run it locally. Always.

4. Two-line fixes need two-hundred-line tests

The production change was ~20 lines. The tests were 223 lines. That's the ratio in production Go. If your fix doesn't have a test that would have failed before the fix, your fix isn't done.

5. Backward compatibility is a constraint, not a suggestion

My first draft worked on Kubernetes 1.31. It would have broken k9s for half the user base still on 1.30 or older. A real fix supports both paths simultaneously. This is what separates "it works on my machine" from "it merged into k9s."

Why This Matters (And Why It Doesn't)

It matters because:

k9s has ~58,000 GitHub stars and thousands of daily users. My fix affects real people with real clusters.
I learned Go by reading client-go source and writing table-driven tests, not by building TodoMVC.
I now understand Kubernetes authorization, HTTP upgrades, and protocol negotiation at a level I didn't before.

It doesn't matter because:

It's a 20-line fix. Senior engineers at Google write diffs like this before coffee.
I didn't architect a new subsystem. I didn't refactor the codebase. I fixed a verb check.
The value isn't in the code. The value is in the proof that I can read an issue, understand the context, write a correct fix, survive review, and ship.

That's the signal I'm building.

What's Next?

I'm continuing to contribute to k9s. The codebase is complex enough to keep teaching me things — how tview renders terminal UIs, how k9s watchers avoid polling the API server, how the dao layer abstracts client-go operations.

I also contributed to Checkov today — adding missing GCP taggable resources for Cloud SQL and GKE clusters. Small PR. Two lines. Merged in 20 minutes. That's what happens when you build the muscle: the second contribution is faster than the first.

TL;DR

I started learning Go three weeks ago. I found a bug in k9s where port-forward authorization only checked the create verb, missing the get verb needed by Kubernetes 1.31's new WebSocket path.

My first fix was wrong — I replaced create with get and nearly broke backward compatibility for legacy SPDY clusters. The final fix checks both verbs independently.

I wrote 223 lines of table-driven tests to cover every permission combination. The PR merged in 48 hours. A data race in my test taught me that parallel tests don't share mock state.

If you're learning a language, don't build tutorial projects. Find a real bug in a real project and fix it. The test suite will teach you more than any course.

What's your first open-source contribution story? Drop it in the comments — especially if you also broke a test suite and lived to tell about it.

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

Le Beltagy — Sat, 25 Jul 2026 17:55:01 +0000

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

From a bricked Raspberry Pi to a 4-node Talos Linux cluster running Cilium eBPF, ArgoCD, and Longhorn — every failure, every fix, and the homelab that now hosts my production side projects.

The Setup

It started with a Dell OptiPlex I found in a recycling pile.

Intel i5-6500, 16GB DDR4, a 256GB SATA SSD. Someone's discarded office PC. I wiped the Windows 10 sticker off the case, installed Proxmox, and told myself: "This is going to be my cloud."

Three months later, that OptiPlex plus three Raspberry Pi 4s, a second-hand managed switch, and a drawer full of dead SD cards became a 4-node bare-metal Kubernetes cluster running:

Talos Linux — immutable, API-driven OS built for K8s
Cilium — eBPF-based CNI replacing kube-proxy and providing cluster mesh
ArgoCD — GitOps continuous delivery
Longhorn — distributed block storage across nodes
MetalLB — bare-metal load balancing with BGP
Prometheus + Grafana — monitoring stack pulling metrics from the cluster and my ISP's router

Total cost: ~$220 (OptiPlex was free, Pis were ~$60 each, switch was $35 on eBay).

This is not a tutorial. This is an autopsy of every mistake I made building it.

Why Not Just Use EKS?

I run AWS infrastructure professionally. I can spin up an EKS cluster in 20 minutes with Terraform. I know what managed Kubernetes costs at scale.

But I was hitting a wall:

1. You can't break AWS

In production, I follow well-architected patterns. VPCs, private subnets, IAM roles with least privilege, GuardDuty, Security Hub. Everything is safe. Everything is boring.

In my homelab, I can delete the CNI and watch the cluster melt. I can spin up a privileged container with hostNetwork: true and see what breaks. I can push a bad Cilium network policy and lock myself out of every node — which I did. Twice.

2. The gap between "I passed CKA" and "I understand Kubernetes"

Certifications teach you how to use the API. They do not teach you what happens when etcd runs out of disk because Longhorn replicas ballooned and you didn't set volume quotas. They do not teach you that Cilium's eBPF kube-proxy replacement silently breaks certain firewall rules in ways that are invisible to kubectl.

3. If I can't self-host it, do I really know it?

This is my rule. I don't put a technology on my CV unless I've operated it without a managed control plane. Talos, Cilium, ArgoCD — I wanted to feel them fail.

The Architecture

┌─────────────────────────────────────────────────────────────┐
│                    DELL OPTIPLEX (Control Plane)             │
│                  Talos Linux · x86_64 · 16GB RAM             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  etcd (q=1) │  │ API Server  │  │  Controller Manager │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  Scheduler  │  │ Cilium Agent│  │  Talos API Server   │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
           ┌──────────────────┼──────────────────┐
           ▼                  ▼                  ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│   RPI-01 (arm64)│ │   RPI-02 (arm64)│ │   RPI-03 (arm64)│
│  4GB RAM · 64GB │ │  4GB RAM · 64GB │ │  4GB RAM · 64GB │
│   USB-SSD boot  │ │   USB-SSD boot  │ │   USB-SSD boot  │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │ Cilium Agent│ │ │ │ Cilium Agent│ │ │ │ Cilium Agent│ │
│ │ kubelet     │ │ │ │ kubelet     │ │ │ │ kubelet     │ │
│ │ Longhorn    │ │ │ │ Longhorn    │ │ │ │ Longhorn    │ │
│ │ (replica)   │ │ │ │ (replica)   │ │ │ │ (replica)   │ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
           │                  │                  │
           └──────────────────┼──────────────────┘
                              ▼
                    ┌─────────────────┐
                    │  Netgear GS308  │
                    │  8-Port Gigabit │
                    └─────────────────┘

Design Decisions

Decision	Chose	Over	Because
OS	Talos Linux	Ubuntu + kubeadm	Immutable, API-driven, minimal attack surface, no SSH
CNI	Cilium (eBPF)	Flannel / Calico	kube-proxy replacement, Hubble observability, cluster mesh ready
Storage	Longhorn	NFS / local-path	Distributed block storage, snapshots, cross-node replication
GitOps	ArgoCD	Flux / manual	UI for visualizing sync state, familiar to teams
LB	MetalLB (L2 mode)	NodePort	True LoadBalancer IPs on bare metal, no cloud provider
Boot	USB SSD (all nodes)	SD cards	SD cards die under etcd I/O. USB SSDs survive.

The Failures (And What Each One Taught Me)

Failure 1: I Burned 3 SD Cards in 6 Weeks

My first cluster booted from 64GB SanDisk Ultra SD cards. They were cheap. They were fast — for the first week.

Then etcd started crashing. The logs showed mvcc: database space exceeded. I ran etcdctl defrag and it helped for a day. Then the Pi locked up. I rebooted. The SD card wasn't recognised anymore. Corrupted beyond repair.

The second SD card died the same way. The third one survived longest because I moved /var/lib/longhorn to an NFS share — but the card still degraded after 5 weeks.

Root cause: Etcd does frequent small writes. SD cards use wear-leveling algorithms designed for photos (large sequential writes), not databases (random small writes). The flash translation layer dies.

The fix: Every node now boots from a $25 USB3 SSD. Talos supports this natively — you flash the installer ISO, boot from USB, and install to the SSD. The difference is staggering. I've been running the current cluster for 8 months with zero storage-related crashes.

Lesson: Storage media is not interchangeable. Understand your write patterns before you pick hardware.

Failure 2: A Bad CiliumNetworkPolicy Locked Me Out of Every Node

I was experimenting with zero-trust network policies. I applied a CiliumNetworkPolicy that allowed ingress only from pods with label app=frontend. I forgot to include the kube-system namespace. I forgot to include Cilium's own health checks. I applied it with kubectl apply -f.

Within 30 seconds, my SSH session to the control plane hung. kubectl timed out. The Pi nodes showed as NotReady. I had physically unplugged and rebooted everything before I could get back in.

What happened: Cilium enforces policies at the eBPF level — before packets even reach iptables or the host network stack. A bad policy doesn't just drop traffic; it drops it invisibly. No logs in dmesg. No iptables -L entries. Just silence.

The fix: Talos has a recovery mechanism. You boot the node with a kernel parameter talos.config=none and it starts without the network stack enabled. From there, you can mount the Talos EPHEMERAL partition, edit the CNI state, and reboot. But I didn't know this at the time.

The real fix is procedural:

Always test network policies with cilium policy trace before applying
Always keep a default-allow-all policy in Git, ready to apply
Never apply network policies on Friday evening (learned the hard way)

Lesson: eBPF is powerful because it's invisible. That same invisibility makes debugging a nightmare. Test before you enforce.

Failure 3: Longhorn Filled My Disks and kubelet Evicted Everything

I deployed a PostgreSQL database for MarketPulse with a 10Gi Longhorn volume. Longhorn replicates volumes across nodes for HA. I set replica count to 3.

What I didn't know: Longhorn stores replicas as sparse files that grow as data is written. It also keeps snapshots for backups. My "10Gi" database had 3 replicas × 10Gi × 2 snapshots = 60Gi actual usage.

Each Pi had a 120Gi SSD. Longhorn consumed 60Gi. Talos system overhead consumed 20Gi. The remaining 40Gi filled up with container images and logs.

One morning, every pod was Evicted. kubelet had hit the 85% disk pressure threshold and started killing pods to reclaim space. But Longhorn volumes are block devices — kubelet can't delete them. The cluster entered a death spiral.

The fix: I now enforce:

Volume quotas in Longhorn (max 2 replicas for non-critical workloads)
Automatic snapshot cleanup (retain 2, delete older than 7 days)
Node disk alerts in Prometheus at 70% usage, not 85%
Container image garbage collection: imageGCThreshold: 70

Lesson: Distributed storage is not free storage. Replicas × snapshots × logs = real disk consumption. Do the math before you deploy.

Failure 4: MetalLB Gave Out IPs Already Used by My Router's DHCP Pool

I installed MetalLB in Layer 2 mode and assigned it the range 192.168.1.100-192.168.1.150. I exposed a LoadBalancer service. It got 192.168.1.105. I could curl it from my laptop. Success!

Three hours later, my wife's laptop couldn't reach the internet. It had been assigned 192.168.1.105 by the router's DHCP pool. Same IP. Two MAC addresses. Chaos.

The fix: I logged into my ISP router (which I hate but cannot replace), found the DHCP range, and moved MetalLB to 192.168.1.200-192.168.1.250 — outside the DHCP pool.

Better fix: I later configured MetalLB in BGP mode. My router (running OpenWrt on a separate access point) peers with MetalLB and learns routes dynamically. No IP conflicts. No L2 ARP spoofing. True routing.

Lesson: IP address management is not optional on bare metal. Your homelab is a real network. Treat it like one.

Failure 5: I Upgraded Talos and Lost the Cluster Because I Didn't Backup etcd

Talos makes OS upgrades trivial: talosctl upgrade --image ghcr.io/siderolabs/installer:v1.7.0. One command, rolling reboot, cluster stays up.

Except I upgraded from v1.6.0 to v1.7.0 while the cluster was under disk pressure (see Failure 3). The control plane rebooted. etcd failed to start because the WAL files were corrupted from the previous disk-full event. The cluster had no quorum. I had no backup.

I spent 6 hours trying to recover etcd with etcdctl snapshot restore. The snapshot I had was 3 weeks old. I lost:

All ArgoCD application definitions (I had them in Git, so I re-synced)
All Prometheus metrics history
All Longhorn volume states (the PVCs survived, but the volume attachments were inconsistent)

The fix:

Automated etcd snapshots every 6 hours to S3 (yes, I back up my homelab to AWS — the irony is not lost on me)
Pre-upgrade health checks: disk <70%, all nodes Ready, all Longhorn volumes healthy
I wrote a tiny Go CLI that wraps talosctl upgrade with these checks. It lives in my homelab-tools repo.

Lesson: Immutable infrastructure doesn't mean unbreakable infrastructure. Back up your control plane state. Or learn etcd disaster recovery at 2 AM like I did.

What's Running on It Now?

Workload	Purpose	Resource Usage
kube-radar	Public Go CLI for K8s RBAC wildcard detection	50m CPU, 128Mi mem
job-digest	Python job scraping pipeline	200m CPU, 256Mi mem
MarketPulse API	Go + FastAPI real-time analytics backend	500m CPU, 512Mi mem
PostgreSQL	Primary DB for MarketPulse (Longhorn-backed)	300m CPU, 512Mi mem
Prometheus	Cluster + app metrics	400m CPU, 1Gi mem
Grafana	Dashboards for everything	100m CPU, 128Mi mem
ArgoCD	GitOps sync for all workloads	200m CPU, 512Mi mem
Cilium Hubble	Network flow observability	100m CPU, 256Mi mem
Longhorn	Storage engine (daemonset)	150m CPU, 512Mi mem

The OptiPlex sits at ~60% CPU under normal load. The Pis hover at 40-70%. I can burst higher, but I know the limits.

3 Lessons That Transfer to Production

1. Resource quotas are not bureaucracy

In my homelab, I have no cost optimization team. When Longhorn eats all disk, I pay with crashed pods and 2 AM debugging. Setting requests and limits, configuring PVC sizes, defining retention policies — these are not "enterprise overhead." They are survival mechanisms.

2. Observability first, workload second

I deployed Prometheus and Grafana before I deployed any application. Not because I'm disciplined — because I learned that debugging a bare-metal cluster without metrics is like performing surgery in the dark. Hubble shows me every dropped packet. Node exporter shows me disk I/O patterns I never would have guessed. In production, I'd push this to Thanos or Cortex. In my homelab, I run a 30-day retention and call it enough.

3. GitOps is the only way I trust myself

I have root access to every node. I can kubectl exec into any pod. Without ArgoCD, I would absolutely make manual changes at 11 PM and forget them by morning. With ArgoCD, the cluster state is in Git. If I change something manually, the UI shows it as OutOfSync and I feel shame. That shame keeps my cluster declarative.

The Hardware List (If You Want to Replicate)

Component	Spec	Cost	Notes
Control plane	Dell OptiPlex 7050, i5-6500, 16GB	Free (recycled)	Any x86_64 box works. Needs VT-x.
Worker nodes	3× Raspberry Pi 4 (8GB)	~$60 each	4GB works, 8GB is comfortable
Worker storage	3× 120GB USB3 SSD	~$25 each	Avoid SD cards for etcd. Just don't.
Switch	Netgear GS308 8-port gigabit	$35 (eBay)	Any managed switch for VLANs
Cables	Cat6 patch cables	~$15	Buy a pack
Total		~$220

The Pi 5 exists now and is faster. I'm staying on Pi 4 because arm64 compat is tested and I'm lazy.

TL;DR

I built a 4-node bare-metal Kubernetes cluster on $220 of scrap hardware and Raspberry Pis. I use Talos Linux, Cilium eBPF, ArgoCD, Longhorn, and MetalLB.

I burned SD cards, locked myself out with network policies, filled disks with storage replicas, caused IP conflicts, and corrupted etcd during an upgrade.

Every failure taught me something I use in production. The cluster now runs my public projects, my monitoring, and my experiments.

If you're learning Kubernetes, break it at home before you break it at work.

What's your homelab stack? Drop it in the comments — always looking for new failure modes to learn from.

I Deployed a Purple Team Inside My Kubernetes Cluster. The Scanners Never Saw It Coming.

Le Beltagy — Thu, 23 Jul 2026 19:50:05 +0000

Three security scanners gave my cluster a clean bill of health. Then I deployed an attacker into a DaemonSet and discovered what static analysis actually misses.

The Setup

My homelab cluster is nothing exotic: a few Hyper-V VMs, a Raspberry Pi worker, and a Talos Linux control plane running the usual suspects: ArgoCD for GitOps, Cilium for CNI, Longhorn for storage. My workload is a typical cloud-native stack: a Go backend, a React frontend, PostgreSQL, the works.

I did exactly what the security playbooks tell you to do:

Trivy for container image scanning
kube-bench for CIS Kubernetes Benchmark compliance
Polaris for best-practice configuration checks

Every single tool reported green. I took a screenshot of the dashboard and felt safe.

Then something felt wrong.

A scanner audits what you declared. It does not test what happens when things go wrong. It is like a home inspector checking your door locks while ignoring the open window around the corner. I wanted a burglar on my side.

So I built one.

Introducing NEMESIS

NEMESIS is a Kubernetes-native purple team platform. It lives inside the cluster as a DaemonSet, not outside as a scheduled scan. It does not read manifests and compare them against registries. It executes attack scenarios from the same pod network your real workloads use.

The Stack

The platform splits along two languages for very specific reasons:

Python Controller — The orchestration brain. Schedules attack scenarios, correlates events, and emits findings. Python was the right call here: rapid scenario development and rich security libraries.
Go Attack Agent — A DaemonSet on every node. Executes the actual attack primitives: token abuse, lateral movement, network probing. Go gives me a single static binary, sub-second cold-start, and zero dependency hell inside a distroless container.

Attack Engine

Scenarios are not hard-coded exploits. They are a composable library of primitives:

Recon — Enumerate service accounts, roles, secrets, and network policies from inside the pod
Token Abuse — Mount and abuse service account tokens against the API server
Lateral Movement — Test east-west connectivity between namespaces with no external tooling
Exfil Probe — DNS and HTTP egress tests to measure actual network policy enforcement

eBPF Integration

Because the cluster runs Cilium, NEMESIS hooks into eBPF-based network telemetry. Every packet the attack agent sends is observable at the kernel layer. This means I do not just know that an attack path exists; I have flow-level proof of exactly which packet traversed which interface.

The Attack Chain That Scanners Missed

Here is one scenario, step by step. I anonymized nothing else because the configuration is common.

Step 1: Recon

The attack agent starts inside a pod in the default namespace. It enumerates service accounts:

curl -s https://kubernetes.default.svc/api/v1/namespaces/default/serviceaccounts \
  --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

It finds the default service account. Then it checks its bindings:

curl -s https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings

Result: a ClusterRoleBinding named cluster-reader-default grants cluster-reader to the default SA across all namespaces.

Step 2: Token Abuse

With cluster-reader permissions, the agent queries every namespace for secrets:

curl -s https://kubernetes.default.svc/api/v1/secrets --all-namespaces

It finds a ConfigMap in the backend namespace named db-config containing a PostgreSQL connection string with a hard-coded password.

What the scanner said: "ConfigMap db-config exists. No hardcoded secrets detected in source image."

What NEMESIS found: The ConfigMap is readable by any pod using the default service account, which is every pod in the cluster that does not explicitly override it.

Step 3: Lateral Movement

The agent lists pods across namespaces:

curl -s https://kubernetes.default.svc/api/v1/pods --all-namespaces

It identifies the backend API pod in the backend namespace. It checks the NetworkPolicy for that namespace:

# backend-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Ingress

What the scanner said: "NetworkPolicy exists. Default-deny posture implemented."

What NEMESIS found: The policy only specifies Ingress. There is no Egress policy. The backend pod can open outbound connections to anything.

Step 4: Data Exfil Test

The agent attempts a DNS lookup to an external domain controlled by the test harness:

nslookup exfil-test.nemesis.internal

Cilium's eBPF probe captures the packet at the kernel socket layer. The DNS query reaches the cluster DNS, which forwards it upstream. No policy blocked it.

NEMESIS logs this as: "DNS exfiltration path confirmed — no egress filtering on namespace backend."

The Gap: What You Declare vs. What You Get

Out of 47 unique findings NEMESIS surfaced across three weeks of continuous testing, here is the breakdown versus what the traditional scanner suite caught:

Finding Category	NEMESIS	Scanner Suite	Gap
Lateral movement paths	12	0	100%
Service account token abuse	9	1	89%
Network exfiltration (DNS/HTTP)	8	0	100%
Network policy bypass	6	0	100%
Runtime container escape	3	1	67%
Image CVEs	2	7	Covered by scanner
CIS misconfigurations	4	5	Partial overlap
Manifest best practices	3	6	Covered by scanner
Total	47	20	~81% unique to NEMESIS

To be completely fair: the scanners caught every image vulnerability and most CIS misconfigurations. They are excellent at what they do. They just do not test runtime behavior.

Why This Changes How You Think About K8s Security

The industry has optimized for speed of detection. But detection speed means nothing if you are detecting the wrong things.

Scanners answer: "Is this configuration compliant?"

Purple teams answer: "What happens when someone already inside tries to move?"

Both questions matter. But in a world where supply chain attacks, compromised CI pipelines, and insider threats are the dominant risk models, the second question is the one that actually keeps you awake.

NEMESIS is not a replacement for scanning. It is an additional layer that validates your assumptions at runtime, continuously, from the same vantage point a real attacker would have.

What I Learned Building This

The Go-vs-Python split was not philosophical. It was operational. The agent needs to start in under a second when a node scales up, handle network timeouts gracefully, and compile to a single binary that fits in a scratch container. Go wins here without debate.

The controller needs to parse YAML attack definitions, talk to the Kubernetes API, and integrate with Cilium's Hubble for flow data. Python's ecosystem made this painless.

The hardest part was not the attack code. It was building the telemetry correlation: matching an eBPF flow event to a specific attack scenario step so you can say with certainty: "The DNS leak on step 4 of scenario backend-exfil-001 originated from pod backend-api-7d9f4b8c5-x2k9m at 14:23:07 UTC."

Without that correlation, purple team findings are just noise.

Where NEMESIS Goes Next

The current milestone focuses on Kubernetes, but the architecture is generic. The same DaemonSet pattern applies to:

Container runtime testing (Docker socket abuse, privileged escalation)
Cloud metadata service abuse (IMDSv2 bypass attempts on AWS/Azure nodes)
Storage layer attacks (Longhorn volume snapshot tampering, RWO bypass)

The goal is not to build a scanner with a different skin. The goal is to shift the security posture from "we read your configuration" to "we prove what an attacker can actually do."

Try It Yourself

If you want to see what your scanners are missing, the fastest path is deliberate:

Spin up a throwaway pod in your cluster
kubectl exec into it
Run curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/default/secrets
See what comes back

If you see more than an empty list, you have found your first gap without writing a single line of code.

NEMESIS is an open-source Kubernetes-native purple team platform. If you want to follow the build, attack scenario library, or contribute, the repository is at github.com/beltagyy/nemesis.

Built with Go, Python, Cilium, and enough caffeine to make a kernel panic look calm.

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

Le Beltagy — Wed, 22 Jul 2026 10:50:05 +0000

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

From manually copy-pasting cover letters to a Python pipeline that hunts Dutch tech roles while I sleep — the architecture, the failures, and why I need your help.

The Setup

I had applied to 73 jobs in 4 weeks.

Not casual browsing. I mean:

Custom cover letters per role
CV tweaks for Platform Eng vs SRE vs Cloud Security
LinkedIn cold messages to hiring managers
Tracking everything in a Notion database

The result? Ghosted by 61. Rejected by 8. Phone screen with 4.

Meanwhile, I was working full-time at Siemens, studying Go for kube-radar, and maintaining a homelab that was literally on fire one Tuesday because a Cilium policy update bricked the cluster network. I was burning out on the process of job searching instead of actually getting better at it.

So I did what any engineer would do: I automated the pain away.

That's how job-digest started — a Python pipeline that scrapes job boards, filters by my exact criteria, ranks roles by match quality, and delivers a daily summary to my Telegram. It didn't get me hired overnight. But it got me interviewed by companies I actually wanted to work for — and it taught me more about web scraping, async Python, and data pipelines than any course.

This is the story of building it. The code is public. I want collaborators. If you're job-hunting in tech and you know Python, read on.

Why This Problem?

Cloud consulting taught me how to build systems. It did not teach me how to sell myself.

When I decided to relocate to Amsterdam 🇳🇱 and target backend/platform roles, I faced three problems no recruiter warns you about:

1. The signal-to-noise ratio is brutal

Job boards are flooded with "Cloud Engineer" roles that are actually MSP helpdesk jobs, "DevOps" roles that mean "you're the only ops person for 40 devs," and "Kubernetes" roles at companies running Docker Compose on a single EC2 instance.

I needed filtering that understood my stack: Go, Kubernetes internals, Cilium/eBPF, Azure + AWS multi-cloud, security compliance (NIS2/DORA).

2. Timing asymmetry

The best roles at Adyen, Booking.com, ASML — they go live on Tuesday morning and have 200 applicants by Thursday afternoon. I was checking boards manually on Sunday evenings. I was always late.

3. Context switching kills deep work

Every 20-minute "let me check LinkedIn" session destroyed my flow state. I wanted one daily digest at 7 AM. Read it on the train. Apply to the top 3 roles. Then close the tab and write Go code.

The Architecture

┌─────────────┐    ┌──────────────┐    ┌──────────────┐    ┌─────────────┐
│  Scrapers   │───►│   Enricher   │───►│    Ranker    │───►│   Notifier  │
│ (async)     │    │ (LLM + rules)│    │ (scoring)    │    │ (Telegram)  │
└─────────────┘    └──────────────┘    └──────────────┘    └─────────────┘
     │                                              │
     ▼                                              ▼
 LinkedIn Jobs                              SQLite cache
 Indeed                                     (dedup + history)
 Glassdoor
 We Work Remotely
 RemoteOK

Design Decisions

Decision	Chose	Over	Because
Language	Python 3.11 + asyncio	Go	Rapid prototyping, BeautifulSoup/httpx ecosystem, LLM SDKs
Storage	SQLite	PostgreSQL	Zero infra, portable, single file. WAL mode for concurrent reads.
Scraping	httpx + BeautifulSoup	Selenium/Playwright	Faster, lighter, cheaper. JavaScript-rendered boards get headless fallback.
Rate limiting	Adaptive backoff (exponential + jitter)	Fixed delays	Respects robots.txt without being slower than necessary.
LLM enrichment	OpenAI GPT-4o-mini	Local LLM	Cost: ~$0.02 per 100 jobs. Quality worth it for extraction.
Delivery	Telegram Bot API	Email	I live in Telegram. Push notification = instant awareness.
Scheduling	cron on homelab RPi	GitHub Actions	Free, persistent, no cold start. Runs at 06:00 CET daily.

The Pipeline (Simplified)

# 1. COLLECT — async scrapers, one per source
async def scrape_linkedin(query: str, location: str) -> list[RawJob]:
    # httpx + BeautifulSoup
    # Extract: title, company, location, description, salary, url, posted_at
    # Returns list of RawJob dataclasses

# 2. NORMALIZE — unify schema across sources
def normalize(raw: RawJob) -> Job:
    # Map "Amsterdam, North Holland" → "Amsterdam"
    # Extract seniority: Junior / Mid / Senior / Staff from title
    # Detect remote policy: onsite / hybrid / remote
    # Parse salary ranges from description text (regex + LLM fallback)

# 3. FILTER — hard rules (cheap, deterministic)
def passes_filter(job: Job) -> bool:
    return (
        job.location in TARGET_CITIES  # Amsterdam, Berlin, Dublin, etc.
        and job.seniority in {"Senior", "Staff", "Principal"}
        and any(kw in job.description.lower() for kw in TECH_KEYWORDS)
        and job.company not in BLOCKLIST  # agencies, body shops
    )

# 4. ENRICH — LLM judges fit quality
def enrich_with_llm(job: Job) -> EnrichedJob:
    prompt = f"""
    Role: {job.title} at {job.company}
    Description excerpt: {job.description[:2000]}

    Candidate profile: 6 years cloud/platform, K8s internals (Cilium, Talos),
    Go learner, Azure+AWS, security compliance (NIS2/DORA).
    Target: Platform Engineer or Backend Engineer in Amsterdam/Berlin/Dublin.

    Tasks:
    1. Is this role genuinely backend/platform work (not IT support, not MSP)?
    2. Does the tech stack mention K8s, Go, distributed systems, or cloud-native?
    3. Visa sponsorship likely? (big tech yes, small local shop probably no)
    4. Score 0-100 for fit.
    5. Is it worth applying? yes/no/maybe.
    """
    response = openai.chat.completions.create(...)
    return EnrichedJob(..., llm_score=response.score, verdict=response.verdict)

# 5. RANK — composite score
def score(job: EnrichedJob) -> float:
    return (
        job.llm_score * 0.5 +
        salary_score(job.salary_max) * 0.2 +
        company_tier_score(job.company) * 0.2 +  # Tier 1 = Adyen/Booking/ASML/etc.
        recency_score(job.posted_at) * 0.1        # Fresh postings score higher
    )

# 6. NOTIFY — formatted Telegram message
async def send_digest(jobs: list[EnrichedJob]):
    msg = format_telegram(jobs[:10])  # top 10 only
    await telegram_bot.send_message(chat_id=MY_CHAT, text=msg)

The Telegram Output

🔍 Daily Job Digest — 2026-07-22
18 new roles | 4 match your filters | Top 3 below

━━━━━━━━━━━━━━━━━━━━
🥇 94 pts — Adyen — Platform Engineer
   📍 Amsterdam 🇳🇱  |  💰 €90-120K  |  🏠 Hybrid
   🛠 Go · K8s · Kafka · Terraform
   📝 "Build internal primitives for developer platforms..."
   ✅ LLM: "Strong match. Go-heavy platform team. Sponsors HSM."
   🔗 https://careers.adyen.com/jobs/...

🥈 88 pts — Booking.com — SRE
   📍 Amsterdam 🇳🇱  |  💰 €85-110K  |  🏠 Hybrid
   🛠 K8s · Prometheus · Python/Go · Cilium
   ✅ LLM: "Excellent infra depth match. Large K8s fleet."

🥉 81 pts — N26 — Platform Engineer
   📍 Berlin 🇩🇪  |  💰 €80-110K  |  🏠 Hybrid
   🛠 Go · K8s · Terraform · Microservices
   ✅ LLM: "Good fit. Fintech, EU Blue Card likely."

📊 Actions: /apply N26  |  /dismiss Adyen  |  /details Booking

5 Failures Before It Worked

Failure 1: LinkedIn Rate-Limited Me in 6 Minutes

My first scraper used a single httpx session with a 2-second delay between requests. I was proud of the backoff. LinkedIn was not impressed.

They returned a 429 after ~30 requests and started serving CAPTCHA pages. I added:

Rotating User-Agent headers (from a pool of 50 real browser strings)
Proxy rotation via free proxy lists (terrible idea — 80% were dead)
Randomized delays (1-5 seconds with jitter)

The fix that actually worked: scrape LinkedIn's public job search JSON endpoint instead of HTML parsing. It returns structured data at /jobs-guest/jobs/api/jobs and is significantly more stable. I also dropped proxy rotation and just respected a 5-second delay. LinkedIn is aggressive but fair.

Lesson: Don't fight the frontend. Find the API the frontend uses.

Failure 2: The LLM Hallucinated a Role at Google That Didn't Exist

I blindly trusted GPT-4o-mini's extraction for the first week. It invented a "Kubernetes Platform Engineer" role at Google Amsterdam with a salary of €500K. The URL 404'd. I had already told my wife about it.

The fix: every scraped job gets a live URL validation (HEAD request, follow redirects). If the URL doesn't return 200, it's dropped before enrichment. LLMs are pattern matchers, not fact checkers.

Lesson: LLMs are enrichment, not validation. Never trust their facts without verification.

Failure 3: I Applied to the Same Job 4 Times

My SQLite schema initially keyed jobs by (title, company). Then a company reposted the same role with a slightly different title. I applied twice. The second time, the recruiter replied: "You already applied last week."

The fix: content hashing. I now hash the normalized job description (after removing whitespace and dates). Same description = same job, even if the title changes. Deduplication is fuzzy, not exact.

Lesson: Job boards are messy. Exact matching is naive.

Failure 4: Running It on My Laptop Meant It Died When I Closed the Lid

I initially ran the script manually: python main.py, wait 8 minutes, get results. But I kept forgetting. And when I closed my MacBook, the script died.

The fix: a headless Raspberry Pi 4 in my homelab runs a cron job at 06:00 CET. It has a persistent SQLite file on an external SSD. The Pi also runs my DNS (Pi-hole), my monitoring (Prometheus), and now my job search. One device, many duties.

Lesson: Automations that depend on you remembering to run them are not automations.

Failure 5: I Got 18 "Matches" and Felt Overwhelmed

Even with filtering and scoring, getting 18 "good" roles in one digest meant I applied to none. Decision paralysis. I would read all 18, open 12 tabs, and close the laptop exhausted.

The fix: the digest shows top 3 by default. To see 4-10, you reply /more. To see the rest, /all. This tiny UX change meant I actually applied to the top 3 instead of bookmarking 12.

Lesson: More data is not better. More decisions is worse.

Results (So Far)

I won't claim this got me hired. I'm still interviewing. But here's what changed:

Metric	Before	After
Time spent browsing jobs/day	45 min	5 min reading digest
Applications/week	~8 (sporadic)	~12 (consistent)
Roles matching my stack	~30% of browsed	~85% of digest
Time-to-apply after posting	3-4 days	<24 hours
Relevant recruiter conversations/month	1-2	4-5

The real win: mental bandwidth. I stopped treating job search as a thing I do when I'm tired at 10 PM. It became a system that runs without me.

Companies where the pipeline surfaced roles I wouldn't have found manually:

A stealth-mode fintech in Amsterdam hiring Go platform engineers (not on LinkedIn, found via We Work Remotely)
A CDN company in Berlin rebuilding their edge on Cilium + eBPF (mentioned in a blog post the scraper indexed)
A Series B startup in Dublin with ex-Google SREs building a K8s-native security product (literally my exact niche)

The Code

git clone https://github.com/beltagyy/job-digest.git
cd job-digest
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp config.example.yaml config.yaml
# Edit: add your OpenAI key, Telegram bot token, target cities, tech keywords
python main.py --run-now

What you get:

Modular scrapers (add a new source in ~50 lines)
SQLite cache with WAL mode
LLM enrichment with prompt templates
Telegram bot interface with simple commands
Docker support for headless deployment

What you need:

Python 3.11+, an OpenAI API key (~$5/month at my volume)
A Telegram bot (free via @botfather)
A server that runs cron (RPi, VPS, old laptop)

I Need Collaborators

This project helps me, but it's built for my profile and my targets. To make it useful for others, I need help with:

1. More scrapers

AngelList / Wellfound (startup roles)
Hacker News "Who is Hiring?" (monthly thread parser)
Greenhouse / Workday ATS backends (many companies use these and have predictable JSON APIs)
Country-specific boards (StepStone.de, Indeed.nl, IrishJobs.ie)

2. Better filtering

Visually parse "years of experience" requirements and flag "possibly underqualified" vs "definitely qualified"
Detect visa sponsorship mentions from description text (pattern-based, not just LLM)
Filter out roles from recruiting agencies vs direct employers

3. Smarter enrichment

Local LLM option (Ollama/Llama 3.1) for people who don't want OpenAI costs
Company research: pull funding stage, Glassdoor rating, team size from public APIs
Salary benchmarking: compare offered range to levels.fyi / Glassdoor data

4. Alternative frontends

Discord bot (many communities live there)
Email digest (for non-Telegram users)
Web dashboard (Streamlit or similar) for browsing history and stats

5. Testing & reliability

Scrapers break when job boards change HTML. We need snapshot tests (save HTML, assert extraction still works).
Mock LLM responses for unit tests (expensive to hit OpenAI in CI).

If any of this sounds like you — especially if you're job-hunting in EU tech and want a tool that actually understands your stack — open an issue or PR.

Even if you're not technical: try it, tell me what job board you want added, and I'll prioritize it.

3 Lessons That Transfer Beyond Job Searching

1. Automate the boring, not the important

I automated discovery and filtering. I did not automate applying or interviewing. The bot finds roles. I still write every cover letter by hand and research every company before applying. The pipeline handles noise; I handle signal.

2. Your side project's tech stack should teach you something

I chose Python + asyncio because I already knew Python but had never built a production async pipeline. I learned:

asyncio.gather for parallel scrapers
SQLite WAL mode and connection pooling
Prompt engineering for structured LLM output (JSON mode, function calling)
Rate limiting and backpressure

If your side project doesn't stretch you, it's just work you don't get paid for.

3. Ship before it's perfect

The first version didn't have LLM enrichment. It was just a list of job titles and URLs sorted by posting date. It was ugly. It still saved me 30 minutes a day.

I added scoring later. Then filtering. Then the Telegram bot. Each increment made it better, but the core value (stop manually browsing job boards) existed on day 1.

What's Next

Feature	Status
Core pipeline (scrape → filter → notify)	✅ shipped
LLM enrichment	✅ shipped
SQLite dedup + history	✅ shipped
More scrapers (HN, Wellfound, StepStone)	🔄 in progress
Company enrichment (funding, Glassdoor)	📋 planned
Web dashboard	📋 planned
Resume-to-job matching score	📋 planned

**
🚀 Try it:** https://github.com/beltagyy/job-digest
**
🤝 Collaborate:** Open an issue with the tag help-wanted or DM me

**

If you're job-hunting right now — what's the most frustrating part? Drop it in the comments. If it's common enough, I'll add it to the pipeline.

TL;DR

I applied to 73 jobs in 4 weeks, got exhausted by the process, and built a Python bot (job-digest) to automate job-board scraping, filtering, and ranking.
It's a pipeline: async scrapers → LLM enrichment → scoring → daily Telegram digest. Runs on a Raspberry Pi at 6 AM.
It cut my daily job-browsing time from 45 minutes to 5, and surfaced roles at companies I actually wanted (Adyen, Booking, N26) that I would have missed manually.
Open source and looking for collaborators: github.com/beltagyy/job-digest. I need help with scrapers, filtering rules, local-LLM support, and a web dashboard.

I Learned Go by Hacking Kubernetes RBAC Security (And Nearly Quit 5 Times)

Le Beltagy — Sun, 19 Jul 2026 22:03:03 +0000

I Learned Go by Hacking Kubernetes RBAC Security (And Nearly Quit 5 Times)

From zero Go experience to shipping kube-radar — a Kubernetes security CLI that detects RBAC wildcard violations. The story, the architecture, the mistakes, and why I did it the hard way.

The Setup

I manage Kubernetes clusters at Siemens. Not "I run kubectl apply from my laptop" manage — I mean Talos Linux on bare metal, Cilium replacing kube-proxy, eBPF policies doing deep packet inspection, ArgoCD running GitOps end-to-end. The kind of infra where one bad RoleBinding can cost you the crown jewels.

But here's the problem: I had never written production Go.

My code was Python — automation, Lambdas, Flask services, AI pipelines. My Go experience was exactly one Pulumi IaC program where I mostly copy-pasted from examples and hoped go build didn't yell at me.

I wanted to become what I call a "backend + cloud-native mega engineer" — someone who can architect a distributed system and ship the code that runs inside it. Not just configure the cluster, but write the controller. Not just deploy the service, but design the API.

So I made a deal with myself:

Build one real, public, useful project in Go. Ship it. Then use it as proof.

That's how kube-radar started.

Why This Project?

Kubernetes RBAC is a nightmare to audit.

If you've ever done a security review on a real cluster, you know the pain:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

This is a cluster-admin equivalent disguised as a single rule. Anyone bound to this can do anything in your cluster. And unlike cluster-admin, it doesn't show up in obvious places like kubectl get clusterrolebindings when you're scanning for privilege escalation.

Wildcard rules (verbs: ["*"], resources: ["*"], apiGroups: ["*"]) are the silent killers of K8s security posture. They slip past most CNAPP tools because they're technically valid — they just give away the kingdom.

I wanted a tool that:

Scans any cluster's RBAC in seconds
Flags wildcard over-permissions with severity
Tracks findings across scans (is this new, or did someone just accept it?)
Gates CI/CD with exit codes — fail the build when someone commits a wildcard rule
Reports in SARIF for GitHub Advanced Security integration

And I wanted to build it from scratch in a language I'd never shipped in. Because if I can ship this, I can ship anything.

The Learning Curve (Or: How I Nearly Gave Up 5 Times)

Attempt 1: The Cathedral Plan

I sat down and wrote a 5,000-word design document.

I'm not kidding:

Full SQLite schema with WAL mode, migrations, and ON DELETE CASCADE
Fingerprinting strategy with SHA-256 canonical hashing
Cobra CLI with 6 subcommands
Rule engine with Go interfaces and a plugin registry
Diff engine with set operations across snapshots
.goreleaser cross-compilation for 4 platforms
SARIF 2.1.0 schema validation

It was beautiful. It was also unbuildable for someone learning Go from zero. After 3 weeks of wrestling with database/sql semantics and SQLite driver imports, I had a schema but no working code.

Lesson 1: A perfect design you can't ship is worthless.

Attempt 2: Copy-Paste Panic

I found a GitHub repo that did something vaguely similar. I copy-pasted Go client-go List calls, cobra wiring, even error wrapping. It compiled! It ran! It... panic'd on startup because I shadowed err in an if block and the outer function didn't know.

Go's error-as-values model broke my brain. In Python, I'd raise an exception and catch it somewhere. In Go, the error is just a return value you need to handle every single time. My code was 40% error handling by line count.

Lesson 2: You can't paste your way to understanding.

Attempt 3: The Interface Rabbit Hole

The design doc said "Rule interface type." I spent a full day understanding why interface{} isn't the same as a typed interface, why empty interfaces are a smell unless you need them, and why Go composition beats inheritance so completely.

It clicked when I realized: a Rule in my analyzer isn't a class. It's a contract.

type Rule interface {
    ID() string
    Describe() Description
    Evaluate(s *model.Snapshot) []model.Finding
}

KR001 (wildcard verbs), KR002 (wildcard resources), KR003 (wildcard apiGroups) — all three are just structs that implement those three methods. No inheritance tree. No base class. Just behavior.

Lesson 3: Go interfaces are small, implicit, and compositional. Stop thinking in classes.

Attempt 4: client-go vs. Kubernetes API

Here's something nobody tells you: kubectl does a lot of heavy lifting that client-go doesn't.

kubectl config current-context uses shell-level kubeconfig resolution. client-go's clientcmd.BuildConfigFromFlags doesn't. It took me 2 hours to figure out why my code was connecting to the wrong cluster — client-go reads the kubeconfig file but doesn't use the current-context field unless you explicitly load it.

// This does NOT use your shell's "kubectl config current-context"
config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)

// This DOES — it loads the kubeconfig and resolves context, cluster, user
config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
    &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},
    &clientcmd.ConfigOverrides{CurrentContext: contextName},
).ClientConfig()

That single discovery changed how I build the kube-radar scan command.

Lesson 4: client-go is lower-level than kubectl in ways that matter.

Attempt 5: Determinism

My first version printed findings in random order every time. Tests flaked. Diff was impossible. Fingerprints changed because I included rule index in the hash.

The fix was a lesson in Go's core philosophy: if the language doesn't guarantee something, you must enforce it yourself. Go map iteration is deliberately randomized. So you sort. Always.

sort.Slice(findings, func(i, j int) bool {
    if findings[i].Severity != findings[j].Severity {
        return findings[i].Severity > findings[j].Severity // desc
    }
    return findings[i].Fingerprint < findings[j].Fingerprint
})

Lesson 5: Randomness is a feature of the language. Determinism is your job.

The Architecture

Here's what I shipped — and more importantly, why the architecture looks like this:

Design Decisions

Decision	Chose	Over	Because
SQLite driver	`modernc.org/sqlite`	`mattn/go-sqlite3`	Zero CGO = trivial cross-compilation. 2× slower, irrelevant at this scale.
State	Local SQLite	Stateless linter	Diff/baseline is the differentiator; stateless tools already exist
Role handling	Unified `RoleLike` type	Separate Role/ClusterRole types	Rules written once; kind preserved for reporting severity
Migrations	Hand-rolled (~20 lines)	goose/golang-migrate	Zero deps, more learning value. Revisit when migrations exceed ~10 files.
Persistence	Per-machine SQLite DB	Centralized API/DB	CLI-first tool; no server to maintain, no network dependency
Fingerprinting	SHA-256 canonical content hash	UUID per finding	Stable across rescans; "same problem" = "same fingerprint"

The Fingerprint System (The Most Underrated Part)

This is where most security tools fail: they generate "new findings" every scan because of random IDs or timestamps. My fingering system:

// v1|<rule_id>|<kind>|<namespace>|<name>|<canonical_rule>
canonical_rule = sort-and-join(apiGroups) + "|" + sort-and-join(resources) + "|" + sort-and-join(verbs)
fp = sha256(v1|KR001|ClusterRole||admin|*|*|*)[:16]

Key properties:

Stable across rescans: Same cluster state → same fingerprint
Survives reordering: If someone reorders rules in a Role, the fingerprint doesn't change
Detects narrowing: verbs: ["*"] → verbs: ["*"] + resources: ["pods"] is a different finding (content changed)

This makes the diff feature in v0.2 trivial: it's just set arithmetic on fingerprint maps.

Exit Codes as CI Contract

0 — Clean scan, no new findings ≥ --fail-on
1 — New unacknowledged findings ≥ --fail-on severity threshold
2 — Operational error (unreachable cluster, bad flags, DB failure)

Why only 3 codes? Because CI pipelines need simple logic: kube-radar scan && deploy || fail works with zero tooling.

What It Looks Like

$ kube-radar scan --context homelab --fail-on medium

RULE    | KIND        | NAMESPACE | NAME          | DETAIL
--------|-------------|-----------|---------------|------------------
KR001   | ClusterRole | -         | admin-legacy  | Wildcard verbs: [*]
KR002   | Role        | default   | app-service   | Wildcard resources: [*]
KR003   | ClusterRole | -         | cluster-admin | Wildcard apiGroups: [*]

3 findings (1 Critical, 1 Medium, 1 Low)  exit code: 1

That exit code: 1 means CI fails. That means the wildcard rule does not get deployed.

3 Lessons That Transfer Beyond This Project

1. Design the hard thing in writing first

My 5,000-word design doc wasn't a waste. It was a map. When I cut the scope to v0.1 (no SQLite, no diff, no SARIF), I knew exactly what each deferred feature looked like because I'd already designed it. The doc became the roadmap.

2. Build for yourself, then write about the pain

Every feature in kube-radar solves a problem I actually have:

Can't find wildcard rules quickly → the scan command
Can't tell if a finding is new → the diff feature (v0.2)
Can't baseline accepted risk → acks (v0.2)
Can't integrate with GitHub Security → SARIF (v0.2)

Tooling built from real pain is better than tooling built for a portfolio.

3. A shipped v0.1 beats a perfect v2.0

I had to fight my own instinct to build everything at once. v0.1 is:

1 command (scan)
3 rules (KR001-003)
Table output only
Fingerprints computed but not yet persisted

But it works. It already catches real violations. It already fails CI builds. Everything else is additive.

The Roadmap

Version	What
v0.1 ✅	Scan, 3 rules, table output, fingerprints, exit codes
v0.2	SQLite store, diff, history, acks/baselines, JSON + SARIF
v0.3	Full rule pack: cluster-admin bindings, secrets read, pods/exec, impersonate, escalate/bind
v1.0	Multi-context fleet scan, config file, goreleaser
v2.0	Controller mode: watch RBAC, Prometheus metrics, admission warnings

Try It

git clone https://github.com/beltagyy/kube-radar.git
cd kube-radar
go build -o kube-radar ./cmd/kube-radar
./kube-radar scan --kubeconfig ~/.kube/config --fail-on medium

Or grab a release binary (when I cut v0.1.0 — soon).

What I need from you:

Does it work on your cluster?
What's missing for your use case?
Are there wildcard variants I'm not catching?

Every issue helps me learn.

About This Series

This is post 1 in the "kube-radar: Building a Security CLI in Go" series. Next up:

Deep-dive: Why I chose SQLite over in-memory state
Deep-dive: Designing deterministic fingerprints for security findings
Deep-dive: The Go interfaces that made the rule engine testable

If you're learning Go from a cloud background like I am — or if you're a backend engineer who wants to understand K8s security internals — follow along.

*Questions? I'm @beltagyy on GitHub and beltagyy on DEV.

Processing 1 Million Events Per Day Without Breaking the Bank (Or Your Database)

Le Beltagy — Sun, 19 Jul 2026 09:27:51 +0000

Processing 1 Million Events Per Day Without Breaking the Bank (Or Your Database)

The real-world math behind event batching, and why it changed everything for me

The Problem

VehicleMetrics receives sensor data from vehicles in real-time.

Each vehicle sends: 10 GPS updates/second + 100 sensor readings/second = 110 messages/second per vehicle

With 50 vehicles: 5,500 messages/second = 473 million messages per day

I needed to:

Ingest fast (< 50ms response time)
Store reliably (no data loss)
Not go bankrupt (costs matter at scale)
Not break the database (PostgreSQL can only write so fast)

This is where most people fail. They think: "Just insert each event into the database."

Wrong.

The Naive Approach (That Destroys Databases)

# ❌ NAIVE: Direct insert per event
@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest(vehicle_id: str, data: SensorData):
    await db.execute(
        "INSERT INTO sensor_data VALUES (...)",
        [vehicle_id, data.timestamp, data.speed, ...]
    )
    return {"status": "created"}

The math:

5,500 events/second × 60 = 330,000 inserts/minute
Each insert: ~1ms database overhead
PostgreSQL write throughput: ~10,000 inserts/second max
Result: Queue builds. Latency explodes. System crashes.

What Actually Works: Batch Processing

Instead of inserting individually, batch events and write them together.

# ✅ REAL: Batch events before writing
from asyncio import gather
import asyncio

class EventBatcher:
    def __init__(self, batch_size=1000, flush_interval=5):
        self.batch_size = batch_size
        self.flush_interval = flush_interval
        self.queue = []
        self.lock = asyncio.Lock()

    async def add(self, event: dict):
        async with self.lock:
            self.queue.append(event)
            if len(self.queue) >= self.batch_size:
                await self._flush()

    async def _flush(self):
        if not self.queue:
            return

        # Batch write to database
        events = self.queue[:]
        self.queue = []

        # Single multi-row insert
        await db.execute(
            """INSERT INTO sensor_data (vehicle_id, timestamp, speed, sensors)
               VALUES """ + ", ".join(["(%s, %s, %s, %s)"] * len(events)),
            [val for event in events for val in [
                event['vehicle_id'],
                event['timestamp'],
                event['speed'],
                event['sensors']
            ]]
        )

        logger.info(f"Flushed {len(events)} events")

batcher = EventBatcher(batch_size=1000, flush_interval=5)

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest(vehicle_id: str, data: SensorData):
    event = {
        'vehicle_id': vehicle_id,
        'timestamp': data.timestamp,
        'speed': data.speed,
        'sensors': data.sensors
    }

    # Add to batch (doesn't block)
    await batcher.add(event)

    # Return immediately (< 5ms)
    return {"status": "queued"}

# Flush periodically
@app.on_event("startup")
async def flush_periodically():
    while True:
        await asyncio.sleep(5)
        await batcher._flush()

The math changes:

Instead of 5,500 individual inserts/second
We do 6 batch inserts/second (5,500 ÷ 1,000 batch size)
Database load: 5,500 events written, but only 6 inserts executed
Result: Database stays relaxed. Latency stays low. We scale.

The Real Numbers

Before Batching:

Throughput: 50 rps (bottlenecked by database)
Latency: 2,500ms (queued inserts waiting)
Database CPU: 95% (constant write pressure)
Cost: $150/month RDS (need bigger instance)

After Batching:

Throughput: 1,000+ rps (limited by API, not database)
Latency: 18ms (batch waits 5 seconds max)
Database CPU: 15% (batches are efficient)
Cost: $100/month RDS (smaller instance works fine)

That's 20x throughput improvement + 33% cost reduction.

The Batching Trade-offs

Trade-off 1: Latency vs Throughput

# Option C: Real-time + Batch split
# - Real-time data: Redis stream (for dashboard)
# - Persistent data: Batch to database (for analytics)

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest(vehicle_id: str, data: SensorData):
    # Real-time: Update Redis immediately
    await redis.hset(
        f"vehicle:{vehicle_id}:latest",
        mapping={
            'speed': data.speed,
            'timestamp': data.timestamp,
        }
    )

    # Persistent: Add to batch (writes later)
    await batcher.add({
        'vehicle_id': vehicle_id,
        'timestamp': data.timestamp,
        'speed': data.speed,
        'sensors': data.sensors
    })

    return {"status": "accepted"}

Result: Dashboard gets data instantly (Redis), analytics get data within 1-5 seconds (database batch).

Advanced: Batch Size Selection

Formula:

batch_size = (target_latency × events_per_second) / batches_per_second

Example:
- Target latency: 5 seconds (acceptable delay)
- Events per second: 5,500
- Desired batches per second: 10 (don't stress DB)

batch_size = (5 × 5,500) / 10 = 2,750

The Real Cost Analysis

Without Batching:

RDS Instance: db.t3.large ($300/month)
Kinesis: 10 shards ($300/month)
Lambda: Heavy ($50/month)
EC2 API servers: 10 instances ($500/month)

Total: ~$1,150/month

With Batching:

RDS Instance: db.t3.small ($100/month)
Kinesis: 1 shard ($30/month)
Lambda: Minimal ($5/month)
EC2 API servers: 2 instances ($100/month)

Total: ~$235/month

Batching saves 80% on infrastructure costs.

TL;DR

Don't insert events one-at-a-time (database will hate you)
Batch 500-2,000 events (sweet spot)
Flush every 1-5 seconds (or when batch is full)
Split real-time (Redis) from persistent (batch) (best of both)
Result: 20x throughput, 80% cost savings

The real lesson: One tiny optimization (batching) can change your entire infrastructure.

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Real-Time Data | Performance | Scale

Kubernetes for Solo Developers: Stop Overthinking It (And Start Using It)

Le Beltagy — Sat, 18 Jul 2026 13:10:33 +0000

Kubernetes for Solo Developers: Stop Overthinking It (And Start Using It)

Why a solo developer should absolutely be running their production system on Kubernetes, and why it's not as scary as you think

The Myth

Most developers think Kubernetes is:

❌ Too complex
❌ For big companies only
❌ Overkill for solo projects
❌ Requires DevOps expertise

I thought that too.

Then I actually deployed VehicleMetrics on EKS and realized:

Kubernetes solved more problems for me than it created.

Why I Actually Use Kubernetes (Solo Dev)

Problem 1: Manual Deployments Are Terrifying

Before Kubernetes:

# SSH into server
ssh ubuntu@prod-server

# Stop old app
sudo systemctl stop myapp

# Deploy new code
cd /app && git pull

# Start new app
sudo systemctl start myapp

# Pray nothing broke

With Kubernetes:

kubectl apply -f deployment.yaml
# Automatic rolling update, zero downtime

That's worth it alone.

Problem 2: When Your App Crashes

Kubernetes automatically restarts crashed pods. No 3 AM alerts for me.

Problem 3: Scaling During Peak Hours

Kubernetes scales automatically. No manual provisioning needed.

The Setup (It's Actually Simple)

Step 1: Create EKS Cluster

resource "aws_eks_cluster" "main" {
  name     = "vehicle-metrics"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.28"
}

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "main"
  node_role_arn   = aws_iam_role.node.arn

  scaling_config {
    desired_size = 3
    max_size     = 10
    min_size     = 1
  }

  instance_types = ["t3.medium"]
}

Step 2: Configure kubectl

aws eks update-kubeconfig --name vehicle-metrics --region us-east-1
kubectl get nodes

Step 3: Deploy

kubectl apply -f deployment.yaml

Day-to-Day Operations

Deploy New Version

docker build -t vehicle-metrics/api:v1.2.3 .
docker push vehicle-metrics/api:v1.2.3

kubectl set image deployment/api-ingestion \
  api=vehicle-metrics/api:v1.2.3

# If something breaks, rollback
kubectl rollout undo deployment/api-ingestion

Check Logs

kubectl logs -f deployment/api-ingestion

SSH Into Pod

kubectl exec -it pod/api-ingestion-xyz -- /bin/bash

Cost Comparison

Manual VMs: ~$300/month
Kubernetes (EKS): ~$263/month

Kubernetes is cheaper AND more reliable.

What Kubernetes Solves

✅ Automatic pod restarts
✅ Zero-downtime deployments
✅ Automatic scaling
✅ Health checks
✅ Secrets management
✅ Load balancing

TL;DR

Solo developers should use Kubernetes
Setup takes 1 hour (Terraform + kubectl)
Day-to-day: one command to deploy
Automatic scaling, restarts, rollbacks
Stop overthinking. Start using it.

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Kubernetes | DevOps | Solo Development

I Almost Leaked My Production Database Password to GitHub (And How I Fixed CI/CD Security)

Le Beltagy — Fri, 17 Jul 2026 11:25:14 +0000

I Almost Leaked My Production Database Password to GitHub (And How I Fixed CI/CD Security)

Why GitHub Actions secrets aren't enough, and what solo developers actually need to know

The Moment

I'm reviewing a GitHub Actions log.

Something catches my eye in the build output:

Step 5: Running tests
DATABASE_URL=postgresql://root:MySecretPassword123!@prod-db.aws.rds.amazonaws.com:5432/vehicle_metrics
Running pytest...

Wait. My production database password is logged in plain text in a GitHub Actions run.

Not in a secret file. Not hidden. In the public build log.

Anyone with access to the repository could scroll through the logs and find it.

I had 30 seconds of pure panic.

How It Happened

Here's what I did wrong:

# ❌ VULNERABLE
name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run tests
        run: pytest backend/tests/
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

This looks safe, but if my test code does this:

# ❌ BAD TEST CODE
def test_database_connection():
    print(f"Connecting to: {os.getenv('DATABASE_URL')}")  # LOGS THE SECRET!

The secret gets printed. GitHub logs it. Everyone sees it.

The Fix

Step 1: Clean Error Messages

# ✅ GOOD
def sanitize_for_logging(connection_string):
    return connection_string.split('@')[1] if '@' in connection_string else "***"

try:
    db_url = os.getenv('DATABASE_URL')
    logger.info(f"Connecting to: {sanitize_for_logging(db_url)}")
    connection = psycopg2.connect(db_url)
except Exception as e:
    logger.error(f"Connection failed: {type(e).__name__}")
    raise

Step 2: Use Temporary Credentials

# ✅ BEST: Use AWS IAM
name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      - uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions
          aws-region: us-east-1

      - name: Run tests
        run: pytest backend/tests/
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

Step 3: Scan for Secrets

# ✅ ADD: Secret scanning
jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: TruffleHog secret scan
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

The Complete Secure Pipeline

name: Secure CI/CD
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD

  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'
      - uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions
          aws-region: us-east-1

      - run: pip install -r requirements.txt pytest pytest-cov
      - run: pytest backend/tests/ --cov=backend --cov-report=xml
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

      - uses: codecov/codecov-action@v3
        with:
          files: ./coverage.xml

  deploy:
    runs-on: ubuntu-latest
    needs: [secret-scan, dependency-scan, test]
    if: github.ref == 'refs/heads/main'
    permissions:
      id-token: write

    steps:
      - uses: actions/checkout@v3
      - uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: us-east-1
      - run: |
          aws eks update-kubeconfig --name vehicle-metrics
          kubectl apply -f infrastructure/kubernetes/

Security Checklist

☐ Secrets in GitHub Secrets (not in code)
☐ Secrets masked in logs
☐ No debug logging with sensitive data
☐ Use IAM roles instead of long-lived credentials
☐ Rotate credentials regularly
☐ Scan for secrets on every push (TruffleHog)
☐ Scan dependencies (Trivy)
☐ SAST scanning (SonarQube)
☐ Container scanning (Grype, Snyk)
☐ Limited permissions per job (RBAC)
☐ Audit logs reviewed regularly
☐ Separate test/staging/prod credentials
☐ Encrypted secrets at rest
☐ MFA for manual deployment
☐ Secrets expire automatically

Real World Gotchas

Gotcha 1: Secrets in Environment Variables

# ❌ BAD
export DATABASE_URL="postgresql://root:password@localhost/db"

# ✅ GOOD
export DATABASE_URL=$(aws secretsmanager get-secret-value --secret-id db-password)

Gotcha 2: Secrets in Docker Build

# ❌ BAD: Secret baked into image
ARG DB_PASSWORD=prod_secret_here
RUN echo $DB_PASSWORD > /app/config

# ✅ GOOD: Only runtime secrets
ARG VERSION=1.0
ENV VERSION=$VERSION

Gotcha 3: Secrets in Error Messages

# ❌ BAD
try:
    connect(os.getenv('DATABASE_URL'))
except Exception:
    print(traceback.format_exc())  # Includes the URL!

# ✅ GOOD
except Exception as e:
    logger.error(f"Connection failed: {type(e).__name__}")

TL;DR

Almost leaked production database password via GitHub Actions logs
Found it in environment variable printed during tests
Fixed: sanitized logs, temporary credentials, secret scanning
Implemented: TruffleHog + Trivy + SonarQube + Snyk in CI/CD
One leaked credential kills your business

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: CI/CD Security | DevOps | Production Hardening

The Architecture Decision That Almost Killed My Project (And How I Fixed It)

Le Beltagy — Thu, 16 Jul 2026 15:14:26 +0000

The Architecture Decision That Almost Killed My Project (And How I Fixed It)

A real story about bottlenecks, debugging, and why the obvious solution isn't always right

The Problem

It's 3 AM. I'm staring at Grafana dashboards. VehicleMetrics is supposed to handle 1,000 requests per second.

It's handling 50.

The thing is breaking. Not crashing. Just... slow. Every request takes 2-3 seconds. Users are complaining. The API is timing out.

And I have no idea why.

The Setup (What I Thought Would Work)

Let me back up. Here's what I built:

Vehicle sends data → FastAPI → PostgreSQL (direct insert)

Simple, right? Beautiful architecture:

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest_data(vehicle_id: str, data: SensorData):
    # Direct write to database
    await db.execute(
        "INSERT INTO sensor_data VALUES (...)",
        [vehicle_id, data.timestamp, data.speed, ...]
    )
    return {"status": "created"}

Problem: I didn't see it coming.

The Debugging Journey (The Hard Part)

Attempt 1: "It's Probably the Database"

$ curl -X POST http://localhost:8000/api/vehicles/V1/sensor-data
  -H "Content-Type: application/json"
  -d '{"timestamp": 1000, "speed": 55, ...}'

real    0m2.834s
user    0m0.012s

2.8 seconds for one request. That's death.

First thought: Database is slow.

I check PostgreSQL:

SELECT count(*) FROM sensor_data;
-- 50 million rows

EXPLAIN ANALYZE INSERT INTO sensor_data VALUES (...);
-- Cost: 0.01, Time: 0.02ms

SELECT indexrelname FROM pg_indexes WHERE tablename = 'sensor_data';
-- 15 indexes

Wait. The INSERT itself is fast (0.02ms). But my API call took 2.8 seconds.

Realization: It's not the database. It's me.

Attempt 2: "Let Me Profile This"

I add timing to my code:

import time

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest_data(vehicle_id: str, data: SensorData):
    start = time.time()

    t1 = time.time()
    # Validation
    validated_data = SensorData(**data)
    print(f"Validation: {time.time() - t1}ms")

    t2 = time.time()
    # Database insert
    await db.execute(
        "INSERT INTO sensor_data VALUES (...)",
        [validated_data.vehicle_id, ...]
    )
    print(f"Insert: {time.time() - t2}ms")

    total = time.time() - start
    print(f"Total: {total}ms")

    return {"status": "created"}

Output:

Validation: 0.2ms
Insert: 2750ms
Total: 2760ms

The insert is taking 2.75 seconds.

That's... weird. The EXPLAIN said 0.02ms.

Attempt 3: "Let Me Check the Connection Pool"

Aha! I found it.

# ❌ OLD CODE
engine = create_async_engine(
    "postgresql+asyncpg://...",
    # No connection pool settings!
)

Default connection pool size: 5 connections.

What does that mean?

First 5 requests: use 5 connections, fast
6th request: wait for a connection to free up...
Wait... wait... wait...

With 50 concurrent requests, every request is waiting in a queue for a connection.

That's the problem.

The Fix (What I Learned)

Solution 1: Bigger Connection Pool

# ✅ BETTER
engine = create_async_engine(
    "postgresql+asyncpg://...",
    pool_size=50,  # 50 connections
    max_overflow=20,  # 20 extra connections if needed
)

Test again:

Request time: 2750ms → 150ms  (18x faster!)

Better. But still not 50ms.

Solution 2: The Real Problem (Async I/O)

I realized: even with a big pool, I'm doing synchronous writes.

Each insert waits for the database to acknowledge. With 1,000 concurrent requests:

Request 1: INSERT → Wait for DB → Response (takes 10ms)
Request 2: INSERT → Wait for DB → Response (takes 10ms)
Request 3: INSERT → Wait for DB → Response (takes 10ms)
...
Request 1000: INSERT → Wait for DB → Response (takes 10ms + 999 queued requests)

The real solution: Fire-and-forget with Kafka.

Solution 3: The Architecture Change

from aiokafka import AIOKafkaProducer

@app.on_event("startup")
async def startup():
    app.kafka = AIOKafkaProducer(
        bootstrap_servers=['kafka:9092'],
        compression_type='gzip',
    )
    await app.kafka.start()

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest_data(vehicle_id: str, data: SensorData):
    # Fire-and-forget to Kafka
    await app.kafka.send(
        "sensor-data",
        value=json.dumps(data.dict()).encode(),
        key=vehicle_id.encode(),
    )

    # Update Redis for real-time dashboard
    await redis.hset(
        f"vehicle:{vehicle_id}:latest",
        mapping=data.dict()
    )

    # Respond immediately (< 50ms)
    return {"status": "accepted"}

# Separate process: consume from Kafka, write to database
async def kafka_consumer():
    consumer = AIOKafkaConsumer(
        'sensor-data',
        bootstrap_servers=['kafka:9092'],
        group_id='ingestion-group',
    )
    await consumer.start()

    async for message in consumer:
        data = json.loads(message.value)
        await db.execute(
            "INSERT INTO sensor_data VALUES (...)",
            [data.vehicle_id, ...]
        )

New flow:

Vehicle sends data → FastAPI (< 5ms)
                  → Kafka (send, < 10ms)
                  → Response (< 20ms total)

Meanwhile:
Lambda/Consumer → Read from Kafka
               → Write to PostgreSQL (async)

Test again:

Request time: 150ms → 18ms  (8x faster!)
Throughput: 50 rps → 1000+ rps  (20x improvement!)

What I Learned (The Real Lesson)

Lesson 1: Premature Optimization is Evil (But Late Optimization is Worse)

I thought: "PostgreSQL is fast, let's just write directly."

Wrong. I didn't think about concurrency. With 1,000 concurrent requests, the database becomes the bottleneck, not because it's slow, but because I'm serializing all writes.

Lesson 2: Profiling Saves Your Life

Without timing, I would have blamed the database forever.

# Add this to EVERY endpoint
import time
@app.middleware("http")
async def add_timing(request: Request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    logger.info(f"{request.method} {request.url.path}: {duration*1000:.1f}ms")
    return response

Lesson 3: Connection Pools Aren't Magic

Even with async/await, if you have 5 connections and 1,000 requests, you're creating a queue.

# The math:
pool_size = 50
avg_request_time = 10ms
requests_per_second = 1000

can_handle = (pool_size / avg_request_time) * 1000
# = (50 / 0.01) * 1000 = 5,000 rps

# With pool_size = 5:
can_handle = (5 / 0.01) * 1000 = 500 rps

Lesson 4: Async All The Way Down

This is where I messed up. Even with async FastAPI, my writes were blocking:

# ❌ BLOCKING (even in async context)
result = await db.execute("INSERT ...")  # Waits for DB
return result  # Only then returns

# ✅ NON-BLOCKING
background_task.add_task(write_to_db, data)
return {"status": "queued"}  # Returns immediately

Lesson 5: Decouple Your Read/Write Paths

The biggest realization:

Writes: Fire-and-forget, async, high throughput
Reads: Cached, can be slightly stale, always fast

Don't make them both go to the same place at the same time.

The Before & After

Before (Direct Database):

Architecture:
  Vehicle → FastAPI → PostgreSQL

Performance:
  - Throughput: 50 rps
  - Latency: 2,500ms (average)
  - P99 latency: 8,000ms
  - Database CPU: 95%

Problem:
  - Connection pool exhaustion
  - All requests serialized on writes
  - Backup building up

After (Kafka + Async):

Architecture:
  Vehicle → FastAPI → Kafka → Lambda → PostgreSQL
                   → Redis (for dashboard)

Performance:
  - Throughput: 1,000+ rps
  - Latency: 20ms (average)
  - P99 latency: 100ms
  - Database CPU: 30%

Benefits:
  - Decoupled writes
  - Can replay data
  - Dashboard updates instantly
  - Database never overwhelmed

20x improvement. Just by fixing the architecture.

The Real Lesson (Not Just Technical)

When you're solo building, you have no one to bounce ideas off.

So here's what I do now:

Ask "Why?" three times
- Slow latency? → Database slow? → No, wait times → Connection pool? → Yes!
Profile first, optimize second
- Don't guess. Measure. Then fix what's actually slow.
Think about the future
- Building for 1 user? Cool. Building for 1,000 concurrent users? Different architecture.
Async everywhere or nowhere
- Mixing sync/async is a trap. Commit to async.
Decouple everything you can
- Writes and reads
- Real-time and batch
- Hot data and cold data

The Metrics That Matter

I obsess over these now:

✓ P50 latency (average user)
✓ P99 latency (worst 1% of users)
✓ Throughput (requests/second)
✓ Connection pool utilization
✓ Database CPU
✓ Kafka lag (messages waiting to process)
✓ Redis hit rate

If any of these look bad, I can drill down instantly.

What This Taught Me About Solo Development

Building alone means:

❌ No code reviews to catch this
❌ No senior engineer to ask "Why are you doing it this way?"
❌ No infrastructure team to handle scaling

✅ But also: No meetings, no politics, no "that's how we've always done it"
✅ And: I get to make these discoveries myself (painful, but educational)

The mistakes hurt more when you're solo.

But the lessons stick harder.

The Gotcha Nobody Tells You

Here's the thing everyone skips over in tutorials:

Your code can be perfect and your architecture can still suck.

FastAPI is fast ✅
PostgreSQL is fast ✅
Your connection pool is wrong ❌ = System is slow

I could have written the most beautiful async code in the world and still gotten 20ms responses per 1,000 requests.

Architecture beats code quality every time.

Design right first. Optimize code second.

TL;DR

Built direct-to-database architecture, got bottlenecked at 50 rps
Found the problem: connection pool too small (5 connections)
Fixed pool, improved to 150ms response time
Real solution: Kafka + async + fire-and-forget
Achieved 1,000+ rps with 20ms latency
Lessons: Profile first, decouple read/write, async everywhere, think about concurrency

Big lesson: One architecture decision (direct DB writes vs Kafka) meant 20x difference in throughput.

That's why design matters more than code quality.

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Architecture | Performance | Real-World Engineering

Building with FastAPI: A Production Journey (Real Code, Real Lessons)

Le Beltagy — Wed, 15 Jul 2026 09:55:49 +0000

Building with FastAPI: A Production Journey (Real Code, Real Lessons)

How I built high-performance microservices for handling 1,000+ requests per second

The FastAPI Choice

When I started VehicleMetrics, I could have chosen Django, Flask, or even Go.

But I chose FastAPI.

Not because it's trendy. Not because of hype. But because it solves real problems that matter at scale.

Let me show you why, with actual code from production.

Why FastAPI Over Django/Flask?

Django

Pros: Batteries included. ORM. Admin panel. Massive ecosystem.
Cons: Bloated for microservices. Slow request handling. Heavyweight framework for what we need.

Flask

Pros: Lightweight. Simple. Easy to start.
Cons: No built-in validation. No async support by default. Manual everything.

FastAPI

Pros:

Built-in async/await (critical for I/O-bound work)
Automatic OpenAPI documentation
Runtime validation with Pydantic
Dependency injection (clean, testable code)
10-100x faster than Flask/Django
Type hints = self-documenting

Decision: FastAPI. It's the sweet spot between simplicity and power.

The Ingestion Service: 1,000 RPS Challenge

Here's the real challenge: receive and process 1,000 sensor updates per second, with < 50ms response time.

The Naive Approach (That Fails)

from fastapi import FastAPI
from sqlalchemy import create_engine

app = FastAPI()
engine = create_engine("postgresql://...")
connection = engine.connect()

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
def ingest_data(vehicle_id: str, data: dict):
    # ❌ BLOCKING: Waits for database
    result = connection.execute(
        "INSERT INTO sensor_data ..."
    )
    return {"status": "saved"}

Problems:

Database insert is synchronous (blocks the thread)
1,000 requests = need 1,000 threads waiting on I/O
Thread overhead kills performance
Response time: 200ms+ (database latency)

The Production Approach (What Works)

from fastapi import FastAPI, Depends, BackgroundTasks
from pydantic import BaseModel, Field, validator
from typing import Optional
import aiokafka
import redis.asyncio as redis
import logging

logger = logging.getLogger(__name__)

# 1. Define your data model with validation
class SensorData(BaseModel):
    vehicle_id: str = Field(..., min_length=1, max_length=50)
    timestamp: int = Field(..., gt=0)
    speed: float = Field(..., ge=0, le=200)
    lat: float = Field(..., ge=-90, le=90)
    lon: float = Field(..., ge=-180, le=180)
    sensors: dict = Field(default={})

    @validator('timestamp')
    def validate_timestamp(cls, v):
        import time
        current = int(time.time() * 1000)
        # Reject data older than 5 minutes
        if abs(current - v) > 300000:
            raise ValueError('Timestamp too old or in future')
        return v

# 2. Initialize async clients (connection pooling)
class AppState:
    kafka: Optional[aiokafka.AIOKafkaProducer] = None
    redis: Optional[redis.Redis] = None

app = FastAPI()
state = AppState()

@app.on_event("startup")
async def startup():
    # Kafka connection pool
    state.kafka = aiokafka.AIOKafkaProducer(
        bootstrap_servers='kafka:9092',
        compression_type='gzip',
        batch_size=16384,
    )
    await state.kafka.start()

    # Redis connection pool
    state.redis = await redis.from_url(
        "redis://localhost:6379",
        encoding="utf8",
        decode_responses=True,
        max_connections=50,
    )
    logger.info("Startup: Kafka and Redis initialized")

@app.on_event("shutdown")
async def shutdown():
    if state.kafka:
        await state.kafka.stop()
    if state.redis:
        await state.redis.close()
    logger.info("Shutdown: Connections closed")

# 3. The actual endpoint
@app.post("/api/vehicles/{vehicle_id}/sensor-data", status_code=201)
async def ingest_sensor_data(
    vehicle_id: str,
    data: SensorData,
    background_tasks: BackgroundTasks,
):
    """
    Ingest vehicle sensor data.

    - Validates payload (Pydantic)
    - De-duplicates (Redis check)
    - Sends to Kafka (fire-and-forget)
    - Returns immediately (< 50ms)
    """

    try:
        # 1. De-duplication check (Redis, async)
        cache_key = f"vehicle:{vehicle_id}:latest:{data.timestamp}"
        is_duplicate = await state.redis.exists(cache_key)

        if is_duplicate:
            logger.debug(f"Duplicate message for {vehicle_id}")
            return {
                "status": "ignored",
                "reason": "duplicate",
                "vehicle_id": vehicle_id,
            }

        # 2. Mark as seen (Redis, 10-second TTL)
        await state.redis.setex(
            cache_key,
            ex=10,
            value="1"
        )

        # 3. Update latest vehicle state (for dashboard)
        latest_key = f"vehicle:{vehicle_id}:latest"
        await state.redis.hset(
            latest_key,
            mapping={
                "speed": data.speed,
                "lat": data.lat,
                "lon": data.lon,
                "timestamp": data.timestamp,
            }
        )
        await state.redis.expire(latest_key, 3600)  # 1 hour TTL

        # 4. Send to Kafka (async, non-blocking)
        message = {
            "vehicle_id": vehicle_id,
            "timestamp": data.timestamp,
            "speed": data.speed,
            "lat": data.lat,
            "lon": data.lon,
            "sensors": data.sensors,
        }

        # Don't await - fire and forget
        background_tasks.add_task(
            state.kafka.send_and_wait,
            "sensor-data",
            value=json.dumps(message).encode(),
            key=vehicle_id.encode(),
            partition=hash(vehicle_id) % 6,  # Route to specific partition
        )

        logger.info(
            f"Ingested {vehicle_id} - speed: {data.speed}",
            extra={"vehicle_id": vehicle_id, "timestamp": data.timestamp}
        )

        return {
            "status": "accepted",
            "vehicle_id": vehicle_id,
            "timestamp": data.timestamp,
        }

    except Exception as e:
        logger.error(f"Ingest error: {str(e)}", exc_info=True)
        raise HTTPException(status_code=500, detail="Ingest failed")

Why this works:

Pydantic validation: Rejects bad data instantly (type-safe)
Async I/O: No threads blocking on I/O
Connection pooling: Reuse connections, not create new ones
Fire-and-forget: Kafka send doesn't block response
Redis caching: Dedup check in < 2ms (memory)
Response time: < 50ms (tested)

Performance:

1 instance handles 100+ rps
10 instances = 1,000+ rps
Each handles concurrent requests with same thread

Dependency Injection: The Secret Weapon

FastAPI's dependency system is game-changing. Here's why:

Without DI (Messy)

@app.get("/api/vehicle/{id}")
async def get_vehicle(id: str):
    # Auth check
    user = await check_auth(request.headers["Authorization"])

    # Database query
    db = get_database()
    vehicle = await db.fetch_one(...)

    # Authorization check
    if vehicle.owner_id != user.id:
        raise PermissionError()

    return vehicle

Problems:

Mixed concerns
Hard to test
Duplicate auth code everywhere
Tight coupling

With DI (Clean)

from fastapi import Depends, HTTPException, Request
from functools import lru_cache

# 1. Define dependencies
async def get_current_user(
    request: Request,
    redis: redis.Redis = Depends(get_redis_client),
) -> User:
    """Extract and validate JWT from Authorization header."""
    token = request.headers.get("Authorization", "").replace("Bearer ", "")

    if not token:
        raise HTTPException(status_code=401, detail="Missing token")

    # Check Redis cache first (avoid JWT parsing)
    cached = await redis.get(f"user_token:{token}")
    if cached:
        return User(**json.loads(cached))

    # Validate JWT
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
        user = User(**payload)

        # Cache for 24 hours
        await redis.setex(
            f"user_token:{token}",
            ex=86400,
            value=json.dumps(user.dict())
        )
        return user
    except jwt.InvalidTokenError:
        raise HTTPException(status_code=401, detail="Invalid token")

async def get_vehicle_or_404(
    vehicle_id: str,
    db = Depends(get_database),
) -> Vehicle:
    """Fetch vehicle or raise 404."""
    vehicle = await db.fetch_one(
        "SELECT * FROM vehicles WHERE id = $1",
        vehicle_id
    )
    if not vehicle:
        raise HTTPException(status_code=404, detail="Vehicle not found")
    return Vehicle(**vehicle)

async def check_vehicle_ownership(
    current_user: User = Depends(get_current_user),
    vehicle: Vehicle = Depends(get_vehicle_or_404),
) -> Vehicle:
    """Verify user owns this vehicle."""
    if vehicle.owner_id != current_user.id:
        raise HTTPException(status_code=403, detail="Not authorized")
    return vehicle

# 2. Use dependencies (clean!)
@app.get("/api/vehicles/{vehicle_id}")
async def get_vehicle_details(
    vehicle: Vehicle = Depends(check_vehicle_ownership),
) -> dict:
    """Get vehicle data. Authentication + authorization automatic."""
    return {
        "id": vehicle.id,
        "name": vehicle.name,
        "status": vehicle.status,
    }

@app.put("/api/vehicles/{vehicle_id}")
async def update_vehicle(
    vehicle: Vehicle = Depends(check_vehicle_ownership),
    update: VehicleUpdate,
    db = Depends(get_database),
) -> Vehicle:
    """Update vehicle. Auth included."""
    await db.execute(
        "UPDATE vehicles SET name = $1 WHERE id = $2",
        update.name,
        vehicle.id,
    )
    return await db.fetch_one("SELECT * FROM vehicles WHERE id = $1", vehicle.id)

Benefits:

Separation of concerns
Reusable dependencies
Easy testing (mock dependencies)
Clear what each endpoint needs
DRY (Don't Repeat Yourself)

Error Handling: Don't Let Exceptions Escape

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from fastapi.exceptions import RequestValidationError
import logging

logger = logging.getLogger(__name__)
app = FastAPI()

# 1. Validation errors (Pydantic)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
    """Return clean validation errors."""
    return JSONResponse(
        status_code=422,
        content={
            "status": "validation_error",
            "errors": [
                {
                    "field": error["loc"][-1],
                    "message": error["msg"],
                    "type": error["type"],
                }
                for error in exc.errors()
            ]
        }
    )

# 2. Application errors (Custom)
class VehicleNotFound(Exception):
    def __init__(self, vehicle_id: str):
        self.vehicle_id = vehicle_id
        self.message = f"Vehicle {vehicle_id} not found"

@app.exception_handler(VehicleNotFound)
async def vehicle_not_found_handler(request: Request, exc: VehicleNotFound):
    """Handle missing vehicle gracefully."""
    logger.warning(f"Vehicle not found: {exc.vehicle_id}")
    return JSONResponse(
        status_code=404,
        content={"status": "error", "message": exc.message}
    )

# 3. Catch-all (unexpected errors)
@app.exception_handler(Exception)
async def general_exception_handler(request: Request, exc: Exception):
    """Log unexpected errors, return safe response."""
    logger.error(
        f"Unexpected error: {str(exc)}",
        exc_info=True,
        extra={"path": request.url.path}
    )

    # Don't leak internal details in production
    return JSONResponse(
        status_code=500,
        content={"status": "error", "message": "Internal server error"}
    )

# Usage
@app.get("/api/vehicles/{vehicle_id}")
async def get_vehicle(vehicle_id: str, db = Depends(get_database)):
    vehicle = await db.fetch_one(
        "SELECT * FROM vehicles WHERE id = $1",
        vehicle_id
    )

    if not vehicle:
        raise VehicleNotFound(vehicle_id)  # Clean, handled above

    return vehicle

Request/Response Logging (Without Slowing Down)

import time
import uuid
from fastapi import Request
from starlette.middleware.base import BaseHTTPMiddleware
from contextvars import ContextVar

# Store request ID in context for logging
request_id_var: ContextVar[str] = ContextVar("request_id", default="")

class LoggingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        # 1. Generate request ID
        request_id = str(uuid.uuid4())
        request_id_var.set(request_id)

        # 2. Log request start
        logger.info(
            f"→ {request.method} {request.url.path}",
            extra={
                "request_id": request_id,
                "method": request.method,
                "path": request.url.path,
                "client": request.client.host if request.client else "unknown",
            }
        )

        # 3. Time the request
        start = time.time()
        response = await call_next(request)
        duration = time.time() - start

        # 4. Log response
        logger.info(
            f"← {response.status_code} ({duration*1000:.1f}ms)",
            extra={
                "request_id": request_id,
                "status_code": response.status_code,
                "duration_ms": duration * 1000,
            }
        )

        # 5. Add request ID to response headers (for debugging)
        response.headers["X-Request-ID"] = request_id

        return response

app.add_middleware(LoggingMiddleware)

Output:

2026-07-15 10:23:45 INFO → POST /api/vehicles/V123/sensor-data (request_id=abc-123)
2026-07-15 10:23:45 INFO ← 201 (12.3ms) (request_id=abc-123)

Database Queries (The Right Way)

from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker

# 1. Create async engine
engine = create_async_engine(
    "postgresql+asyncpg://user:password@localhost/vehicle_metrics",
    echo=False,  # Set to True for SQL logging
    pool_size=20,
    max_overflow=10,
)

# 2. Session factory
async_session = async_sessionmaker(
    engine,
    class_=AsyncSession,
    expire_on_commit=False,
)

async def get_database():
    async with async_session() as session:
        yield session

# 3. Parameterized queries (prevent SQL injection)
@app.get("/api/vehicles/{vehicle_id}/analytics")
async def get_vehicle_analytics(
    vehicle_id: str,
    days: int = 30,
    db: AsyncSession = Depends(get_database),
):
    """Get vehicle analytics for last N days."""

    # Use parameters, not string concatenation
    query = text("""
        SELECT 
            DATE_TRUNC('hour', timestamp) as hour,
            AVG(speed) as avg_speed,
            MAX(speed) as max_speed,
            COUNT(*) as samples
        FROM sensor_data
        WHERE vehicle_id = :vehicle_id
          AND timestamp > NOW() - INTERVAL ':days days'
        GROUP BY hour
        ORDER BY hour DESC
    """)

    result = await db.execute(
        query,
        {
            "vehicle_id": vehicle_id,
            "days": days,
        }
    )

    return [dict(row) for row in result.fetchall()]

# 4. Transactions (for consistency)
@app.post("/api/vehicles/{vehicle_id}/alert")
async def create_alert(
    vehicle_id: str,
    alert: AlertCreate,
    db: AsyncSession = Depends(get_database),
):
    """Create alert + update vehicle status (atomic)."""

    try:
        # Both queries run in a transaction
        await db.execute(
            text("INSERT INTO alerts (vehicle_id, message) VALUES (:vehicle_id, :message)"),
            {"vehicle_id": vehicle_id, "message": alert.message}
        )

        await db.execute(
            text("UPDATE vehicles SET status = 'alert' WHERE id = :vehicle_id"),
            {"vehicle_id": vehicle_id}
        )

        await db.commit()  # Commit both at once
        return {"status": "created"}

    except Exception as e:
        await db.rollback()  # Rollback both if error
        logger.error(f"Alert creation failed: {str(e)}")
        raise

Testing FastAPI (So Much Easier)

from fastapi.testclient import TestClient
import pytest

@pytest.fixture
def client():
    return TestClient(app)

def test_ingest_valid_data(client):
    """Test successful data ingestion."""
    response = client.post(
        "/api/vehicles/V123/sensor-data",
        json={
            "timestamp": 1000000,
            "speed": 55.5,
            "lat": 37.7749,
            "lon": -122.4194,
            "sensors": {}
        }
    )

    assert response.status_code == 201
    assert response.json()["status"] == "accepted"

def test_ingest_invalid_speed(client):
    """Test validation rejects invalid speed."""
    response = client.post(
        "/api/vehicles/V123/sensor-data",
        json={
            "timestamp": 1000000,
            "speed": 999,  # ❌ Exceeds max of 200
            "lat": 37.7749,
            "lon": -122.4194,
            "sensors": {}
        }
    )

    assert response.status_code == 422
    assert "speed" in str(response.json())

def test_ingest_missing_vehicle_id(client):
    """Test validation requires vehicle_id."""
    response = client.post(
        "/api//sensor-data",  # Missing vehicle_id
        json={
            "timestamp": 1000000,
            "speed": 55.5,
            "lat": 37.7749,
            "lon": -122.4194,
            "sensors": {}
        }
    )

    assert response.status_code == 404

Configuration Management

from pydantic_settings import BaseSettings
from functools import lru_cache

class Settings(BaseSettings):
    # Database
    database_url: str = "postgresql://localhost/vehicle_metrics"
    database_pool_size: int = 20

    # Redis
    redis_url: str = "redis://localhost:6379"
    redis_max_connections: int = 50

    # Kafka
    kafka_bootstrap_servers: str = "localhost:9092"
    kafka_topic: str = "sensor-data"

    # API
    api_title: str = "VehicleMetrics API"
    api_version: str = "1.0.0"
    debug: bool = False

    # Security
    secret_key: str = "change-me-in-production"
    jwt_algorithm: str = "HS256"
    jwt_expiration_hours: int = 24

    class Config:
        env_file = ".env"  # Read from .env file
        case_sensitive = False

@lru_cache()
def get_settings():
    return Settings()

# Usage
@app.get("/api/health")
async def health_check(settings: Settings = Depends(get_settings)):
    return {
        "status": "healthy",
        "version": settings.api_version,
    }

.env file:

DATABASE_URL=postgresql://user:pass@localhost/vehicle_metrics
REDIS_URL=redis://localhost:6379
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
SECRET_KEY=your-super-secret-key
DEBUG=False

Monitoring & Metrics

from prometheus_client import Counter, Histogram, generate_latest
import time

# Metrics
request_count = Counter(
    'fastapi_requests_total',
    'Total requests',
    ['method', 'endpoint', 'status']
)

request_duration = Histogram(
    'fastapi_request_duration_seconds',
    'Request duration',
    ['method', 'endpoint'],
    buckets=(0.01, 0.05, 0.1, 0.5, 1.0)
)

@app.middleware("http")
async def add_metrics(request: Request, call_next):
    start = time.time()
    response = await call_next(request)

    duration = time.time() - start
    request_count.labels(
        method=request.method,
        endpoint=request.url.path,
        status=response.status_code,
    ).inc()

    request_duration.labels(
        method=request.method,
        endpoint=request.url.path,
    ).observe(duration)

    return response

@app.get("/metrics")
async def metrics():
    """Return Prometheus metrics."""
    return generate_latest()

Performance Optimization Checklist

# ✅ Async all the way
@app.post("/api/data")
async def process(data: MyModel):  # Not def, use async def
    result = await async_operation()
    return result

# ✅ Connection pooling
# Don't create new DB connections per request
# Use connection pools (20-50 connections)

# ✅ Caching hot data
# Store latest vehicle state in Redis
# TTL: 30 seconds for real-time data

# ✅ Compression
# Enable gzip compression on responses
app = FastAPI()
from fastapi.middleware.gzip import GZIPMiddleware
app.add_middleware(GZIPMiddleware, minimum_size=1000)

# ✅ Rate limiting
from slowapi import Limiter
limiter = Limiter(key_func=get_remote_address)
@app.post("/api/data")
@limiter.limit("1000/minute")
async def process(data: MyModel):
    pass

# ✅ Pagination for large datasets
@app.get("/api/analytics")
async def get_analytics(skip: int = 0, limit: int = 100):
    if limit > 1000:
        limit = 1000  # Cap maximum
    return await db.fetch(f"LIMIT {limit} OFFSET {skip}")

Deployment (Docker + Kubernetes)

# Dockerfile
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy app
COPY . .

# Run with Uvicorn (4 workers for 4 CPUs)
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

# Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-ingestion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-ingestion
  template:
    metadata:
      labels:
        app: api-ingestion
    spec:
      containers:
      - name: api
        image: vehicle-metrics/api:latest
        ports:
        - containerPort: 8000
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /api/health
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /api/health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: database-url
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-ingestion-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-ingestion
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Real-World Gotchas

1. Connection Pool Exhaustion

# ❌ Wrong: Creates new connection each time
@app.get("/api/data")
async def get_data():
    db = create_connection()  # New connection!
    result = await db.query(...)
    return result

# ✅ Right: Reuse from pool
@app.get("/api/data")
async def get_data(db = Depends(get_database)):  # From pool
    result = await db.query(...)
    return result

2. Async All the Way

# ❌ Wrong: Blocking call in async context
@app.get("/api/data")
async def get_data():
    result = requests.get("http://...")  # BLOCKS!
    return result

# ✅ Right: Use async HTTP client
import httpx

@app.get("/api/data")
async def get_data():
    async with httpx.AsyncClient() as client:
        result = await client.get("http://...")
    return result

3. Memory Leaks with Background Tasks

# ❌ Wrong: Task references prevent garbage collection
background_tasks.add_task(
    process_data,
    large_data_object,  # Stays in memory!
)

# ✅ Right: Pass only what you need
background_tasks.add_task(
    process_data,
    large_data_object.id,  # Just the ID
)

TL;DR

Choose FastAPI for real-time, high-throughput APIs
Use async/await everywhere (not one sync function!)
Validate with Pydantic (free type safety)
Dependency injection keeps code clean and testable
Connection pooling is non-negotiable
Fire-and-forget for non-blocking work
Logging + Metrics = visibility into production
Test everything (TestClient makes it easy)
Deploy on Kubernetes with auto-scaling

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: FastAPI | Python | Microservices

Cloud Architecture for Autonomous Systems: The 3-Tier Design & Why It Works

Le Beltagy — Tue, 14 Jul 2026 12:18:49 +0000

Cloud Architecture for Autonomous Systems: The 3-Tier Design & Why It Works

How I designed a production-grade system for handling real-time vehicle data at scale

The Problem I Solved

When I started building VehicleMetrics, I faced a choice that every backend engineer faces when tackling real-time systems:

Do I over-engineer and build something that scales to millions? Or do I start lean and regret it when things break?

I chose a third path: build a system that scales intelligently from day one, but doesn't waste resources on premature optimization.

This is the story of how I designed a 3-tier cloud architecture that handles autonomous vehicle data in real-time, scales horizontally, maintains data integrity, and stays operationally simple enough for one developer to manage.

Why 3 Tiers? Why Not Serverless? Why Not Monolith?

Let me be honest: I considered all approaches.

Serverless (Lambda/DynamoDB)

Pros: No servers to manage. Auto-scales. Pay per invocation.
Cons: Cold starts (critical for real-time data). Vendor lock-in. Query limitations with DynamoDB. Debugging nightmares when things go wrong.

Decision: No. Real-time vehicle data can't tolerate 500ms cold starts.

Monolith (Single large service)

Pros: Simple to deploy. Easier debugging. Single database.
Cons: Can't scale individual components. Technology lock-in. Deployment = all-or-nothing risk.

Decision: No. Different components scale differently. Data ingestion ≠ API response times.

3-Tier Microservices

Pros: Independent scaling. Technology flexibility. Clear separation of concerns. Battle-tested pattern.
Cons: More complex. Requires orchestration (Kubernetes). Multiple databases = consistency challenges.

Decision: Yes. This is the Goldilocks zone.

The Architecture: Layer by Layer

┌─────────────────────────────────────────────────────────────┐
│                    PRESENTATION TIER                        │
│        React + TypeScript + Grafana Dashboards              │
│              (Real-time Analytics UI)                        │
│                                                             │
│  • User Dashboards (React + WebSocket)                      │
│  • Grafana (System Metrics & Alerts)                        │
│  • Real-time Analytics Display                              │
└─────────────────┬───────────────────────────────────────────┘
                  │
         ┌────────┴──────────┐
         │ HTTPS + WebSocket │
         │    TLS 1.3        │
         └────────┬──────────┘
                  │
┌─────────────────▼───────────────────────────────────────────┐
│                  APPLICATION TIER                           │
│  FastAPI Microservices on EKS (Kubernetes)                  │
│                                                             │
│  ┌──────────────────┐  ┌──────────────────┐                │
│  │ Ingestion Svc    │  │ Analytics Svc    │                │
│  │ (High throughput)│  │ (Complex queries)│                │
│  │ Auto-scale: 1-10 │  │ Auto-scale: 1-5  │                │
│  └──────────────────┘  └──────────────────┘                │
│                                                             │
│  ┌──────────────────┐  ┌──────────────────┐                │
│  │ Auth Svc         │  │ Aggregation Svc  │                │
│  │ (Stateless)      │  │ (Batch processes)│                │
│  │ Auto-scale: 1-3  │  │ Auto-scale: 1-2  │                │
│  └──────────────────┘  └──────────────────┘                │
│                                                             │
│  All behind: AWS Application Load Balancer                 │
│  All in: Private subnets with NAT gateway                  │
└─────────────────┬───────────────────────────────────────────┘
                  │
      ┌───────────┼───────────┐
      │           │           │
   (Private Network - VPC)
      │           │           │
      ▼           ▼           ▼
┌─────────────────────────────────────────────────────────────┐
│                    DATA TIER                                │
│                                                             │
│  ┌────────────────────────────────────────┐                │
│  │ PostgreSQL 14 + TimescaleDB            │                │
│  │ • Time-series data (optimized)         │                │
│  │ • Retention: 90 days hot, 1yr archive  │                │
│  │ • Backup: Continuous PITR              │                │
│  │ • Read replicas for analytics queries  │                │
│  │ • Multi-AZ for high availability       │                │
│  └────────────────────────────────────────┘                │
│                                                             │
│  ┌────────────────────────────────────────┐                │
│  │ Redis 7 (Cluster Mode)                 │                │
│  │ • Real-time data streams               │                │
│  │ • Session storage                      │                │
│  │ • Cache layer for hot queries          │                │
│  │ • 6 nodes, 1GB each                    │                │
│  └────────────────────────────────────────┘                │
│                                                             │
│  ┌────────────────────────────────────────┐                │
│  │ S3 Data Lake                           │                │
│  │ • Raw sensor data (parquet format)     │                │
│  │ • Lifecycle: 30 days → Glacier         │                │
│  │ • Partition by vehicle_id/date         │                │
│  │ • Versioning enabled                   │                │
│  └────────────────────────────────────────┘                │
│                                                             │
│  ┌────────────────────────────────────────┐                │
│  │ Kinesis Data Streams                   │                │
│  │ • Real-time ingest (1000 rps)          │                │
│  │ • Lambda consumers for processing      │                │
│  │ • 24-hour retention                    │                │
│  └────────────────────────────────────────┘                │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Tier 1: Presentation (Frontend)

Why React + TypeScript?

React:

Component-driven = easier to maintain
Virtual DOM = predictable performance
Ecosystem = everything you need exists
WebSocket support = real-time data

TypeScript:

Catches bugs at compile time
Self-documenting code
Refactoring confidence
Better IDE support

Why Grafana Dashboards?

Grafana wasn't in the original plan. But I realized:

Users expect dashboards. Giving them a custom React dashboard = I maintain it forever.
Grafana solves this. Pre-built panels, alerting, user management, all free.
It integrates. Prometheus metrics flow directly into Grafana.

Decision: Custom React for business dashboards. Grafana for ops/metrics dashboards.

Connection Pattern: WebSocket

Why not REST polling?

REST Polling:
Client → POST /api/latest → Server (every 5 seconds)
= 12 requests per minute per client
= 12,000 requests/min with 1000 clients
= Unnecessary load

WebSocket (Server-Sent Events):
Server opens persistent connection
Server pushes updates when available
= Only real changes transmitted
= 1/10th the bandwidth

Decision: WebSocket for real-time data. REST for historical queries.

Tier 2: Application (Backend Microservices)

Why Microservices? Why Not One Giant Service?

Here's the problem with monoliths:

Ingestion Service receives 1,000 requests/second of sensor data. It needs:

Minimal processing
Immediate response
Horizontal scaling

Analytics Service runs complex queries across months of data. It needs:

Heavy CPU
Large memory
Fewer instances (expensive to scale)

These services have opposite needs.

A monolith forces you to scale the entire application. Microservices let you scale only what you need.

Service Breakdown

1. Ingestion Service

Endpoint: POST /api/vehicles/{id}/sensor-data
Input: Raw sensor telemetry (30KB per message)
Processing: Validation + De-duplication
Output: 
  - Write to Kinesis (real-time stream)
  - Write to PostgreSQL (persistent storage)
Response: 201 Created (< 50ms)

Scaling: Auto-scale 1-10 instances
Reasoning: This is your bottleneck. High throughput, simple logic.

Why this design?

Fire-and-forget writes to Kinesis = immediate response
Async processing = no blocking operations
Database writes are asynchronous (Lambda consumes Kinesis)
One instance handles 100+ messages/second

2. Analytics Service

Endpoint: GET /api/analytics/vehicle/{id}?from=date&to=date
Input: Query parameters (dates, filters)
Processing: Complex SQL joins, aggregations
Output: Processed metrics (vehicle efficiency, anomalies, trends)
Response: 200 OK (may take 2-5 seconds)

Scaling: Auto-scale 1-5 instances
Reasoning: CPU-bound, fewer instances needed, heavy queries.

Why this design?

Queries run against read replicas (no impact on write performance)
Results cached in Redis (repeat queries = 50ms response)
Batch processing at off-peak hours
One instance per 5-10 concurrent users

3. Auth Service

Endpoint: POST /api/auth/token
Input: credentials or refresh_token
Processing: OAuth2/OIDC with AWS Cognito
Output: JWT token + claims
Response: 200 OK (< 100ms)

Scaling: Auto-scale 1-3 instances
Reasoning: Stateless, lightweight. Rarely the bottleneck.

Why this design?

Stateless = trivial to scale
Outsourced to AWS Cognito = we validate JWTs
No database lookups
Can run anywhere

4. Aggregation Service

Schedule: Every 10 minutes
Input: Raw sensor data from yesterday
Processing: Compress + rollup data
Output: Daily summaries written to cold storage
Storage: S3 (cheap long-term)

Scaling: Batch job, auto-scale 1-2 instances
Reasoning: Runs off-peak, one instance sufficient.

Why this design?

PostgreSQL only keeps 90 days hot
S3 keeps 1 year (Parquet format = 1/10th the space)
Old data queries redirect to S3 (Athena)
Cost reduction: hot vs cold data storage

Orchestration: Kubernetes (EKS)

Why not Docker Compose?

Docker Compose:

✅ Great for local development
✅ Simple deployments
❌ No auto-scaling
❌ No self-healing
❌ No rolling updates
❌ Doesn't survive node failures

Kubernetes (EKS):

✅ Auto-scaling based on CPU/memory
✅ Self-healing (pod dies? restart automatically)
✅ Rolling updates (zero downtime)
✅ Multi-AZ resilience
✅ Proven at scale

Decision: EKS for production. Docker Compose for local dev + CI/CD testing.

Load Balancing: ALB (Application Load Balancer)

Why ALB over NLB?

NLB (Network Load Balancer):
- Ultra-high throughput (millions/sec)
- Layer 4 (connection-level)
- Cost: High

ALB (Application Load Balancer):
- Layer 7 (application-level)
- URL-based routing
- Cost: Reasonable

Your load: ~5,000 requests/second total
→ ALB handles this easily
→ Save money

If you hit 50,000+ requests/second:
→ Upgrade to NLB

Decision: ALB. Easy path to upgrade if needed.

Tier 3: Data (Persistence & Processing)

PostgreSQL + TimescaleDB: Why Time-Series Specialization?

Standard PostgreSQL for time-series data is... bad.

Your data:
- Millions of rows per day
- Queries: "show me data from this vehicle last 30 days"
- Traditional indexing: SLOW

TimescaleDB advantage:
- Hypertables = automatic partitioning by time
- Native time-series functions (rollup, downsample)
- 10-100x faster queries
- Compression = 1/10th storage

Example:

-- Standard PostgreSQL
SELECT AVG(speed) FROM sensor_data 
WHERE vehicle_id = 'V123' 
AND timestamp > NOW() - INTERVAL '30 days'
-- Scans millions of rows, slow

-- TimescaleDB
SELECT time_bucket('1 hour', timestamp) as hour,
       AVG(speed)
FROM sensor_data
WHERE vehicle_id = 'V123'
  AND timestamp > NOW() - INTERVAL '30 days'
GROUP BY hour
-- Uses hypertable partitions, 100x faster

Data Retention Strategy:

Hot (PostgreSQL): 90 days
- Real-time queries
- Full resolution
- Fast

Warm (S3 Parquet): 1 year
- Historical analysis
- Aggregated
- Cheap (Glacier)

Cold (Archival): Indefinite
- Regulatory compliance
- Deep archive (AWS Glacier Deep Archive)

Redis: Why Not Just Use PostgreSQL Cache?

PostgreSQL has built-in caching...

Real-time dashboard needs latest speed for vehicle V123:
- Query PostgreSQL: 200ms (disk I/O)
- Query Redis: 2ms (memory)
- 100x difference

With 1000 concurrent users:
- PostgreSQL: Database collapses
- Redis: Handles trivially

Redis Specific Use Cases:

Session Storage: User login tokens (TTL: 24 hours)
Real-Time Streams: Vehicle location updates (using Redis Streams)
Query Cache: "Latest metrics for vehicle V123" (TTL: 30 seconds)
Rate Limiting: "API key used 950/1000 requests" (TTL: 1 hour)

Why Cluster Mode?

Single Redis instance:
- Holds entire dataset in memory
- 6GB vehicle data = 6GB RAM
- Cost: ~$0.50/hour

Redis Cluster (3 nodes):
- Partitions data across nodes
- Each node holds 2GB
- More resilient
- Cost: ~$0.80/hour (33% premium for resilience)

Decision: Cluster mode. The 33% cost for high availability is worth it.

Kinesis: Real-Time Data Pipeline

Why not just write directly to PostgreSQL?

Direct writes:
POST /ingest → PostgreSQL (synchronous)
Problems:
- Ingest service waits for disk I/O
- Database is bottleneck
- Can't exceed database write throughput

Kinesis pattern:
POST /ingest → Kinesis (fast, async)
Lambda → PostgreSQL (asynchronous processing)
Benefits:
- Ingest responds in 50ms (only writes to Kinesis buffer)
- Database is decoupled
- Can retry failed writes
- Can replay data if needed
- Can add new consumers without changing ingest

Why 1,000 requests/second is our target:

Single vehicle:
- GPS: 10Hz (10 updates/second)
- Sensors: 100Hz (speed, temp, pressure, etc)
- 1 vehicle = 110 messages/second

50 vehicles in production:
50 × 110 = 5,500 messages/second needed
Kinesis shard handles 1,000/sec
→ Need 6 shards
→ Cost: ~$300/month

1 million messages/second:
→ 1,000 shards
→ Cost: $300,000/month (enterprise scale)

Decision: Kinesis with on-demand scaling. Pay per GB ingested.

Design Decisions I Made (and Why)

1. Private Subnets for Everything

Why? Reduces attack surface. No direct internet access to databases.

Internet → ALB (public subnet)
         → EKS (private subnet)
         → RDS (private subnet)

Attacker gets ALB? Can't reach databases directly.

2. Multi-AZ Deployment

AWS Region (e.g., us-east-1)
├── AZ 1: RDS Primary, EKS nodes, Redis nodes
├── AZ 2: RDS Replica, EKS nodes, Redis nodes
└── AZ 3: EKS nodes, backup

One AZ goes down?
→ System still running (automatic failover)

3. Stateless Services

Every service is 100% stateless:

No local files
No in-memory caches
No session storage

Why? Easy horizontal scaling. Kill pod, create new one, no state loss.

4. API Rate Limiting

Per API key: 1,000 requests/minute
Enforced at: ALB + Auth service
Benefit: Prevents one bad actor from crushing system

Data Flow: A Single Vehicle's Update

┌─────────────────────────────────────────────────────────────┐
│ 1. Vehicle Telemetry (GPS + 50 sensors)                     │
│    Sent: Every 100ms                                        │
│    Payload: ~30KB                                           │
└─────────────────┬───────────────────────────────────────────┘
                  │ HTTPS
┌─────────────────▼───────────────────────────────────────────┐
│ 2. ALB (Application Load Balancer)                          │
│    • Rate limit check                                       │
│    • Route to Ingestion service                             │
└─────────────────┬───────────────────────────────────────────┘
                  │
┌─────────────────▼───────────────────────────────────────────┐
│ 3. Ingestion Service (Pod)                                  │
│    • Validate payload (< 5ms)                               │
│    • De-duplicate (Redis check)                             │
│    • Write to Kinesis (async, < 10ms)                       │
│    • Return 201 Created (< 50ms total)                      │
└─────────────────┬───────────────────────────────────────────┘
                  │
   ┌──────────────┴───────────────┐
   │                              │
   ▼                              ▼
┌──────────────────┐      ┌──────────────────┐
│ Kinesis Stream   │      │ Redis Cache      │
│ (Real-time)      │      │ (Latest value)   │
└─────────┬────────┘      └──────────────────┘
          │
┌─────────▼─────────────────────────────────────────────────┐
│ 4. Lambda (Kinesis Consumer)                              │
│    • Runs every 1 second                                  │
│    • Batch processes messages (efficiency)                │
│    • Writes to PostgreSQL + TimescaleDB                   │
│    • Updates Redis streams (real-time feed)               │
└─────────┬───────────────────────────────────────────────┘
          │
┌─────────▼───────────────────────────────────────────────────┐
│ 5. Dashboard (User's Browser)                               │
│    • Subscribes to Redis Streams (WebSocket)               │
│    • Receives updates instantly                            │
│    • Displays latest metrics                               │
│    • Historical queries hit PostgreSQL (cached in Redis)   │
└───────────────────────────────────────────────────────────────┘

Total latency: ~100ms from vehicle sensor to dashboard display

Cost Breakdown (Rough Monthly)

Service              Usage              Cost
─────────────────────────────────────────────
EKS Compute         6 nodes (t3.medium) $150
RDS PostgreSQL      db.t3.small + replica $100
ElastiCache Redis   3-node cluster      $60
S3 Storage          100GB per month     $2
Kinesis Streams     5-10M records       $25
Lambda Processing   1B invocations      $20
Data Transfer       ~50GB out           $5
NAT Gateway         200GB               $30
─────────────────────────────────────────────
TOTAL (monthly):                        ~$392

Annual: ~$4,700 for production system

What I'd Change If Building Today

1. Add a Message Queue (SQS)

Between Ingestion and Kinesis:

Current: POST → Kinesis → Lambda
Problem: Kinesis can get backlogged

Better: POST → SQS → Kinesis → Lambda
Benefit: SQS is cheaper, provides buffering

2. Use EventBridge Instead of Kinesis

EventBridge (AWS's event bus) is newer and:

More flexible routing
Better filtering at source
Native integration with Lambda, SNS, etc.

3. Kafka for Very High Volume

If hitting 100,000+ messages/second:

Kinesis: $300,000+/month
Kafka (self-managed): $2,000/month
Kafka (Confluent managed): $10,000/month

Lessons for Your Architecture

1. Match Tool to Problem

Low latency, high throughput: Use Kinesis + Lambda
Complex queries, low throughput: Use PostgreSQL + read replicas
Session state, caching: Use Redis
Long-term storage: Use S3 with Parquet

Don't try to use one tool for everything.

2. Separate Read and Write Paths

Writes are different from reads:

Writes: Fast, simple, durable (Kinesis)
Reads: Complex, slow, can be stale (Redis cache)

Build them separately.

3. Auto-Scaling Is Non-Negotiable

For autonomous vehicle data, you can't predict demand:

3 AM Tuesday: 0 vehicles
3 PM Friday: 50 vehicles
3 AM Sunday: 10 vehicles

Every service must auto-scale based on load.

4. Think in Tiers

Tier 1 (Hot): Everything in memory, milliseconds
Tier 2 (Warm): Disk-based, seconds
Tier 3 (Cold): Archive, days

This matches cost to need.

Next Steps

This architecture handles Phase 1 comfortably. For Phase 2/3:

Add ML pipeline: Model training on historical data
Add notifications: Alert when anomalies detected
Add analytics: Trend analysis across vehicles
Multi-region: Deploy to multiple AWS regions

TL;DR

3-tier architecture separates concerns: frontend, backend, data
Microservices scale independently (1,000 rps ingestion ≠ analytics queries)
Kubernetes provides auto-scaling, self-healing, and resilience
Polyglot persistence: PostgreSQL (structured) + Redis (real-time) + S3 (archive)
Async pipelines: Kinesis decouples ingest from processing
Cost: ~$400/month for production system handling 50 vehicles

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Cloud Architecture | Microservices | Autonomous Systems

DEV Community: Le Beltagy

Great article for security concepts

I Deployed a Purple Team Inside My Kubernetes Cluster. The Scanners Never Saw It Coming.

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s.

I Learned Go in 3 Weeks. Yesterday, My Code Merged into k9s.

The Setup

Why Not Just Use kubectl?

The k9s Architecture

The Decision I Missed

The Failures (And What Each One Taught Me)

Failure 1: I Changed One Word and Called It a Day

Failure 2: I Blew Up the Test Suite Because I Didn't Understand t.Parallel()

Failure 3: I Didn't Know What an HTTP/1.1 Upgrade Was

The PR

What I Learned About Contributing to Big Projects

1. Start with the issue, not the code

2. Read the test file before the source file

3. go test -race is not optional

4. Two-line fixes need two-hundred-line tests

5. Backward compatibility is a constraint, not a suggestion

Why This Matters (And Why It Doesn't)

It matters because:

It doesn't matter because:

What's Next?

TL;DR

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

The Setup

Why Not Just Use EKS?

The Architecture

Design Decisions

The Failures (And What Each One Taught Me)

Failure 1: I Burned 3 SD Cards in 6 Weeks

Failure 2: A Bad CiliumNetworkPolicy Locked Me Out of Every Node

Failure 3: Longhorn Filled My Disks and kubelet Evicted Everything

Failure 4: MetalLB Gave Out IPs Already Used by My Router's DHCP Pool

Failure 5: I Upgraded Talos and Lost the Cluster Because I Didn't Backup etcd

What's Running on It Now?

3 Lessons That Transfer to Production

1. Resource quotas are not bureaucracy

2. Observability first, workload second

3. GitOps is the only way I trust myself

The Hardware List (If You Want to Replicate)

TL;DR

I Deployed a Purple Team Inside My Kubernetes Cluster. The Scanners Never Saw It Coming.

The Setup

Introducing NEMESIS

The Stack

Attack Engine

eBPF Integration

The Attack Chain That Scanners Missed

Step 1: Recon

Step 2: Token Abuse

Step 3: Lateral Movement

Step 4: Data Exfil Test

The Gap: What You Declare vs. What You Get

Why This Changes How You Think About K8s Security

What I Learned Building This

Where NEMESIS Goes Next

Try It Yourself

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

The Setup

Why This Problem?

The Architecture

Design Decisions

The Pipeline (Simplified)

The Telegram Output

5 Failures Before It Worked

Failure 1: LinkedIn Rate-Limited Me in 6 Minutes

Failure 2: The LLM Hallucinated a Role at Google That Didn't Exist

Failure 3: I Applied to the Same Job 4 Times

Failure 4: Running It on My Laptop Meant It Died When I Closed the Lid

Failure 5: I Got 18 "Matches" and Felt Overwhelmed

Results (So Far)

The Code

I Need Collaborators

3 Lessons That Transfer Beyond Job Searching

1. Automate the boring, not the important

2. Your side project's tech stack should teach you something

Failure 2: I Blew Up the Test Suite Because I Didn't Understand `t.Parallel()`

3. `go test -race` is not optional