I Learned Go by Hacking Kubernetes RBAC Security (And Nearly Quit 5 Times)

From zero Go experience to shipping kube-radar — a Kubernetes security CLI that detects RBAC wildcard violations. The story, the architecture, the mistakes, and why I did it the hard way.

The Setup

I manage Kubernetes clusters at Siemens. Not "I run kubectl apply from my laptop" manage — I mean Talos Linux on bare metal, Cilium replacing kube-proxy, eBPF policies doing deep packet inspection, ArgoCD running GitOps end-to-end. The kind of infra where one bad RoleBinding can cost you the crown jewels.

But here's the problem: I had never written production Go.

My code was Python — automation, Lambdas, Flask services, AI pipelines. My Go experience was exactly one Pulumi IaC program where I mostly copy-pasted from examples and hoped go build didn't yell at me.

I wanted to become what I call a "backend + cloud-native mega engineer" — someone who can architect a distributed system and ship the code that runs inside it. Not just configure the cluster, but write the controller. Not just deploy the service, but design the API.

So I made a deal with myself:

Build one real, public, useful project in Go. Ship it. Then use it as proof.

That's how kube-radar started.

Why This Project?

Kubernetes RBAC is a nightmare to audit.

If you've ever done a security review on a real cluster, you know the pain:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

This is a cluster-admin equivalent disguised as a single rule. Anyone bound to this can do anything in your cluster. And unlike cluster-admin, it doesn't show up in obvious places like kubectl get clusterrolebindings when you're scanning for privilege escalation.

Wildcard rules (verbs: ["*"], resources: ["*"], apiGroups: ["*"]) are the silent killers of K8s security posture. They slip past most CNAPP tools because they're technically valid — they just give away the kingdom.

I wanted a tool that:

1. Scans any cluster's RBAC in seconds
2. Flags wildcard over-permissions with severity
3. Tracks findings across scans (is this new, or did someone just accept it?)
4. Gates CI/CD with exit codes — fail the build when someone commits a wildcard rule
5. Reports in SARIF for GitHub Advanced Security integration

And I wanted to build it from scratch in a language I'd never shipped in. Because if I can ship this, I can ship anything.

The Learning Curve (Or: How I Nearly Gave Up 5 Times)

Attempt 1: The Cathedral Plan

I sat down and wrote a 5,000-word design document.

I'm not kidding:

• Full SQLite schema with WAL mode, migrations, and ON DELETE CASCADE
• Fingerprinting strategy with SHA-256 canonical hashing
• Cobra CLI with 6 subcommands
• Rule engine with Go interfaces and a plugin registry
• Diff engine with set operations across snapshots
• .goreleaser cross-compilation for 4 platforms
• SARIF 2.1.0 schema validation

It was beautiful. It was also unbuildable for someone learning Go from zero. After 3 weeks of wrestling with database/sql semantics and SQLite driver imports, I had a schema but no working code.

Lesson 1: A perfect design you can't ship is worthless.

Attempt 2: Copy-Paste Panic

I found a GitHub repo that did something vaguely similar. I copy-pasted Go client-go List calls, cobra wiring, even error wrapping. It compiled! It ran! It... panic'd on startup because I shadowed err in an if block and the outer function didn't know.

Go's error-as-values model broke my brain. In Python, I'd raise an exception and catch it somewhere. In Go, the error is just a return value you need to handle every single time. My code was 40% error handling by line count.

Lesson 2: You can't paste your way to understanding.

Attempt 3: The Interface Rabbit Hole

The design doc said "Rule interface type." I spent a full day understanding why interface{} isn't the same as a typed interface, why empty interfaces are a smell unless you need them, and why Go composition beats inheritance so completely.

It clicked when I realized: a Rule in my analyzer isn't a class. It's a contract.

type Rule interface {
    ID() string
    Describe() Description
    Evaluate(s *model.Snapshot) []model.Finding
}

KR001 (wildcard verbs), KR002 (wildcard resources), KR003 (wildcard apiGroups) — all three are just structs that implement those three methods. No inheritance tree. No base class. Just behavior.

Lesson 3: Go interfaces are small, implicit, and compositional. Stop thinking in classes.

Attempt 4: client-go vs. Kubernetes API

Here's something nobody tells you: kubectl does a lot of heavy lifting that client-go doesn't.

kubectl config current-context uses shell-level kubeconfig resolution. client-go's clientcmd.BuildConfigFromFlags doesn't. It took me 2 hours to figure out why my code was connecting to the wrong cluster — client-go reads the kubeconfig file but doesn't use the current-context field unless you explicitly load it.

// This does NOT use your shell's "kubectl config current-context"
config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)

// This DOES — it loads the kubeconfig and resolves context, cluster, user
config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
    &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},
    &clientcmd.ConfigOverrides{CurrentContext: contextName},
).ClientConfig()

That single discovery changed how I build the kube-radar scan command.

Lesson 4: client-go is lower-level than kubectl in ways that matter.

Attempt 5: Determinism

My first version printed findings in random order every time. Tests flaked. Diff was impossible. Fingerprints changed because I included rule index in the hash.

The fix was a lesson in Go's core philosophy: if the language doesn't guarantee something, you must enforce it yourself. Go map iteration is deliberately randomized. So you sort. Always.

sort.Slice(findings, func(i, j int) bool {
    if findings[i].Severity != findings[j].Severity {
        return findings[i].Severity > findings[j].Severity // desc
    }
    return findings[i].Fingerprint < findings[j].Fingerprint
})

Lesson 5: Randomness is a feature of the language. Determinism is your job.

The Architecture

Here's what I shipped — and more importantly, why the architecture looks like this:

Design Decisions

Decision	Chose	Over	Because
SQLite driver	modernc.org/sqlite	mattn/go-sqlite3	Zero CGO = trivial cross-compilation. 2× slower, irrelevant at this scale.
State	Local SQLite	Stateless linter	Diff/baseline is the differentiator; stateless tools already exist
Role handling	Unified RoleLike type	Separate Role/ClusterRole types	Rules written once; kind preserved for reporting severity
Migrations	Hand-rolled (~20 lines)	goose/golang-migrate	Zero deps, more learning value. Revisit when migrations exceed ~10 files.
Persistence	Per-machine SQLite DB	Centralized API/DB	CLI-first tool; no server to maintain, no network dependency
Fingerprinting	SHA-256 canonical content hash	UUID per finding	Stable across rescans; "same problem" = "same fingerprint"

The Fingerprint System (The Most Underrated Part)

This is where most security tools fail: they generate "new findings" every scan because of random IDs or timestamps. My fingering system:

// v1|<rule_id>|<kind>|<namespace>|<name>|<canonical_rule>
canonical_rule = sort-and-join(apiGroups) + "|" + sort-and-join(resources) + "|" + sort-and-join(verbs)
fp = sha256(v1|KR001|ClusterRole||admin|*|*|*)[:16]

Key properties:

• Stable across rescans: Same cluster state → same fingerprint
• Survives reordering: If someone reorders rules in a Role, the fingerprint doesn't change
• Detects narrowing: verbs: ["*"] → verbs: ["*"] + resources: ["pods"] is a different finding (content changed)

This makes the diff feature in v0.2 trivial: it's just set arithmetic on fingerprint maps.

Exit Codes as CI Contract

0 — Clean scan, no new findings ≥ --fail-on
1 — New unacknowledged findings ≥ --fail-on severity threshold
2 — Operational error (unreachable cluster, bad flags, DB failure)

Why only 3 codes? Because CI pipelines need simple logic: kube-radar scan && deploy || fail works with zero tooling.

What It Looks Like

$ kube-radar scan --context homelab --fail-on medium

RULE    | KIND        | NAMESPACE | NAME          | DETAIL
--------|-------------|-----------|---------------|------------------
KR001   | ClusterRole | -         | admin-legacy  | Wildcard verbs: [*]
KR002   | Role        | default   | app-service   | Wildcard resources: [*]
KR003   | ClusterRole | -         | cluster-admin | Wildcard apiGroups: [*]

3 findings (1 Critical, 1 Medium, 1 Low)  exit code: 1

That exit code: 1 means CI fails. That means the wildcard rule does not get deployed.

3 Lessons That Transfer Beyond This Project

1. Design the hard thing in writing first

My 5,000-word design doc wasn't a waste. It was a map. When I cut the scope to v0.1 (no SQLite, no diff, no SARIF), I knew exactly what each deferred feature looked like because I'd already designed it. The doc became the roadmap.

2. Build for yourself, then write about the pain

Every feature in kube-radar solves a problem I actually have:

• Can't find wildcard rules quickly → the scan command
• Can't tell if a finding is new → the diff feature (v0.2)
• Can't baseline accepted risk → acks (v0.2)
• Can't integrate with GitHub Security → SARIF (v0.2)

Tooling built from real pain is better than tooling built for a portfolio.

3. A shipped v0.1 beats a perfect v2.0

I had to fight my own instinct to build everything at once. v0.1 is:

• 1 command (scan)
• 3 rules (KR001-003)
• Table output only
• Fingerprints computed but not yet persisted

But it works. It already catches real violations. It already fails CI builds. Everything else is additive.

The Roadmap

Version	What
v0.1 ✅	Scan, 3 rules, table output, fingerprints, exit codes
v0.2	SQLite store, diff, history, acks/baselines, JSON + SARIF
v0.3	Full rule pack: cluster-admin bindings, secrets read, pods/exec, impersonate, escalate/bind
v1.0	Multi-context fleet scan, config file, goreleaser
v2.0	Controller mode: watch RBAC, Prometheus metrics, admission warnings

Try It

git clone https://github.com/beltagyy/kube-radar.git
cd kube-radar
go build -o kube-radar ./cmd/kube-radar
./kube-radar scan --kubeconfig ~/.kube/config --fail-on medium

Or grab a release binary (when I cut v0.1.0 — soon).

What I need from you:

• Does it work on your cluster?
• What's missing for your use case?
• Are there wildcard variants I'm not catching?

Every issue helps me learn.

About This Series

This is post 1 in the "kube-radar: Building a Security CLI in Go" series. Next up:

• Deep-dive: Why I chose SQLite over in-memory state
• Deep-dive: Designing deterministic fingerprints for security findings
• Deep-dive: The Go interfaces that made the rule engine testable

If you're learning Go from a cloud background like I am — or if you're a backend engineer who wants to understand K8s security internals — follow along.

---

*Questions? I'm @beltagyy on GitHub and beltagyy on DEV.

Comments

[Mustafa ERBAY]

I like that this started as a learning project but immediately focused on a real operational problem instead of another toy CLI. The deterministic fingerprinting is probably the most interesting part for me, because stable findings are what make diffing, baselines, and CI gating actually usable over time. One thing I’d be interested in seeing as the project grows is moving from rule detection to effective permission analysis. Wildcard rules are important, but the real security posture depends on how Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, aggregation rules, and subjects combine into the permissions a workload or user actually receives. That’s where many privilege-escalation paths hide. The roadmap already points in that direction, and I think that’s where kube-radar could really differentiate itself from a simple RBAC linter.

[Le Beltagy]

That's exactly the direction I want to take it. Right now it's intentionally focused on deterministic rule detection, but the long-term goal is to build an effective permission graph that resolves Roles, ClusterRoles, bindings, aggregation, and subjects into the permissions a workload actually has. I agree that's where the really interesting security insights and most privilege escalation paths start to appear. Glad you noticed the roadmap, because that's definitely the direction I'm aiming for.

[Mustafa ERBAY]

That’s great to hear. I think an effective permission graph could become one of the biggest differentiators of the project. Once you can answer questions like “what can this ServiceAccount actually do?” or “which workloads inherit this permission through multiple bindings?”, the tool moves beyond linting into security analysis. At that point, I could also imagine exporting the graph for visualization or integrating it with attack-path analysis. Looking forward to seeing where you take it.

[UnitBuilds]

There's just something different about frustration driven development. Not frustrated development, innovation that is fueled by the passion of 'I hate this so much, I dont care how long it takes me, I will build a system so I never have to deal with this again' 😂 that's actually the same reason I had built V.A.L.I.D. as a personal challenge, because I had barely used Roslyn prior and I was annoyed at CSLA, so I used Roslyn to build a better CSLA, now everything is so much easier and nicer. A shame I cant use it for my day job... But it's the thought that counts, atleast I built the solution to my daily pains, even if I cant use it.

[Le Beltagy]

That's such a relatable mindset. Some of the best tools come from someone finally saying, "I'm done solving this problem manually." 😄 Even if you can't use V.A.L.I.D. at work, building it forced you to learn Roslyn deeply and gave you a solution that reflects how you think software should be built. That's rarely wasted effort.

[UnitBuilds]

Honestly, it's made my side-projects alot nicer. Lets face it, if you've ever dealt with Blazor, you'd know it's a steaming pile... It's slow, it's bloated, but it's nice to write for the most part. V.A.L.I.D. didnt just replace CSLA, it fundamentally fixed Blazor at a level I never imagined. From 600 mutations per second, to over 3600/s with multi-parameter rule validation in nanoseconds. It ended up exceeding my expectations, because I used my experience in F# and Rust to optimize it to a point where it's easier to write, easier to maintain and performs far better than practically anything else.

[GoluScriptMage]

I just started learning from yesterday
Any advice?