DEV Community

Mahesh Rayas
Mahesh Rayas

Posted on

Part 1: Least-privilege Kubernetes, generated from what eBPF saw

Your pods already tell the kernel what they connect to. kguardian listens with eBPF and writes the NetworkPolicy from what it saw.

Part 1 of a series on kguardian and eBPF. Parts 1–3 are the tour: what the tool does, from the operator's chair. Parts 4–7 open the hood on the eBPF underneath. No kernel knowledge needed for this one.

Where this came from

kguardian started as a conversation between Michael Fornaro and me about five years ago. We had both been running Kubernetes long enough to have the same complaint.

Every cluster we worked on wrote security policy the same way: from a template, or from whatever the last team had left behind. Nobody wrote a NetworkPolicy from evidence, because there wasn't any. You either shipped something so permissive it was decorative, or you shipped something tight, broke a workload nobody had told you about, and rolled it back. Seccomp was worse. Most people skipped it entirely.

What nagged at us was that the information existed. The kernel watches every connection and every syscall these workloads make. It just had no way of reaching us. Around then we both wanted an excuse to get properly into eBPF, and this turned out to be the right size of problem. Big enough to be worth solving, specific enough to finish.

We also wanted one tool. Not a collector plus a database plus a dashboard plus three scripts, stitched together and re-explained to every new engineer. One thing you install that watches what your applications do and writes the policy they need.

kguardian is under heavy development and I'd call it early rather than finished. The pace has picked up in the last year, partly because AI tooling has made the unglamorous parts.The tests, the plumbing, the docs, cheap enough to keep up with the interesting parts. Features are landing faster than this blog can keep up with. Expect the screenshots here to age.

The question nobody can answer

Pick a pod in your cluster. Now answer, without guessing: what does it connect to, what connects to it, and does it reach anything outside the cluster?

Almost nobody can. That's not a skills problem. The information isn't anywhere convenient. It's not in the manifest, it's not in the Helm chart, and your CNI only knows about traffic it already allowed.

Which is why least-privilege stays theoretical. Writing a NetworkPolicy by hand means answering those questions from memory, being wrong about at least one of them, and finding out during an incident. It's always DNS. Someone forgets DNS egress, the pod can't resolve anything, the policy gets reverted, and nobody proposes it again.

kguardian's premise is that you don't have to guess. The kernel already knows. eBPF is how you ask.

Install

You need Kubernetes v1.19+, kubectl v1.19+, and Linux kernel 6.2 or newer on every node. Check the kernel first, because it's the requirement people trip on:

kubectl get nodes -o wide
Enter fullscreen mode Exit fullscreen mode

Then:

helm install kguardian oci://ghcr.io/kguardian-dev/charts/kguardian --namespace kguardian --create-namespace

sh -c "$(curl -fsSL https://raw.githubusercontent.com/kguardian-dev/kguardian/main/scripts/quick-install.sh)"
Enter fullscreen mode Exit fullscreen mode

The first command installs the in-cluster pieces: an eBPF DaemonSet on every node, a broker with a PostgreSQL database, an evaluator, and the web UI. The second installs the kubectl kguardian plugin locally.

On k3s, point the controller at the right containerd socket:

--set controller.containerdSockPath=/run/k3s/containerd/containerd.sock
Enter fullscreen mode Exit fullscreen mode

Then nothing happens

Open the UI locally.

kubectl port-forward svc/kguardian-frontend 5173 -n kguardian
Enter fullscreen mode Exit fullscreen mode

kguardian doesn't read your manifests. It reports what it observed, and right after install it has observed nothing. Give it a few minutes and pods appear as they make their first connections.

Here's the thing to understand about the tool: a generated policy is only as complete as the window it was learned from. Your workload probably has a path that runs rarely. A nightly batch job, a failover, something that only fires on the first of the month. Generate a policy before those have run and it won't allow them. It will look fine. It will break later.

Observe for a full business cycle before you generate anything you intend to enforce.

The map

Every node in the diagram above is a workload. Every edge is a connection eBPF saw, not one your CNI allows, not one implied by a Service definition. Something on this graph means a socket was opened.

The legend names the edge types. Trusted is internal pod-to-pod, Egress is traffic leaving the cluster, and Denied is a connection that was attempted and never completed. There's a fourth kind, Contention, which isn't about traffic at all; that's part 3 in the series.

Node colours carry a similar signal: workloads in the namespace, external peers, and DaemonSet peers. That last group is node-level agents like your CNI or log shipper, which appear in almost every namespace's traffic and would otherwise drown out the rest.

Look at the edge labels. They carry port and protocol: 3306/TCP into cmangos-database, 8086/TCP and 8085/TCP into the game servers, and 25575/TCP from rcon-web-admin to minecraft. That last one is Minecraft's RCON port, which is exactly the sort of thing you want to know is reachable and exactly the sort of thing nobody writes down.

Note the naming. kguardian labels nodes by workload identity rather than pod name, so the node reads cmangos-ptr, and azerothcore appears as one node marked three replicas instead of three separate blobs. That matters because pod names are ephemeral and the policy you're about to write is not. A rule pinned to cmangos-ptr-77558bd57d-82ps7 is wrong the moment that pod is rescheduled.

Click a node to expand it and you get the receipts: how many connections and how many distinct syscalls this workload has been seen making. There's a Focus control that collapses the graph to just this node and what it talks to, which on a namespace with forty workloads is the difference between a hairball and an answer. And there's a Build Policy button, which we'll get to.

Scroll down and the flow table sits underneath the graph.

Every observed connection as a row, and both names are here, which is the point. The identity cmangos-ptr on top, the pod cmangos-ptr-77558bd57d-82ps7 and its IP underneath. The graph is the shape of things; this is the evidence, and it's what you scroll when someone asks when that actually happened.

It reaches outside the namespace you're looking at, too. There's ingress from envoy over in network-system, and egress to kube-dns in kube-system over UDP. That last row is the DNS rule everybody forgets, showing up as an observed fact rather than something you have to remember.

One small thing that turns out to matter a lot in practice: the view, the namespace, the selected workload and the focus state all live in the URL hash. The map you're looking at is a link. Pasting "here's what the payments namespace talks to" into an incident channel is a different conversation from "let me screenshare and click around."

Findings

The map shows you everything. Findings shows you what's wrong.

Five counters across the top, with the same things as reviewable sections below. Three of them are about traffic.

Would-deny policies is every flow your AuditNetworkPolicy resources would have dropped, had they been enforcing. This is the list you want empty before you switch a policy on, and there's more on that at the end of this post.

Blocked connections is outbound connections that never completed. The UI shows the cause per flow and is careful to say that a policy is only one possibility. A connection that never established might be a NetworkPolicy. It might equally be a missing Service, a crashed backend, or a real outage. kguardian reports what happened and what it can infer, then stops.

High egress fan-out catches workloads reaching an unusual number of distinct destinations. Sometimes that's a legitimate crawler or a service mesh. Sometimes it's the first thing you'd want to know about.

The other two counters, Sensitive syscalls and Compute, are parts 2 and 3.

Rows link straight into the work. Policy opens the builder on the tab relevant to the finding, and the chevron jumps to that workload on the map, so triage doesn't require you to remember where anything lives.

Generating a policy

Open the Policy Builder from a node, from a finding, or from the command palette, and pick a workload. The subtitle says what it's doing: generate a policy from its observed traffic and syscalls.

Out comes a policy where every rule corresponds to something eBPF saw. The ingress rule above exists because Prometheus really did scrape that pod on 9094, not because a template guessed it would.

Then look at the comment in the generated YAML: # unattributed peer 10.244.13.94 at 2026-09-15T05:01:37. The generator saw a flow it couldn't resolve to a workload, and rather than dropping it or inventing a peer, it left you the address and the timestamp to go and find out.

Three output formats sit as tabs across the top. Audit (kguardian CR) produces an AuditNetworkPolicy with the same spec, except nothing is dropped and the evaluator reports what it would deny. NetworkPolicy is upstream Kubernetes. CiliumNetworkPolicy is there if you run Cilium, and the editor warns you when your detected CNI doesn't match the tab you've picked this saves a specific kind of bad afternoon, because a Cilium policy on a non-Cilium cluster is a YAML file that does nothing, silently.

There's a seccomp tab in the same builder. That's part 2 in the series.

Copy it, or download it. And then kguardian does not apply it.

Don't be misled by the Save Policy button; it downloads the file. Nothing in this dialog touches your cluster, and the CLI likewise writes files to a directory and stops. The footer says as much: review and customize before applying. You review the YAML, commit it, and apply it through whatever process you already trust. A tool that both invents your security policy and enforces it unsupervised is not one I'd want running privileged on every node either.

For anything repeatable, the plugin does the same job:

# One pod
kubectl kguardian gen networkpolicy my-pod -n default --output-dir ./policies

# Cilium policies for an entire namespace
kubectl kguardian gen netpol --all -n staging --type cilium --output-dir ./policies
Enter fullscreen mode Exit fullscreen mode

Turning it on without an incident

A generated policy is a hypothesis. The AuditNetworkPolicy tab is how you test it.

It's byte-identical in spec to the NetworkPolicy, but nothing is dropped. The evaluator watches live traffic against it and reports every flow it would have denied, which lands back in the Would-deny section of Findings. Apply the audit version, leave it for a cycle, and read the verdicts. The ones you expected mean the policy is doing its job. The ones you didn't mean you generated too early, and the workload has a path you hadn't observed yet.

When the list goes quiet, promote it:

kubectl kguardian audit promote <name> -n <ns> | kubectl apply -f -
Enter fullscreen mode Exit fullscreen mode

Observe, audit, enforce. Skipping the middle step is how people end up reverting NetworkPolicies and never trying again.

What it costs

From a 3-node cluster with 18 vCPU and 47 GiB per node running Cilium, watching 234 pods across 26 namespaces: the eBPF DaemonSet sits at roughly 60 MiB and 0.1–0.6 vCPU per node, the evaluator at about 26 MiB with negligible CPU, and PostgreSQL is the dominant consumer, so size it generously.

The number I'd actually pay attention to is that storage growth is dedup-bounded. Once a workload's set of flows and syscalls is learned, new rows drop to roughly zero per minute. The database grows with new behaviour, not with time or traffic volume. A monitoring tool that costs more every week doesn't survive contact with a real cluster, and getting that right is most of what parts 5 and 6 are about.

What it doesn't do

People arrive at this with a Falco or Pixie mental model, so it's worth being clear about the boundaries.

It doesn't enforce anything, ever; it writes YAML. It doesn't do runtime threat detection but it builds baselines, it doesn't alert on live attacks. And it can't tell you about behaviour it never saw, which is the whole ballgame, and why "observe for a full business cycle" is the advice this post keeps repeating.

Next in this series

Part 2 covers seccomp: syscall allow-lists, capture tiers, and the drift number that stops generated policy from rotting.
Part 3 is everything about compute, live gauges,noisy neighbour and naming the pod on the same node that's starving this one.

After that the series goes under the hood.
Part 4 is the decisions every probe inherits: CO-RE, why kernel 6.2, hooking sockets instead of packets.
Part 5 is network capture, including how you detect a blocked connection when the kernel never tells you one happened.
Part 6 is syscall capture and how tracing every syscall on the node ends up costing almost nothing.
Part 7 is sched_contention: run-queue latency histograms and a per-pair blame matrix, in kernel.
... and more

Come and talk about it

If you try this on your own cluster I'd like to hear how it goes, especially the awkward cases. An odd CNI, a workload whose traffic pattern doesn't fit, a policy that turned out wrong.

Join the discussion on Discord if you have any questions, bug reports and "is this supposed to happen?" all welcome. Bugs and feature requests are fine as GitHub issues too.

And if the idea appeals, star the repo. It's the cheapest signal that this is worth continuing to build, and at this stage it genuinely helps.

kguardian is on GitHub (BSL 1.1, converting to Apache 2.0 on 2029-01-01). Docs at docs.kguardian.dev.

Top comments (0)