DEV Community: Granite

Kubernetes Audit Policy: A Checklist Against Common Blind Spots in Your Rules

Alex Gorshkov — Thu, 17 Sep 2026 10:11:43 +0000

Hi everyone! We want to share our experience writing and reviewing a Kubernetes audit policy.

A Kubernetes audit policy is one of those things that gets written once during cluster setup and then lives "as is" for years, accumulating exceptions for new components and almost never being reviewed as a whole. At some point it's no longer a security policy — it's a layer of three-year-old comments.

We recently sat down to revisit our own 500+ line config. We had assembled it from several sources, primarily based on the Kubernetes Threat Matrix. We want to share not so much the specific findings (those are specific to our cluster) as the principles and common traps that surfaced during the review. If you're writing or reviewing an audit policy for your own cluster, this checklist should save you some time.

How Audit Policy Works

In short: it's a list of rules, each describing which users/groups/verbs/resources it matches and which logging level to apply — from None (don't log at all) to RequestResponse (log the full request and response, including the body).

A key detail that's easy to miss: rules are evaluated top to bottom, and the first match wins. If a broad exception rule sits above a narrow, security-critical rule, the second one will never fire for the same requests. Most of the problems actually worth hunting for in a review are ordering mistakes, not mistakes in individual rules.

Principle 1: Separate Noise from Signal

In any reasonably active cluster, 90% of API traffic is get/list/watch from system components: the kubelet polling node status, Prometheus scraping metrics, controllers watching their resources. If you log all of that at a meaningful level, the log drowns in noise and finding a real incident becomes impossible.

The right pattern is to silence these streams precisely:

- level: None
  users: ["system:serviceaccount:kube-system:coredns"]
  verbs: ["get", "list", "watch"]

Important: silence as narrowly as possible — by specific user, specific verbs, and ideally specific resources. This is where the first common mistake hides.

Trap #1: An Exception Without Verb/Resource Restrictions

This one shows up regularly:

- level: None
  users: ["system:serviceaccount:some-ns:some-controller"]

Without verbs and resources, this rule silences every action by that user or service account — not just the routine read traffic it was written for, but any mutations that service account is ever granted (accidentally through an RBAC mistake, or intentionally when functionality expands). If that service account's token is compromised, the attack goes entirely unnoticed by the audit log — not because anyone planned a hole, but because the exception was written "by eye" for the component's current behavior, without explicitly fixing its boundaries.

Practical rule: when adding a new exception, always explicitly list verbs, even if the component currently only does get/list/watch. This pins down the boundaries — it defines what we see and what matters to us.

Trap #2: Stale Comments

# Any actions with secrets and configs (except list/watch) - critical to monitor
# for potential data leaks or unauthorized configuration changes
- level: None
  verbs: ["get", "create", "update", "patch", "delete"]
  resources:
    - group: ""
      resources: ["secrets", "configmaps"]
  users: [...]

The comment says "critical to monitor," but the rule actually disables logging for the listed service accounts.

This doesn't affect how the policy works, but inconsistencies like this are a direct path to someone reading the comment a year from now, taking it at face value, and spending a day debugging "why aren't events being logged when it says they should be." Reviewing comments is as much a part of code review as reviewing the logic.

Trap #3: Leftover Rules from Past Migrations

The CNI was migrated from kube-router to Cilium six months ago, but the exception for system:kube-router is still in the file. On its own it's harmless — it simply never fires, because that service account no longer exists. But:

it clutters the file and confuses reviewers;
if someone ever creates a service account with the same name for a different component, it inherits someone else's stale exemptions.

Simple check: periodically (say, once a quarter) compare the list of users and service accounts in your exception rules against the service accounts that actually exist in the cluster — kubectl get sa -A — and clean up the mismatches.

Principle 2: For Sensitive Data, Cap at Metadata

A separate pattern from the reviewed config:

# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
# so only log at the Metadata level.
- level: Metadata
  verbs: ["get", "create", "update", "patch", "delete"]
  resources:
    - group: ""
      resources: ["secrets", "configmaps"]
    - group: "authentication.k8s.io"
      resources: ["tokenreviews"]

The logic is simple: the fact that a secret was accessed must be recorded (who, when, which secret), but the contents of the secret must never end up in the audit log under any circumstances — otherwise the audit log itself becomes a secret store, just a less protected one.

Principle 3: Use RequestResponse Where It's Justified

On the other end of the spectrum are events where you need to see not just the fact, but the full request body:

- level: RequestResponse
  resources:
    - group: ""
      resources: ["pods/exec", "pods/attach", "pods/portforward"]
- level: RequestResponse
  verbs: ["create", "update", "patch", "delete"]
  resources:
    - group: "rbac.authorization.k8s.io"
      resources: ["clusterrolebindings", "clusterroles", "rolebindings", "roles"]

exec/attach/portforward is direct access inside a running container — a classic vector when investigating incidents. RBAC changes are potential privilege escalation. In both cases, skimping on the logging level isn't worth it: the difference in log volume is small, and the value during an investigation is enormous.

Principle 4: Use Rule Ordering Deliberately, Not Accidentally

A good example of using ordering intentionally is how events are handled:

# Defense evasion tactic - detect attempts to delete k8s events
- level: Metadata
  verbs: ["delete"]
  resources:
    - group: ""
      resources: ["events"]
    - group: "events.k8s.io"
      resources: ["events"]

# Don't log events requests.
- level: None
  resources:
    - group: ""
      resources: ["events"]
    - group: "events.k8s.io"
      resources: ["events"]

Events themselves are a very noisy resource, and logging every access to them isn't practical. But deleting events is a classic defense evasion technique (an attacker cleaning up traces of their activity). By placing the narrow delete rule above the general "silence everything" rule, we get exactly the behavior we want: normal event traffic doesn't clutter the log, but an attempt to delete events is recorded.

Keep this in mind as a general pattern: if the same resource needs different sensitivity depending on the verb, the specific rule must always come before the general one.

Mini-Checklist for Reviewing Your Audit Policy

Is there a rule for system:unauthenticated, and does it sit above all exception rules?
Is there any exception rule (level: None) without explicit verbs and/or resources? If so, can it be narrowed?
Are secrets/configmaps/tokens never logged above Metadata?
Are RBAC changes, exec/attach/portforward, and network policy deletions logged at a sufficient level (Request/RequestResponse)?
Do any exception rules reference users/service accounts that no longer exist in the cluster (leftovers from migrations)?
Do the comments match what the rules actually do?
Is there a catch-all rule at the very end of the file (usually level: Metadata) so nothing slips past the log by default?
Are there any obviously temporary rules without a date or ticket for their removal?

Conclusion

A Kubernetes audit policy is not a "set it and forget it" thing. It grows with the cluster, and every new exception added in a hurry to silence yet another noisy component is a potential blind spot if it isn't explicitly bounded. The difference between "convenient" and "secure" here usually comes down to a single line — an explicit list of verbs.

If you have your own findings or patterns from working with audit policies, share them in the comments — we'd love to compare approaches.

Read-Only Kubernetes Access for AI Agents: Why "Please Don't Delete Anything" Isn't a Security Boundary

Alex Gorshkov — Tue, 18 Aug 2026 13:36:09 +0000

I keep seeing the same pattern: take an LLM, give it access to kubectl or the k8s API, write something like "you can only read, don't delete or modify anything" in the system prompt or an attached skill, and consider the problem solved. I went through this myself and at some point realized that this isn't a security boundary — it's a polite request.

This isn't a hypothetical risk: you probably remember how in July 2025, the Replit agent deleted the SaaStr database despite a direct prohibition on making any changes — not Kubernetes and not MCP, but the same pattern. The "don't touch anything" instruction was right there in the context, and there was simply no one to enforce it except the model itself. Giving an agent write access to a k8s cluster means assembling exactly the same construct that already cost SaaStr their database.

I'm far from the first to cover this topic, and a lot of read-only MCP servers have appeared recently. But it's surprising how often "read-only" is misunderstood in them. For example, in MCP servers for Kubernetes, "read-only" is often implemented as an environment variable that filters the tools/list response, rather than as the absence of a function in the registry. That's exactly how mcp-server-kubernetes (20k weekly downloads on npm) worked: the ALLOW_ONLY_READONLY_TOOLS flag hid mutating tools from the list, while tools/call still accepted kubectl_delete directly, bypassing the filter.

This became CVE-2026-46519, CVSS 8.8 — the same principle this article is about, taken to an actual exploit: a function hidden from the list is not the same as a function that doesn't exist. And this isn't just a community problem — Azure/mcp-kubernetes, Microsoft's official MCP server for Kubernetes, is built exactly the same way: --access-level readonly|readwrite instead of simply not having mutating tools in the first place.

Why Instructions Don't Work as a Restriction

A model is not a sandbox. If it has delete_pod or scale_deployment in its list of available tools, it can technically call it regardless of what's written in the system prompt. For example, an attacker doesn't need cluster access for this — a regular HTTP request with a spoofed header value like User-Agent is enough. Nginx or the app itself will log it as-is: Kubernetes simply captures the container's stdout/stderr without any sanitization.

Then someone (or the assistant itself) asks the model to "check this pod's logs" — a completely innocuous request — and the model reads that line as part of the context, without distinguishing it from the system prompt. The same story applies to instructions from an attached skill or another plugin, which end up in the context as equally "trusted"; to jailbreaks; to ordinary hallucinations in an attempt to "fix" a problem you just asked it to explain. An instruction — whether in a system prompt, a skill, or a container log — is data that the model interprets, not code that constrains it.

This means the only boundary that actually holds is which tools exist in the registry available to it. If the delete_pod function doesn't exist, it doesn't matter what prompt injection, jailbreak, or the model itself in a fit of "helpfulness" says: there's nothing to call.

What This Looks Like in a Tool Registry

Let's take a concrete example: an MCP server for Kubernetes. In its registry, it makes sense to register only read tools — list_pods, list_deployments, get_yaml, get_events, read_pod_logs, start_pod_log_stream, and so on, around thirty in total. And not a single delete_*, scale_*, exec_*, apply_*, or port_forward_* — not because they're disabled by some flag, but because those functions simply don't exist in the code.

All mutating operations — scale, rollout restart, delete, cordon/drain — in this design live in a separate, human path: through a GUI with a confirmation dialog, through a CLI with an explicit flag or a "y/n" prompt. It doesn't matter which — what matters is that a human confirms it, and only then does a direct Kubernetes API call happen, without the model and without the AI tool registry at all. These are two different code paths, not one with a "allowed/forbidden" flag.

One Server, Two Transports

A separate problem arises when an AI assistant is available in two forms: as a built-in panel inside a larger tool, and as a standalone binary for external MCP clients (Claude Desktop, Claude Code, etc.). The temptation is to quickly assemble a separate set of tools for the built-in variant. Over time, the sets diverge, and one accidentally ends up with an extra tool that the other doesn't have.

It's more reliable to run the same MCP server in both cases and communicate with it genuinely through the MCP protocol, just over different transports: in-memory for the built-in variant, stdio for the external client:

server, shutdown := mcpserver.NewServer(ctx)
serverTransport, clientTransport := mcp.NewInMemoryTransports()
server.Connect(ctx, serverTransport, nil)

client := mcp.NewClient(&mcp.Implementation{Name: "desktop-assistant"}, nil)
session, _ := client.Connect(ctx, clientTransport, nil)
list, _ := session.ListTools(ctx, nil) // the same ListTools any external MCP client would call

This means the codebase physically has one server and one list of read-only tools — not an original and a separate copy for the GUI that someone will forget about. In practice, this eliminates exactly one class of bugs: when six months of refactoring later, a tool gets added to one list and forgotten in the other.

Read-Only Doesn't Mean "Nothing Is Visible"

Read-only solves the problem of state mutation, but not the problem of leaking data that's already in the cluster. If your kubeconfig has access to read Secrets in a namespace, the model theoretically can too by calling get_yaml. This needs to be addressed at the data level, not the prompt level: Secret values are redacted before the YAML reaches the tool response —

sec.Data[k] = []byte("<redacted>")

— and the AI (whether the built-in assistant or the MCP client) physically never sees the decrypted value, because it's replaced with a placeholder before the YAML string is even constructed. In this design, it also makes sense to reject apply if <redacted> remains in the YAML — otherwise the placeholder could accidentally overwrite the real value.

What This Approach Doesn't Solve

The model can still read a lot of data you already have RBAC access to. Read-only limits what can be done, not what can be seen within the same permissions.

Load on the API server from a chatty tool-calling loop isn't limited by read-only alone — that requires reasonable measures (a limit on the number of iterations in a conversation, a byte limit on tool results, timeouts on log reads, capped and idle-reaped streams), not cryptographic guarantees.

If keeping prompts and tool outputs on the machine entirely is important, that's a separate configuration (a local model via Ollama/vLLM/LM Studio), not a consequence of read-only architecture by itself.

A compromised registry or poisoned tool descriptions are a separate issue: tool poisoning works against read-only tools too if the model trusts instructions inside a description just as much as the system prompt. A list of thirty read tools doesn't by itself guarantee that each one does exactly what it says — for a detailed breakdown of this topic, see "MCP and Agent Security."

The entire approach described here is simple precisely because it deliberately doesn't solve the more general problem — giving an agent any write access at all. If you really need it (for example, for production debugging with the ability to fix things), that's a fundamentally different, much heavier architecture: whitelisting specific commands, rate-limiting, role-based restrictions, an immutable audit log with alerts.

How Kubernetes Storage Actually Works - A Guide for Sysadmins

Alex Gorshkov — Fri, 07 Aug 2026 22:51:47 +0000

Pods are ephemeral. They restart, get rescheduled, get deleted - and everything written inside the container goes with them. That's fine for a stateless API. It's a disaster for a database, a log directory, or a file upload folder.

Kubernetes solves this with a chain of abstractions that, at first glance, look like bureaucracy: a PVC that asks for a PV, a StorageClass that describes a PV you don't have yet, and a CSI driver that talks to something outside the cluster entirely. Once you see how the pieces hand off to each other, though, the design is straightforward - and, more usefully, so is debugging it.

Here's the whole chain, end to end.

The big picture

The shortest possible summary of Kubernetes storage is: Pod → PVC → CSI → physical disk → PV.

If you follow a request through the system, it travels along a very specific route. First, an application asks for storage by creating a PersistentVolumeClaim (PVC) - essentially saying, "I need 5 GB that survives restarts." Kubernetes inspects this claim and searches for a compatible PersistentVolume (PV) that already exists in the cluster. If a suitable PV isn't sitting idle, Kubernetes doesn't panic; it calls in a CSI driver as an intermediary. The CSI driver translates the request and talks to the actual storage provider - a cloud block-storage API, a SAN, or an NFS appliance - in whatever language that provider speaks. Once the disk physically exists, the cluster binds the newly created PV to the PVC, flipping the PVC status from Pending to Bound. Finally, the pod mounts the volume and gets its own persistent scratchpad for databases, logs, and files.

The key insight that makes this entire architecture click: Kubernetes doesn't store your data. It manages a reliable route to whoever does. Every component below exists to build and maintain one segment of that route.

CSI and StorageClass: who creates the disk

The Container Storage Interface (CSI) is the standard plugin interface that sits between Kubernetes and a storage backend. Kubernetes has no native idea how to carve out a volume on OpenStack Cinder, AWS EBS, or a Ceph cluster, and it doesn't need to. It simply calls the CSI driver and lets the driver deal with vendor-specific API calls. These drivers run as regular pods inside your cluster, and they are typically split into two distinct roles: a Controller plugin, which handles the volume lifecycle (create, delete, resize, snapshot) and runs centrally, and a Node plugin, which handles the messy work of attaching and mounting the volume on a specific host. The Node plugin runs as a DaemonSet, placing one instance on every schedulable node. This split isn't just an implementation detail; it’s the key to debugging. If the disk was never created, you have a Controller problem. If the disk exists but the pod won't start, it's almost certainly a Node problem.

To tell Kubernetes what kind of disk you want, you use a StorageClass. Think of it as a configuration profile for your storage. It specifies the provisioner (which CSI driver to call), the reclaimPolicy (what happens to the disk when the claim is deleted), and the volumeBindingMode. The volumeBindingMode is particularly critical because it dictates the timing of the disk creation. Immediate mode creates the volume as soon as the PVC is created, regardless of whether any pod will ever use it. In a multi-zone cluster, this can be a trap because the volume might be created in an availability zone where the pod can’t run. WaitForFirstConsumer, on the other hand, delays everything until a pod is scheduled. This ensures the disk lands in the same zone as the pod, avoiding a "volume stuck in the wrong zone" headache. The rest of the StorageClass is a set of parameters - options like disk type, IOPS tier, or replication settings - that are passed directly to the CSI driver.

To see what your cluster offers, kubectl get storageclass and kubectl describe storageclass are your starting points. Pay close attention to which class is marked as default. Any PVC that doesn't explicitly name a storageClassName silently uses that default, which is a fast track to accidentally provisioning production data onto a slow, non-replicated tier.

PV and PVC: the claim and the volume

These two objects are easy to mix up conceptually, so it helps to think of them as physical property vs. a ticket. A PersistentVolume (PV) is the actual, provisioned piece of storage - a cluster-level resource, like a node. It's "the disk." A PersistentVolumeClaim (PVC) is a namespaced request for storage with certain properties. It's "the ticket." The PVC specifies the size, access mode, and StorageClass it needs, and if a matching PV exists, Kubernetes binds them together.

Binding isn't a fuzzy match; it requires three conditions to line up exactly. The StorageClass must match, the access modes must be compatible, and the capacity must be sufficient. If the PV is 10 GB and the PVC asks for 5 GB, they bind - but you lose access to the extra 5 GB because a PV can only satisfy a single claim. If any of these conditions fails, the PVC doesn't throw an error and it doesn't retry with looser requirements; it just sits in Pending indefinitely. That silent waiting is the single most common "why won't my pod start" scenario in Kubernetes storage.

You also need to understand the access modes, because they dictate what topologies your pods can use. ReadWriteOnce (RWO) creates a volume that can only be mounted as read/write by a single node. This is a crucial distinction: the lock is per node, not per pod. Multiple pods on the same node can share an RWO volume just fine. However, if you scale a Deployment beyond one replica and the new pod lands on a different node, it will sit in Pending forever because it can't reach the volume. ReadWriteMany (RWX) lifts this restriction, allowing many nodes to mount the volume simultaneously, but this mode is typically only available with shared filesystems like NFS or CephFS, not standard block storage. ReadOnlyMany (ROX) is the read-only counterpart, mountable across many nodes.

When the PVC finally goes away, what happens to your data is dictated by the reclaimPolicy on the StorageClass. The Delete policy cleans up everything - the PV and the underlying disk in your cloud provider - the moment the PVC is removed. It’s convenient for CI and aggressively unforgiving in production. The Retain policy preserves the data, leaving the PV in a Released state. An administrator then has to manually tidy it up or re-attach it, making this the only safe choice for databases or anything containing state you’d be sad to lose. (You might see a Recycle policy in legacy manifests, which wiped the data and put the PV back in the pool, but it’s deprecated in favor of dynamic provisioning).

Broadly, provisioning happens in one of two ways. In Static provisioning, an admin creates PV objects by hand, pointing to pre-existing disks - useful when managing an NFS export or specialized hardware outside the cluster. In Dynamic provisioning, the PVC references a StorageClass, and the CSI driver creates the disk on demand without any pre-made PV objects. This is the default operational mode for practically every modern cluster. When things go wrong, kubectl describe pvc <name> is the most valuable tool you have; the Events section at the bottom usually names the exact reason a binding or provisioning call failed.

Mounting: what happens on the node

Once the binding is resolved, the heavy lifting shifts to kubelet, the agent on each node. When a pod with an attached volume starts, kubelet reads the pod spec, asks the CSI Node driver to attach the disk to the physical server, mounts the storage into the node’s global filesystem, and finally bind-mounts that specific directory into the container's isolated namespace.

Every Kubernetes volume lands under a predictable, well-hidden path on the node: /var/lib/kubelet/pods/<pod-uuid>/volumes/. Memorizing this path pays off when a pod claims its data is gone. You can SSH into the node and check the directory. An empty directory at that path means the mount step failed entirely. A populated directory that the container can’t see points to a problem in the container’s mount propagation settings or a simple typo in the mountPath definition. This distinction saves hours of staring at YAML when you're under pressure.

The pod spec connects everything with two interlocking sections: the volumes block names the source, referencing the PVC by its claim name, and the volumeMounts block inside the container spec declares where that volume appears inside the filesystem tree. The shared name field is the only glue linking the two.

A debugging checklist

When storage misbehaves, walking the chain in order is the fastest way to find the single broken link. Always start by checking if the claim is bound: kubectl get pvc -A and kubectl describe pvc <name>, focusing on the Events section. If the PVC is stuck in Pending, you immediately know the problem is that no matching PV exists, or the CSI Controller can't provision one - so you check the provisioner logs. Next, verify the volume itself with kubectl get pv and kubectl describe pv <name>. A mismatch here means looking at the StorageClass definition. If the PVC is Bound but the pod is stuck in ContainerCreating, the fault has moved to the node; kubectl describe pod <name> will show attach or mount errors in its events, and you should be looking at the CSI Node pod logs on that specific node using kubectl logs. The final fallback is the raw driver diagnostics, found with kubectl -n kube-system get pods | grep csi to locate the controller and node pods, and kubectl -n kube-system logs to see the verbose error from the CSI driver itself.

Wrapping up

The whole system is one sequential handoff: you create a PVC; Kubernetes calls a CSI driver via the StorageClass; the CSI driver creates the real disk on the storage backend; a PV appears and binds to the PVC; the pod mounts the volume and starts writing. Once that chain - Pod → PVC → CSI → external storage → PV - is in your head, the layers stop looking like ceremony. Each object owns exactly one segment of the route, which means every failure has exactly one place to look. And it's worth repeating the point that grounds the entire design: Kubernetes never stores your data. It builds and maintains a reliable path to whatever does.