DEV Community

kozhevniko
kozhevniko

Posted on

Kubernetes audit webhooks need a backpressure and evidence plan

Kubernetes audit webhooks need a backpressure and evidence plan

A Kubernetes audit webhook is part of the evidence pipeline. The kube-apiserver can write audit events to files, local logs, or a remote webhook. The webhook backend sends events to a remote API and can batch them. The mode determines how much the API server prioritizes the audit write over the request path.

The default webhook mode is batch. In batch mode, the API server buffers events and writes them asynchronously. Kubernetes documents that events are dropped when the incoming event rate overflows the buffer. Throttling and batch size also affect how quickly events leave the buffer. A collector outage or a sudden increase in audited request rate can therefore produce a gap that is not visible from the audit policy alone.

The blocking modes change the tradeoff. blocking waits for individual event processing before the API server responds. blocking-strict does the same and fails the whole request when audit logging fails at the RequestReceived stage. A strict mode can make the audit backend part of the API server's availability boundary. Choosing it without sizing the backend can turn an observability outage into a control-plane outage.

Define the evidence requirement

Not every request needs the same guarantee. Security-relevant changes to authentication, authorization, workload admission, and secret access may need durable delivery. High-volume read traffic may be sampled or sent to a separate backend. The audit policy's rules, stages, and levels define what is generated. The backend mode defines how the generated events are buffered and written.

Use a local audit log as an independent copy when the remote collector is not trusted to survive load. The local backend has its own mode and retention settings, and it can be placed on a separate disk or log path. The two backends should not share a failure domain if the goal is evidence continuity. Encrypt or protect the local files with the same care used for remote transport.

Record the effective flags for webhook mode, buffer size, batch size, maximum wait, throttle QPS, throttle burst, and truncation. Compare the values across control-plane replicas. A cluster with different settings on each API server can produce inconsistent delivery during a rollout.

Test the collector failure

Use a test cluster or an isolated control-plane instance. Stop the collector, generate a known number of audited requests, and observe which events arrive after recovery. Confirm the documented buffer behavior and the time to drop. Repeat with the collector responding slowly, returning an error, and returning a malformed response. Capture only counts and status information, not request bodies or credentials.

Check the effect on the API server. In batch mode, verify that ordinary requests continue while events queue. In blocking-strict mode, verify that the intended request fails when the RequestReceived stage cannot be written. Then restart the collector and confirm delivery resumes without duplicate records being treated as new security events.

Size the pipeline from the observed audit event rate. Count each configured stage, not only each request. The Kubernetes documentation gives an example where auditing ResponseStarted and ResponseComplete produces roughly twice as many events as requests. Use the cluster's real peak rate and add margin for incident response, controller retries, and bulk changes.

Protect the audit channel

Authenticate the webhook endpoint and protect the connection with TLS. Restrict which workloads and networks can reach the collector. Validate the webhook configuration so that a compromised collector cannot redirect events or accept a weaker protocol than the platform allows. Keep the kubeconfig or certificate material outside public repositories.

Audit the audit configuration itself. Changes to the audit policy, webhook URL, buffer, mode, and truncation settings can remove evidence without changing application manifests. Use a protected configuration path, change review, and a startup check that fails when the required backend is missing.

Have a second detection path for control-plane actions that does not depend solely on the same audit pipeline. Cloud control-plane logs, node logs, and application authorization logs can provide corroboration. Retention and access control need to cover the full investigation window.

An audit webhook is not a fire-and-forget log sink. Its mode and buffer determine what happens under pressure, and its collector determines whether the evidence survives. Define the requirement, test the failure path, and monitor the gap.

References

Top comments (1)

Collapse
 
macro8n profile image
macro8n | Desktop Automation •

Good plan on failure-domain separation. One extra signal I’d expose is the oldest buffered-event age alongside drop counters; a synthetic event that must reach durable storage can test the full chain. That catches a collector that returns healthy while evidence is still stalled.