DEV Community

josepraveen
josepraveen

Posted on

Configure Audit Logging in Kubernetes

#kubernetes #devops #sre #security

Picture this: Your team just wrapped up a security incident simulation. The dust settles, and you huddle up for the postmortem. You ask, "Who modified that deployment?" or "What exact payload was sent to the API?" The room goes silent. You check the logs, only to realize there are no logs of what was done via the Kubernetes API.

Without audit logging, your cluster is essentially a black box. To fix this blind spot, we are going to look at how to implement robust audit policy rules and configure the Kubernetes API server to track threats both in real-time and postmortem.

🧠 Understanding the 4 Audit Levels

Before diving into config files, you need to understand the four Audit Levels in Kubernetes. They dictate the depth of data you collect (and how fast your storage will fill up!):

  • None: Don't log this event at all.
  • Metadata: Log the basics only (who did it, when, from where, and to what resource). It excludes the request and response bodies.
  • Request: Log the Metadata + the exact content/payload sent to the cluster.
  • RequestResponse: Log Metadata + the Request content + the exact response returned by the cluster. This provides total visibility but generates massive amounts of data.

🛠️ Step 1: Crafting the Audit Policy Rules

Let's configure a smart, security-first audit policy. Open your audit policy file:

sudo vi /etc/kubernetes/audit-policy.yaml
Enter fullscreen mode Exit fullscreen mode

Paste the following YAML configuration. This policy is highly optimized to balance deep visibility, disk space saving, and credential security:

# 1. Log request and response bodies for all changes to Namespaces.

  • level: RequestResponse resources:
    • group: "" resources: ["namespaces"]

# 2. Log request bodies (but not response bodies) for changes to Pods and Services in the 'web' Namespace.

  • level: Request resources:
    • group: "" resources: ["pods", "services"] namespaces: ["web"]

# 3. Log metadata ONLY for all changes to Secrets.

  • level: Metadata resources:
    • group: "" resources: ["secrets"]

# 4. Create a catch-all rule to log metadata for all other requests.

  • level: Metadata

💡 Why did we set it up this way?

  • Namespaces (RequestResponse): Creating or deleting a namespace is a major structural change. We want maximum data to see exactly who requested it and what the cluster returned.
  • Pods & Services (Request): These resources change constantly. Logging the full response bodies would fill up your disks rapidly. Request ensures you see what a user tried to do while saving storage.
  • Secrets (Metadata): Crucial security practice! If you use Request or RequestResponse here, your plain-text passwords and sensitive TLS keys will be written directly into your log files. Metadata captures who touched the secret without leaking the secret data itself.
  • The Catch-all (Metadata): Ensures that if someone modifies something else (like a Deployment or ConfigMap), we still capture the baseline who, what, and when.

Save and exit the file (Esc, then :wq, then Enter).

audit policy

⚙️ Step 2: Configuring the API Server

The API Server (kube-apiserver) is the brain of your control plane. Every single command runs through it. To activate our new policy, we need to pass these rules to it.

Open the static pod manifest for the API server:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
Enter fullscreen mode Exit fullscreen mode

Under the - command arguments block, append the following flags:

- command:
  - kube-apiserver
  # ... existing flags ...
  - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
  - --audit-log-path=/var/log/kubernetes/k8s-audit.log
  - --audit-log-maxage=60
  - --audit-log-maxbackup=1
Enter fullscreen mode Exit fullscreen mode

Breakdown of the flags:

  • --audit-policy-file: Tells the API Server where to find the rules we wrote in Step 1.
  • --audit-log-path: Specifies the exact file path on the master node where the JSON logs will be written.
  • --audit-log-maxage=60: Automatically retains old log files for a maximum of 60 days.
  • --audit-log-maxbackup=1: Restricts log rotation to keep only 1 archived backup file, preventing out-of-disk crashes.

Save and exit the file.

kubeapiserver

Because this is a static pod, the kube-apiserver will automatically restart to apply the changes. Give it a minute, then verify the cluster is healthy:

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

🧪 Step 3: Testing the Setup

Let's test our security guardrails by creating a dummy secret and seeing how it logs.

kubectl create secret generic my-secret --from-literal=password=SuperSecret123
Enter fullscreen mode Exit fullscreen mode

Now, check the audit log file:

sudo tail -f /var/log/kubernetes/k8s-audit.log
Enter fullscreen mode Exit fullscreen mode

audit log

Look closely at the JSON block generated for my-secret. Because of our Metadata rule, you will see a trail showing that a secret was created, but you will not see SuperSecret123 anywhere in the logs.

🔑 Key Takeaways

  1. Always implement a Catch-all rule at the end of your policy so nothing slips through.
  2. Never log Request or RequestResponse on Secrets.
  3. Manage your log sizes aggressively using --audit-log-maxage and --audit-log-maxbackup.

Top comments (0)