DEV Community

Cheedge Lee
Cheedge Lee

Posted on • Originally published at notes-renovation.hashnode.dev

CKS Notes - Audit log

#cks #kubernetes #audit #policy

This is a quick notes and summary from office doc.

  1. this is a quick look at log audit.

  2. more detail about the log audit, pls refer to the official doc about audit

  3. and also can check my previous article: kubernets related logs & configurations which shows clear log file paths and the config related files and their locations.

Audit workflow

edit the audit-policy.yaml:

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
Enter fullscreen mode Exit fullscreen mode

add flags to /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=8
    - --audit-log-maxsize=9
    - --audit-log-maxbackup=3
Enter fullscreen mode Exit fullscreen mode

  • --audit-policy-file=/etc/kubernetes/audit-policy.yaml

  • --audit-log-path=/var/log/kubernetes/audit/audit.log

  • --audit-log-maxage=8 : max number of days

  • --audit-log-maxsize=9 : max file size in megabytes before rotate

  • --audit-log-maxbackup=3 : copy of audit log files

edit the volumeMounts

volumeMounts:
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    readOnly: true
  - mountPath: /var/log/kubernetes/audit/
    name: audit-log
    readOnly: false
Enter fullscreen mode Exit fullscreen mode

edit the hostPath corresponding with --audit-policy-file and --audit-log-path

volumes:
- name: audit
  hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: File

- name: audit-log
  hostPath:
    path: /var/log/kubernetes/audit/
    type: DirectoryOrCreate
Enter fullscreen mode Exit fullscreen mode

and then wait for some time and we can check the logs on path /var/log/kubernetes/audit/.

Notice:

in above Volume and VolumeMount, we see some difference settings for audit policy and audit logs.

Audit policy

  • Path is a single YAML file: /etc/kubernetes/audit-policy.yaml

  • readOnly: API server only needs to read it

  • type: File: The file already exists, Kubelet should fail to start the apiserver if the file is missing

Audit logs

  • Path is a directory /var/log/kubernetes/audit/, API server writes out multiple log files (with rotation) audit.log, audit.log.1, etc

  • readOnly: false

  • type: DirectoryOrCreate: It may not exist yet on a fresh system, Kubelet will automatically create it so the API server can write logs

Policy

A Kubernetes audit policy has three important layers:

1. Resources you target 

resources:
  - group: ""    # core API group
    resources: ["pods", "configmaps", "secrets", "persistentvolumeclaims", ...]
Enter fullscreen mode Exit fullscreen mode

group: "" means Core API group, like pods, configmaps, secrets, services, pv, pvc.

but there are also non-core-api

Resource Group
deployments apps
daemonsets apps
jobs batch
ingresses networking.k8s.io
certificatesigningrequests certificates.k8s.io

so the resource eg.

resources:
  - group: "apps"
    resources: ["deployments"]
Enter fullscreen mode Exit fullscreen mode

2. Which stages you log 

stages: ["Request", "Response", "Metadata"]
Enter fullscreen mode Exit fullscreen mode

  • Request: log the incoming request before it hits handlers

  • Response: log the full response (including responseBody if enabled)

  • Metadata: only logs the request metadata (verb, user, resource, namespace)

Normally not used, but we often define omitStages to reduce noise.

omitStages: ["RequestReceived"] is used to: Reduce extremely noisy logs with no loss of useful information.

Stage Meaning
RequestReceived kube-apiserver got the request
ResponseStarted apiserver is sending stream response
ResponseComplete finished sending full response
Panic apiserver crashed during request

3. Log level fields 

level: None | Metadata | Request | RequestResponse
Enter fullscreen mode Exit fullscreen mode

3.1. level: None

No audit event produced, which is used to ignore noisy resources. eg.

- level: None
  resources:
    - group: ""
      resources: ["events"]
Enter fullscreen mode Exit fullscreen mode

level: Metadata

Logs:

  • user (e.g., system:serviceaccount:default:sa)

  • verb (get, list, update, patch…)

  • resource (configmaps, pods…)

  • namespace

  • request URI

  • response code

No request/response bodies. This is lightweight and safe for Secrets.

3.2. level: Request

Logs everything from “Metadata” plus the request body. eg.

  • the new version of a ConfigMap

  • the PVC spec submitted by kubectl

  • the PATCH operations

Does NOT log the response body.

3.3. level: RequestResponse

Logs:

  • metadata

  • request body

  • response body

eg. when we run command kubectl apply -f my-pv.yaml

the audit logs include:

  • the PV YAML sent

  • the PV YAML that apiserver stored after admission

This is the most expensive but also the most complete.

3.4 Summary

Level Logs Notes
None nothing Useful to exclude noisy resources
Metadata verb, user, resource, namespace, stage Very cheap, no bodies
Request request headers + request body More expensive
RequestResponse request + response body Most expensive

Top comments (0)