Hardening the Control Plane: Auditing a Cluster with kube-bench, kubescape and CIS Rules

Most clusters are installed once, pass a smoke test, and are never audited again. Meanwhile the control plane keeps running with whatever flags the installer picked two years ago: no audit log, an API server that still accepts anonymous requests on more endpoints than it should, an etcd data directory readable by anything on the node. None of this shows up in a dashboard. It shows up in a post-incident review.

This article walks through building an automated CIS compliance audit for a Kubernetes or OpenShift cluster using kube-bench and kubescape, interpreting the results without drowning in false positives, fixing the three most common gaps (anonymous-auth, audit logging, etcd), and wiring the whole thing into CI so that drift is caught in hours instead of years.

Context: what the CIS Benchmark actually measures

The CIS Kubernetes Benchmark is a prescriptive list of configuration checks, grouped by component:

• Section 1 — control plane node configuration: kube-apiserver, kube-controller-manager, kube-scheduler flags and file permissions.
• Section 2 — etcd configuration.
• Section 3 — control plane authentication/authorization and logging.
• Section 4 — worker node configuration, mostly kubelet.
• Section 5 — policies: RBAC, service accounts, Pod Security, network policies.

Two things matter before you run anything.

First, the benchmark must match your distribution. The upstream cis-1.x profile assumes kubeadm-style static pod manifests in /etc/kubernetes/manifests. On OpenShift, the control plane is managed by operators, files live elsewhere, and roughly a third of the upstream checks are structurally inapplicable. Red Hat maintains a dedicated benchmark (rh-1.0 / rh-1.6 in kube-bench, ocp4-cis in the Compliance Operator). Running the wrong profile produces a report nobody trusts, and an untrusted report is worse than no report.

Second, a managed control plane is not yours. On EKS, GKE, AKS or ROSA, sections 1 and 2 are the provider's responsibility. Use the eks-1.x, gke-1.x or aks-1.x benchmarks, which skip them and focus on nodes and policies.

Running kube-bench

kube-bench is a single Go binary that reads the running process arguments and the config files on disk, then compares them to the benchmark. It needs to run on the node, with host PID and host paths mounted.

As a one-shot Job on a control plane node

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-master
  namespace: kube-bench
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: kube-bench
    spec:
      hostPID: true
      restartPolicy: Never
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: kube-bench
          image: docker.io/aquasec/kube-bench:v0.10.4
          command: ["kube-bench"]
          args:
            - "run"
            - "--targets=master,etcd,controlplane,policies"
            - "--json"
          securityContext:
            runAsUser: 0
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - { name: var-lib-etcd,  mountPath: /var/lib/etcd,  readOnly: true }
            - { name: var-lib-kubelet, mountPath: /var/lib/kubelet, readOnly: true }
            - { name: etc-kubernetes, mountPath: /etc/kubernetes, readOnly: true }
            - { name: etc-systemd,   mountPath: /etc/systemd,   readOnly: true }
            - { name: usr-bin,       mountPath: /usr/local/mount-from-host/bin, readOnly: true }
      volumes:
        - { name: var-lib-etcd,    hostPath: { path: /var/lib/etcd } }
        - { name: var-lib-kubelet, hostPath: { path: /var/lib/kubelet } }
        - { name: etc-kubernetes,  hostPath: { path: /etc/kubernetes } }
        - { name: etc-systemd,     hostPath: { path: /etc/systemd } }
        - { name: usr-bin,         hostPath: { path: /usr/bin } }

kubectl create namespace kube-bench
kubectl apply -f kube-bench-master.yaml
kubectl wait --for=condition=complete job/kube-bench-master -n kube-bench --timeout=120s
kubectl logs job/kube-bench-master -n kube-bench > kube-bench-master.json

For worker nodes, run the same Job as a DaemonSet with --targets=node and without the control plane nodeSelector.

Making the output usable

Raw kube-bench output is a wall of [PASS] / [FAIL] / [WARN]. Extract only what is actionable:

jq -r '
  .Controls[] |
  .text as $section |
  .tests[].results[] |
  select(.status == "FAIL") |
  [$section, .test_number, .status, .test_desc] | @tsv
' kube-bench-master.json | column -t -s$'\t'

And a one-line verdict for CI:

jq '[.Controls[].total_fail] | add' kube-bench-master.json

kube-bench also has --exit-code 1, which makes the container exit non-zero as soon as one check fails. Use it only once you have triaged your baseline, otherwise every pipeline turns red on day one.

On OpenShift

Force the Red Hat benchmark, because auto-detection on RHCOS is not always reliable:

kube-bench run --benchmark rh-1.0 --targets=master,node,policies --json

But on OpenShift the native path is better: the Compliance Operator runs OpenSCAP-based ocp4-cis profiles, produces ComplianceCheckResult objects in the API, and can auto-remediate through MachineConfig.

apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: cis-compliance
  namespace: openshift-compliance
profiles:
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default
  apiGroup: compliance.openshift.io/v1alpha1
  kind: ScanSetting

oc get compliancescan -n openshift-compliance
oc get compliancecheckresult -n openshift-compliance \
  -l compliance.openshift.io/check-status=FAIL \
  -o custom-columns='NAME:.metadata.name,SEVERITY:.severity'

Adding kubescape for the workload and RBAC side

kube-bench stops at the node. kubescape talks to the API server and evaluates controls across RBAC, Pod Security, network policies, image provenance and secrets — exactly the part of the CIS benchmark (section 5) that kube-bench marks as "manual".

# list available frameworks for your kubescape version
kubescape list frameworks

# CIS scan, JSON + human output, fail the shell on low compliance
kubescape scan framework cis-v1.23-t1.0.1 \
  --format json --output kubescape-cis.json \
  --compliance-threshold 80 \
  --severity-threshold high \
  --verbose

Two flags do the heavy lifting in CI:

• --compliance-threshold 80 — exit non-zero if the overall compliance score drops below 80%.
• --severity-threshold high — exit non-zero if any High/Critical control fails, regardless of score.

Legitimate deviations go into an exceptions file instead of being silently ignored:

[
  {
    "name": "allow-host-path-for-csi-driver",
    "policyType": "postureExceptionPolicy",
    "actions": ["alertOnly"],
    "resources": [
      { "designatorType": "Attributes",
        "attributes": { "namespace": "openshift-storage" } }
    ],
    "posturePolicies": [ { "controlID": "C-0048" } ]
  }
]

kubescape scan framework cis-v1.23-t1.0.1 \
  --exceptions ./exceptions.json --format sarif --output kubescape.sarif

The SARIF output drops straight into GitHub code scanning; --format junit plugs into GitLab or Jenkins test reporting.

Fixing the three failures you will always see

1. --anonymous-auth=true on the API server

CIS 1.2.1 wants --anonymous-auth=false on kube-apiserver. Before you change it, understand the trade-off: anonymous requests are mapped to the user system:anonymous in group system:unauthenticated, and many load balancers and monitoring probes hit /livez, /readyz or /healthz without credentials. Setting the flag to false on a kubeadm cluster will break those probes unless you reconfigure them.

The real risk is not anonymous authentication itself — it is anonymous authentication combined with a permissive RBAC binding. Audit that first:

kubectl get clusterrolebindings,rolebindings -A -o json | jq -r '
  .items[] |
  select(.subjects[]? | .name == "system:unauthenticated" or .name == "system:anonymous") |
  "\(.kind)/\(.metadata.name) -> \(.roleRef.kind)/\(.roleRef.name)"'

Anything other than system:public-info-viewer deserves an explanation. On kubeadm, apply the flag in the static pod manifest (the kubelet restarts the pod automatically):

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --anonymous-auth=false
        - --authorization-mode=Node,RBAC
        - --profiling=false

On OpenShift you do not edit these manifests. The API server is rendered by the kube-apiserver-operator, and anonymous access is already constrained by the default RBAC. Treat kube-bench's finding as informational and verify the bindings instead.

The kubelet-side check (CIS 4.2.1) is the one you should always enforce:

# /var/lib/kubelet/config.yaml (excerpt)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook
readOnlyPort: 0

2. No audit log

CIS 1.2.19–1.2.23 require an audit policy and retention. Without it you have no record of who read which Secret. A balanced starting policy:

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # Never log request/response bodies of secret-bearing resources
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]

  # Drop high-volume, low-value read traffic
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get", "list", "watch"]
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version", "/metrics"]

  # Full payload on RBAC and admission changes
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
      - group: "admissionregistration.k8s.io"
      - group: "policy"

  # Everything else: metadata
  - level: Metadata

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
        - --audit-log-path=/var/log/kubernetes/audit/audit.log
        - --audit-log-maxage=30
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
      volumeMounts:
        - { name: audit-policy, mountPath: /etc/kubernetes/audit-policy.yaml, readOnly: true }
        - { name: audit-log, mountPath: /var/log/kubernetes/audit }
  volumes:
    - name: audit-policy
      hostPath: { path: /etc/kubernetes/audit-policy.yaml, type: File }
    - name: audit-log
      hostPath: { path: /var/log/kubernetes/audit, type: DirectoryOrCreate }

On OpenShift, audit logging is on by default and you select a profile declaratively:

oc patch apiserver cluster --type=merge \
  -p '{"spec":{"audit":{"profile":"WriteRequestBodies"}}}'

# logs are already collected per control plane node
oc adm node-logs --role=master --path=kube-apiserver/ | head

3. etcd exposure

CIS section 2 checks that etcd requires client certificates, uses peer TLS, and that /var/lib/etcd is 0700 and owned by etcd:etcd. Verify on a control plane node:

stat -c "%a %U:%G %n" /var/lib/etcd
ps -ef | grep -o -- '--client-cert-auth=[a-z]*\|--peer-client-cert-auth=[a-z]*\|--auto-tls=[a-z]*'

The check kube-bench cannot do for you is encryption at rest (CIS 1.2.31/1.2.34). Secrets are stored in etcd base64-encoded, not encrypted, unless you configure a provider:

# /etc/kubernetes/enc/encryption-config.yaml  -- never commit this file
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets", "configmaps"]
    providers:
      - aescbc:
          keys:
            - name: key-2024-06
              secret: <BASE64_32_BYTE_KEY>   # head -c 32 /dev/urandom | base64
      - identity: {}

Add --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml to the API server, then rewrite existing objects:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

On OpenShift this is a single field, and the operator handles key rotation:

oc patch apiserver cluster --type=merge -p '{"spec":{"encryption":{"type":"aescbc"}}}'
oc get apiserver cluster -o jsonpath='{.status.conditions[?(@.type=="Encrypted")].reason}{"\n"}'

Wiring it into CI

Run kubescape on every merge request (it only needs API access) and kube-bench nightly as a CronJob (it needs node access).

# .gitlab-ci.yml
cluster-compliance:
  stage: audit
  image: quay.io/kubescape/kubescape-cli:v3.0.8
  script:
    - kubescape scan framework cis-v1.23-t1.0.1
        --exceptions ./compliance/exceptions.json
        --compliance-threshold 80
        --severity-threshold high
        --format junit --output kubescape-junit.xml
        --format json  --output kubescape.json
  artifacts:
    when: always
    reports:
      junit: kubescape-junit.xml
    paths: [kubescape.json]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_PIPELINE_SOURCE == "schedule"

The nightly kube-bench run, with results pushed to a bucket and a summary into the daily report:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench-nightly
  namespace: kube-bench
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          restartPolicy: Never
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
            - { key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }
          containers:
            - name: kube-bench
              image: docker.io/aquasec/kube-bench:v0.10.4
              args: ["run", "--targets=master,etcd,controlplane,policies",
                     "--json", "--outputfile=/reports/kube-bench-$(NODE_NAME).json"]
              env:
                - name: NODE_NAME
                  valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
              volumeMounts:
                - { name: reports, mountPath: /reports }
                - { name: etc-kubernetes, mountPath: /etc/kubernetes, readOnly: true }
                - { name: var-lib-etcd, mountPath: /var/lib/etcd, readOnly: true }
          volumes:
            - { name: reports, persistentVolumeClaim: { claimName: compliance-reports } }
            - { name: etc-kubernetes, hostPath: { path: /etc/kubernetes } }
            - { name: var-lib-etcd, hostPath: { path: /var/lib/etcd } }

A readable Markdown report for the weekly review:

{
  echo "# CIS audit — $(date -I) — cluster $(kubectl config current-context)"
  echo
  echo "| Check | Severity | Description | Remediation |"
  echo "|---|---|---|---|"
  jq -r '.Controls[].tests[].results[]
    | select(.status=="FAIL")
    | "| \(.test_number) | \(.scored // false | if . then "scored" else "not scored" end) | \(.test_desc) | \(.remediation | gsub("\n"; " ")) |"' \
    kube-bench-master.json
} > report-cis.md

Security best practices

• Baseline first, gate second. Capture the current failure list, file it as exceptions with an owner and an expiry date, then make CI fail on anything new. Gating on absolute compliance from day one just teaches people to add || true.
• Pin the benchmark version in your pipeline (cis-v1.23-t1.0.1, rh-1.0, v0.10.4). A benchmark bump is a change you review, not a surprise red build.
• Never auto-remediate the control plane. kube-bench remediations edit static pod manifests; a bad flag means an API server that does not come back. Apply through your installer (kubeadm config, MachineConfig, Cluster API) and roll one node at a time.
• Ship audit logs off the node. A log sitting on a compromised control plane node is evidence an attacker can delete. Forward to a write-once destination and alert on secrets access by non-system identities.
• Keep the identity provider last in EncryptionConfiguration during migration, and rotate keys by prepending a new one and re-running the kubectl replace pass.
• Audit the auditor. kube-bench runs as root with hostPID and host mounts. Scope it to a dedicated namespace, a dedicated ServiceAccount, and a restrictive Pod Security / SCC exception — not a blanket privileged grant.
• Correlate with runtime. A passing CIS score with a cluster-admin ServiceAccount token mounted in a public-facing pod is still a breach waiting to happen. Section 5 controls in kubescape are the ones that catch this.

Key takeaways

Control plane hardening is not a one-off project; it is a measurement you take every night. kube-bench tells you what the node configuration looks like, kubescape tells you what the API objects allow, and the two together cover the CIS benchmark end to end. On OpenShift, let the Compliance Operator own sections 1–4 and use kubescape for the workload layer.

Start with the three findings that appear on nearly every cluster: review who is bound to system:unauthenticated, turn on a real audit policy with retention, and encrypt Secrets at rest while locking down etcd's files and client certificates. Then convert the remaining failures into a tracked exception list, set --compliance-threshold and --severity-threshold in CI, and archive the JSON so you can show a trend line rather than a snapshot. A compliance report that nobody reads is an artifact; a compliance report that fails a pipeline is a control.