Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

#aws #kubernetes #security #devops

Kubernetes security failures usually do not look like network intrusions. An attacker who gets code execution inside a container, through an application bug, a poisoned dependency, or a policy gap that let a privileged pod schedule, is already inside the perimeter the firewall was watching. From there the firewall verified nothing: which secrets the process can reach, which syscalls it can make, whether the running image is even the one you signed.

That is the useful reframe. Kubernetes security is not one problem at the boundary. It is three problems inside it, and each one needs its own control that keeps working when the others fail:

Admission decides what is allowed to deploy, and with what configuration.
Identity decides what each running pod is allowed to reach.
Runtime watches what a pod actually does, which is often not what its manifest claimed.

None of the individual tools below are new, and none of this is a new technique. What is worth sharing is how the layers fit together, so a failure in any one of them does not open an exploitable gap. The model maps cleanly onto NIST SP 800-207 (zero trust) and NIST SP 800-190 (container security), and you can hold it to the CIS Kubernetes Benchmark as a measurable floor.

Layer 1: admission, decide what gets in

Start with what Kubernetes gives you for free. Pod Security Admission (PSA) enforces security profiles per namespace with no external tooling. The Restricted profile blocks host namespace sharing, privilege escalation, root contexts, and unsafe seccomp settings. A practical split is Restricted for externally facing namespaces and Baseline for internal platform namespaces with documented exceptions.

PSA is a floor, not a scalpel. It enforces at the namespace level, so it cannot say "this specific Deployment must run as UID 1000 with this seccomp profile." That is where Kyverno comes in. Its policies are plain YAML, they live in the same GitOps repo as your manifests, and they go through the same pull request review as any other change. Four rule types cover most needs:

validate rejects non-compliant resources (for example, require every image to come from an approved ECR registry).
mutate corrects omissions automatically, which turns "you forgot a security context" from an admission failure into a silent fix.
generate creates dependent resources on creation.
verifyImages enforces signature checks at admission time (more on that below).

For constraint logic that depends on the state of other cluster resources, or cross-namespace rules, OPA Gatekeeper and Rego express things Kyverno's YAML cannot. Running both is a reasonable choice: Kyverno for the common case, Gatekeeper where you need Rego's relational power.

One rollout lesson worth stating plainly: both tools have an audit mode, and you should use it first. Measure violations with enforcement off, remediate systematically, then switch to enforce. Flipping straight to enforcement is how a security rollout turns into a production incident.

Layer 2: identity, kill the static credential

A static AWS key in a ConfigMap or an environment variable is a persistent risk with a wide blast radius. IRSA (IAM Roles for Service Accounts) removes the category entirely. The AWS SDK in a pod annotated with an IRSA service account exchanges a short-lived, audience-scoped JWT for temporary IAM credentials through STS. Nothing static is created, stored, or rotated. The IAM trust policy validates the token's subject claim against a specific service account and namespace, so a compromised pod in one namespace cannot assume another namespace's role.

The payoff is blast-radius reduction you can describe concretely. Without IRSA, a compromised pod inherits whatever the node's instance profile can reach. With IRSA, it is limited to that one pod's scoped role.

Application secrets need a parallel mechanism, and that is the ExternalSecrets Operator. It syncs values from AWS Secrets Manager or Parameter Store into Kubernetes Secrets and refreshes them on the source's rotation schedule. When Secrets Manager rotates a database password, the new value propagates without a deployment. IRSA plus ExternalSecrets together removes every static credential: cloud access through IRSA, application secrets through ESO, nothing hardcoded in a Helm values file.

Scaling per-workload IAM roles across hundreds of workloads is the part teams tend to defer. Attribute-based access control with IAM permission boundaries makes it tractable: a role that expresses "this service account, in this namespace, for this environment" can be generated from workload metadata instead of hand-authored each time.

Layer 3: runtime, watch what actually happens

Admission decides what can deploy. Identity decides what it can reach. Neither sees what a pod does after it is running. Falco does, using eBPF probes at the kernel level to observe every syscall from every container on the node. A shell spawned in a container whose image has no shell in its approved process list raises an alert, regardless of how the compromise happened or whether the attacker edited the deployment manifest. Because eBPF sits below the container runtime, it keeps working even if the runtime is tampered with, and it needs no change to the image or the application.

Two practical notes. First, uniform rules across every workload generate false positives that erode trust in the alerts. A data-processing pod that legitimately reads many files will trip rules tuned for an application server. Per-workload rule sets, stored in Git next to the manifests, let you tighten baselines for payment or secret-handling services and loosen them for CI runners. Second, route Falco alerts into a SIEM so individual detections become correlated events. A shell-spawn alert followed within seconds by an outbound connection to an unknown IP is a pattern a correlation engine can act on, including automated network-policy quarantine of the affected pod.

Plan for a calibration period. Default Falco rules produce false-positive volume that takes iterative tuning, with both security and application knowledge, before the layer runs at production-grade sensitivity.

The layer most teams skip: supply chain

An image that passes admission is trusted to be what its manifest says, but only if the reference is cryptographically verified. Tags are mutable. Someone can overwrite the content behind a trusted tag and the cluster will deploy it under the same name, which bypasses every admission policy that checks the image's properties rather than its identity.

Cosign signs images at the registry, and Kyverno verifyImages rules reject anything at admission that is not signed by an approved identity. That closes the tag-mutability gap. SLSA provenance is the next step up: signing proves an image came from your build system, provenance proves what went into that build (source commit, build platform, resolved dependencies). A Software Bill of Materials attached as an attestation then lets you answer "which running images contain this newly disclosed CVE?" by querying the SBOM store instead of inspecting containers one by one.

Putting it together

The point of three independent layers is that no single failure is fatal. A policy gap that lets a bad pod through is still caught by scoped identity and runtime monitoring. A runtime compromise still cannot reach credentials the pod's IAM role never granted. A swapped image is rejected before it schedules.

If you are starting from scratch, a reasonable order is:

Turn on Pod Security Admission at Baseline or Restricted. Free, immediate.
Add Kyverno in audit mode, learn your violations, then enforce.
Move workloads to IRSA and ExternalSecrets to delete static credentials.
Deploy Falco, then budget real time to tune per-workload rules.
Add Cosign signing and verifyImages, then SLSA provenance and SBOMs.

The honest tradeoff is maintenance. Policy-as-code needs upkeep as Kubernetes APIs and workloads change, and Falco rule tuning is ongoing work. The reason to pay that cost is coverage no single control gives you: admission, identity, and runtime failures are different problems, and defending only one of them leaves the other two open.

DEV Community