DEV Community: kubernetes

CloudNativePG: Running PostgreSQL in Kubernetes Without the Pain

Guatu — Tue, 16 Jun 2026 00:15:32 +0000

A CloudNativePG cluster that sits in Setting up primary forever, with zero error events on the Cluster resource and a perfectly healthy operator, is one of the more frustrating ways to spend an afternoon. The operator says it's working. The pods never appear. And the actual cause has nothing to do with the database at all.

Running stateful databases on Kubernetes used to be the thing everyone told you not to do. CloudNativePG (CNPG) changed that calculus for a lot of people, including me. It's a proper operator: it handles failover, backups, connection routing, and rolling upgrades through native Kubernetes primitives instead of bolting Postgres onto a StatefulSet and praying. If you run a hardened cluster with admission controllers, network policies, and least-privilege RBAC, this post is about the friction you'll hit that the quickstart never mentions.

Who should care

If your cluster is vanilla, kubectl apply the operator and a Cluster manifest, and you're done in ten minutes. The CNPG docs are genuinely good for that path. This is for the rest of us: people running Kyverno or OPA Gatekeeper, self-signed cert chains, and the kind of policy-as-code setup where every workload has to justify its existence. That's where CNPG stops being a ten-minute install and starts being an integration project.

What I tried first

The first instinct, when a CNPG cluster hangs, is to assume you got the database config wrong. So you go read your Cluster manifest line by line. You check the storage class. You check that the PVC bound. You bump the operator log level and watch it cheerfully report that it's reconciling, over and over, with no complaints.

Here's the trap: the CNPG operator doesn't run initdb itself. It creates a Kubernetes Job to bootstrap the primary. That Job spawns a Pod. And in a hardened cluster, the Pod is where everything dies, because your admission controller is judging it against policies the operator's own Pods were exempted from but the bootstrap Job was not.

The mistake I see constantly is reading the wrong resource. People kubectl describe cluster and kubectl describe pod on the operator, find nothing, and conclude CNPG is broken. The events you need are on the Job and on the Pod the Job tries to create. A blocked Pod creation shows up as an event on the Job's owning controller, not on the Cluster:

# The Cluster looks stuck here, but says nothing useful
kubectl get cluster -n databases
# NAME       AGE   INSTANCES   READY   STATUS                    PRIMARY
# pg-main    8m    3           0       Setting up primary

# The real story is on the bootstrap Job's events
kubectl describe job -n databases pg-main-1-initdb

If a policy is the culprit, that describe output is where you'll finally see something like admission webhook "validate.kyverno.svc" denied the request: validation error: every container must define resource limits. The bootstrap Job's Pod template didn't set CPU/memory limits, your require-resource-limits policy rejected it, and the operator quietly retries forever because, from its perspective, it asked Kubernetes nicely and Kubernetes said no.

I spent longer than I'd like to admit assuming the storage layer was at fault before I went and looked at the Job. The lesson stuck: when an operator hangs, find the resource the operator creates, not the resource it manages.

The actual solution

1. Exempt CNPG lifecycle resources from blocking policies

CNPG generates Jobs and Pods on your behalf, and you can't directly edit their pod templates the way you would a Deployment you wrote. So the fix isn't to add resource limits to the Job. It's to teach your policy engine that CNPG-owned resources are allowed to skip the rule that's blocking them.

Every resource CNPG creates carries the cnpg.io/cluster label. That label is your exclusion key. For Kyverno, add an exclude block to the rule that's firing:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              # CNPG-managed Pods (instances + bootstrap Jobs) carry this label
              selector:
                matchLabels:
                  cnpg.io/cluster: "*"
      validate:
        message: "Every container must define CPU and memory limits."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

This is a deliberately narrow exclusion. You're not disabling the policy. You're carving out resources that match a specific operator-owned label, which means a developer can't accidentally smuggle a limitless Pod past the gate by slapping a random label on it. If you want to be stricter, scope the exclusion to the databases namespace as well so the label only grants an exemption where CNPG is actually allowed to run.

The same idea applies to OPA Gatekeeper, just expressed differently: add the label to the constraint's match.excludedNamespaces or write a labelSelector exclusion in the constraint spec. The principle doesn't change. Match the operator's label, exempt the lifecycle resources, leave everything else under enforcement. I wrote about the general shape of this in Kyverno Admission Controllers: Policy-as-Code That Actually Works, and CNPG's initdb Job is the cleanest real-world example I've found of policy breaking infrastructure in a way that's invisible until you know where to look.

2. Give the operator the RBAC it actually needs

If you provision service accounts by hand instead of trusting the operator's defaults, remember that CNPG needs to manage Jobs, Pods, PVCs, Secrets, and Services on your behalf. A read-only or overly-scoped account will fail in the same silent way a policy block does: the reconcile loop runs, the create call gets a 403, and nothing visible happens.

The operator's ClusterRole covers this out of the box. If you're tightening it, the non-obvious permissions are the ability to create and delete Jobs (for initdb and restores) and to manage PVCs (for volume expansion and replica provisioning). Strip those and your cluster bootstraps fine until the first time it needs to scale or recover, then breaks. I go deeper on scoping accounts like this in Kubernetes RBAC: Building Least-Privilege Service Accounts.

3. Pin your PostgreSQL minor version away from 16.4

There's a known regression in PostgreSQL 16.4 where the server can hit a segmentation fault under certain memory conditions on nodes with large amounts of RAM available. If you're running CNPG on beefy worker nodes (16GB+ of available memory is the trigger zone), this is exactly the kind of thing that looks like a CNPG bug, a storage bug, or a kernel OOM, when it's actually upstream Postgres.

The fix is boring and effective: pin the image to a known-good minor and don't float the tag.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-main
  namespace: databases
spec:
  instances: 3
  # Pin explicitly. Do not use a floating major-version tag in production.
  imageName: ghcr.io/cloudnative-pg/postgresql:16.6
  storage:
    size: 20Gi
    storageClass: longhorn
  resources:
    requests:
      memory: "2Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1"

Note the memory requests and limits are set to the same value. For a database, you almost never want Postgres getting throttled or evicted because a noisy neighbor ballooned and the scheduler decided your requests were a polite suggestion. Equal requests and limits put the Pod in the Guaranteed QoS class, which is what you want for a stateful workload you can't afford to lose to memory pressure.

4. Understand the three Services CNPG hands you

This is the part that pays off long after install. For a cluster named pg-main, CNPG creates a set of Services automatically, and each one routes to a different role:

Service	Routes to	Use it for
`pg-main-rw`	Current primary	Writes, migrations, anything that mutates
`pg-main-ro`	Replicas only	Read-only queries, reporting, analytics
`pg-main-r`	Any instance (primary or replica)	Reads where you don't care which node

The -rw Service is the important one: when CNPG fails over, it repoints -rw at the new primary. Your application doesn't need to know a failover happened. It keeps connecting to pg-main-rw.databases.svc.cluster.local and the operator handles the rest. That's the entire value proposition of running Postgres under an operator instead of as a hand-rolled StatefulSet.

For read/write splitting, point your app at two connection strings instead of one. Most ORMs and connection libraries support a primary/replica config:

# In your app's config or Secret
env:
  - name: DATABASE_URL_PRIMARY
    value: "postgresql://app:$(PGPASSWORD)@pg-main-rw.databases.svc.cluster.local:5432/appdb"
  - name: DATABASE_URL_REPLICA
    value: "postgresql://app:$(PGPASSWORD)@pg-main-ro.databases.svc.cluster.local:5432/appdb"

Send SELECTs that tolerate slight replication lag to -ro, and send everything else to -rw. The catch worth stating plainly: replicas are asynchronous by default, so a read immediately after a write can return stale data. If you need read-your-writes consistency for a given query, send it to -rw. Don't blanket-route all reads to replicas and then act surprised when a user doesn't see the row they just created.

5. Connection SSL: the untrusted-certificate wall

CNPG enables TLS by default and issues its own certificates through an internal CA. That's good for in-cluster security and annoying the first time a client refuses to connect because it doesn't trust the CA.

The error you'll see from a client is some flavor of SSL error: certificate verify failed or self-signed certificate in certificate chain. The wrong reaction is to globally disable TLS on the cluster. The right reaction depends on who's connecting:

# In-cluster clients: trust CNPG's CA. The operator publishes it as a Secret.
kubectl get secret pg-main-ca -n databases -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
# Then point the client at it:
# postgresql://...?sslmode=verify-full&sslrootcert=/etc/pg/ca.crt

For clients that genuinely can't do certificate verification (some managed platforms and serverless backends only support a binary "SSL on/off" toggle and can't be handed a custom CA), you have two honest options. Either set sslmode=require on the client, which encrypts the connection but skips CA verification, or terminate trust at a proxy you control. sslmode=require is the pragmatic middle ground: you keep encryption in transit and drop only the identity check. It's not as strong as verify-full, but it's a deliberate, documented tradeoff rather than turning TLS off entirely.

Here's the quick reference I keep around for the sslmode ladder:

`sslmode`	Encrypted?	Verifies CA?	Verifies hostname?
`disable`	No	No	No
`require`	Yes	No	No
`verify-ca`	Yes	Yes	No
`verify-full`	Yes	Yes	Yes

Aim for verify-full for anything in-cluster, where you control the CA distribution. Drop to require only for external clients that can't be handed the CA, and never to disable. If you're already running cluster-wide TLS automation, the CA-distribution problem is the same one cert-manager solves for ingress; I covered that workflow in cert-manager + Cloudflare DNS-01: Automated TLS for Everything.

6. Exposing pgAdmin without poking a hole in the cluster

You'll eventually want a GUI to poke at the database. The pattern I'd reach for is pgAdmin4 in its own namespace, reachable through your existing ingress controller, never exposed directly. Keep it in a separate namespace from the database so your network policies can treat it as an external-ish client that's explicitly allowed to reach the -rw/-ro Services, rather than something that lives inside the data tier.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin
  namespace: pgadmin
  annotations:
    # Force HTTPS and lean on cert-manager for the cert
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # pgAdmin needs a bigger body size for imports/exports
    nginx.ingress.kubernetes.io/proxy-body-size: "16m"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["pgadmin.example.com"]
      secretName: pgadmin-tls
  rules:
    - host: pgadmin.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pgadmin
                port:
                  number: 80

Put authentication in front of it. pgAdmin's own login is fine, but I'd add an ingress-level auth layer (OAuth proxy or basic auth) so a leaked pgAdmin password isn't a direct line to your database. And lock down the NetworkPolicy so only the pgAdmin namespace can reach the database Services. A database admin GUI on the public internet with default credentials is how clusters become someone else's crypto miner.

Why it works

The thing that finally made CNPG click for me is that it's not pretending Postgres is stateless. It embraces the fact that a database has a primary and replicas, that failover is a real event, and that bootstrapping is a one-time Job rather than a steady-state process. Every piece of the design maps a Postgres concept onto a native Kubernetes object you can inspect with kubectl.

That's also why the failure modes are sneaky. The operator delegates the actual work to Jobs and Pods, so when an admission controller or RBAC rule blocks one of those, the operator has no good way to surface it beyond a stalled status. There's no exception thrown into your terminal. The reconcile loop is doing exactly what it's designed to do, which is keep trying, and "keep trying against a wall" looks identical to "working" until you go read the Job's events.

The Service abstraction works because CNPG owns the failover decision and the endpoint update atomically. When it promotes a replica, it updates the -rw Service's selector in the same control loop. There's no DNS TTL to wait out, no client-side failover logic to get wrong, no floating VIP to manage. Kubernetes Service routing was already solving "send traffic to whichever Pod currently has this role," and CNPG just plugs the primary/replica roles into that existing machinery. Running databases reliably on Kubernetes is the kind of platform-engineering work that separates a homelab toy from production infrastructure, and it's a chunk of what I do in consulting engagements.

Lessons learned

The biggest shift was learning to debug the resources the operator creates, not the ones it manages. kubectl describe cluster will lie to you by omission. The Job and its Pod tell the truth. If a CNPG cluster hangs in Setting up primary, my first move now is straight to the bootstrap Job's events, and nine times out of ten it's a policy or RBAC denial, not a database problem.

What surprised me was how much the hardened-cluster setup matters. Every CNPG tutorial assumes a permissive cluster, so the exact features that make a cluster production-grade (enforced resource limits, least-privilege RBAC, default-deny network policies) are the features that break the install. None of them are CNPG's fault. They're the cost of doing security right, and the fix is always a narrow, labeled exclusion rather than a blanket exception. If you run CNPG via GitOps, put those policy exclusions in the same ArgoCD app as the operator so they're never out of sync; the App-of-Apps pattern handles this cleanly.

If I were starting over, I'd pin the PostgreSQL minor version from day one and treat floating tags as a production smell, set Guaranteed QoS on the database Pods before the first incident rather than after, and write the read/write split into the application from the start instead of routing everything at the primary and refactoring later. None of those are hard. They're just the kind of decision that's cheap to make early and expensive to retrofit once you have data and uptime to protect.

CNPG genuinely delivers on running Postgres in Kubernetes without the pain, but only if you account for the cluster you actually have, not the empty one the docs assume. The operator is excellent. The integration with your security posture is the part you own.

From Pull Request to Preview Environment: How We Built Disposable Cloud Environments at RTL

Rajesh Gunasekaran — Mon, 15 Jun 2026 21:36:40 +0000

Introduction

The same developer conversation kept coming up on the platform team: "I can't really verify this Pull Request (PR) without merging it." Local dev handled simple cases, but the moment a change touched a real cloud resource (a database, a queue, an object store, an external API) the only honest test was to merge and watch.

This post is how we replaced that with on-demand ephemeral environments, one per PR: fully provisioned from cloud resources down to running Pods, and torn down automatically when the PR closes or merges. It's built on two open-source tools that compose well: ArgoCD's ApplicationSet PR generator and Crossplane.

Why per-PR environments?

Shared "dev" or "staging" environments fail in two ways:

Contention: two PRs touch the same shared queue or schema, and neither developer can tell whose change broke what.
Drift: what you tested in staging isn't what reaches production, because ten more PRs landed before yours got promoted.

The fix: give every PR its own slice of the world. Open a PR with a preview label, a fresh namespace appears, the PR's image is deployed, backing infra is provisioned, a preview URL goes live. Merge or close the PR, and the whole slice disappears.

That sounds expensive. With Crossplane + ArgoCD it isn't, because everything (manifests and cloud resources) lives behind the same GitOps loop. No separate Terraform pipeline. ArgoCD watches PRs; Crossplane watches Custom Resources; the rest follows.

The architecture, end-to-end

Three zones: Source Code Management (SCM), the Kubernetes cluster, and the cloud. Two signals connect them; the PR generator polling SCM, and Crossplane authenticating to the cloud via Workload Identity Federation (WIF).

Every arrow in this diagram is triggered by one event: a PR opening, updating, or closing in GitHub. Start there, and follow the arrows.

The building blocks

Component	What it does
ArgoCD ApplicationSet with PR generator	Watches PRs in source repos. Emits one `Application` per matching PR, parameterised by PR number / branch / labels.
Helm chart (tenant)	Renders workload manifests for one PR: `Deployment`, `Service`, `Ingress`, plus Crossplane `Claim`s for backing infra.
Crossplane	Reconciles Claims into real cloud resources via providers for your cloud.
Kyverno	Admission webhook that stamps every Crossplane Managed Resource at the point of creation, injects the correct `providerConfigRef` and `resourceGroupName` from namespace labels. Tenant chart authors never write (or hold) platform credentials.
Workload Identity Federation	Lets the Crossplane provider authenticate to the cloud without a static credential. The provider's Kubernetes ServiceAccount is federated to a cloud Managed Identity via the cluster's OIDC issuer URL. No secret, no rotation, no leak surface.

ArgoCD never talks to the cloud directly. It deploys the chart, the chart contains a DatabaseClaim, Crossplane provisions the database. Both directions are GitOps-pure.

ApplicationSet + PR generator

A sanitised tenant ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: tenant-a-previews, namespace: argocd }
spec:
  goTemplate: true
  generators:
    - pullRequest:
        github:
          owner: tenant-a
          repo: tenant-a-app
          tokenRef: { secretName: github-token-tenant-a, key: token }
        filters: [{ labels: ["preview"] }]
        requeueAfterSeconds: 60
  template:
    metadata: { name: 'tenant-a-pr-{{ .number }}' }
    spec:
      project: tenant-a
      source:
        repoURL: https://github.com/platform/tenant-charts.git
        path: charts/tenant-a-preview
        targetRevision: main
        helm:
          parameters:
            - { name: pr.number, value: '{{ .number }}' }
            - { name: pr.branch, value: '{{ .branch }}' }
      destination: { server: https://kubernetes.default.svc, namespace: 'tenant-a-pr-{{ .number }}' }
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

A few things worth noting:

Two SCMs, one architecture. The PR generator supports GitHub and Azure DevOps (plus GitLab, Bitbucket). Tenant onboarding values declare scm.provider: github | azuredevops and the rest is derived.
Token storage. The SCM token never lives in values.yaml. We sync a Personal Access Token (PAT) from a cloud Key Vault into the cluster via a vault-sync operator, and the ApplicationSet's tokenRef points at the resulting Kubernetes Secret.
requeueAfterSeconds. Default is 30 minutes, unusable for developer experience. Drop it to 60 seconds.
Filter on a label, not every PR. Tagging the PR preview keeps the noise down for repos where most PRs don't need a preview.

Crossplane — the cloud resources half

ArgoCD gets the Pod running. The Pod usually wants a database, a queue, an object store. Without Crossplane, your options are: (a) one giant shared DB with per-PR schemas (contention, drift, awful teardown), or (b) a separate Terraform pipeline per PR (slow, asynchronous, two tools to reason about).

Crossplane gives you a third: the cloud resource is a Kubernetes object. The tenant chart contains:

apiVersion: db.platform.example.io/v1alpha1
kind: PostgresClaim
metadata: { name: pr-{{ .Values.pr.number }}-db }
spec:
  compositionRef: { name: postgres-small }
  parameters: { storageGB: 10, version: "16" }
  writeConnectionSecretToRef: { name: db-connection }

A platform-owned Composition translates that into real cloud resources via Crossplane's provider for your cloud. The connection Secret lands in the PR's namespace; the Deployment mounts it.

Close the PR → ArgoCD prunes the Application → the Claim disappears → Crossplane garbage-collects the cloud resource. No orphaned databases.

Habits we picked up:

Compositions are platform-owned, Claims are tenant-owned. Tenants pick from a curated menu (postgres-small, queue-default, storage-bucket).
Set deletionPolicy: Delete explicitly. Some providers default to Orphan, which is the wrong default for ephemeral envs.
Tag everything with the PR number so every resource in the cloud console is traceable back to the PR that created it.

The lifecycle of a preview PR

Here's what happens when a developer opens a labelled PR, and what gets cleaned up when they close it:

Open. A developer opens a PR and adds the preview label. The ApplicationSet PR generator polls SCM every 60 seconds, sees the new labelled PR, and emits one Application (tenant-pr-N). That Application renders the tenant Helm chart at the PR's exact commit into a fresh namespace, covering both the workload manifests and the Crossplane Claims.

Provision + serve. Crossplane reconciles the Claims into real cloud resources and writes a connection Secret into the namespace. The Pod pulls the pr-N image, mounts that Secret, and the preview URL goes live. (How each Claim is routed to the right tenant identity and Resource Group is the fence; see Stamping the fence at admission below.)

Iterate. Every push updates the generated revision; ArgoCD re-syncs, so the preview always tracks the latest commit. No manual redeploy.

Close. Merge or close the PR and it drops off the generator's list on the next 60-second poll. ArgoCD removes the Application and prunes the namespace; the Claim disappears and Crossplane garbage-collects the cloud resource. The whole slice is gone, no orphaned infra to bill for.

The teardown has ordering subtleties (what must drain before what, and what happens when a finalizer gets stuck) but those are operational lessons, so they live in the companion post.

The multi-tenant pattern

One tenant module onboards a team in a couple of lines of values:

tenants:
  - name: tenant-a
    scm:
      provider: github
      github: { owner: tenant-a, repo: tenant-a-app, keyvaultKey: tenant-a-pat }
    chart: { path: charts/tenant-a-preview }
    composition: postgres-small
  - name: tenant-b
    scm:
      provider: azuredevops
      azuredevops: { organization: contoso, project: TenantB, repo: tenant-b-app, keyvaultKey: tenant-b-pat }
    chart: { path: charts/tenant-b-preview }
    composition: queue-default

The module renders the per-tenant ApplicationSet, an AppProject for isolation, the secret wiring, and default RBAC. Adding a new team is a five-line PR. Adoption becomes a conversation about fit, not effort.

Stamping the fence at admission

The values block above declares which Resource Group (RG) and which Managed Identity (MI) belong to a tenant. By itself, that fences nothing. A team's helm chart that writes its own providerConfigRef could route a PostgresClaim at another team's identity. A team's chart that picks any resourceGroupName could pour resources into the wrong RG. Values are not policy.

What actually fences the tenant is a three-layer arrangement:

ArgoCD owns the namespace labels. Every per-PR namespace is born with platform.example.io/tenant: <team>, platform.example.io/tenant-rg: <team-rg>, and platform.example.io/tenant-scope: pr-preview, set via managedNamespaceMetadata on the per-tenant ApplicationSet. The team chart cannot edit them: ArgoCD reconciles them away on every sync.
Kyverno stamps the Managed Resources at admission. A ClusterPolicy matches every Crossplane Managed Resource (MR) created in a pr-preview namespace and mutates the spec: it adds providerConfigRef derived from the tenant label, and forProvider.resourceGroupName derived from tenant-rg. Team chart authors never write either field. They cannot: Kyverno overrides whatever they put there.
The cloud enforces the blast radius. The MI for the team has exactly one RoleAssignment: Contributor, scoped to the team's RG. Even if a tenant somehow got past layers 1 and 2, the cloud's own Role-Based Access Control (RBAC) would refuse the call.

# Kyverno ClusterPolicy (abridged) — runs at admission on every Crossplane MR
# created in a tenant namespace.
match:
  any:
    - resources:
        kinds: ["*/*"]
        namespaceSelector:
          matchLabels:
            platform.example.io/tenant-scope: pr-preview
preconditions:
  all:
    - key: "{{ regex_match('^.*\\.upbound\\.io$', '{{ request.kind.group }}') }}"
      operator: Equals
      value: true
mutate:
  patchesJson6902: |-
    - op: add
      path: /spec/providerConfigRef
      value:
        kind: ClusterProviderConfig
        name: '{{ nsLabels."platform.example.io/tenant" }}-azure'

The shape kinds: ["*/*"] is deliberate: Kubernetes admission webhooks reject partial group wildcards, so the group filter lives in preconditions.

The pattern inverts the trust model: the chart is the least-trusted artefact, the namespace label is the trust anchor, and Kyverno is the seam. Team chart authors get fast iteration without ever holding platform credentials. Platform owns the fence and can change it in one place when something needs hardening.

Authentication — no static cloud credentials

Crossplane needs to authenticate to the cloud. We refused to put a static credential in a Kubernetes Secret. The pattern that worked:

A Managed Identity (or its cloud equivalent) for the Crossplane provider.
Workload Identity Federation: the ServiceAccount the provider runs as is federated to that identity via the cluster's OpenID Connect (OIDC) issuer URL.
The provider's ProviderConfig references the identity by client ID. No secret, no rotation, no leak surface.

Azure: User-Assigned Managed Identity + FederatedIdentityCredential. AWS: IAM Roles for Service Accounts (IRSA). GCP: Workload Identity. Same principle, different flavour.

One honest caveat. In a multi-tenant control plane like ours, one Crossplane provider Pod talks to many tenant identities. Its ServiceAccount mounts a single OIDC token, and the subject claim in that token is shared across all tenants. Per-tenant isolation comes from the fence above: Kyverno picks the right ProviderConfig per namespace, and the RG-scoped RoleAssignment enforces blast radius at the cloud. It is not cryptographic isolation. We made our peace with the trade-off; worth naming out loud.

The SCM PAT is the one static credential we still need: the PR generator does not yet federate to SCMs natively. We rotate it on a schedule and hope upstream catches up.

The half of the loop that bit us

The flow above assumes the image tagged pr-<N> exists when ArgoCD spawns the Application. That image comes from the tenant's own CI pipeline, not the platform.

We discovered this during demo validation: every platform piece worked beautifully, but the Pod hit ErrImagePull because nothing had built pr-42. The fix on the tenant side is small: a PR-triggered pipeline that builds and tags pr-$(PR_NUMBER):

trigger: none
pr: { branches: { include: ['*'] } }
stages:
  - stage: BuildPushImage
    jobs:
      - job:
        steps:
          - task: Docker@2
            inputs:
              command: buildAndPush
              repository: $(REGISTRY)/$(IMAGE)
              tags: pr-$(System.PullRequest.PullRequestId)

The bigger lesson: write the platform/tenant contract down before onboarding tenant #2. What the platform owns vs. what the tenant owns. Without it, the seam between "platform did its part" and "tenant did its part" becomes a debugging nightmare.

When the workaround became the bug

A day after a mitigation landed, a wave of fresh-PR Pods started failing with:

AADSTS700211: No matching federated identity record found for presented assertion issuer

kubectl get fic -o yaml on the affected tenants showed exactly the OIDC issuer URL we expected in spec.forProvider. So we asked Azure directly:

$ az identity federated-credential show \
    --identity-name mi-tenant-a \
    --name mi-tenant-a-fed \
    --resource-group rg-tenant-a \
  | jq .issuer
"https://oidc.<old-cluster>.../..."   # not what the spec said

Azure was still serving the pre-migration issuer. The cluster had just migrated to a new OIDC issuer URL; the corrected URL was already in the values file, ArgoCD had synced the Composition, and the corrected spec had never reached Azure.

To understand why, you have to know about the workaround already living in the FIC's Composition. A day earlier we had hit a different bug: legitimate FIC spec changes were being overwritten back to stale values within seconds of applying them. The symptom looked exactly like a stale-state cache in the upstream provider. The mitigation we shipped was minimal: strip Update from spec.managementPolicies, leaving [Create, Delete, Observe]. Crossplane could still create and observe FICs but could no longer push corrections. The symptom went away. We marked it "interim, pending upstream fix" in the docs and moved on.

What we did not realise: we had just disabled the only path through which legitimate future spec changes could reach Azure. The OIDC issuer correction sat in git, reconciled, observed, and stranded.

The real fix landed in three commits:

Find the second writer. The earlier "stale writes" symptom was not a cache. Crossplane had been running on our previous cluster (call it cluster-A) while we migrated the platform to a new cluster (cluster-B). We updated the OIDC issuer URL in cluster-B's values file and replaced all cluster-A references, but forgot to disable Crossplane on cluster-A in the app-of-apps. Both clusters held valid Workload Identity Federation credentials scoped to the same Managed Identities. Both were reconciling the same Azure Federated Identity Credentials. When cluster-B wrote the corrected OIDC issuer URL, cluster-A's Crossplane saw drift from its own (stale) spec and immediately wrote it back. The two providers were racing over every field, continuously. Disabling Crossplane on cluster-A removed the second writer.
Put Update back. With the real cause neutralised, restoring the default managementPolicies (Create, Update, Delete, Observe) is safe again.
Recreate the affected composites. Existing FICs in Azure still carried the wrong pre-migration issuer. kubectl delete xteamidentityazure <tenant> for each affected tenant; ArgoCD re-renders the claim; the Composition re-emits FICs with the corrected spec; UserAssignedIdentity adopts the existing Azure MI via pinned external-name.

Three lessons we will not forget:

A mitigation that disables a reconcile loop is a time bomb for legitimate change. Stripping Update made the original symptom go away and blocked the very next legitimate spec change from reaching the cloud, inside 24 hours. Workarounds shipped mid-migration are especially dangerous: that's exactly when legitimate spec changes are flying past, any one of which the mitigation could silently swallow.
When migrating Crossplane to a new cluster, disable it on the old one first. Two Crossplane providers authenticated to the same cloud identity will race over the same cloud resources. The symptom looks like a stale-state cache or a misbehaving provider, but it isn't. If both clusters hold valid Workload Identity Federation credentials scoped to the same Managed Identity, both reconcilers will fight over every field on every Azure resource. Disable (or fully remove) Crossplane on the source cluster before the destination cluster takes over. App-of-apps templating makes this easy to miss: the old cluster's entry is still there, still healthy, still reconciling.
When the Kubernetes side and the cloud side disagree, ask the cloud. kubectl get fic agreed with the corrected spec. Azure did not. az identity federated-credential show was the call that broke the illusion. If the symptom screams trust, don't trust what Kubernetes tells you about the cloud. Go to the cloud.

Lessons learned

Pick your secret-injection operator deliberately. We migrated from the External Secrets Operator (ESO) to a cloud-native vault-sync operator (akv2k8s on Azure). The motivation was workload-identity hygiene: ESO needed its own cluster ServiceAccount with cloud-side credentials, while the cloud-native operator authenticates with the cluster's existing federated identity. One fewer credential, one fewer rotation. Either pattern works, but pick early; mid-stream migration is annoying.
Tear-down ordering matters. Set PrunePropagationPolicy=foreground on the per-tenant Application's syncOptions. That tells the Kubernetes garbage collector to wait for Crossplane finalizers to drain before deleting the namespace. Get this wrong and the namespace disappears first, orphaning any cloud resource whose Claim was still finalizing.
Cap the cost. Set a TTL: auto-close PRs idle for N days, or scale preview Pods to zero overnight. Ephemeral envs are cheap individually and expensive in aggregate.
Validate end-to-end, on a real tenant repo, by opening a PR yourself. "Platform delivered" is not the same as "the loop closes." The PR is the only acceptance test that matters.
A live demo on a tenant's actual codebase converts more skeptics than any docs page. Build one, run it, click the preview URL together.
One public feedback channel (Slack thread or Confluence page) where every team's pain lives. Triage weekly. The platform stays honest.

What's next

Federated SCM auth: kill the last static credential when upstream catches up.
Per-PR cost estimate posted back to the PR as a comment.
Smoke test on spin-up: is the preview URL 200? posted back as a PR check. Half the time when a preview "doesn't work" it's the tenant's app, not the platform; making that visible saves the back-and-forth.

Closing thought

The best part of this setup isn't the technology: it's that "did this PR work?" stops being a debate. The developer opens the PR, clicks the preview link, exercises the change end-to-end against real cloud resources, and either ships it or doesn't.

If you're standing this up at your own org, start with one tenant repo, one type of backing resource, and a labelled PR as the trigger. Get the loop closing end-to-end before you generalise. Everything else is iteration.

The Ultimate Kubernetes Checklist: Unlocking Performance While Slashing Costs

Nandini — Mon, 15 Jun 2026 16:57:38 +0000

Build a Strong Kubernetes Foundation

Before optimizing costs or performance, organizations must ensure their Kubernetes environment is healthy and reliable. A strong foundation reduces operational risks and creates a stable platform for future growth.

Key Areas to Review
Control plane health monitoring
Kubernetes version management
Automated backup strategies
Multi-zone deployments
Node health monitoring Why It Matters

Many performance issues and unexpected outages originate from poor cluster maintenance. Establishing strong operational practices early prevents larger problems later and improves overall cluster reliability.

Optimize Resources and Scale Efficiently

Resource waste is one of the biggest contributors to rising Kubernetes costs. Overprovisioned CPU and memory allocations often leave clusters running far below their actual capacity.

Resource Optimization Checklist
Review CPU and memory requests
Compare allocated resources with actual usage
Remove unused workloads
Optimize scheduled batch jobs
Right-size applications regularly
Smart Scaling Checklist
Configure Horizontal Pod Autoscaler (HPA)
Enable Cluster Autoscaler
Test scaling thresholds
Simulate peak traffic conditions
Monitor node utilization Why It Matters

Organizations frequently reduce cloud spending by 20–40% simply by right-sizing workloads and implementing effective autoscaling strategies.

Improve Visibility, Security, and Operational Control

Optimization becomes difficult when teams lack visibility into resource consumption, application performance, and infrastructure costs.

Observability Checklist
Enable metrics collection
Configure centralized logging
Implement distributed tracing
Establish cost monitoring
Create performance dashboards
Security Checklist
Enforce RBAC policies
Configure Network Policies
Scan container images
Secure secrets management
Enable audit logging Why It Matters

Visibility helps teams identify inefficiencies, while security ensures workloads remain protected without sacrificing performance.

Create a Culture of Continuous Optimization

Kubernetes environments constantly evolve. New applications, changing traffic patterns, and increasing infrastructure demands can quickly introduce inefficiencies.

Continuous Improvement Checklist
Monthly cost audits
Quarterly architecture reviews
Resource utilization analysis
Performance benchmarking
FinOps collaboration
Kubernetes Maturity Scorecard

Evaluate your environment regularly across:

Infrastructure Health
Resource Efficiency
Scalability
Observability
Security
Cost Governance
Reliability
Final Takeaway

The most successful Kubernetes teams don't optimize once—they optimize continuously. By regularly reviewing costs, performance, scalability, and security, organizations can build Kubernetes platforms that are both high-performing and cost-efficient.

Remember: Kubernetes success isn't about running more containers. It's about running them smarter.

The Human-in-the-Loop SRE: Designing Automation Escalation Policies for AI-Assisted Operations

Nijo George Payyappilly — Mon, 15 Jun 2026 16:00:00 +0000

On April 23, 2021, a Fastly CDN configuration change triggered a global outage that took down the UK government website, the New York Times, Reddit, and hundreds of other major internet properties for approximately one hour. The triggering event was a configuration push. The propagation mechanism was automated. The time between the configuration being pushed and the global impact becoming visible was under a minute. The time required for a human operator to identify the cause and initiate the rollback was approximately forty-nine minutes longer than that.

The Fastly incident is not primarily a story about automation failure. It is a story about the speed asymmetry between automated propagation and human response — and about what happens when the automation layer between a human decision and its production consequence moves faster than the accountability layer designed to govern it.

This asymmetry is the defining operational challenge of AI-assisted SRE. The capability to automate incident detection, root cause hypothesis generation, and even remediation is now accessible at costs and latencies that were unavailable five years ago. The operational risk is not that this capability will be under-used. The risk is that it will be deployed without a rigorous escalation policy — a formal framework that defines exactly where automated execution ends and human judgement begins, under what conditions the boundary shifts, and how accountability is preserved for every action the AI takes on behalf of an operator who may not have been in the room when it was taken.

The Human-in-the-Loop Spectrum

AI-assisted SRE operations do not exist at a single point on the autonomy spectrum. They exist across a range, and the appropriate position on that range is a function of confidence, blast radius, novelty, and regulatory context — not of how sophisticated the AI system is.

THE AUTOMATION AUTONOMY SPECTRUM
────────────────────────────────────────────────────────────────────────────

LEVEL 0 — MANUAL
  AI generates no recommendations. Human observes raw telemetry and decides.
  Appropriate when: AI system is unavailable, untrusted, or context is
  outside AI training distribution entirely.

LEVEL 1 — ASSISTED
  AI surfaces relevant context, correlated signals, and historical patterns.
  Human makes all decisions. AI does not recommend actions.
  Appropriate when: novel failure pattern; first occurrence of incident type;
  regulated change requiring documented human judgement.

LEVEL 2 — SUPERVISED
  AI recommends specific actions with confidence scores. Human approves
  each action before execution. AI does not execute autonomously.
  Appropriate when: high blast radius; unfamiliar but not novel pattern;
  action is reversible but consequential.

LEVEL 3 — CONDITIONAL AUTONOMOUS
  AI executes actions autonomously within pre-approved policy boundaries.
  Human is notified after execution. Human can abort within a defined window.
  Appropriate when: well-characterised failure pattern; low blast radius;
  action is fully reversible; pattern seen > N times with consistent outcome.

LEVEL 4 — AUTONOMOUS
  AI executes and verifies remediation without human notification unless
  verification fails. Audit trail maintained.
  Appropriate when: toil pattern fully characterised; action is idempotent;
  blast radius is bounded to a single service; recurrence rate justifies
  zero-latency response.

────────────────────────────────────────────────────────────────────────────
CRITICAL CONSTRAINT: No action may exist permanently at Level 4.
Every Level 4 automation must have a scheduled re-qualification review
that reassesses whether the failure pattern is still well-characterised
and the blast radius assumption still holds.
────────────────────────────────────────────────────────────────────────────

The critical constraint — that no action may exist permanently at Level 4 — is not conservatism. It is the engineering response to a specific failure mode: automation that was correctly calibrated at deployment time and has silently drifted out of calibration as the system evolved. An OOM restart automation that was safe when first deployed becomes unsafe the moment the underlying cause shifts from a memory leak to a data corruption event that is triggering the same symptom. The re-qualification review is the mechanism that catches this drift before it produces an incident.

The Four Escalation Triggers

Every escalation policy is built from four primitive triggers. Each trigger defines a condition under which the automation level must shift upward — toward more human involvement, not less.

Trigger 1 — Confidence Threshold Breach

The AI system's confidence in its diagnosis or recommended action has fallen below a defined threshold. In the context of LLM-based operations (HolmesGPT, LiteLLM Proxy routing), confidence is expressed as a combination of model-reported token probability distributions and domain-specific heuristics applied to the recommendation output.

A low-confidence diagnosis means the AI has identified a plausible pattern match but lacks sufficient corroborating signal to recommend action without human review. Executing actions based on low-confidence diagnoses is the operational equivalent of acting on a single data point in a monitoring dashboard: occasionally correct, reliably dangerous as a policy.

Trigger 2 — Blast Radius Threshold

The proposed action affects more infrastructure than the policy authorises for autonomous execution. Blast radius is assessed across three dimensions: service count (how many services are affected), traffic fraction (what percentage of user requests are served by the affected infrastructure), and reversibility (can the action be undone in under five minutes with a single command).

High blast radius is not a disqualifying condition for automation. It is a condition that requires the automation level to shift to at least Level 2 (supervised) regardless of confidence score.

Trigger 3 — Novelty Detection

The failure pattern does not match any pattern in the AI system's training corpus or historical incident database. Novelty is the most dangerous condition for autonomous execution because it is precisely the condition where the AI's pattern-matching capability provides the least value — and where a confident-sounding but incorrect recommendation carries the highest operational cost.

Novelty detection is the hardest trigger to implement well, because it requires the AI system to accurately assess the boundaries of its own knowledge. A system that cannot reliably distinguish "I have seen this pattern and am confident" from "I have seen a superficially similar pattern and am extrapolating" should not be operating at Level 3 or Level 4.

Trigger 4 — Regulatory Boundary

The proposed action would touch a regulated asset, require a documented change record, affect a system subject to NERC CIP, PCI-DSS, HIPAA, or equivalent obligations, or generate a compliance event. In regulated environments, no automated action may bypass the change management governance framework, regardless of confidence score or blast radius.

This trigger is absolute. It does not have a confidence threshold exception. An AI system that correctly diagnoses a production issue with 99% confidence and proposes a remediation that would constitute an undocumented change to a regulated asset must escalate to Level 2 and generate a change record, even if the remediation would restore service faster without it.

Designing the Escalation Policy Document

The escalation policy is an operational governance document, not a configuration file. It must be version-controlled, reviewed and approved by SRE leadership and compliance, and referenced in every AI-assisted automation's runtime configuration. Its authority derives from human review, not from the AI system that consults it.

ESCALATION POLICY: AI-ASSISTED INCIDENT RESPONSE
────────────────────────────────────────────────────────────────────────────
Service:       production-platform (all services)
AI System:     HolmesGPT + LiteLLM Proxy + Ollama / GitHub Models
Policy Version: v1.3  |  Approved: SRE Lead + VP Engineering
Last Reviewed: 2025-Q1  |  Next Review: 2025-Q2
────────────────────────────────────────────────────────────────────────────

SECTION 1: AUTONOMOUS EXECUTION AUTHORISED (Level 4)
  Conditions required (ALL must be true):
    ✓ Confidence score ≥ 0.85 (model-reported + heuristic composite)
    ✓ Pattern seen ≥ 10 times in incident history with consistent outcome
    ✓ Blast radius: single service, single namespace, ≤ 20% of replicas
    ✓ Action is idempotent and fully reversible in ≤ 5 minutes
    ✓ No regulated asset in scope
    ✓ Error budget > 25% remaining (not in Tier 3 freeze)
  Authorised actions at Level 4:
    → Rolling restart of single stateless deployment (OOM, deadlock)
    → Scale-up of single HPA-managed deployment by ≤ 2 replicas
    → Certificate rotation on non-production workloads
    → Log pipeline gateway restart (telemetry outage, no production impact)
  Required logging: structured Splunk event per action (mandatory)
  Re-qualification: every 90 days or after any incident where autonomous
                   action was taken and outcome was suboptimal

SECTION 2: SUPERVISED EXECUTION (Level 2 — Human Approval Required)
  Conditions triggering Level 2 (ANY is sufficient):
    ⚠ Confidence score 0.60–0.84
    ⚠ Blast radius: > 20% of replicas OR > 1 service OR cross-namespace
    ⚠ First or second occurrence of this failure pattern
    ⚠ Error budget between 25–75% (Tier 2 degraded)
    ⚠ Action affects shared infrastructure (Argo CD, Prometheus, Istio)
  Approval mechanism: Slack approval button with 10-minute timeout
  Timeout behaviour: escalate to on-call if no response in 10 minutes
  Required logging: recommendation + approval/rejection + outcome

SECTION 3: ASSISTED ONLY (Level 1 — No Action Authorised)
  Conditions triggering Level 1 (ANY is sufficient):
    ✗ Confidence score < 0.60
    ✗ Novel failure pattern (no match in incident history)
    ✗ Regulated asset in scope (NERC CIP, PCI-DSS, HIPAA boundary)
    ✗ Error budget < 25% (Tier 3 freeze — deployment freeze active)
    ✗ Active P0 incident in progress (human incident commander owns scope)
    ✗ Multiple simultaneous incidents (blast radius assessment unreliable)
  AI role at Level 1: surface correlated signals, historical context only
  Human owns: diagnosis, action decision, execution, verification

SECTION 4: ACCOUNTABILITY CHAIN
  Every AI-assisted action must trace to one of:
    a) Direct human approval (Level 2 Slack approval button)
    b) This policy document (Level 4 autonomous execution)
  "The AI decided" is not a complete accountability chain.
  Policy document owner: SRE Lead
  Policy review and approval authority: SRE Lead + VP Engineering
────────────────────────────────────────────────────────────────────────────

HolmesGPT Escalation Architecture

The escalation policy document defines the governance rules. The escalation architecture implements those rules as runtime logic in the AI-assisted operations stack. The architecture shown here is specific to the HolmesGPT + LiteLLM Proxy + Ollama deployment pattern in a regulated on-premises environment.

# HolmesGPT Escalation Policy ConfigMap
# Consumed by HolmesGPT at runtime to determine autonomy level per action
# Version-controlled in git; updated only via Argo CD sync (change record enforced)

apiVersion: v1
kind: ConfigMap
metadata:
  name: holmesgpt-escalation-policy
  namespace: holmesgpt
  annotations:
    sre.internal/policy-version: "v1.3"
    sre.internal/approved-by: "sre-lead,vp-engineering"
    sre.internal/approved-date: "2025-03-15"
    sre.internal/next-review: "2025-06-15"
    sre.internal/review-enforced-by: "kyverno-policy/ai-ops-policy-review"
data:
  escalation_policy.yaml: |
    confidence_thresholds:
      autonomous:   0.85
      supervised:   0.60
      assisted_only: 0.0

    blast_radius_limits:
      autonomous:
        max_replica_fraction: 0.20
        max_service_count: 1
        max_namespace_count: 1
        cross_namespace_allowed: false
        regulated_assets_allowed: false

    autonomous_actions_allowlist:
      - action: rolling_restart_stateless
        max_replicas_affected: 5
        requires_pdb_check: true
      - action: hpa_scale_up
        max_replica_delta: 2
        requires_current_below_sot: true
      - action: log_pipeline_restart
        namespaces: [monitoring, sre-platform]
        production_namespaces_blocked: true

    error_budget_gates:
      tier_3_freeze_blocks_autonomous: true
      tier_2_degrades_to_supervised: true

    regulatory_boundary:
      always_level_1_namespaces:
        - pci-zone
        - hipaa-zone
        - nerc-cip-zone
      always_level_1_labels:
        - "compliance.internal/regulated=true"

    novelty_detection:
      min_historical_occurrences_for_autonomous: 10
      similarity_threshold: 0.80
      unknown_pattern_forces_level_1: true

    approval_workflow:
      slack_channel: "sre-aiops-approvals"
      timeout_minutes: 10
      timeout_action: escalate_to_oncall

    audit:
      splunk_sourcetype: "sre:holmesgpt:decisions"
      log_all_recommendations: true
      log_operator_overrides: true
      override_feeds_prompt_review: true

Model Routing for Escalation Quality

The LiteLLM Proxy's model routing configuration is a first-class component of the escalation architecture. Routing to the right model at the right confidence tier is not a performance optimisation — it is a safety mechanism.

# LiteLLM Proxy — Model Routing for Escalation Tiers
# Smaller local models for low blast radius / routine patterns
# Larger models with greater context window for high blast radius / novel patterns
# On-premises models for regulated asset investigations (data sovereignty)

model_list:
  # Tier 1: Routine investigation — local Ollama model
  # Low latency, no data egress, adequate for well-characterised patterns
  - model_name: holmesgpt-routine
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://ollama.ai-ops.svc.cluster.local:11434
      timeout: 30
      max_tokens: 2048

  # Tier 2: Complex investigation — larger local model
  # Higher accuracy for multi-service correlation and novel patterns
  - model_name: holmesgpt-complex
    litellm_params:
      model: ollama/llama3.1:70b
      api_base: http://ollama.ai-ops.svc.cluster.local:11434
      timeout: 90
      max_tokens: 8192

  # Tier 3: High-stakes / novel pattern — GitHub Models
  # Largest context window for multi-service incident correlation
  # Data classification check required before routing: no PII, no regulated data
  - model_name: holmesgpt-highstakes
    litellm_params:
      model: github/gpt-4o
      api_base: https://models.inference.ai.azure.com
      api_key: "os.environ/GITHUB_MODELS_PAT"
      timeout: 120
      max_tokens: 16384

router_settings:
  routing_strategy: custom
  routing_logic: |
    # Route by blast_radius_tier header set by HolmesGPT pre-routing assessment
    if blast_radius_tier == "low" and pattern_novelty == "known":
        return "holmesgpt-routine"
    elif blast_radius_tier == "high" or pattern_novelty == "novel":
        # Data classification gate before external model routing
        if data_contains_regulated_fields:
            return "holmesgpt-complex"  # Stay on-premises
        return "holmesgpt-highstakes"
    else:
        return "holmesgpt-complex"

  fallback_model: holmesgpt-complex    # Always fall back to on-premises
  fallback_on_status_codes: [429, 500, 503]

The Recommendation Quality Feedback Loop

The operational risk of AI-assisted recommendations is not static. It evolves as the system changes and as the model's training distribution diverges from the current operational reality. An AI recommendation quality feedback loop is the mechanism that makes this drift visible before it produces a damaging autonomous action.

# Prometheus Recording Rules — AI Recommendation Quality Tracking
# Measures whether HolmesGPT recommendations are operationally valuable
# High override rate or low action rate = recommendation quality degrading

groups:
  - name: holmesgpt.recommendation_quality
    rules:

      # Recommendation acceptance rate: fraction of recommendations
      # that operators acted on (approved or executed autonomously)
      # versus rejected or ignored
      - record: holmesgpt:recommendation_acceptance_rate:rate7d
        expr: |
          sum(rate(holmesgpt_recommendations_acted_on_total[7d]))
          /
          sum(rate(holmesgpt_recommendations_total[7d]))

      # Operator override rate: fraction of autonomous actions that
      # were manually reversed by an operator after execution
      # High rate = autonomous confidence thresholds are too permissive
      - record: holmesgpt:autonomous_override_rate:rate7d
        expr: |
          sum(rate(holmesgpt_autonomous_actions_reversed_total[7d]))
          /
          sum(rate(holmesgpt_autonomous_actions_total[7d]))

      # False positive rate: recommendations made but outcome was
      # NOT the recommended action resolving the incident
      - record: holmesgpt:false_positive_rate:rate7d
        expr: |
          sum(rate(holmesgpt_recommendations_outcome_mismatch_total[7d]))
          /
          sum(rate(holmesgpt_recommendations_acted_on_total[7d]))

      # Alert: recommendation quality degrading
      - alert: HolmesGPT_RecommendationQualityDegrading
        expr: |
          holmesgpt:autonomous_override_rate:rate7d > 0.15
          OR
          holmesgpt:false_positive_rate:rate7d > 0.20
        for: 1d
        labels:
          severity: ticket
          domain: ai_ops_quality
        annotations:
          summary: >
            HolmesGPT recommendation quality below threshold.
            Override rate: {{ with query "holmesgpt:autonomous_override_rate:rate7d" }}
            {{ . | first | value | humanizePercentage }}{{ end }}.
            Action: review recent overrides, update prompt context,
            consider reducing autonomous confidence threshold.
          runbook: "https://wiki.internal/sre/runbooks/holmesgpt-quality-review"

      # Alert: recommendation volume causing alert fatigue risk
      # More than 3 recommendations per incident = cognitive overload signal
      - alert: HolmesGPT_RecommendationVolumeHigh
        expr: |
          sum(rate(holmesgpt_recommendations_total[1h]))
          /
          sum(rate(incidents_opened_total[1h])) > 3
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: >
            HolmesGPT generating > 3 recommendations per incident on average.
            Risk: alert fatigue causing operators to ignore recommendations.
            Action: tighten confidence floor or reduce recommendation scope.

The Accountability Chain Principle and NIST AI RMF Alignment

The accountability chain principle — that every AI-assisted action must trace back to a human decision, either a direct approval or a policy that a human wrote and approved — is the operational implementation of the NIST AI Risk Management Framework's GOVERN function.

The NIST AI RMF establishes four core functions for AI risk management: GOVERN (policies, accountability), MAP (risk identification), MEASURE (risk quantification), and MANAGE (risk response). Each function maps directly to components of the escalation policy architecture.

NIST AI RMF MAPPING: AI-ASSISTED SRE OPERATIONS
────────────────────────────────────────────────────────────────────────────

GOVERN — Accountability and Policy
  Who owns the AI system's outputs?
    → SRE Lead owns escalation policy; VP Engineering co-approves
  Who approves autonomous action boundaries?
    → Policy document with named approvers and review cadence
  How are accountability chains maintained?
    → Splunk audit trail: every recommendation, decision, and outcome
  SRE implementation: escalation policy document + approval workflow

MAP — Risk Identification
  What failure modes does the AI system face?
    → Confidence decay: model accuracy degrades as system evolves
    → Distribution shift: production patterns diverge from training data
    → Novel pattern extrapolation: confident recommendation on unfamiliar input
    → Blast radius miscalculation: action scope larger than assessed
  SRE implementation: four escalation triggers + novelty detection

MEASURE — Risk Quantification
  How do you measure AI recommendation quality over time?
    → Acceptance rate: fraction of recommendations acted on
    → Override rate: fraction of autonomous actions manually reversed
    → False positive rate: recommendations where predicted outcome was wrong
    → Confidence calibration: does 85% confidence actually mean 85% accuracy?
  SRE implementation: Prometheus quality recording rules + 7-day rolling metrics

MANAGE — Risk Response
  What happens when AI recommendation quality degrades?
    → Automatic downgrade of autonomous confidence threshold
    → Prompt context refresh from recent incident postmortems
    → Temporary suspension of Level 4 autonomy pending review
  SRE implementation: quality alert → runbook → policy review cadence
────────────────────────────────────────────────────────────────────────────

Splunk Audit Trail: The Irreplaceable Governance Layer

In regulated environments, the audit trail for AI-assisted actions is not optional. It is the documentary evidence that demonstrates human accountability over automated decisions — the record that answers the auditor's question: "Who authorised this change to your production system?"

# Splunk HEC Forwarder — HolmesGPT Decision Audit Trail
# Every recommendation, escalation decision, and outcome → Splunk
# This record is the accountability chain in documentary form

# Splunk event structure (sourcetype: sre:holmesgpt:decisions):
# {
#   "timestamp": "2025-04-15T14:23:07Z",
#   "incident_id": "INC-20250415-0047",
#   "alert_name": "KubePodOOMKilled",
#   "service": "payments-api",
#   "namespace": "production",
#
#   "investigation": {
#     "model_used": "holmesgpt-routine",
#     "model_backend": "ollama/llama3.1:8b",
#     "confidence_score": 0.91,
#     "diagnosis": "Memory limit (2Gi) exceeded by 847MB under high load...",
#     "recommended_action": "rolling_restart_stateless",
#     "blast_radius_assessment": {
#       "services_affected": 1,
#       "replica_fraction": 0.15,
#       "reversible": true,
#       "regulated_asset": false
#     }
#   },
#
#   "escalation_decision": {
#     "autonomy_level": 4,
#     "policy_version": "v1.3",
#     "triggers_evaluated": ["confidence", "blast_radius", "novelty", "regulatory"],
#     "triggers_fired": [],
#     "decision": "AUTONOMOUS_EXECUTE",
#     "policy_authority": "holmesgpt-escalation-policy v1.3 (approved: sre-lead)"
#   },
#
#   "execution": {
#     "action_taken": "rolling_restart_stateless",
#     "execution_start": "2025-04-15T14:23:09Z",
#     "verification_result": "HEALTHY",
#     "mttr_seconds": 67,
#     "operator_override": false
#   },
#
#   "quality_signals": {
#     "prediction_matched_outcome": true,
#     "error_budget_consumed_pct": 0.002,
#     "operator_satisfaction": null    # Populated by post-incident feedback
#   }
# }

The policy_authority field in the escalation decision block is the accountability chain closure. It names the specific policy document version and its human approvers. When an auditor asks who authorised the autonomous action, the answer is not "the AI decided" — it is "the SRE Lead and VP Engineering approved escalation policy v1.3 on 2025-03-15, and this action fell within the boundaries of Section 1 of that policy."

The Confidence Calibration Problem

A confidence score of 0.85 from a language model does not intrinsically mean that the recommendation is correct 85% of the time. Language models are notoriously poorly calibrated — they express high confidence in incorrect outputs and sometimes express low confidence in correct ones. The confidence threshold in the escalation policy must be calibrated against the AI system's actual historical accuracy, not against the model's self-reported certainty.

-- Splunk SPL: Confidence Calibration Assessment
-- Compares model-reported confidence bands against actual outcome accuracy
-- Run monthly; output informs confidence threshold calibration in policy

index=sre_holmesgpt sourcetype="sre:holmesgpt:decisions"
| eval confidence_band = case(
    confidence_score >= 0.90, "90-100%",
    confidence_score >= 0.85, "85-89%",
    confidence_score >= 0.80, "80-84%",
    confidence_score >= 0.70, "70-79%",
    confidence_score >= 0.60, "60-69%",
    true(),                   "<60%"
  )
| stats
    count                                          as total_recommendations,
    sum(prediction_matched_outcome)                as correct_predictions,
    avg(prediction_matched_outcome)                as empirical_accuracy,
    sum(operator_override)                         as operator_overrides
    by confidence_band, model_used
| eval
    calibration_delta = empirical_accuracy - (tonumber(substr(confidence_band,1,2))/100),
    calibration_status = if(abs(calibration_delta) < 0.10, "CALIBRATED", "MISCALIBRATED")
| table
    confidence_band, model_used, total_recommendations,
    empirical_accuracy, calibration_delta, calibration_status, operator_overrides
| sort confidence_band

-- If empirical_accuracy at "85-89%" band is actually 0.71:
-- The 0.85 autonomous threshold is accepting actions that are only
-- correct 71% of the time. Raise threshold or re-evaluate model.

Common Antipatterns

The Confidence Theatre antipattern → Using model-reported confidence scores as the primary autonomous execution gate without calibration against empirical outcome accuracy. A model that reports 0.92 confidence but is empirically correct 68% of the time is a dangerous basis for autonomous action. Calibration against historical outcomes must precede the deployment of any confidence-based gate.
The Policy-as-Default antipattern → Deploying the AI system with permissive defaults and planning to tighten the escalation policy "after we see how it performs in production." The escalation policy must be the first artefact produced, not a retroactive constraint on a system that is already taking autonomous actions. Permissive defaults in AI operations systems are not starting points; they are incident preconditions.
The Accountability Diffusion antipattern → Designing the system so that no single person is clearly accountable for an autonomous AI action. "The AI did it" is not an accountability chain. "The escalation policy approved by [names] on [date] authorised this class of action" is. In regulated environments, the inability to name a responsible human for a production change is itself a compliance finding.
The Alert Fatigue Transfer antipattern → Moving from a system that generates too many monitoring alerts to a system that generates too many AI recommendations. If HolmesGPT surfaces seven recommendations per incident, operators will start ignoring them at the same rate they ignore high-volume monitoring alerts. Recommendation volume should be governed by the same principles as alert volume: every recommendation must be actionable, and the threshold for surfacing should be higher than the threshold for suppressing.
The Permanent Level 4 antipattern → Classifying an autonomous action as Level 4 and never re-qualifying it. The re-qualification cadence is the mechanism that prevents a well-calibrated autonomous action from silently becoming a dangerous one as the system evolves. Every Level 4 action must carry a sre.internal/sot-next-review equivalent annotation and a Kyverno policy that generates a ticket when the date passes.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        AI-OPS ESCALATION STATE             NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     No AI-assisted operations.          All investigation is
             Operators work from raw             manual. MTTR limited
             telemetry only.                     by human availability.

Defined      HolmesGPT deployed at              AI operating at Level
             Level 1 only. Escalation           1–2 only. Context
             policy drafted but not             surfacing measurably
             yet governing autonomous           reduces investigation
             action.                            time.

Measured     Escalation policy governs          Recommendation quality
             Level 3–4 boundaries.              metrics tracked. Confidence
             Audit trail in Splunk.             calibration assessed
             Quality metrics active.            monthly. Override rate
                                                below 15%.

Optimised    Confidence calibration             Level 4 actions cover
             cycle running quarterly.           top-5 toil remediations.
             Model routing by blast             MTTR for covered patterns
             radius operational.                < 5 minutes (automated).
             NIST AI RMF aligned.               Audit trail satisfies
                                                regulatory review.

Generative   Escalation policy published        Policy cited in industry
             as reference architecture.         guidance. Recommendation
             Feedback loop feeds               quality above 85%.
             prompt engineering cycle.          AI-ops layer itself
             AI-ops treated as a               has SLO and error budget.
             production service.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Draft your escalation policy document before configuring any autonomous action in HolmesGPT. Start with the accountability chain section: who owns the policy, who approves autonomous action boundaries, and what the change record looks like. A policy document that exists on paper but has not been approved by SRE leadership and VP Engineering is not a governance artefact — it is a draft. The approval is the governance act.
Run the Splunk confidence calibration query against your last 90 days of HolmesGPT decisions. If you do not yet have 90 days of data, start collecting it now at Level 1 only. Calibration data must precede autonomous execution boundaries. The calibration query is the empirical basis for your confidence thresholds — thresholds chosen without it are guesses with operational consequences.
Map every existing automated remediation to an autonomy level and a blast radius assessment. For each automation in your Class 1 (Reactive Remediation) category from the automation taxonomy post, assess: what is its blast radius under worst-case conditions, and what confidence mechanism governs when it executes? Automations with no explicit blast radius boundary and no confidence mechanism are operating at implicit Level 4 without a policy. Make the policy explicit before the next incident.
Configure the recommendation quality Prometheus rules and set a 30-day baseline. Even if you are operating at Level 1 only, begin measuring acceptance rate and false positive rate now. The first meaningful governance conversation about elevating to Level 3 or Level 4 should be anchored in empirical quality data, not in enthusiasm about the capability.
Add the four escalation triggers as literal fields to your HolmesGPT Splunk audit events. Every decision event should record: confidence_trigger_fired: true/false, blast_radius_trigger_fired: true/false, novelty_trigger_fired: true/false, regulatory_trigger_fired: true/false. Over time, this data reveals which triggers are governing your escalation decisions most frequently — and which failure modes your autonomous boundary is most exposed to.

"The risk in AI-assisted SRE is not that the automation will fail to act. The risk is that it will act confidently, at scale, on a pattern it has only partially understood — and that the human who approved the policy that authorised the action will not be reachable, will not remember what the policy said, or will not realise the policy applied to this situation. The escalation policy is not a constraint on AI capability. It is the engineering discipline that makes AI capability safe to deploy in systems where the cost of being confidently wrong is borne by users, not by the model."

What Comes Next

The escalation policy governs how AI recommendations become actions. The harder engineering problem is the quality of the recommendations themselves — specifically, how to evaluate LLM reliability for incident diagnosis with the same rigour that SRE applies to any other production dependency. The next post examines what it means to apply an SLO framework to an AI system: defining SLIs for recommendation accuracy, precision, and recall; setting error budgets for the AI-ops layer; and designing the automated quality gates that prevent a degrading LLM backend from silently undermining the operational decisions that depend on it.

Why vLLM autoscaling on Kubernetes breaks (and what to use instead)

Sonia — Mon, 15 Jun 2026 15:10:34 +0000

If you deploy vLLM on Kubernetes and reach for the standard HPA-on-CPU autoscaling, you will ship something that looks fine in testing and falls apart under real traffic.
Here is why, and what to do instead.

The problem: Kubernetes can't see your inference load

HPA scales on CPU and memory by default. Both are useless signals for LLM inference.
CPU stays low because the GPU does the work. A vLLM pod serving zero requests and one serving 100 show nearly identical CPU.
GPU memory stays constant because vLLM pre-allocates it for the KV cache at startup. It never moves, so it never triggers a scaling decision.
So under heavy load, your vLLM deployment looks idle to Kubernetes while requests pile up inside the engine and latency climbs. The scheduler has no idea anything is wrong.

The fix: autoscale on the metrics that reflect real load

The signals that matter live inside vLLM and are exported on its Prometheus endpoint:

vllm:num_requests_waiting : how many requests are queued.
vllm:gpu_cache_usage_perc : how full the KV cache is.

Wire these into KEDA, not HPA:

yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
  namespace: llm-inference
spec:
  scaleTargetRef:
    name: vllm-inference
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: vllm_queue_depth
      threshold: "10"
      query: |
        sum(vllm:num_requests_waiting) /
        count(kube_deployment_status_replicas_ready{deployment="vllm-inference"})
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: vllm_kv_cache
      threshold: "0.8"
      query: avg(vllm:gpu_cache_usage_perc)

The cooldownPeriod: 300 is not optional. vLLM cold start takes minutes. A short cooldown thrashes, scaling down then immediately back up, paying the cold-start cost on repeat.

Three more things that bite in production

Cold start. Pre-cache model weights on a ReadOnlyMany PVC. Downloading a 7B model from HuggingFace on every scale-up adds 2-5 minutes. Mounting from a pre-populated PVC cuts that to 30-60 seconds. Set a startupProbe with a high failureThresholdso Kubernetes doesn't kill the pod mid-load.
KV cache OOM. The most common vLLM crash. Size it with params_B × 2 for FP16 weights plus 25% for the cache, and set --gpu-memory-utilization 0.85 for headroom. 0.95 will OOM under concurrent load.
Preemption. When the KV cache fills, vLLM silently preempts older requests to make room. P99 latency can spike 8x. Alert on rate(vllm:num_preemptions_total[1m]) > 0.05 for 30s. It's the earliest warning you'll get.

The metric that matters most

Watch preemption rate. Standard Kubernetes monitoring (CPU, memory, restarts) tells you nothing about inference health. Preemption rate precedes the latency spikes your users actually feel. If you alert on one vLLM metric, alert on that one.

I wrote the full version with the GPU memory sizing rules, Spot vs On-Demand node strategy, cold-start mitigation including NVIDIA Dynamo Snapshot, the four metrics to monitor, and a production checklist over on our blog: [https://thegoodshell.com/vllm-kubernetes/]
What signals are you autoscaling LLM inference on? Curious if anyone has found something better than queue depth + KV cache.

K9s — The Terminal UI That Made Me Love K8s Again

Sohana Akbar — Mon, 15 Jun 2026 14:58:20 +0000

Let me be honest with you.

For the first two years of my Kubernetes journey, I was that person. You know the one. The person who says, “Kubernetes is amazing, but the CLI friction kills my flow.”

I had aliases for my aliases. I used kubectx and kubens religiously. I even built a messy bash script to tail logs from five pods at once. And still, debugging a rollout felt like doing surgery with oven mitts.

Then I found K9s.

And I swear to you: within ten minutes, I actually enjoyed managing my cluster again.

The Problem: Too Much Typing, Not Enough Seeing
Here’s the dirty secret of kubectl: it’s a single-purpose knife in a multi-course meal. Want to check pod status? kubectl get pods. Want to see logs? kubectl logs -f pod-name. Want to describe a deployment? New command. Scale it? Another command. Jump into a shell? You get the idea.

The cognitive load isn't the cluster—it’s the context switching between commands and namespaces.

K9s solves that by turning your terminal into a live, interactive cockpit.

What Is K9s, Really?
K9s is a Terminal UI (TUI) that watches your cluster in real time. It’s not a dashboard you open in a browser. It lives inside your terminal, respects your SSH config, uses your existing kubeconfig, and weighs essentially nothing.

But the magic isn't the tech specs. The magic is the ergonomics.

When you launch k9s, you don’t type --namespace. You don’t remember flags. You press : (colon) and type /pods, /deploy, /svc, or /ctx to switch contexts instantly.

Let me show you how my workflow changed.

Before vs. After K9s
Before (pure kubectl)
bash
kubectl get pods -n myapp

oh, pod is crashing

kubectl logs myapp-pod-7d8f9-abc -n myapp --tail=50

hmm, need to see events

kubectl describe pod myapp-pod-7d8f9-abc -n myapp | grep -A 5 Events

ok scale down

kubectl scale deploy/myapp -n myapp --replicas=0
That’s 4 commands, 2 copy-pastes, and one grep.

After (K9s)
k9s (enter)

: then ns (switch namespace to myapp)

Arrow keys to the crashing pod → press l (logs automatically stream)

Press d (describe) – reads like a man page, but instant

Press /deploy → select deployment → press s (scale) → type 0 → enter

No typing resource names. No looking up pod hashes. No --tail. No namespace typos.

That’s the difference between “managing YAML” and “piloting a cluster.”

The Features That Stole My Heart

Vim-like navigation (if you know, you know)
j/k to scroll, / to filter, : for commands, ? for help. It feels like home for anyone who lives in the terminal.
Logs with infinite scroll and live tail
Press l on any pod and watch logs update in real time. Press ctrl-s to save them to a file. Press esc to go back. No ctrl-c | kubectl logs dance.
Popeye integration for cluster health
K9s includes a built-in “Popeye” sanitizer. Press : then popeye and it scans your cluster for misconfigurations, deprecated APIs, and resource waste. It’s like a linter for your entire K8s setup.
Command mode for everything
Want to restart a deployment? Select it, press r. Port-forward? Select pod, press shift-f. Shell into a container? Select pod, press s. Everything is two keystrokes away.
Screens and skins
You can save multiple screens (pods + logs + events side by side) and customize the color theme. I use a purple-and-cyan theme that makes CrashLoopBackOff stand out like a warning light.

Real Talk: Is It Perfect?
No. Nothing is.

It’s not for scripting – obviously. You’ll still need kubectl in CI/CD.

Large clusters can lag – if you have 2,000 pods, initial load takes a few seconds.

The shortcut list is daunting at first. But after a week, you’ll be pressing ctrl-d to kill a pod without thinking.

But here’s the thing: I don’t want it to replace kubectl. I want it to replace fatigue.

How to Start (in 60 seconds)
bash

macOS

brew install k9s

Linux (via snap)

sudo snap install k9s

Or download from GitHub releases

https://github.com/derailed/k9s/releases

Then just run:

bash
k9s
That’s it. It reads your ~/.kube/config. It respects your current context. It just works.

The Moment I Knew I Was Hooked
We had a production incident. Three microservices failing. A ConfigMap typo. I needed to check logs from service A, events from namespace B, and scale down service C simultaneously.

Normally, I’d have five terminal tabs open, each with a different kubectl command running.

With K9s, I opened two splits in the same TUI: logs on the right, pod list on the left. I watched the error appear, fixed the ConfigMap in another window, and saw the pods restart in real time without refreshing anything.

My lead engineer looked over and said, “What the hell is that? Install it on my machine right now.”

That’s K9s.

Final Verdict
Kubernetes is powerful, but power without visibility is just chaos.

K9s doesn’t make K8s easier — it makes it visible, tactile, and fast. It turns a firehose of YAML and API calls into a dashboard that fits inside a single terminal window.

If you’re tired of typing kubectl get pods --all-namespaces | grep Pending, do yourself a favor.

Try K9s for one day.

You might just fall in love with K8s all over again.

Bonus tip: Add alias kk='k9s' to your .zshrc and thank me later.

Have you used K9s? What’s your favorite shortcut? Drop a comment below — I’ll trade you my skin config for yours. 🚀

Secrets Management Across Multi-Cloud Pipelines

Nerav Doshi — Mon, 15 Jun 2026 14:51:40 +0000

🛠️ Pipelines in the Wild #3

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

Secret management failures are invisible until they cause a production incident — start with RBAC and namespace isolation before the first workload goes live

Storing secrets in a central vault solves the sprawl problem but introduces a new failure mode: rotation lag between the vault and the namespace-level Kubernetes secret

The real unsolved problem is not technical — it is knowing who owns the approval and escalation path when a credential rotates at 2 AM across a multi-timezone team

The Story

The deployment had been running fine in dev for two days. Same manifests, same pipeline, same container images. We promoted to production and the pods went straight into ImagePullBackOff.

Not a misconfigured resource limit. Not a broken liveness probe. A pull secret that existed in the dev namespace and nowhere else.

The registry was internal. The credential was real. Nobody had thought to check whether the secret had been created in the production namespace — because it had been created ad hoc during initial testing, stored on a local notepad, and everyone assumed someone else had handled it for prod.

What followed was several hours of degraded production, a delayed platform release, and five or six people across multiple time zones working from memory and Slack threads with no runbook in sight. The fix, once identified, took minutes. Finding the fix took hours.

That incident was the starting point of a long education in secret management. The immediate problem was a missing pull secret in the wrong namespace. The real problem ran deeper — and it took an audit, an enterprise approval process, a failed secret rotation, and one very sharp observation from a more experienced engineer to understand what it actually was.

The Problem

In the early stages of a Kubernetes adoption, secrets are almost always an afterthought. The team is focused on getting workloads running, learning the platform, and delivering against commitments. Secrets get created when something fails, stored wherever is convenient, and recreated from memory the next time something breaks.

This works until it doesn't.

The failure mode is not just operational — a wrong namespace, a stale credential, a missed rotation. The deeper failure is structural. Kubernetes base64 encoding is not encryption. Any service account with read access to a namespace can retrieve every secret in that namespace and decode the values in seconds. Without RBAC, dev service accounts can read prod database credentials. Without namespace isolation, a misconfigured workload in one environment can inadvertently consume secrets intended for another.

Platform engineers moving into multi-cloud environments compound this problem. Each cloud has its own native secrets service. Each pipeline has its own credential requirements. Each environment has its own namespace structure. Without a deliberate architecture, secrets sprawl across notepads, environment variables, ConfigMaps used as secret storage, and Git commits that are very hard to fully expunge once they are pushed.

The incident cost was one day's delay on a significant platform release, discovered manually by a human checking on a deployment that had been quietly failing for hours. There was no alert. No monitor. No automated detection. Just someone who happened to look.

Why Existing Approaches Fall Short

Ad hoc secret creation per namespace

The natural first step. Create the secret where you need it, when you need it. Fast to start, impossible to maintain. Secrets diverge between environments, rotation becomes manual per namespace, and the source of truth is whoever created the secret last.

Kubernetes Secrets without RBAC

Kubernetes Secrets are base64 encoded, not encrypted at rest by default on vanilla Kubernetes. OpenShift 4.x enables etcd encryption for Secrets by default — but without RBAC, any pod's service account with namespace access can still read any secret in that namespace. In a shared cluster with dev and prod namespaces side by side, this is not a theoretical risk — it is a standing exposure that an audit will find immediately.

Cluster separation as a security boundary

Separating prod and dev onto different clusters contains blast radius but does not fix the underlying problem. Ad hoc secrets still get created. Rotation is still manual. Tribal knowledge still owns the recovery path. The incident can no longer cross environments, but within each environment, the same exposure exists.

Cloud-native secrets managers without a sync strategy

Centralizing secrets in a cloud-native vault is the right architectural move. But it introduces a new failure mode that most documentation does not cover: the sync gap. When a secret rotates in the vault, the namespace-level Kubernetes Secret object is a separate artifact. If the sync between vault and namespace fails — or if the pod is not restarted after a successful sync — the running workload is using a stale credential. The vault shows the rotation succeeded. The pod disagrees.

The Architecture

The diagram above proves one thing: secret management is a routing problem with two distinct failure points — the trust boundary between namespaces, and the sync gap between the central vault and the Kubernetes Secret object.

The architecture has three layers.

Layer 1 — Central Secrets Store

A cloud-native or self-hosted secrets manager holds the canonical value for every credential. Access to this layer is controlled by service account tokens scoped per environment. No developer has direct write access to production secrets in the central store. The CI/CD pipeline has read-only access, scoped to the secrets it needs for the environment it is deploying to. Human write access to prod secrets requires a break-glass process outside of automated rotation.

Layer 2 — Sync Operator

The External Secrets Operator (ESO) runs inside the cluster and watches for changes in the central store. When a rotation event occurs, ESO reconciles the namespace-level Kubernetes Secret objects. This is the critical seam. If the operator fails, is misconfigured, or runs behind its refresh interval, the Kubernetes secret is stale even though the vault value is current. ESO must be monitored and alerted on — it is a critical path dependency, not background infrastructure.

Layer 3 — Namespace Isolation with RBAC

Prod and dev namespaces are isolated with explicit RBAC. Service accounts are scoped to their namespace. The prod service account cannot read dev secrets. The dev service account cannot read prod secrets. This is enforced at the API server level, not by convention.

The rotation lag problem is architectural, not operational. A pod that started before a secret rotation uses the credential that was mounted at pod startup. Restarting the pod after a confirmed sync is the only way to guarantee the running workload is using the current credential. Without a process that enforces this, rotation and running workload credential state are eventually consistent at best.

How It Works: Step by Step

Prerequisites

OpenShift 4.12+ or Kubernetes 1.26+
Helm 3.x installed locally
A central secrets manager — this article covers AWS Secrets Manager (IRSA via STS), Azure Key Vault (Workload Identity), and HashiCorp Vault (Kubernetes auth)
Cluster-admin access to install the ESO operator and configure RBAC

Step 1 — Install the External Secrets Operator

# Add the External Secrets Operator Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

# Install ESO 0.10.0+ into its own namespace
# [AUTHOR TO VALIDATE] — confirm latest stable chart version before repo build
helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true \
  --version 0.10.0

Verify the operator is running before proceeding:

oc get pods -n external-secrets
# All pods should show Running status before applying any SecretStore or ExternalSecret

Step 2 — Create a SecretStore scoped to each namespace

A SecretStore is namespace-scoped. Prod and dev each get their own — they never share one. Choose the provider block that matches your environment.

AWS Secrets Manager — IRSA via STS

# prod-secretstore-aws.yaml
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: prod-secretstore
  namespace: prod
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1  # [AUTHOR TO VALIDATE] — set your region
      auth:
        jwt:
          serviceAccountRef:
            name: prod-workload-sa
            # This SA must carry the IAM role annotation — see Step 4

Annotate the service account with the IAM role ARN:

oc annotate serviceaccount prod-workload-sa \
  -n prod \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/prod-secrets-reader
  # [AUTHOR TO VALIDATE] — replace account ID and role name

The IAM role requires a trust policy scoped to the cluster OIDC provider and a permissions policy granting secretsmanager:GetSecretValue against specific secret ARNs — not *.

Azure Key Vault — Workload Identity

# prod-secretstore-azure.yaml
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: prod-secretstore
  namespace: prod
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://<YOUR-KEYVAULT-NAME>.vault.azure.net"
      # [AUTHOR TO VALIDATE] — replace with your Key Vault URL
      serviceAccountRef:
        name: prod-workload-sa
        # This SA must carry the Workload Identity annotation — see Step 4

Annotate the service account with the managed identity client ID:

oc annotate serviceaccount prod-workload-sa \
  -n prod \
  azure.workload.identity/client-id=<MANAGED_IDENTITY_CLIENT_ID>
  # [AUTHOR TO VALIDATE] — replace with your managed identity client ID

The managed identity needs the Key Vault Secrets User role scoped to the specific Key Vault — not the subscription. The pod spec also requires this label in the Deployment's pod template metadata:

labels:
  azure.workload.identity/use: "true"

HashiCorp Vault — Kubernetes Auth

Kubernetes auth is the recommended starting point for Vault in an OpenShift environment. It uses the pod's projected service account token to authenticate — no static credentials stored anywhere.

# prod-secretstore-vault.yaml
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: prod-secretstore
  namespace: prod
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      # [AUTHOR TO VALIDATE] — replace with your Vault server URL
      path: "secret"
      version: "v2"  # KV v2 is the current default secrets engine
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "prod-secret-reader"
          # [AUTHOR TO VALIDATE] — replace with your Vault role name
          serviceAccountRef:
            name: prod-workload-sa

Configure the Kubernetes auth backend on Vault once per cluster:

# Run against your Vault instance — not inside OpenShift
vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://<OPENSHIFT_API_SERVER>:6443"
  # [AUTHOR TO VALIDATE] — replace with your OpenShift API server URL

vault write auth/kubernetes/role/prod-secret-reader \
  bound_service_account_names=prod-workload-sa \
  bound_service_account_namespaces=prod \
  policies=prod-secrets-policy \
  ttl=1h

Create a minimal Vault policy scoped to the specific secret path — never use wildcards in prod:

# prod-secrets-policy.hcl
path "secret/data/prod/registry/pull-secret" {
  capabilities = ["read"]
}

Apply the SecretStore manifest for your provider:

oc apply -f prod-secretstore-aws.yaml    # if using AWS
oc apply -f prod-secretstore-azure.yaml  # if using Azure
oc apply -f prod-secretstore-vault.yaml  # if using Vault

Step 3 — Define an ExternalSecret to sync the pull secret

The ExternalSecret fetches individual credential fields from the vault and assembles them into a valid kubernetes.io/dockerconfigjson secret in the namespace. The template below works for all three providers — only the secretStoreRef name changes per provider.

# prod-pull-secret-external.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: registry-pull-secret
  namespace: prod
spec:
  refreshInterval: 1h
  # Note: 1h means up to 60 minutes rotation lag before the
  # namespace Secret reflects a vault change. Reduce for
  # time-sensitive credentials. Minimum recommended: 15m.
  secretStoreRef:
    name: prod-secretstore   # matches whichever SecretStore you applied in Step 2
    kind: SecretStore
  target:
    name: registry-pull-secret
    creationPolicy: Owner
    # Owner means ESO controls the lifecycle of this Secret.
    # If this ExternalSecret is deleted, the Secret is deleted with it.
    # Do not delete ExternalSecrets without understanding this behavior.
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "{{ .registryHost }}": {
                "username": "{{ .registryUsername }}",
                "password": "{{ .registryPassword }}",
                "auth": "{{ printf "%s:%s" .registryUsername .registryPassword | b64enc }}"
              }
            }
          }
  data:
    - secretKey: registryHost
      remoteRef:
        key: prod/registry/pull-secret    # [AUTHOR TO VALIDATE] — Vault path to your secret
        property: host                    # [AUTHOR TO VALIDATE] — field name for registry hostname
    - secretKey: registryUsername
      remoteRef:
        key: prod/registry/pull-secret
        property: username                # [AUTHOR TO VALIDATE] — field name for username
    - secretKey: registryPassword
      remoteRef:
        key: prod/registry/pull-secret
        property: password                # [AUTHOR TO VALIDATE] — field name for password

oc apply -f prod-pull-secret-external.yaml

Verify the sync completed and the Secret was created:

oc get externalsecret registry-pull-secret -n prod
# STATUS column must show: SecretSynced
# READY column must show: True

# Confirm the Secret exists and is correctly typed
oc get secret registry-pull-secret -n prod -o jsonpath='{.type}'
# Expected output: kubernetes.io/dockerconfigjson

If STATUS shows SecretSyncedError, check the ESO operator logs:

oc logs -n external-secrets \
  -l app.kubernetes.io/name=external-secrets \
  --tail=50

Step 4 — Apply RBAC to lock down namespace secret access

# prod-secret-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: prod
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
    resourceNames: ["registry-pull-secret"]
    # Scoped to the named secret only — not wildcard access to all secrets
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workload-secret-reader
  namespace: prod
subjects:
  - kind: ServiceAccount
    name: prod-workload-sa
    namespace: prod
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

oc apply -f prod-secret-rbac.yaml

This scopes the prod service account to read only the specific named secret it needs. Apply the equivalent for the dev namespace, scoped to dev secrets only. Neither service account should have cross-namespace access.

Step 5 — Reference the secret in your workload

# prod-deployment.yaml (relevant section)
spec:
  template:
    metadata:
      labels:
        azure.workload.identity/use: "true"  # include only if using Azure Workload Identity
    spec:
      imagePullSecrets:
        - name: registry-pull-secret
      serviceAccountName: prod-workload-sa
      containers:
        - name: app
          image: registry.internal/org/app:latest

Step 6 — Handle rotation explicitly

When a credential rotates in the central store, the ExternalSecret will re-sync within the refreshInterval. The running pod will not automatically pick up the new credential — it uses the value that was mounted at startup. A rollout restart is required after every confirmed sync.

# Confirm the sync has completed before restarting
oc get externalsecret registry-pull-secret -n prod
# Confirm: STATUS = SecretSynced and READY = True

# Restart the deployment to pick up the rotated credential
oc rollout restart deployment/app -n prod

# Verify the rollout completes cleanly
oc rollout status deployment/app -n prod

Add this as an explicit named step in your rotation runbook — not a footnote. It is not optional and it is not automatic.

Rollback consideration

If a rotation introduces a bad credential — wrong value, wrong format, access not yet propagated in the provider — roll back the deployment to the previous revision first, then investigate:

oc rollout undo deployment/app -n prod
oc rollout status deployment/app -n prod

Note that oc rollout undo rolls back the deployment configuration, not the secret value. If the vault value itself is wrong, rolling back the deployment buys time but does not fix the underlying problem. Correct the value in the vault first, wait for ESO to re-sync, then trigger a new rollout. Do not attempt to fix the secret in place while the deployment is actively failing.

Security and Operational Considerations

RBAC is the first thing to configure, not the last

Kubernetes Secrets are base64 encoded. Any service account with get or list access to secrets in a namespace can retrieve and decode every credential stored there. OpenShift 4.x enables etcd encryption for Secrets by default — vanilla Kubernetes does not. Verify your cluster's encryption at rest configuration before assuming the storage layer is protected. Apply Role and RoleBinding before the first secret is created in any namespace, and scope them to named resources, not wildcard access.

The sync operator is a critical dependency — treat it as one

Once ESO is part of your architecture it is a critical path component. Monitor it. Alert on sync failures. ESO exposes the externalsecret_sync_calls_error metric — wire this to your alerting platform. A silent sync failure means your workload is running with a stale credential and you will not know until something breaks.

# Check ESO sync status across all ExternalSecrets in a namespace
oc get externalsecret -n prod
# Any STATUS other than SecretSynced needs immediate investigation

The central secrets store itself needs RBAC

If the engineering team has full read/write access to the secrets manager, the blast radius of a compromised account is the entire vault. Separate write access from read access. Human write access to prod secrets should require a break-glass process outside of automated rotation. Document who holds that access and review it quarterly.

creationPolicy: Owner has a destructive side effect

When ESO owns a Secret's lifecycle, deleting the ExternalSecret deletes the Secret with it. In a multi-team environment, a developer deleting what appears to be a stale or misconfigured ExternalSecret will drop the credential from the namespace immediately. Make sure your team understands this behavior before granting delete access to ExternalSecret resources.

Define the rotation approval path before you need it

This is the thing that documentation does not cover. When a credential rotates at 2 AM in a multi-cloud environment with a team spread across time zones, who has the authority to approve the rotation in the central store? Who runs the oc rollout restart? Who confirms the rollout completed cleanly and signs off that prod is healthy?

Write this down before it happens. Name the people, define the escalation path, and put it somewhere a new team member can find it without a Slack thread.

Audit logs need active review, not passive collection

Most secrets managers generate audit logs for every read and write operation. These logs are only useful if someone is reviewing them. Wire secret access events into your SIEM or log aggregator and create alerts for anomalous patterns — unexpected reads, access from unrecognized service accounts, bulk secret reads that do not match a known pipeline run.

What Breaks at Scale

Rotation lag multiplies across namespaces

With one namespace and one workload, a manual oc rollout restart after rotation is manageable. With ten namespaces, thirty deployments, and a rotation event that cascades across dependent credentials, it does not scale. You need a rotation event handler — a pipeline step or operator webhook that triggers a rolling restart of affected workloads automatically after a confirmed sync. This is not a day-one problem. It becomes one at day ninety when the first coordinated rotation happens and nobody has automated the downstream restart.

Cross-cloud secret identity is unsolved by most teams

In a true multi-cloud deployment — workloads on AWS, Azure, and an on-premises OpenShift cluster all consuming secrets — each cloud has its own identity model for authenticating to the central store. The pipeline service account on AWS uses an IAM role. The OpenShift cluster on-premises uses a service account token projected via OIDC. Keeping these identity bindings consistent, rotated, and auditable across three clouds is an operational challenge that most tooling handles partially at best.

The 2 AM problem at scale

With one team and one cluster, Slack and tribal knowledge is expensive but survivable. With multiple teams, multiple clusters, and a secrets manager that is a shared dependency, a rotation failure at 2 AM is a cross-team incident. The human routing problem — who owns the approval, who runs the restart, who confirms health across environments — does not get easier with scale. It gets harder. The runbook is not optional at this point. It is the difference between a thirty-minute recovery and a three-hour incident bridge.

Regulated environments add approval gates to the rotation path

In financial services or healthcare environments, credential rotation often requires a change approval before the rotation runs, not just after. This means the automated rotation flow needs to integrate with your change management tooling — a ServiceNow ticket, a Jira issue, an approval gate in the pipeline. The technical implementation is straightforward. Getting it through the approval process for a new tooling integration is the actual work.

What I'd Do Differently

Start with encrypted Git secrets before the first workload enters a namespace. Not as the end state — as the minimum bar that establishes the habit. Leaked Git history is incredibly difficult to clean completely. An encrypted Git secret is easy to upgrade to an enterprise vault later. And it builds a security-first mindset within the engineering team from day one, before there is an incident to justify it.

The harder lesson: define the rotation runbook before the first secret is created in prod, not after the first rotation failure. The technical architecture is the easy part. Knowing who clicks approve at 2 AM is what breaks in production — and no documentation covers it because it is a people and process problem, not a Kubernetes problem.

Quick Recap

RBAC first, secrets second — configure namespace-level RBAC before the first secret is created; base64 encoding is not access control, and etcd encryption at rest is not enabled by default on vanilla Kubernetes
The sync gap is the rotation failure — a successful rotation in your central vault does not mean running pods are using the new credential; an explicit rollout restart after a confirmed ESO sync is required and must be in the runbook
Secret management is a human routing problem — the technical architecture is solvable; who owns the 2 AM approval and the cross-timezone escalation path is what breaks in production

GitHub Repo

Full implementation with working manifests for all three providers, RBAC templates, and rotation runbook:

[PLACEHOLDER — repo content in progress: pipelineandprompts-labs/secrets-management-multi-cloud]

What's Next?

Secret management is one half of the pipeline security conversation. The other half is what happens when the pipeline itself is the attack surface — supply chain security, signed commits, and verifying that the image running in prod is exactly the image that passed your tests.

Next in Pipelines in the Wild: Pipeline Supply Chain Security — Signing, Provenance, and Why Your CI/CD Pipeline is a Target.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Found this useful? Share it with the engineer on your team who is still creating secrets manually — and forward it to whoever owns the rotation runbook. If there is no rotation runbook, this article is for them.

MCP Server Architecture for Platform Teams — Giving AI Live Access to Your Infrastructure

Nerav Doshi — Mon, 15 Jun 2026 13:43:44 +0000

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

AI in the Stack #3

⚡ Byte Size Summary

MCP (Model Context Protocol) is the standard that lets AI agents interact with external systems — your cluster, your observability stack, your ticketing system — without bespoke integration code for every tool.

MCP directly addresses AI hallucination and 2AM incident response by grounding AI answers in live system state. It does not solve tribal knowledge alone — that needs RAG alongside it.

This article covers the production-grade architecture: what MCP servers are, how to design them for platform engineering use cases, and what you need to get right before running them anywhere near production.

In logistics, the hardest problems rarely come from missing data.

They come from disconnected systems.

The warehouse knows one thing. The transportation management system knows another. Inventory systems lag behind reality by hours. Operators work around the gaps manually — copying numbers between screens, making calls to confirm what the system should already know, carrying context in their heads because no single system has the full picture.

I spent years watching intelligent people solve problems that should not have existed, because the systems around them were designed to optimise locally rather than coordinate globally. The data was there. The capability was there. The coordination layer was not.

Modern infrastructure operations feel surprisingly similar.

Your Kubernetes cluster knows the state of every pod. Your observability stack knows the error rates and latency trends. Your ticketing system knows what changes were deployed in the last 24 hours. Your CI/CD pipeline knows what is currently in flight. And your AI assistant — the tool you are increasingly asking to help you reason about incidents — knows none of it, unless you paste it in manually.

Model Context Protocol is the coordination layer that changes this. Not by giving AI access to everything at once, but by giving it a structured, auditable, controlled way to request the context it needs, from the systems that have it, at the moment it needs it.

That is what this article is about.

What MCP Actually Is

Model Context Protocol (MCP) is an open standard, introduced by Anthropic, that defines how AI models communicate with external tools and data sources. Think of it as a common language that sits between an AI assistant and the systems it needs to interact with.

Before MCP, every AI integration was bespoke. You wanted your LLM to query your Kubernetes cluster? Write a custom function. You wanted it to check PagerDuty? Write another one. You wanted it to search your runbooks and open a Jira ticket? Three separate integrations, all maintained independently, all breaking in different ways when APIs change.

MCP replaces that with a standard. An MCP server exposes a set of tools — defined capabilities the AI can invoke — plus resources — data it can read. The AI client (Claude, Cursor, any MCP-compatible host) discovers what tools are available, decides which to call based on the user's question, calls them, and incorporates the results into its response.

The AI does not have direct access to your systems. It has access to an MCP server that mediates that access. That distinction matters enormously for security and governance — which is why this article spends as much time on architecture as on implementation.

Why Platform Engineers Should Care

The RAG pipeline from Article 02 was useful for static knowledge — runbooks, documentation, past incident reports. MCP is useful for live state.

When an engineer asks "what is causing the latency spike in the payments service right now?" — that is not a runbook question. It requires current pod status, recent deployment events, live error rates, and possibly the last three alerts that fired. None of that lives in a document. All of it lives in systems your MCP server can reach.

The distinction between what MCP solves and what it does not matters before you design anything.

AI hallucination — yes, directly. Hallucination happens when an LLM answers from training data instead of ground truth. MCP forces the AI to retrieve live, authoritative state before responding. It does not eliminate hallucination entirely — an LLM can still misinterpret what it retrieves — but it directly attacks the root cause for infrastructure questions.

2AM incidents — yes, directly. This is the primary operational use case. Instead of an engineer manually checking five systems in sequence while half-asleep, an AI with MCP access can pull pod status, recent events, and active alerts in a single query and reason across all of it simultaneously. Speed and context at the moment they are hardest to find.

Too many dashboards — partially. MCP does not reduce the number of dashboards in your environment. It gives an AI a way to query across the systems those dashboards represent, so an engineer asks one question instead of navigating five screens. The dashboards still exist. You stop having to drive them manually during an incident.

Tribal knowledge — not alone. MCP surfaces what your systems know. It does not surface what your team knows — the undocumented context that lives in people's heads, the runbook that exists nowhere in any system, the reason a service is named what it is. That is a RAG problem. The combination of RAG (for historical and human knowledge) and MCP (for live system state) is where the tribal knowledge gap actually starts to close. Neither alone is sufficient.

An AI that can read your runbooks and query your cluster simultaneously is a meaningful operational tool. An AI that can only do one of those things is a limited one.

MCP Server Architecture for Platform Engineering

A production-grade MCP server for a platform team has four layers:

Every tool invocation travels this path: the AI client sends a request, the Auth Gateway validates identity before anything reaches your infrastructure, the MCP server processes it through governance and audit controls, and the Kubernetes API Server enforces access policy independently of the application layer. Two enforcement gates — not one. That is the architecture the implementation sections below are built around.

The four layers in code:

Layer 1 — Governance First

Before writing a single tool definition, decide and enforce these three things:

Read-only by default. Every tool that touches production infrastructure should be read-only unless you have explicitly designed the write path with human approval steps. An MCP server that can kubectl delete anything is an incident waiting to happen. Start with read, earn trust, expand deliberately.

Audit logging. Every tool call should be logged with: timestamp, tool name, input parameters, calling session identity, and response status. This is your audit trail when something goes wrong. It is also how you demonstrate to your security team that AI is not a black box.

Rate limiting. An AI in an agentic loop can call tools hundreds of times in seconds. Without rate limiting, a runaway agent can exhaust your Kubernetes API quota, spam your ticketing system, or trigger alert storms in your observability stack. Set per-session and per-tool limits before you deploy.

Layer 2 — Backend Clients

The MCP server needs clients for each system it connects to. Keep these thin — their job is to call APIs and return structured data, not to contain business logic.

For a Kubernetes-connected MCP server, using the official kubernetes Python client:

# k8s_client.py
from kubernetes import client, config
from typing import Optional

class KubernetesClient:
    def __init__(self, in_cluster: bool = False):
        if in_cluster:
            config.load_incluster_config()
        else:
            config.load_kube_config()
        self.v1 = client.CoreV1Api()
        self.apps_v1 = client.AppsV1Api()

    def get_pod_status(self, namespace: str, pod_name: str) -> dict:
        pod = self.v1.read_namespaced_pod(name=pod_name, namespace=namespace)
        return {
            "name": pod.metadata.name,
            "namespace": pod.metadata.namespace,
            "phase": pod.status.phase,
            "conditions": [
                {"type": c.type, "status": c.status, "reason": c.reason}
                for c in (pod.status.conditions or [])
            ],
            "container_statuses": [
                {
                    "name": cs.name,
                    "ready": cs.ready,
                    "restart_count": cs.restart_count,
                    "state": str(cs.state)
                }
                for cs in (pod.status.container_statuses or [])
            ]
        }

    def list_failing_pods(self, namespace: Optional[str] = None) -> list[dict]:
        if namespace:
            pods = self.v1.list_namespaced_pod(namespace=namespace)
        else:
            pods = self.v1.list_pod_for_all_namespaces()

        failing = []
        for pod in pods.items:
            if pod.status.phase not in ("Running", "Succeeded"):
                failing.append({
                    "name": pod.metadata.name,
                    "namespace": pod.metadata.namespace,
                    "phase": pod.status.phase,
                    "reason": pod.status.reason
                })
        return failing

    def get_recent_events(self, namespace: str, limit: int = 20) -> list[dict]:
        events = self.v1.list_namespaced_event(
            namespace=namespace,
            limit=limit
        )
        return [
            {
                "type": e.type,
                "reason": e.reason,
                "message": e.message,
                "involved_object": e.involved_object.name,
                "count": e.count,
                "last_timestamp": str(e.last_timestamp)
            }
            for e in sorted(
                events.items,
                key=lambda x: x.last_timestamp or "",
                reverse=True
            )
        ]

Layer 3 — Tool Definitions

This is the layer the AI interacts with directly. Tool descriptions are not just documentation — they are what the LLM reads to decide whether to call the tool and how to format its inputs. Write them precisely.

# tools.py
from mcp.server import Server
from mcp.types import Tool, TextContent
import json
import logging

from k8s_client import KubernetesClient
from audit import log_tool_call

logger = logging.getLogger(__name__)
k8s = KubernetesClient(in_cluster=False)  # Set True when running inside the cluster


def register_tools(server: Server):

    @server.list_tools()
    async def list_tools():
        return [
            Tool(
                name="get_pod_status",
                description=(
                    "Get the current status of a specific Kubernetes pod, including phase, "
                    "readiness conditions, container states, and restart counts. "
                    "Use this when investigating why a specific pod is unhealthy or not ready."
                ),
                inputSchema={
                    "type": "object",
                    "properties": {
                        "namespace": {
                            "type": "string",
                            "description": "The Kubernetes namespace the pod is in"
                        },
                        "pod_name": {
                            "type": "string",
                            "description": "The exact name of the pod"
                        }
                    },
                    "required": ["namespace", "pod_name"]
                }
            ),
            Tool(
                name="list_failing_pods",
                description=(
                    "List all pods that are not in Running or Succeeded state across the cluster "
                    "or within a specific namespace. Use this as a first step when an incident "
                    "is reported and you need to identify which pods are affected."
                ),
                inputSchema={
                    "type": "object",
                    "properties": {
                        "namespace": {
                            "type": "string",
                            "description": "Optional: filter to a specific namespace"
                        }
                    }
                }
            ),
            Tool(
                name="get_recent_events",
                description=(
                    "Retrieve recent Kubernetes events for a namespace, ordered by most recent first. "
                    "Events capture warnings, errors, and state changes. Use this to understand "
                    "what happened in the cluster leading up to an issue."
                ),
                inputSchema={
                    "type": "object",
                    "properties": {
                        "namespace": {
                            "type": "string",
                            "description": "The namespace to retrieve events from"
                        },
                        "limit": {
                            "type": "integer",
                            "description": "Maximum number of events to return (default 20)",
                            "default": 20
                        }
                    },
                    "required": ["namespace"]
                }
            )
        ]

    @server.call_tool()
    async def call_tool(name: str, arguments: dict):
        log_tool_call(tool=name, inputs=arguments)  # Always audit first

        try:
            if name == "get_pod_status":
                result = k8s.get_pod_status(
                    namespace=arguments["namespace"],
                    pod_name=arguments["pod_name"]
                )
            elif name == "list_failing_pods":
                result = k8s.list_failing_pods(
                    namespace=arguments.get("namespace")
                )
            elif name == "get_recent_events":
                result = k8s.get_recent_events(
                    namespace=arguments["namespace"],
                    limit=arguments.get("limit", 20)
                )
            else:
                return [TextContent(type="text", text=f"Unknown tool: {name}")]

            return [TextContent(type="text", text=json.dumps(result, indent=2))]

        except Exception as e:
            logger.error(f"Tool {name} failed: {str(e)}")
            return [TextContent(type="text", text=f"Tool execution failed: {str(e)}")]

Layer 4 — Transport and Auth

MCP supports two transport modes:

stdio — the server runs as a subprocess of the AI client. Simple, local, no network exposure. Right for developer workstations and local tooling.

HTTP with SSE (Server-Sent Events) — the server runs as a persistent service, reachable over the network. Required for shared team tooling, remote access, and running inside a cluster. For production deployments, SSE transport with mutual TLS (mTLS) is the hardened path; API key authentication is acceptable for internal cluster traffic with network policy controls in place.

For a platform team MCP server running on Kubernetes:

# main.py
import asyncio
import logging
from mcp.server import Server
from mcp.server.sse import SseServerTransport
from starlette.applications import Starlette
from starlette.routing import Route
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from tools import register_tools

logging.basicConfig(level=logging.INFO)

server = Server("platform-mcp")
register_tools(server)


class APIKeyMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        api_key = request.headers.get("X-API-Key")
        if api_key != EXPECTED_API_KEY:  # Load from env, not hardcoded
            from starlette.responses import JSONResponse
            return JSONResponse({"error": "Unauthorised"}, status_code=401)
        return await call_next(request)


transport = SseServerTransport("/messages")

async def handle_sse(request):
    async with transport.connect_sse(
        request.scope, request.receive, request._send
    ) as streams:
        await server.run(
            streams[0], streams[1], server.create_initialization_options()
        )

app = Starlette(
    routes=[Route("/sse", endpoint=handle_sse)],
    middleware=[Middleware(APIKeyMiddleware)]
)

Kubernetes Deployment

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: platform-mcp-server
  namespace: platform-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: platform-mcp-server
  template:
    metadata:
      labels:
        app: platform-mcp-server
    spec:
      serviceAccountName: platform-mcp-sa  # Read-only SA — see RBAC below
      containers:
        - name: mcp-server
          image: your-registry/platform-mcp:latest
          ports:
            - containerPort: 8080
          env:
            - name: MCP_API_KEY
              valueFrom:
                secretKeyRef:
                  name: platform-mcp-secrets
                  key: api-key
---
# k8s/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: platform-mcp-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "events", "namespaces", "nodes"]
    verbs: ["get", "list", "watch"]   # Read-only — no create, update, delete
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: platform-mcp-reader-binding
subjects:
  - kind: ServiceAccount
    name: platform-mcp-sa
    namespace: platform-tools
roleRef:
  kind: ClusterRole
  name: platform-mcp-reader
  apiGroup: rbac.authorization.k8s.io

The RBAC configuration enforces the governance constraint at the Kubernetes level — not just in application code. Even if a bug in the tool definitions allowed a write operation to reach the Kubernetes client, the service account has no permission to execute it.

Defence in depth. Not one gate — two.

What This Unlocks

With a platform MCP server running, a Claude-powered assistant can handle questions like these using live cluster data:

"What pods are failing in the payments namespace right now?" → calls list_failing_pods
"Why did the checkout service restart three times this morning?" → calls get_pod_status + get_recent_events
"Is there anything unusual happening across the cluster before I deploy?" → calls list_failing_pods across all namespaces

This is the coordination layer the opening story was pointing at. In logistics, the fix for disconnected systems was never better dashboards — it was a shared integration layer that let every system speak to every other system through a common protocol. MCP is that layer for AI and infrastructure.

Combined with the RAG pipeline from Article 02, the same assistant can cross-reference live cluster state against your runbooks — returning answers grounded in documentation and informed by current reality simultaneously. That is the operational use case MCP was built for.

What to Build Next

The server in this article covers Kubernetes read operations. The natural extensions, covered in the GitHub repo, are:

Prometheus integration — add a get_metrics tool that queries PromQL (Prometheus Query Language) and returns current error rates and latency percentiles
PagerDuty integration — add get_active_incidents and get_recent_alerts tools
Write operations with human approval — a restart_pod tool that creates a Jira ticket and waits for human sign-off before executing; this is the governance pattern that makes agentic write operations safe in production

The write operation pattern — where the AI prepares an action, a human approves it, and the MCP server executes — is covered in Article 05 of this series.

What's Next

Article 04 — Prompt Versioning in Production: Treat Prompts Like Infrastructure Artifacts

System prompts are configuration. Changing them without version control, testing, or rollback strategy is the same mistake engineers made with infrastructure before Terraform existed. Next: how to version, test, and deploy prompts with the same discipline you apply to everything else in your stack.

How Do You Integrate Penetration Testing into CI/CD?

varun varde — Mon, 15 Jun 2026 12:50:55 +0000

Modern software delivery pipelines can deploy code dozens or even hundreds of times per day. Traditional penetration testing models, where security teams perform assessments quarterly or before major releases, simply cannot keep pace.

Attackers do not wait for the next security review.

Every pull request, dependency update, infrastructure change, or container image introduces potential risk. Integrating penetration testing into CI/CD enables organizations to identify vulnerabilities before they reach production.

The goal is not replacing human penetration testers. The goal is automating everything that can be automated so security experts can focus on complex attack paths and business logic flaws.

Understanding Security Testing Layers in CI/CD

Security testing is often misunderstood because multiple categories overlap.

Testing Type	Purpose
SAST	Analyze source code
SCA	Detect vulnerable dependencies
DAST	Test running applications
IAST	Runtime security analysis
Penetration Testing	Simulate attacker behavior

Penetration testing combines elements of all these approaches.

A mature CI/CD pipeline continuously performs automated penetration testing while reserving manual testing for sophisticated attack scenarios.

Designing a Security-First CI/CD Architecture

A security-centric pipeline typically looks like:

Developer Commit
      ↓
Pre-Commit Security Checks
      ↓
Pull Request Validation
      ↓
Build Stage
      ↓
Container Security Scan
      ↓
Infrastructure Validation
      ↓
Deploy to Staging
      ↓
Automated Penetration Testing
      ↓
Security Gate
      ↓
Production Deployment

Each stage eliminates vulnerabilities before they become more expensive to fix.

Stage 1: Pre-Commit Security Controls

The cheapest vulnerability is the one that never reaches Git.

Secret Detection

Install TruffleHog or Gitleaks before code reaches the repository.

repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.20.0
  hooks:
    - id: gitleaks

Developer installation:

pip install pre-commit

pre-commit install

Now every commit is automatically scanned.

Dependency Security Validation

Use dependency auditing tools.

Python

pip install pip-audit

pip-audit

Node.js

npm audit --production

govulncheck ./...

Stage 2: Automated Penetration Testing in Pull Requests

Pull requests provide the earliest opportunity to validate attack surfaces.

Authentication Testing

Example automated API validation:

import requests

response = requests.get(
    "https://staging.example.com/api/admin"
)

assert response.status_code in [401,403]

This simple test catches accidental authorization bypasses.

Authorization Validation

Testing role separation:

admin_token = get_admin_token()
user_token = get_user_token()

admin = requests.get(
    endpoint,
    headers={"Authorization": f"Bearer {admin_token}"}
)

user = requests.get(
    endpoint,
    headers={"Authorization": f"Bearer {user_token}"}
)

assert admin.status_code == 200
assert user.status_code == 403

These tests often discover privilege escalation vulnerabilities before release.

Stage 3: Dynamic Application Security Testing (DAST)

DAST evaluates a running application exactly as attackers would.

OWASP ZAP Automation

Deploy the application into a temporary staging environment.

GitHub Actions example:

- name: OWASP ZAP Scan
  uses: zaproxy/action-full-scan@v0.10.0
  with:
    target: "https://staging.example.com"

Baseline Scan

docker run -t owasp/zap2docker-stable zap-baseline.py \
  -t https://staging.example.com

Findings may include:

Missing security headers
XSS exposure
Cookie weaknesses
Directory traversal
Information disclosure

Stage 4: API Penetration Testing Automation

APIs have become the primary attack surface.

OpenAPI Security Testing

Tools like Schemathesis can automatically generate attack cases.

schemathesis run openapi.yaml \
  --base-url=https://staging.example.com

Generated tests include:

Invalid inputs
Boundary conditions
Injection attempts
Authentication failures

Fuzz Testing

schemathesis run \
  openapi.yaml \
  --checks all

This uncovers hidden edge cases frequently missed by developers.

Stage 5: Container and Kubernetes Penetration Testing

Containers introduce a different attack surface.

Container Image Scanning

Trivy example:

trivy image myapp:latest

CI integration:

- name: Container Scan
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myapp:latest
    severity: HIGH,CRITICAL

Kubernetes Security Assessment

Using Kubescape:

kubescape scan framework nsa

Checks include:

Privileged containers
Host networking
Excessive capabilities
Missing security contexts

Stage 6: Infrastructure Penetration Testing

Infrastructure is code. It should be tested like code.

Terraform Security Validation

Checkov example:

checkov -d terraform/

GitHub Actions:

- name: Terraform Security Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: terraform

Cloud Configuration Testing

AWS example:

prowler aws

Findings include:

Public S3 buckets
Weak IAM policies
Missing encryption
Excessive permissions

Integrating Security Gates into CI/CD

Not every vulnerability should block deployment.

A practical policy:

Severity	Action
Critical	Fail build
High	Fail production deployment
Medium	Ticket creation
Low	Backlog

GitHub Actions gate:

- name: Fail on Critical Issues
  run: |
    if [ "$CRITICAL_FINDINGS" -gt 0 ]; then
      exit 1
    fi

Reporting, Dashboards, and Vulnerability Management

Security data scattered across tools creates chaos.

Centralize findings.

Popular platforms:

DefectDojo
Security Hub
Grafana
ELK Stack

Example DefectDojo import:

curl -X POST \
  -H "Authorization: Token $TOKEN" \
  -F "file=@zap-report.xml" \
  https://defectdojo.example.com/api/v2/import-scan/

Building a Production-Ready DevSecOps Pipeline

Complete GitHub Actions workflow:

name: Continuous Security Testing

on:
  pull_request:
  push:

jobs:

  sast:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Semgrep
        uses: returntocorp/semgrep-action@v1

  dependency-scan:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Trivy FS
        run: trivy fs .

  container-scan:
    runs-on: ubuntu-latest

    steps:
      - name: Build Image
        run: docker build -t app:${{ github.sha }} .

      - name: Scan Image
        run: trivy image app:${{ github.sha }}

  iac-scan:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Checkov
        run: checkov -d terraform/

  dast:
    needs:
      - sast
      - dependency-scan
      - container-scan
      - iac-scan

    runs-on: ubuntu-latest

    steps:
      - name: OWASP ZAP
        uses: zaproxy/action-full-scan@v0.10.0
        with:
          target: https://staging.example.com

This provides comprehensive automated penetration testing from commit to deployment.

Common Mistakes and Lessons Learned

Treating DAST as Penetration Testing

DAST is valuable but incomplete. Human attackers exploit business logic flaws that scanners cannot detect.

Running Every Scan on Every Commit

Excessive scanning creates developer fatigue.

Use:

Fast scans on pull requests
Full penetration testing on staging
Deep assessments before production

Ignoring Authentication Testing

Many breaches result from authorization flaws rather than software vulnerabilities.

Focus heavily on:

RBAC validation
Token abuse testing
API authorization checks

Failing to Prioritize Findings

Thousands of low-risk findings provide little value.

Security teams should prioritize:

Critical
High
Exploitable
Internet-facing

Everything else comes later.

Integrating penetration testing into CI/CD transforms security from a periodic activity into a continuous engineering practice. By combining pre-commit validation, SAST, dependency scanning, container assessment, infrastructure testing, API security analysis, and automated DAST, organizations can identify vulnerabilities at the earliest possible stage.

The strongest DevSecOps programs do not rely on a single security tool. They build layered defenses throughout the entire software delivery lifecycle, ensuring that every commit, build, deployment, and infrastructure change is evaluated through an attacker's lens before it reaches production.

Network Policies with Calico: Default Deny and Namespace Isolation

Guatu — Mon, 15 Jun 2026 12:38:04 +0000

A default-deny NetworkPolicy is five lines of spec. Those five lines will also kill DNS resolution for every pod they select, because an egress deny blocks UDP packets to kube-dns just as happily as it blocks the traffic you were actually worried about. The distance between "I understand network policies" and "I rolled out default deny without an outage" is mostly three blind spots: DNS, your ingress controller, and admission webhooks.

Out of the box, Kubernetes runs a flat pod network. Every pod can open a connection to every other pod in the cluster, across namespaces, no questions asked. If you've already done the work of building least-privilege service accounts, a flat network is the same problem one layer down: identity is locked tight while the network is wide open. This post is about closing that gap with Calico on a bare-metal cluster (K8s 1.31, Calico 3.x), in an order that doesn't take the cluster down while you do it.

One prerequisite worth stating plainly: the NetworkPolicy API objects exist in every cluster, but they do nothing unless your CNI enforces them. Calico does. If you're on a CNI without policy support, you can apply these manifests all day and traffic flows anyway, which is its own special category of false confidence.

The rollout that looks right and isn't

The tempting approach goes like this: write one default-deny policy, template it across every namespace, apply, done. Security checkbox ticked before lunch.

Here's the policy everyone starts with:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}        # selects every pod in the namespace
  policyTypes:
    - Ingress
    - Egress

The empty podSelector selects all pods, and listing both policy types makes them isolated in both directions. Correct, minimal, and the moment it lands cluster-wide, three things break in a predictable order.

Failure one: DNS dies first, and it dies slowly

Every pod in a selected namespace loses the ability to resolve names, because queries to kube-dns in kube-system are egress traffic like any other. The nasty part is the failure mode. Connections to a denied endpoint fail fast with a timeout you'll notice. DNS failures look different: each lookup waits out a 5-second timeout per attempt, multiplied by the search domain list your ndots config generates. Apps get slow before they get broken, which sends you debugging application performance instead of network policy. I wrote about how the search domain expansion amplifies this in the ndots:5 post; default deny turns every one of those expanded lookups into a 5-second black hole.

Failure two: your ingress controller can't reach anything

Traffic from Traefik or ingress-nginx to your backend pods is just pod-to-pod traffic crossing a namespace boundary. Default deny on the application namespace blocks it, and every service behind the ingress starts returning 502s and 504s. The application pods are healthy, the Service endpoints are populated, readiness probes pass (kubelet probes come from the node, and Calico permits them). Everything looks green except the part where users reach it. This also bites cert-manager: an HTTP-01 challenge needs the ingress controller to reach the temporary solver pod, so default deny can silently stall certificate issuance long after the initial rollout.

Failure three: the webhook deadlock

This is the one that turns a degraded cluster into a stuck one. Admission webhooks (Kyverno, cert-manager's webhook, anything with a ValidatingWebhookConfiguration) receive calls from the API server. Deny ingress to the webhook pod and those calls time out. With failurePolicy: Fail, the API server now rejects the operations that webhook gates, and the trap closes: the NetworkPolicy you're trying to apply to fix the problem is itself an API operation that flows through admission. You're locked out of the fix by the thing you broke.

It gets worse if the policies are managed by automation. With a Kyverno generate rule or a GitOps controller syncing the policy, deleting the offending NetworkPolicy by hand buys you a few seconds before it's regenerated. You end up playing whack-a-mole against your own reconciliation loop while the cluster burns. The escape hatch is to pause the automation first (scale down Kyverno, disable ArgoCD auto-sync for that app), then remove the policy.

A detail that matters here: API server traffic to webhooks often originates from the control plane host network, not from a pod you can match with a podSelector. Allowing it means an ipBlock rule for your control plane CIDR, or excluding webhook namespaces from default deny entirely. I do the latter.

A rollout order that works

The fix for all three failures is the same discipline: never apply a deny you haven't already written the allows for, and never apply it wider than you can watch.

Step 1: one namespace, not the cluster

Pick a single application namespace with low blast radius. Resist the urge to start cluster-wide; the whole point of the first namespace is to discover the flows you forgot existed. kubectl get networkpolicy -A should stay boring while you learn.

Step 2: the baseline trio

Default deny ships as a set of three policies applied together, in one kubectl apply -f of one directory. The deny:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

The DNS allow, which goes everywhere the deny goes, no exceptions:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Both protocols matter. DNS falls back to TCP for large responses, and an egress rule that only allows UDP produces intermittent failures that are miserable to track down.

The intra-namespace and ingress-controller allow:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-baseline-ingress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    # any pod in this same namespace
    - from:
        - podSelector: {}
    # everything in the ingress controller's namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress

That kubernetes.io/metadata.name label is the load-bearing trick here. Since K8s 1.22, every namespace carries it automatically with its own name as the value, which gives you a stable way to select namespaces without inventing and maintaining your own labeling scheme.

With the trio applied, check behavior from inside the namespace before moving on:

# throwaway pod inside the locked-down namespace
kubectl -n team-a run probe --rm -it --image=busybox:1.36 --restart=Never -- sh
# inside the pod:
nslookup kubernetes.default                                  # should answer instantly
wget -qO- -T 2 http://api.team-b.svc.cluster.local           # should time out

Fast DNS plus a slow, eventually-failing cross-namespace connection is the signature of a healthy baseline. Instant DNS failure means the allow-dns policy didn't land; an instant cross-namespace success means the deny didn't.

Step 3: log before you deny

Calico's Log rule action is the visibility tool the vanilla NetworkPolicy API doesn't have. Before tightening further, I put a logging policy behind the allows so I can see what the deny is about to catch:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: log-unmatched
spec:
  order: 4000                    # evaluated after everything else
  namespaceSelector: projectcalico.org/name == 'team-a'
  types: [Ingress, Egress]
  ingress:
    - action: Log
  egress:
    - action: Log

With the iptables dataplane, Log uses the kernel LOG target, so dropped-candidate packets show up in the kernel log with a calico-packet: prefix (configurable via logPrefix in FelixConfiguration):

journalctl -k --grep calico-packet

Two caveats. Kernel logging is noisy, so treat this as a diagnostic you enable for hours, not a permanent fixture. And the eBPF dataplane doesn't support the Log action, so if you've switched dataplanes this tool isn't available.

This step is where "set and forget" turns into something closer to auditing. Run a logging policy for a day against a namespace before enforcing, and you find the flows nobody documented: the metrics scraper, the backup job, the sidecar that phones a service in another namespace.

One class of flow deserves special mention: anything running with hostNetwork: true. Node-level monitoring agents and some bare-metal ingress deployments source their traffic from the node's IP, not a pod IP, so podSelector and namespaceSelector rules never match them. If scraping or health checks break only after enforcement, this is usually why, and the fix is an ipBlock rule covering your node CIDR rather than another selector you'll fight with.

Step 4: the cluster-wide backstop

Once the per-namespace pattern is proven, Calico's GlobalNetworkPolicy enforces namespace isolation as a guardrail across every tenant namespace at once, with infrastructure explicitly carved out:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: tenant-isolation-backstop
spec:
  order: 3000
  namespaceSelector: >-
    projectcalico.org/name not in
    {"kube-system", "calico-system", "calico-apiserver",
     "ingress", "argocd", "cert-manager", "kyverno"}
  types: [Ingress, Egress]
  egress:
    # DNS keeps working even where namespace policies are missing
    - action: Allow
      protocol: UDP
      destination:
        selector: k8s-app == 'kube-dns'
        ports: [53]
    - action: Allow
      protocol: TCP
      destination:
        selector: k8s-app == 'kube-dns'
        ports: [53]

No explicit Deny rule, and that's deliberate. In Calico, when at least one policy selects an endpoint and no rule allows the packet, the packet is dropped at the end of evaluation. The backstop selects everything outside the exclusion list, allows DNS, and lets the implicit deny do the rest.

The order: 3000 is doing real work. Calico assigns Kubernetes NetworkPolicies an order of 1000, and lower order means earlier evaluation. An allow in a namespace's own policy terminates evaluation before the backstop is ever consulted. The backstop only catches traffic nothing else has claimed, which means namespaces with proper policies behave per their policies, and namespaces without any get isolation by default instead of the flat network.

That exclusion list is the "infrastructure exclusion" pattern, and I'd argue it's the single most important decision in the whole rollout. The namespaces that run your CNI, your ingress, your GitOps controller, and your admission webhooks are the namespaces where a policy mistake costs you the ability to fix policy mistakes. Leave them out of automated enforcement. Write their policies by hand, later, one at a time, with the logging step in between.

Step 5: automate generation, with the same exclusions

For new namespaces, a Kyverno generate rule stamps the baseline trio in automatically:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-deny
spec:
  rules:
    - name: default-deny
      match:
        any:
          - resources:
              kinds: [Namespace]
      exclude:
        any:
          - resources:
              kinds: [Namespace]
              names: ["kube-system", "kube-public", "kube-node-lease",
                      "calico-system", "ingress", "argocd", "kyverno"]
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        data:
          spec:
            podSelector: {}
            policyTypes: [Ingress, Egress]

Two operational notes. synchronize: true is what creates the regeneration loop from failure three: hand-deleting the generated policy gets it recreated within seconds, so during an incident you pause the ClusterPolicy before touching its output. And Kyverno treats generate rules as effectively immutable: if the generated resource definition is wrong, plan on deleting and recreating the ClusterPolicy rather than patching it in place.

Why this works

The mental model that makes all of this predictable: Kubernetes NetworkPolicies are additive allow-lists with an implicit deny that activates the moment any policy selects a pod. There is no deny rule in the vanilla API. A pod selected by zero policies accepts everything; a pod selected by any policy accepts only what the union of matching policies allows. That's why the baseline trio works as a set: the deny policy flips the pod into isolated mode, and the other two define the allowed surface.

Calico layers an ordered evaluation model on top. Policies are sorted by order, rules within a policy run top to bottom, and the first Allow or Deny terminates evaluation. Kubernetes-native policies slot in at order 1000 (you can see the converted versions with calicoctl get networkpolicy --all-namespaces, prefixed knp.default.). Pods matched by no policy at all fall through to Calico's per-namespace profiles, which default to allow. That layering is exactly what makes the backstop-at-3000 pattern safe: specific intent at 1000 wins, the guardrail catches the remainder, and the logging policy at 4000 sees only what's about to die.

Felix, Calico's per-node agent, also quietly saves you from the worst self-own. Its failsafe port list (SSH on 22, the API server on 6443, BGP on 179, etcd, Typha) is exempt from policy on host endpoints by default, so a bad policy can break your workloads without also locking you out of the nodes you need to fix it from. Don't shrink that list without a very specific reason.

Lessons learned

The failure modes are knowable in advance. DNS, ingress, and webhooks fail in that order every time, and writing the allows before the deny is cheaper in every way than discovering them from a monitoring graph. If a rollout plan doesn't mention kube-dns, port 53, or failurePolicy, it isn't done.

Namespace-by-namespace beats cluster-wide, even though it feels slower. The first namespace takes a day because you're discovering undocumented flows. The tenth takes ten minutes because there's nothing left to discover. Going cluster-wide first inverts that: you discover everything at once, in production, with automation re-applying the breakage faster than you can remove it.

Exclude infrastructure from automation permanently, not temporarily. Every system that can generate or sync policies (Kyverno, ArgoCD, your own scripts) should carry the same exclusion list for kube-system, the CNI namespace, ingress, GitOps, and webhook namespaces. The asymmetry is stark: a missing policy in those namespaces costs you some security posture, while a wrong policy there costs you the control plane's ability to accept the fix.

Logging is the difference between policy as guesswork and policy as engineering. The Log action is crude (kernel log lines, iptables dataplane only), but it converts "why is this connection failing" from a hypothesis into a grep. I'd take crude visibility over elegant blindness in any network debugging session. This pattern, restrict by default and watch the boundary, is the same shape as the guardrails I build around autonomous agent infrastructure: the deny is easy, and the engineering is in the observability that tells you what the deny will cost before you pay it.

The thing the docs undersell is that default deny is a migration, not a manifest. The YAML is trivial. The work is the inventory of flows your cluster actually depends on, and you only get that inventory by watching one namespace at a time with the logs on.

CKS Success Story -44 Days Walking with AI-

Aoi Takahashi — Mon, 15 Jun 2026 11:54:45 +0000

Hi! I'm Aoi! I Passed the CKS Exam — Here's My Story!

I passed the CKS exam and I'm so happy I had to write about it!

This time I asked AI to accompany me throughout my study journey, so I've added the subtitle "44 Days Walking with AI."

This article is translated from the original article written in Japanese.
https://zenn.dev/aoi/articles/5a816271cbb174

My Skill Set

7 years working with Kubernetes as an SRE
Experience with Linux server setup and administration
CKA and CKAD certified

SRE stands for Site Reliability Engineer — I work on platform maintenance (including Kubernetes) for developers. So I have a solid foundation in Kubernetes basics. Since we use AWS EKS, I rarely touch the Control Plane directly at work.

Summary

Since I had AI accompanying me, I asked it to write the summary. Here's what it came up with:

Study Period

April 21, 2026 (Tue) – June 4, 2026 (Wed) = approximately 44 days

I failed the first attempt (5/26) with 50 points, but regrouped quickly and passed on the second attempt (6/4).

Study Resources Used

KodeKloud CKS Course (video + hands-on labs)
Killercoda (killercoda.com/killer-shell-cks) — all 34 scenarios completed twice
Killer.sh (practice exam included with the certification purchase, taken twice)
Killer Shell official YouTube (full course, approximately 12 hours)
Official curriculum PDF (CKS_Curriculum_v1.34.pdf)

Study Flow

Phase 1: Input Phase (4/21–5/6)

Watched all KodeKloud video sections to build conceptual understanding of each domain.

System Hardening (AppArmor / Seccomp / kube-bench)
Cluster Setup (RBAC / NetworkPolicy / Ingress TLS / CSR)
Microservice Vulnerabilities (PSA / gVisor / Cilium / Istio mTLS)
Supply Chain Security (Trivy / SBOM / image signing / static analysis)
Monitoring & Runtime Security (Falco / strace / Audit Logs)

Golden Week (Japanese national holidays) let me get significantly ahead of schedule.

Phase 2: Hands-on Phase (4/27–5/13)

Worked through Killercoda scenarios in order. I also reviewed the official curriculum PDF myself and confirmed things like OPA Gatekeeper being out of scope — staying precise about what's actually on the exam was a priority.

Phase 3: Practice Exam Phase (5/15–5/20)

Killer.sh attempt 1: 30-something points (ran out of time)
Killer.sh attempt 2: 17 points

The scores were low, but Killer.sh is intentionally harder than the real exam — the goal here was to identify weak spots.

Phase 4: First Exam Attempt (5/26)

Result: 50 points — failed

Domains flagged as weak:

Supply Chain Security
System Hardening
Cluster Setup

Phase 5: Re-exam Prep (5/28–6/3)

Thoroughly analyzed where I got stuck in the first attempt and ran through all Killercoda scenarios again.

Phase 6: Second Attempt (6/4)

Passed!

Study Hours

Life with a dog keeps me busy, so I managed about 0–1 hours on weekdays and 1–3 hours on weekends. Since Golden Week fell during this period, I probably clocked around 10 hours over those few days.

Prerequisites

You must already hold the CKA to sit the CKS exam.

What I Did First

Learning from Those Who Came Before

I started by reading other people's exam write-ups to benefit from their experience.

Asami's article in particular was extremely clear — I came back to it many times.

Asking AI to Accompany Me

After reading those articles and forming a rough picture of my study path, I asked AI: "I want to pass CKS in one month — please be my study companion."

It then put together a study plan, and I followed it throughout.

Study Methods

I worked through the following in order.

KodeKloud CKS Course

https://learn.kodekloud.com/courses/certified-kubernetes-security-specialist-cks

Requires a paid KodeKloud plan, but you can watch topic videos and do hands-on practice side by side. The standard plan runs $35/month (roughly ¥6,000), which is enough to make you feel like you absolutely have to pass within the month. (Looking just now, the standard plan is apparently $29 — a price cut!?)

Some content is a bit dated, so double-check the official exam requirements before your attempt.

I got through about 80% of KodeKloud before running low on time, so I switched to the Killercoda-focused approach described below. (I also consulted AI on when to make that switch.)

I ended up not passing within the one-month window, but I felt confident enough that "cycling through Killercoda is sufficient" to cancel the subscription. I was pleasantly surprised that KodeKloud sends a friendly "your month is almost up!" reminder notification.

Killercoda

https://killercoda.com/killer-shell-cks

A free hands-on environment. After getting through the KodeKloud input phase, I moved here for hands-on practice. KodeKloud walks you step-by-step toward the answer, so Killercoda is closer to the actual exam format.

Killer.sh

The official practice exam bundled with the CKS purchase. I didn't use the practice exam when I took the CKA and regretted it, so this time I planned my schedule to include proper review time before sitting it.

Quick AI Quizzes in Spare Moments

Video watching and hands-on practice are heavy study methods. For tired moments or short waits during commutes, I'd ask AI to "give me a quiz" so I could keep studying in small windows.

One funny thing: the first answer choice was always the correct one 😂 (I caught on eventually and asked it to randomize the order.)

Sometimes I'd request multiple-choice questions; other times I'd ask for free-response prompts to check my command recall.

Studying After Failing the First Time

I only got 50 points on my first attempt — devastating! The result breakdown tells you which domains you struggled with, so I focused on those weak areas and just hammered Killercoda.

It paid off: I passed the second attempt with 80 points!!!!

Final Thoughts

It was rough! There were plenty of days I couldn't carve out study time, and the stress was through the roof. So relieved to be free of it now!

I learned a lot — there were things I didn't know at all, and things I'd only half-understood that I finally got a solid grasp on.

Troubleshooting Kubernetes Events with TKE and Tencent Cloud CLS

Tencent Cloud -Cloud Log Service — Mon, 15 Jun 2026 11:06:29 +0000

Troubleshooting Kubernetes Events with TKE and Tencent Cloud CLS

Cluster problems rarely appear from nowhere. Before a service outage becomes visible, Kubernetes often records smaller state changes: node pressure, Pod scheduling, Pod eviction, and cluster autoscaler decisions.

Tencent Kubernetes Engine can send those Events into Tencent Cloud CLS, where they become searchable logs and dashboard data. This gives operators a central way to answer what changed, when it changed, which object was involved, and which component reported it.

What an Event tells you

Kubernetes Events describe state transitions. The useful fields are:

Field	What to look for
`Type`	`Normal`, `Warning`, or a custom type.
`Involved Object`	Pod, Deployment, Node, or another Kubernetes object.
`Source`	Component such as Scheduler or Kubelet.
`Reason`	Short reason enum.
`Message`	Detailed explanation.
`Count`	How many times it happened.

The core flow is: Kubernetes emits a state-change record, CLS stores it as a log event, and the operator filters by object, component, reason, message, count, and timestamp.

Open Event Search

In TKE, go to Cluster Operations -> Event Search. CLS provides collection, storage, search, analysis, and dashboards for the event stream.

Use the overview when you need warning distribution, affected object types, and event trends. Use global search when you already know the component or object name and need a row-level timeline.

Runbook 1: an abnormal node

Filter by the abnormal node name in the event overview. In this example, the result included a node disk-space warning.

The timeline showed that on 2020-11-25, node 172.16.18.13 became abnormal because disk space was insufficient. Kubelet then tried to evict Pods from the node to reclaim disk space.

That sequence gives you a clean next step: check node disk usage, eviction thresholds, and workload placement before treating it as a generic application failure.

Runbook 2: autoscaler expansion

For node pool autoscaling, query the autoscaler component:

event.source.component:"cluster-autoscaler"

Display these fields:

event.reason
event.message
event.involvedObject.name

Sort by log time descending. The result should work like a compact ledger of autoscaler decisions: workload object, reason, message, and the timestamp of each scaling step.

The event stream showed scale-out around 2020-11-25 20:35:45, triggered by three nginx Pods:

nginx-5dbf784b68-tq8rd
nginx-5dbf784b68-fpvbx
nginx-5dbf784b68-v9jv5

Three nodes were added. Later scale-out did not continue because the node pool had reached its maximum node count.

Checklist

Use Events to understand state changes, not only current state.
Start with overview dashboards, then filter by object name.
For node issues, inspect reason, message, source component, and count.
For autoscaling, query cluster-autoscaler and reconstruct the event timeline.
Use metrics and logs after Events point you to the right object and time window.

FAQ

Why not only use `kubectl describe`?

kubectl describe is useful for one object. CLS is better when you need searchable history, dashboards, and cross-object analysis.

What is the fastest autoscaler query?

Start with event.source.component:"cluster-autoscaler" and sort by log time descending.