DEV Community: Florient DOGBE

Supply Chain Security: Generate an SBOM and Block Critical CVEs Before the Merge, Not in Prod

Florient DOGBE — Tue, 03 Nov 2026 09:22:06 +0000

Most teams discover a critical CVE in a container image the same way: a scanner running against the production cluster fires an alert at 3pm on a Friday, and someone has to figure out whether the vulnerable library is actually reachable, who owns the base image, and how fast a patched version can be shipped. That workflow is expensive and stressful. The fix is unglamorous: move the detection left, into the pull request, where the cost of a change is a one-line bump in a lockfile instead of an emergency change request.

This article shows how to build that guardrail with two small, focused tools — Syft to produce a CycloneDX SBOM, and Grype to scan it — wired into GitHub Actions, with a severity policy that fails the build and an exception mechanism that is documented, time-boxed, and reviewable. We will also cover where Trivy fits, and how to keep the same policy enforced at admission time on OpenShift/Kubernetes so the CI gate is not the only line of defence.

The target reader is someone who has never generated an SBOM before. Every command below is runnable as-is.

Context: why a scanner in prod is already too late

A vulnerability scan tells you three things: what software you ship, which known CVEs affect it, and whether a fix exists. Running that scan in production answers the question after you have already taken the risk. Running it in CI answers it before the artifact exists.

There are three concrete reasons this matters:

1. You cannot patch what you cannot enumerate. When log4shell landed, the teams that answered "are we affected?" in minutes were the ones with a stored SBOM per build. Everyone else spent two days running find / -name "log4j*" across clusters.

2. The cost of a fix grows with distance from the commit. Bumping express from 4.17.1 to 4.19.2 in a PR is a 10-minute job. Doing it on a running production service means a change window, a rollback plan, and a regression test campaign.

3. Regulations now require the artefact. The EU Cyber Resilience Act, the US Executive Order 14028, and most enterprise vendor questionnaires ask for a machine-readable bill of materials. CycloneDX and SPDX are the two accepted formats. Generating one as a build side effect costs you about 15 seconds of CI time.

The guardrail we are building has a simple contract: no pull request merges if it introduces a Critical (or High, depending on your policy) vulnerability with a known fix, unless an exception is explicitly recorded in the repository.

The tools, in 90 seconds

Syft (Anchore, Apache-2.0) walks a filesystem, a directory, or a container image and lists every package it can identify: OS packages (apk/deb/rpm), language packages (npm, pip, go modules, Maven, Cargo...), and binaries. Output: an SBOM in CycloneDX, SPDX, or its own JSON format. Syft does not know about vulnerabilities.
Grype (same vendor) takes an SBOM — or an image directly — and matches each package against a vulnerability database (NVD, GitHub Security Advisories, distro trackers). It knows about fix state, which is the single most useful field for a sane policy.
Trivy (Aqua, Apache-2.0) does both jobs in one binary, plus IaC and secret scanning. It is an excellent alternative, and many teams pick it precisely because it is one tool instead of two.

Why split Syft and Grype? Because the SBOM becomes an artifact in its own right. You generate it once, attach it to the release, and can re-scan it six months later against a newer vulnerability DB without rebuilding anything. That decoupling is the whole point of an SBOM.

Install locally first

Always validate the pipeline on your laptop before writing YAML.

# Syft and Grype (binaries land in ./bin or /usr/local/bin)
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh \
  | sh -s -- -b /usr/local/bin v1.18.1

curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh \
  | sh -s -- -b /usr/local/bin v0.87.0

syft version
grype version

Pin the versions. An unpinned install.sh is itself a supply chain risk, and a tool upgrade that changes default severity mappings will break builds at the worst moment.

Step 1 — Generate a CycloneDX SBOM

Syft accepts several source types. The two you will use most:

# From a container image in a registry (no Docker daemon required)
syft registry:quay.io/myorg/payment-api:1.4.2 \
  -o cyclonedx-json=sbom.cdx.json

# From the local source tree (catalogs lockfiles: package-lock.json, go.sum, poetry.lock...)
syft dir:. -o cyclonedx-json=sbom-src.cdx.json

Scan both. The source scan catches your direct and transitive application dependencies early, even before the image is built. The image scan catches everything: OS packages from the base image, binaries copied in, and anything a build stage pulled in.

Inspect what you got:

# Human-readable table
syft registry:quay.io/myorg/payment-api:1.4.2 -o table | head -20

# Count components in the SBOM
jq '.components | length' sbom.cdx.json

# List components by package ecosystem
jq -r '.components[].purl' sbom.cdx.json | cut -d: -f2 | cut -d/ -f1 | sort | uniq -c

A CycloneDX document is just JSON with a bomFormat, a specVersion, a metadata block describing the subject, and a components array of packages with their purl (package URL) identifiers. That purl is what makes the SBOM matchable against vulnerability databases.

Cut the noise at the source

Syft catalogs everything it finds, including build-time tooling that never reaches the runtime image. If you use multi-stage builds correctly, this mostly solves itself:

FROM registry.access.redhat.com/ubi9/nodejs-20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev

FROM registry.access.redhat.com/ubi9/nodejs-20-minimal
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER 1001
CMD ["node", "dist/server.js"]

The final stage contains production dependencies only, so the SBOM reflects what actually runs. A -minimal or distroless base image typically removes 100+ OS packages and with them a long tail of irrelevant CVEs.

Step 2 — Scan the SBOM with Grype

# Scan the SBOM you just produced
grype sbom:sbom.cdx.json -o table

# Fail the process if anything Critical is found
grype sbom:sbom.cdx.json --fail-on critical

# Only report vulnerabilities that have an available fix
grype sbom:sbom.cdx.json --only-fixed --fail-on high

--only-fixed is the setting that turns a noisy scanner into a usable gate. A Critical CVE in a distro package where the vendor has not shipped a patch is not something your PR author can act on. Blocking the merge on it just teaches people to bypass the gate. Report it, track it, but do not fail the build on it.

Exit code is 1 when the threshold is hit, 0 otherwise — that is all CI needs.

Policy as a file, not as CLI flags

Grype reads .grype.yaml from the working directory. Commit it; it is your policy document.

# .grype.yaml
# Severity threshold that breaks the build.
fail-on-severity: high

# Only consider vulnerabilities with a known fix available.
only-fixed: true

output:
  - table

# Documented, time-boxed exceptions.
ignore:
  # CVE-2024-21538 (cross-spawn ReDoS) - build-time only, not reachable at runtime.
  # Owner: platform-team  Ticket: SEC-1427  Review by: 2025-09-30
  - vulnerability: CVE-2024-21538
    package:
      name: cross-spawn
      type: npm

  # GHSA-xxxx: triggers only when the parser is fed untrusted XML; our input is
  # internal and schema-validated. Owner: payments  Ticket: SEC-1455  Review by: 2025-08-15
  - vulnerability: GHSA-2c8p-7m4v-h3cp
    package:
      type: npm

Three rules make exceptions safe rather than a loophole:

Every entry carries an owner, a ticket, and a review date. No anonymous ignore lines.
The file is protected by CODEOWNERS, so a security reviewer must approve additions.
A scheduled job fails when a review date has passed. An exception that nobody re-validates silently becomes permanent.

# scripts/check-exceptions.sh — fail if any "Review by" date is in the past
#!/usr/bin/env bash
set -euo pipefail
today=$(date +%F)
expired=0
while read -r line; do
  d=$(grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' <<<"$line" || true)
  [[ -z "$d" ]] && continue
  if [[ "$d" < "$today" ]]; then
    echo "EXPIRED exception (review date $d): $line"
    expired=1
  fi
done < <(grep -i 'Review by' .grype.yaml)
exit "$expired"

Step 3 — Wire it into GitHub Actions

Here is the complete workflow. It runs on pull requests, builds the image, generates both SBOMs, scans, and uploads results.

# .github/workflows/supply-chain.yml
name: supply-chain

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  schedule:
    # Re-scan main nightly: new CVEs appear without any code change.
    - cron: "0 3 * * *"

permissions:
  contents: read
  security-events: write   # required to upload SARIF
  pull-requests: write

jobs:
  sbom-and-scan:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4

      - name: Verify exception review dates
        run: bash scripts/check-exceptions.sh

      - name: Build image (local, not pushed)
        run: |
          docker build -t app:${{ github.sha }} .

      - name: Generate CycloneDX SBOM (image)
        uses: anchore/sbom-action@v0
        with:
          image: app:${{ github.sha }}
          format: cyclonedx-json
          output-file: sbom.cdx.json
          artifact-name: sbom-${{ github.sha }}.cdx.json

      - name: Generate CycloneDX SBOM (source tree)
        uses: anchore/sbom-action@v0
        with:
          path: .
          format: cyclonedx-json
          output-file: sbom-src.cdx.json
          artifact-name: sbom-src-${{ github.sha }}.cdx.json

      - name: Scan SBOM with Grype
        id: grype
        uses: anchore/scan-action@v6
        with:
          sbom: sbom.cdx.json
          fail-build: true
          severity-cutoff: high
          only-fixed: true
          output-format: sarif

      - name: Upload SARIF to code scanning
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: ${{ steps.grype.outputs.sarif }}

      - name: Keep SBOM for 90 days
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: sbom-${{ github.sha }}
          path: |
            sbom.cdx.json
            sbom-src.cdx.json
          retention-days: 90

Two details that matter more than they look:

The nightly schedule. A repository whose code has not changed in three months can still become vulnerable overnight, because the vulnerability database changed, not your code. Scanning only on PR gives you a false sense of safety. The scheduled run catches drift.

SARIF upload. Results land in the GitHub Security tab with per-finding deduplication and history, rather than being buried in a job log that nobody opens.

Finally, make the job a required status check in branch protection — otherwise it is advisory and will be ignored under deadline pressure.

The same thing with Trivy

If you prefer a single binary, Trivy covers both steps:

# CycloneDX SBOM
trivy image --format cyclonedx --output sbom.cdx.json quay.io/myorg/payment-api:1.4.2

# Scan the SBOM, fail on HIGH/CRITICAL with a fix available
trivy sbom --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 sbom.cdx.json

Exceptions live in a .trivyignore file (one CVE ID per line, comments with #) or, for richer rules, in trivyignore.yaml with per-entry statement and expired_at fields — the expiry date being built in is a genuinely nice touch.

Pick one and standardise. Running Syft/Grype and Trivy on the same artifact produces slightly different result sets and endless "which one is right?" discussions.

Step 4 — Enforce at admission, not just in CI

A CI gate stops your pipeline from shipping a vulnerable image. It does nothing about an image someone deploys by hand, or a third-party chart pulling some/image:latest. On OpenShift/Kubernetes, close that gap with a cluster-side policy.

Minimal example with Kyverno — only allow images from trusted registries:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: allowed-registries
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: ["prod-*"]
      validate:
        message: >-
          Images must come from quay.io/myorg/ or
          registry.access.redhat.com/ (scanned in CI).
        pattern:
          spec:
            =(initContainers):
              - image: "quay.io/myorg/* | registry.access.redhat.com/*"
            containers:
              - image: "quay.io/myorg/* | registry.access.redhat.com/*"

And a CronJob that re-scans what is actually running, independently of CI:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: grype-runtime-rescan
  namespace: security-scanning
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: image-scanner
          containers:
            - name: grype
              image: anchore/grype:v0.87.0
              args:
                - "registry:quay.io/myorg/payment-api:1.4.2"
                - "--only-fixed"
                - "-o"
                - "json"
                - "--file"
                - "/reports/payment-api.json"
              env:
                - name: GRYPE_DB_CACHE_DIR
                  value: /tmp/grype-db
                - name: GRYPE_CHECK_FOR_APP_UPDATE
                  value: "false"
              securityContext:
                allowPrivilegeEscalation: false
                runAsNonRoot: true
                capabilities:
                  drop: ["ALL"]
                seccompProfile:
                  type: RuntimeDefault
              resources:
                requests:
                  cpu: 200m
                  memory: 512Mi
                limits:
                  memory: 1Gi
              volumeMounts:
                - name: reports
                  mountPath: /reports
                - name: tmp
                  mountPath: /tmp
          volumes:
            - name: reports
              persistentVolumeClaim:
                claimName: scan-reports
            - name: tmp
              emptyDir: {}

If the registry is private, mount a pull secret and point Grype at it:

oc -n security-scanning create secret docker-registry quay-pull \
  --docker-server=quay.io \
  --docker-username='myorg+scanner' \
  --docker-password="$(cat ~/.quay-robot-token)"

oc -n security-scanning secrets link image-scanner quay-pull --for=pull

Never inline credentials in a manifest. Use a Secret, an External Secrets Operator reference, or OpenShift's built-in service account pull secrets.

Security best practices

Pin every tool version. install.sh ... v0.87.0, anchore/grype:v0.87.0, actions/checkout@v4 — ideally pinned to a commit SHA for third-party actions. A supply chain gate built on floating tags is self-defeating.

Restrict GitHub Actions permissions. Declare permissions: explicitly at workflow or job level. The default token is far broader than a scan job needs.

Never scan with pull_request_target on untrusted forks. That trigger runs with repository secrets available to code the PR author controls. Use pull_request and accept that forks cannot access secrets.

Store the SBOM as a durable artifact. 90-day retention in CI is a minimum; for released versions, push the SBOM to your registry as an OCI artifact or attach it to the GitHub release. The value of an SBOM is being able to answer "which of our 200 services ship library X version Y?" in under a minute.

Sign images and SBOMs. An unsigned SBOM proves nothing about provenance. cosign attest --predicate sbom.cdx.json --type cyclonedx binds the SBOM to the image digest, and Kyverno or Sigstore Policy Controller can verify that attestation at admission.

Always reference images by digest in production manifests. quay.io/myorg/payment-api@sha256:... is the only way to guarantee that what you scanned is what runs. Tags are mutable.

Set the threshold where the team can actually operate. Start with fail-on-severity: critical plus only-fixed: true, clear the backlog, then tighten to high. A gate that fails 80% of builds on day one gets disabled on day two.

Separate "block" from "report". Vulnerabilities without a fix, and Medium/Low findings, go to the Security tab and the nightly report — not to a red X on a PR.

Keep the vulnerability database fresh. Grype downloads it on each run by default; in air-gapped environments, mirror it and set GRYPE_DB_UPDATE_URL. A stale DB turns a green build into a lie.

Treat .grype.yaml as a security-reviewed file. CODEOWNERS entry, mandatory ticket reference, mandatory review date, and a job that fails when a date expires.

Conclusion

The gap between "we have a scanner" and "we have a guardrail" is mostly process, not tooling. The technical part takes an afternoon:

Syft produces a CycloneDX SBOM from your source tree and from your image. That file is the inventory you will need the next time a log4shell-class event happens — generate it on every build and keep it.
Grype scans the SBOM and returns a non-zero exit code against a policy you commit to the repository. only-fixed: true plus a severity threshold the team can realistically meet is the difference between a useful gate and an ignored one.
GitHub Actions runs it on every pull request, uploads SARIF to the Security tab, stores the SBOM as an artifact, and re-runs nightly to catch CVEs published after the merge. The job must be a required status check.
Exceptions are explicit, owned, ticketed, and expiring. An exception mechanism that nobody can audit is just a disabled scanner with extra steps.
CI is not the last line of defence. Admission policies on OpenShift/Kubernetes and a scheduled re-scan of running images cover the paths that bypass your pipeline.

Start small: generate an SBOM on your most critical service this week, in report-only mode. Look at the numbers, fix the trivially fixable, document the rest with review dates, then flip fail-build to true. Six weeks later, nobody on the team will remember what it was like to find out about a critical CVE from a production alert.

Writing Your First Kubernetes Controller in Go: Reconcile, Finalizers, and Mistakes to Avoid

Florient DOGBE — Mon, 02 Nov 2026 13:33:38 +0000

A Kubernetes controller is deceptively simple: watch objects, call Reconcile, make the world match the spec. Then you ship it, and three weeks later someone reports that deleting a namespace hangs forever, your API server audit log shows 40 000 PATCH requests per minute from a single controller pod, and an orphaned bucket is still billing you in production.

All three failures come from the same three concepts: idempotency, finalizers, and status write discipline. This article builds a minimal but correct controller with controller-runtime, scaffolded by kubebuilder, that manages a CRD representing an external resource. We focus on the parts that actually break in production.

Context: why the reconcile loop is harder than it looks

Reconcile is level-triggered, not edge-triggered. You never receive "the user changed spec.size from 10 to 20". You receive "something about default/my-bucket may have changed; go look". The event may be a resync, a status update your own controller just wrote, a cache re-list after an API server restart, or a genuine user edit. It may arrive twice for the same change, or once for three changes.

Three consequences that drive the entire design:

Reconcile must be idempotent. Running it 100 times must produce the same cluster state as running it once, and must not create 100 external resources.
Reconcile must be restartable. The process can die between "create the external bucket" and "record the bucket ID in status". The next pass must recover without leaking.
Garbage collection only covers in-cluster objects. ownerReferences will clean up a Secret you created. Nothing in Kubernetes knows about an S3 bucket, a DNS record, or a row in an external IPAM. That is exactly what finalizers are for.

Scaffolding the project

mkdir bucket-operator && cd bucket-operator
kubebuilder init \
  --domain example.com \
  --repo github.com/acme/bucket-operator \
  --plugins go/v4

kubebuilder create api \
  --group storage \
  --version v1alpha1 \
  --kind BucketClaim \
  --resource --controller

The API type. Note observedGeneration and the conditions slice — both are load-bearing, not decoration.

// api/v1alpha1/bucketclaim_types.go
type BucketClaimSpec struct {
    // +kubebuilder:validation:MinLength=3
    // +kubebuilder:validation:MaxLength=63
    Region string `json:"region"`

    // +kubebuilder:validation:Enum=Retain;Delete
    // +kubebuilder:default=Delete
    ReclaimPolicy string `json:"reclaimPolicy,omitempty"`
}

type BucketClaimStatus struct {
    // BucketID is the opaque identifier returned by the storage backend.
    BucketID string `json:"bucketID,omitempty"`

    // ObservedGeneration is the .metadata.generation this status reflects.
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`

    // +listType=map
    // +listMapKey=type
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Region",type=string,JSONPath=`.spec.region`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
type BucketClaim struct { /* ... */ }

The status subresource is non-negotiable: without it, every Update on status bumps metadata.generation and re-triggers your own controller. That is infinite loop number one.

The generated CRD can carry CEL validation so you reject bad specs at admission time instead of in Go:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bucketclaims.storage.example.com
spec:
  group: storage.example.com
  scope: Namespaced
  names:
    kind: BucketClaim
    listKind: BucketClaimList
    plural: bucketclaims
    singular: bucketclaim
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [region]
              x-kubernetes-validations:
                - rule: "self.region == oldSelf.region"
                  message: "region is immutable"
              properties:
                region:
                  type: string
                reclaimPolicy:
                  type: string
                  enum: [Retain, Delete]
                  default: Delete
            status:
              type: object
              properties:
                bucketID: { type: string }
                observedGeneration: { type: integer, format: int64 }
                conditions:
                  type: array
                  items:
                    type: object
                    required: [type, status, lastTransitionTime, reason]
                    properties:
                      type: { type: string }
                      status: { type: string }
                      reason: { type: string }
                      message: { type: string }
                      observedGeneration: { type: integer, format: int64 }
                      lastTransitionTime: { type: string, format: date-time }

Immutability enforced by CEL removes an entire class of reconcile branches ("what do I do when the user moves the bucket to another region?").

The reconcile loop, step by step

const finalizerName = "storage.example.com/bucket-cleanup"

func (r *BucketClaimReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := logf.FromContext(ctx)

    var bc storagev1alpha1.BucketClaim
    if err := r.Get(ctx, req.NamespacedName, &bc); err != nil {
        // NotFound means the object is gone and fully finalized. Nothing to do.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if !bc.DeletionTimestamp.IsZero() {
        return r.reconcileDelete(ctx, &bc)
    }

    // 1. Finalizer BEFORE any external side effect.
    if controllerutil.AddFinalizer(&bc, finalizerName) {
        if err := r.Update(ctx, &bc); err != nil {
            return ctrl.Result{}, err
        }
        // This Update generates a new event; stop here, the next pass continues.
        return ctrl.Result{}, nil
    }

    // 2. Idempotent provisioning, keyed on a deterministic name.
    name := bucketName(&bc) // e.g. fmt.Sprintf("%s-%s-%s", bc.Namespace, bc.Name, bc.UID[:8])
    id, err := r.Storage.EnsureBucket(ctx, name, bc.Spec.Region)
    if err != nil {
        meta.SetStatusCondition(&bc.Status.Conditions, metav1.Condition{
            Type: "Ready", Status: metav1.ConditionFalse,
            Reason: "BackendError", Message: err.Error(),
            ObservedGeneration: bc.Generation,
        })
        _ = r.patchStatus(ctx, &bc)
        return ctrl.Result{}, err // exponential backoff from the workqueue
    }

    // 3. Owned in-cluster object: GC handles deletion, no finalizer needed.
    if err := r.ensureCredentialsSecret(ctx, &bc, id); err != nil {
        return ctrl.Result{}, err
    }

    // 4. Status last, and only status.
    bc.Status.BucketID = id
    bc.Status.ObservedGeneration = bc.Generation
    meta.SetStatusCondition(&bc.Status.Conditions, metav1.Condition{
        Type: "Ready", Status: metav1.ConditionTrue,
        Reason: "Provisioned", ObservedGeneration: bc.Generation,
    })
    if err := r.patchStatus(ctx, &bc); err != nil {
        return ctrl.Result{}, err
    }

    // Re-check the backend periodically: no watch exists on external systems.
    return ctrl.Result{RequeueAfter: 10 * time.Minute}, nil
}

Why the finalizer goes first

If you create the bucket and then add the finalizer, there is a window where the controller can crash, the user deletes the CR, the API server removes it immediately, and the bucket leaks. Adding the finalizer is a cheap, idempotent metadata write. Do it before you touch anything outside the cluster, always.

Why provisioning must be keyed on a deterministic name

EnsureBucket must be "create if absent, return existing otherwise" and derive its key from immutable CR data (namespace, name, UID). Never key on status.BucketID alone: if the process dies between the backend call and the status write, the next pass must still find the existing bucket instead of creating a second one. Reading the status you just wrote back from the cache is also unreliable — the informer cache is eventually consistent.

Patch, don't update, for status

func (r *BucketClaimReconciler) patchStatus(ctx context.Context, bc *storagev1alpha1.BucketClaim) error {
    // base is the object as it was read; MergeFrom computes a minimal patch.
    return r.Status().Patch(ctx, bc, client.MergeFrom(bc.DeepCopy()))
}

In practice, capture the base object right after Get (base := bc.DeepCopy()), mutate bc, then r.Status().Patch(ctx, bc, client.MergeFrom(base)). A merge patch that computes to an empty diff is a no-op on the API server: no write, no event, no loop. A blind Status().Update() on every reconcile rewrites resourceVersion every time and is the second classic infinite loop — made worse by lastTransitionTime if you rebuild conditions by hand instead of using meta.SetStatusCondition, which only touches the timestamp when the status actually flips.

Deletion: the finalizer path

func (r *BucketClaimReconciler) reconcileDelete(ctx context.Context, bc *storagev1alpha1.BucketClaim) (ctrl.Result, error) {
    if !controllerutil.ContainsFinalizer(bc, finalizerName) {
        return ctrl.Result{}, nil // someone else's finalizer, not our turn
    }

    if bc.Spec.ReclaimPolicy == "Delete" {
        err := r.Storage.DeleteBucket(ctx, bucketName(bc))
        switch {
        case err == nil, errors.Is(err, backend.ErrNotFound):
            // Idempotent: already gone is success.
        default:
            r.Recorder.Eventf(bc, corev1.EventTypeWarning, "CleanupFailed",
                "bucket cleanup failed: %v", err)
            return ctrl.Result{}, err
        }
    }

    controllerutil.RemoveFinalizer(bc, finalizerName)
    if err := r.Update(ctx, bc); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    return ctrl.Result{}, nil
}

Four rules for finalizers:

Idempotent cleanup. "Resource not found in the backend" is a success, not an error. Otherwise an already-deleted bucket blocks the CR forever.
Remove the finalizer only after cleanup succeeded. The order in the code above is the whole point.
Never block forever. A finalizer that always errors makes the CR undeletable and, worse, makes its namespace stuck in Terminating. Expose the failure as an Event and a condition so an operator can see why, and support an escape hatch (for example, honour storage.example.com/skip-cleanup: "true" as an annotation).
Check it is your finalizer. Other controllers (and the kubernetes namespace finalizer) add their own. Only ever touch your own string.

Emergency unblocking, for the runbook:

# Inspect what is holding the object
oc get bucketclaim my-bucket -o jsonpath='{.metadata.finalizers}{"\n"}'

# Last resort: strip the finalizer manually (this LEAKS the external resource)
oc patch bucketclaim my-bucket --type=merge -p '{"metadata":{"finalizers":null}}'

Wiring the controller and filtering events

func (r *BucketClaimReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&storagev1alpha1.BucketClaim{},
            builder.WithPredicates(predicate.Or(
                predicate.GenerationChangedPredicate{},
                predicate.AnnotationChangedPredicate{},
            ))).
        Owns(&corev1.Secret{}).
        WithOptions(controller.Options{MaxConcurrentReconciles: 4}).
        Named("bucketclaim").
        Complete(r)
}

GenerationChangedPredicate drops update events where metadata.generation is unchanged — that includes your own status writes. It does not drop deletions: when the API server marks an object for deletion while finalizers are pending, it bumps generation, so the delete-intent event still reaches you. The trade-off: it also drops periodic resync events, so never rely on resync for drift detection — use the explicit RequeueAfter in the success path instead.

MaxConcurrentReconciles > 1 means the same controller processes different objects in parallel, but the workqueue guarantees a given key is never reconciled concurrently with itself. Your Reconcile must still be safe against two different CRs touching the same backend.

Returning the right thing

Situation	Return
Done, nothing more to do	`ctrl.Result{}, nil`
Transient failure	`ctrl.Result{}, err` — exponential backoff, logged, metered
Waiting on something with no watch	`ctrl.Result{RequeueAfter: d}, nil`
Permanent, user-caused error	`ctrl.Result{}, nil` + a `Ready=False` condition

Never return an error for a user mistake like an invalid region string: the workqueue will retry it forever, spam your logs, and inflate controller_runtime_reconcile_errors_total. Surface it in status.conditions and stop. Also, do not hand-roll time.Sleep inside Reconcile — you are blocking a worker goroutine; use RequeueAfter.

Conflicts (apierrors.IsConflict) are normal under optimistic concurrency. Just return the error and let the backoff re-read a fresh object; do not wrap your whole reconcile in retry.RetryOnConflict.

Testing against a real API server

envtest runs a real kube-apiserver and etcd binary, which is the only way to validate finalizer semantics and the status subresource.

make test            # kubebuilder wires setup-envtest + ginkgo
make manifests generate
make install         # applies config/crd into the cluster
make run             # runs the manager out-of-cluster against your kubeconfig

Two tests worth writing on day one: delete the CR while the fake backend returns an error, and assert the object still exists with its finalizer; then make the backend succeed and assert the object disappears.

Security best practices

Least-privilege RBAC. Kubebuilder markers generate the role, so keep them narrow — no groups=* or resources=*:

// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims/finalizers,verbs=update
// +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch

Note the dedicated /finalizers subresource permission: it is required as soon as OwnerReferencesPermissionEnforcement is enabled, which is the default on OpenShift.

Scope the cache. A cluster-wide watch on Secrets pulls every secret in the cluster into controller memory. Restrict it:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    Cache: cache.Options{
        ByObject: map[client.Object]cache.ByObject{
            &corev1.Secret{}: {
                Label: labels.SelectorFromSet(labels.Set{"app.kubernetes.io/managed-by": "bucket-operator"}),
            },
        },
    },
})

Harden the pod. On OpenShift, target the restricted-v2 SCC: do not pin runAsUser, let the namespace range assign it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bucket-operator-controller-manager
  namespace: bucket-operator-system
spec:
  replicas: 2
  selector:
    matchLabels: { control-plane: controller-manager }
  template:
    metadata:
      labels: { control-plane: controller-manager }
    spec:
      serviceAccountName: bucket-operator-controller-manager
      automountServiceAccountToken: true
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: manager
          image: quay.io/acme/bucket-operator:v0.1.0
          args: ["--leader-elect", "--health-probe-bind-address=:8081"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          resources:
            requests: { cpu: 10m, memory: 64Mi }
            limits:   { memory: 256Mi }
          livenessProbe:
            httpGet: { path: /healthz, port: 8081 }
          readinessProbe:
            httpGet: { path: /readyz, port: 8081 }

Leader election (--leader-elect with replicas: 2) is a correctness control, not just HA: two active managers racing on the same external backend will double-provision. It needs leases in coordination.k8s.io, namespace-scoped.

Never log secret material. logf.FromContext(ctx) is structured — log the object key and the backend resource ID, never credentials or full object dumps. Credentials belong in a Secret with an ownerReference pointing at the CR, so they are garbage-collected automatically and never written to status.

Validate at admission, not in Go. CEL rules in the CRD (and a validating webhook with a service-ca managed certificate on OpenShift) block bad input before it ever reaches a worker. A NetworkPolicy restricting egress to your backend endpoint closes the loop.

Key takeaways

Reconcile is level-triggered: make it idempotent, restartable, and keyed on deterministic names derived from immutable fields.
Add the finalizer before the first external side effect; remove it only after cleanup succeeds, and treat "already gone" as success.
Use the status subresource with Status().Patch(client.MergeFrom(base)) and meta.SetStatusCondition — blind Update calls are the most common infinite loop.
Filter events with GenerationChangedPredicate, but keep an explicit RequeueAfter to detect drift in systems you cannot watch.
Return an error only for transient failures; report user errors as conditions so the workqueue does not retry them forever.
A finalizer that can never complete is a namespace that can never terminate: always expose the failure and provide an escape hatch.
Lock down RBAC, scope the informer cache, and run the manager as a non-root, read-only, capability-dropped pod.

Gateway API in Production: Replacing Ingress NGINX Without Breaking Your TLS Certificates

Florient DOGBE — Sun, 01 Nov 2026 17:07:21 +0000

Ingress NGINX is in maintenance mode, and the Kubernetes project has been nudging everyone toward Gateway API for a while now. The networking spec graduated to GA (v1) for GatewayClass, Gateway and HTTPRoute, the ecosystem has converged, and your next cluster upgrade is a good excuse to stop postponing the migration.

The problem is that "replace your ingress controller" is a sentence that hides a lot of risk: a single typo in a TLS listener and every browser in production gets a certificate error. This article walks through a migration strategy that never requires a big-bang cutover: run Gateway API next to your existing Ingress NGINX, move one hostname at a time, shift traffic by weight, and validate the TLS chain at every step.

We'll use Envoy Gateway as the controller and cert-manager for certificates, but the pattern applies to any conformant implementation (Istio, Cilium, Traefik, NGINX Gateway Fabric).

Context: why this migration is different

Swapping an ingress controller is not like swapping a Deployment image. Three things make it risky:

TLS termination moves. Your certificates live in Secrets referenced by Ingress objects. The new data plane has to reference the same Secrets (or new ones), possibly from a different namespace, under a stricter cross-namespace model.
The public IP changes. A new controller means a new LoadBalancer Service, a new external IP, and a DNS change. DNS changes are not atomic — you cannot roll them back in 30 seconds.
Annotations don't translate. Rewrites, CORS, rate limits, auth, timeouts — everything encoded in nginx.ingress.kubernetes.io/* annotations becomes either native Gateway API fields, filters, or implementation-specific policies.

The good news: Gateway API was explicitly designed for coexistence. An Ingress and a Gateway can serve the same hostname from the same cluster, and nothing in the spec forbids two controllers from watching different resources. That's the lever we're going to pull.

Target architecture

The model splits responsibilities across three object types, which maps nicely onto real org boundaries:

GatewayClass — cluster-scoped, owned by the platform team. Points at a controller.
Gateway — the listener (ports, protocols, TLS certs, allowed routes). Lives in an infrastructure namespace.
HTTPRoute — owned by application teams, in application namespaces, attached to a Gateway via parentRefs.
ReferenceGrant — the explicit opt-in that allows a cross-namespace reference (Gateway → Secret, HTTPRoute → Service).

Concretely: one shared Gateway in gateway-infra, TLS Secrets managed by cert-manager, and each team shipping its own HTTPRoute in its own namespace.

Step 1 — Install the CRDs and the controller

On a vanilla Kubernetes cluster, install the standard channel CRDs first:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
kubectl get crd | grep gateway.networking.k8s.io

OpenShift note: since 4.19, the Ingress Operator can manage the Gateway API CRDs itself. Check for ownership before applying anything, otherwise the operator will fight you:
oc get crd gateways.gateway.networking.k8s.io \
  -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}'
If the CRDs are operator-managed, skip the kubectl apply and only install your controller. Envoy Gateway's data plane runs unprivileged on ports 10080/10443, so the restricted-v2 SCC is sufficient — no SCC patching required.

Install Envoy Gateway:

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.6 \
  -n envoy-gateway-system --create-namespace \
  --set config.envoyGateway.provider.kubernetes.rateLimitDeployment.container.resources.requests.cpu=100m

kubectl -n envoy-gateway-system rollout status deploy/envoy-gateway

Then declare the GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg-external
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

kubectl get gatewayclass eg-external \
  -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}{"\n"}'
# expected: True

Step 2 — Wire cert-manager to Gateway API

cert-manager ships a "gateway shim" that watches Gateway objects for the cert-manager.io/cluster-issuer annotation and creates Certificate resources from the listeners. It's off by default:

helm upgrade --install cert-manager jetstack/cert-manager \
  -n cert-manager --create-namespace \
  --version v1.16.2 \
  --set crds.enabled=true \
  --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \
  --set config.kind="ControllerConfiguration" \
  --set config.enableGatewayAPI=true

For HTTP-01 solving, cert-manager can create a temporary HTTPRoute instead of an Ingress:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: apps-gw
                namespace: gateway-infra
                kind: Gateway

Critical detail during coexistence: if your existing Ingress NGINX still serves the hostname and holds the DNS record, HTTP-01 challenges will land on NGINX, not on the Gateway. Two safe options: keep issuing certificates through the existing Ingress solver until DNS is switched, or — far better — use DNS-01. DNS-01 is independent of which data plane owns the IP, which removes an entire class of migration failures.

Step 3 — The Gateway, reusing the existing certificate

Here's the trick that avoids re-issuing certs on day one: point the new Gateway at the same Secret your Ingress already uses. No new ACME order, no rate-limit risk, identical chain.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: apps-gw
  namespace: gateway-infra
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: eg-external
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      hostname: "*.example.com"
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"
    - name: https-shop
      protocol: HTTPS
      port: 443
      hostname: shop.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: shop-example-com-tls
            namespace: shop
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"

Two things to notice. First, allowedRoutes uses a label selector, not All — namespaces must explicitly opt in. Second, the certificate Secret lives in the shop namespace while the Gateway lives in gateway-infra. That cross-namespace reference is denied by default. You need a ReferenceGrant in the namespace that owns the Secret:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-gw-to-read-shop-tls
  namespace: shop
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: gateway-infra
  to:
    - group: ""
      kind: Secret
      name: shop-example-com-tls

Note the name field in to: — grant access to that one Secret, not to every Secret in the namespace.

Verify the listener is actually programmed. This is the single most useful command of the whole migration:

kubectl -n gateway-infra get gateway apps-gw \
  -o jsonpath='{range .status.listeners[*]}{.name}{"\t"}{range .conditions[*]}{.type}={.status}{" "}{end}{"\n"}{end}'

A ResolvedRefs=False with reason RefNotPermitted means your ReferenceGrant is missing or mismatched.

Step 4 — The HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
    - name: apps-gw
      namespace: gateway-infra
      sectionName: https-shop
  hostnames:
    - shop.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            set:
              - name: X-Forwarded-Proto
                value: https
      backendRefs:
        - name: shop-api
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: shop-frontend
          port: 8080

Don't forget the namespace label required by allowedRoutes:

kubectl label namespace shop gateway-access=true

If a rule needs to target a Service in another namespace (say payments), add a ReferenceGrant in payments with from.kind: HTTPRoute. Same mechanism, same least-privilege rule: name the target Service explicitly.

Step 5 — Validate TLS before touching DNS

At this point both data planes serve shop.example.com, but only NGINX has the DNS record. Test the Gateway directly by resolving the hostname yourself:

GW_IP=$(kubectl -n envoy-gateway-system get svc \
  -l gateway.envoyproxy.io/owning-gateway-name=apps-gw \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

curl -sSv --resolve shop.example.com:443:$GW_IP https://shop.example.com/healthz

Then compare the certificate served by each data plane. Identical fingerprints mean zero client-visible change:

fp() { openssl s_client -connect "$1:443" -servername shop.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256; }

NGINX_IP=$(dig +short shop.example.com | head -1)
echo "nginx:   $(fp $NGINX_IP)"
echo "gateway: $(fp $GW_IP)"

Also check the full chain and SAN list — a frequent failure mode is a Gateway serving only the leaf certificate because the Secret was built by hand:

openssl s_client -connect $GW_IP:443 -servername shop.example.com -showcerts </dev/null 2>/dev/null \
  | grep -E 's:|i:'

openssl s_client -connect $GW_IP:443 -servername shop.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -ext subjectAltName

And make sure an unknown SNI doesn't silently fall through to the wrong certificate:

openssl s_client -connect $GW_IP:443 -servername notmydomain.example.com </dev/null 2>&1 | head -20

Step 6 — Shift traffic by weight, without DNS

DNS-based cutovers are coarse: TTLs lie, resolvers cache, and rollback takes minutes. A much tighter loop is to make Ingress NGINX itself forward a percentage of traffic to the Gateway, using its canary annotations plus an ExternalName Service.

apiVersion: v1
kind: Service
metadata:
  name: gw-bridge
  namespace: shop
spec:
  type: ExternalName
  externalName: envoy-gateway-infra-apps-gw-7f9c2d11.envoy-gateway-system.svc.cluster.local
  ports:
    - name: https
      port: 443

Get the real Service name first:

kubectl -n envoy-gateway-system get svc \
  -l gateway.envoyproxy.io/owning-gateway-name=apps-gw -o name

Then a canary Ingress on the same hostname:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-canary-to-gateway
  namespace: shop
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/upstream-vhost: "shop.example.com"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-name: "shop.example.com"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [shop.example.com]
      secretName: shop-example-com-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gw-bridge
                port:
                  number: 443

Now you have a real dial. Ramp it with a single patch, watch your SLOs, and roll back in seconds:

for w in 5 25 50 100; do
  kubectl -n shop annotate ingress shop-canary-to-gateway \
    nginx.ingress.kubernetes.io/canary-weight="$w" --overwrite
  sleep 600
done

# instant rollback
kubectl -n shop annotate ingress shop-canary-to-gateway \
  nginx.ingress.kubernetes.io/canary-weight="0" --overwrite

You can also pin the canary to a header (nginx.ingress.kubernetes.io/canary-by-header: x-use-gateway) so QA can target the new path deterministically before any percentage of real users is affected.

Once 100% has been stable for a few days, switch DNS to the Gateway's IP, wait out the TTL, then delete the canary Ingress and the legacy Ingress. The same weighting primitive exists natively inside Gateway API for backend-level canaries:

      backendRefs:
        - name: shop-frontend
          port: 8080
          weight: 90
        - name: shop-frontend-next
          port: 8080
          weight: 10

Security best practices

Never use allowedRoutes.namespaces.from: All on an internet-facing Gateway. Any namespace in the cluster could then attach a route to your public listener and claim a hostname. Use Same or a label Selector, and treat the label as a privileged operation controlled by the platform team.

Scope every ReferenceGrant. Always set to[].name. A grant without a name allows every Secret (or every Service) of that kind in the namespace — including, potentially, the Secret holding a wildcard certificate.

Enforce TLS floor and cipher policy. Gateway API leaves this to implementation-specific policies. With Envoy Gateway:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: tls-hardening
  namespace: gateway-infra
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: apps-gw
  tls:
    minVersion: "1.2"
    maxVersion: "1.3"
    ciphers:
      - ECDHE-ECDSA-AES256-GCM-SHA384
      - ECDHE-RSA-AES256-GCM-SHA384
      - ECDHE-ECDSA-CHACHA20-POLY1305
  headers:
    xForwardedClientCert:
      mode: Sanitize

Redirect HTTP to HTTPS at the route level, so a misconfigured app can never be served in clear text:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-redirect
  namespace: shop
spec:
  parentRefs:
    - name: apps-gw
      namespace: gateway-infra
      sectionName: http
  hostnames: ["shop.example.com"]
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

Encrypt the last hop too. If your backend speaks TLS, use BackendTLSPolicy with explicit CA validation and hostname verification rather than disabling verification:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: shop-api-upstream-tls
  namespace: shop
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: shop-api
      sectionName: "8443"
  validation:
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: internal-ca
    hostname: shop-api.shop.svc.cluster.local

Lock down the data plane with NetworkPolicy. The Envoy pods should accept traffic from the LoadBalancer and reach only the namespaces they route to. On OpenShift, confirm the proxy runs under restricted-v2 and not a relaxed SCC:

oc -n envoy-gateway-system get pod -l app.kubernetes.io/name=envoy \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.openshift\.io/scc}{"\n"}{end}'

Audit certificate ownership after the switch. When you finally delete the legacy Ingress, make sure cert-manager still owns the Certificate object — deleting an Ingress with the cert-manager.io/cluster-issuer annotation can garbage-collect the Certificate and, depending on your setup, the Secret with it. Create a standalone Certificate resource explicitly rather than relying on annotation-driven shims for anything production-critical:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shop-example-com
  namespace: shop
spec:
  secretName: shop-example-com-tls
  duration: 2160h
  renewBefore: 720h
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  dnsNames:
    - shop.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Alert on expiry independently. Scrape certmanager_certificate_expiration_timestamp_seconds and alert at 14 days, plus a black-box probe that checks the certificate actually served on port 443. A valid Secret that the Gateway failed to load is a silent outage waiting for renewal day.

Conclusion

Migrating off Ingress NGINX doesn't have to be a maintenance-window event. The key takeaways:

Coexist first. Gateway API and Ingress can serve the same hostname simultaneously. Build the Gateway, validate it out-of-band with curl --resolve, and only then send it traffic.
Reuse the existing TLS Secret for the first cutover. Comparing SHA-256 fingerprints between the old and new data plane is a 10-second check that eliminates the scariest failure mode.
ReferenceGrant is a security boundary, not a formality. Scope it to a named Secret or Service, and keep allowedRoutes on a label selector.
Shift traffic with weights, not DNS. The NGINX canary + ExternalName bridge gives you a percentage dial with sub-second rollback. Move DNS only once 100% has been stable for days.
Prefer DNS-01 during the transition, so certificate issuance never depends on which data plane currently owns the public IP.
Harden explicitly. TLS floor, HTTP→HTTPS redirect, upstream TLS validation and NetworkPolicy are all policy objects you have to write — nothing is secure by default just because it's new.

Do one low-traffic hostname end to end first. The second one takes an afternoon, and by the fifth you'll be deleting Ingress objects without thinking about it.

Upgrading an OpenShift Cluster Without Waking the On-Call: Channels, MCPs and Controlled Pauses

Florient DOGBE — Sun, 01 Nov 2026 17:03:07 +0000

An OpenShift upgrade is two very different operations wearing the same trench coat. The first one is a control-plane operator rollout driven by the Cluster Version Operator (CVO): fast, mostly invisible, and reversible only in narrow cases. The second one is a fleet-wide node reboot campaign driven by the Machine Config Operator (MCO): slow, disruptive, and entirely at the mercy of your PodDisruptionBudgets.

On a 12-node lab cluster nobody notices. On a 300-node cluster with stateful workloads, a default maxUnavailable: 1 on the worker pool means a reboot campaign that runs for 20 hours straight — right across your 3 a.m. quiet window, with no natural checkpoint. That is how upgrades end up paging the on-call.

This article breaks down the machinery — ClusterVersion, update channels, MachineConfigPool — and shows how to turn a monolithic upgrade into a wave-based rollout with real maintenance windows and a usable abort path.

Context: the two clocks of an OpenShift upgrade

The CVO reconciles a single cluster-scoped object, ClusterVersion/version. You declare a target payload; the CVO walks the manifests of the release image in dependency order, updating every ClusterOperator. Control-plane nodes get their new RHCOS config from the MCO as part of that process, and because there are only three of them with maxUnavailable: 1 hardcoded, it is bounded and predictable.

Then the worker pools start. The MCO renders a new MachineConfig per pool, cordons a node, drains it, applies the OS update via rpm-ostree, reboots, uncordons, and moves to the next. The CVO reports Completed for the version long before the last worker has rebooted — oc get clusterversion will happily show 4.16.21 while 180 nodes are still pending. That gap is exactly where unplanned incidents live.

The lever you control is the MachineConfigPool. Everything below is about using it.

The CVO side: ClusterVersion and channels

Reading the graph before touching anything

oc adm upgrade
oc get clusterversion version -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'
oc get co

Three things matter in that output:

Upgradeable=False blocks minor upgrades. The message tells you why: a pending admin ack for removed Kubernetes APIs, an operator in a bad state, or an OLM operator with no compatible version for the target.
Recommended vs conditional updates. oc adm upgrade only lists recommended targets. Use oc adm upgrade --include-not-recommended to see conditional ones along with the risk reason (a known bug affecting a specific platform or configuration). Read the risk before you accept it.
Degraded ClusterOperators. The CVO refuses to start if an operator is Degraded=True, unless you pass --allow-upgrade-with-warnings. Don't.

Channel semantics

Channels are not "quality tiers" in the sense people assume. A given 4.16.z release lands in candidate-4.16, then fast-4.16, then stable-4.16 after a soak period driven by telemetry. The payload is identical; only the recommendation timing differs. eus-4.16 exists only for even-numbered minors and carries the extended-support stream plus the edges needed for EUS-to-EUS hops.

A sane fleet policy: non-production on fast-4.y, production on stable-4.y, and EUS-pinned clusters on eus-4.y. Switching channel is declarative and non-destructive:

# Move to the next minor's channel to expose the upgrade edges
oc adm upgrade channel stable-4.16

# Target a specific patch, never "latest"
oc adm upgrade --to=4.16.21

Always pin the exact version. --to-latest is a moving target between the moment you write the change request and the moment you execute it.

If you need to abort before any manifest has been applied:

oc adm upgrade --clear

This only works while the update has not actually started rolling. Once ClusterOperators begin moving, the forward path is the only path.

The MCO side: MachineConfigPool as a rollout controller

maxUnavailable is your throughput dial

The worker pool defaults to maxUnavailable: 1. On a large cluster, compute your window budget first:

window_duration / (drain_time + reboot_time) * maxUnavailable = nodes per window

With a 12-minute average per node (drain included) and a 4-hour window, maxUnavailable: 1 gives you 20 nodes. maxUnavailable: 10% on a 200-node pool gives 20 concurrent reboots — far too aggressive for most workloads. Something between 3 and 8 % is usually the sweet spot, but validate against your PDBs:

oc get pdb -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed'

Any PDB with disruptionsAllowed: 0 will stall a drain indefinitely. The MCO retries the drain for an hour, then marks the pool Degraded with NodeDrainFailed. Fix those before the window, not during.

Custom pools are your waves

A node carrying a custom role label leaves the worker pool and joins the matching custom pool, which inherits worker MachineConfigs through machineConfigSelector. That is the primitive for wave-based rollouts:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: wave-a
spec:
  # Pick up every rendered worker config, plus anything scoped to wave-a
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values:
          - worker
          - wave-a
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/wave-a: ""
  maxUnavailable: 2
  paused: true

# Canary wave: 5 nodes spread across failure domains
for n in worker-a-01 worker-b-01 worker-c-01 worker-a-02 worker-b-02; do
  oc label node "$n" node-role.kubernetes.io/wave-a=
done

oc get mcp wave-a -o jsonpath='{.status.machineCount}{"\n"}'

Create wave-a, wave-b, wave-c (and leave the remainder in worker), all with paused: true. A node must never match two custom pools — the MCO will mark it degraded with node has been in multiple pools.

One caveat on labelling: moving a node between pools triggers a config change and therefore a reboot. Do the labelling work before the upgrade, in a quiet period, so that upgrade day only flips paused.

Orchestrating the rollout

Phase 0 — preflight, D-2

# etcd backup on a control-plane node
oc debug node/master-0 -- chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup

# No firing alerts that would mask upgrade symptoms
oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
  curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing") | .labels.alertname' | sort -u

# Admin acks for removed APIs (key name is in the Upgradeable message)
oc -n openshift-config patch cm admin-acks --type=merge \
  -p '{"data":{"ack-4.15-kube-1.29-api-removals-in-4.16":"true"}}'

Also confirm every OLM-managed operator has a channel compatible with the target minor; an operator stuck on an incompatible channel is a classic post-upgrade outage.

Phase 1 — pause the fleet, upgrade the control plane

for p in worker wave-a wave-b wave-c; do
  oc patch mcp/$p --type=merge -p '{"spec":{"paused":true}}'
done

oc adm upgrade channel stable-4.16
oc adm upgrade --to=4.16.21

# Follow progress (oc 4.16+ ships a richer status view)
watch -n 30 'oc get clusterversion version; oc get co | grep -v "True        False         False"'

With all worker pools paused, the blast radius is the control plane only: three sequential master reboots, operators rolled out, zero worker drain. This typically fits in a 90-minute window.

Phase 2 — canary wave

oc patch mcp/wave-a --type=merge -p '{"spec":{"paused":false}}'

watch -n 20 'oc get mcp wave-a -o custom-columns=\
NAME:.metadata.name,UPDATED:.status.updatedMachineCount,\
TOTAL:.status.machineCount,DEGRADED:.status.degradedMachineCount'

Per-node visibility when something stalls:

oc get nodes -l node-role.kubernetes.io/wave-a -o custom-columns=\
'NAME:.metadata.name,STATE:.metadata.annotations.machineconfiguration\.openshift\.io/state,\
CURRENT:.metadata.annotations.machineconfiguration\.openshift\.io/currentConfig,\
DESIRED:.metadata.annotations.machineconfiguration\.openshift\.io/desiredConfig'

# machine-config-daemon logs for a stuck node
oc -n openshift-machine-config-operator logs -l k8s-app=machine-config-daemon \
  --field-selector spec.nodeName=worker-a-01 --tail=200

Then bake. Give the canary 24 to 48 hours of real traffic before opening the next wave. That bake time is the entire point of the exercise.

Phase 3 — remaining waves

Unpause one pool per maintenance window, and re-pause it at the end of the window if it has not finished:

# End of window: freeze whatever is left, mid-flight nodes finish cleanly
oc patch mcp/wave-b --type=merge -p '{"spec":{"paused":true}}'

Pausing mid-rollout does not interrupt a node already draining; it only stops the MCO from picking up new ones. That is your natural checkpoint — and the reason the on-call phone stays quiet.

EUS-to-EUS: skipping a minor without double-draining

EUS channels let you go from 4.14 to 4.16 with a single worker reboot campaign instead of two. The mechanics are exactly the pause trick, applied across two consecutive control-plane upgrades:

# 1. Pause every worker-bearing pool
for p in worker wave-a wave-b wave-c; do
  oc patch mcp/$p --type=merge -p '{"spec":{"paused":true}}'
done

# 2. 4.14 -> 4.15 (control plane only)
oc adm upgrade channel eus-4.16
oc adm upgrade --to=4.15.44

# 3. 4.15 -> 4.16 (control plane only)
oc adm upgrade --to=4.16.21

# 4. Only now, unpause wave by wave
oc patch mcp/wave-a --type=merge -p '{"spec":{"paused":false}}'

Two constraints are non-negotiable. First, the control plane can only ever be two minors ahead of the kubelet (the Kubernetes version-skew policy), so you cannot chain three minors. Second, a paused pool stops receiving the rotated kubelet client CA. The MachineConfigControllerPausedPoolKubeletCA alert fires when that becomes dangerous; if you ignore it long enough, nodes lose their ability to authenticate to the API server. Treat a paused pool as a resource with a TTL measured in days, not weeks, and track it explicitly.

Rollback: what is actually possible

Be honest with the change advisory board: a completed minor upgrade is not reversible. CRD schema migrations, etcd storage version bumps and operator-managed data changes have no downgrade path. What you do have:

Abort before start — oc adm upgrade --clear, valid only while no manifest has been applied.
Z-stream rollback — recent oc builds expose oc adm upgrade rollback, restricted to returning to the previously installed patch release within the same minor. Check oc adm upgrade --help on your client version before relying on it.
Partial rollback at pool level — this is the realistic one. Pools still paused have never left the old OS version. If the canary wave misbehaves, you keep 95 % of the fleet on the known-good RHCOS image while you investigate, and the cluster keeps running in a supported skew.
MachineConfig rollback — deleting a custom MachineConfig makes the MCO render the previous configuration and roll the pool back to it. This covers your own kernel args, chrony or kubelet tweaks, not the OS payload itself.

On 4.17+, nodeDisruptionPolicy removes reboots entirely for many file and unit changes, shrinking the rollback surface:

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
spec:
  nodeDisruptionPolicy:
    files:
      - path: /etc/chrony.conf
        actions:
          - type: Reload
            reload:
              serviceName: chronyd.service

Security best practices

Never use --force. It bypasses release-image signature verification and precondition checks. If oc adm upgrade refuses a target, the answer is to fix the precondition, not to disable the guardrail. For disconnected clusters, verify the mirrored payload explicitly and declare mirrors through ImageDigestMirrorSet so signature validation still resolves.

Lock down who can write to ClusterVersion and MachineConfigPool. Both are cluster-scoped and both are production-destroying. patch on machineconfigpools is effectively "reboot the fleet" or "silently stop security patches" — audit it like a privileged credential and restrict it to a dedicated upgrade ServiceAccount used by your pipeline.

Treat paused: true as a security exception. A paused pool stops receiving RHCOS CVE fixes and certificate rotations. Add a monitoring rule that alerts on any pool paused for more than 72 hours, and require a ticket reference in the pool annotations.

Do not use spec.overrides in ClusterVersion to silence a stuck operator. It makes the CVO stop managing that component — including its security updates — and it sets Upgradeable=False for everything that follows.

Handle admin acks deliberately. ack-*-api-removals-* keys exist because workloads still calling removed APIs will break. Query the apirequestcounts resource (oc get apirequestcounts -o json) to identify the callers before acking — not after.

Protect the etcd backup. It contains every Secret in the cluster in plaintext unless etcd encryption is enabled. Encrypt at rest, restrict access, and set a retention policy.

Key takeaways

ClusterVersion drives the control plane; MachineConfigPool drives the nodes. They complete at very different times, and only the second one touches your workloads.
Pin an exact patch version, read conditional-update risks, and clear Upgradeable=False properly instead of forcing through it.
Pre-create custom pools (wave-a, wave-b, …) with tuned maxUnavailable, label nodes ahead of time, and upgrade by unpausing one pool per maintenance window.
Pausing a pool is a checkpoint, not a parking spot: kubelet CA rotation and CVE patching stop with it.
EUS-to-EUS halves your reboot campaigns, at the cost of a longer pause window you must actively monitor.
Rollback is a pool-level concept, not a cluster-level one. Design the waves so that "stop here" is always a valid, supported state.

OpenShift Pipelines (Tekton): From a Monolithic Jenkinsfile to Reusable Tasks

Florient DOGBE — Fri, 30 Oct 2026 15:13:24 +0000

Most teams that adopt OpenShift arrive with luggage: a Jenkinsfile that started at 60 lines and now weighs 800, with nested script {} blocks, a shared library nobody dares to touch, and three agents whose Docker images were last rebuilt in 2021. It works — until a new service needs the same build logic, and the only migration path is copy-paste.

OpenShift Pipelines (the supported downstream of Tekton) changes the unit of reuse. Instead of one file describing the whole flow, you get Task objects: versioned, namespaced (or clusterwide), testable in isolation, and composed into Pipeline objects. This article is a pragmatic migration guide: how to cut a monolithic Jenkinsfile into Tasks, how to handle workspaces, dependency resolution and build cache, and how to trigger everything from Git with an EventListener.

Context: why the monolith hurts on Kubernetes

A Jenkinsfile is executable Groovy. That is its strength and its curse. Logic, orchestration, and infrastructure assumptions all live in the same file:

pipeline {
  agent { label 'maven-jdk17' }
  stages {
    stage('Checkout') { steps { checkout scm } }
    stage('Build')    { steps { sh 'mvn -B -s settings.xml clean package' } }
    stage('Test')     { steps { sh 'mvn -B verify' } }
    stage('Image')    { steps { sh 'podman build -t $REG/$APP:$TAG . && podman push ...' } }
    stage('Deploy')   { steps { sh 'oc apply -f k8s/ && oc rollout status deploy/$APP' } }
  }
}

Three problems surface once you run this at scale on OpenShift:

No reuse boundary. stage('Image') is identical in 40 repositories, but each repo owns its own copy. One CVE in the build tooling means 40 pull requests.
Implicit state. Jenkins gives you a persistent workspace directory on the agent, so .m2 cache, node_modules and the Git checkout "just exist". On Kubernetes, every step is a container in a Pod; nothing persists unless you declare it.
Agent drift. The maven-jdk17 label points to a VM or image maintained outside Git. Tekton forces each step to name its own image, which is both more verbose and far more auditable.

The goal of migration is not a 1:1 translation. It is to extract the five or six genuinely reusable Tasks your organisation needs, version them, and leave the per-application glue in a thin Pipeline.

Concept mapping

Jenkins	Tekton / OpenShift Pipelines
`stage`	`Task` (or a `step` inside a Task)
`agent` / label	`step.image` + `podTemplate` / `taskRunTemplate`
Workspace directory	`Workspace` backed by PVC, emptyDir, ConfigMap or Secret
Shared library	Catalog of `Task` / `StepAction` resources, versioned in Git or OCI bundles
Credentials binding	`Secret` + ServiceAccount, or a Secret-backed workspace
Multibranch webhook	`EventListener` + `TriggerBinding` + `TriggerTemplate`
Build history	`PipelineRun` objects (pruned by the operator)

Getting the platform ready

OpenShift Pipelines ships as an operator. Verify it and install the CLI:

# Operator and version
oc get csv -n openshift-operators | grep pipelines

# Cluster-wide Tasks shipped by Red Hat (git-clone, buildah, s2i, ...)
oc get clustertasks 2>/dev/null | head
oc get tasks -n openshift-pipelines | head

# tkn CLI, matched to the cluster
tkn version

Create a dedicated namespace and service account for the pipeline — do not reuse pipeline SA defaults blindly across teams:

oc new-project payments-ci
oc create sa pipeline-runner -n payments-ci

Step 1 — Cut the Jenkinsfile into Tasks

Apply one rule: a Task is reusable only if it takes everything it needs as params and workspaces, and exposes everything downstream needs as results. No hardcoded registry, no hardcoded namespace.

Here is the Maven build extracted as a Task, with an explicit cache workspace:

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: maven-build
  namespace: payments-ci
  labels:
    app.kubernetes.io/version: "0.4"
spec:
  description: Build a Maven project, reusing a persistent ~/.m2 cache.
  params:
    - name: GOALS
      type: array
      default: ["-B", "clean", "package"]
    - name: MAVEN_IMAGE
      type: string
      default: registry.access.redhat.com/ubi9/openjdk-17:1.20
    - name: CONTEXT_DIR
      type: string
      default: "."
  workspaces:
    - name: source
      description: Checked-out sources.
    - name: maven-cache
      description: Persistent local repository.
      optional: true
    - name: maven-settings
      description: ConfigMap or Secret holding settings.xml.
      optional: true
  results:
    - name: ARTIFACT_VERSION
      description: Project version read from the POM.
  steps:
    - name: build
      image: $(params.MAVEN_IMAGE)
      workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
      args: ["$(params.GOALS[*])"]
      env:
        - name: MAVEN_OPTS
          value: -Dmaven.repo.local=$(workspaces.maven-cache.path)/repository
      script: |
        #!/usr/bin/env bash
        set -euo pipefail
        SETTINGS=""
        if [ -f "$(workspaces.maven-settings.path)/settings.xml" ]; then
          SETTINGS="-s $(workspaces.maven-settings.path)/settings.xml"
        fi
        mvn ${SETTINGS} "$@"
        mvn -q help:evaluate -Dexpression=project.version -DforceStdout \
          | tee $(results.ARTIFACT_VERSION.path)

Two details matter here:

optional: true workspaces let the same Task run with or without a cache. Local developers can start a PipelineRun with an emptyDir; CI mounts a PVC.
Results are files, not environment variables. Write to $(results.<name>.path) with no trailing newline issues (tee is fine, Tekton trims trailing whitespace).

Versioning Tasks

Treat Tasks like libraries. Two viable strategies:

Git + resolver. Keep Tasks in a platform-tekton-catalog repo and reference them by tag with the Git resolver. No cluster-side copy to keep in sync.
OCI bundles. Push Tasks to your registry with tkn bundle push and reference them by digest. Best for air-gapped clusters and for signing.

# OCI bundle, immutable reference
tkn bundle push quay.io/acme/tekton-catalog/maven-build:0.4 -f task-maven-build.yaml

In a Pipeline, pin the version:

- name: build
  taskRef:
    resolver: bundles
    params:
      - name: bundle
        value: quay.io/acme/tekton-catalog/maven-build@sha256:3f1d...   # digest, not tag
      - name: name
        value: maven-build
      - name: kind
        value: task

Never reference a mutable tag in production pipelines. A digest is the Tekton equivalent of a pinned shared-library version.

Step 2 — Workspaces, dependency resolution and cache

This is where most migrations fail. Jenkins gave you a sticky filesystem; Tekton gives you declarative volumes.

Source workspace

Use a volumeClaimTemplate so each PipelineRun gets a fresh, automatically deleted PVC:

workspaces:
  - name: source
    volumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2Gi

With RWO storage, every Task sharing that PVC must be scheduled on the same node. Tekton's affinity assistant handles this automatically (it creates a placeholder StatefulSet that pins the Pods). Check the feature flag before you fight phantom scheduling issues:

oc get cm feature-flags -n openshift-pipelines -o jsonpath='{.data.coschedule}{"\n"}'
# expected: workspaces  (default) — pins Pods sharing a workspace to one node

Cache workspace

The source workspace dies with the run; the cache must not. Create a long-lived PVC per team and language runtime:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: maven-cache
  namespace: payments-ci
spec:
  accessModes: ["ReadWriteMany"]   # RWX: concurrent PipelineRuns
  resources:
    requests:
      storage: 20Gi
  storageClassName: ocs-storagecluster-cephfs

If you only have RWO storage (common on bare metal with LVM or on some cloud CSI drivers), you have two options:

One cache PVC per pipeline, ReadWriteOnce, and accept serialised runs (PipelineRun concurrency 1 via a mutex Task or an admission policy).
Externalise the cache: a cache-fetch / cache-upload Task pair that pulls a tarball from S3/ODF object storage keyed by the hash of pom.xml or package-lock.json. Slower cold start, but fully parallel and node-independent.

The key-based approach is worth the extra two Tasks once you exceed ~10 concurrent builds:

- name: cache-key
  image: registry.access.redhat.com/ubi9/ubi-minimal:9.4
  script: |
    #!/bin/sh
    set -e
    sha256sum $(workspaces.source.path)/pom.xml | cut -c1-16 \
      > $(results.CACHE_KEY.path)

Step 3 — Compose the Pipeline

The Pipeline is now thin: ordering, parameter plumbing, and workspace binding.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: payments-api
  namespace: payments-ci
spec:
  params:
    - name: git-url
    - name: git-revision
      default: main
    - name: image
      default: image-registry.openshift-image-registry.svc:5000/payments-ci/payments-api
  workspaces:
    - name: source
    - name: maven-cache
    - name: maven-settings
    - name: dockerconfig
  results:
    - name: image-digest
      value: $(tasks.build-image.results.IMAGE_DIGEST)
  tasks:
    - name: clone
      taskRef: { name: git-clone, kind: Task }
      params:
        - name: url
          value: $(params.git-url)
        - name: revision
          value: $(params.git-revision)
      workspaces:
        - name: output
          workspace: source

    - name: build
      runAfter: ["clone"]
      taskRef: { name: maven-build, kind: Task }
      params:
        - name: GOALS
          value: ["-B", "clean", "verify"]
      workspaces:
        - name: source
          workspace: source
        - name: maven-cache
          workspace: maven-cache
        - name: maven-settings
          workspace: maven-settings

    - name: build-image
      runAfter: ["build"]
      taskRef: { name: buildah, kind: Task }
      params:
        - name: IMAGE
          value: $(params.image):$(tasks.build.results.ARTIFACT_VERSION)
        - name: TLSVERIFY
          value: "true"
      workspaces:
        - name: source
          workspace: source
        - name: dockerconfig
          workspace: dockerconfig

  finally:
    - name: report
      taskRef: { name: slack-notify, kind: Task }
      params:
        - name: status
          value: $(tasks.status)

finally replaces the Jenkins post { always { ... } } block, and $(tasks.status) gives you Succeeded, Failed or Completed.

Run and watch it with the CLI:

tkn pipeline start payments-api \
  -p git-url=https://github.com/acme/payments-api.git \
  -p git-revision=main \
  -w name=source,volumeClaimTemplateFile=./wsp-source.yaml \
  -w name=maven-cache,claimName=maven-cache \
  -w name=maven-settings,config=maven-settings \
  -w name=dockerconfig,secret=registry-push \
  --serviceaccount pipeline-runner \
  --use-param-defaults --showlog

tkn pr list
tkn pr logs -f --last
tkn pr describe --last
tkn taskrun logs payments-api-run-abc123-build -f

tkn pr logs -f --last is the command you will type a thousand times; make an alias.

Step 4 — Trigger from Git with an EventListener

Triggers replace the Jenkins multibranch webhook. Three objects: a TriggerBinding (extract fields from the payload), a TriggerTemplate (render a PipelineRun), an EventListener (HTTP endpoint + interceptors).

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
  name: github-push
  namespace: payments-ci
spec:
  params:
    - name: git-url
      value: $(body.repository.clone_url)
    - name: git-revision
      value: $(body.after)
---
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: payments-api-template
  namespace: payments-ci
spec:
  params:
    - name: git-url
    - name: git-revision
  resourcetemplates:
    - apiVersion: tekton.dev/v1
      kind: PipelineRun
      metadata:
        generateName: payments-api-
      spec:
        taskRunTemplate:
          serviceAccountName: pipeline-runner
        pipelineRef:
          name: payments-api
        params:
          - name: git-url
            value: $(tt.params.git-url)
          - name: git-revision
            value: $(tt.params.git-revision)
        timeouts:
          pipeline: "1h0m0s"
        workspaces:
          - name: source
            volumeClaimTemplate:
              spec:
                accessModes: ["ReadWriteOnce"]
                resources:
                  requests:
                    storage: 2Gi
          - name: maven-cache
            persistentVolumeClaim:
              claimName: maven-cache
          - name: maven-settings
            configMap:
              name: maven-settings
          - name: dockerconfig
            secret:
              secretName: registry-push
---
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: payments-el
  namespace: payments-ci
spec:
  serviceAccountName: tekton-triggers-sa
  triggers:
    - name: github-push-main
      interceptors:
        - ref:
            name: github
          params:
            - name: secretRef
              value:
                secretName: github-webhook-secret
                secretKey: secretToken
            - name: eventTypes
              value: ["push"]
        - ref:
            name: cel
          params:
            - name: filter
              value: "body.ref == 'refs/heads/main'"
      bindings:
        - ref: github-push
      template:
        ref: payments-api-template

Expose it with a TLS-terminated route (never plain HTTP for a webhook):

oc create secret generic github-webhook-secret \
  --from-literal=secretToken="$(openssl rand -hex 20)" -n payments-ci

oc create route edge payments-el --service=el-payments-el \
  --port=8080 --insecure-policy=Redirect -n payments-ci

oc get route payments-el -o jsonpath='{.spec.host}{"\n"}'
tkn eventlistener logs payments-el -f

Point the GitHub/GitLab webhook at https://<route-host> with the same secret. The github interceptor validates the HMAC signature; the CEL interceptor filters branches so you do not start a build for every tag push.

Security best practices

CI is the most privileged system in most organisations: it can read all source code and push to production registries. Treat it accordingly.

One ServiceAccount per pipeline, not a shared one. Give pipeline-runner only the RBAC it needs (push to its own ImageStream, deploy to its own namespace). The Triggers SA needs create on pipelineruns and nothing else — bind the shipped tekton-triggers-eventlistener-roles ClusterRole, not edit.
Avoid privileged builds. The Red Hat buildah Task runs rootless with the pipelines-scc SCC. Do not grant privileged to work around a storage driver error; set STORAGE_DRIVER=vfs or use fuse-overlayfs instead.
Never pass secrets as params. Params end up in the PipelineRun spec, visible to anyone with get pipelineruns. Mount secrets as workspaces (secret: secretName: ...) or bind them to the ServiceAccount.
Scope registry credentials. Prefer the internal registry with a ServiceAccount token over a long-lived dockerconfigjson for quay.io. If you need an external registry, use a robot account restricted to one repository.
Validate webhook payloads. The github/gitlab interceptors verify the HMAC signature — always configure secretRef. An unauthenticated EventListener is remote code execution with extra steps.
Pin Task references by digest (OCI bundles) or by Git tag with commit verification. A mutable :latest catalog Task is a supply-chain backdoor.
Enable Tekton Chains for SLSA provenance and image signing; it is bundled with OpenShift Pipelines and signs TaskRun outputs automatically.
Apply a NetworkPolicy on the CI namespace: allow egress to the registry, Git and artifact proxy; deny lateral traffic to application namespaces.
Set timeouts and resource limits. A Pipeline without a timeout is a PVC leak and a node-pressure incident waiting to happen. Also configure the operator's pruner to delete old PipelineRun objects.

Conclusion

Migrating from Jenkins to Tekton is not a file conversion, it is a refactoring exercise. The mental shift is straightforward once you accept it: a Task is a function with typed inputs (params, workspaces) and typed outputs (results), and a Pipeline is the call graph.

Key takeaways:

Start by extracting 5–6 Tasks, not 30. Build, test, image, deploy, notify. Everything else stays inline until a second consumer appears.
Declare your state. The source workspace is ephemeral (volumeClaimTemplate), the dependency cache is persistent (RWX PVC or object storage keyed by lockfile hash). Getting this wrong is the main cause of "Tekton is slower than Jenkins" complaints.
Version Tasks as artifacts, referenced by digest through the bundles or git resolver. That is what finally kills the copy-paste problem.
Triggers replace the multibranch job: EventListener + interceptors give you signature validation and CEL filtering in a declarative, reviewable form.
Live in tkn. tkn pr logs -f --last, tkn pr describe, tkn eventlistener logs will cover 90% of your debugging.

Run both systems in parallel for a few sprints: let Jenkins keep deploying while Tekton builds and publishes images. When the Tekton PipelineRun has been green for two weeks, flip the webhook and delete the Jenkinsfile. The diff will be the best part of the migration.

Illusory Cloud Backups: Actually Testing Your DR Plan with Velero and a Timed Restore

Florient DOGBE — Thu, 29 Oct 2026 14:27:28 +0000

Your backup job has been green for 14 months. Your Grafana dashboard shows a reassuring velero_backup_success_total counter climbing steadily. Your compliance auditor ticked the box. And yet, if your production cluster disappeared tonight, you have no idea whether you could bring the platform back — nor how long it would take.

This is not a hypothetical. The failure mode is boringly consistent: backups succeed, restores fail. Custom Resource Definitions come back after the Custom Resources that depend on them. PersistentVolumes restore empty because the CSI driver was swapped. Secrets are present but the operator that consumes them crash-loops because an owner reference points at a UID that no longer exists. The Completed status in velero backup get tells you a tarball landed in an S3 bucket. It tells you nothing about recoverability.

This article builds a complete, timed restore scenario — PVs, CRDs, secrets — on OpenShift/Kubernetes with Velero, and turns it into an automated test you can run on a schedule. The goal is simple: replace your declared RTO/RPO with measured RTO/RPO.

Context: why "backup success" is a meaningless metric

RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are contractual numbers. In most organisations they were written in a Word document by someone who never ran velero restore create.

RPO is bounded by your backup schedule and by the consistency of what was captured. A backup taken every 6 hours with no filesystem quiescing gives you a 6-hour RPO on paper, and an unknown RPO in practice if your database was mid-write.
RTO is the full wall-clock time from "we decide to restore" to "the service answers requests correctly". That includes cluster provisioning, Velero installation, CRD reconciliation, volume rehydration from object storage, operator convergence, and DNS/ingress propagation. The velero restore command itself is often the fastest part.

Velero's architecture explains where the time goes. Kubernetes object manifests are tiny JSON blobs in an S3 bucket — restoring 5,000 objects takes seconds. Volume data is the expensive part, and the mechanism matters enormously:

Mechanism	Backup target	Restore speed	Cross-cluster / cross-region
CSI snapshots (`VolumeSnapshotClass`)	Storage backend	Very fast (CoW clone)	Usually no, unless the snapshot is replicated
File System Backup (restic / Kopia)	S3 bucket	Slow, data-size bound	Yes
CSI snapshot + data movement	S3 bucket	Medium	Yes

A team that benchmarked restores using local CSI snapshots and then has to rebuild in another region from restic will discover a 20x gap in RTO at the worst possible moment.

Setting up Velero correctly for a testable DR

Install with the right plugins

On OpenShift, prefer the OADP operator (OpenShift API for Data Protection) which packages Velero. On vanilla Kubernetes, the Velero CLI is fine. The key decision is enabling both CSI snapshots and File System Backup, so you can measure both paths.

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0,velero/velero-plugin-for-csi:v0.7.1 \
  --bucket velero-dr-eu-west-3 \
  --prefix prod-cluster \
  --backup-location-config region=eu-west-3 \
  --snapshot-location-config region=eu-west-3 \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --uploader-type kopia \
  --features=EnableCSI \
  --namespace velero

--use-node-agent deploys the DaemonSet that performs File System Backup. --uploader-type kopia is the modern replacement for restic; restic remains supported and the CLI flags are identical (--default-volumes-to-fs-backup). If you are on an older Velero, substitute --uploader-type restic.

On OpenShift, the node agent needs a privileged SCC to read pod volumes from the host:

oc adm policy add-scc-to-user privileged -z velero -n openshift-adp
oc adm policy add-scc-to-user privileged -z node-agent -n openshift-adp

Declare a VolumeSnapshotClass Velero can use

Velero only picks up a VolumeSnapshotClass that carries the right label. Without it, CSI backups silently fall back to nothing.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-gce-pd-snapclass
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: pd.csi.storage.gke.io
deletionPolicy: Retain

deletionPolicy: Retain matters: with Delete, removing the VolumeSnapshot object during cleanup destroys the underlying storage snapshot and your restore point with it.

A schedule that encodes your RPO

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: prod-apps-hourly
  namespace: velero
spec:
  schedule: "0 * * * *"
  useOwnerReferencesInBackup: false
  template:
    includedNamespaces:
      - payments
      - identity
    excludedResources:
      - events
      - events.events.k8s.io
    snapshotVolumes: true
    defaultVolumesToFsBackup: false
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 168h
    hooks:
      resources:
        - name: pg-consistent-snapshot
          includedNamespaces:
            - payments
          labelSelector:
            matchLabels:
              app: postgresql
          pre:
            - exec:
                container: postgresql
                command:
                  - /bin/bash
                  - -c
                  - "psql -U postgres -c \"SELECT pg_backup_start('velero');\""
                onError: Fail
                timeout: 3m
          post:
            - exec:
                container: postgresql
                command:
                  - /bin/bash
                  - -c
                  - "psql -U postgres -c \"SELECT pg_backup_stop();\""
                onError: Continue
                timeout: 3m

The hooks are what turn a crash-consistent snapshot into an application-consistent one. onError: Fail on the pre-hook is deliberate — a backup you cannot trust is worse than a missing backup, because it gives false confidence.

Building the timed restore scenario

The test must run against a separate cluster (or at minimum a separate namespace with a rewritten namespace mapping). Restoring into the live namespace proves nothing and risks production.

Step 1: define what "restored" means

Before touching Velero, write the assertions. A restore is successful when:

All Deployments/StatefulSets in the target namespace are Available with full replica count.
The PVC contains a known canary file or database row written just before the backup.
The application answers a functional HTTP probe (not just a liveness probe).
CRs reconciled by operators reach their expected status.phase.

Plant the canary before the backup so you can verify the actual data, not just the volume's existence:

oc -n payments exec deploy/ledger -- \
  sh -c 'echo "canary-$(date -u +%Y%m%dT%H%M%SZ)" > /data/dr-canary.txt'

Step 2: the restore manifest

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: dr-drill-20240612
  namespace: velero
spec:
  backupName: prod-apps-hourly-20240612010014
  includedNamespaces:
    - payments
  namespaceMapping:
    payments: payments-dr
  restorePVs: true
  existingResourcePolicy: none
  includeClusterResources: true
  hooks:
    resources:
      - name: wait-for-db
        includedNamespaces:
          - payments-dr
        labelSelector:
          matchLabels:
            app: postgresql
        postHooks:
          - exec:
              container: postgresql
              command:
                - /bin/bash
                - -c
                - "until pg_isready -U postgres; do sleep 2; done"
              execTimeout: 5m
              waitTimeout: 10m

includeClusterResources: true pulls in CRDs, ClusterRoles and PVs. This is the setting most often forgotten — without it, a restore into a fresh cluster produces namespaces full of CRs whose CRDs do not exist, and Velero will log error restoring ...: no matches for kind.

Step 3: the timing harness

Here is the core of the drill: a script that measures each phase and emits structured results.

#!/usr/bin/env bash
set -euo pipefail

NS_SRC="payments"
NS_DST="payments-dr"
RESTORE="dr-drill-$(date -u +%Y%m%d%H%M%S)"
BACKUP="$(velero backup get -o json \
  | jq -r '.items[] | select(.status.phase=="Completed")
           | [.metadata.name, .status.completionTimestamp] | @tsv' \
  | sort -k2 | tail -1 | cut -f1)"

echo "Using backup: ${BACKUP}"

# --- RPO: age of the restore point at the moment the drill starts
BACKUP_TS=$(velero backup get "${BACKUP}" -o json | jq -r '.status.completionTimestamp')
RPO_SECONDS=$(( $(date -u +%s) - $(date -u -d "${BACKUP_TS}" +%s) ))

T0=$(date -u +%s)

velero restore create "${RESTORE}" \
  --from-backup "${BACKUP}" \
  --namespace-mappings "${NS_SRC}:${NS_DST}" \
  --include-cluster-resources=true \
  --wait

T_VELERO=$(date -u +%s)

# --- Wait for workloads to converge
oc -n "${NS_DST}" wait --for=condition=Available deployment --all --timeout=15m
oc -n "${NS_DST}" rollout status statefulset/postgresql --timeout=15m

T_WORKLOAD=$(date -u +%s)

# --- Functional assertion: canary file present on the restored PV
CANARY=$(oc -n "${NS_DST}" exec deploy/ledger -- cat /data/dr-canary.txt)
[[ -n "${CANARY}" ]] || { echo "FAIL: canary missing"; exit 1; }

# --- Functional assertion: application answers correctly
oc -n "${NS_DST}" run dr-probe --rm -i --restart=Never \
  --image=registry.access.redhat.com/ubi9/ubi-minimal:latest -- \
  curl -sf --max-time 10 http://ledger:8080/healthz/ready > /dev/null

T_READY=$(date -u +%s)

cat <<EOF
{
  "restore": "${RESTORE}",
  "backup": "${BACKUP}",
  "rpo_seconds": ${RPO_SECONDS},
  "velero_restore_seconds": $((T_VELERO - T0)),
  "workload_convergence_seconds": $((T_WORKLOAD - T_VELERO)),
  "functional_ready_seconds": $((T_READY - T_WORKLOAD)),
  "rto_seconds": $((T_READY - T0)),
  "canary": "${CANARY}"
}
EOF

Three measurements, not one. If your RTO regresses, the breakdown tells you whether the cause is Velero (more objects, slower S3), workload convergence (operator reconcile loops, image pulls), or application warm-up (cache rebuild, JIT).

Step 4: automate the drill

Run it weekly as a Kubernetes CronJob in the DR cluster, with a ServiceAccount scoped tightly:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: dr-drill
  namespace: velero
spec:
  schedule: "0 3 * * 1"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 10
  jobTemplate:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 5400
      template:
        spec:
          serviceAccountName: dr-drill
          restartPolicy: Never
          containers:
            - name: drill
              image: quay.io/myorg/dr-drill:1.4.0
              command: ["/usr/local/bin/dr-drill.sh"]
              env:
                - name: PUSHGATEWAY_URL
                  value: http://prometheus-pushgateway.monitoring:9091
              resources:
                requests:
                  cpu: 100m
                  memory: 256Mi
                limits:
                  memory: 512Mi
              securityContext:
                allowPrivilegeEscalation: false
                runAsNonRoot: true
                capabilities:
                  drop: ["ALL"]
                seccompProfile:
                  type: RuntimeDefault

Push the JSON output to Prometheus Pushgateway and alert on it. The alert that matters is not "backup failed" but:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: dr-objectives
  namespace: monitoring
spec:
  groups:
    - name: dr
      rules:
        - alert: RTOBudgetExceeded
          expr: dr_drill_rto_seconds > 1800
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Measured RTO ({{ $value }}s) exceeds the 30 min objective"
        - alert: DRDrillStale
          expr: time() - dr_drill_last_success_timestamp > 10 * 24 * 3600
          labels:
            severity: critical
          annotations:
            summary: "No successful DR drill in the last 10 days"

DRDrillStale is the single most valuable alert in this article. A DR plan that is not exercised decays silently.

Handling the hard parts

CRDs and ordering

Velero restores resources in a built-in priority order (namespaces, then CRDs, then cluster-scoped, then namespaced). It does not wait for operators to reconcile. If your CR requires a webhook served by a Deployment that is itself being restored, the API server will reject the CR.

Two mitigations: split the restore into two passes (infrastructure first, workloads second), or use --existing-resource-policy=update on the second pass. For OLM-managed operators, restoring Subscription and ClusterServiceVersion objects is usually a mistake — let OLM reinstall the operator from the catalog and restore only the CRs.

velero restore create dr-infra --from-backup "${BACKUP}" \
  --include-resources crds,namespaces,storageclasses,volumesnapshotclasses \
  --wait

velero restore create dr-apps --from-backup "${BACKUP}" \
  --exclude-resources crds \
  --namespace-mappings payments:payments-dr --wait

Secrets that cannot be restored as-is

Service account tokens, kubernetes.io/service-account-token secrets and anything bound to a cluster UID must be excluded — they will be regenerated. Externally managed secrets (External Secrets Operator, Vault Agent) should also be excluded and resynced from source of truth.

velero backup create prod-apps \
  --include-namespaces payments \
  --exclude-resources 'secrets.bootstrap.kubernetes.io' \
  --selector 'velero.io/exclude-from-backup notin (true)'

Label the secrets you do not want captured:

oc -n payments label secret vault-synced-db-creds velero.io/exclude-from-backup=true

StorageClass mismatch across clusters

A restore into a different cloud region or provider will fail if the PVC references a StorageClass that does not exist. Use a Velero ConfigMap to remap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  gp3-encrypted: standard-csi
  premium-rwo: standard-csi

Security best practices

Dedicated bucket, dedicated credentials. The Velero IAM principal needs s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket scoped to one prefix — never s3:* on *. Prefer IRSA (EKS), Workload Identity (GKE) or OpenShift's CloudCredential operator over long-lived access keys.
Object Lock in compliance mode. Enable S3 Object Lock with a retention period at least equal to your backup TTL. This is the only control that survives an attacker with cluster-admin who runs velero backup delete --all. Note that Velero's own deletion of expired backups will fail against locked objects; manage expiry with S3 lifecycle rules instead.
Encrypt at rest and in transit. SSE-KMS on the bucket with a customer-managed key, and a key policy that denies the Velero role kms:ScheduleKeyDeletion. Kopia/restic repositories are additionally encrypted with a repository password — store it in a separate secret and back it up out-of-band, because without it your S3 data is unrecoverable.
Treat backups as production data for compliance. A backup of a namespace containing PII is PII. Same retention rules, same region constraints, same access logging (s3:GetObject events in CloudTrail with alerting on unusual read volume).
Isolate the DR cluster's RBAC. The drill ServiceAccount needs read on Velero CRs and write on the target namespace only. It should never hold cluster-admin, even though the restore operation itself requires broad permissions performed by the Velero controller.
Never restore into production by accident. Guard the drill script with an explicit check on the cluster's API server URL or a well-known ConfigMap flag, and refuse to run if the target namespace lacks a dr-drill=true label.

Key takeaways

A green backup is a necessary condition for recovery, never a sufficient one. The only evidence of recoverability is a successful restore with functional assertions.
Measure RTO in three segments — Velero restore, workload convergence, application readiness — so regressions are attributable.
Measure real RPO as the age of the restore point at drill time, not as the cron interval.
includeClusterResources: true plus a two-pass restore (CRDs first, workloads second) solves the large majority of restore failures in operator-heavy clusters.
CSI snapshots and File System Backup have radically different restore times. Benchmark the path you will actually use in a disaster, which is usually the slow one.
Alert on drill staleness, not just drill failure. An untested plan degrades to zero value within months.

Start small: pick one namespace with a PV, plant a canary, run the timed restore manually this week, and write down the number. Whatever it is, it will be more useful than the number currently in your DR document.

GCP: Cutting Cross-Region Latency with Cloud Load Balancing and Anycast IP

Florient DOGBE — Wed, 28 Oct 2026 10:19:20 +0000

A user in Frankfurt hitting an API hosted in us-central1 pays a fixed tax: roughly 90–110 ms of round-trip time before a single byte of application logic runs. Add TLS handshake and you have burned 300 ms before your handler is invoked. No amount of code optimization fixes the speed of light.

Google Cloud's global external Application Load Balancer solves part of this problem by terminating the connection at the closest of ~190 Google Front End (GFE) points of presence, then carrying the request over Google's private backbone to the nearest healthy backend. The entry point is a single anycast IPv4/IPv6 address — one DNS record, no GeoDNS, no per-region hostnames.

This article walks through building that front end: multi-region backends (MIGs and serverless NEGs), health checks tuned for fast failover, gcloud and Terraform implementations, GKE standalone NEGs, and a Cloud Armor baseline.

Context: why DNS-based geo-routing is not enough

The classic pattern is a weighted/geo DNS record pointing at regional load balancers: api.example.com resolves to the EU IP for European resolvers, the US IP for American ones. It works, but it has three structural problems.

TTL is your failover budget. If a region dies, clients keep hammering the dead IP until their resolver cache expires. Browsers and JVMs cache aggressively and ignore your 30s TTL. Real failover time is measured in minutes.

Resolver location ≠ client location. A client using a public DNS resolver may be geolocated hundreds of kilometres away. EDNS Client Subnet helps, but coverage is partial.

TCP and TLS still start at the origin. With DNS routing, the three-way handshake and the TLS handshake travel the full distance to the region. With an anycast front end, the handshake terminates at the PoP a few milliseconds away; only the HTTP request crosses the ocean — and it does so on a warm, multiplexed connection over Google's backbone, not the public internet.

Measured effect on a Paris → us-central1 path: a cold HTTPS request drops from roughly 480 ms to 180–220 ms, mostly because handshake round-trips are localised.

Architecture overview

The target topology:

                    api.example.com → 34.x.x.x (global anycast)
                                 │
                        ┌────────┴────────┐
                        │  GFE / PoP      │  TLS termination, Cloud Armor, Cloud CDN
                        └────────┬────────┘
                                 │  Google backbone
          ┌──────────────────────┼──────────────────────┐
          │                      │                      │
   europe-west1            us-central1            asia-southeast1
   MIG / GKE NEG           MIG / GKE NEG          Cloud Run NEG

One global backend service holds all regional backends. The GFE picks the closest backend that still has capacity and passes health checks; when a region is unhealthy or saturated, traffic spills to the next-closest one automatically.

Components, in dependency order: health check → backend service → backends (NEGs/MIGs) → URL map → target HTTPS proxy → global forwarding rule + static IP.

Building the front end with gcloud

1. Reserve the anycast address

export PROJECT_ID=my-project
gcloud config set project "$PROJECT_ID"

# Global external IPv4 — anycast, announced from every Google PoP
gcloud compute addresses create api-global-ipv4 \
  --ip-version=IPV4 \
  --network-tier=PREMIUM \
  --global

gcloud compute addresses create api-global-ipv6 \
  --ip-version=IPV6 \
  --network-tier=PREMIUM \
  --global

gcloud compute addresses describe api-global-ipv4 --global --format='value(address)'

PREMIUM tier is mandatory for the global ALB. Standard tier gives you regional routing over the public internet — exactly what we are trying to avoid.

2. A health check that actually fails fast

Default health check settings (5s interval, 2 failures) mean up to 15 seconds of blackholing. Tighten it, and point it at a dedicated endpoint that checks dependencies — not at /.

gcloud compute health-checks create http hc-api-readyz \
  --port=8080 \
  --request-path=/readyz \
  --check-interval=2s \
  --timeout=1s \
  --healthy-threshold=2 \
  --unhealthy-threshold=2 \
  --enable-logging \
  --global

With 2s × 2 = 4s detection plus propagation, a region is drained in under 10 seconds. --enable-logging emits a Cloud Logging entry on every health state transition — invaluable during incidents:

gcloud logging read \
  'resource.type="gce_instance_group" AND jsonPayload.healthCheckProbeResult.detailedHealthState!=""' \
  --limit=20 --format='table(timestamp, jsonPayload.healthCheckProbeResult.detailedHealthState)'

Do not go below 1s interval: probes come from every GFE region, so a 1s interval on a 50-instance MIG generates meaningful load.

3. Backend service with multi-region backends

gcloud compute backend-services create api-backend \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --protocol=HTTP \
  --port-name=http \
  --health-checks=hc-api-readyz \
  --global-health-checks \
  --timeout=30s \
  --connection-draining-timeout=60s \
  --enable-logging --logging-sample-rate=1.0 \
  --global

# Europe: managed instance group
gcloud compute backend-services add-backend api-backend \
  --instance-group=api-mig-euw1 \
  --instance-group-region=europe-west1 \
  --balancing-mode=RATE \
  --max-rate-per-instance=120 \
  --capacity-scaler=1.0 \
  --global

# US: managed instance group
gcloud compute backend-services add-backend api-backend \
  --instance-group=api-mig-usc1 \
  --instance-group-region=us-central1 \
  --balancing-mode=RATE \
  --max-rate-per-instance=120 \
  --capacity-scaler=1.0 \
  --global

EXTERNAL_MANAGED selects the modern (Envoy-based) global ALB rather than the classic one — required for advanced traffic management such as outlier detection and weighted routing.

--balancing-mode=RATE with --max-rate-per-instance defines when a region is "full". Once Europe exceeds its aggregate capacity, overflow goes to the US instead of queuing. --capacity-scaler=0.0 drains a region without removing it — your manual kill switch during a regional rollout.

4. Serverless NEGs for Cloud Run regions

A serverless NEG maps a region's Cloud Run (or App Engine/Cloud Functions) service into the same backend service:

for REGION in europe-west1 us-central1 asia-southeast1; do
  gcloud compute network-endpoint-groups create neg-api-"$REGION" \
    --region="$REGION" \
    --network-endpoint-type=SERVERLESS \
    --cloud-run-service=api

  gcloud compute backend-services add-backend api-backend \
    --network-endpoint-group=neg-api-"$REGION" \
    --network-endpoint-group-region="$REGION" \
    --global
done

Important gotcha: serverless NEG backends do not support health checks, balancing modes, or capacity scalers. Availability is handled by the Cloud Run control plane, and the GFE routes to the closest region with a deployed revision. If you need explicit health-based failover semantics, use MIGs or GKE NEGs — or mix: serverless NEGs for low-traffic regions, MIGs where you need capacity control.

5. URL map, proxy, forwarding rule

gcloud compute url-maps create api-urlmap --default-service=api-backend --global

gcloud compute ssl-certificates create api-cert \
  --domains=api.example.com --global

gcloud compute target-https-proxies create api-https-proxy \
  --url-map=api-urlmap \
  --ssl-certificates=api-cert \
  --ssl-policy=tls12-modern \
  --global

gcloud compute forwarding-rules create api-fr-https \
  --load-balancing-scheme=EXTERNAL_MANAGED \
  --address=api-global-ipv4 \
  --target-https-proxy=api-https-proxy \
  --ports=443 \
  --global

Repeat the forwarding rule for the IPv6 address. Add an HTTP→HTTPS redirect URL map on port 80; never leave port 80 serving content.

GKE: standalone zonal NEGs

If your workloads run on GKE in several regions, expose each cluster's pods as standalone NEGs and attach them to the same global backend service. Pods become load balancer endpoints directly — no extra hop through a node port.

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: production
  annotations:
    cloud.google.com/neg: '{"exposed_ports": {"8080": {"name": "neg-api-gke-euw1"}}}'
    cloud.google.com/backend-config: '{"default": "api-backendconfig"}'
spec:
  type: ClusterIP
  selector:
    app: api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: api-backendconfig
  namespace: production
spec:
  timeoutSec: 30
  connectionDraining:
    drainingTimeoutSec: 60
  healthCheck:
    type: HTTP
    requestPath: /readyz
    port: 8080
    checkIntervalSec: 2
    timeoutSec: 1
    healthyThreshold: 2
    unhealthyThreshold: 2
  securityPolicy:
    name: api-armor-policy
  logging:
    enable: true
    sampleRate: 1.0

The NEG controller creates one zonal NEG per zone where pods run. Attach them all:

for ZONE in europe-west1-b europe-west1-c europe-west1-d; do
  gcloud compute backend-services add-backend api-backend \
    --network-endpoint-group=neg-api-gke-euw1 \
    --network-endpoint-group-zone="$ZONE" \
    --balancing-mode=RATE --max-rate-per-endpoint=100 \
    --global
done

Make the readiness probe meaningful, since it drives both Kubernetes and GFE decisions:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  replicas: 6
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
      containers:
        - name: api
          image: europe-docker.pkg.dev/my-project/api/api:1.8.3
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 2
            failureThreshold: 2
          lifecycle:
            preStop:
              exec: { command: ["sleep", "15"] }
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits: { memory: "512Mi" }

The preStop sleep of 15s covers the window between pod termination and NEG endpoint removal — without it you will see 502s on every rollout.

Terraform

The same stack, declaratively. This is the version you actually want in production.

resource "google_compute_global_address" "api" {
  name       = "api-global-ipv4"
  ip_version = "IPV4"
}

resource "google_compute_health_check" "api" {
  name                = "hc-api-readyz"
  check_interval_sec  = 2
  timeout_sec         = 1
  healthy_threshold   = 2
  unhealthy_threshold = 2

  http_health_check {
    port         = 8080
    request_path = "/readyz"
  }

  log_config { enable = true }
}

resource "google_compute_region_network_endpoint_group" "api" {
  for_each              = toset(["europe-west1", "us-central1", "asia-southeast1"])
  name                  = "neg-api-${each.key}"
  region                = each.key
  network_endpoint_type = "SERVERLESS"

  cloud_run { service = "api" }
}

resource "google_compute_backend_service" "api" {
  name                  = "api-backend"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  protocol              = "HTTP"
  timeout_sec           = 30
  security_policy       = google_compute_security_policy.api.id

  dynamic "backend" {
    for_each = google_compute_region_network_endpoint_group.api
    content {
      group = backend.value.id
    }
  }

  log_config {
    enable      = true
    sample_rate = 1.0
  }
}

resource "google_compute_url_map" "api" {
  name            = "api-urlmap"
  default_service = google_compute_backend_service.api.id
}

resource "google_compute_managed_ssl_certificate" "api" {
  name = "api-cert"
  managed { domains = ["api.example.com"] }
}

resource "google_compute_target_https_proxy" "api" {
  name             = "api-https-proxy"
  url_map          = google_compute_url_map.api.id
  ssl_certificates = [google_compute_managed_ssl_certificate.api.id]
  ssl_policy       = google_compute_ssl_policy.modern.id
}

resource "google_compute_ssl_policy" "modern" {
  name            = "tls12-modern"
  profile         = "MODERN"
  min_tls_version = "TLS_1_2"
}

resource "google_compute_global_forwarding_rule" "api" {
  name                  = "api-fr-https"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  ip_address            = google_compute_global_address.api.address
  port_range            = "443"
  target                = google_compute_target_https_proxy.api.id
}

For MIG or GKE NEG backends, add balancing_mode, max_rate_per_instance and capacity_scaler inside the backend block, plus health_checks = [google_compute_health_check.api.id] on the backend service.

Testing geographic routing and failover

Have your application echo the serving region (K_SERVICE / GOOGLE_CLOUD_REGION on Cloud Run, the node label on GKE) in a response header such as X-Served-By.

Measure from VMs in several regions:

for R in europe-west1-b us-central1-a asia-southeast1-a; do
  gcloud compute instances create probe-$R --zone=$R \
    --machine-type=e2-micro --image-family=debian-12 \
    --image-project=debian-cloud --no-address 2>/dev/null
done

gcloud compute ssh probe-europe-west1-b --zone=europe-west1-b --command \
  'curl -s -o /dev/null -D - -w "ttfb=%{time_starttransfer}s total=%{time_total}s\n" https://api.example.com/healthz | grep -Ei "x-served-by|ttfb"'

Expected: the European probe reports a European backend with a TTFB under 30 ms; the Asian probe reports asia-southeast1.

Force a failover by blocking health check probes in one region. Google's probes come from 35.191.0.0/16 and 130.211.0.0/22:

gcloud compute firewall-rules create deny-hc-euw1 \
  --network=prod-vpc --action=DENY --rules=tcp:8080 --priority=900 \
  --source-ranges=35.191.0.0/16,130.211.0.0/22 \
  --target-tags=api-euw1

# Watch the backend flip to UNHEALTHY
watch -n 2 'gcloud compute backend-services get-health api-backend --global \
  --format="table(status.healthStatus[].instance, status.healthStatus[].healthState)"'

While the rule is active, European clients should transparently receive responses from us-central1 with a higher TTFB. Delete the rule and verify traffic returns. Do this in a staging project first, and script the rollback.

A less invasive drain, for rollouts:

gcloud compute backend-services update-backend api-backend \
  --instance-group=api-mig-euw1 --instance-group-region=europe-west1 \
  --capacity-scaler=0.0 --global

Security best practices

Put Cloud Armor in front of every backend service. A global anycast IP is reachable by every scanner on the internet within minutes of creation.

gcloud compute security-policies create api-armor-policy \
  --description="Baseline WAF + rate limiting"

# OWASP preconfigured rules (sensitivity 1 = low false positives)
gcloud compute security-policies rules create 1000 \
  --security-policy=api-armor-policy \
  --expression="evaluatePreconfiguredExpr('sqli-v33-stable', ['owasp-crs-v030301-id942251-sqli'])" \
  --action=deny-403

gcloud compute security-policies rules create 1100 \
  --security-policy=api-armor-policy \
  --expression="evaluatePreconfiguredExpr('xss-v33-stable')" \
  --action=deny-403

# Per-IP rate limiting with temporary ban
gcloud compute security-policies rules create 2000 \
  --security-policy=api-armor-policy \
  --action=rate-based-ban \
  --rate-limit-threshold-count=600 \
  --rate-limit-threshold-interval-sec=60 \
  --ban-duration-sec=300 \
  --conform-action=allow \
  --exceed-action=deny-429 \
  --enforce-on-key=IP \
  --src-ip-ranges='*'

gcloud compute backend-services update api-backend \
  --security-policy=api-armor-policy --global

Always deploy new WAF rules with --preview first, inspect jsonPayload.enforcedSecurityPolicy.outcome in the load balancer logs, then enforce.

Lock down the backends. Instances and GKE nodes must only accept traffic from GFE ranges. Everything else is denied:

gcloud compute firewall-rules create allow-gfe-to-api \
  --network=prod-vpc --action=ALLOW --rules=tcp:8080 --priority=1000 \
  --source-ranges=35.191.0.0/16,130.211.0.0/22 --target-tags=api

gcloud compute firewall-rules create deny-all-ingress-api \
  --network=prod-vpc --action=DENY --rules=all --priority=65000 \
  --source-ranges=0.0.0.0/0 --target-tags=api

Backends should have no external IP; use Cloud NAT for egress. For GKE, use private clusters with authorized networks on the control plane.

TLS and transport. Enforce a MODERN SSL policy with TLS_1_2 minimum, enable HSTS at the application layer, and use managed certificates so renewal is not a human task. If your backends require encryption in transit, switch the backend protocol to HTTPS or HTTP2.

Identity. For internal tooling behind the same LB, enable Identity-Aware Proxy instead of IP allowlists. For service-to-service calls into Cloud Run, keep the service private (--no-allow-unauthenticated) and let the serverless NEG handle ingress with --ingress=internal-and-cloud-load-balancing.

Observability as a control. Enable LB logging at 100% during rollouts, export to BigQuery, and alert on backend_latencies p95 per backend_target_name and on request_count filtered by response_code_class=500. A latency regression in one region is the earliest signal of a partial outage.

Key takeaways

A single global anycast IP removes DNS TTL from your failover path and moves TLS termination within a few milliseconds of the user. That is the bulk of the perceived latency win.
One global backend service, many regional backends. The GFE handles proximity routing and overflow; you only decide capacity through --balancing-mode, --max-rate-per-instance and --capacity-scaler.
Health checks are the failover SLA. A 2s interval with a 2-failure threshold, probing a dependency-aware /readyz, gives sub-10-second drains. Enable health check logging before you need it.
Serverless NEGs trade control for simplicity — no health checks, no capacity scalers. Mix them with MIG or GKE NEG backends where you need deterministic failover behaviour.
GKE standalone NEGs route directly to pods; pair them with BackendConfig, topology spread constraints and a preStop delay to avoid 502s on rollout.
Ship the whole thing as Terraform, and keep capacity-scaler as an operational escape hatch you can flip from a runbook.
Cloud Armor is not optional on an internet-facing anycast IP. Preconfigured OWASP rules in preview mode, rate-based bans, and GFE-only firewall rules are the minimum baseline.

Deploy it, then prove it: measure TTFB from three continents, kill a region's health checks, and watch traffic move. An untested failover is a hypothesis, not an architecture.

Isolating East-West Traffic: Default-Deny NetworkPolicy and Observability with Cilium Hubble

Florient DOGBE — Mon, 26 Oct 2026 13:36:55 +0000

Most Kubernetes clusters I audit are flat networks wearing a service mesh costume. Ingress is locked down with a WAF, mTLS terminates at the edge, secrets live in Vault — and any compromised pod in namespace-a can open a TCP connection to the PostgreSQL primary in namespace-b, or to the cloud metadata endpoint at 169.254.169.254, without a single log line being written.

The fix is well known: default-deny NetworkPolicy. The reason it is rarely applied to an existing cluster is also well known: nobody knows what actually talks to what. Applying a blanket deny to a production namespace and waiting for pager duty to tell you what broke is not a migration strategy.

This article describes the approach that works: observe first, generate policies from real traffic, enforce in audit mode, then flip to deny. Cilium Hubble gives you the flow data; CiliumNetworkPolicy gives you the enforcement primitives that standard NetworkPolicy lacks (DNS-aware egress, L7 rules, cluster-wide policies).

Context: why east-west is the blind spot

Kubernetes has no native network isolation. The default model is "every pod can reach every pod, in every namespace, on every port". NetworkPolicy is an opt-in allowlist mechanism: as soon as one policy selects a pod, that pod becomes default-deny for the direction(s) covered by that policy. That nuance is where most teams get it wrong — a policy with only an ingress block leaves egress wide open.

The attack path that matters:

An RCE in a Java app pod (Log4Shell-style, or a vulnerable dependency).
The attacker enumerates the service CIDR and the cluster DNS.
Lateral movement to the Redis cache with no auth, to an internal admin API, or to the Kubernetes API server with the pod's ServiceAccount token.
Egress to a C2 server, or to the cloud provider IMDS to steal node credentials.

Steps 2 to 4 are all east-west or egress. None of them touch your Ingress controller. Without network policies, none of them produce a denial you can alert on.

Three additional reasons this matters now: PCI-DSS 4.0 and SOC2 both explicitly ask for network segmentation between workloads of different trust levels; the CIS Kubernetes Benchmark flags namespaces without NetworkPolicy; and egress filtering is the cheapest reliable detection point for data exfiltration.

Step 0: verify your CNI actually enforces policies

Before anything else, confirm that your cluster's CNI enforces NetworkPolicy. On OpenShift, OVN-Kubernetes does. On a self-managed cluster running flannel, it does not — the API objects are accepted silently and nothing happens. This silent no-op has burned more than one team into thinking they were segmented.

# Which CNI?
kubectl get pods -n kube-system -l k8s-app=cilium -o wide
kubectl -n openshift-ovn-kubernetes get pods 2>/dev/null | head

# Cilium status and whether Hubble is on
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | grep -iE 'hubble|policy'

# Install the CLI tools if needed
cilium status --wait
cilium hubble enable --ui

A quick functional smoke test is worth the two minutes:

kubectl create ns np-test
kubectl -n np-test run victim --image=registry.access.redhat.com/ubi9/ubi-minimal --command -- sleep 3600
kubectl -n np-test expose pod victim --port=8080
kubectl apply -n np-test -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector:
    matchLabels:
      run: victim
  policyTypes:
    - Ingress
EOF
kubectl -n np-test run probe --rm -it --image=registry.access.redhat.com/ubi9/ubi-minimal \
  --restart=Never -- curl -m 3 -s -o /dev/null -w '%{http_code}\n' http://victim:8080
# Expect a timeout / exit 28, not a connection refused.

Step 1: observe before you enforce

Hubble is Cilium's flow observability layer. It reads flow events from the eBPF datapath, so it sees every packet decision including the ones a sidecar-based mesh never sees (raw TCP, UDP, DNS, ICMP).

Enable the Hubble Relay port-forward and start looking:

cilium hubble port-forward &

# Every flow in a namespace, live
hubble observe --namespace payments --follow

# Only flows leaving the namespace (the ones that need egress rules)
hubble observe --namespace payments --not --to-namespace payments --follow

# DNS resolution performed by a given workload — critical for egress design
hubble observe --namespace payments --pod checkout -t l7 --protocol dns

# Anything already being dropped (misconfigured policy, or legitimate noise)
hubble observe --verdict DROPPED --last 500

Collecting a representative sample

A ten-minute capture during a quiet afternoon is useless. You need a window that covers your batch jobs, CronJobs, certificate renewals, nightly reconciliation, and your monthly billing run. In practice: capture for at least a full business cycle, 7 days minimum, 30 if you have monthly jobs.

Hubble's in-memory ring buffer is bounded (default 4095 flows per node). For a real capture, export to a file and aggregate:

# Long-running capture to JSONL
hubble observe --namespace payments --follow -o jsonl >> /tmp/payments-flows.jsonl

# Reduce to a unique (source identity -> destination identity : port) tuple set
jq -r 'select(.flow.verdict=="FORWARDED")
  | [ .flow.source.namespace, (.flow.source.labels | join(",")),
      .flow.destination.namespace, (.flow.destination.labels | join(",")),
      .flow.l4.TCP.destination_port // .flow.l4.UDP.destination_port ]
  | @tsv' /tmp/payments-flows.jsonl | sort -u

For production, do not rely on a port-forward and a shell loop. Enable the Hubble metrics exporter and the file-based flow log on each node:

# Helm values for cilium (hubble section)
hubble:
  enabled: true
  relay:
    enabled: true
  metrics:
    enableOpenMetrics: true
    enabled:
      - "dns:query;ignoreAAAA"
      - "drop:sourceContext=workload-name;destinationContext=workload-name"
      - "flow:sourceContext=workload-name;destinationContext=workload-name"
      - "policy:sourceContext=workload-name;destinationContext=workload-name"
  export:
    static:
      enabled: true
      filePath: /var/run/cilium/hubble/events.log
      allowList:
        - '{"verdict":["DROPPED","ERROR"]}'

Ship events.log to Loki or your SIEM with your existing log agent. You now have a durable, queryable flow archive instead of a ring buffer.

Step 2: derive policies from observed flows

Cilium can do a first pass for you. hubble observe combined with the policy generation in Cilium Enterprise, or the open-source cilium policy tooling, produces a draft — but treat the output as a starting point, never as the final artifact. Generated policies encode accidents (a developer's curl from a debug pod) as permanent allow rules.

The discipline that works: generate, then rewrite by hand using label selectors that express intent, not the labels that happen to exist.

The baseline: cluster-wide default deny

Start with a CiliumClusterwideNetworkPolicy that denies everything except the non-negotiable primitives (DNS, health checks). Apply it in audit mode first — see Step 3.

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny-all
spec:
  description: "Baseline default-deny for all namespaces except control plane"
  endpointSelector:
    matchExpressions:
      - key: k8s:io.kubernetes.pod.namespace
        operator: NotIn
        values:
          - kube-system
          - cilium-system
          - openshift-dns
          - openshift-ingress
          - openshift-monitoring
  ingress:
    - fromEntities:
        - health
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
          rules:
            dns:
              - matchPattern: "*"

Two things to note. First, the rules.dns block makes Cilium proxy DNS at L7 — this is what enables toFQDNs later. Second, excluding the platform namespaces is pragmatic: isolating openshift-monitoring is a separate project with its own blast radius.

Per-workload allowlists

Now express the real flows. Example for a checkout service that talks to Postgres, Redis, and one external payment provider:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: checkout-allow
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: checkout
  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/name: api-gateway
            k8s:io.kubernetes.pod.namespace: edge
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "POST"
                path: "/v1/checkout.*"
              - method: "GET"
                path: "/healthz"
  egress:
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: postgres
            k8s:io.kubernetes.pod.namespace: data
      toPorts:
        - ports: [{ port: "5432", protocol: TCP }]
    - toEndpoints:
        - matchLabels:
            app.kubernetes.io/name: redis
            k8s:io.kubernetes.pod.namespace: data
      toPorts:
        - ports: [{ port: "6379", protocol: TCP }]
    - toFQDNs:
        - matchName: "api.stripe.com"
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]

toFQDNs is the feature that makes egress filtering practical. Cilium's DNS proxy observes the resolution, caches the returned IPs, and programs them into the eBPF policy map for the TTL duration. You write api.stripe.com, not a CIDR list that rotates weekly. It requires the DNS L7 rule from the baseline policy to be in effect for that endpoint.

If you prefer to stay on upstream NetworkPolicy for portability, the same intent in vanilla form — note that FQDN egress is simply not expressible:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-allow
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: checkout
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: edge
          podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - port: 8080
          protocol: TCP
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: data
          podSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
      ports:
        - port: 5432
          protocol: TCP
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP

Remember the kubernetes.io/metadata.name label is set automatically by the API server on every namespace since 1.21 — use it instead of hand-maintained namespace labels.

Step 3: audit mode, the no-downtime switch

This is the part that lets you ship to production without an outage. Cilium supports a per-endpoint policy audit mode: policy is evaluated, verdicts are recorded as AUDIT in Hubble, but packets are still forwarded.

# Find the endpoint IDs for a workload on a given node
NODE_POD=$(kubectl -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker-02 -o name | head -1)

kubectl -n kube-system exec $NODE_POD -- cilium-dbg endpoint list | grep checkout

# Enable audit mode on endpoint 1234
kubectl -n kube-system exec $NODE_POD -- cilium-dbg endpoint config 1234 PolicyAuditMode=Enabled

# Watch what *would* be denied
hubble observe --namespace payments --verdict AUDIT --follow

Run in audit mode for a full business cycle. Every AUDIT verdict is a flow your policy does not cover. Triage each one into two buckets: legitimate (add a rule) or unexpected (you just found an undocumented dependency, or something worse).

For a cluster-wide rollout, set policyAuditMode: true in the Cilium Helm values on a canary node pool first, rather than per-endpoint by hand.

# Count the AUDIT flows per source/destination pair to prioritise
hubble observe --verdict AUDIT --last 5000 -o jsonl \
  | jq -r '[.flow.source.pod_name, .flow.destination.pod_name,
            (.flow.l4.TCP.destination_port // .flow.l4.UDP.destination_port)] | @tsv' \
  | sort | uniq -c | sort -rn | head -30

When the audit stream is empty for 24 hours, disable audit mode and the policy becomes enforcing. Nothing changed in the datapath configuration except one flag.

Step 4: validate and keep it validated

Enforcement without a regression test decays. Add negative tests to CI:

# Should FAIL (blocked by policy)
kubectl -n payments run redteam --rm -i --restart=Never \
  --image=registry.access.redhat.com/ubi9/ubi-minimal -- \
  timeout 5 bash -c 'curl -s -o /dev/null http://169.254.169.254/latest/meta-data/' \
  && echo "FAIL: IMDS reachable" || echo "OK: IMDS blocked"

# Should SUCCEED (allowed path)
kubectl -n payments exec deploy/checkout -- \
  timeout 5 bash -c 'echo > /dev/tcp/postgres.data.svc.cluster.local/5432' \
  && echo "OK: DB reachable" || echo "FAIL: DB unreachable"

# Cilium's own connectivity test suite
cilium connectivity test --test-namespace cilium-test

Alert on drops. A sustained increase in hubble_drop_total for a workload means either a deployment introduced a new dependency or something is probing:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: hubble-policy-drops
  namespace: kube-system
spec:
  groups:
    - name: network-policy
      rules:
        - alert: PolicyDropSpike
          expr: |
            sum by (source_workload, destination_workload) (
              rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])
            ) > 1
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Policy drops between {{ $labels.source_workload }} and {{ $labels.destination_workload }}"

Security best practices

Always set policyTypes explicitly. A NetworkPolicy with only an ingress: key does not restrict egress at all. Half-isolated workloads are the most common finding in cluster audits.
Never allow 0.0.0.0/0 egress. If you must allow broad internet access, use toCIDRSet with except: for RFC1918 ranges, link-local (169.254.0.0/16), and your cloud metadata endpoint. Metadata endpoint access from a pod is an instance-credential theft primitive.
Protect the DNS path deliberately. Allowing port 53 to 0.0.0.0/0 recreates a covert channel. Restrict DNS to the cluster resolver and enable the L7 DNS proxy so you get query visibility.
Use CiliumClusterwideNetworkPolicy for the baseline, namespaced policies for exceptions. A namespaced default-deny can be deleted by anyone with namespace edit rights; a cluster-wide one cannot.
Select on stable, semantic labels (app.kubernetes.io/name, app.kubernetes.io/part-of), never on pod-template-hash or Helm release labels that change on upgrade.
Keep policies next to the workload in the same Helm chart or Kustomize overlay, so a new dependency and its allow rule land in the same pull request.
Do not treat NetworkPolicy as authentication. Labels are mutable by anyone who can patch pods; identity is derived from labels. Combine with mTLS and proper authn/authz for anything that matters.
Use enableDefaultDeny: false sparingly. Cilium 1.14+ lets a policy add allowances without triggering default-deny — useful for shared add-on policies, dangerous if you forget which policy actually establishes the deny.
Audit egress rules on a schedule. toFQDNs entries accumulate. An allow rule to a SaaS vendor you stopped using three quarters ago is dead weight and attack surface.

Conclusion

Zero-trust networking in an existing cluster is not a big-bang migration; it is a measurement problem followed by a configuration problem. The sequence that works on live production:

Verify your CNI enforces policy at all — flannel and some managed CNIs silently ignore NetworkPolicy objects.
Observe with Hubble for a full business cycle, exported to durable storage, not a ring buffer. Pay special attention to DNS and to CronJobs that only run monthly.
Generate a draft from real flows, then rewrite it by hand with semantic label selectors. Generated policy is evidence, not an artifact.
Enforce in audit mode with PolicyAuditMode=Enabled — this is the step that removes the downtime risk entirely. Drain the AUDIT verdict stream to zero before flipping.
Validate continuously with negative connectivity tests in CI and a Prometheus alert on hubble_drop_total{reason="POLICY_DENIED"}.

The payoff is not only prevention. Once default-deny is in place, every policy drop becomes a high-signal event: legitimate traffic no longer gets denied, so a drop means either an undeclared dependency or an intrusion. You have converted a flat, silent network into one that tells you when something is wrong.

Hardening the Control Plane: Auditing a Cluster with kube-bench, kubescape and CIS Rules

Florient DOGBE — Mon, 26 Oct 2026 13:35:48 +0000

Most clusters are installed once, pass a smoke test, and are never audited again. Meanwhile the control plane keeps running with whatever flags the installer picked two years ago: no audit log, an API server that still accepts anonymous requests on more endpoints than it should, an etcd data directory readable by anything on the node. None of this shows up in a dashboard. It shows up in a post-incident review.

This article walks through building an automated CIS compliance audit for a Kubernetes or OpenShift cluster using kube-bench and kubescape, interpreting the results without drowning in false positives, fixing the three most common gaps (anonymous-auth, audit logging, etcd), and wiring the whole thing into CI so that drift is caught in hours instead of years.

Context: what the CIS Benchmark actually measures

The CIS Kubernetes Benchmark is a prescriptive list of configuration checks, grouped by component:

Section 1 — control plane node configuration: kube-apiserver, kube-controller-manager, kube-scheduler flags and file permissions.
Section 2 — etcd configuration.
Section 3 — control plane authentication/authorization and logging.
Section 4 — worker node configuration, mostly kubelet.
Section 5 — policies: RBAC, service accounts, Pod Security, network policies.

Two things matter before you run anything.

First, the benchmark must match your distribution. The upstream cis-1.x profile assumes kubeadm-style static pod manifests in /etc/kubernetes/manifests. On OpenShift, the control plane is managed by operators, files live elsewhere, and roughly a third of the upstream checks are structurally inapplicable. Red Hat maintains a dedicated benchmark (rh-1.0 / rh-1.6 in kube-bench, ocp4-cis in the Compliance Operator). Running the wrong profile produces a report nobody trusts, and an untrusted report is worse than no report.

Second, a managed control plane is not yours. On EKS, GKE, AKS or ROSA, sections 1 and 2 are the provider's responsibility. Use the eks-1.x, gke-1.x or aks-1.x benchmarks, which skip them and focus on nodes and policies.

Running kube-bench

kube-bench is a single Go binary that reads the running process arguments and the config files on disk, then compares them to the benchmark. It needs to run on the node, with host PID and host paths mounted.

As a one-shot Job on a control plane node

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-master
  namespace: kube-bench
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: kube-bench
    spec:
      hostPID: true
      restartPolicy: Never
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: kube-bench
          image: docker.io/aquasec/kube-bench:v0.10.4
          command: ["kube-bench"]
          args:
            - "run"
            - "--targets=master,etcd,controlplane,policies"
            - "--json"
          securityContext:
            runAsUser: 0
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - { name: var-lib-etcd,  mountPath: /var/lib/etcd,  readOnly: true }
            - { name: var-lib-kubelet, mountPath: /var/lib/kubelet, readOnly: true }
            - { name: etc-kubernetes, mountPath: /etc/kubernetes, readOnly: true }
            - { name: etc-systemd,   mountPath: /etc/systemd,   readOnly: true }
            - { name: usr-bin,       mountPath: /usr/local/mount-from-host/bin, readOnly: true }
      volumes:
        - { name: var-lib-etcd,    hostPath: { path: /var/lib/etcd } }
        - { name: var-lib-kubelet, hostPath: { path: /var/lib/kubelet } }
        - { name: etc-kubernetes,  hostPath: { path: /etc/kubernetes } }
        - { name: etc-systemd,     hostPath: { path: /etc/systemd } }
        - { name: usr-bin,         hostPath: { path: /usr/bin } }

kubectl create namespace kube-bench
kubectl apply -f kube-bench-master.yaml
kubectl wait --for=condition=complete job/kube-bench-master -n kube-bench --timeout=120s
kubectl logs job/kube-bench-master -n kube-bench > kube-bench-master.json

For worker nodes, run the same Job as a DaemonSet with --targets=node and without the control plane nodeSelector.

Making the output usable

Raw kube-bench output is a wall of [PASS] / [FAIL] / [WARN]. Extract only what is actionable:

jq -r '
  .Controls[] |
  .text as $section |
  .tests[].results[] |
  select(.status == "FAIL") |
  [$section, .test_number, .status, .test_desc] | @tsv
' kube-bench-master.json | column -t -s$'\t'

And a one-line verdict for CI:

jq '[.Controls[].total_fail] | add' kube-bench-master.json

kube-bench also has --exit-code 1, which makes the container exit non-zero as soon as one check fails. Use it only once you have triaged your baseline, otherwise every pipeline turns red on day one.

On OpenShift

Force the Red Hat benchmark, because auto-detection on RHCOS is not always reliable:

kube-bench run --benchmark rh-1.0 --targets=master,node,policies --json

But on OpenShift the native path is better: the Compliance Operator runs OpenSCAP-based ocp4-cis profiles, produces ComplianceCheckResult objects in the API, and can auto-remediate through MachineConfig.

apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: cis-compliance
  namespace: openshift-compliance
profiles:
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default
  apiGroup: compliance.openshift.io/v1alpha1
  kind: ScanSetting

oc get compliancescan -n openshift-compliance
oc get compliancecheckresult -n openshift-compliance \
  -l compliance.openshift.io/check-status=FAIL \
  -o custom-columns='NAME:.metadata.name,SEVERITY:.severity'

Adding kubescape for the workload and RBAC side

kube-bench stops at the node. kubescape talks to the API server and evaluates controls across RBAC, Pod Security, network policies, image provenance and secrets — exactly the part of the CIS benchmark (section 5) that kube-bench marks as "manual".

# list available frameworks for your kubescape version
kubescape list frameworks

# CIS scan, JSON + human output, fail the shell on low compliance
kubescape scan framework cis-v1.23-t1.0.1 \
  --format json --output kubescape-cis.json \
  --compliance-threshold 80 \
  --severity-threshold high \
  --verbose

Two flags do the heavy lifting in CI:

--compliance-threshold 80 — exit non-zero if the overall compliance score drops below 80%.
--severity-threshold high — exit non-zero if any High/Critical control fails, regardless of score.

Legitimate deviations go into an exceptions file instead of being silently ignored:

[
  {
    "name": "allow-host-path-for-csi-driver",
    "policyType": "postureExceptionPolicy",
    "actions": ["alertOnly"],
    "resources": [
      { "designatorType": "Attributes",
        "attributes": { "namespace": "openshift-storage" } }
    ],
    "posturePolicies": [ { "controlID": "C-0048" } ]
  }
]

kubescape scan framework cis-v1.23-t1.0.1 \
  --exceptions ./exceptions.json --format sarif --output kubescape.sarif

The SARIF output drops straight into GitHub code scanning; --format junit plugs into GitLab or Jenkins test reporting.

Fixing the three failures you will always see

1. `--anonymous-auth=true` on the API server

CIS 1.2.1 wants --anonymous-auth=false on kube-apiserver. Before you change it, understand the trade-off: anonymous requests are mapped to the user system:anonymous in group system:unauthenticated, and many load balancers and monitoring probes hit /livez, /readyz or /healthz without credentials. Setting the flag to false on a kubeadm cluster will break those probes unless you reconfigure them.

The real risk is not anonymous authentication itself — it is anonymous authentication combined with a permissive RBAC binding. Audit that first:

kubectl get clusterrolebindings,rolebindings -A -o json | jq -r '
  .items[] |
  select(.subjects[]? | .name == "system:unauthenticated" or .name == "system:anonymous") |
  "\(.kind)/\(.metadata.name) -> \(.roleRef.kind)/\(.roleRef.name)"'

Anything other than system:public-info-viewer deserves an explanation. On kubeadm, apply the flag in the static pod manifest (the kubelet restarts the pod automatically):

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --anonymous-auth=false
        - --authorization-mode=Node,RBAC
        - --profiling=false

On OpenShift you do not edit these manifests. The API server is rendered by the kube-apiserver-operator, and anonymous access is already constrained by the default RBAC. Treat kube-bench's finding as informational and verify the bindings instead.

The kubelet-side check (CIS 4.2.1) is the one you should always enforce:

# /var/lib/kubelet/config.yaml (excerpt)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
authorization:
  mode: Webhook
readOnlyPort: 0

2. No audit log

CIS 1.2.19–1.2.23 require an audit policy and retention. Without it you have no record of who read which Secret. A balanced starting policy:

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  # Never log request/response bodies of secret-bearing resources
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]

  # Drop high-volume, low-value read traffic
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get", "list", "watch"]
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version", "/metrics"]

  # Full payload on RBAC and admission changes
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "rbac.authorization.k8s.io"
      - group: "admissionregistration.k8s.io"
      - group: "policy"

  # Everything else: metadata
  - level: Metadata

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
        - --audit-log-path=/var/log/kubernetes/audit/audit.log
        - --audit-log-maxage=30
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
      volumeMounts:
        - { name: audit-policy, mountPath: /etc/kubernetes/audit-policy.yaml, readOnly: true }
        - { name: audit-log, mountPath: /var/log/kubernetes/audit }
  volumes:
    - name: audit-policy
      hostPath: { path: /etc/kubernetes/audit-policy.yaml, type: File }
    - name: audit-log
      hostPath: { path: /var/log/kubernetes/audit, type: DirectoryOrCreate }

On OpenShift, audit logging is on by default and you select a profile declaratively:

oc patch apiserver cluster --type=merge \
  -p '{"spec":{"audit":{"profile":"WriteRequestBodies"}}}'

# logs are already collected per control plane node
oc adm node-logs --role=master --path=kube-apiserver/ | head

3. etcd exposure

CIS section 2 checks that etcd requires client certificates, uses peer TLS, and that /var/lib/etcd is 0700 and owned by etcd:etcd. Verify on a control plane node:

stat -c "%a %U:%G %n" /var/lib/etcd
ps -ef | grep -o -- '--client-cert-auth=[a-z]*\|--peer-client-cert-auth=[a-z]*\|--auto-tls=[a-z]*'

The check kube-bench cannot do for you is encryption at rest (CIS 1.2.31/1.2.34). Secrets are stored in etcd base64-encoded, not encrypted, unless you configure a provider:

# /etc/kubernetes/enc/encryption-config.yaml  -- never commit this file
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets", "configmaps"]
    providers:
      - aescbc:
          keys:
            - name: key-2024-06
              secret: <BASE64_32_BYTE_KEY>   # head -c 32 /dev/urandom | base64
      - identity: {}

Add --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml to the API server, then rewrite existing objects:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

On OpenShift this is a single field, and the operator handles key rotation:

oc patch apiserver cluster --type=merge -p '{"spec":{"encryption":{"type":"aescbc"}}}'
oc get apiserver cluster -o jsonpath='{.status.conditions[?(@.type=="Encrypted")].reason}{"\n"}'

Wiring it into CI

Run kubescape on every merge request (it only needs API access) and kube-bench nightly as a CronJob (it needs node access).

# .gitlab-ci.yml
cluster-compliance:
  stage: audit
  image: quay.io/kubescape/kubescape-cli:v3.0.8
  script:
    - kubescape scan framework cis-v1.23-t1.0.1
        --exceptions ./compliance/exceptions.json
        --compliance-threshold 80
        --severity-threshold high
        --format junit --output kubescape-junit.xml
        --format json  --output kubescape.json
  artifacts:
    when: always
    reports:
      junit: kubescape-junit.xml
    paths: [kubescape.json]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_PIPELINE_SOURCE == "schedule"

The nightly kube-bench run, with results pushed to a bucket and a summary into the daily report:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench-nightly
  namespace: kube-bench
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          restartPolicy: Never
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
            - { key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }
          containers:
            - name: kube-bench
              image: docker.io/aquasec/kube-bench:v0.10.4
              args: ["run", "--targets=master,etcd,controlplane,policies",
                     "--json", "--outputfile=/reports/kube-bench-$(NODE_NAME).json"]
              env:
                - name: NODE_NAME
                  valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
              volumeMounts:
                - { name: reports, mountPath: /reports }
                - { name: etc-kubernetes, mountPath: /etc/kubernetes, readOnly: true }
                - { name: var-lib-etcd, mountPath: /var/lib/etcd, readOnly: true }
          volumes:
            - { name: reports, persistentVolumeClaim: { claimName: compliance-reports } }
            - { name: etc-kubernetes, hostPath: { path: /etc/kubernetes } }
            - { name: var-lib-etcd, hostPath: { path: /var/lib/etcd } }

A readable Markdown report for the weekly review:

{
  echo "# CIS audit — $(date -I) — cluster $(kubectl config current-context)"
  echo
  echo "| Check | Severity | Description | Remediation |"
  echo "|---|---|---|---|"
  jq -r '.Controls[].tests[].results[]
    | select(.status=="FAIL")
    | "| \(.test_number) | \(.scored // false | if . then "scored" else "not scored" end) | \(.test_desc) | \(.remediation | gsub("\n"; " ")) |"' \
    kube-bench-master.json
} > report-cis.md

Security best practices

Baseline first, gate second. Capture the current failure list, file it as exceptions with an owner and an expiry date, then make CI fail on anything new. Gating on absolute compliance from day one just teaches people to add || true.
Pin the benchmark version in your pipeline (cis-v1.23-t1.0.1, rh-1.0, v0.10.4). A benchmark bump is a change you review, not a surprise red build.
Never auto-remediate the control plane. kube-bench remediations edit static pod manifests; a bad flag means an API server that does not come back. Apply through your installer (kubeadm config, MachineConfig, Cluster API) and roll one node at a time.
Ship audit logs off the node. A log sitting on a compromised control plane node is evidence an attacker can delete. Forward to a write-once destination and alert on secrets access by non-system identities.
Keep the identity provider last in EncryptionConfiguration during migration, and rotate keys by prepending a new one and re-running the kubectl replace pass.
Audit the auditor. kube-bench runs as root with hostPID and host mounts. Scope it to a dedicated namespace, a dedicated ServiceAccount, and a restrictive Pod Security / SCC exception — not a blanket privileged grant.
Correlate with runtime. A passing CIS score with a cluster-admin ServiceAccount token mounted in a public-facing pod is still a breach waiting to happen. Section 5 controls in kubescape are the ones that catch this.

Key takeaways

Control plane hardening is not a one-off project; it is a measurement you take every night. kube-bench tells you what the node configuration looks like, kubescape tells you what the API objects allow, and the two together cover the CIS benchmark end to end. On OpenShift, let the Compliance Operator own sections 1–4 and use kubescape for the workload layer.

Start with the three findings that appear on nearly every cluster: review who is bound to system:unauthenticated, turn on a real audit policy with retention, and encrypt Secrets at rest while locking down etcd's files and client certificates. Then convert the remaining failures into a tracked exception list, set --compliance-threshold and --severity-threshold in CI, and archive the JSON so you can show a trend line rather than a snapshot. A compliance report that nobody reads is an artifact; a compliance report that fails a pipeline is a control.

Priorities, Preemption and QoS: Making Sure Your Critical Pods Survive Saturation

Florient DOGBE — Mon, 26 Oct 2026 13:25:15 +0000

When a node runs out of memory at 3 a.m., three different subsystems can decide which container dies: the scheduler (preemption at admission time), the kubelet eviction manager (node-pressure eviction at runtime), and the Linux OOM killer (cgroup-level, no Kubernetes semantics at all). They do not share the same ranking rules, they do not read the same fields, and only one of them cares about your PodDisruptionBudget.

This article unpacks that interaction: how PriorityClass, QoS class and evictionHard thresholds combine, why your logging DaemonSet gets killed before a batch job, and how to design a priority taxonomy that does not produce cascading preemption.

Context: three killers, one node

A saturated node is not a single failure mode. You need to distinguish:

Mechanism	Trigger	Ranking input	Honors PDB?
Scheduler preemption	A pending pod cannot be scheduled	`PriorityClass.value`, `preemptionPolicy`	Best-effort only
Kubelet node-pressure eviction	`memory.available`, `nodefs.available`, `pid.available`… crossing a threshold	usage vs. requests, then priority, then overage	No
Kernel OOM killer	cgroup limit or global memory exhaustion	`oom_score_adj` derived from QoS	No

The classic incident: an operator sets priorityClassName: high-priority on a batch workload "to make sure it runs", the scheduler preempts a handful of pods on node A, those pods reschedule onto node B, push node B above memory.available<100Mi, the kubelet evicts there too — and because the evicted pods have no memory requests at all (BestEffort), the cascade continues across the cluster. Priority was set, QoS was not, and nobody looked at evictionHard.

QoS classes: the only thing the kubelet really ranks on

QoS is derived, never declared. The rules are strict:

Guaranteed: every container in the pod sets requests == limits for both CPU and memory.
Burstable: at least one container sets a CPU or memory request (or limit), but the Guaranteed condition is not met.
BestEffort: no requests and no limits anywhere in the pod.

Check it, do not assume it:

oc get pods -n payments -o custom-columns=\
'NAME:.metadata.name,QOS:.status.qosClass,PRIO:.spec.priority,PC:.spec.priorityClassName'

QoS drives two things. First, oom_score_adj on every container process:

Guaranteed → -997
BestEffort → 1000
Burstable → min(max(2, 1000 - (1000 × memoryRequest) / nodeAllocatableMemory), 999)

So under a global kernel OOM event, a Guaranteed pod is almost the last victim, and a Burstable pod with a large request is far safer than one with a token 64Mi. Priority plays no role here — the kernel has never heard of PriorityClass.

# Verify on a node (OpenShift)
oc debug node/worker-02 -- chroot /host /bin/bash -c \
  'for p in $(pgrep -f "nginx"); do echo -n "$p "; cat /proc/$p/oom_score_adj; done'

Second, QoS shapes the kubelet's eviction ordering — indirectly.

Kubelet node-pressure eviction: the real ranking

The eviction manager does not sort by "BestEffort first, then Burstable, then Guaranteed", despite what many blog posts claim. For memory pressure it applies a three-level comparator:

Is the pod's memory usage above its memory request? Pods above request sort first.
Pod priority (lower priority sorts first).
Memory usage minus request (largest overage first).

The practical consequence is elegant: a Guaranteed pod can never exceed its request (request == limit), and a Burstable pod staying inside its request is in the same bucket. They are only considered after every pod that is borrowing memory it never asked for. BestEffort pods have a request of zero, so any usage puts them in bucket 1 — they always die first.

Priority only breaks ties inside a bucket. A system-cluster-critical pod that consumes 4 GiB with a 512 MiB request will be evicted before a priority-0 pod that respects its request. Setting a high PriorityClass on a pod with no memory request buys you almost nothing.

Disk pressure (nodefs.available, imagefs.available, inodes) follows the same pattern using ephemeral-storage requests and local volume usage. The kubelet first garbage-collects dead containers and unused images; only if that is insufficient does it start evicting.

Configuring thresholds on OpenShift

Thresholds are node-level kubelet configuration, delivered through a KubeletConfig bound to a MachineConfigPool:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-eviction-tuning
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    evictionHard:
      memory.available: "500Mi"
      nodefs.available: "10%"
      nodefs.inodesFree: "5%"
      imagefs.available: "15%"
    evictionSoft:
      memory.available: "1Gi"
      nodefs.available: "15%"
    evictionSoftGracePeriod:
      memory.available: "1m30s"
      nodefs.available: "2m"
    evictionMaxPodGracePeriod: 60
    evictionPressureTransitionPeriod: 5m
    evictionMinimumReclaim:
      memory.available: "300Mi"
      nodefs.available: "2Gi"
    systemReserved:
      cpu: "500m"
      memory: "1Gi"
      ephemeral-storage: "2Gi"

oc apply -f kubeletconfig-eviction.yaml
oc get mcp worker -w   # rollout is node-by-node, expect reboots-free kubelet restarts

Three details that bite in production:

memory.available is computed from cgroup working set (memory.usage_in_bytes minus inactive file cache), not from free -m. A node showing "plenty of free memory" in top can still be under pressure.
The hard eviction threshold is subtracted from Allocatable. Raising memory.available to 500Mi removes 400 MiB of schedulable capacity per node compared to the default 100Mi. That is the point: you buy headroom so the kubelet reacts before the kernel OOM killer does.
evictionSoft gives containers a real SIGTERM grace period (capped by evictionMaxPodGracePeriod); evictionHard kills immediately. Always define both, with the soft threshold more conservative.

# What did the kubelet actually load?
oc debug node/worker-02 -- chroot /host \
  cat /etc/kubernetes/kubelet.conf | grep -A12 eviction

# Who got evicted and why
oc get events -A --field-selector reason=Evicted \
  -o custom-columns='TIME:.lastTimestamp,NS:.involvedObject.namespace,POD:.involvedObject.name,MSG:.message'

PriorityClass and preemption: the scheduler's side

PriorityClass is a cluster-scoped object. Its value lands in pod.spec.priority at admission time (mutation is immutable afterwards).

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: platform-critical
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Node agents, ingress, logging, service mesh data plane."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-tier1
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Customer-facing APIs with an SLO."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-non-preempting
value: 10000
globalDefault: false
preemptionPolicy: Never
description: "Scheduled ahead of default workloads in the queue, never evicts anyone."

preemptionPolicy: Never is the single most underrated field here. The pod keeps its place in the scheduling queue (it is considered before lower-priority pending pods) but it will never trigger victim selection. This is exactly what you want for batch, CI runners, Spark executors and anything that can simply wait.

Reserved bands: system-cluster-critical = 2000000000 and system-node-critical = 2000001000 ship with every cluster. Do not create user classes above 1000000000 — leave the top band to the control plane. On OpenShift you will also find openshift-user-critical and openshift-monitoring as managed classes.

oc get priorityclass -o custom-columns='NAME:.metadata.name,VALUE:.value,DEFAULT:.globalDefault,POLICY:.preemptionPolicy'

Avoiding cascading preemption

Preemption cascades happen when high-priority pods are produced faster than capacity, or when victim selection repeatedly fails to actually unblock the pending pod (anti-affinity, topology spread, volume node affinity). Guardrails that work:

1. Few, wide priority bands. Four or five classes with gaps of 10× between values. Dozens of classes with values 100, 101, 102 produce long preemption chains with no operational meaning.

2. Quota the expensive bands. A ResourceQuota with a scopeSelector on PriorityClass caps how much a namespace can consume at a given priority:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tier1-budget
  namespace: payments
spec:
  hard:
    cpu: "64"
    memory: 128Gi
    pods: "120"
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["business-tier1", "platform-critical"]

3. Make DaemonSets explicitly critical. The kubelet grants no special treatment to DaemonSet pods during eviction; only priority and requests count. Node agents must carry system-node-critical (or your own high band) and real requests.

4. Watch the right metrics.

# Scheduler-side
scheduler_preemption_victims
scheduler_preemption_attempts_total

# Kubelet-side
kubelet_evictions{eviction_signal="memory.available"}
kubelet_eviction_stats_age_seconds

A PromQL alert worth having:

sum(rate(kubelet_evictions[15m])) by (node, eviction_signal) > 0

PodDisruptionBudget: what it does and does not protect

This is the most common misconception in the whole topic.

API-initiated eviction (oc adm drain, descheduler, cluster-autoscaler scale-down) goes through the pods/eviction subresource and respects PDBs. A violated PDB returns 429 TooManyRequests.
Scheduler preemption tries to pick victims that do not violate PDBs, but if no such set exists it preempts anyway. It is a preference, not a guarantee.
Kubelet node-pressure eviction bypasses PDBs entirely. The kubelet deletes the pod directly. No API eviction call, no budget check, no minAvailable.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: payments
spec:
  minAvailable: 80%
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: checkout-api

oc get pdb -n payments
oc adm drain worker-02 --ignore-daemonsets --delete-emptydir-data --timeout=300s

unhealthyPodEvictionPolicy: AlwaysAllow (policy/v1, GA since 1.27) prevents the classic deadlock where a PDB blocks a drain because the already-broken pods are counted as "must not be disrupted".

The takeaway: PDBs protect against planned disruption only. Protection against node saturation comes from requests, limits and eviction thresholds — never from a budget.

A reference manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: payments
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout-api }
  template:
    metadata:
      labels: { app: checkout-api }
    spec:
      priorityClassName: business-tier1
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: checkout-api }
      containers:
        - name: api
          image: registry.example.com/payments/checkout:1.14.2
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
              ephemeral-storage: "256Mi"
            limits:
              cpu: "500m"
              memory: "1Gi"
              ephemeral-storage: "1Gi"
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            capabilities: { drop: ["ALL"] }
            seccompProfile: { type: RuntimeDefault }

requests == limits on CPU and memory → Guaranteed → oom_score_adj: -997, never in the "above request" eviction bucket, and an explicit ephemeral-storage request so disk-pressure ranking treats it fairly.

Security best practices

Priority is a capacity-theft primitive. A tenant able to set priorityClassName: system-cluster-critical can evict your control-plane add-ons from every node.

Lock down PriorityClass creation. scheduling.k8s.io/priorityclasses must be restricted to cluster admins. Audit with oc adm policy who-can create priorityclasses.
Restrict consumption, not just creation. RBAC cannot filter which priorityClassName a pod may reference. Use a ValidatingAdmissionPolicy:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: restrict-priority-classes
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >
        !has(object.spec.priorityClassName) ||
        object.spec.priorityClassName in ['batch-non-preempting','business-tier1']
      message: "This namespace may only use batch-non-preempting or business-tier1."

Bind it with a ValidatingAdmissionPolicyBinding scoped by namespace selector so platform namespaces keep access to the high bands.

Never set globalDefault: true on a high-value class. It silently promotes every pod in the cluster, including untrusted ones, and makes preemption meaningless.
Enforce requests with LimitRange. BestEffort pods are a denial-of-service vector: they consume memory they never declared and drag the whole node to the eviction threshold. A namespace LimitRange with defaultRequest removes the BestEffort class entirely.
Reserve capacity for the system. systemReserved plus a non-default evictionHard keeps the kubelet, CRI-O and sshd alive so you can still reach the node during an incident.
Audit priority changes. Add scheduling.k8s.io resources to the audit policy at RequestResponse level; a new PriorityClass with value 2000000001 is a credible escalation signal.

Conclusion

QoS is derived from requests/limits — it is the primary defense. requests == limits yields Guaranteed, oom_score_adj: -997, and immunity from the "above request" eviction bucket. No PriorityClass compensates for missing requests.
The kubelet ranks by overage first, priority second. A high-priority pod exceeding its request dies before a low-priority pod respecting its own.
The scheduler ranks by priority only. preemptionPolicy: Never lets batch workloads queue ahead without triggering evictions — use it aggressively.
PDBs do not apply to node-pressure eviction. They cover drains and API-initiated eviction. Stop relying on them for saturation scenarios.
Tune evictionHard / evictionSoft deliberately. Defaults (memory.available<100Mi) leave almost no reaction time before the kernel OOM killer takes over with no Kubernetes semantics.
Treat priority as a privilege. Restrict it through admission policy and ResourceQuota scope selectors, and never make a high class the global default.

Design the taxonomy once — four bands, documented, quota-backed — and enforce requests everywhere. Saturation then becomes a predictable degradation instead of a cluster-wide cascade.

Debugging a Stubborn CrashLoopBackOff: Ephemeral Containers, kubectl debug, and Init Pitfalls

Florient DOGBE — Thu, 22 Oct 2026 08:43:27 +0000

Your deployment rolled out. Thirty seconds later, kubectl get pods shows CrashLoopBackOff and a restart counter climbing. You reach for the usual reflex — kubectl exec -it pod/... -- sh — and get:

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Welcome to the modern container world: your image is distroless (or scratch, or a FROM scratch Go binary), there is no shell, no ps, no curl, no cat. And even if there were, the container is dead most of the time anyway — you can't exec into something that isn't running.

This article is a step-by-step method to diagnose that pod without ever rebuilding the image with "debug tools". We'll read exit codes properly, decode init container failures, attach ephemeral containers with kubectl debug, and clone a pod into a debuggable copy.

Context: why CrashLoopBackOff got harder

CrashLoopBackOff is not an error in itself. It's the kubelet telling you: "the container's main process exited, I restarted it, it exited again, so now I'm waiting before trying once more." The backoff starts at 10 seconds and doubles up to a 5-minute cap. That cap is why a crashing pod feels frozen after a few minutes — you get one short window of life every 300 seconds.

Two trends made this harder to debug:

Minimal images. Distroless, scratch, and Chainguard-style images ship a binary and little else. No package manager, no shell, often no writable filesystem. Great for the attack surface, terrible for kubectl exec.
Hardened Pod Security. On OpenShift, the restricted-v2 SCC forces a random UID, drops all capabilities, and blocks privilege escalation. Your old trick of "just run a privileged busybox" needs explicit authorization.

The good news: Kubernetes shipped ephemeral containers (stable since 1.25) precisely for this. You inject a throwaway container into the running pod — same network namespace, same volumes, optionally the same process namespace — without restarting anything.

Step 1: read the status before touching anything

Ninety percent of CrashLoopBackOff cases are solved here. Don't skip it.

kubectl -n shop get pod checkout-7d9f6c4b8-xk2lq -o wide
kubectl -n shop describe pod checkout-7d9f6c4b8-xk2lq

In describe, go straight to the Last State block. But the fastest, most reliable read is JSONPath, because describe truncates and reorders:

kubectl -n shop get pod checkout-7d9f6c4b8-xk2lq \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.message}{"\n"}{end}'

And do the same for init containers — this is the single most forgotten command:

kubectl -n shop get pod checkout-7d9f6c4b8-xk2lq \
  -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}'

Decoding exitCode correctly

exitCode is not noise. It's a precise signal:

exitCode	Meaning	Typical cause
0	Clean exit	The process finished its job; a `Deployment` expects a long-running process. Wrong workload type.
1	Generic application error	Unhandled exception, missing env var, bad config. Read the logs.
2	Shell misuse / CLI arg error	Bad flag in `args`.
126	Cannot execute	Binary present but not executable, or wrong architecture (arm64 image on amd64 node).
127	Command not found	`command:` points to a path that doesn't exist in the image — classic on distroless.
128+N	Killed by signal N	See below.
137	128+9, SIGKILL	OOMKilled, or liveness probe + failed graceful shutdown.
139	128+11, SIGSEGV	Native crash, corrupt binary, incompatible CPU instruction set.
143	128+15, SIGTERM	Something asked it to stop: liveness probe, eviction, rollout.

Two important distinctions a beginner must internalize:

137 + reason: OOMKilled means the memory limit was hit. 137 without OOMKilled usually means a liveness probe killed the container after it ignored SIGTERM for terminationGracePeriodSeconds.
exitCode 0 in CrashLoopBackOff is not a bug in Kubernetes. With restartPolicy: Always (the only option for Deployments), even a successful exit triggers a restart. If your container is a one-shot task, it belongs in a Job.

Logs of the previous instance

The container that failed is gone. Its logs are not in kubectl logs — they're one container generation back:

kubectl -n shop logs checkout-7d9f6c4b8-xk2lq -c checkout --previous
kubectl -n shop logs checkout-7d9f6c4b8-xk2lq --all-containers --previous --timestamps

If --previous returns nothing, the process died before writing anything — which itself points at exit 126/127 or a missing config file.

Make the crash message visible in the status

A trick almost nobody uses: terminationMessagePolicy. By default the kubelet only reads /dev/termination-log. Set it to FallbackToLogsOnError and the last lines of stderr are copied into lastState.terminated.message, visible in kubectl describe forever:

apiVersion: v1
kind: Pod
metadata:
  name: checkout
  namespace: shop
spec:
  containers:
    - name: checkout
      image: registry.example.com/shop/checkout:1.8.3
      terminationMessagePolicy: FallbackToLogsOnError
      terminationMessagePath: /dev/termination-log

Put this on every container in your base templates. It costs nothing.

Step 2: the init container pitfalls

If kubectl get pod shows Init:CrashLoopBackOff or Init:Error, your app container never even started. Init containers run sequentially, and each one must exit 0 before the next starts. A single failing init container restarts the whole pod.

Here's a realistic failing example — a distroless app behind a busybox init that waits for a database:

apiVersion: v1
kind: Pod
metadata:
  name: checkout
  namespace: shop
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  initContainers:
    - name: wait-for-db
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          until nc -z postgres.shop.svc.cluster.local 5432; do
            echo "waiting for postgres..." >&2
            sleep 2
          done
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 65532
        capabilities:
          drop: ["ALL"]
  containers:
    - name: checkout
      image: gcr.io/distroless/static-debian12:nonroot
      args: ["--config=/etc/checkout/config.yaml"]
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
        - name: config
          mountPath: /etc/checkout
          readOnly: true
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 65532
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
  volumes:
    - name: config
      configMap:
        name: checkout-config

Classic traps:

Infinite wait, not a crash. Init:0/1 that never progresses isn't CrashLoopBackOff — it's a DNS or NetworkPolicy issue. Check with kubectl logs pod/checkout -c wait-for-db.
The init container has a shell, the app doesn't. You can debug the init phase easily, which lulls you into thinking the problem is in the app.
Sidecars as init containers (restartPolicy: Always on an init container, Kubernetes 1.29+) behave differently: they're not expected to exit, and the next init container starts as soon as they're Started. Mixing a classic init and a native sidecar in the wrong order is a common source of hangs.
Init container logs disappear on restart. Use --previous on init containers too: kubectl logs pod/checkout -c wait-for-db --previous.

Step 3: attach an ephemeral container with kubectl debug

The pod still has a running sandbox (pause container, network namespace, volumes) even while the app container is dead. That's where you inject your toolbox:

kubectl -n shop debug -it checkout-7d9f6c4b8-xk2lq \
  --image=nicolaka/netshoot:v0.13 \
  --target=checkout \
  --profile=general \
  -- bash

What each flag does:

--image — your debug toolbox. busybox:1.36, nicolaka/netshoot, or an internal image with curl, dig, ss, strace.
--target=checkout — share the process namespace of that specific container. Without it you can't see or inspect the app process.
--profile — general, baseline, restricted, netadmin, sysadmin. Always pass it explicitly; the implicit default has changed across versions. Use restricted on hardened namespaces, netadmin only when you need NET_ADMIN / NET_RAW for tcpdump.

Once inside, you are in the same network namespace as the pod:

# Is the config file actually mounted and readable?
ls -l /proc/1/root/etc/checkout/ 2>/dev/null

# Does the service resolve from inside the pod?
nslookup postgres.shop.svc.cluster.local
nc -vz postgres.shop.svc.cluster.local 5432

# Who is listening in this pod's netns?
ss -lntp

# What environment did the app actually get?
cat /proc/1/environ | tr '\0' '\n'

The /proc/<pid>/root trick is the key to distroless debugging: with --target and a shared PID namespace, you can read the other container's filesystem through /proc, even though your debug container has its own image. Find the PID with ps aux first — it won't always be 1.

Limits you must know

Ephemeral containers cannot be removed. Once added, they stay in the pod spec until the pod is deleted. Debug, then delete the pod.
They cannot define resources, ports, probes, or lifecycle. They have no resource requests — mind your node pressure.
They need the pods/ephemeralcontainers subresource RBAC verb (see security section).
If the pod has no sandbox at all — Pending, ImagePullBackOff, CreateContainerConfigError — ephemeral containers won't help. That's a scheduling/image/config problem; kubectl describe events are your answer.

Step 4: clone the pod when you need to change the command

Sometimes the container dies in 200 ms and you need it alive to inspect it. You can't change the command of a running pod — but you can copy it:

kubectl -n shop debug checkout-7d9f6c4b8-xk2lq \
  --copy-to=checkout-debug \
  --container=checkout \
  --keep-init-containers=false \
  -it -- sleep infinity

This creates a new pod checkout-debug with the same volumes, env, and service account, but the checkout container's command replaced by sleep infinity. Note: if the image is distroless, sleep doesn't exist — combine with --set-image:

kubectl -n shop debug checkout-7d9f6c4b8-xk2lq \
  --copy-to=checkout-debug \
  --set-image=checkout=busybox:1.36 \
  --keep-init-containers=false \
  -it -- sh

Now you have a shell with the exact same mounted ConfigMaps, Secrets, and ServiceAccount token. Validate the hypothesis:

cat /etc/checkout/config.yaml
ls -ln /etc/checkout         # UID/GID mismatch with runAsUser?
env | sort
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace

Flags that matter on --copy-to:

--keep-labels=false (the default) strips labels so the copy does not receive Service traffic. Never set it to true in production.
--keep-init-containers=true is the default — set it to false when the init container is exactly what's failing, otherwise your copy crashes the same way.
--keep-liveness, --keep-readiness, --keep-startup default to false: probes are removed, so nothing kills your debug pod.
--share-processes makes all containers in the copy share the PID namespace.

Delete the copy when you're done: kubectl -n shop delete pod checkout-debug.

OpenShift specifics

oc debug predates kubectl debug and does the copy-and-neutralize dance for you:

# Debug a copy of the pod, probes removed, command replaced by a shell
oc -n shop debug pod/checkout-7d9f6c4b8-xk2lq --image=registry.access.redhat.com/ubi9/ubi-minimal

# Debug from a workload controller directly
oc -n shop debug deployment/checkout --image=registry.access.redhat.com/ubi9/ubi-minimal -- /bin/bash

# Keep the same UID/SCC context instead of asking for root
oc -n shop debug deployment/checkout --as-user=1000650000

If the problem looks node-level (disk pressure, CNI, image GC):

oc debug node/worker-02
# then, inside:
chroot /host
crictl ps -a | head
journalctl -u kubelet --since "10 min ago" | tail -50

The kubectl equivalent, with the node filesystem under /host:

kubectl debug node/worker-02 -it --image=busybox:1.36 --profile=sysadmin

Security best practices

Debugging tooling is a privilege-escalation vector. Treat it like one.

1. Scope RBAC to the pods/ephemeralcontainers subresource. Creating an ephemeral container means running arbitrary code inside an existing pod, with that pod's ServiceAccount token. Don't bundle it into a generic edit role.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-debugger
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "pods/exec"]
    verbs: ["get", "create"]
  - apiGroups: [""]
    resources: ["pods/ephemeralcontainers"]
    verbs: ["get", "patch", "update"]

Bind it to an on-call group, not to every developer, and prefer a time-bound binding you remove after the incident.

2. Never default to --profile=sysadmin. It grants privileged: true and host namespaces. Start with restricted, escalate to netadmin only for packet capture, and to sysadmin only for node-level work. On OpenShift, needing sysadmin means you need an SCC exception — make it explicit and auditable.

3. Pin and curate your debug images. nicolaka/netshoot:v0.13, not :latest. Mirror it into your internal registry, scan it, and allow it in your admission policy. A debug image with network tools running in a pod with a mounted ServiceAccount token is a lateral-movement toolkit.

4. Remember that a copied pod inherits the Secrets. --copy-to mounts the same Secrets and SA token. Anyone with shell access in the copy can read them. Delete the copy immediately, and never leave it running overnight "just in case".

5. Keep labels off the copies. The default --keep-labels=false prevents the debug pod from being picked up by a Service and serving real traffic. Verify with kubectl get endpoints -n shop before and after.

6. Audit it. Ephemeral container creation appears in the API audit log as a patch on pods/ephemeralcontainers. Add an alert on it in production namespaces — it's a low-volume, high-signal event.

7. Don't "fix" the problem by shipping a shell. Adding busybox to your production image to make debugging easier undoes the entire point of distroless. The whole workflow above exists so you don't have to.

Conclusion: the checklist

When a pod loops and you have no shell, run these in order:

describe + JSONPath on lastState.terminated — get the exitCode and reason. 127 = wrong command, 137 + OOMKilled = memory limit, 143 = something killed it, 0 = wrong workload type.
kubectl logs --previous, on the app container and every init container. Enable terminationMessagePolicy: FallbackToLogsOnError everywhere so the message survives in the status.
Check initContainerStatuses before blaming the app. Init: prefixes in the pod status mean the app never ran.
kubectl debug --target=<container> --profile=general for live inspection: DNS, connectivity, mounted files via /proc/<pid>/root, real environment via /proc/<pid>/environ.
kubectl debug --copy-to --set-image --keep-init-containers=false when you need the container to stay alive with a different command.
kubectl debug node/<node> --profile=sysadmin or oc debug node/... only when the evidence points below the pod.

And the mental model to keep: an exit code tells you how it died, the logs tell you why, and the ephemeral container tells you what the environment actually looked like — which, in nine cases out of ten, is different from what your manifest led you to believe.

Azure Workload Identity: Remove Secrets From Your AKS Pods in 45 Minutes

Florient DOGBE — Wed, 21 Oct 2026 14:43:21 +0000

If you are still mounting a Kubernetes Secret containing an AZURE_CLIENT_SECRET into your pods, you are one kubectl get secret -o yaml away from an incident report. And if you are using AAD Pod Identity, you are running a component that reached end of support in September 2023.

Azure Workload Identity replaces both. It uses the OIDC issuer built into your AKS cluster to let a Kubernetes ServiceAccount exchange its projected token for an Azure AD (Microsoft Entra ID) access token. No secret, no rotation job, no NodeManagedIdentity DaemonSet intercepting IMDS traffic.

This walkthrough takes about 45 minutes end to end, assuming you already have an AKS cluster and the Azure CLI installed.

The problem: static credentials are a liability

A typical "pre-Workload-Identity" deployment looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: app-azure-credentials
  namespace: payments
type: Opaque
stringData:
  AZURE_TENANT_ID: 00000000-0000-0000-0000-000000000000
  AZURE_CLIENT_ID: 11111111-1111-1111-1111-111111111111
  AZURE_CLIENT_SECRET: "<never do this>"

Three structural problems:

Base64 is not encryption. Anyone with get secrets RBAC in that namespace — or read access to your etcd backups, or to the Git repo where the manifest lives — owns that identity.
Rotation is manual. Client secrets expire (6, 12, 24 months). Your rotation is a calendar reminder and a pipeline run, so it fails at 2 a.m. on a Sunday.
No blast radius control. The same secret works from a laptop, a CI runner, or a compromised pod in another namespace. Nothing ties it to a workload.

AAD Pod Identity tried to fix this by intercepting calls to the Instance Metadata Service (169.254.169.254) and injecting a managed identity token. It worked, but it was node-scoped, raced at pod startup (AzureIdentityBinding not reconciled yet → 404 from IMDS), and required privileged DaemonSets with iptables rules. It is deprecated and removed from new clusters.

Workload Identity takes a different path: federation. The cluster becomes an OIDC identity provider. Entra ID trusts tokens it issues, for one specific subject. No network interception, no DaemonSet, no secret.

How the token exchange actually works

Understanding the flow makes debugging trivial later:

The kubelet projects a short-lived (default 1 hour), audience-scoped JWT for the pod's ServiceAccount at /var/run/secrets/azure/tokens/azure-identity-token. The audience is api://AzureADTokenExchange.
The Azure SDK (or DefaultAzureCredential) in your app reads that file, plus the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_AUTHORITY_HOST environment variables injected by the mutating webhook.
The SDK calls Entra ID with a client_assertion = the projected token.
Entra ID validates the signature against the cluster's public JWKS (fetched from the OIDC issuer URL), checks iss, sub and aud against the federated identity credential you registered, and returns an Azure access token.
Your app calls Key Vault / Storage / Cosmos DB with that token.

The only thing you configure is step 4: which issuer, which subject, which audience. Everything else is plumbing.

Step 1 — Enable the OIDC issuer and the webhook

Set your variables first:

export RG=rg-platform-prod
export CLUSTER=aks-platform-prod
export LOCATION=westeurope
export SUBSCRIPTION=$(az account show --query id -o tsv)

Both features are flags on the cluster. On an existing cluster:

az aks update \
  --resource-group "$RG" \
  --name "$CLUSTER" \
  --enable-oidc-issuer \
  --enable-workload-identity

This takes 5 to 10 minutes and performs a control-plane reconcile. It is non-disruptive for running workloads, but the mutating webhook only applies to newly created pods — existing pods will not get the projected token until they are recreated.

For a new cluster:

az aks create \
  --resource-group "$RG" \
  --name "$CLUSTER" \
  --location "$LOCATION" \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --node-count 2 \
  --generate-ssh-keys

Retrieve the issuer URL — you will need it twice:

export AKS_OIDC_ISSUER=$(az aks show \
  --resource-group "$RG" \
  --name "$CLUSTER" \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

echo "$AKS_OIDC_ISSUER"
# https://westeurope.oic.prod-aks.azure.com/<tenant-guid>/<cluster-guid>/

Sanity check that it is publicly reachable (Entra ID must be able to fetch the JWKS):

curl -s "${AKS_OIDC_ISSUER}.well-known/openid-configuration" | jq .jwks_uri

Confirm the webhook is running:

kubectl get pods -n kube-system -l azure-workload-identity.io/system=true
# azure-wi-webhook-controller-manager-xxxxx   1/1   Running

Step 2 — Create the managed identity

Use a user-assigned managed identity (UAMI). You can also federate an App Registration, but a UAMI has no secret at all and lives in your resource group, which keeps RBAC and lifecycle in one place.

export UAMI=id-payments-api

az identity create \
  --name "$UAMI" \
  --resource-group "$RG" \
  --location "$LOCATION"

export USER_ASSIGNED_CLIENT_ID=$(az identity show \
  --name "$UAMI" --resource-group "$RG" \
  --query 'clientId' -o tsv)

export USER_ASSIGNED_PRINCIPAL_ID=$(az identity show \
  --name "$UAMI" --resource-group "$RG" \
  --query 'principalId' -o tsv)

Grant it something concrete — read-only access to one Key Vault:

export KV_NAME=kv-payments-prod
export KV_ID=$(az keyvault show --name "$KV_NAME" --query id -o tsv)

az role assignment create \
  --assignee-object-id "$USER_ASSIGNED_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

Note: Key Vault Secrets User only works if your vault uses Azure RBAC for data plane authorization (--enable-rbac-authorization true). If it still uses access policies, use az keyvault set-policy --object-id "$USER_ASSIGNED_PRINCIPAL_ID" --secret-permissions get list instead.

Step 3 — Create the ServiceAccount

The ServiceAccount is the Kubernetes-side identity. The annotation tells the webhook which client ID to inject.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "11111111-1111-1111-1111-111111111111"

Apply it with the real value substituted:

kubectl create namespace payments --dry-run=client -o yaml | kubectl apply -f -

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"
EOF

Optional annotations worth knowing:

azure.workload.identity/tenant-id — override the tenant (multi-tenant scenarios).
azure.workload.identity/service-account-token-expiration — token lifetime in seconds, between 3600 and 86400. Leave it at the default unless you have a specific reason.

Step 4 — Create the federated identity credential

This is the trust anchor. The subject must match exactly the format system:serviceaccount:<namespace>:<name>.

az identity federated-credential create \
  --name fc-payments-api \
  --identity-name "$UAMI" \
  --resource-group "$RG" \
  --issuer "$AKS_OIDC_ISSUER" \
  --subject "system:serviceaccount:payments:payments-api" \
  --audience "api://AzureADTokenExchange"

A few rules that bite people:

One federated credential = one (issuer, subject) pair. A ServiceAccount in staging needs its own credential, even for the same identity. You can attach up to 20 per identity.
The subject is case-sensitive and has no wildcards (except for the newer flexible federated credentials with * support, which are not GA everywhere — assume exact match).
Propagation takes a few seconds to a couple of minutes. If your first test fails, wait and retry before debugging.

Verify:

az identity federated-credential list \
  --identity-name "$UAMI" --resource-group "$RG" -o table

Step 5 — Label the pod and deploy

The webhook only mutates pods carrying the label azure.workload.identity/use: "true" on the pod template, not on the Deployment metadata. This is the single most common mistake.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: payments-api
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: ghcr.io/example/payments-api:1.4.2
          env:
            - name: KEYVAULT_URL
              value: "https://kv-payments-prod.vault.azure.net/"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi

automountServiceAccountToken: false disables the default Kubernetes API token. The Workload Identity projected volume is added independently by the webhook, so Azure auth still works while your pod loses needless API server access.

Verify the mutation happened:

kubectl -n payments get pod -l app=payments-api \
  -o jsonpath='{.items[0].spec.containers[0].env}' | jq .

You should see AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE and AZURE_AUTHORITY_HOST.

Testing from inside a pod

A quick smoke test without touching your application code:

kubectl -n payments run wi-test \
  --image=mcr.microsoft.com/azure-cli:latest \
  --labels="azure.workload.identity/use=true" \
  --overrides='{"spec":{"serviceAccountName":"payments-api"}}' \
  --restart=Never -it --rm -- bash

Inside the pod:

az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" \
  --service-principal \
  -u "$AZURE_CLIENT_ID" \
  -t "$AZURE_TENANT_ID"

az keyvault secret show --vault-name kv-payments-prod --name db-password

If that works, DefaultAzureCredential in your Python, .NET, Java, Go or Node app will work with zero code changes — the SDKs have supported WorkloadIdentityCredential since 2023.

Debugging token errors

`AADSTS70021: No matching federated identity record found`

The most frequent error. The subject or issuer does not match. Decode the token the pod actually received:

kubectl -n payments exec deploy/payments-api -- \
  cat /var/run/secrets/azure/tokens/azure-identity-token \
  | cut -d. -f2 | base64 -d 2>/dev/null | jq '{iss, sub, aud, exp}'

Compare iss and sub character by character with the federated credential. Typical causes: namespace mismatch, trailing slash difference on the issuer, or the app was copied from another environment.

`AZURE_FEDERATED_TOKEN_FILE` is not set

The webhook did not mutate the pod. Check, in order:

# 1. Is the label on the POD, not just the Deployment?
kubectl -n payments get pod -l app=payments-api --show-labels

# 2. Is the annotation on the ServiceAccount?
kubectl -n payments get sa payments-api -o yaml

# 3. Is the webhook alive?
kubectl -n kube-system logs -l azure-workload-identity.io/system=true --tail=50

Remember: the mutation happens at pod admission. Changing a ServiceAccount annotation requires a kubectl rollout restart deployment/payments-api.

`AADSTS700016: Application not found in the directory`

The client-id annotation points to an identity that does not exist, or exists in a different tenant. Re-read it from the source: az identity show --name "$UAMI" -g "$RG" --query clientId -o tsv.

`403 Forbidden` from Key Vault / Storage

Authentication succeeded, authorization failed. The federation is fine — the role assignment is missing or scoped wrong. Check:

az role assignment list \
  --assignee "$USER_ASSIGNED_PRINCIPAL_ID" \
  --all -o table

Also check Key Vault firewall rules: a correct token still gets blocked if your AKS egress IP is not allowed.

Security best practices

One identity per workload, not per cluster. The temptation is a single id-aks-prod with Contributor at subscription scope. That rebuilds the exact blast radius problem you were trying to remove. One UAMI per application, scoped to the specific vault, storage account or database.

Keep subjects narrow. Avoid wildcard federated credentials if your Azure tenant supports them. system:serviceaccount:payments:payments-api is auditable; system:serviceaccount:payments:* means any new ServiceAccount in that namespace silently inherits production access.

Lock down ServiceAccount creation. Anyone who can create a ServiceAccount named payments-api in namespace payments plus a pod using it can assume the identity. Enforce it with RBAC and an admission policy. Example with a Kyverno-style guard or a plain ValidatingAdmissionPolicy:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: restrict-wi-annotation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["serviceaccounts"]
  validations:
    - expression: >
        !has(object.metadata.annotations) ||
        !('azure.workload.identity/client-id' in object.metadata.annotations) ||
        object.metadata.namespace in ['payments', 'billing']
      message: "Workload Identity annotations are restricted to approved namespaces"

Disable the default API token. automountServiceAccountToken: false as shown above. The projected Azure token is independent.

Never disable issuer validation. If a tool offers to skip JWKS verification, that tool is not production-grade.

Rotate the OIDC signing keys periodically. AKS supports it; it invalidates all outstanding projected tokens, which is exactly what you want after a suspected compromise:

az aks oidc-issuer rotate-signing-keys --resource-group "$RG" --name "$CLUSTER"

Expect a short window where existing tokens fail until pods refresh them.

Audit federated credentials like you audit SSH keys. They are long-lived trust statements. Export them regularly and diff against your IaC:

for id in $(az identity list -g "$RG" --query "[].name" -o tsv); do
  echo "== $id"
  az identity federated-credential list --identity-name "$id" -g "$RG" \
    --query "[].{name:name, subject:subject, issuer:issuer}" -o table
done

Codify everything. The four moving parts (UAMI, role assignment, federated credential, ServiceAccount annotation) belong in Terraform or Bicep plus GitOps. Manual az commands are fine for the first 45 minutes; they are not fine for the 50th application.

Conclusion

The migration path is short and mechanical:

az aks update --enable-oidc-issuer --enable-workload-identity — once per cluster.
Create a user-assigned managed identity per workload and give it a narrowly scoped Azure role.
Annotate the ServiceAccount with azure.workload.identity/client-id.
Register a federated identity credential matching system:serviceaccount:<ns>:<sa> with audience api://AzureADTokenExchange.
Add the azure.workload.identity/use: "true" label to the pod template and restart.

Three things to remember when it breaks: the label goes on the pod template, the subject string must match exactly, and the webhook only acts at pod creation time — so restart after any annotation change.

What you gain is not just convenience. You remove an entire class of credentials from Git, from etcd, from CI logs and from your rotation backlog. The token your pod holds lives one hour, is bound to one ServiceAccount in one namespace in one cluster, and is useless anywhere else. That is a meaningfully smaller attack surface for less than an hour of work.