DEV Community: ThankGod Chibugwum Obobo

GitOps in Practice: How to Manage Kubernetes State with ArgoCD and Helm

ThankGod Chibugwum Obobo — Sun, 06 Sep 2026 17:00:00 +0000

Kubernetes configuration drift is a silent killer of operational confidence. A developer applies a quick fix directly with kubectl during an incident. A CI pipeline pushes an update that partially applies before timing out. An engineer edits a ConfigMap in production to test a theory and forgets to update the repository. Six weeks later, nobody is sure what is actually running, the cluster state has diverged from the repository that supposedly defines it.

GitOps solves this by making Git the single source of truth for cluster state. Every change, application version, configuration, replica count, ingress rule, is expressed as a commit. The cluster continuously reconciles toward that committed state. Unauthorized changes are detected and can be automatically reverted. The audit trail is your Git log.

ArgoCD is the most widely adopted GitOps controller for Kubernetes. Combined with Helm for templated application packaging, it provides a complete GitOps platform, continuous reconciliation, multi-environment promotion, RBAC-controlled access, and a UI that shows the real-time gap between desired and actual cluster state.

This guide covers how to implement a production-grade GitOps workflow with ArgoCD and Helm, from installation and Application setup through the App of Apps pattern, sync policies, secrets management, and progressive delivery.

The GitOps Principles in Practice

Before implementation, understand the four principles GitOps enforces:

Declarative: the entire system state is expressed declaratively in Git. Not scripts, not imperative commands, YAML manifests and Helm values that describe the desired end state.

Versioned and immutable: Git history is the audit trail. Every state change is a commit with author, timestamp, and diff. Rolling back is a revert commit, not a manual procedure.

Pulled automatically: the cluster pulls configuration from Git, rather than CI/CD pushing configuration to the cluster. This eliminates the need for CI systems to have cluster credentials, a significant security improvement.

Continuously reconciled: ArgoCD continuously compares desired state (Git) against actual state (cluster) and alerts on drift. With auto-sync enabled, it corrects drift automatically.

Step 1 - Repository Structure for GitOps

A clear repository structure is the foundation of a maintainable GitOps setup. The most common pattern separates application code from deployment configuration:

/gitops-repo                    ← dedicated GitOps repository
  /apps                         ← ArgoCD Application manifests
    /root-app.yaml              ← App of Apps root
    /platform/
      platform-apps.yaml        ← platform-level applications
    /teams/
      team-alpha-apps.yaml
      team-beta-apps.yaml
  /charts                       ← Helm charts for your services
    /orders-service/
      Chart.yaml
      values.yaml               ← default values
      templates/
        deployment.yaml
        service.yaml
        ingress.yaml
        hpa.yaml
  /envs                         ← environment-specific values
    /staging/
      orders-service.yaml       ← staging overrides
    /production/
      orders-service.yaml       ← production overrides
  /infrastructure               ← cluster-level resources
    /cert-manager/
    /ingress-nginx/
    /monitoring/

Key design decisions:
Separate application charts (/charts) from environment values (/envs). The chart defines the template, the environment values define the configuration. Promoting from staging to production is a values change, not a chart change.

Keep all ArgoCD Application manifests in Git (/apps). ArgoCD manages itself, the App of Apps pattern means ArgoCD's own configuration is version-controlled and reconciled like everything else.

Step 2 - Installing ArgoCD

# Create ArgoCD namespace and install
kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all components to be ready
kubectl wait --for=condition=available --timeout=300s \
  deployment/argocd-server -n argocd

# Retrieve initial admin password
kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath='{.data.password}' | base64 -d

# Port-forward the UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

For production installations, manage ArgoCD itself via Helm and the ArgoCD Helm chart, enabling version-controlled ArgoCD configuration and upgrades:

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --values argocd-values.yaml

Step 3 - Creating Your First ArgoCD Application

An ArgoCD Application is the core resource, it links a Git source to a Kubernetes destination and defines how reconciliation should behave:

# apps/orders-service-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-service-staging
  namespace: argocd
  labels:
    environment: staging
    team: backend
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade delete on app removal
spec:
  project: backend-team

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: charts/orders-service
    helm:
      valueFiles:
        - ../../envs/staging/orders-service.yaml  # relative to chart path

  destination:
    server: https://kubernetes.default.svc
    namespace: orders-staging

  syncPolicy:
    automated:
      prune: true         # delete resources removed from Git
      selfHeal: true      # revert unauthorized manual changes
      allowEmpty: false   # never sync an empty resource set (safety guard)
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - RespectIgnoreDifferences=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The selfHeal: true option is the GitOps enforcement mechanism, if someone runs kubectl edit deployment orders-service directly, ArgoCD detects the drift within 3 minutes and reverts it to match Git. This is what makes Git the actual source of truth rather than a suggestion.

Step 4 - Helm Values for Multi-Environment Configuration

Structure Helm values for environment-specific overrides:

# charts/orders-service/values.yaml — defaults
replicaCount: 1
image:
  repository: your-org/orders-service
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80
  targetPort: 3000

ingress:
  enabled: false
  host: ""

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70

env: {}

# envs/staging/orders-service.yaml — staging overrides
replicaCount: 1

image:
  tag: "2.4.1-staging"    # pinned to staging image tag

ingress:
  enabled: true
  host: orders.staging.your-domain.com

resources:
  requests:
    cpu: 50m
    memory: 64Mi

env:
  NODE_ENV: staging
  DATABASE_URL: postgresql://staging-db:5432/orders
  LOG_LEVEL: debug

# envs/production/orders-service.yaml — production overrides
replicaCount: 3

image:
  tag: "2.4.1"            # pinned to production image tag

ingress:
  enabled: true
  host: orders.your-domain.com

resources:
  requests:
    cpu: 100m
    memory: 224Mi
  limits:
    cpu: 500m
    memory: 360Mi

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20

env:
  NODE_ENV: production
  LOG_LEVEL: warn

Image tags are pinned, never latest and promotion from staging to production is a single-line diff in envs/production/orders-service.yaml. That diff is a pull request, code-reviewed, and merged. ArgoCD detects the merge and applies the change to the production cluster automatically.

Step 5 - The App of Apps Pattern

Managing dozens of ArgoCD Application resources individually doesn't scale. The App of Apps pattern treats the collection of applications as a managed set, a root Application that points to a directory of Application manifests, and ArgoCD manages them all:

# apps/root-app.yaml — the root of everything
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: apps            # ArgoCD watches this directory for Application manifests

  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply this single manifest manually once:

kubectl apply -f apps/root-app.yaml

ArgoCD discovers every Application manifest in the apps/ directory and creates/manages them all. Adding a new application is adding a new YAML file to the apps/ directory and committing. Removing an application is removing its manifest, with prune: true, ArgoCD cleans up the cluster resources automatically.

This is the GitOps pattern in its purest form, the cluster configuration, including its own management configuration, is fully defined in and controlled by Git.

Step 6 - Projects and RBAC for Multi-Team Environments

ArgoCD AppProject resources define boundaries between teams, controlling which repositories, clusters, and namespaces each team's Applications can target:

# infrastructure/argocd/projects/backend-team.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: backend-team
  namespace: argocd
spec:
  description: Backend engineering team applications

  # Which Git repositories this project can deploy from
  sourceRepos:
    - https://github.com/your-org/gitops-repo

  # Which clusters and namespaces this project can deploy to
  destinations:
    - server: https://kubernetes.default.svc
      namespace: orders-*     # wildcard — any namespace starting with orders-
    - server: https://kubernetes.default.svc
      namespace: users-*

  # Which Kubernetes resource types this project can manage
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace

  namespaceResourceWhitelist:
    - group: 'apps'
      kind: Deployment
    - group: ''
      kind: Service
    - group: 'networking.k8s.io'
      kind: Ingress
    - group: 'autoscaling'
      kind: HorizontalPodAutoscaler

  # Deny deploying privileged containers
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota

  roles:
    - name: developer
      description: Read-only access plus sync trigger
      policies:
        - p, proj:backend-team:developer, applications, get, backend-team/*, allow
        - p, proj:backend-team:developer, applications, sync, backend-team/*, allow
      groups:
        - backend-developers     # maps to your IdP group

    - name: lead
      description: Full access to backend team applications
      policies:
        - p, proj:backend-team:lead, applications, *, backend-team/*, allow
      groups:
        - backend-leads

RBAC is enforced at the ArgoCD level, developers can trigger syncs and view application state, but cannot modify Application resources, change sync policies, or deploy to namespaces outside their project's boundaries.

Step 7 - Secrets Management with ArgoCD

ArgoCD should never store secrets in Git, even encrypted. The two recommended approaches:

Sealed Secrets

# Install Sealed Secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Encrypt a secret for Git storage
kubectl create secret generic orders-db-credentials \
  --from-literal=DATABASE_URL=postgresql://user:password@db:5432/orders \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > charts/orders-service/templates/db-secret.sealed.yaml

The SealedSecret in Git can only be decrypted by the Sealed Secrets controller running in your specific cluster, safe to commit.

External Secrets Operator with Vault

For teams already using HashiCorp Vault (as covered in our secrets management guide):

# charts/orders-service/templates/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: orders-service-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: orders-service-env
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: secret/production/orders-service/database
        property: url
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: secret/production/orders-service/stripe
        property: api_key

ArgoCD deploys the ExternalSecret resource, the External Secrets Operator fetches the actual values from Vault and creates the Kubernetes Secret. Vault credentials never appear in Git.

Step 8 - Progressive Delivery with ArgoCD Rollouts

For production deployments where blue/green or canary strategies are required, Argo Rollouts extends ArgoCD with progressive delivery capabilities:

# charts/orders-service/templates/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: orders-service
spec:
  replicas: 5
  selector:
    matchLabels:
      app: orders-service
  template:
    spec:
      containers:
        - name: orders-service
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

  strategy:
    canary:
      steps:
        - setWeight: 10           # route 10% of traffic to new version
        - pause: { duration: 5m } # wait 5 minutes — observe error rate
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100          # full traffic if no issues detected

      # Automatically roll back if error rate exceeds threshold
      analysis:
        templates:
          - templateName: success-rate-check
        startingStep: 1
        args:
          - name: service-name
            value: orders-service

# infrastructure/argocd/analysis-templates/success-rate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.98
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus-server.monitoring.svc.cluster.local
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

The canary automatically rolls back if the success rate drops below 98% for three consecutive minutes, combining GitOps promotion (the image tag change in Git) with data-driven delivery (the rollback decision from Prometheus).

Step 9 - Handling Sync Waves and Dependencies

Some resources must exist before others, namespaces before deployments, CRDs before custom resources, Secrets before Deployments that mount them. ArgoCD sync waves control this ordering:

# Apply this first (wave -1) — before all other resources
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
# Apply this second (wave 0, default)
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Apply this last (wave 1) — after secrets and config are ready
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Use sync waves for:

Database migrations (wave: 0) before the application deployment (wave: 1)
CRD installation (wave: -1) before custom resources (wave: 0)
Secrets (wave: -1) before Deployments that reference them (wave: 0)

Common Pitfalls to Avoid

Using latest image tags. ArgoCD reconciles to the state in Git. If latest resolves to a different image without a Git commit, ArgoCD won't detect it as a change and won't reflect it in the UI. Always pin image tags and update them via Git commits.

Auto-sync without selfHeal in production. Auto-sync without selfHeal applies changes from Git but doesn't revert unauthorized manual changes. Enable both for full GitOps enforcement in production environments.

Storing unencrypted secrets in Git. Even in private repositories, plaintext secrets in Git are a security risk. Use Sealed Secrets or External Secrets Operator, there is no legitimate reason for plaintext credentials in a GitOps repository.

No sync windows for production. Configure sync windows to restrict when ArgoCD can apply changes to production clusters, preventing automated syncs during peak traffic hours or business-critical periods:

syncPolicy:
  syncWindows:
    - kind: allow
      schedule: '0 2-6 * * 1-5'  # weekdays 2am-6am only
      duration: 4h
      applications: ['*']

Ignoring ArgoCD health status. ArgoCD tracks application health (Healthy, Progressing, Degraded) based on Kubernetes resource conditions. A Degraded health status on a synced application means the desired state was applied but the application isn't healthy, a deployment with crashing pods, for example. Treat health status with the same urgency as a failed deployment.

Conclusion

GitOps with ArgoCD and Helm is not just a deployment mechanism, it is a complete operational model that eliminates an entire category of operational risk. Cluster drift, unclear ownership, undocumented changes, and failed rollbacks are structural problems that Git-driven reconciliation solves at the architecture level rather than the process level.

The App of Apps pattern scales the model across dozens of services and teams. Projects and RBAC enforce team boundaries without ticket-based access control. Sync policies enforce Git as the source of truth, not a guideline. Progressive delivery with Argo Rollouts brings data-driven safety to the final mile of deployment.

The investment is real, repository structure, RBAC design, secrets management, and sync policy tuning require deliberate engineering. But once the platform is in place, the operational discipline it enforces is automatic rather than cultural, which is the only kind of discipline that holds at scale.

Running multiple Kubernetes clusters across environments or cloud providers? ArgoCD's multi-cluster support with ApplicationSets scales the App of Apps pattern to fleet-level management without per-cluster configuration duplication.

GitOps #ArgoCD #Helm #Kubernetes #DevOps

Kubernetes Cost Optimization: How to Right-Size Workloads with Goldilocks and VPA

ThankGod Chibugwum Obobo — Sun, 30 Aug 2026 17:00:00 +0000

Kubernetes resource requests and limits are one of the most consequential configuration decisions in cloud-native infrastructure, and one of the least data-driven. Most teams set them once during initial deployment, based on estimates or copied from a Stack Overflow answer, and never revisit them. The result is predictable, workloads are systematically over-provisioned to avoid OOM kills, node utilization stays at 20–40%, and cloud bills grow faster than the business demands it.

The irony is that right-sizing is not guesswork, it's a measurement problem. Every Kubernetes cluster already collects the utilization data needed to set accurate resource requests. The gap is tooling that surfaces those measurements as actionable recommendations without requiring engineers to manually analyze Prometheus queries for every workload.

Goldilocks and the Vertical Pod Autoscaler (VPA) close that gap. VPA observes actual resource consumption and computes right-sized recommendations. Goldilocks surfaces those recommendations in a readable dashboard, organized by namespace, with diff-ready output that shows exactly what to change in your Helm values or Kubernetes manifests.

This guide covers how to deploy and configure both tools, interpret their recommendations, implement right-sizing changes safely, and integrate continuous cost optimization into your engineering workflow.

The Resource Request Problem

To understand why right-sizing matters, start with how Kubernetes uses resource requests:

Resource requests are what the Kubernetes scheduler uses to decide which node to place a pod on. A pod requesting 500m CPU and 512Mi memory will only be scheduled on a node with that much unallocated capacity.

Resource limits cap what a pod can actually consume. A pod that exceeds its CPU limit is throttled. A pod that exceeds its memory limit is OOM killed.

The consequence of over-provisioning requests is that capacity is reserved on nodes that the workload never uses. A pod requesting 2000m CPU that actually uses 200m is holding a slot for 1800m CPU that no other pod can use, you're paying for 10x the compute the workload needs.

The consequence of under-provisioning limits is that pods get OOM killed, generating false incidents. CPU throttling degrades latency without generating visible errors, a silent performance problem that many teams misdiagnose.

Accurate resource requests and limits require knowing what workloads actually consume under real load, which requires measurement, not estimation.

Architecture: How VPA and Goldilocks Work Together

Vertical Pod Autoscaler (VPA) is a Kubernetes controller that observes pod resource utilization over time and computes recommendation ranges:

Lower Bound - the minimum request that prevents resource starvation
Target - the recommended request for typical workload behavior
Upper Bound - the maximum consumed under peak conditions

VPA has three modes: Auto (applies recommendations automatically, with pod restarts), Recreate (applies on pod recreation only), and Off (computes recommendations but applies nothing). For most production environments, Off mode (recommendations without automatic application) is the right starting point.

Goldilocks wraps VPA's Off mode with a namespace-level dashboard that aggregates VPA recommendations across all deployments in a namespace. It creates a VerticalPodAutoscaler object in Off mode for every deployment it monitors, collects the resulting recommendations, and presents them in a UI showing current requests vs. recommended requests with quality-of-service (QoS) class implications.

The workflow is:

Workloads run → Metrics Server collects utilization →
VPA Recommender analyzes history →
Goldilocks reads VPA recommendations →
Dashboard shows what to change →
Engineers apply changes to Helm values / manifests →
Next cycle validates the new settings

Step 1 - Install Metrics Server

VPA depends on the Kubernetes Metrics Server for resource utilization data. Verify it is running:

kubectl top nodes
kubectl top pods -A

If kubectl top returns an error, install Metrics Server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Verify
kubectl get deployment metrics-server -n kube-system

Step 2 - Install the Vertical Pod Autoscaler

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Install VPA components
./hack/vpa-up.sh

# Verify all three VPA components are running
kubectl get pods -n kube-system | grep vpa
# vpa-admission-controller-xxx   1/1   Running
# vpa-recommender-xxx            1/1   Running
# vpa-updater-xxx                1/1   Running

Alternatively, install via Helm:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa \
  --namespace kube-system \
  --set recommender.enabled=true \
  --set updater.enabled=false \    # disable auto-updates - use Off mode
  --set admissionController.enabled=false

The VPA recommender is the only component required for Goldilocks, it computes recommendations. The updater and admission controller are needed only for automatic recommendation application, which we'll address separately.

Step 3 - Install Goldilocks

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace \
  --set dashboard.enabled=true \
  --set dashboard.service.type=ClusterIP

Verify Goldilocks is running:

kubectl get pods -n goldilocks
# goldilocks-controller-xxx    1/1   Running
# goldilocks-dashboard-xxx     1/1   Running

Step 4 - Enable Goldilocks Per Namespace

Goldilocks only monitors namespaces explicitly labeled for analysis. Label the namespaces you want to optimize:

# Enable Goldilocks for your application namespaces
kubectl label namespace orders-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace users-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace payments-service goldilocks.fairwinds.com/enabled=true

Once labeled, Goldilocks automatically creates a VerticalPodAutoscaler object in Off mode for every Deployment in the namespace. VPA begins observing resource utilization immediately.

Important: Allow at least 24–72 hours of observation before acting on recommendations. VPA's accuracy improves with more history, recommendations based on 15 minutes of data reflect startup behavior, not steady-state production utilization.

Step 5 - Accessing the Goldilocks Dashboard

# Port-forward the dashboard
kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80

# Open http://localhost:8080

The dashboard displays, per namespace and per deployment:

Current requests and limits - what your manifests currently specify
VPA recommendations - Lower Bound, Target, and Upper Bound for CPU and memory
QoS class impact - whether your current and recommended settings result in Guaranteed, Burstable, or BestEffort quality-of-service
Helm values diff - the exact resources block to paste into your Helm values file

A typical recommendation output looks like:

# Current (what you have)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "1Gi"

# Goldilocks recommended (VPA Target)
resources:
  requests:
    cpu: "87m"
    memory: "179Mi"
  limits:
    cpu: "262m"
    memory: "358Mi"

This workload is requesting 5.7x more CPU than it actually uses. Applying the recommendation would reduce its node footprint by ~83%, allowing the cluster to either schedule significantly more workloads on existing nodes or rightsize to fewer nodes.

Step 6 - Interpreting Recommendations Safely

Goldilocks recommendations are starting points, not gospel. Apply these interpretation rules before implementing changes:

Use the Upper Bound for Memory Limits

VPA's memory Target represents typical consumption. Set your memory limit to the Upper Bound, the peak observed usage, to avoid OOM kills during traffic spikes. Memory limits below observed peak are the primary cause of OOM kills.

resources:
  requests:
    memory: "179Mi"      # VPA Target → use as request
  limits:
    memory: "358Mi"      # VPA Upper Bound → use as limit (not Target)

Add a Safety Buffer for Memory Requests

For stateful or memory-sensitive workloads, add a 20–30% buffer above the VPA Target for memory requests:

# VPA Target: 179Mi → apply with 25% buffer
requests:
  memory: "224Mi"    # 179 × 1.25 = 223.75 → round up

Be Conservative with CPU Limits

Unlike memory, CPU throttling does not kill pods, it degrades latency. Setting CPU limits close to observed peak reduces burst headroom. Consider setting CPU limits to the Upper Bound and leaving headroom for traffic spikes:

requests:
  cpu: "87m"         # VPA Target - accurate for scheduling
limits:
  cpu: "500m"        # Upper Bound + buffer - prevents pathological consumption

Ignore Recommendations for Stateful Workloads

VPA recommendations for StatefulSets (databases, message brokers, caches) should be applied with extra caution, memory consumption patterns for stateful workloads are often highly variable and spike during compaction, reindexing, or failover operations. Treat VPA recommendations for stateful workloads as lower bounds, not targets.

Step 7 - Implementing Changes Safely

Never apply all VPA recommendations at once. A phased approach minimizes risk:

Phase 1 - Non-production environments first. Apply recommendations to staging and development. Observe for one week. Verify no OOM kills, no CPU throttling-related latency degradation, no unexpected behavior.

Phase 2 - Lowest-traffic production workloads. Apply to internal tools, admin services, and background workers, workloads with the lowest blast radius if a recommendation is wrong.

Phase 3 - Core application services. Apply to API services and user-facing workloads after observing Phase 2 outcomes. Monitor error rates, latency p95/p99, and OOM kill rates for 48 hours after each change.

For Helm-managed deployments, apply recommendations via values override:

# values/production/orders-service.yaml — after Goldilocks review
resources:
  requests:
    cpu: "100m"       # up from VPA's 87m — minor safety buffer
    memory: "224Mi"   # VPA Target (179Mi) + 25% buffer
  limits:
    cpu: "500m"       # VPA Upper Bound + margin
    memory: "360Mi"   # VPA Upper Bound

helm upgrade orders-service ./charts/orders-service \
  -f values/production/orders-service.yaml \
  --namespace orders-service

Step 8 - VPA Auto Mode for Non-Critical Workloads

For non-critical workloads (batch jobs, internal tools, development namespaces) where pod restarts are acceptable, VPA's Auto mode applies recommendations without manual intervention:

# vpa/orders-worker-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-worker-vpa
  namespace: orders-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-worker
  updatePolicy:
    updateMode: "Auto"   # applies recommendations automatically
  resourcePolicy:
    containerPolicies:
      - containerName: orders-worker
        minAllowed:
          cpu: "50m"
          memory: "64Mi"
        maxAllowed:
          cpu: "2000m"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]

The minAllowed and maxAllowed bounds prevent VPA from recommending values outside your operational comfort zone, a safeguard against runaway recommendations if a workload behaves unusually during the observation window.

Step 9 - CI/CD Integration for Continuous Right-Sizing

Make right-sizing a recurring engineering practice rather than a one-time event by integrating Goldilocks into your workflow:

# .github/workflows/cost-review.yml
name: Monthly Cost Optimization Review

on:
  schedule:
    - cron: '0 9 1 * *'   # first day of each month at 9am

jobs:
  generate-recommendations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Export Goldilocks recommendations
        run: |
          kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80 &
          sleep 5

          # Export recommendations via Goldilocks API
          curl -s http://localhost:8080/api/v1/namespaces \
            | jq '.[] | {namespace: .name, recommendations: .recommendations}' \
            > recommendations.json

      - name: Create GitHub Issue with recommendations
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const recs = JSON.parse(fs.readFileSync('recommendations.json'));

            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Monthly Cost Optimization Review — ${new Date().toISOString().slice(0, 7)}`,
              body: `## Goldilocks Recommendations\n\n\`\`\`json\n${JSON.stringify(recs, null, 2)}\n\`\`\`\n\nReview and apply relevant recommendations following the right-sizing runbook.`,
              labels: ['cost-optimization', 'infrastructure'],
              assignees: ['platform-team'],
            });

A monthly GitHub issue with current recommendations creates a lightweight, trackable cost optimization cadence, without requiring engineers to proactively access the dashboard.

Measuring the Impact

Track right-sizing outcomes with these Grafana/Prometheus metrics:

# Node CPU utilization — target 60-70% for cost efficiency
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (node) * 100

# Pod CPU request vs actual usage (request efficiency)
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, namespace)

# OOM kill rate — should be zero after right-sizing
sum(increase(kube_pod_container_status_restarts_total[24h])) by (pod, namespace)

# Memory request efficiency
sum(container_memory_working_set_bytes) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="memory"}) by (pod, namespace)

A request efficiency ratio below 0.3 (pods using less than 30% of requested resources) is a strong signal of over-provisioning. A ratio above 0.85 sustained means requests are too tight, burst traffic risks throttling or OOM.

Common Pitfalls to Avoid

Acting on fresh recommendations. VPA recommendations based on less than 24 hours of data reflect startup patterns, not steady-state behavior. Wait at least 72 hours before applying recommendations to production workloads.

Setting memory limits equal to requests. This creates Guaranteed QoS class pods, Kubernetes evicts Burstable pods before Guaranteed ones, which sounds good but means any memory spike above the limit results in an OOM kill with no burst headroom. Set limits meaningfully above requests.

Applying VPA Auto mode to HPA-managed deployments. VPA and HPA conflict when both target the same metric. If you use HPA for CPU-based scaling, configure VPA to manage only memory: controlledResources: ["memory"].

Ignoring the QoS class implications. Guaranteed QoS (requests = limits) gets the highest scheduling priority but no burst headroom. Burstable QoS (requests < limits) allows bursting but can be evicted under node pressure. Understand which class your workloads land in after right-sizing.

Right-sizing without load testing. VPA recommendations are based on historical traffic. If your staging environment doesn't reflect production load patterns, its recommendations don't reflect production requirements. Right-size production workloads using production observation data.

Conclusion

Kubernetes cost optimization is not a one-time project, it's an ongoing engineering discipline driven by measurement rather than estimation. VPA provides the utilization data. Goldilocks makes that data actionable. The right-sizing process, observe, recommend, review, apply, measure, is the feedback loop that keeps cluster efficiency high as workloads evolve.

The typical outcome of a systematic right-sizing initiative is a 30–50% reduction in resource requests, translating directly to fewer nodes needed to run the same workloads, and a proportional reduction in cloud spend. More importantly, accurate resource requests improve scheduling density, reduce node pressure evictions, and eliminate the silent CPU throttling that degrades latency without generating visible errors.

Start with one namespace. Label it for Goldilocks, wait 72 hours, review the dashboard, and apply the most conservative recommendations first. The data will tell you what to do, you just have to make time to look at it.

Using Karpenter for node provisioning alongside VPA for pod right-sizing? The combination is particularly effective, Karpenter provisions exactly the node types needed for right-sized pod requests, eliminating both over-provisioned pods and over-provisioned nodes simultaneously.

Kubernetes #CostOptimization #Goldilocks #VPA #CloudCost

Durable Execution with Temporal.io: Building Workflows That Never Lose State

ThankGod Chibugwum Obobo — Sun, 23 Aug 2026 17:00:00 +0000

Distributed systems fail in partial and unpredictable ways. A payment workflow that processes a charge, updates inventory, sends a confirmation email, and notifies a fulfilment service has seven places to fail, and no built-in mechanism to pick up where it left off. The traditional response is a combination of message queues, idempotency keys, retry logic, state machines stored in a database, and compensating transactions, each adding complexity, each requiring its own failure handling, and each representing a subtle opportunity for state to be lost or corrupted.

Temporal.io is built on a different premise, what if your workflow code simply continued executing after a failure, a restart, or a crash, exactly where it left off, with all local variables intact, all function calls remembered, and no manual state management required?

This is durable execution, the guarantee that a workflow's progress is preserved indefinitely, regardless of what happens to the infrastructure running it. Temporal achieves this by recording every workflow event to a persistent log and replaying the workflow's history to reconstruct state after any failure. From the developer's perspective, you write ordinary sequential code. Temporal makes it fault-tolerant.

This guide covers Temporal's core concepts and shows how to implement durable workflows in TypeScript, including activities, retries, signals, timers, and child workflows, for the kinds of long-running business processes that break traditional queue-based approaches.

How Temporal Works: The Core Model

Understanding Temporal's execution model upfront prevents confusion later.

Workflows are deterministic functions that orchestrate work. They define what happens and in what order. Workflow code must be deterministic, the same inputs must always produce the same sequence of events, because Temporal replays workflow history to reconstruct state after failures.

Activities are non-deterministic units of work, the things that actually talk to the outside world, database writes, API calls, file operations, sending emails. Activities are executed by workers, can be retried independently, and their results are recorded in the workflow history.

Workers are long-running processes that poll Temporal's task queues and execute workflow and activity code. You run workers, Temporal runs the orchestration.

The Temporal Server persists workflow state, manages task queues, handles timers, and routes signals. It is the durable backbone, workers are stateless and replaceable.

The key insight: workflow code never talks to the outside world directly. It schedules activities and waits for results. Temporal records the activity result to the event history. If the worker dies before the workflow advances, Temporal replays the history, skipping already-completed activities using their recorded results, and the workflow continues from where it stopped. The activity does not re-execute. No duplicate charges. No double emails.

Step 1 - Installation and Project Setup

# Create a new TypeScript project
mkdir temporal-demo && cd temporal-demo
npm init -y
npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity
npm install -D typescript ts-node @types/node

# Start Temporal development server (requires Docker)
docker run --rm -p 7233:7233 -p 8080:8080 temporalio/auto-setup:latest

Project structure:

/temporal-demo
  /src
    /workflows
      order-fulfillment.workflow.ts
      subscription-billing.workflow.ts
    /activities
      payment.activities.ts
      inventory.activities.ts
      notification.activities.ts
    worker.ts        ← registers and runs workflows + activities
    client.ts        ← starts workflows from your application

Step 2 - Defining Activities

Activities are where side effects live. Each activity is a plain async function, no Temporal-specific decorators required:

// src/activities/payment.activities.ts
import { ApplicationFailure } from '@temporalio/activity';

export interface ChargeResult {
  transactionId: string;
  amount: number;
  status: 'succeeded' | 'failed';
}

export async function chargeCustomer(
  customerId: string,
  amount: number,
  idempotencyKey: string,
): Promise<ChargeResult> {
  try {
    // Call your payment provider (Stripe, etc.)
    const charge = await stripeClient.charges.create({
      customer: customerId,
      amount,
      currency: 'usd',
      idempotency_key: idempotencyKey,   // prevent duplicate charges on retry
    });

    return {
      transactionId: charge.id,
      amount: charge.amount,
      status: 'succeeded',
    };
  } catch (error) {
    if (error.code === 'card_declined') {
      // Non-retryable failure — throw ApplicationFailure with retryable: false
      throw new ApplicationFailure(
        'Card declined',
        'CARD_DECLINED',
        false,            // do not retry this activity
        [],
        error,
      );
    }
    // Retryable error — Temporal will retry according to retry policy
    throw error;
  }
}

export async function refundCustomer(
  transactionId: string,
  amount: number,
): Promise<void> {
  await stripeClient.refunds.create({
    charge: transactionId,
    amount,
  });
}

// src/activities/inventory.activities.ts
export async function reserveInventory(
  items: OrderItem[],
  orderId: string,
): Promise<void> {
  await inventoryService.reserve(items, orderId);
}

export async function releaseInventory(
  orderId: string,
): Promise<void> {
  await inventoryService.release(orderId);
}

// src/activities/notification.activities.ts
export async function sendOrderConfirmation(
  customerId: string,
  orderId: string,
  items: OrderItem[],
): Promise<void> {
  await emailService.send({
    to: customerId,
    template: 'order-confirmation',
    data: { orderId, items },
  });
}

export async function sendOrderFailureNotification(
  customerId: string,
  reason: string,
): Promise<void> {
  await emailService.send({
    to: customerId,
    template: 'order-failed',
    data: { reason },
  });
}

Step 3 - Building the Order Fulfillment Workflow

This workflow orchestrates the full order lifecycle, charge, reserve inventory, confirm, with automatic compensation (refund + release inventory) if any step fails:

// src/workflows/order-fulfillment.workflow.ts
import { proxyActivities, sleep, defineSignal, setHandler, condition, ApplicationFailure } from '@temporalio/workflow';
import type * as PaymentActivities from '../activities/payment.activities';
import type * as InventoryActivities from '../activities/inventory.activities';
import type * as NotificationActivities from '../activities/notification.activities';

// Configure activity options - retry policy, timeouts
const { chargeCustomer, refundCustomer } = proxyActivities<typeof PaymentActivities>({
  startToCloseTimeout: '30 seconds',
  retry: {
    maximumAttempts: 3,
    initialInterval: '1 second',
    backoffCoefficient: 2,
    nonRetryableErrorTypes: ['CARD_DECLINED'],
  },
});

const { reserveInventory, releaseInventory } = proxyActivities<typeof InventoryActivities>({
  startToCloseTimeout: '10 seconds',
  retry: {
    maximumAttempts: 5,
    initialInterval: '500ms',
    backoffCoefficient: 1.5,
  },
});

const { sendOrderConfirmation, sendOrderFailureNotification } = proxyActivities<typeof NotificationActivities>({
  startToCloseTimeout: '15 seconds',
  retry: {
    maximumAttempts: 10,           // emails should retry aggressively
    initialInterval: '5 seconds',
  },
});

// Signals - external events that can influence workflow execution
export const cancelOrderSignal = defineSignal<[reason: string]>('cancelOrder');

export interface OrderInput {
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
}

export async function orderFulfillmentWorkflow(input: OrderInput): Promise<string> {
  const { orderId, customerId, items, totalAmount } = input;

  let cancelRequested = false;
  let cancelReason = '';

  // Register signal handler - cancellation can arrive at any time
  setHandler(cancelOrderSignal, (reason: string) => {
    cancelRequested = true;
    cancelReason = reason;
  });

  // Step 1 - Check for cancellation before starting
  if (cancelRequested) {
    await sendOrderFailureNotification(customerId, cancelReason);
    return 'CANCELLED';
  }

  // Step 2 - Charge the customer
  let chargeResult;
  try {
    chargeResult = await chargeCustomer(
      customerId,
      totalAmount,
      orderId,     // use orderId as idempotency key — safe to retry
    );
  } catch (error) {
    await sendOrderFailureNotification(customerId, 'Payment could not be processed.');
    throw error;  // fail the workflow — no compensation needed, no charge was made
  }

  // Check for cancellation after payment
  if (cancelRequested) {
    await refundCustomer(chargeResult.transactionId, totalAmount);
    await sendOrderFailureNotification(customerId, cancelReason);
    return 'CANCELLED_AND_REFUNDED';
  }

  // Step 3 — Reserve inventory
  try {
    await reserveInventory(items, orderId);
  } catch (error) {
    // Inventory failed — refund the charge (compensating transaction)
    await refundCustomer(chargeResult.transactionId, totalAmount);
    await sendOrderFailureNotification(customerId, 'Items are out of stock.');
    throw error;
  }

  // Step 4 — Send confirmation
  await sendOrderConfirmation(customerId, orderId, items);

  return 'FULFILLED';
}

The compensation pattern (refund on inventory failure) is expressed as ordinary sequential code, not a saga framework, not a choreography of events, not a state machine in a database. Temporal records every activity result. If the worker crashes during sendOrderConfirmation, Temporal replays the history, sees that chargeCustomer and reserveInventory already completed, and re-executes only from the sendOrderConfirmation call.

Step 4 - Long-Running Workflows with Timers

Temporal timers are durable, they survive server restarts and worker crashes, firing at the correct time regardless of what happened to the infrastructure while they were waiting:

// src/workflows/subscription-billing.workflow.ts
import { proxyActivities, sleep, continueAsNew } from '@temporalio/workflow';

const { chargeCustomer, sendInvoice, handlePaymentFailure } =
  proxyActivities<typeof BillingActivities>({
    startToCloseTimeout: '30 seconds',
    retry: { maximumAttempts: 3 },
  });

export interface SubscriptionInput {
  customerId: string;
  planId: string;
  monthlyAmount: number;
  billingCycleCount: number;   // how many cycles have run
}

export async function subscriptionBillingWorkflow(
  input: SubscriptionInput,
): Promise<void> {
  const { customerId, planId, monthlyAmount, billingCycleCount } = input;

  // Wait for next billing date — durable 30-day timer
  await sleep('30 days');

  // Attempt billing
  try {
    const result = await chargeCustomer(
      customerId,
      monthlyAmount,
      `${planId}-cycle-${billingCycleCount}`,
    );
    await sendInvoice(customerId, result.transactionId, monthlyAmount);
  } catch (error) {
    await handlePaymentFailure(customerId, billingCycleCount);
  }

  // Temporal event history grows with each cycle — use continueAsNew
  // to reset history while preserving workflow identity
  await continueAsNew<typeof subscriptionBillingWorkflow>({
    customerId,
    planId,
    monthlyAmount,
    billingCycleCount: billingCycleCount + 1,
  });
}

continueAsNew is critical for perpetual workflows, it closes the current workflow history and starts a new execution with fresh history, preserving the workflow ID. Without it, a subscription billing workflow running for years would accumulate an unbounded event history.

A durable sleep('30 days') in Temporal is not a 30-day setTimeout. If the worker is restarted, the server rebooted, or the entire Temporal cluster is upgraded during those 30 days, the timer fires correctly when the time comes. This is qualitatively different from any timer mechanism available in a standard application runtime.

Step 5 - Signals: Communicating with Running Workflows

Signals allow external systems to send events to a running workflow, without polling, without a database flag, without a queue:

// From your NestJS API — send a cancellation signal to a running workflow
import { Client } from '@temporalio/client';
import { cancelOrderSignal } from './workflows/order-fulfillment.workflow';

const client = new Client();

// Cancel a specific order — signal the running workflow directly
async function cancelOrder(orderId: string, reason: string): Promise<void> {
  const handle = client.workflow.getHandle(orderId);  // orderId is the workflow ID
  await handle.signal(cancelOrderSignal, reason);
}

// Query workflow state without interrupting it
async function getOrderStatus(orderId: string): Promise<string> {
  const handle = client.workflow.getHandle(orderId);
  return handle.query(orderStatusQuery);
}

The signal is delivered to the workflow exactly once, durably, even if the workflow is in the middle of a sleep, waiting for an activity, or processing another signal. The workflow receives it on the next evaluation cycle.

Step 6 - Child Workflows for Decomposition

Large workflows decompose into child workflows, independently observable, independently retryable units of orchestration:

// Parent workflow — orchestrates the overall order process
import { executeChild } from '@temporalio/workflow';
import { paymentWorkflow } from './payment.workflow';
import { fulfilmentWorkflow } from './fulfilment.workflow';

export async function orderOrchestrationWorkflow(
  order: Order,
): Promise<void> {
  // Execute payment as a child workflow — has its own retry policy and history
  const paymentResult = await executeChild(paymentWorkflow, {
    args: [{ orderId: order.id, customerId: order.customerId, amount: order.total }],
    workflowId: `payment-${order.id}`,
    taskQueue: 'payment-queue',
  });

  if (paymentResult.status !== 'succeeded') {
    throw new ApplicationFailure('Payment failed', 'PAYMENT_FAILED');
  }

  // Execute fulfilment as a child workflow in parallel for each warehouse
  await Promise.all(
    order.warehouses.map(warehouseId =>
      executeChild(fulfilmentWorkflow, {
        args: [{ orderId: order.id, warehouseId }],
        workflowId: `fulfilment-${order.id}-${warehouseId}`,
        taskQueue: 'fulfilment-queue',
      })
    )
  );
}

Child workflows are independently visible in the Temporal UI, you can observe the payment workflow and the fulfilment workflows as separate entities, each with their own event history, each independently retryable.

Step 7 - Running Workers

// src/worker.ts
import { Worker } from '@temporalio/worker';
import * as PaymentActivities from './activities/payment.activities';
import * as InventoryActivities from './activities/inventory.activities';
import * as NotificationActivities from './activities/notification.activities';

async function runWorker() {
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows/order-fulfillment.workflow'),
    activities: {
      ...PaymentActivities,
      ...InventoryActivities,
      ...NotificationActivities,
    },
    taskQueue: 'orders-queue',
    maxConcurrentActivityTaskExecutions: 20,
    maxConcurrentWorkflowTaskExecutions: 10,
  });

  await worker.run();
}

runWorker().catch(console.error);

Workers are stateless. Run as many as you need for throughput, Temporal distributes tasks across all available workers automatically. Scale workers horizontally without coordination.

Temporal vs. Traditional Approaches

Problem	Traditional Approach	Temporal
Retry failed steps	Manual retry logic + dead letter queues	Configurable retry policy per activity
Long-running timers	Cron jobs + database state	Durable `sleep()`, survives restarts
Parallel execution	Manual async coordination	`Promise.all()` over activity calls
Cancel in-flight process	Poll a database flag	Send a signal
Compensating transactions	Saga framework or manual rollback	Sequential code with try/catch
Observe running processes	Query database state	Temporal UI + queries
Workflow versioning	Schema migrations	Built-in versioning API

Common Pitfalls to Avoid

Non-deterministic workflow code. Calling Date.now(), Math.random(), or accessing external state directly in workflow code breaks replay. Use workflow.now() for timestamps, pass random values as workflow inputs, and read external state only through activities.

Blocking in workflows. Never use setTimeout, setInterval, or promise-based delays in workflow code. Use Temporal's sleep(), it is durable, the others are not.

Too much logic in activities. Activities should do one thing, one API call, one database write. Complex coordination logic belongs in the workflow, not the activity. An activity that does five things is an activity that partially fails and leaves unclear what to retry.

Missing idempotency keys on activities. Activities retry. If the activity makes a charge or sends an email, it must include an idempotency key derived from the workflow input, not a randomly generated UUID per attempt.

Unbounded workflow history. Perpetual workflows that never use continueAsNew accumulate event history indefinitely. This degrades performance and eventually hits Temporal's history size limits. Use continueAsNew for any workflow with recurring cycles.

Conclusion

Temporal.io solves a category of problem that traditional queues, cron jobs, and state machines handle poorly, long-running, multi-step business processes that must survive infrastructure failures without losing progress.

The programming model is its defining advantage. You write sequential TypeScript code, charge, reserve, confirm, and Temporal makes it fault-tolerant. No saga framework to configure. No compensating transaction registry to maintain. No polling loops to implement. The workflow code is the specification of the business process, readable by any engineer on the team.

The learning curve is real, the determinism constraint, replay semantics, and continueAsNew pattern require careful understanding. But once internalized, the model eliminates an entire category of distributed systems complexity that most engineering teams carry as permanent operational debt.

Migrating from a queue-based architecture (BullMQ, SQS, RabbitMQ) to Temporal? The migration is incremental, Temporal can consume from existing queues via activities while you migrate orchestration logic workflow by workflow.

Temporal #DurableExecution #WorkflowOrchestration #NodeJS #TypeScript

Goodbye YAML? My First Month with Pkl (Apple's Configuration Language)

ThankGod Chibugwum Obobo — Sun, 16 Aug 2026 17:00:00 +0000

YAML has been the configuration language of the cloud-native era by default, not by design. It is whitespace-sensitive in ways that cause real production incidents. It has no type system. It provides no mechanism for reuse, short of anchors and aliases that quickly become unreadable. It silently coerces values, the notorious NO becoming false, 1.0 becoming a float, and on becoming true, in ways that require memorization rather than reasoning.

Engineering teams have tolerated these limitations because the alternatives were worse, or at least less familiar. HCL is powerful but Terraform-specific. CUE is expressive but has a steep learning curve. Dhall is principled but alien to most developers. Jsonnet is underused and under-tooled.

Then Apple published Pkl (pronounced "pickle"), an open-source configuration language designed explicitly for the problems YAML fails to solve, type safety, reuse, abstraction, and validation built into the language rather than bolted on afterward.

After one month of using Pkl across real infrastructure configuration, Kubernetes manifests, application configs, and CI/CD templates, here is an honest account of what it does well, where it still struggles, and whether it is ready to replace YAML in a production environment.

What Is Pkl?

Pkl (Package Konfig Language) is a configuration language developed by Apple and open-sourced in February 2024. It compiles to JSON, YAML, XML, or plist, meaning it integrates with any toolchain that reads those formats, without requiring the entire ecosystem to adopt Pkl natively.

Its core design goals:

Type safety - every value has a type, enforced at evaluation time
Reuse through templates - configurations extend and override base templates
Validation built in - constraints are expressed in the language, not in external schema validators
Readability - syntax designed to be familiar to developers without YAML's whitespace traps
Evaluation, not just declaration - expressions, conditionals, and computed values are first-class

The compiler outputs are standard formats, so adopting Pkl is a source change, not an output change. Your Kubernetes cluster still receives YAML. Your application still reads JSON. Pkl lives in your repository and your CI pipeline, not in your runtime.

Syntax First: What Pkl Actually Looks Like

The fastest way to understand Pkl is to see it alongside the YAML it replaces.

Basic Value Declaration

# YAML
app:
  name: orders-service
  port: 3000
  debug: false
  replicas: 3

# Pkl
app {
  name = "orders-service"
  port = 3000
  debug = false
  replicas = 3
}

The structural difference is immediately visible: Pkl uses braces instead of indentation, = for assignment instead of :, and quoted strings that are unambiguously strings, "false" is a string, false is a boolean. There is no YAML parser needed to figure out which is which.

Type Annotations

// Pkl - explicit types on every field
class AppConfig {
  name: String
  port: Int(this > 1024 && this < 65535)   // inline constraint
  debug: Boolean
  replicas: Int(this >= 1 && this <= 10)
  environment: "staging" | "production" | "development"  // union type
}

app: AppConfig = new {
  name = "orders-service"
  port = 3000
  debug = false
  replicas = 3
  environment = "production"
}

Setting replicas = 0 against this class fails at evaluation time with a clear error message, not at runtime when Kubernetes tries to create zero pods. Setting environment = "prod" (a typo) fails immediately because the value is not in the union type. This is the type safety YAML schemas attempt to provide through external validators, Pkl builds it directly into the language.

YAML Coercion Problems - Solved

# YAML - notorious coercion traps
enabled: yes        # becomes true
version: 1.0        # becomes float in some parsers
country: NO         # becomes false (Norway's ISO code!)
port: 8080          # integer
tag: "8080"         # string — same value, different type, different behavior

# Pkl - unambiguous
enabled: Boolean = true
version: String = "1.0"
country: String = "NO"
port: Int = 8080
tag: String = "8080"

Every value's type is explicit. The Norway NO incident, a real class of YAML bug that has affected Kubernetes configurations in production, cannot happen in Pkl.

Templates and Inheritance: The Killer Feature

YAML's anchor-and-alias system for reuse looks like this:

# YAML anchors - functional but unreadable at scale
.defaults: &defaults
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"

deployments:
  orders-service:
    <<: *defaults
    replicas: 3
  users-service:
    <<: *defaults
    replicas: 2
    resources:
      limits:
        memory: "1Gi"    # override — but does this merge or replace?

The << merge key behavior is ambiguous in some parsers and was officially deprecated in YAML 1.2. Pkl's approach is unambiguous:

// Pkl - templates with explicit inheritance
class DeploymentDefaults {
  imagePullPolicy: String = "IfNotPresent"
  memoryRequest: String = "256Mi"
  cpuRequest: String = "100m"
  memoryLimit: String = "512Mi"
  cpuLimit: String = "500m"
}

class Deployment extends DeploymentDefaults {
  name: String
  replicas: Int = 1
}

ordersService: Deployment = new {
  name = "orders-service"
  replicas = 3
  // inherits all defaults
}

usersService: Deployment = new {
  name = "users-service"
  replicas = 2
  memoryLimit = "1Gi"    // explicitly override one field, all others inherited
}

The inheritance is explicit, the override is unambiguous, and the IDE can show you exactly which fields are inherited versus overridden, something YAML anchors fundamentally cannot do.

Real-World Usage: Kubernetes Manifests

Here is a production-representative Pkl template for a Kubernetes Deployment, showing how templating applies to the configuration I write most often:

// k8s/templates/deployment.pkl
module deployment

import "package://pkg.pkl-lang.org/pkl-k8s/k8s@1.0.0#/api/apps/v1/Deployment.pkl"

class AppDeployment {
  name: String
  namespace: String = "default"
  image: String
  tag: String
  replicas: Int(this >= 1) = 1
  port: Int = 8080
  cpuRequest: String = "100m"
  memoryRequest: String = "256Mi"
  cpuLimit: String = "500m"
  memoryLimit: String = "512Mi"
  env: Listing<EnvVar> = new {}
}

class EnvVar {
  name: String
  value: String
}

function toK8sDeployment(config: AppDeployment): Deployment = new {
  metadata {
    name = config.name
    namespace = config.namespace
    labels { app = config.name }
  }
  spec {
    replicas = config.replicas
    selector { matchLabels { app = config.name } }
    template {
      metadata { labels { app = config.name } }
      spec {
        containers {
          new {
            name = config.name
            image = "\(config.image):\(config.tag)"
            ports { new { containerPort = config.port } }
            resources {
              requests {
                cpu = config.cpuRequest
                memory = config.memoryRequest
              }
              limits {
                cpu = config.cpuLimit
                memory = config.memoryLimit
              }
            }
            envFrom = config.env
          }
        }
      }
    }
  }
}

// k8s/services/orders-service.pkl
import "../templates/deployment.pkl"

output {
  value = deployment.toK8sDeployment(new deployment.AppDeployment {
    name = "orders-service"
    namespace = "production"
    image = "your-org/orders-service"
    tag = "2.4.1"
    replicas = 3
    env {
      new { name = "DATABASE_URL"; value = "postgresql://db:5432/orders" }
      new { name = "NODE_ENV"; value = "production" }
    }
  })
}

Compile to YAML for Kubernetes:

pkl eval k8s/services/orders-service.pkl -f yaml > k8s/manifests/orders-service.yaml

The manifest is generated, not handwritten. The template enforces the constraint that replicas is at least 1. The image and tag are separate fields, no string concatenation errors. The output is standard Kubernetes YAML that kubectl apply consumes without modification.

CI/CD Integration

Pkl integrates into CI/CD pipelines anywhere you currently generate or validate YAML:

# .github/workflows/pkl-generate.yml
name: Generate and Validate Configs

on: [push, pull_request]

jobs:
  pkl-generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Pkl CLI
        run: |
          curl -L https://github.com/apple/pkl/releases/latest/download/pkl-linux-amd64 \
            -o /usr/local/bin/pkl
          chmod +x /usr/local/bin/pkl

      - name: Validate all Pkl configurations
        run: pkl eval --multiple-file-output /dev/null pkl/**/*.pkl

      - name: Generate Kubernetes manifests
        run: |
          for file in k8s/services/*.pkl; do
            service=$(basename "$file" .pkl)
            pkl eval "$file" -f yaml > "k8s/manifests/${service}.yaml"
          done

      - name: Commit generated manifests
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add k8s/manifests/
          git diff --cached --quiet || git commit -m "chore: regenerate manifests from Pkl"
          git push

Honest Assessment After One Month

What Pkl Gets Right

Type safety is genuinely useful. Finding configuration errors at compile time rather than at kubectl apply time, or worse, at runtime, is a real quality improvement. Inline constraints like Int(this > 0) replace entire schema validation workflows.

Templates and inheritance are the best solution I've seen. The YAML anchor problem is real. Pkl's class-based inheritance with explicit overrides is significantly cleaner than any YAML reuse mechanism.

String interpolation without escaping traps. "\(config.image):\(config.tag)" is readable and unambiguous. YAML multiline string handling is notoriously confusing.

Evaluation-time computation. Generating environment-specific configuration from a single template with conditional expressions reduces the number of files to maintain.

Where Pkl Still Struggles

Ecosystem maturity is early. The Kubernetes package library (pkl-k8s) covers most resources but not all. For anything not in the package, you're writing raw mappings to target types, which works but loses the ergonomic advantage.

IDE support is improving but incomplete. The VS Code extension provides syntax highlighting and basic completion. IntelliJ support is more mature. Neither yet matches the experience of working in a fully-supported language.

Team adoption friction. Pkl is not YAML. Developers familiar with YAML need to learn new syntax, new mental models for inheritance, and a new compilation step. This is a real cost that requires team buy-in.

Debugging compilation errors. When a constraint fails or a type mismatch occurs in a deeply nested template, the error messages are sometimes less precise than you'd want. This is improving with each release.

Not every toolchain accepts Pkl natively. Tools that consume configuration at runtime, some CI/CD platforms, older infrastructure tools, can't read Pkl directly. The compilation step to YAML/JSON is necessary, which adds pipeline complexity.

Should You Adopt Pkl?

Adopt Pkl today if:

You maintain large, complex YAML configurations with significant duplication
Type errors and coercion bugs have caused real incidents in your environment
Your team writes configuration as code and values IDE support
You're already compiling or generating YAML (Helm charts, Kustomize overlays), Pkl replaces the templating layer cleanly

Wait on Pkl if:

Your YAML configs are small and simple, the overhead isn't justified
Your team doesn't have bandwidth for adoption friction right now
You need native tool support without a compilation step
You're evaluating for a production migration within the next quarter, the ecosystem is maturing fast, but not yet complete

Conclusion

After one month, Pkl has earned a permanent place in my infrastructure toolchain, but not as a wholesale YAML replacement, yet. It is the right tool for complex, template-heavy configuration that benefits from type safety and reuse. It is not yet the obvious choice for simple, stable configuration that existing YAML handles without friction.

The trajectory is clear, Pkl is solving real problems that YAML ignores and doing so with a thoughtfulness that most configuration languages lack. The ecosystem will catch up. The IDE tooling will improve. Team familiarity will build.

YAML isn't dead. But for the first time in years, there's a credible successor that makes the case on engineering merit rather than novelty.

Have you tried Pkl in your infrastructure? What was your first-month experience, similar frustrations, different use cases? Share in the comments.

Pkl #YAML #ConfigurationAsCode #DevOps #InfrastructureAsCode

Green DevOps: How to Optimize Your Cloud Carbon Footprint Using Carbon-Aware SDKs

ThankGod Chibugwum Obobo — Sun, 09 Aug 2026 17:00:00 +0000

Cloud computing accounts for approximately 2–3% of global electricity consumption a figure that rivals the aviation industry and continues to grow as engineering teams scale infrastructure to meet demand. For most of that growth, carbon impact has been invisible, infrastructure decisions are made on cost, latency, and availability, with sustainability treated as someone else's problem.

Green DevOps changes the accounting. It integrates carbon awareness directly into the engineering decision loop, workload scheduling, region selection, CI/CD pipeline timing, and autoscaling policies, making the carbon cost of infrastructure decisions as visible and actionable as the financial cost.

The Carbon-Aware SDK, developed by the Green Software Foundation, provides the tooling layer that makes this practical. A standardized API for querying real-time and forecast carbon intensity data, enabling your applications and pipelines to make time and location-based decisions that minimize carbon emissions without sacrificing availability or performance.

This guide covers how to integrate carbon awareness into your cloud infrastructure and DevOps workflows, from SDK setup and workload time-shifting to Kubernetes scheduling policies and measurable sustainability metrics.

Understanding Carbon Intensity

Before implementing carbon-aware systems, understand what you're measuring:

Carbon intensity measures the amount of CO₂ equivalent (gCO₂eq) emitted per kilowatt-hour of electricity consumed. It varies dramatically by:

Location: A data center in Norway running on hydroelectric power has a carbon intensity of ~20 gCO₂eq/kWh. The same workload in a coal-heavy grid region might emit 700+ gCO₂eq/kWh.
Time: Renewable energy sources like solar and wind are intermittent. Carbon intensity on the same grid can vary by 3–5x between peak solar hours and overnight periods when fossil fuel plants pick up demand.

The two levers carbon-aware systems pull are:

Spatial shifting, running workloads in the region with the lowest current carbon intensity, where latency requirements permit.

Temporal shifting, delaying non-time-sensitive workloads (batch jobs, CI pipelines, data exports, model training) until carbon intensity in your region is lower, typically during high renewable generation periods.

The Carbon-Aware SDK

The Carbon-Aware SDK (Green Software Foundation) provides a unified interface to multiple carbon intensity data providers, Electricity Maps, WattTime, and the UK National Grid ESO, through a consistent REST API and CLI.

Installation and Setup

# Clone and run the Carbon-Aware SDK as a local API server
git clone https://github.com/Green-Software-Foundation/carbon-aware-sdk.git
cd carbon-aware-sdk/src/CarbonAware.WebApi

# Configure your data provider
cat > appsettings.json << EOF
{
  "DataSources": {
    "EmissionsDataSource": "ElectricityMaps",
    "ForecastDataSource": "ElectricityMaps"
  },
  "ElectricityMaps": {
    "APITokenHeader": "auth-token",
    "APIToken": "${ELECTRICITY_MAPS_API_TOKEN}",
    "BaseURL": "https://api.electricitymap.org/v3/"
  }
}
EOF

dotnet run
# API now running at http://localhost:5073

Alternatively, deploy as a container alongside your infrastructure:

# docker-compose.yml
services:
  carbon-aware-api:
    image: ghcr.io/green-software-foundation/carbon-aware-sdk:latest
    ports:
      - "5073:5073"
    environment:
      DataSources__EmissionsDataSource: "ElectricityMaps"
      DataSources__ForecastDataSource: "ElectricityMaps"
      ElectricityMaps__APIToken: ${ELECTRICITY_MAPS_API_TOKEN}
    restart: unless-stopped

Querying Carbon Intensity

// src/carbon/carbon-aware.client.ts
export class CarbonAwareClient {
  private readonly baseUrl: string;

  constructor(baseUrl = process.env.CARBON_AWARE_API_URL ?? 'http://localhost:5073') {
    this.baseUrl = baseUrl;
  }

  // Get current carbon intensity for a location
  async getCurrentIntensity(location: string): Promise<number> {
    const response = await fetch(
      `${this.baseUrl}/emissions/current?location=${location}`
    );
    const data = await response.json();
    return data[0]?.rating ?? 0; // gCO₂eq/kWh
  }

  // Get forecast to find optimal execution window
  async getForecast(location: string, windowMinutes: number = 360): Promise<CarbonForecast[]> {
    const now = new Date();
    const end = new Date(now.getTime() + windowMinutes * 60 * 1000);

    const response = await fetch(
      `${this.baseUrl}/emissions/forecasts/current?` +
      `location=${location}&` +
      `dataStartAt=${now.toISOString()}&` +
      `dataEndAt=${end.toISOString()}&` +
      `windowSize=60`
    );
    const data = await response.json();
    return data[0]?.optimalDataPoints ?? [];
  }

  // Find the lowest-carbon region from a set of candidates
  async getBestRegion(locations: string[]): Promise<string> {
    const params = locations.map(l => `location=${l}`).join('&');
    const response = await fetch(
      `${this.baseUrl}/emissions/current?${params}`
    );
    const data = await response.json();

    return data.reduce((best: any, current: any) =>
      current.rating < best.rating ? current : best
    ).location;
  }
}

Step 1 - Time-Shifting Batch Workloads

The highest-value application of carbon awareness is temporal shifting of batch workloads, jobs that must complete within a window but don't need to start immediately. Data exports, report generation, ML model training, database maintenance, and archive compression are all candidates.

// src/carbon/workload-scheduler.ts
import { CarbonAwareClient } from './carbon-aware.client';

interface WorkloadConfig {
  location: string;
  maxDelayMinutes: number;       // how long can we wait?
  carbonThreshold: number;       // only run if below this gCO₂eq/kWh
}

export class CarbonAwareScheduler {
  private readonly client: CarbonAwareClient;

  constructor() {
    this.client = new CarbonAwareClient();
  }

  async findOptimalExecutionTime(config: WorkloadConfig): Promise<Date> {
    const forecast = await this.client.getForecast(
      config.location,
      config.maxDelayMinutes
    );

    if (forecast.length === 0) {
      // No forecast data — run now rather than block indefinitely
      return new Date();
    }

    // Find the lowest carbon intensity window within our delay budget
    const optimal = forecast.reduce((best, current) =>
      current.value < best.value ? current : best
    );

    const optimalTime = new Date(optimal.timestamp);
    const maxWaitTime = new Date(Date.now() + config.maxDelayMinutes * 60 * 1000);

    // Don't wait longer than our deadline
    return optimalTime < maxWaitTime ? optimalTime : new Date();
  }

  async shouldRunNow(config: WorkloadConfig): Promise<boolean> {
    const current = await this.client.getCurrentIntensity(config.location);
    return current <= config.carbonThreshold;
  }
}

Integrate the scheduler into your batch job runner:

// src/jobs/data-export.job.ts
import { CarbonAwareScheduler } from '../carbon/workload-scheduler';
import { scheduleJob } from 'node-schedule';

export async function scheduleDataExport() {
  const scheduler = new CarbonAwareScheduler();

  const optimalTime = await scheduler.findOptimalExecutionTime({
    location: 'eastus',
    maxDelayMinutes: 360,     // can delay up to 6 hours
    carbonThreshold: 200,     // prefer below 200 gCO₂eq/kWh
  });

  console.log(`Data export scheduled for: ${optimalTime.toISOString()}`);
  console.log(`Delay: ${Math.round((optimalTime.getTime() - Date.now()) / 60000)} minutes`);

  scheduleJob(optimalTime, async () => {
    console.log('Executing carbon-optimized data export...');
    await runDataExport();
  });
}

Step 2 - Carbon-Aware CI/CD Pipelines

CI/CD pipelines are one of the most consistently overlooked sources of cloud carbon emissions, running on every commit, often around the clock, with flexible timing requirements. A test suite that takes 15 minutes can run at 2am during low-carbon hours just as effectively as at 2pm during peak demand.

# .github/workflows/carbon-aware-ci.yml
name: Carbon-Aware CI Pipeline

on:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 */6 * * *'  # Check every 6 hours for deferred runs

jobs:
  check-carbon:
    name: Carbon Intensity Check
    runs-on: ubuntu-latest
    outputs:
      should-run: ${{ steps.carbon-check.outputs.should-run }}
      intensity: ${{ steps.carbon-check.outputs.intensity }}

    steps:
      - name: Check current carbon intensity
        id: carbon-check
        run: |
          INTENSITY=$(curl -s \
            "https://api.electricitymap.org/v3/carbon-intensity/latest?zone=US-MIDA" \
            -H "auth-token: ${{ secrets.ELECTRICITY_MAPS_TOKEN }}" \
            | jq '.carbonIntensity')

          echo "intensity=$INTENSITY" >> $GITHUB_OUTPUT

          # Run immediately for pushes to main regardless of carbon
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "should-run=true" >> $GITHUB_OUTPUT
            echo "Main branch push — running regardless of carbon intensity ($INTENSITY gCO₂eq/kWh)"
          elif (( $(echo "$INTENSITY < 250" | bc -l) )); then
            echo "should-run=true" >> $GITHUB_OUTPUT
            echo "Carbon intensity acceptable: $INTENSITY gCO₂eq/kWh — running pipeline"
          else
            echo "should-run=false" >> $GITHUB_OUTPUT
            echo "Carbon intensity too high: $INTENSITY gCO₂eq/kWh — deferring pipeline"
          fi

  build-and-test:
    name: Build and Test
    runs-on: ubuntu-latest
    needs: check-carbon
    if: needs.check-carbon.outputs.should-run == 'true'

    steps:
      - uses: actions/checkout@v4

      - name: Log carbon context
        run: |
          echo "Running pipeline at carbon intensity: ${{ needs.check-carbon.outputs.intensity }} gCO₂eq/kWh"

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Build
        run: npm run build

The key design decision: main branch pushes always run, carbon awareness is applied to non-critical developer branch pipelines, not to production deployments where timing is dictated by business requirements, not grid carbon intensity.

Step 3 - Carbon-Aware Kubernetes Scheduling

For Kubernetes workloads with flexible placement, integrate carbon intensity into pod scheduling decisions using a custom scheduler plugin or node labels updated by a controller:

// carbon-node-labeler/src/labeler.ts
// A controller that labels Kubernetes nodes with current carbon intensity zones
import { KubeConfig, CoreV1Api } from '@kubernetes/client-node';
import { CarbonAwareClient } from './carbon-aware.client';

const kc = new KubeConfig();
kc.loadFromDefault();
const k8sApi = kc.makeApiClient(CoreV1Api);
const carbonClient = new CarbonAwareClient();

const REGION_TO_LOCATION: Record<string, string> = {
  'us-east-1':    'eastus',
  'eu-west-1':    'northeurope',
  'ap-southeast-1': 'southeastasia',
};

async function labelNodesWithCarbonZone(): Promise<void> {
  const nodes = await k8sApi.listNode();

  for (const node of nodes.body.items) {
    const region = node.metadata?.labels?.['topology.kubernetes.io/region'];
    if (!region || !REGION_TO_LOCATION[region]) continue;

    const intensity = await carbonClient.getCurrentIntensity(REGION_TO_LOCATION[region]);

    // Classify carbon zone
    const zone = intensity < 150 ? 'green'
               : intensity < 300 ? 'yellow'
               : 'red';

    // Label node with current carbon zone
    await k8sApi.patchNode(node.metadata!.name!, {
      metadata: {
        labels: {
          'carbon.fdn.dev/intensity-zone': zone,
          'carbon.fdn.dev/intensity-value': Math.round(intensity).toString(),
        }
      }
    }, undefined, undefined, undefined, undefined, undefined, {
      headers: { 'Content-Type': 'application/merge-patch+json' }
    });
  }
}

// Run every 15 minutes
setInterval(labelNodesWithCarbonZone, 15 * 60 * 1000);
labelNodesWithCarbonZone();

Use node affinity in batch job manifests to prefer green-zone nodes:

# k8s/batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing-job
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          # Prefer green nodes — schedule here if available
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: In
                    values: ["green"]
            - weight: 50
              preference:
                matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: In
                    values: ["yellow"]

          # Never schedule on red-zone nodes for non-critical jobs
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: NotIn
                    values: ["red"]

      containers:
        - name: data-processor
          image: your-org/data-processor:latest

Step 4 - Spatial Shifting for Multi-Region Workloads

For workloads without strict data residency requirements, route to the lowest-carbon region at scheduling time:

// src/carbon/region-selector.ts
export class CarbonAwareRegionSelector {
  private readonly client: CarbonAwareClient;

  // Candidate regions for this workload
  private readonly candidateRegions = [
    { id: 'eastus',         name: 'us-east-1' },
    { id: 'northeurope',    name: 'eu-west-1' },
    { id: 'westus2',        name: 'us-west-2' },
    { id: 'australiaeast',  name: 'ap-southeast-2' },
  ];

  async selectOptimalRegion(): Promise<string> {
    const bestLocation = await this.client.getBestRegion(
      this.candidateRegions.map(r => r.id)
    );

    const region = this.candidateRegions.find(r => r.id === bestLocation);
    console.log(`Carbon-optimal region selected: ${region?.name} (${bestLocation})`);
    return region?.name ?? 'us-east-1';
  }
}

// Use in your infrastructure provisioning or job dispatch
const selector = new CarbonAwareRegionSelector();
const region = await selector.selectOptimalRegion();
await dispatchBatchJob({ region, jobConfig });

Step 5 - Measuring and Reporting Carbon Impact

Carbon awareness only delivers accountability if emissions are measured and reported. The Cloud Carbon Footprint open-source tool aggregates billing data from AWS, GCP, and Azure into CO₂ estimates:

# Install Cloud Carbon Footprint
npx create-cloud-carbon-footprint-app@latest

# Configure AWS credentials
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_DEFAULT_REGION=us-east-1

# Run the estimation
npx ts-node packages/cli/src/CreateLookupTableAndEstimate.ts

Build carbon metrics into your observability stack:

// src/carbon/carbon-metrics.ts
import { MeterProvider } from '@opentelemetry/sdk-metrics';

export class CarbonMetricsReporter {
  private readonly meter = new MeterProvider().getMeter('carbon-metrics');

  private readonly intensityGauge = this.meter.createObservableGauge(
    'carbon.grid.intensity',
    { description: 'Current grid carbon intensity (gCO₂eq/kWh)', unit: 'gCO2eq/kWh' }
  );

  private readonly emissionsCounter = this.meter.createCounter(
    'carbon.workload.emissions',
    { description: 'Estimated carbon emitted by workloads (gCO₂eq)', unit: 'gCO2eq' }
  );

  async recordWorkloadEmissions(
    workloadName: string,
    durationHours: number,
    powerWatts: number,
    location: string,
  ): Promise<void> {
    const intensity = await new CarbonAwareClient().getCurrentIntensity(location);

    // Emissions (gCO₂eq) = Power (kW) × Duration (h) × Intensity (gCO₂eq/kWh)
    const emissionsGrams = (powerWatts / 1000) * durationHours * intensity;

    this.emissionsCounter.add(emissionsGrams, {
      workload: workloadName,
      location,
    });

    console.log(`${workloadName}: estimated ${emissionsGrams.toFixed(2)} gCO₂eq`);
  }
}

Surface carbon metrics in Grafana alongside your standard engineering dashboards:

# Average carbon intensity by region over the last 24 hours
avg_over_time(carbon_grid_intensity{location="eastus"}[24h])

# Total estimated emissions from CI pipelines this week
sum(increase(carbon_workload_emissions{workload="ci-pipeline"}[7d]))

# Carbon savings from time-shifted jobs vs immediate execution baseline
(carbon_baseline_emissions - carbon_actual_emissions) / carbon_baseline_emissions * 100

Step 6 - Green DevOps Principles Checklist

Beyond tooling, Green DevOps is a set of engineering practices:

Right-size before you optimize carbon. An oversized VM running at 5% CPU is wasting energy regardless of grid carbon intensity. Right-size your infrastructure first, carbon optimization on already-efficient infrastructure has a multiplied impact.

Prefer managed services. Cloud providers' managed services (RDS, Fargate, Cloud Functions) are typically more energy-efficient than self-managed equivalents at the same compute capacity, cloud hyperscalers achieve PUE (Power Usage Effectiveness) ratios that on-premise or self-managed rarely match.

Implement autoscaling aggressively. Scale to zero during off-hours for non-production environments. A staging environment running 24/7 at minimum capacity consumes energy around the clock, including during high-carbon grid periods.

Choose renewable-powered regions. AWS regions like eu-north-1 (Stockholm) and us-gov-west-1 run on high renewable percentages. GCP and Azure publish similar sustainability region data. When latency requirements permit region flexibility, factor carbon into the selection.

Add carbon to your architecture decision records. When making infrastructure decisions, region selection, instance type, deployment frequency, document the carbon trade-offs alongside cost and performance trade-offs. Visibility drives accountability.

Conclusion

Green DevOps is not a compliance exercise or a marketing initiative, it is an engineering discipline that optimizes cloud workloads against a metric that has been invisible in most infrastructure decision-making, carbon emissions.

The Carbon-Aware SDK makes the data available. Time-shifting batch jobs, deferring non-critical CI pipelines, routing spatially flexible workloads to lower-carbon regions, and scheduling Kubernetes batch jobs on green-zone nodes are all achievable with the patterns in this guide, with measurable impact on your infrastructure's carbon footprint.

Start with the highest-leverage intervention, identify your largest batch workloads and shift their execution timing by a few hours based on carbon intensity forecasts. Measure the before-and-after emissions. Once the pattern is proven, expand to CI pipeline scheduling, region selection policies, and Kubernetes node affinity rules.

Sustainable infrastructure is not slower infrastructure. Done right, it is simply infrastructure that knows what time it is, and chooses wisely.

What's your team's current approach to cloud sustainability — are you tracking emissions today? Share your stack and what you've measured in the comments.

The Flutter-to-DevOps Pipeline: How to Automate Play Store Deployments with GitHub Actions

ThankGod Chibugwum Obobo — Sun, 02 Aug 2026 17:00:00 +0000

Mobile app deployment is one of the last frontiers where many engineering teams still rely on manual processes. A developer builds a release locally, signs it on their machine, uploads it through the Play Console UI, and fills out a release form by hand. It works, until it doesn't. The signing keystore lives on one laptop. The Play Console credentials are shared over Slack. The versioning is managed in a spreadsheet.

This is not a deployment pipeline. It is a fragile ritual that couples your release process to a specific person, a specific machine, and a specific sequence of manual steps that nobody has fully documented.

GitHub Actions combined with Fastlane transforms Flutter's release process into a reproducible, auditable, automated pipeline, building, testing, signing, and deploying to the Google Play Store on every merge to your release branch, with no developer manual intervention required.

This guide covers the complete Flutter-to-Play Store pipeline, project configuration, automated testing, keystore management, build versioning, AAB generation, and staged Play Store deployment, all wired into a GitHub Actions workflow that runs without human input.

Pipeline Architecture

Before writing any YAML, understand what the complete pipeline does and in what order:

Push to release branch
           │
           ▼
┌─────────────────────┐
│   1. Setup          │  Install Flutter SDK, Java, dependencies
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   2. Test           │  Unit tests, widget tests, integration tests
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   3. Code Analysis  │  flutter analyze, dart format check
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   4. Version Bump   │  Auto-increment build number from Git
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   5. Build AAB      │  flutter build appbundle --release
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   6. Sign           │  Sign with keystore via Gradle
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   7. Deploy         │  Upload to Play Store internal track via Fastlane
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   8. Notify         │  Slack notification with release summary
└─────────────────────┘

Each stage gates the next, a failed test prevents a build, a failed build prevents a deployment. The pipeline is a sequence of quality gates, not just a delivery mechanism.

Step 1 - Flutter Project Configuration

Ensure your Flutter project is configured for automated builds before touching CI:

# pubspec.yaml - version format: major.minor.patch+buildNumber
name: your_app
version: 1.4.2+87       # +87 is the build number (versionCode in Play Store)

Configure Gradle to read signing credentials from environment variables rather than hardcoded file paths, critical for CI environments where keystores are injected at runtime:

// android/app/build.gradle
def keystoreProperties = new Properties()
def keystorePropertiesFile = rootProject.file('key.properties')

// Load from file if it exists (local development)
// Otherwise read from environment variables (CI)
if (keystorePropertiesFile.exists()) {
  keystoreProperties.load(new FileInputStream(keystorePropertiesFile))
}

android {
  signingConfigs {
    release {
      keyAlias     keystoreProperties['keyAlias']     ?: System.getenv('KEY_ALIAS')
      keyPassword  keystoreProperties['keyPassword']  ?: System.getenv('KEY_PASSWORD')
      storeFile    keystoreProperties['storeFile']    ? file(keystoreProperties['storeFile'])
                                                      : file(System.getenv('KEY_STORE_PATH') ?: 'keystore.jks')
      storePassword keystoreProperties['storePassword'] ?: System.getenv('KEY_STORE_PASSWORD')
    }
  }

  buildTypes {
    release {
      signingConfig     signingConfigs.release
      minifyEnabled     true
      shrinkResources   true
      proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }
  }
}

This pattern works identically in local development (reads from key.properties) and CI (reads from environment variables injected by GitHub Actions).

Step 2 - Secrets Setup in GitHub

Before writing the workflow, store all sensitive values as GitHub Actions secrets:

Repository → Settings → Secrets and Variables → Actions

KEYSTORE_BASE64          ← Base64-encoded keystore file
KEY_ALIAS                ← Key alias within the keystore
KEY_PASSWORD             ← Key password
KEY_STORE_PASSWORD       ← Keystore password
PLAY_STORE_JSON_KEY      ← Google Play service account JSON
SLACK_WEBHOOK_URL        ← Slack notification webhook (optional)

Encode your keystore for storage as a GitHub secret:

# On your local machine - encode keystore to base64
base64 -i your-release-keystore.jks | pbcopy   # macOS
base64 -w 0 your-release-keystore.jks | xclip  # Linux

# Paste the output as the KEYSTORE_BASE64 secret value

The base64 encoding is necessary because GitHub Secrets store string values, binary files must be encoded for storage and decoded at runtime.

Step 3 - Google Play Service Account

Fastlane's supply tool authenticates to the Play Store API using a Google service account, not your personal Google account. This is the secure, auditable approach, the service account has only the permissions it needs, and credentials can be rotated independently.

Set up the service account:

Go to Google Play Console → Setup → API access
Link to an existing Google Cloud project or create a new one
Create a service account in Google Cloud IAM with the Editor role on the linked project
Download the service account JSON key file
Grant the service account Release Manager permissions in Play Console
Paste the JSON key contents as the PLAY_STORE_JSON_KEY GitHub secret

Step 4 - Fastlane Configuration

Fastlane handles the Play Store upload. Initialize it in your Flutter project:

# From your Flutter project root
gem install fastlane
cd android
fastlane init

Configure the Fastfile with lanes for each deployment target:

# android/fastlane/Fastfile
default_platform(:android)

platform :android do

  desc "Run all tests"
  lane :test do
    gradle(task: "test")
  end

  desc "Deploy to Play Store internal testing track"
  lane :deploy_internal do
    upload_to_play_store(
      track:              "internal",
      aab:                "../build/app/outputs/bundle/release/app-release.aab",
      json_key_data:      ENV["PLAY_STORE_JSON_KEY"],
      release_status:     "draft",
      skip_upload_apk:    true,
      skip_upload_images: true,
      skip_upload_screenshots: true,
    )
  end

  desc "Promote internal build to beta"
  lane :promote_to_beta do
    upload_to_play_store(
      track:          "internal",
      track_promote_to: "beta",
      json_key_data:  ENV["PLAY_STORE_JSON_KEY"],
      skip_upload_apk: true,
      skip_upload_aab: true,
    )
  end

  desc "Promote beta to production with staged rollout"
  lane :promote_to_production do
    upload_to_play_store(
      track:              "beta",
      track_promote_to:   "production",
      rollout:            "0.1",   # start with 10% rollout
      json_key_data:      ENV["PLAY_STORE_JSON_KEY"],
      skip_upload_apk:    true,
      skip_upload_aab:    true,
    )
  end

end

The three-lane structure mirrors the Play Store track progression, internal -> beta -> production. Each lane is independently callable from GitHub Actions, giving you granular control over the promotion workflow.

Step 5 - The Complete GitHub Actions Workflow

# .github/workflows/flutter-deploy-android.yml
name: Flutter Android Deploy

on:
  push:
    branches: [release]

  # Manual trigger for promoting between tracks
  workflow_dispatch:
    inputs:
      action:
        description: "Deployment action"
        required: true
        type: choice
        options:
          - deploy-internal
          - promote-beta
          - promote-production

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.19.0'
          channel: 'stable'
          cache: true

      - name: Install dependencies
        run: flutter pub get

      - name: Verify formatting
        run: dart format --output=none --set-exit-if-changed .

      - name: Analyze code
        run: flutter analyze --fatal-infos

      - name: Run unit and widget tests
        run: flutter test --coverage

      - name: Upload coverage report
        uses: codecov/codecov-action@v4
        with:
          files: coverage/lcov.info

  build-and-deploy:
    name: Build & Deploy
    runs-on: ubuntu-latest
    needs: test                    # only runs if tests pass
    if: github.event_name == 'push' || github.event.inputs.action == 'deploy-internal'

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0           # full history for build number calculation

      - name: Set up Java
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Set up Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.19.0'
          channel: 'stable'
          cache: true

      - name: Install dependencies
        run: flutter pub get

      - name: Calculate build number from Git commit count
        id: versioning
        run: |
          BUILD_NUMBER=$(git rev-list --count HEAD)
          echo "BUILD_NUMBER=$BUILD_NUMBER" >> $GITHUB_OUTPUT
          echo "Build number: $BUILD_NUMBER"

      - name: Update pubspec version
        run: |
          CURRENT_VERSION=$(grep '^version:' pubspec.yaml | sed 's/version: //' | cut -d'+' -f1)
          NEW_VERSION="${CURRENT_VERSION}+${{ steps.versioning.outputs.BUILD_NUMBER }}"
          sed -i "s/^version: .*/version: ${NEW_VERSION}/" pubspec.yaml
          echo "Version set to: ${NEW_VERSION}"

      - name: Decode keystore from secret
        run: |
          echo "${{ secrets.KEYSTORE_BASE64 }}" | base64 --decode > android/app/keystore.jks

      - name: Build Android App Bundle
        run: |
          flutter build appbundle \
            --release \
            --build-number=${{ steps.versioning.outputs.BUILD_NUMBER }}
        env:
          KEY_ALIAS: ${{ secrets.KEY_ALIAS }}
          KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }}
          KEY_STORE_PASSWORD: ${{ secrets.KEY_STORE_PASSWORD }}
          KEY_STORE_PATH: keystore.jks

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Deploy to Play Store internal track
        working-directory: android
        run: bundle exec fastlane deploy_internal
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

      - name: Clean up keystore
        if: always()            # clean up even if build fails
        run: rm -f android/app/keystore.jks

      - name: Notify Slack on success
        if: success()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "✅ *Flutter Android Deploy* — Build ${{ steps.versioning.outputs.BUILD_NUMBER }} deployed to internal track.\nCommit: `${{ github.sha }}`\nAuthor: ${{ github.actor }}"
            }'

      - name: Notify Slack on failure
        if: failure()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "❌ *Flutter Android Deploy Failed* — Build ${{ steps.versioning.outputs.BUILD_NUMBER }}\nCommit: `${{ github.sha }}`\nAuthor: ${{ github.actor }}\nCheck: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }'

  promote-beta:
    name: Promote to Beta
    runs-on: ubuntu-latest
    if: github.event.inputs.action == 'promote-beta'
    environment: beta-promotion    # requires manual approval gate in GitHub environments

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Promote internal build to beta
        working-directory: android
        run: bundle exec fastlane promote_to_beta
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

  promote-production:
    name: Promote to Production
    runs-on: ubuntu-latest
    if: github.event.inputs.action == 'promote-production'
    environment: production-promotion   # requires mandatory reviewer approval

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Promote beta to production (10% rollout)
        working-directory: android
        run: bundle exec fastlane promote_to_production
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

Step 6 - Build Number Strategy

The git rev-list --count HEAD approach ties the build number to the Git commit count, it's monotonically increasing, reproducible from any clone, and doesn't require external state management:

# Build number = total number of commits on the branch
git rev-list --count HEAD
# Output: 847

# Reproducible - the same commit always produces the same build number
# Monotonically increasing - merges to main always increase the count
# No external state - no counter stored in a database or file

This eliminates the classic CI problem of "where is the build number stored?" the answer is always "in Git history."

Step 7 - Environment Protection Rules for Promotion

GitHub Environments add a required human approval gate before the promote-production job runs:

Repository → Settings → Environments → New Environment

Name: production-promotion
Required Reviewers: [release-managers team]
Wait Timer: 0 minutes
Deployment Branches: release

With this configuration, the promote-production workflow job pauses and sends a notification to the release-managers team requesting approval. The 10% staged rollout in Fastlane gives you a further safety buffer, a bad build affects 10% of users before you manually expand the rollout in Play Console.

Step 8 - Caching for Pipeline Speed

Flutter and Gradle dependencies are large, caching them dramatically reduces pipeline runtime:

# Cache Flutter SDK and pub packages
- name: Set up Flutter
  uses: subosito/flutter-action@v2
  with:
    flutter-version: '3.19.0'
    cache: true   # built-in Flutter cache

# Cache Gradle dependencies separately
- name: Cache Gradle
  uses: actions/cache@v4
  with:
    path: |
      ~/.gradle/caches
      ~/.gradle/wrapper
    key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
    restore-keys: |
      ${{ runner.os }}-gradle-

Common Pitfalls to Avoid

Keystore on disk after workflow completes. The if: always() cleanup step is non-negotiable. A keystore file left in the workspace of a GitHub Actions runner even ephemerally, is a security risk. Always delete it, even on failure.

Using personal Google accounts for Play Store API. Personal account credentials tied to a developer are revoked when that person leaves the organization. Service accounts are credential artifacts owned by the organization, rotate them independently of any individual.

Hardcoding the Flutter version. Pin the Flutter version in your workflow (flutter-version: '3.19.0') and update it deliberately. Floating to latest means a Flutter SDK update can silently break your build.

Skipping the test job on the release branch. The test job is the most important gate in the pipeline, removing it "to speed up releases" eliminates the quality guarantee the pipeline is built to provide.

No staged rollout for production. Deploying directly to 100% of production users with no staged rollout turns every release into a high-stakes gamble. Start at 10%, monitor crash rates and ANR rates in the Play Console, and expand manually when metrics confirm stability.

Conclusion

A Flutter deployment pipeline built on GitHub Actions and Fastlane transforms the release process from a fragile, person-dependent ritual into a reproducible, auditable, automated workflow. Tests gate builds. Builds gate deployments. Deployments require approval before reaching production users. Staged rollouts limit the blast radius of regressions.

Every element of the manual process, the keystore on a developer's laptop, the shared Play Console login, the manual version bump, the upload through the UI, has a secure, automated equivalent in this pipeline. The result is a release process your team can run confidently, repeatedly, and without heroics.

Building for iOS alongside Android? The same GitHub Actions workflow structure applies for App Store Connect deployment via Fastlane's deliver action, with Xcode cloud signing replacing the Android keystore step.

Platform Engineering with Spacelift: Building a Self-Service Infrastructure Platform

ThankGod Chibugwum Obobo — Sun, 26 Jul 2026 17:00:00 +0000

Platform engineering has emerged as the discipline that sits between DevOps tooling and the application teams that consume it. Its goal is deceptively simple, make it easy for developers to do the right thing, provision infrastructure, deploy services, manage environments, without requiring deep infrastructure expertise or opening a ticket to a platform team.

The gap between that goal and reality is often filled by a patchwork of CI/CD pipelines, manual approvals, Slack messages, and Terraform scripts that only two people understand. Developer experience suffers. Compliance and security controls exist only in documentation that nobody reads. Infrastructure sprawls across environments with no consistent enforcement.

Spacelift is a platform engineering tool designed specifically to close this gap, providing a collaborative, policy-driven, GitOps-native control plane for Terraform, OpenTofu, Pulumi, Ansible, and Kubernetes. It sits above your IaC tooling and gives platform teams the controls they need to enforce standards, while giving application teams the self-service access they need to move fast without breaking things.

This guide covers how to build a self-service infrastructure platform with Spacelift, from stack architecture and policy enforcement to drift detection, module registries, and the developer experience patterns that make platform engineering actually deliver on its promise.

What Makes Spacelift Different

Before diving into implementation, it helps to understand what Spacelift provides that raw CI/CD pipelines don't:

Capability	Raw CI/CD (GitHub Actions)	Spacelift
Terraform plan/apply	Manual pipeline config	Native, zero-config
Policy enforcement	Shell scripts or OPA sidecars	Built-in OPA policies (plan, access, trigger)
Drift detection	Custom cron jobs	Native, scheduled
Module registry	Terraform Registry or S3	Built-in, versioned
Stack dependencies	Manual pipeline ordering	Native dependency graph
Self-service provisioning	PR-based or ticket-based	Self-service catalog
Audit trail	CI logs	Dedicated, tamper-evident
Secret management	GitHub Secrets	Native Vault/cloud integration
Cost estimation	None	Infracost integration

The fundamental difference is intent, CI/CD pipelines are general-purpose automation. Spacelift is purpose-built infrastructure orchestration, with the safety, compliance, and developer experience features that platform engineering requires.

Core Concepts

Before building, understand Spacelift's key abstractions:

Stack, the fundamental unit in Spacelift. A stack wraps a single IaC configuration (a Terraform root module, a Pulumi program, an Ansible playbook) with its state backend, environment variables, policies, and deployment settings. One stack equals one deployable infrastructure unit.

Space, a hierarchical namespace for grouping and access-controlling stacks. Spaces mirror your organizational structure: root → platform → team-a, root → platform → team-b. Policies and access rules defined at a parent space are inherited by child spaces.

Policy, an OPA (Open Policy Agent) Rego rule that governs stack behavior. Spacelift ships four policy types: plan (approve or reject based on plan output), access (control who can do what), trigger (control when stacks run), and git push (control which commits trigger runs).

Module, a versioned, reusable Terraform module published to Spacelift's built-in registry. Modules are the mechanism through which platform teams publish approved, tested infrastructure patterns that application teams consume via self-service.

Run, a single execution of a stack's IaC code. Every run is tracked with full input context, plan output, policy decisions, approval state, and apply output, providing an immutable audit trail.

Step 1 - Repository and Stack Architecture

Design your repository structure before creating stacks. A common pattern for multi-team platform engineering:

/infrastructure
  /platform           ← platform team owns this
    /networking
      main.tf         ← VPCs, subnets, TGW
    /security
      main.tf         ← IAM roles, security groups, KMS keys
    /eks
      main.tf         ← shared EKS clusters
  /modules            ← approved module library
    /vpc
    /rds
    /s3-secure
    /eks-nodegroup
  /teams              ← application teams own this (self-service)
    /team-alpha
      /staging
        main.tf
      /production
        main.tf
    /team-beta
      /staging
        main.tf
      /production
        main.tf
  /policies           ← Spacelift policies as code
    plan-enforce-tagging.rego
    plan-cost-guard.rego
    access-team-boundaries.rego
    trigger-dependency-chain.rego

This structure enforces a clear ownership boundary, the platform team manages foundational infrastructure and approved modules. Application teams self-service within /teams/{team-name}, constrained by policies that prevent them from touching what they shouldn't.

Step 2 - Creating and Configuring Stacks

Create stacks programmatically using the Spacelift Terraform provider, treating your platform configuration as code:

# spacelift/stacks/platform-networking.tf
resource "spacelift_stack" "platform_networking" {
  name        = "platform-networking"
  description = "Core VPC and network infrastructure"
  space_id    = spacelift_space.platform.id

  repository   = "your-org/infrastructure"
  branch       = "main"
  project_root = "platform/networking"

  terraform_version = "1.6.0"

  autodeploy         = false    # require explicit approval for platform changes
  protect_from_deletion = true

  labels = ["platform", "networking", "critical"]
}

resource "spacelift_stack" "team_alpha_staging" {
  name        = "team-alpha-staging"
  description = "Team Alpha staging environment"
  space_id    = spacelift_space.team_alpha.id

  repository   = "your-org/infrastructure"
  branch       = "main"
  project_root = "teams/team-alpha/staging"

  terraform_version = "1.6.0"

  autodeploy = true    # staging auto-deploys on merge
  labels     = ["team-alpha", "staging"]
}

Configure environment variables and secret injection per stack:

# Environment variables, non-sensitive configuration
resource "spacelift_environment_variable" "team_alpha_region" {
  stack_id  = spacelift_stack.team_alpha_staging.id
  name      = "AWS_DEFAULT_REGION"
  value     = "us-east-1"
  write_only = false
}

# Secret injection from AWS Secrets Manager via Spacelift context
resource "spacelift_context" "aws_credentials" {
  name        = "aws-production-credentials"
  description = "AWS credentials for production deployments"
  space_id    = spacelift_space.platform.id
}

resource "spacelift_context_attachment" "team_alpha_prod" {
  context_id = spacelift_context.aws_credentials.id
  stack_id   = spacelift_stack.team_alpha_staging.id
  priority   = 1
}

Contexts are the Spacelift equivalent of shared environment variable sets, define credentials once at the platform level, attach to any number of stacks, and rotate in one place.

Step 3 - Policies as Code with OPA

Policies are where platform engineering leverage lives. A policy written once enforces a standard across every stack in your platform, no per-team configuration, no tribal knowledge, no exceptions that slip through code review.

Plan Policy - Enforce Resource Tagging

Reject any Terraform plan that creates resources without mandatory tags:

# policies/plan-enforce-tagging.rego
package spacelift

# Required tags for all resources in production
required_tags := {"Environment", "Team", "CostCenter", "ManagedBy"}

# Deny if any resource is missing required tags
deny[sprintf("Resource '%s' is missing required tags: %v", [resource.address, missing])] {
  resource := input.terraform.resource_changes[_]
  resource.change.actions[_] == "create"

  # Identify which required tags are missing
  provided_tags := {tag | resource.change.after.tags[tag]}
  missing := required_tags - provided_tags
  count(missing) > 0
}

# Warn if tags exist but Environment doesn't match a known value
warn[sprintf("Resource '%s' has unrecognized Environment tag: '%s'", [resource.address, env])] {
  resource := input.terraform.resource_changes[_]
  env := resource.change.after.tags.Environment
  not env in {"staging", "production", "development"}
}

Plan Policy - Cost Guard

Reject plans that would exceed a per-run cost threshold:

# policies/plan-cost-guard.rego
package spacelift

# Block plans with monthly cost increase above $500
deny["Monthly cost increase exceeds $500 threshold — escalate to platform team for approval"] {
  input.spacelift.run.cost_estimate.monthly_cost_increase > 500
}

warn["Monthly cost increase above $100 — ensure this is expected"] {
  input.spacelift.run.cost_estimate.monthly_cost_increase > 100
  input.spacelift.run.cost_estimate.monthly_cost_increase <= 500
}

Access Policy - Team Boundary Enforcement

Prevent teams from reading or modifying stacks they don't own:

# policies/access-team-boundaries.rego
package spacelift

# Extract team from the user's IdP groups
user_teams := {team |
  group := input.session.teams[_]
  team := trim_prefix(group, "team-")
}

# Allow access only to stacks labeled with the user's team
allow {
  stack_team := input.stack.labels[_]
  stack_team in user_teams
}

# Platform team members have access to all stacks
allow {
  "platform" in input.session.teams
}

# Block all other access
deny {
  not allow
}

Trigger Policy - Stack Dependency Chains

Automatically trigger downstream stacks when upstream infrastructure changes:

# policies/trigger-dependency-chain.rego
package spacelift

# When platform-networking applies successfully, trigger EKS stack
trigger["platform-eks"] {
  input.run.stack.id == "platform-networking"
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"
}

# When platform-eks applies successfully, trigger team stacks
trigger["team-alpha-staging"] {
  input.run.stack.id == "platform-eks"
  input.run.state == "FINISHED"
}

Attach policies to stacks or spaces:

resource "spacelift_policy" "enforce_tagging" {
  name = "enforce-resource-tagging"
  type = "PLAN"
  body = file("policies/plan-enforce-tagging.rego")
}

# Attach to the entire platform space — all child stacks inherit
resource "spacelift_policy_attachment" "enforce_tagging_platform" {
  policy_id = spacelift_policy.enforce_tagging.id
  stack_id  = "*"
  space_id  = spacelift_space.platform.id
}

Step 4 - Module Registry for Self-Service

The Spacelift module registry is the mechanism through which platform teams publish approved infrastructure patterns for self-service consumption. A module published to the registry is versioned, tested, and policy-compliant by definition, application teams consume it without needing to understand the underlying infrastructure design.

Publish a module to Spacelift's registry:

# spacelift/modules/rds-module.tf
resource "spacelift_module" "rds_postgres" {
  name           = "rds-postgres"
  space_id       = spacelift_space.platform.id
  repository     = "your-org/infrastructure"
  project_root   = "modules/rds"
  branch         = "main"
  description    = "Production-grade PostgreSQL RDS with encryption, backups, and parameter group"

  terraform_provider = "aws"

  labels = ["database", "postgres", "approved"]
}

Application teams consume it in their stacks like any Terraform module:

# teams/team-alpha/staging/main.tf
module "orders_database" {
  source  = "spacelift.io/your-org/rds-postgres/aws"
  version = "~> 2.0"

  identifier        = "orders-staging"
  instance_class    = "db.t3.medium"
  allocated_storage = 20
  environment       = "staging"
  team              = "team-alpha"
}

The version constraint (~> 2.0) means Team Alpha gets patch updates automatically but won't be silently upgraded to a breaking major version. The platform team controls what's published, application teams control what version they pin to.

Step 5 - Drift Detection

Infrastructure drift, the gap between what Terraform's state says exists and what actually exists in your cloud account, is one of the most persistent operational problems in IaC. Manual changes through the AWS console, emergency hotfixes applied directly, or race conditions between concurrent applies all cause drift.

Spacelift's built-in drift detection runs Terraform plan on a schedule and alerts when the live state diverges from the declared state:

resource "spacelift_stack" "platform_networking" {
  # ... other config ...

  # Run drift detection daily at 2am UTC
  drift_detection {
    enabled   = true
    reconcile = false    # alert on drift but don't auto-remediate — require human review
    schedule  = ["0 2 * * *"]
  }
}

For non-critical stacks where auto-remediation is safe:

drift_detection {
  enabled   = true
  reconcile = true     # automatically apply to reconcile drift
  schedule  = ["0 */6 * * *"]  # every 6 hours
}

Configure Slack or PagerDuty notifications for drift alerts:

resource "spacelift_webhook" "drift_alert" {
  stack_id = spacelift_stack.platform_networking.id
  endpoint = var.slack_webhook_url
  enabled  = true

  # Trigger on drift detection and failed reconciliation
  secret = var.webhook_secret
}

Step 6 - Self-Service Developer Experience

The final layer is the developer experience that makes self-service actually self-service. Spacelift's self-service catalog (Spaces + module registry + approval workflows) lets application teams provision infrastructure through a UI or API without opening tickets:

For application teams: a curated list of available modules (RDS, S3, SQS, ElastiCache) with a form-based provisioning interface, no Terraform knowledge required.

For platform teams: full visibility into what every team has provisioned, policy enforcement that fires automatically, drift detection that runs continuously, and an audit trail that captures every change with the identity of who made it and what the plan output was.

Connect Spacelift to your IdP (Okta, GitHub, Google) to enforce team boundaries automatically via SSO group membership, the access policies defined in Step 3 apply automatically as team membership changes.

Step 7 - CI/CD Integration

Spacelift integrates with GitHub, GitLab, Bitbucket, and Azure DevOps for Git-driven runs. A push to main in the infrastructure repo triggers plan runs on affected stacks automatically:

# .github/workflows/spacelift-preview.yml
name: Spacelift Plan Preview

on: [pull_request]

jobs:
  plan-preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Comment plan output on PR
        uses: spacelift-io/spacelift-pull-request-action@v1
        with:
          spacelift-api-key-id: ${{ secrets.SPACELIFT_API_KEY_ID }}
          spacelift-api-key-secret: ${{ secrets.SPACELIFT_API_KEY_SECRET }}

For stacks with autodeploy: false, Spacelift posts the plan output as a PR comment, requiring a human to review and approve before the apply runs. For stacks with autodeploy: true (staging environments), the apply runs automatically on merge to main.

Common Pitfalls to Avoid

Flat space hierarchy. A single space for all stacks gives no access boundary between teams. Design your space hierarchy to mirror your team structure from day one, retrofitting access controls onto a flat hierarchy is painful.

Policies without testing. A deny policy that fires incorrectly can block all deployments across your entire platform. Test policies against representative plan inputs using OPA's opa eval before attaching them to production stacks.

Auto-reconciling critical infrastructure. Drift auto-remediation is powerful but dangerous for foundational infrastructure. Networking, security groups, and IAM roles should alert on drift and require human review, not automatically apply. Reserve auto-reconciliation for stateless, idempotent resource sets.

No module versioning discipline. Publishing a breaking change to a module and expecting all consumers to upgrade simultaneously doesn't scale. Adopt semantic versioning for modules and maintain at least one prior major version while teams migrate.

Conclusion

Platform engineering is not about giving developers access to infrastructure tooling, it's about designing guardrails and self-service patterns so that developers can provision what they need safely, consistently, and without creating operational debt for the platform team.

Spacelift makes this achievable by providing a purpose-built control plane for IaC, policy enforcement that fires at plan time, a module registry that codifies approved patterns, drift detection that catches manual changes, and a space hierarchy that enforces team boundaries without manual intervention.

The platform team's job evolves from running infrastructure to building the platform that lets others run theirs safely, at scale, and with a complete audit trail.

Running OpenTofu instead of Terraform, or mixing Pulumi with Ansible? Spacelift's multi-IaC support handles heterogeneous toolchains in the same platform all under the same policy and audit model. Drop your stack in the comments.

PlatformEngineering #Spacelift #Terraform #InfrastructureAsCode #DevOps

Implementing mTLS in Kubernetes: How to Secure Service-to-Service Communication

ThankGod Chibugwum Obobo — Sun, 19 Jul 2026 17:00:00 +0000

Introduction

In a microservices architecture running on Kubernetes, services communicate constantly, the orders-service calls the users-service, the payments-service calls the fraud-detection-service, and the API gateway routes to a dozen downstream services. By default, most of this traffic travels unencrypted inside the cluster, authenticated only by the fact that the caller has network access.

That assumption that anything inside the cluster boundary is trustworthy, is the network-level equivalent of leaving every internal door unlocked because the building has a lock on the front entrance. A single compromised pod, a misconfigured network policy, or a lateral movement attack can reach any service in the cluster with no additional authentication required.

Mutual TLS (mTLS) closes this gap. Unlike standard TLS where only the server presents a certificate, mTLS requires both parties, client and server, to present and verify certificates before any data is exchanged. Every service-to-service connection is cryptographically authenticated. A compromised pod cannot impersonate payments-service to fraud-detection-service because it cannot produce a valid certificate for that identity.

This guide covers how to implement mTLS in Kubernetes using a service mesh, comparing Istio and Linkerd, configuring automatic certificate issuance, enforcing strict mTLS policies, and validating that your network is actually secured.

How mTLS Works in Kubernetes

Standard TLS in a web context works like this: the client (browser) verifies the server's certificate, establishing that the server is who it claims to be, but the server accepts any client. mTLS adds the reverse, the server also verifies the client's certificate, establishing mutual identity before the connection proceeds.

In Kubernetes, the mechanics are:

Each service is issued a certificate tied to its SPIFFE identity, a URI in the format spiffe://cluster.local/ns/{namespace}/sa/{service-account}.
When orders-service connects to users-service, both services present their certificates.
The receiving service verifies the caller's certificate against the cluster's trust anchor (a root CA).
If verification passes, the connection proceeds. If not, wrong identity, expired certificate, or untrusted issuer, the connection is rejected at the TLS layer, before any application code runs.

The critical operational property: certificates are automatically rotated by the service mesh control plane, typically every 24 hours. No manual certificate management, no expiry incidents, no long-lived credentials.

Choosing a Service Mesh: Istio vs. Linkerd

Both Istio and Linkerd implement mTLS via sidecar proxies injected into each pod. The proxies intercept all inbound and outbound traffic, transparently handling TLS without requiring application code changes.

Feature	Istio	Linkerd
mTLS implementation	Envoy sidecar	Linkerd2-proxy sidecar
Certificate authority	Istiod (built-in) or external	Built-in or cert-manager
mTLS enforcement	PeerAuthentication policy	Server policy
Observability	Rich (Kiali, Jaeger integration)	Lightweight (built-in dashboard)
Resource overhead	Higher (~200–350MB per sidecar)	Lower (~10–20MB per sidecar)
Configuration complexity	Higher	Lower
FIPS 140-2 compliance	Enterprise	Enterprise
Multi-cluster support	Native	Native

Choose Istio if you need advanced traffic management (canary deployments, fault injection, circuit breaking), rich observability, or WebAssembly-based policy extensibility.

Choose Linkerd if you prioritize simplicity, low resource overhead, and want mTLS with strong defaults out of the box with minimal configuration.

This guide covers both installation and policy configuration for each.

Approach 1 - mTLS with Istio

Step 1 - Install Istio

Install Istio using the istioctl CLI with the minimal production profile:

# Install istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-*/bin:$PATH

# Install Istio with production profile
istioctl install --set profile=default \
  --set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true \
  -y

# Verify installation
istioctl verify-install

Step 2 - Enable Automatic Sidecar Injection

Label namespaces where you want mTLS enforced. Istio automatically injects the Envoy sidecar proxy into every pod in labeled namespaces:

# Enable sidecar injection for your application namespaces
kubectl label namespace orders-service istio-injection=enabled
kubectl label namespace users-service istio-injection=enabled
kubectl label namespace payments-service istio-injection=enabled

# Verify injection is active
kubectl get namespace -L istio-injection

Rolling restart existing deployments to pick up sidecar injection:

kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service

Verify sidecars are injected, each pod should now show 2/2 containers (application + Envoy proxy):

kubectl get pods -n orders-service
# NAME                             READY   STATUS    RESTARTS   AGE
# orders-api-7d8b9f5c4-xk2pq      2/2     Running   0          45s

Step 3 - Configure PeerAuthentication for Strict mTLS

By default, Istio operates in permissive mode, it accepts both plaintext and mTLS traffic. This is safe for migration but provides no security guarantee. Switch to strict mode to enforce mTLS and reject all plaintext traffic:

# istio/peer-authentication-strict.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system   # mesh-wide policy
spec:
  mtls:
    mode: STRICT

kubectl apply -f istio/peer-authentication-strict.yaml

This single resource enforces mTLS for all service-to-service communication across the entire mesh. Any pod that attempts to connect to another pod without a valid certificate, including debug containers, legacy services without sidecars, or external probes, will have their connection rejected.

Step 4 - AuthorizationPolicy for Fine-Grained Access Control

mTLS establishes who a caller is. AuthorizationPolicy controls what authenticated identities are allowed to do, combining mTLS identity with RBAC-style rules:

# istio/authorization-payments.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payments-service-policy
  namespace: payments-service
spec:
  selector:
    matchLabels:
      app: payments-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              # Only orders-service can call payments-service
              - "cluster.local/ns/orders-service/sa/orders-service-sa"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/payments/charge", "/payments/refund"]

# Deny all traffic not explicitly allowed, defense in depth
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: payments-service
spec:
  {}  # Empty spec = deny all by default

Apply deny-all first, then layer specific ALLOW policies on top. This is the zero-trust principle applied at the service level: every inter-service call must be explicitly authorized.

Step 5 - Verify mTLS is Active with Kiali

Kiali, Istio's observability console, provides a visual confirmation of mTLS status:

# Install Kiali and supporting tools
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml

# Access Kiali dashboard
istioctl dashboard kiali

In Kiali's service graph, a padlock icon on each edge confirms mTLS is active on that connection. Missing padlocks indicate a service communicating without mTLS, a misconfiguration or missing sidecar.

Approach 2 - mTLS with Linkerd

Step 1 - Install Linkerd

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Validate cluster compatibility
linkerd check --pre

# Install Linkerd control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify installation
linkerd check

Step 2 - Inject Linkerd Proxy into Namespaces

# Annotate namespaces for automatic injection
kubectl annotate namespace orders-service linkerd.io/inject=enabled
kubectl annotate namespace users-service linkerd.io/inject=enabled
kubectl annotate namespace payments-service linkerd.io/inject=enabled

# Restart deployments to pick up proxy injection
kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service

Step 3 - Enforce mTLS with Server Policies

Linkerd enables mTLS by default in injected namespaces. To enforce authenticated traffic only (equivalent to Istio's strict mode), configure Server policies:

# linkerd/server-policy-payments.yaml
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: payments-server
  namespace: payments-service
spec:
  podSelector:
    matchLabels:
      app: payments-service
  port: 8080
  proxyProtocol: HTTP/2


apiVersion: policy.linkerd.io/v1beta2
kind: ServerAuthorization
metadata:
  name: payments-authz
  namespace: payments-service
spec:
  server:
    name: payments-server
  client:
    meshTLS:
      serviceAccounts:
        - name: orders-service-sa
          namespace: orders-service

kubectl apply -f linkerd/server-policy-payments.yaml

# Verify mTLS is active on connections
linkerd viz edges deployment -n orders-service
# Shows which connections are secured with mTLS (✓) vs plaintext (✗)

Certificate Management with cert-manager

Both Istio and Linkerd can integrate with cert-manager for certificate issuance, enabling integration with external CAs (HashiCorp Vault, AWS ACM PCA, Let's Encrypt) rather than relying solely on the mesh's built-in CA.

# cert-manager/cluster-issuer-vault.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.internal:8200
    path: pki/sign/kubernetes-mesh
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager
        secretRef:
          name: cert-manager-vault-token
          key: token

For Istio, plug the external CA into the control plane:

# Use cert-manager as Istio's certificate provider
istioctl install --set profile=default \
  --set values.pilot.env.EXTERNAL_CA=true \
  --set values.pilot.env.K8S_SIGNER=cert-manager.io/cluster-issuer/vault-issuer

External CA integration provides:

Centralized audit trail of every certificate issuance across your mesh
Cross-cluster trust using a shared root CA for multi-cluster mTLS
Compliance alignment with PKI policies already in use in your organization

Migrating to Strict mTLS Without Downtime

Flipping directly to strict mode on a production cluster breaks any service that doesn't yet have a sidecar. Use a phased migration:

Phase 1 - Permissive mode (baseline). Install the mesh with mTLS in permissive mode. All traffic is accepted, both plaintext and mTLS. Use Kiali or linkerd viz to identify which connections are NOT using mTLS.

Phase 2 - Namespace-by-namespace injection. Enable sidecar injection one namespace at a time. Verify each namespace's workloads after injection before proceeding.

Phase 3 - Strict mode per namespace. Apply PeerAuthentication or Server policies with strict enforcement to already-injected namespaces:

# Namespace-scoped strict policy (safer than mesh-wide during migration)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: payments-service   # scoped to one namespace
spec:
  mtls:
    mode: STRICT

Phase 4 - Mesh-wide strict enforcement. Once all namespaces are injected and verified, apply the mesh-wide PeerAuthentication in istio-system and remove namespace-scoped policies.

This phased approach takes longer but eliminates the risk of a mesh-wide strict enforcement breaking services that weren't ready.

Validating mTLS Enforcement

After enabling strict mode, validate it is actually enforced, don't assume:

# Attempt to connect to payments-service from a pod WITHOUT a sidecar
kubectl run debug-pod --image=curlimages/curl --restart=Never -- \
  curl -s http://payments-service.payments-service.svc.cluster.local:8080/health

# Expected result: connection refused or reset, mTLS is enforced
# Unexpected result: 200 OK, mTLS is NOT enforced, investigate

# Verify certificate details on an active connection
kubectl exec -n orders-service deployment/orders-api -c istio-proxy -- \
  openssl s_client \
    -connect payments-service.payments-service.svc.cluster.local:8080 \
    -showcerts 2>/dev/null | grep -A2 "subject="

# Expected output shows SPIFFE URI:
# subject=URI:spiffe://cluster.local/ns/payments-service/sa/payments-service-sa

# Check Istio's view of mTLS status per service
istioctl x describe service payments-service.payments-service

# Output confirms:
# mTLS: yes
# Policy: STRICT

Observability and Troubleshooting

mTLS adds authentication at the network layer, but it also adds debugging complexity. Keep these tools in your operational toolkit:

Istio proxy logs reveal TLS handshake failures with specific error codes:

# View Envoy access logs for a specific pod
kubectl logs -n orders-service deployment/orders-api -c istio-proxy | \
  grep -E "UF|UC|UH"  # Upstream connection failure codes

Linkerd tap provides real-time traffic inspection with mTLS status:

linkerd viz tap deployment/orders-api -n orders-service \
  --to deployment/payments-service -n payments-service

Certificate expiry monitoring even with automatic rotation, monitor certificate validity as a health signal:

# Prometheus alert for certificates expiring within 24 hours
- alert: MeshCertificateExpiringSoon
  expr: |
    (citadel_server_csr_count - citadel_server_success_cert_issuance_count) > 0
  for: 5m
  labels:
    severity: high
  annotations:
    summary: "Certificate issuance failures in Istio CA"

Common Pitfalls to Avoid

Enabling strict mode before all services have sidecars. Services without sidecars cannot participate in mTLS and will have their connections rejected in strict mode. Always verify 100% sidecar coverage before enabling mesh-wide strict enforcement.

Forgetting health check probes. Kubernetes liveness and readiness probes originate from the kubelet, outside the mesh. Configure Istio to rewrite probe URLs or exclude probe ports from mTLS enforcement:

# Exclude health check port from mTLS, kubelet doesn't have a sidecar
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    8081:          # health check port
      mode: DISABLE

Not testing AuthorizationPolicy before applying. A misconfigured deny-all policy can cut off legitimate service traffic in production. Test policies in a staging environment first, and use Istio's AUDIT action mode to log would-be denials before enforcing them.

Ignoring certificate rotation failures. The automatic certificate rotation is the operational backbone of mesh mTLS. Monitor certificate issuance metrics, a rotation failure that goes undetected leads to widespread connection failures when certificates expire.

Conclusion

Mutual TLS in Kubernetes transforms your cluster's internal network from an implicit trust zone into a zero-trust environment where every connection is cryptographically authenticated and explicitly authorized. A compromised pod cannot impersonate a legitimate service. A lateral movement attack cannot freely reach payment processing or user data services. Every inter-service call is auditable.

The operational cost, a service mesh, sidecar overhead, policy management, is real. But for systems where data sensitivity, regulatory requirements, or blast-radius concerns justify it, mTLS via a service mesh is the most complete implementation of zero-trust networking currently available for Kubernetes workloads.

Start with permissive mode, inject sidecars namespace by namespace, validate mTLS coverage with your mesh's observability tools, and graduate to strict enforcement once you're confident in coverage. The security guarantee at the end is worth the careful migration.

Running a multi-cluster Kubernetes setup? Extending mTLS across cluster boundaries with Istio's multi-cluster federation or Linkerd's cluster linking uses the same SPIFFE identity model, but adds a trust bundle exchange step between clusters.

DRY Infrastructure: How to Architect Modular Ansible Playbooks for Next.js

ThankGod Chibugwum Obobo — Sun, 12 Jul 2026 17:00:00 +0000

Infrastructure code suffers from the same antipatterns as application code, duplication, tight coupling, undocumented assumptions, and a growing resistance to change. A playbook written for one environment gets copy-pasted for the next, variables get hardcoded, and six months later you have five slightly different versions of the same deployment logic, each drifting independently and none of them trustworthy.

The DRY (Don't Repeat Yourself) principle applies to infrastructure as directly as it does to software. In Ansible, DRY infrastructure means roles for reusable task logic, variable hierarchies for environment-specific overrides, handlers for event-driven operations, and a playbook composition pattern that assembles production-grade deployments from interchangeable, independently testable units.

For Next.js, with its hybrid rendering modes, standalone server output, static asset optimization, and Node.js runtime requirements, a modular Ansible approach gives you a repeatable, environment-aware provisioning and deployment pipeline that works consistently from local staging to production at scale.

This guide covers how to design and implement a modular Ansible architecture for Next.js from the ground up.

Why Ansible for Next.js Infrastructure

Before diving into structure, it's worth addressing why Ansible fits Next.js provisioning well, especially alongside more modern IaC tools:

Ansible excels at configuration management and application deployment, the layer between raw infrastructure (where Terraform lives) and application orchestration (where Kubernetes lives). For teams running Next.js on VMs, bare metal, or alongside a lightweight container setup, Ansible bridges that gap cleanly.

Specifically for Next.js:

Node.js runtime management (version pinning, nvm, process managers)
Build artifact deployment (transferring .next/standalone output, managing static assets to CDN)
Nginx reverse proxy configuration (SSL termination, caching headers, compression)
Environment variable injection (.env files templated from Vault or group vars)
PM2 or systemd process management (zero-downtime restarts, health checks)
Log rotation and monitoring agent configuration

This is precisely where Ansible's agentless, SSH-based, idempotent task execution shines.

Step 1 - Project Structure for Modular Playbooks

Start with a structure that enforces separation of concerns from day one:

/ansible
  /inventories
    /staging
      hosts.yml
      group_vars/
        all.yml
        nextjs_servers.yml
    /production
      hosts.yml
      group_vars/
        all.yml
        nextjs_servers.yml
  /roles
    /common           ← base system configuration
    /nodejs           ← Node.js runtime installation and management
    /nextjs-app       ← Next.js application deployment
    /nginx            ← Nginx configuration and SSL
    /pm2              ← PM2 process management
    /monitoring       ← monitoring agent setup
    /logrotate        ← log rotation configuration
  /playbooks
    site.yml          ← master playbook (composes all roles)
    provision.yml     ← first-time server provisioning
    deploy.yml        ← application deployment only
    rollback.yml      ← rollback to previous release
  /group_vars
    all.yml           ← variables shared across all environments
  ansible.cfg

This structure reflects three key design decisions:

Separate provisioning from deployment. provision.yml runs once (or rarely) to set up a server. deploy.yml runs on every release. Conflating them means every deployment re-runs expensive provisioning tasks unnecessarily.

Separate inventories per environment. Staging and production are entirely isolated inventory trees, different hosts, different group vars, no risk of accidentally running a production playbook against staging or vice versa.

Roles are the unit of reuse. Every discrete concern is a role. Roles are independently versioned, testable with Molecule, and composable in any playbook.

Step 2 - Ansible Configuration

# ansible.cfg
[defaults]
inventory          = ./inventories
roles_path         = ./roles
host_key_checking  = False
retry_files_enabled = False
stdout_callback    = yaml
interpreter_python = auto_silent
forks              = 10

[ssh_connection]
ssh_args           = -o ControlMaster=auto -o ControlPersist=60s
pipelining         = True   # significant performance improvement

pipelining = True reduces the number of SSH connections per task, one of the most impactful performance improvements for large fleets. It requires requiretty to be disabled in sudoers on your target hosts.

Step 3 - Variable Hierarchy and Environment Overlays

Ansible's variable precedence is the mechanism for DRY environment-specific configuration. Define defaults at the broadest scope, override only what differs per environment:

# group_vars/all.yml - shared across ALL environments
nodejs_version: "20.11.0"
app_user: "nextjs"
app_group: "nextjs"
app_base_dir: "/opt/nextjs"
releases_to_keep: 5
pm2_instances: "max"
nginx_worker_processes: "auto"

# inventories/staging/group_vars/nextjs_servers.yml - staging overrides
app_environment: "staging"
app_port: 3000
next_public_api_url: "https://api.staging.your-domain.com"
pm2_instances: 2                  # fewer instances on smaller staging servers
nginx_ssl_certificate: "/etc/letsencrypt/live/staging.your-domain.com/fullchain.pem"
nginx_ssl_key: "/etc/letsencrypt/live/staging.your-domain.com/privkey.pem"
app_releases_dir: "{{ app_base_dir }}/releases"

# inventories/production/group_vars/nextjs_servers.yml - production overrides
app_environment: "production"
app_port: 3000
next_public_api_url: "https://api.your-domain.com"
pm2_instances: "max"              # use all available CPU cores
nginx_ssl_certificate: "/etc/letsencrypt/live/your-domain.com/fullchain.pem"
nginx_ssl_key: "/etc/letsencrypt/live/your-domain.com/privkey.pem"
app_releases_dir: "{{ app_base_dir }}/releases"

Sensitive values (database URLs, API keys, secrets) should never appear in group vars. Use Ansible Vault for sensitive variable encryption or integrate with HashiCorp Vault for dynamic secret injection, as covered in our guide on replacing hardcoded credentials with Vault and GitHub OIDC.

Step 4 - The Node.js Role

The Node.js role manages runtime installation via nvm, ensuring version pinning and easy upgrades without system package manager conflicts:

roles/nodejs/
  tasks/
    main.yml
  defaults/
    main.yml
  handlers/
    main.yml

# roles/nodejs/defaults/main.yml
nodejs_version: "20.11.0"
nvm_version: "0.39.7"
nvm_dir: "/home/{{ app_user }}/.nvm"
npm_global_packages:
  - pm2
  - typescript

# roles/nodejs/tasks/main.yml
---
- name: Create application user
  ansible.builtin.user:
    name: "{{ app_user }}"
    group: "{{ app_group }}"
    system: true
    shell: /bin/bash
    home: "/home/{{ app_user }}"
    create_home: true
  tags: [nodejs, user]

- name: Check if nvm is installed
  ansible.builtin.stat:
    path: "{{ nvm_dir }}/nvm.sh"
  register: nvm_stat
  tags: [nodejs, nvm]

- name: Install nvm
  ansible.builtin.shell:
    cmd: >
      curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v{{ nvm_version }}/install.sh | bash
  become: true
  become_user: "{{ app_user }}"
  when: not nvm_stat.stat.exists
  tags: [nodejs, nvm]

- name: Install Node.js via nvm
  ansible.builtin.shell:
    cmd: >
      source {{ nvm_dir }}/nvm.sh &&
      nvm install {{ nodejs_version }} &&
      nvm alias default {{ nodejs_version }}
  args:
    executable: /bin/bash
  become: true
  become_user: "{{ app_user }}"
  tags: [nodejs]

- name: Install global npm packages
  ansible.builtin.shell:
    cmd: >
      source {{ nvm_dir }}/nvm.sh &&
      npm install -g {{ item }}
  args:
    executable: /bin/bash
  become: true
  become_user: "{{ app_user }}"
  loop: "{{ npm_global_packages }}"
  tags: [nodejs, npm]

Step 5 - The Next.js Application Role

The Next.js role implements a release directory pattern - the key mechanism for zero-downtime deployments and fast rollbacks:

roles/nextjs-app/
  tasks/
    main.yml
    deploy.yml
    rollback.yml
    cleanup.yml
  templates/
    env.j2            ← .env.local template
    pm2.config.js.j2  ← PM2 ecosystem config template
  handlers/
    main.yml
  defaults/
    main.yml

# roles/nextjs-app/tasks/deploy.yml
- name: Generate release timestamp
  ansible.builtin.set_fact:
    release_timestamp: "{{ ansible_date_time.epoch }}"
  tags: [deploy]

- name: Create release directory
  ansible.builtin.file:
    path: "{{ app_releases_dir }}/{{ release_timestamp }}"
    state: directory
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
    mode: "0755"
  tags: [deploy]

- name: Upload Next.js standalone build
  ansible.builtin.synchronize:
    src: "{{ local_build_path }}/.next/standalone/"
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}"
    delete: true
    rsync_opts:
      - "--exclude=.git"
  tags: [deploy]

- name: Upload static assets
  ansible.builtin.synchronize:
    src: "{{ local_build_path }}/.next/static/"
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}/.next/static/"
  tags: [deploy]

- name: Template environment file
  ansible.builtin.template:
    src: env.j2
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}/.env.local"
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
    mode: "0640"           # restrict read to app user only
  tags: [deploy]

- name: Update current symlink atomically
  ansible.builtin.file:
    src: "{{ app_releases_dir }}/{{ release_timestamp }}"
    dest: "{{ app_base_dir }}/current"
    state: link
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
  notify: Reload PM2
  tags: [deploy]

The atomic symlink swap (current → new release) is the mechanism that makes zero-downtime deployments possible. PM2 reloads against the new current directory; if the reload fails, the symlink points back to the previous release.

# roles/nextjs-app/tasks/rollback.yml
- name: List available releases
  ansible.builtin.find:
    paths: "{{ app_releases_dir }}"
    file_type: directory
  register: available_releases
  tags: [rollback]

- name: Identify previous release
  ansible.builtin.set_fact:
    previous_release: >-
      {{
        available_releases.files
        | sort(attribute='mtime', reverse=True)
        | map(attribute='path')
        | list
        | nth(1)
      }}
  tags: [rollback]

- name: Rollback symlink to previous release
  ansible.builtin.file:
    src: "{{ previous_release }}"
    dest: "{{ app_base_dir }}/current"
    state: link
  notify: Reload PM2
  tags: [rollback]

- name: Report rollback target
  ansible.builtin.debug:
    msg: "Rolled back to: {{ previous_release }}"
  tags: [rollback]

Step 6 - The Nginx Role

# roles/nginx/templates/nextjs.conf.j2
upstream nextjs_upstream {
  server 127.0.0.1:{{ app_port }};
  keepalive 64;
}

server {
  listen 80;
  server_name {{ nginx_server_name }};
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  server_name {{ nginx_server_name }};

  ssl_certificate     {{ nginx_ssl_certificate }};
  ssl_certificate_key {{ nginx_ssl_key }};
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
  ssl_prefer_server_ciphers off;

  # Next.js static assets, long-lived cache with content hashing
  location /_next/static/ {
    alias {{ app_base_dir }}/current/.next/static/;
    expires 1y;
    add_header Cache-Control "public, immutable";
  }

  # Public assets
  location /public/ {
    alias {{ app_base_dir }}/current/public/;
    expires 30d;
    add_header Cache-Control "public";
  }

  # Next.js server
  location / {
    proxy_pass         http://nextjs_upstream;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection 'upgrade';
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_cache_bypass $http_upgrade;
  }
}

# roles/nginx/tasks/main.yml
- name: Install Nginx
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true
  tags: [nginx]

- name: Template Next.js site configuration
  ansible.builtin.template:
    src: nextjs.conf.j2
    dest: /etc/nginx/sites-available/nextjs
    owner: root
    group: root
    mode: "0644"
  notify: Reload Nginx
  tags: [nginx]

- name: Enable site
  ansible.builtin.file:
    src: /etc/nginx/sites-available/nextjs
    dest: /etc/nginx/sites-enabled/nextjs
    state: link
  notify: Reload Nginx
  tags: [nginx]

# roles/nginx/handlers/main.yml
- name: Reload Nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded
  listen: Reload Nginx

Step 7 - Composing Playbooks

With roles defined, playbooks become thin composition layers:

# playbooks/provision.yml - run once to set up new servers
- name: Provision Next.js servers
  hosts: nextjs_servers
  become: true
  roles:
    - role: common
      tags: [common]
    - role: nodejs
      tags: [nodejs]
    - role: nginx
      tags: [nginx]
    - role: pm2
      tags: [pm2]
    - role: logrotate
      tags: [logrotate]
    - role: monitoring
      tags: [monitoring]

# playbooks/deploy.yml - run on every release
- name: Deploy Next.js application
  hosts: nextjs_servers
  become: true
  vars:
    local_build_path: "{{ lookup('env', 'NEXT_BUILD_PATH') }}"

  pre_tasks:
    - name: Validate build artifact exists
      ansible.builtin.stat:
        path: "{{ local_build_path }}/.next/standalone"
      delegate_to: localhost
      register: build_stat
      failed_when: not build_stat.stat.exists

  roles:
    - role: nextjs-app
      tags: [deploy]

  post_tasks:
    - name: Verify application is healthy
      ansible.builtin.uri:
        url: "https://{{ nginx_server_name }}/api/health"
        status_code: 200
        timeout: 30
      retries: 5
      delay: 10
      register: health_check
      until: health_check.status == 200

# playbooks/rollback.yml
- name: Rollback Next.js to previous release
  hosts: nextjs_servers
  become: true
  tasks:
    - name: Execute rollback
      ansible.builtin.include_role:
        name: nextjs-app
        tasks_from: rollback

Step 8 - CI/CD Integration

# .github/workflows/deploy-nextjs.yml
name: Deploy Next.js

on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install and build
        run: |
          npm ci
          npm run build

      - name: Install Ansible
        run: pip install ansible ansible-lint --break-system-packages

      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
          chmod 600 ~/.ssh/deploy_key
          echo "StrictHostKeyChecking no" >> ~/.ssh/config

      - name: Deploy to production
        run: |
          ansible-playbook playbooks/deploy.yml \
            -i inventories/production/hosts.yml \
            --private-key ~/.ssh/deploy_key \
            -e "NEXT_BUILD_PATH=${{ github.workspace }}"
        env:
          ANSIBLE_HOST_KEY_CHECKING: "False"

Common Pitfalls to Avoid

Using shell tasks where native modules exist. ansible.builtin.shell bypasses idempotency, it runs every time regardless of whether the state has changed. Always prefer native modules (apt, copy, template, service) and fall back to shell only when no native module covers the task.

Hardcoding environment-specific values in roles. Role defaults should be genuinely sensible defaults, not production values masquerading as defaults. Production specifics belong in inventory group_vars.

Not cleaning up old releases. The release directory pattern accumulates historical deployments. Add a cleanup task to retain only {{ releases_to_keep }} releases, the rest is disk usage with no operational value.

Skipping the health check post-task. A deploy playbook that doesn't verify the application is healthy after deployment is incomplete. The health check is the difference between a successful deploy and an undetected outage.

Conclusion

Modular Ansible playbooks are not just a style preference, they are an operational necessity at any scale beyond a single server. Roles as units of reuse, variable hierarchies for environment overlays, the release directory pattern for zero-downtime deployments, and thin playbooks for composition give you an infrastructure codebase that is maintainable, testable, and genuinely DRY.

For Next.js specifically, this architecture handles the full lifecycle, provisioning the Node.js runtime, deploying standalone build artifacts, configuring Nginx for optimal caching of static assets, managing PM2 for process reliability, and providing a one-command rollback when things go wrong.

Infrastructure code that can't be understood, changed, and trusted by the team that operates it is a liability. Infrastructure code built to the same quality standards as your application code is a competitive advantage.

Beyond the Basics: How to Build a Multi-Region EKS Cluster with Terraform

ThankGod Chibugwum Obobo — Sun, 05 Jul 2026 17:00:00 +0000

A single-region Kubernetes cluster is a liability in disguise. When AWS experiences a regional outage, and every major cloud provider does periodically, your entire application goes with it. For systems where availability is a business-critical requirement, a single region is not a deployment strategy, it's a single point of failure at planetary scale.

Multi-region EKS distributes your workloads across two or more AWS regions, giving you geographic redundancy, reduced latency for globally distributed users, and the ability to survive a regional failure without a production incident. But the operational complexity is real, managing two clusters, synchronizing deployments, routing traffic intelligently, and designing for state consistency across regions requires deliberate architecture.

Terraform makes this complexity manageable. With its provider aliasing, module composition, and remote state capabilities, you can define your entire multi-region infrastructure as version-controlled, reviewable, reproducible code.

This guide covers how to architect and implement a multi-region EKS cluster with Terraform, from VPC design and cluster provisioning to cross-region traffic routing, state management, and disaster recovery patterns.

Architectural Overview

Before writing configuration, establish the target architecture. A production multi-region EKS setup typically follows one of two patterns:

Active-Active: Both regions serve production traffic simultaneously. Traffic is load-balanced across regions via a global load balancer (AWS Global Accelerator or Route 53 latency routing). Each region is a full deployment of your application stack. This maximizes availability and minimizes latency but requires careful state synchronization between regions.

Active-Passive (Warm Standby): One region handles all production traffic, the second region runs a scaled-down but fully deployed replica, ready to receive traffic within minutes of a failover. Simpler to operate than active-active, at the cost of idle capacity in the passive region.

For this guide, we'll implement active-passive with warm standby, the more achievable starting point for most teams, with a clear upgrade path to active-active.

┌─────────────────────────────────────────────────────────────┐
│                    Route 53 / Global Accelerator            │
│              (Primary: eu-east-1 | Failover: eu-west-1)     │
└────────────────────┬───────────────────────────┬────────────┘
                     │                           │
          ┌──────────▼──────────┐    ┌──────────▼──────────┐
          │    EKS Cluster      │    │    EKS Cluster      │
          │    eu-east-1        │    │    eu-west-1        │
          │    (PRIMARY)        │    │    (WARM STANDBY)   │
          │                     │    │                     │
          │  ┌───────────────┐  │    │  ┌───────────────┐  │
          │  │ App Workloads │  │    │  │ App Workloads │  │
          │  │ (Full Scale)  │  │    │  │ (Min Scale)   │  │
          │  └───────────────┘  │    │  └───────────────┘  │
          │  ┌───────────────┐  │    │  ┌───────────────┐  │
          │  │  RDS Primary  │◄─┼────┼──│  RDS Replica  │  │
          │  └───────────────┘  │    │  └───────────────┘  │
          └─────────────────────┘    └─────────────────────┘

Step 1 - Repository and Terraform Structure

Organize your Terraform repository to support multi-region deployments without duplication:

/infrastructure
  /modules
    /vpc              ← reusable VPC module
    /eks              ← reusable EKS module
    /rds              ← reusable RDS module
    /alb              ← reusable ALB module
  /regions
    /eu-east-1        ← primary region configuration
      main.tf
      variables.tf
      outputs.tf
      backend.tf
    /eu-west-1        ← standby region configuration
      main.tf
      variables.tf
      outputs.tf
      backend.tf
  /global             ← cross-region resources (Route 53, Global Accelerator, IAM)
    main.tf
    variables.tf
    outputs.tf

Each region directory consumes shared modules with region-specific variable overrides. The global directory manages resources that span regions and depends on outputs from both regional deployments.

Step 2 - Configure Terraform Providers with Aliases

Multi-region Terraform requires provider aliasing, defining multiple AWS provider configurations, one per region:

# infrastructure/global/providers.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
  }
}

hcl

infrastructure/regions/eu-east-1/providers.tf

provider "aws" {
alias = "primary"
region = "eu-east-1"

default_tags {
tags = {
ManagedBy = "terraform"
Environment = var.environment
Region = "eu-east-1"
Role = "primary"
}
}
}

hcl

infrastructure/regions/eu-west-1/providers.tf

provider "aws" {
alias = "standby"
region = "eu-west-1"

default_tags {
tags = {
ManagedBy = "terraform"
Environment = var.environment
Region = "eu-west-1"
Role = "standby"
}
}
}

hcl

The default_tags block applies consistent tagging to every resource in each region, critical for cost attribution and operational clarity when managing multi-region infrastructure.

Step 3 - VPC Module Design

The VPC module provisions the network foundation. Both regions use the same module with non-overlapping CIDR ranges, a requirement for VPC peering or Transit Gateway connectivity:

# infrastructure/modules/vpc/main.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = var.vpc_cidr

  azs             = var.availability_zones
  private_subnets = var.private_subnet_cidrs
  public_subnets  = var.public_subnet_cidrs

  enable_nat_gateway     = true
  single_nat_gateway     = false       # one NAT gateway per AZ for HA
  enable_dns_hostnames   = true
  enable_dns_support     = true

  # Required tags for EKS to discover subnets
  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"             = "1"
    "kubernetes.io/cluster/${var.cluster_name}"   = "owned"
  }

  public_subnet_tags = {
    "kubernetes.io/role/elb"                      = "1"
    "kubernetes.io/cluster/${var.cluster_name}"   = "owned"
  }
}

# infrastructure/regions/eu-east-1/main.tf
module "vpc_primary" {
  source = "../../modules/vpc"
  providers = { aws = aws.primary }

  cluster_name         = "prod-eu-east-1"
  vpc_cidr             = "10.0.0.0/16"       # primary CIDR range
  availability_zones   = ["eu-east-1a", "eu-east-1b", "eu-east-1c"]
  private_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnet_cidrs  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
}

# infrastructure/regions/eu-west-1/main.tf
module "vpc_standby" {
  source = "../../modules/vpc"
  providers = { aws = aws.standby }

  cluster_name         = "prod-eu-west-1"
  vpc_cidr             = "10.1.0.0/16"       # non-overlapping CIDR range
  availability_zones   = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnet_cidrs = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
  public_subnet_cidrs  = ["10.1.101.0/24", "10.1.102.0/24", "10.1.103.0/24"]
}

Non-overlapping CIDRs (10.0.x.x for primary, 10.1.x.x for standby) are mandatory. Overlapping ranges prevent VPC peering and Transit Gateway routing between regions.

Step 4 - EKS Cluster Module

The EKS module provisions the control plane and managed node groups:

# infrastructure/modules/eks/main.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.kubernetes_version

  vpc_id                         = var.vpc_id
  subnet_ids                     = var.private_subnet_ids
  cluster_endpoint_public_access = false          # private endpoint only
  cluster_endpoint_private_access = true

  cluster_addons = {
    coredns    = { most_recent = true }
    kube-proxy = { most_recent = true }
    vpc-cni    = { most_recent = true }
    aws-ebs-csi-driver = { most_recent = true }
  }

  eks_managed_node_groups = {
    system = {
      name           = "${var.cluster_name}-system"
      instance_types = ["m6i.large"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2

      labels = { role = "system" }
      taints = [{
        key    = "CriticalAddonsOnly"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]
    }

    application = {
      name           = "${var.cluster_name}-application"
      instance_types = ["m6i.xlarge", "m6i.2xlarge"]
      min_size       = var.min_app_nodes
      max_size       = var.max_app_nodes
      desired_size   = var.desired_app_nodes

      labels = { role = "application" }
    }
  }

  # Enable IRSA for pod-level IAM roles
  enable_irsa = true
}

In the standby region, pass reduced node counts to minimize idle capacity costs:

# infrastructure/regions/eu-west-1/main.tf - standby node counts
module "eks_standby" {
  source = "../../modules/eks"
  providers = { aws = aws.standby }

  cluster_name        = "prod-eu-west-1"
  kubernetes_version  = "1.29"
  vpc_id              = module.vpc_standby.vpc_id
  private_subnet_ids  = module.vpc_standby.private_subnets

  # Warm standby — minimum viable footprint, scales up on failover
  min_app_nodes     = 1
  max_app_nodes     = 20      # headroom for failover traffic burst
  desired_app_nodes = 2
}

Step 5 - Remote State Configuration and Cross-Region References

Each region's Terraform state is stored independently in S3. The global configuration reads outputs from both regional states:

# infrastructure/regions/eu-east-1/backend.tf

hcl
terraform {
backend "s3" {
bucket = "your-org-terraform-state"
key = "eks/eu-east-1/terraform.tfstate"
region = "eu-east-1"
encrypt = true
dynamodb_table = "terraform-state-lock"
}
}

hcl

infrastructure/global/main.tf - reading cross-region outputs

data "terraform_remote_state" "primary" {
backend = "s3"
config = {
bucket = "your-org-terraform-state"
key = "eks/eu-east-1/terraform.tfstate"
region = "eu-east-1"
}
}

data "terraform_remote_state" "standby" {
backend = "s3"
config = {
bucket = "your-org-terraform-state"
key = "eks/eu-west-1/terraform.tfstate"
region = "eu-east-1" # state bucket is in primary region; key points to standby state
}
}


Remote state data sources allow your global configuration to consume the ALB DNS names, cluster endpoint ARNs, and VPC IDs from both regions, enabling Route 53 health-check-based failover configuration without manual copy-pasting.

## Step 6 - Cross-Region Traffic Routing with Route 53
Configure Route 53 health-check-based failover routing, automatically switching traffic to the standby region when the primary region's health check fails:

hcl

infrastructure/global/route53.tf

Health check against primary region ALB

resource "aws_route53_health_check" "primary" {
fqdn = data.terraform_remote_state.primary.outputs.alb_dns_name
port = 443
type = "HTTPS"
resource_path = "/health"
failure_threshold = 3
request_interval = 30

tags = { Name = "primary-health-check" }
}

Primary record - active under normal conditions

resource "aws_route53_record" "primary" {
zone_id = var.route53_zone_id
name = "api.your-domain.com"
type = "A"

failover_routing_policy {
type = "PRIMARY"
}

set_identifier = "primary"
health_check_id = aws_route53_health_check.primary.id

alias {
name = data.terraform_remote_state.primary.outputs.alb_dns_name
zone_id = data.terraform_remote_state.primary.outputs.alb_zone_id
evaluate_target_health = true
}
}

Standby record - receives traffic when primary health check fails

resource "aws_route53_record" "standby" {
zone_id = var.route53_zone_id
name = "api.your-domain.com"
type = "A"

failover_routing_policy {
type = "SECONDARY"
}

set_identifier = "standby"

alias {
name = data.terraform_remote_state.standby.outputs.alb_dns_name
zone_id = data.terraform_remote_state.standby.outputs.alb_zone_id
evaluate_target_health = true
}
}


With this configuration, Route 53 continuously polls the primary region's health check endpoint. If three consecutive checks fail (90 seconds), DNS failover to the standby region initiates automatically, no human intervention required.

## Step 7 - GitOps Deployment with Flux Across Regions
Managing application deployments across two clusters requires a GitOps controller in each region pulling from the same source of truth. **Flux CD** is purpose-built for this:

yaml

gitops/clusters/eu-east-1/flux-system/kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 5m
path: ./gitops/apps/production
prune: true
sourceRef:
kind: GitRepository
name: fleet-repository

yaml

gitops/clusters/eu-west-1/flux-system/kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 5m
path: ./gitops/apps/production # same app manifests, different cluster
prune: true
sourceRef:
kind: GitRepository
name: fleet-repository


Both Flux instances reconcile against the same `gitops/apps/production` path in your Git repository. A single push to main deploys to both regions, maintaining configuration parity automatically without manual multi-cluster kubectl commands.

## Step 8 - Database Strategy for Multi-Region
Application state is the hardest problem in multi-region architecture. For PostgreSQL:

hcl

infrastructure/modules/rds/main.tf - primary RDS instance

resource "aws_db_instance" "primary" {
provider = aws.primary
identifier = "${var.cluster_name}-primary"
engine = "postgres"
engine_version = "16.1"
instance_class = "db.r6g.xlarge"
allocated_storage = 100
storage_encrypted = true

backup_retention_period = 7
backup_window = "03:00-04:00"

# Enable automated backups to S3 for cross-region restore
enabled_cloudwatch_logs_exports = ["postgresql"]
}

Cross-region read replica in standby region

resource "aws_db_instance" "standby_replica" {
provider = aws.standby
identifier = "${var.cluster_name}-standby-replica"
replicate_source_db = aws_db_instance.primary.arn # cross-region replication
instance_class = "db.r6g.large" # smaller read-only in standby
storage_encrypted = true

# Replica can be promoted to primary during failover
# Promotion is a manual or automated operational step, not instantaneous
}


The read replica in the standby region receives continuous replication from the primary, typically with 1–5 seconds of replication lag. During a failover, promoting the replica to a standalone primary takes several minutes, during which write operations are unavailable. Design your application's failover behavior (queuing writes, surfacing a maintenance page) around this window.

## CI/CD Integration

yaml

.github/workflows/terraform-deploy.yml

name: Terraform Multi-Region Deploy

on:
push:
branches: [main]
paths: ['infrastructure/**']

jobs:
deploy-primary:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3

  - name: Terraform Apply - eu-east-1
    working-directory: infrastructure/regions/eu-east-1
    run: |
      terraform init
      terraform apply -auto-approve
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

deploy-standby:
runs-on: ubuntu-latest
needs: deploy-primary # apply standby after primary succeeds
environment: production
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3

  - name: Terraform Apply — eu-west-1
    working-directory: infrastructure/regions/eu-west-1
    run: |
      terraform init
      terraform apply -auto-approve
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

deploy-global:
runs-on: ubuntu-latest
needs: [deploy-primary, deploy-standby] # global depends on both regions
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3

  - name: Terraform Apply — global
    working-directory: infrastructure/global
    run: |
      terraform init
      terraform apply -auto-approve




The `needs` dependency chain ensures the global configuration (Route 53 failover) is applied only after both regional clusters are provisioned and their outputs are available in remote state.

## Common Pitfalls to Avoid
**Overlapping CIDR ranges.** The most common setup mistake, and the one that forces a complete VPC rebuild to fix. Plan your IP address space before writing any Terraform.

**Single NAT gateway per region.** Using one NAT gateway per region (instead of one per AZ) is cheaper but creates an availability zone as a single point of failure. Use `single_nat_gateway = false` in production.

**Not testing failover.** A failover procedure that has never been practiced will fail under pressure. Schedule a quarterly failover drill, redirect traffic to the standby region, validate application behavior, then cut back.

**Treating the standby as truly passive.** A warm standby that receives no real traffic is also receiving no validation that it works. Consider routing a small percentage of read traffic (5–10%) to the standby region to continuously validate its health.

**State drift between regions.** Without GitOps enforcing parity, application deployments to the primary region can drift from the standby. Flux or ArgoCD reconciliation against a single Git source is the most reliable prevention.

## Conclusion
Building a multi-region EKS cluster with Terraform is one of the highest-leverage investments a platform team can make in system reliability. It eliminates the single largest uncontrolled failure mode in cloud-native infrastructure, regional availability, and does so in a way that is reproducible, version-controlled, and operationally predictable.

Start with active-passive. Get the Terraform modules right, validate the Route 53 failover, practice the database promotion procedure, and establish GitOps parity across both clusters. Once that foundation is solid, the path to active-active is an evolution of what you've already built, not a rewrite.

The blueprint takes time to draw. The construction, as always, is where automation earns its keep.

*Using GKE or AKS instead of EKS? The multi-region architecture patterns and GitOps strategy in this guide apply directly, the Terraform provider and cluster module differ, but the design is cloud-agnostic. Drop your platform in the comments.*

The Blueprint for 2026: Why Architecture Must Precede Automation

ThankGod Chibugwum Obobo — Sun, 28 Jun 2026 17:01:00 +0000

2026 has brought no shortage of automation tooling, AI coding agents that scaffold entire features, CI/CD pipelines that deploy on every commit, and infrastructure-as-code platforms that provision environments in minutes. The temptation is to treat automation as a substitute for architectural discipline: if the tooling is fast enough, do the underlying design decisions even matter?

They matter more than ever. Automation amplifies whatever you point it at, including bad architecture. A poorly bounded service automated into a CI/CD pipeline doesn't become well-bounded, it becomes a poorly bounded service that ships faster and breaks more often. An AI coding agent given a vague, undocumented module doesn't infer good structure from chaos, it replicates the chaos at speed.

This article makes the case for a principle that's easy to state and hard to practice: architecture must precede automation. We'll look at why this ordering matters, what happens when teams get it backward, and how to build the discipline to get it right.

Automation Is a Force Multiplier

The core insight is simple, automation doesn't fix structural problems. It multiplies the rate at which those problems manifest.

Consider a backend system with tangled service boundaries, where orders-service reaches directly into users-service's database, where business logic is duplicated across three modules, where there's no clear ownership of shared data models. Without automation, these problems surface slowly, a developer hits friction, works around it, ships anyway. Painful, but contained.

Now introduce aggressive CI/CD automation, AI-assisted code generation, and infrastructure-as-code into that same system. Deployments that used to take a day now take ten minutes. AI agents generate new endpoints against the tangled data model in seconds. The same structural problems that used to surface occasionally now surface constantly and faster than your team can absorb the friction.

This is the central danger of automating before architecting: you don't remove the bottleneck, you relocate it, usually to incident response, on-call burden, or a production outage that automation made possible to ship faster but no easier to recover from.

What "Architecture First" Actually Means

"Architecture first" is not a call to return to multi-month design documents before writing a line of code. In 2026's engineering environment, that approach is its own form of dysfunction, over-planning that delays value delivery without guaranteeing better outcomes. Architecture-first means something more specific and more practical:

Define boundaries before you automate across them. If you don't know where one service's responsibility ends and another's begins, automating deployment, testing, or infrastructure across that ambiguous boundary only makes the ambiguity move faster. Service boundaries, data ownership, and API contracts are architectural decisions that must be made deliberately, not discovered accidentally through automation tooling.

Establish your domain model before generating code against it. AI coding agents are remarkably effective at extending an existing, well-structured codebase. They are equally effective at extending a poorly structured one, producing more code that follows the same bad patterns, at a volume no human reviewer can reasonably keep up with.

Decide your failure model before automating recovery. Self-healing systems, automated rollbacks, and circuit breakers are powerful, but only if you've already decided what failure looks like, what's recoverable, and what requires human judgment. Automating recovery for a system whose failure modes haven't been mapped is automating guesswork.

Choose your data architecture before automating data pipelines. Once an ETL pipeline, a caching layer, or an event stream is automated and running, the underlying data model becomes extremely expensive to change, every downstream automated process depends on it remaining stable.

The 2026 Pressure Toward Automation-First

It's worth naming why this principle is harder to follow now than it was five years ago.

AI coding agents lower the cost of generating code dramatically, which makes "just generate it and see" feel like a reasonable substitute for upfront design. The cost of writing code has dropped, the cost of living with the wrong architecture has not.

CI/CD maturity has made deployment nearly frictionless, which removes a natural pause point that used to force teams to think before shipping. When deploying took an afternoon of manual steps, teams were implicitly forced to batch decisions and consider them more carefully. When deploying takes ninety seconds, that forcing function disappears.

Competitive pressure rewards visible velocity over invisible architectural soundness. Shipping ten features this quarter is visible to leadership. The clean service boundary that made shipping the eleventh feature possible without a rewrite is invisible, until it's missing.

None of this means automation is the problem. It means automation has gotten powerful enough that it no longer provides natural friction against bad architectural decisions, that friction now has to be deliberately engineered back in, through team practices and review discipline.

What Architecture-First Looks like in Practice

This doesn't require slowing down, it requires sequencing.

Before automating CI/CD for a new service: define its API contract, its data ownership boundary, and its failure isolation strategy. This can take an afternoon of focused design discussion, not a quarter of documentation. The output is a lightweight decision record, not a 40-page architecture document.

Before pointing an AI coding agent at a codebase: ensure the codebase has clear module boundaries, consistent naming conventions, and a small number of canonical patterns the agent can learn from and extend. An agent given a clean example to follow produces dramatically better output than one given a blank slate or a tangled mess.

Before automating infrastructure provisioning: decide your environment topology, your network boundaries, and your secrets management approach (see our guide on replacing hardcoded credentials with Vault and GitHub OIDC). Terraform modules generated against an undefined topology become technical debt the moment they're applied.

Before automating test generation: define what correctness actually means for the system under test, the contracts, the invariants, the edge cases that matter to the business. AI-generated tests against an undefined specification produce tests that pass without verifying anything meaningful.

The pattern across all of these is, a small deliberate architectural decision made before automation gives the automation something solid to amplify. Skipping that decision gives automation chaos to amplify instead.

The Cost of Getting the Order Wrong

Teams that automate before architecting tend to discover the cost in a predictable sequence:

First, velocity feels great. Automation is shipping fast. Dashboards show high deployment frequency, high PR throughput, high AI-assisted code volume.

Then, defects compound faster than they can be fixed. Because the underlying structure was never sound, every new automated change has a wider blast radius than expected. Bugs surface in unexpected places because boundaries were never enforced.

Then, the team builds automation to manage the automation. More alerting to catch what the velocity is breaking. More rollback automation to recover from what the velocity is shipping. This is a sign the team is now spending its automation investment compensating for an architectural debt it never paid down, not multiplying genuine engineering leverage.

Finally, a rewrite becomes the only path forward. At this point, the team faces the choice no one wants, a costly, risky architectural rewrite, executed under the same time pressure that caused the original problem.

This sequence is avoidable, not by rejecting automation, but by insisting on the right order of operations.

A Practical Framework: The Architecture Gate

For teams who want a concrete practice rather than just a principle, introduce an architecture gate before any significant automation investment:

Before automating: [CI/CD pipeline | AI code generation | infra provisioning | test generation]

Ask:

Are the relevant service/module boundaries explicitly defined?
Is data ownership for this domain unambiguous?
Is there a documented contract (API, schema, interface) the automation will rely on?
Have failure modes for this component been identified and categorized?
Is there a clear owner for decisions this automation will encode at scale?

If any answer is "no", pause and resolve it first.
The fix is usually a focused half-day design discussion, not a lengthy process.

This isn't bureaucracy for its own sake. It's a five-question checklist that takes less time than debugging the first production incident caused by skipping it.

Architecture and Automation Are Not in Tension

The framing of "architecture vs. automation" is itself a false choice. The most effective engineering organizations in 2026 don't choose one over the other, they sequence them correctly. Sound architecture is what makes aggressive automation safe. Aggressive automation is what makes sound architecture deliver value at speed.

A well-bounded microservices architecture (see our guide on migrating from monolith to microservices with NestJS and Postgres) becomes dramatically more valuable once CI/CD, contract testing, and observability are automated around it, because the boundaries hold under the increased velocity rather than buckling.

The lesson for 2026 is not "slow down." It's "decide deliberately, then move fast." Architecture is the blueprint. Automation is the construction crew. No amount of construction speed compensates for a blueprint that was never drawn.

Conclusion

As AI tooling and automation platforms continue to compress the time between idea and deployed code, the temptation to skip architectural thinking will only grow stronger, because skipping it no longer feels like it costs anything, in the moment. It does. It simply defers the cost to a less convenient time, at a larger scale, with less context available to fix it.

The teams that will build the most reliable, adaptable systems in 2026 are not the ones who automate the most. They're the ones who know precisely what to automate, because they took the time to architect it first.

Draw the blueprint. Then build fast.

What's your team's process for architectural review before adopting new automation tooling? Share your approach in the comments.

Dependency Auditing at Scale: How to Automate Supply Chain Security with Dependabot and Snyk

ThankGod Chibugwum Obobo — Sun, 21 Jun 2026 17:00:00 +0000

The average modern web application depends on hundreds, sometimes thousands, of transitive npm packages. Each one is a potential attack vector, a vulnerability waiting to be disclosed, a maintainer account waiting to be compromised, or a malicious update waiting to be published. Manually tracking the security posture of a dependency tree this large is not just impractical at any meaningful scale, it is impossible.

Supply chain security has moved from a niche compliance concern to a top-tier engineering priority, driven by high-profile incidents like the event-stream backdoor, the ua-parser-js compromise, and the XZ Utils backdoor that nearly compromised SSH across major Linux distributions. Attackers have learned that compromising one widely-used package is more efficient than attacking a thousand individual applications.

This guide covers how to build an automated dependency auditing pipeline using GitHub Dependabot and Snyk, two complementary tools that, used together, give you continuous vulnerability detection, automated remediation, and CI/CD gates that stop vulnerable code before it ships.

Why Manual Dependency Management Doesn't Scale

Consider a mid-sized application with 40 direct dependencies. Each of those typically pulls in 10–30 transitive dependencies of its own, meaning your actual dependency tree can easily exceed 800–1200 packages. Each package:

Has its own maintainers, release cadence, and security practices
Can be updated at any time, potentially introducing a vulnerability or malicious code
May depend on other packages with their own vulnerabilities
Often includes postinstall scripts that execute arbitrary code on npm install

Manually monitoring CVE databases, cross-referencing them against your package-lock.json, and tracking remediation status across dozens of services is not a sustainable practice. It requires automation, both for detection and for response.

Understanding the Two-Tool Strategy

Dependabot (native to GitHub) and Snyk (a dedicated security platform) overlap in purpose but differ in depth and capability. Understanding the distinction helps you use both effectively rather than redundantly.

Capability	Dependabot	Snyk
Vulnerability database	GitHub Advisory Database	Snyk's proprietary database (broader, faster disclosure)
Automated PRs for fixes	Yes	Yes
License compliance scanning	No	Yes
Container image scanning	Limited	Yes
Infrastructure as Code scanning	No	Yes
Code-level vulnerability scanning (SAST)	No	Yes (Snyk Code)
Native GitHub integration	Built-in, free	Requires app installation
Fix prioritization by exploitability	Basic	Advanced risk scoring
Cost	Free for public & private repos	Free tier is limited, pay for scale

Recommended approach: Use Dependabot as your baseline, always-on dependency update mechanism, it's free, native, and requires minimal setup. Layer Snyk on top for deeper vulnerability intelligence, license compliance, and broader scanning coverage (containers, IaC, code-level issues) across your most critical services.

Step 1 - Configure Dependabot

Dependabot is configured via a single YAML file per repository. Start with version updates and security updates enabled:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "monday"
      time: "09:00"
    open-pull-requests-limit: 10
    groups:
      production-dependencies:
        dependency-type: "production"
        update-types: ["minor", "patch"]
      development-dependencies:
        dependency-type: "development"
        update-types: ["minor", "patch", "major"]
    ignore:
      - dependency-name: "legacy-package"
        update-types: ["version-update:semver-major"]
    labels:
      - "dependencies"
      - "automated"
    reviewers:
      - "platform-team"
    commit-message:
      prefix: "chore(deps)"

  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"

  - package-ecosystem: "github-actions"
    directory: "/.github/workflows"
    schedule:
      interval: "weekly"

Key configuration decisions:

groups consolidates multiple minor/patch updates into a single PR, dramatically reducing PR noise for large dependency trees.
open-pull-requests-limit prevents Dependabot from overwhelming your review queue.
Separate ecosystems (docker, github-actions) ensure your container base images and CI workflow actions are also kept current, both common and overlooked attack surfaces.

Enabling Security-Only Updates

Security updates are enabled by default once Dependabot alerts are turned on for the repository, and they bypass the schedule, firing immediately when a new vulnerability is disclosed for a dependency you use:

Repository → Settings → Code security and analysis
Dependabot alerts
Dependabot security updates
Dependabot version updates

Step 2 - Configure Snyk for Deeper Scanning

Install the Snyk GitHub integration and add a .snyk policy file to your repository:

npm install -g snyk
snyk auth
snyk test

Generate a baseline .snyk policy file to manage known, accepted-risk vulnerabilities with documented justification:

# .snyk
version: v1.5.0
ignore:
  SNYK-JS-LODASH-1040724:
    - '*':
        reason: >
          Vulnerability is in a dev-only build tool, not present in
          production bundle. Tracked in JIRA-4521 for upgrade in Q3.
        expires: 2026-09-30T00:00:00.000Z
patch: {}

Never use .snyk to silently suppress vulnerabilities. Every ignored entry requires a documented reason and an expiration date, forcing periodic re-evaluation rather than permanent suppression.

Snyk CI Integration

# .github/workflows/snyk-security.yml
name: Snyk Security Scan

on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  snyk-dependencies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Snyk to check for vulnerabilities
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high --fail-on=upgradable

      - name: Run Snyk Code (SAST)
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          command: code test
          args: --severity-threshold=high

  snyk-container:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build container image
        run: docker build -t app:${{ github.sha }} .

      - name: Run Snyk Container scan
        uses: snyk/actions/docker@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          image: app:${{ github.sha }}
          args: --severity-threshold=high

The --fail-on=upgradable flag is important, it only fails the build for vulnerabilities that have an available fix, preventing your pipeline from blocking on issues you currently have no way to remediate.

Step 3 - License Compliance Scanning

Open source licenses carry legal obligations, some permissive (MIT, Apache 2.0), others restrictive (GPL, AGPL) in ways that can create compliance risk for commercial products. Snyk's license scanning catches these automatically:

# Add license policy enforcement
- name: Snyk License Compliance
  uses: snyk/actions/node@master
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
  with:
    command: test
    args: --severity-threshold=high --fail-on=all
            --policy-path=.snyk-license-policy.json

Configure license policies in the Snyk dashboard to flag or block specific license types:

Disallow:  GPL-3.0, AGPL-3.0, SSPL-1.0
Warn:      LGPL-2.1, MPL-2.0
Allow:     MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC

This is particularly critical for companies building commercial or closed-source products, an AGPL-licensed transitive dependency can create unexpected obligations to release your own source code.

Step 4 - Prioritizing Remediation with Risk Scoring

Not every vulnerability deserves the same urgency. A critical CVE in a dependency that's only used in your test suite is lower risk than a medium-severity CVE in a package that processes user input in production.

Snyk's risk scoring considers exploit maturity, whether the vulnerable code path is actually reachable, and the dependency's position in your tree (direct vs. transitive) to prioritize remediation:

# View prioritized vulnerabilities sorted by risk score
snyk test --json | snyk-to-html -o report.html

# Or query via API for custom dashboards
curl -H "Authorization: token $SNYK_TOKEN" \
  "https://api.snyk.io/v1/org/$ORG_ID/projects/$PROJECT_ID/issues"

Establish a remediation SLA based on severity and reachability:

Severity	Reachable in Production Code	Remediation SLA
Critical	Yes	24–48 hours
Critical	No (dev/test only)	1 week
High	Yes	1 week
High	No	2 weeks
Medium	Yes	2 weeks
Medium	No	Next regular update cycle
Low	Any	Tracked, no SLA

Step 5 - Automating Patch PRs and Auto-Merge

For low-risk updates, patch and minor version bumps with passing tests, automate the entire remediation loop:

# .github/workflows/automerge-dependabot.yml
name: Auto-merge Dependabot PRs

on: pull_request

permissions:
  pull-requests: write
  contents: write

jobs:
  automerge:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'
    steps:
      - name: Fetch Dependabot metadata
        id: metadata
        uses: dependabot/fetch-metadata@v2
        with:
          github-token: "${{ secrets.GITHUB_TOKEN }}"

      - name: Auto-approve patch and minor updates
        if: steps.metadata.outputs.update-type == 'version-update:semver-patch' || steps.metadata.outputs.update-type == 'version-update:semver-minor'
        run: gh pr review --approve "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Enable auto-merge
        if: steps.metadata.outputs.update-type == 'version-update:semver-patch' || steps.metadata.outputs.update-type == 'version-update:semver-minor'
        run: gh pr merge --auto --squash "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Auto-merge should only apply to patch/minor updates and require your full CI suite (tests, linting, security scans) to pass first. gh pr merge --auto waits for all required status checks before merging. Major version updates should always require human review, since they're more likely to introduce breaking changes.

Step 6 - Software Bill of Materials (SBOM) Generation

Beyond vulnerability scanning, regulatory and enterprise procurement requirements increasingly demand a Software Bill of Materials, a complete inventory of every component in your software supply chain:

# Generate an SBOM in CycloneDX format using Snyk
snyk sbom --format=cyclonedx1.5+json --org=$ORG_ID > sbom.json

# .github/workflows/sbom.yml
- name: Generate SBOM
  run: snyk sbom --format=cyclonedx1.5+json > sbom.json
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

- name: Upload SBOM as release artifact
  uses: actions/upload-artifact@v4
  with:
    name: sbom
    path: sbom.json

Store SBOMs alongside each release artifact, they provide auditable proof of exactly what shipped, which is increasingly required for enterprise customers and government contracts (e.g., U.S. Executive Order 14028 compliance).

Step 7 - Blocking Malicious Packages Proactively

Beyond known CVEs, supply chain attacks often involve packages with no disclosed vulnerability, because the malicious behavior was intentionally introduced by a compromised maintainer account. Snyk and GitHub both maintain databases of known-malicious packages distinct from standard CVE tracking:

# Block installation of known-malicious packages at install time
- name: Check for malicious packages
  run: |
    npx socket-security audit  # cross-references known malicious package database

Combine this with npm ci --ignore-scripts in CI to prevent postinstall scripts from executing arbitrary code during dependency installation, a common vector for credential exfiltration:

npm ci --ignore-scripts
# Run necessary build scripts explicitly and individually instead of trusting postinstall hooks
npm run build

Common Pitfalls to Avoid

Treating Dependabot PR volume as noise to ignore. A repository with 60 open Dependabot PRs signals the automation is not being acted on, defeating its purpose. Use grouping and auto-merge for low-risk updates to keep the queue manageable.

Scanning only direct dependencies. The majority of real-world vulnerabilities live in transitive dependencies your team has never directly chosen. Ensure your scanning tools traverse the full dependency tree, not just package.json.

No reachability analysis. Flagging every CVE in every dependency, even ones whose vulnerable code path is never executed, creates alert fatigue. Use tools with reachability analysis (Snyk Code, Socket) to prioritize what's actually exploitable in your application.

Ignoring container and IaC layers. Application dependencies are only one layer of your supply chain. Base container images and Terraform providers carry their own vulnerabilities and deserve the same scanning rigor.

Conclusion

Supply chain security at scale cannot be a manual process, the sheer size of modern dependency trees makes that mathematically impossible to sustain. Dependabot provides a free, always-on baseline for dependency updates and security patches. Snyk adds depth: license compliance, container and IaC scanning reachability-aware risk scoring, and code-level SAST.

Used together, with auto-merge for low-risk updates, documented exception policies for accepted risk, and SBOM generation for auditability, you transform dependency management from a reactive scramble after each disclosed CVE into a continuous, automated engineering practice.

The goal isn't zero vulnerabilities, that's not achievable in any non-trivial dependency tree. The goal is continuous visibility, fast remediation for what's exploitable, and an audit trail for everything else.

Running a polyglot stack with Python, Go, and Java services alongside Node.js? Both Dependabot and Snyk support multi-ecosystem scanning natively — the configuration patterns in this guide apply with ecosystem-specific adjustments. Drop your stack in the comments.