DEV Community: ThankGod Chibugwum Obobo

Internal Developer Platforms: How to Build a Golden Path for Your Engineering Team

ThankGod Chibugwum Obobo — Sun, 27 Sep 2026 17:00:00 +0000

Every engineering organization above a certain size develops the same invisible problem. Some teams ship confidently and fast, deploying multiple times a day, spinning up new services without friction, debugging production issues with full observability. Other teams spend a disproportionate amount of time navigating infrastructure, waiting for platform approvals, reinventing tooling that already exists somewhere else in the organization, or simply not knowing what the right approach is.

The difference between those teams is rarely talent. It is almost always platform.

An Internal Developer Platform (IDP) is the curated set of tools, workflows, standards, and self-service capabilities that an organization provides to its engineering teams to accelerate and standardize how software gets built and delivered. When done well, it creates a golden path, a paved road of sensible defaults that makes doing the right thing the easy thing, without mandating it.

This guide covers how to design and build an IDP that actually gets adopted, from the layers of a platform to service catalog design, self-service provisioning, developer portals, and the feedback loops that make a platform evolve rather than stagnate.

The Platform Engineering Problem

The instinct that creates IDPs is sound, instead of every team solving the same infrastructure, CI/CD, observability, and security problems independently, a dedicated platform team solves them once and makes the solution available to everyone.

The execution often fails for two reasons:

Building what the platform team thinks developers need instead of what developers actually need. A platform built on assumptions rather than feedback becomes a set of tools nobody uses, maintained by a team that wonders why adoption is low.

Building a wall, not a road. An IDP that forces developers through approval processes, ticket queues, and bureaucratic gates in the name of standardization creates friction rather than removing it. Developers route around friction, they find workarounds, create shadow IT, and build their own tooling rather than use a platform that slows them down.

The golden path metaphor captures the right design intent, a golden path is not a mandatory route. It is the path that is so clearly well-maintained, well-lit, and well-documented that choosing it is easier than going off-road. Developers take it because it is the better option, not because they have no alternative.

The Layers of an Internal Developer Platform

An IDP is not a single tool, it is a stack of integrated capabilities that collectively reduce the cognitive load of building and operating software:

┌─────────────────────────────────────────────────────────┐
│                  Developer Portal                       │
│     (Service Catalog, Docs, Templates, Dashboards)      │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│               Self-Service Layer                        │
│   (Provisioning, Environments, Pipelines, Secrets)      │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│              Orchestration Layer                        │
│      (Kubernetes, ArgoCD, Helm, Spacelift, Vault)       │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│              Infrastructure Layer                       │
│         (AWS/GCP/Azure, Terraform, Networking)          │
└─────────────────────────────────────────────────────────┘

Infrastructure layer - the cloud resources, networks, and compute that everything runs on. Platform engineers own this entirely, application teams never interact with it directly.

Orchestration layer - the tools that manage how applications are packaged, deployed, and operated. Kubernetes, ArgoCD, Helm, Vault, and Spacelift live here. Application teams interact with this layer through abstracted interfaces, not directly.

Self-service layer - the APIs, CLIs, and forms that allow application teams to provision environments, spin up new services, manage secrets, and trigger deployments without opening tickets or waiting for platform team approval.

Developer portal - the unified interface where developers discover what the platform provides, browse available services, read documentation, access templates, and monitor the health of their systems.

Step 1 - Start with Developer Research, Not Technology

The most common IDP failure mode is building the platform before understanding what slows developers down. Before choosing any technology, spend four weeks doing structured developer research.

Platform Research Interview Guide

Onboarding

How long did it take you to deploy your first change to production?
What was the hardest part of understanding how deployment works here?

Day-to-Day Development

What takes longer than it should?
What do you find yourself doing manually that you wish was automated?
What do you copy from one service to another when starting something new?

Production Operations

When something breaks in production, what information do you reach for first?
How confident are you in your ability to diagnose and fix a production incident?

Collaboration with Platform

How do you currently request infrastructure or tooling changes?
How long does it typically take to get what you need?
What would make your relationship with the platform team more effective?

Run this interview with 8-10 engineers across different teams and tenure levels. The patterns that emerge, the repeated friction points, the consistent gaps, the workarounds that everyone has independently invented, are your platform backlog. Build against that backlog, not against a preconceived vision of what a modern platform looks like.

Step 2 - Define Your Golden Path

A golden path is a specific, opinionated recommendation for how to build and operate a service at your organization. It is not a mandate, it is the answer to "if I'm starting something new, what should I use?".Document your golden path explicitly. e.g:

Golden Path: New Backend Service

Language and Framework

Recommended: TypeScript with NestJS
Why: Consistent with 80% of existing services, best internal support.

Packaging and Deployment

Recommended: Docker → Helm chart → ArgoCD
Template: github.com/your-org/service-template

Database

Recommended: PostgreSQL (managed RDS via Terraform module)
Self-service: Use the rds-postgres module in the platform registry

Observability

Required: Structured logging (Pino), OpenTelemetry tracing, health check endpoint
Dashboard template: Auto-provisioned on service creation

Secrets Management

Recommended: External Secrets Operator + Vault
Never: Hardcoded credentials or plaintext .env files in Git

CI/CD

Recommended: GitHub Actions with the platform's standard workflow templates
Templates: Available in platform/ci-templates

Testing Requirements

Unit test coverage ≥ 70%
Contract tests for all service-to-service dependencies
E2E tests for critical user journeys

The golden path document is a living artifact, it evolves as the platform evolves and as teams provide feedback on what works and what doesn't. Version-control it and treat updates as pull requests with engineering review.

Step 3 - Service Templates and Scaffolding

The first concrete experience most developers have with the platform is creating a new service. If that experience requires reading 15 pages of documentation and making 40 manual configuration decisions, the platform has failed at the moment of first contact.

Backstage software templates (or equivalent) solve this by making new service creation a form-filling exercise that produces a fully configured repository:

# backstage/templates/nestjs-service/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nestjs-backend-service
  title: NestJS Backend Service
  description: Production-ready NestJS service with observability, CI/CD, and Helm chart
  tags: [typescript, nestjs, backend, recommended]

spec:
  owner: platform-team
  type: service

  parameters:
    - title: Service Details
      required: [name, description, owner]
      properties:
        name:
          title: Service Name
          type: string
          pattern: '^[a-z][a-z0-9-]*$'
          description: Lowercase, hyphenated (e.g. orders-service)
        description:
          title: Description
          type: string
        owner:
          title: Owning Team
          type: string
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group

    - title: Infrastructure
      properties:
        database:
          title: PostgreSQL Database
          type: boolean
          default: false
        redis:
          title: Redis Cache
          type: boolean
          default: false
        initialReplicas:
          title: Initial Replica Count
          type: integer
          default: 2
          minimum: 1
          maximum: 10

  steps:
    - id: fetch-template
      name: Fetch Service Template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          description: ${{ parameters.description }}
          owner: ${{ parameters.owner }}
          database: ${{ parameters.database }}
          redis: ${{ parameters.redis }}

    - id: create-repo
      name: Create GitHub Repository
      action: publish:github
      input:
        repoUrl: github.com?repo=${{ parameters.name }}&owner=your-org
        description: ${{ parameters.description }}
        defaultBranch: main
        repoVisibility: private
        topics: [${{ parameters.owner }}, backend, nestjs]

    - id: register
      name: Register in Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['create-repo'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

  output:
    links:
      - title: Repository
        url: ${{ steps['create-repo'].output.remoteUrl }}
      - title: Open in Catalog
        icon: catalog
        entityRef: ${{ steps['register'].output.entityRef }}

The scaffold output is a complete repository with: NestJS boilerplate, Dockerfile, Helm chart, GitHub Actions CI workflow, catalog-info.yaml for Backstage registration, OpenTelemetry instrumentation, structured logging configuration, and a health check endpoint. A developer fills out a form and receives a production-ready repository in under two minutes.

Step 4 - The Developer Portal with Backstage

Backstage (open-sourced by Spotify) is the most widely adopted developer portal framework. It provides the unified interface layer of your IDP, service catalog, documentation, templates, and plugin integrations:

# catalog-info.yaml — every service registers itself
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: orders-service
  description: Order management and fulfilment service
  annotations:
    github.com/project-slug: your-org/orders-service
    grafana/dashboard-selector: "title=Orders Service"
    pagerduty.com/service-id: P1234AB
    argocd/app-name: orders-service-production
    sonarqube.org/project-key: your-org_orders-service
  tags: [backend, typescript, nestjs]
  links:
    - url: https://orders.your-domain.com/api/docs
      title: API Documentation
    - url: https://runbooks.internal/orders-service
      title: Runbook

spec:
  type: service
  lifecycle: production
  owner: backend-team
  system: order-management
  dependsOn:
    - component:users-service
    - component:payments-service
  providesApis:
    - orders-api

With every service registered, Backstage provides:

Service catalog - searchable inventory of every service, its owner, its status, and its dependencies
TechDocs - docs-as-code rendered from Markdown in each service's repository
Dependency graph - visual map of which services depend on the other
Plugin integrations - ArgoCD deployment status, Grafana dashboards, SonarQube code quality, PagerDuty incidents, GitHub pull requests, all surfaced in a single UI per service

The principle: a developer should be able to open Backstage, find their service, and immediately see its deployment status, recent incidents, code quality score, current pull requests, and documentation, without navigating five different tools.

Step 5 - Self-Service Environment Provisioning

One of the highest-friction developer experiences is getting a new environment. In organizations without self-service, this means a ticket to the platform team, a wait of days to weeks, and a result that may or may not match what was requested.

Self-service provisioning eliminates the ticket. Using Spacelift, Terraform modules, and Backstage actions, a developer can provision a complete environment by selecting options from a catalog:

# Backstage action - provision environment from catalog
- id: provision-environment
  name: Provision Development Environment
  action: platform:provision-environment
  input:
    name: ${{ parameters.environmentName }}
    team: ${{ parameters.owner }}
    services:
      - name: orders-service
        version: ${{ parameters.ordersVersion }}
      - name: users-service
        version: latest
    database: true
    expiresIn: 7d   # auto-destroy after 7 days to control costs

Behind the scenes, this triggers a Spacelift stack that applies a Terraform configuration, provisions an EKS namespace, deploys the specified service versions via ArgoCD, and returns a URL to the environment, all within minutes, with no platform team involvement.

Ephemeral environments that auto-expire prevent environment sprawl and the associated cost. Developers get exactly the environment they need, for exactly as long as they need it, without creating permanent infrastructure debt.

Step 6 - Platform Observability and DORA Metrics

A platform team that cannot measure its impact cannot improve it. Track DORA metrics (Developer Research and Assessment) as the primary signal of platform effectiveness:

Metric	Definition	Target (Elite)
Deployment Frequency	How often code deploys to production	Multiple times per day
Lead Time for Changes	Commit to production time	Less than one hour
Change Failure Rate	Percentage of deployments causing incidents	Less than 5%
Failed Deployment Recovery Time	Time to restore after a failure	Less than one hour

Instrument these metrics from your CI/CD pipeline and present them in Backstage per team:

// platform/metrics/dora-collector.ts
export async function collectDeploymentFrequency(
  team: string,
  windowDays: number = 30
): Promise<number> {
  const deployments = await githubClient.actions.listWorkflowRuns({
    owner: 'your-org',
    repo: '*',  // across all repos for this team
    status: 'success',
    created: `>${subDays(new Date(), windowDays).toISOString()}`,
  });

  return deployments.data.total_count / windowDays;  // deployments per day
}

Beyond DORA, track platform-specific adoption metrics:

Golden path adoption rate - percentage of services using the standard template
Self-service provisioning rate - percentage of environment requests fulfilled via self-service vs. tickets
Platform NPS - developer satisfaction score from quarterly surveys
Time to first deployment - for new engineers, from day one to first production deployment

These metrics tell you whether the platform is achieving its purpose, reducing developer friction and increasing engineering velocity.

Step 7 - The Platform Team Operating Model

A platform is a product. It needs a product owner, a backlog, a roadmap, and a feedback mechanism. The most common failure of platform engineering is treating it as an infrastructure team that happens to build tools, rather than a product team whose customers are developers.

Treat developers as customers. Platform teams should maintain a developer experience feedback channel (Slack, GitHub Discussions, regular office hours) and respond to feedback with the same urgency they'd apply to a production incident. A developer who can't get a question answered about the platform will build around it.

Ship incrementally. An IDP does not need to be complete before it launches. A golden path with a service template, a basic CI/CD workflow, and a simple developer portal is more valuable than a comprehensive platform that's six months from delivery. Ship the minimum viable platform, measure adoption, and iterate.

Dogfood your own platform. Every internal tool the platform team builds should run on the platform. If the platform team bypasses its own standards, using different deployment mechanisms, different observability tools, different secret management, it signals that the platform isn't good enough even for the people who built it.

Establish an API for the platform itself. The self-service layer should be programmable, accessible via CLI, GitHub Actions, and API, not just a web UI. Developers who can script the platform can build their own workflows on top of it, multiplying the platform's reach beyond what the platform team alone can provide.

Common Pitfalls to Avoid

Building for the ideal developer, not the actual developer. A platform that assumes developers know Kubernetes, understand Terraform, and have read all the documentation will be used only by developers who already don't need it. Design the golden path for the median engineer on your team, not the most experienced.

Deprecating old paths before the golden path is ready. Removing existing tools and workflows before self-service alternatives exist forces developers to wait rather than incentivizing them to adopt. Always build the new path first, then migrate, then deprecate.

Measuring adoption instead of outcomes. High adoption of the platform is not the goal, high developer velocity and high system reliability are the goals. If the platform is widely adopted but DORA metrics aren't improving, the platform is being used but not delivering value.

Under-resourcing the platform team. A platform serving 100 developers with a team of one is a support ticket queue with a nice interface. Platform teams typically need one platform engineer for every 10–15 application engineers to be effective, depending on scope and maturity.

Conclusion

An Internal Developer Platform is not a one-time project. It is a product that evolves continuously in response to how developers needs change as the organization scales. The golden path is not a fixed route it is a living recommendation that reflects the organization's current best understanding of how to build reliable software efficiently.

The investment is significant. A platform team, a developer portal, service templates, self-service provisioning, and DORA metric tracking require sustained engineering effort and organizational commitment. The return on that investment is compounding, every developer who doesn't spend two days setting up a new service, every incident that gets diagnosed in minutes instead of hours, every security practice that is adopted automatically rather than manually, represents developer time recovered and directed toward building the product.

The platform is never finished. But a platform that is thoughtfully built, honestly measured, and continuously improved is one of the highest-leverage investments a growing engineering organization can make.

What does your team's current "golden path" look like, documented or de facto? And what's the single biggest friction point it hasn't yet solved? Share in the comments.

PlatformEngineering #InternalDeveloperPlatform #Backstage #DeveloperExperience #DevOps

Why I Switched from Software Development to DevOps (and Why You Should Too)

ThankGod Chibugwum Obobo — Sun, 20 Sep 2026 17:00:00 +0000

I spent five years writing application code, TypeScript backends, React frontends, the occasional GraphQL API. I was good at it. I understood the domain, I could ship features, and I had strong opinions about clean architecture and test coverage. Then I spent three months debugging a production incident that turned out to be a misconfigured load balancer, a memory leak in a sidecar container I didn't know existed, and a Terraform module someone had modified directly in the AWS console without updating the code.

That incident didn't make me want to become a DevOps engineer. But it made me realize I was writing software on top of infrastructure I didn't understand, deploying to environments I couldn't reason about, and debugging failures in systems I had no visibility into. I was operating with half the picture and treating the other half as someone else's problem.

The switch to DevOps was not a career pivot away from software. It was a career expansion toward the full scope of how software actually runs.

This is an honest account of what that transition looked like, what was harder than expected, what became clearer once I made it, and why I think certain kinds of developers should seriously consider it.

What "DevOps" Actually Means in Practice

The word is overloaded. Job descriptions titled "DevOps Engineer" range from glorified sysadmin work to platform engineering to SRE to CI/CD pipeline maintenance. Before I made the switch, I had a vague understanding that DevOps was about pipelines and Kubernetes and "the cloud." That understanding was correct but incomplete in the same way that "software development is about writing functions" is correct but incomplete.

In practice, DevOps engineering is about owning the system that software runs on, not just the software itself. That means:

Designing and maintaining the infrastructure that hosts applications
Building the CI/CD pipelines that deliver code from a developer's laptop to production
Implementing observability so teams can understand what their systems are doing
Enforcing security and compliance at the infrastructure level
Managing the cost, capacity, and reliability of cloud resources
Building the internal developer platform that makes other engineers productive

The developer instinct is to ask "does this code work?" The DevOps instinct is to ask "does this system work across all environments, under all load conditions, after all possible failures?" Both questions matter. Most software teams are better at the first than the second.

What I Left Behind

I want to be honest about what the transition cost, because most "you should do this" career articles gloss over it.

Domain expertise takes time to build. I spent five years developing strong intuitions about software design, when to abstract, how to structure modules, when a codebase is healthy versus deteriorating. Those intuitions transferred partially but not completely. Infrastructure has its own design principles (idempotency, immutability, least privilege, blast radius) that felt foreign initially and required deliberate learning rather than osmosis.

The feedback loop is different. Application code fails fast, your tests catch it, your linter catches it, your PR reviewer catches it. Infrastructure mistakes often fail slowly, sometimes in production, sometimes weeks after the change that caused them. Learning to write infrastructure code defensively, with the same discipline I'd applied to application code, took longer than I expected.

You write less code. A significant portion of DevOps work is configuration, tooling, automation scripting, and architectural decision-making. Engineers who derive deep satisfaction from the act of writing complex application logic may find the shift jarring. Not every DevOps problem is best solved by writing more code, sometimes it's solved by choosing the right managed service, configuring an existing tool, or documenting a process.

None of these are reasons not to make the transition. They are reasons to go in with accurate expectations rather than inflated ones.

What Became Clearer

The payoff was not immediate but it was real.

The full system became legible. Before the switch, I understood the application layer of the systems I worked on. After spending serious time in infrastructure, I could read the whole picture, from the VPC topology to the Kubernetes cluster to the service mesh to the application code to the observability stack. When something breaks, I know where to look. That diagnostic breadth is genuinely rare and genuinely valuable.

I understand why certain software patterns exist. Retry logic, idempotency keys, circuit breakers, graceful shutdown handlers, health check endpoints, these are application-layer patterns that exist because of infrastructure realities. Having lived those realities, I understand why the patterns matter.

Production is less mysterious. There is a specific kind of developer anxiety that comes from deploying to an environment you don't fully understand, a system that might work or might not, for reasons you can't reliably predict. That anxiety is not irrational, it reflects a real information gap. Closing that gap changes your relationship to production from one of nervous deference to one of informed confidence.

I became a better collaborator. Understanding both the application and infrastructure layers means I can have substantive conversations with both developers and platform engineers. I can translate between the two perspectives rather than being fully embedded in only one. On cross-functional teams, that translation ability is consistently undervalued and consistently useful.

The Skills That Transferred Directly

Not everything was a steep learning curve. Several software development skills transferred directly and gave me an early advantage:

Debugging methodology. The discipline of forming hypotheses, testing them systematically, and narrowing the problem space applies equally to application bugs and infrastructure incidents. DevOps engineers who come from software development tend to be more methodical debuggers than those who came from traditional operations.

Code quality instincts. Treating infrastructure code (Terraform, Ansible, CI/CD pipelines) with the same discipline as application code, clear naming, DRY principles, meaningful abstractions, code review, is not universal in operations teams. A background in software development makes this instinctive.

Automation-first thinking. The developer instinct to automate repetitive work translates directly into better DevOps practice. Clicking through cloud consoles is to DevOps what manual testing is to software development, necessary sometimes, but never the long-term answer.

Reading and writing documentation. This sounds trivial but is not. Engineers who can write a clear runbook, a meaningful ADR, or an incident postmortem that actually changes behavior are not common in any specialty. The writing habits built in good engineering cultures transfer.

The Skills That Took Deliberate Effort

Networking. I had a surface-level understanding of TCP/IP, DNS, and HTTP from the application side. Translating that into the ability to design VPCs, debug routing tables, configure security groups, and reason about load balancer behavior required deliberate, structured learning. Start here before anything else.

Security thinking at the infrastructure layer. Application security, input validation, authentication, authorization, was familiar. Infrastructure security, IAM policies, network policies, secret management, supply chain security, required building a new mental model. The threat models are different, the habits required to reason about them are different.

Cost as a first-class concern. Application developers rarely think about compute costs as a design constraint. DevOps engineers think about it constantly. Learning to reason about instance types, reserved versus on-demand pricing, egress costs, and right-sizing required developing a new layer of awareness that simply wasn't part of my previous context.

The breadth of the tooling ecosystem. The DevOps tooling landscape is vast and changes fast. Kubernetes, Terraform, Ansible, ArgoCD, Helm, Prometheus, Grafana, Vault, Istio, Linkerd, Spacelift, each is a significant tool with its own learning curve. The developer ecosystem is also vast, but you tend to go deep in a smaller set of tools. DevOps requires comfort with breadth before depth, which is a different mode of learning.

Who Should Make This Switch

Not every developer should move into DevOps. But the following profile is a strong indicator that the transition would be rewarding:

You find yourself frustrated that you can't diagnose production failures. If the inability to follow a problem from the application layer into the infrastructure layer consistently bothers you, not just occasionally, but as a persistent professional frustration, that frustration is pointing at something real.

You are interested in the system, not just the software. There are developers who care deeply about the elegance of the code and developers who care deeply about the reliability of the system the code runs in. Both are valid, they tend to lead to different satisfying careers. If reliability, observability, and infrastructure design genuinely interest you, DevOps is a natural fit.

You want to have more leverage over production. Developers propose and build. DevOps engineers own the system that proposals get built on. If you want the kind of leverage that comes from owning the platform rather than just contributing to it, the switch offers that.

You are comfortable with ambiguity and breadth over depth. A DevOps career rewards generalists who go deep when needed. If you thrive in a context where today's problem might be a Kubernetes scheduling issue and tomorrow's problem is a TLS certificate misconfiguration, the variety will energize rather than exhaust you.

A Realistic Starting Point

If the above resonates, here is where to begin, not a comprehensive curriculum, but a practical first six months:

Months 1–2: Understand Linux deeply. Networking fundamentals (TCP/IP, DNS, HTTP, TLS). Write real shell scripts. Learn Git beyond the basics. These are the foundations that everything else sits on.

Months 3–4: Terraform for infrastructure provisioning. Docker for containerization. A real cloud provider (AWS is the most employable; GCP and Azure are legitimate alternatives). Build something and deploy it, not a tutorial, a real project.

Months 5–6: Kubernetes, not as a spectator, but as an operator. Set up a cluster, deploy workloads, configure networking, debug pod failures. CI/CD with GitHub Actions. Basic observability with Prometheus and Grafana.

By month six, you will have enough foundation to contribute meaningfully in a DevOps role and enough context to know which directions to specialize in. The learning continues well beyond six months, but so does it in software development.

Conclusion

The switch from software development to DevOps is not a step away from engineering. It is a step toward the full scope of what engineering means in a production system. The code you write is only part of the picture. The infrastructure it runs on, the pipelines that deliver it, the observability that monitors it, and the platform that enables other engineers to work on it, all of that is also engineering, and all of it matters.

I don't think every developer should make this switch. But I think the developers who are frustrated by half the picture, who want to understand the full system, own reliability end to end, and build the platform rather than just contribute to it, will find the transition more rewarding than they expect.

The incident that pushed me toward DevOps was frustrating at the time. Looking back, it was the most useful thing that happened to my career.

DevOps #CareerChange #SoftwareDevelopment #DevOpsEngineer #TechCareer

Multi-Environment Terraform: How to Structure Workspaces for Dev, Staging, and Production

ThankGod Chibugwum Obobo — Sun, 13 Sep 2026 16:00:00 +0000

Every infrastructure codebase eventually confronts the same problem, you need dev, staging, and production environments that are structurally identical but configured differently, different instance sizes, different replica counts, different domain names, different secrets, and critically, different state files that must never cross-contaminate.

How you solve this problem determines the maintainability of your infrastructure for years. Get it right and promoting a configuration change from dev to staging to production is a deliberate, reviewable, auditable workflow. Get it wrong and you have three copies of the same Terraform module drifting apart, environment-specific hacks embedded in shared code, and a lingering fear that terraform apply in the wrong directory will touch production.

Terraform offers two primary approaches: workspaces (single configuration, multiple state files) and directory-based layouts (separate directories per environment). Both are valid, neither is universally correct. Understanding their trade-offs, and when to combine them, is the skill this guide develops.

Approach 1 - Terraform Workspaces

Terraform workspaces allow a single configuration to maintain multiple independent state files. The workspace name is available as terraform.workspace inside configuration, enabling environment-specific branching:

# main.tf - single configuration, workspace-aware
locals {
  environment = terraform.workspace   # "dev", "staging", "production"

  config = {
    dev = {
      instance_type  = "t3.small"
      replica_count  = 1
      min_nodes      = 1
      max_nodes      = 3
    }
    staging = {
      instance_type  = "t3.medium"
      replica_count  = 2
      min_nodes      = 2
      max_nodes      = 5
    }
    production = {
      instance_type  = "m6i.large"
      replica_count  = 3
      min_nodes      = 3
      max_nodes      = 20
    }
  }
}

resource "aws_instance" "app_server" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = local.config[local.environment].instance_type

  tags = {
    Environment = local.environment
    ManagedBy   = "terraform"
  }
}

Working with workspaces:

# Create environments
terraform workspace new dev
terraform workspace new staging
terraform workspace new production

# List workspaces
terraform workspace list
#   default
# * dev
#   staging
#   production

# Switch and apply
terraform workspace select staging
terraform plan
terraform apply

# Each workspace has its own state file in the backend
# terraform.tfstate.d/dev/terraform.tfstate
# terraform.tfstate.d/staging/terraform.tfstate
# terraform.tfstate.d/production/terraform.tfstate

When Workspaces Work Well

Workspaces shine for environments that are structurally identical with minor configuration differences, the same resources, same modules, same topology, different sizes. A simple web application with three environments of the same architecture is an ideal workspace candidate.

Workspace Limitations

No access control between workspaces. Anyone with Terraform access to the configuration can apply to any workspace, including production. There is no native mechanism to require different approval processes for different workspaces.

Drift between environments is easy to introduce. A conditional block added for production (if terraform.workspace == "production") that grows over time creates structural differences that erode the "identical architecture" premise workspaces rely on.

Config-in-code environment branching becomes unwieldy. When local.config maps grow to cover 20+ configuration values across three environments, they become difficult to read and error-prone to maintain.

Approach 2 - Directory-Based Layout

The directory-based approach gives each environment its own Terraform root module, its own main.tf, variables.tf, outputs.tf, and backend configuration, while sharing infrastructure logic through reusable modules:

/infrastructure
  /modules                    ← shared, reusable infrastructure logic
    /vpc/
      main.tf
      variables.tf
      outputs.tf
    /eks/
      main.tf
      variables.tf
      outputs.tf
    /rds/
      main.tf
      variables.tf
      outputs.tf
  /environments               ← environment-specific root modules
    /dev/
      main.tf                 ← composes modules with dev config
      variables.tf
      outputs.tf
      backend.tf
      terraform.tfvars        ← dev-specific values
    /staging/
      main.tf
      variables.tf
      outputs.tf
      backend.tf
      terraform.tfvars
    /production/
      main.tf
      variables.tf
      outputs.tf
      backend.tf
      terraform.tfvars

Each environment directory is a complete, independent Terraform root module. Shared infrastructure logic lives in /modules and is consumed by all environments.

Module Design for Reuse

# modules/rds/main.tf - reusable RDS module
variable "identifier"          { type = string }
variable "instance_class"      { type = string }
variable "allocated_storage"   { type = number }
variable "environment"         { type = string }
variable "vpc_id"              { type = string }
variable "subnet_ids"          { type = list(string) }
variable "deletion_protection" { type = bool    default = false }
variable "backup_retention"    { type = number  default = 7 }
variable "multi_az"            { type = bool    default = false }

resource "aws_db_instance" "this" {
  identifier          = var.identifier
  engine              = "postgres"
  engine_version      = "16.1"
  instance_class      = var.instance_class
  allocated_storage   = var.allocated_storage
  storage_encrypted   = true
  deletion_protection = var.deletion_protection
  backup_retention_period = var.backup_retention
  multi_az            = var.multi_az

  db_subnet_group_name   = aws_db_subnet_group.this.name
  vpc_security_group_ids = [aws_security_group.rds.id]

  tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

output "endpoint"    { value = aws_db_instance.this.endpoint }
output "instance_id" { value = aws_db_instance.this.id }

Environment Root Modules

# environments/dev/main.tf
module "vpc" {
  source = "../../modules/vpc"

  name                 = "dev-vpc"
  cidr                 = "10.0.0.0/16"
  availability_zones   = ["us-east-1a", "us-east-1b"]
  private_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnet_cidrs  = ["10.0.101.0/24", "10.0.102.0/24"]
  environment          = "dev"
}

module "rds" {
  source = "../../modules/rds"

  identifier          = "orders-dev"
  instance_class      = "db.t3.micro"   # smallest for dev
  allocated_storage   = 20
  deletion_protection = false            # allow easy teardown in dev
  backup_retention    = 1               # minimal backups in dev
  multi_az            = false           # single-AZ for dev
  environment         = "dev"
  vpc_id              = module.vpc.vpc_id
  subnet_ids          = module.vpc.private_subnet_ids
}

# environments/production/main.tf
module "vpc" {
  source = "../../modules/vpc"

  name                 = "production-vpc"
  cidr                 = "10.2.0.0/16"
  availability_zones   = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnet_cidrs = ["10.2.1.0/24", "10.2.2.0/24", "10.2.3.0/24"]
  public_subnet_cidrs  = ["10.2.101.0/24", "10.2.102.0/24", "10.2.103.0/24"]
  environment          = "production"
}

module "rds" {
  source = "../../modules/rds"

  identifier          = "orders-production"
  instance_class      = "db.r6g.xlarge"   # production-grade
  allocated_storage   = 100
  deletion_protection = true              # never accidentally destroy
  backup_retention    = 30               # 30 days of backups
  multi_az            = true             # high availability
  environment         = "production"
  vpc_id              = module.vpc.vpc_id
  subnet_ids          = module.vpc.private_subnet_ids
}

Isolated Backend Configuration

Each environment gets its own backend, a separate S3 key ensures state files are completely independent:

# environments/dev/backend.tf
terraform {
  backend "s3" {
    bucket         = "your-org-terraform-state"
    key            = "environments/dev/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
  }
}

# environments/production/backend.tf
terraform {
  backend "s3" {
    bucket         = "your-org-terraform-state"
    key            = "environments/production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
  }
}

Separate state files with separate DynamoDB lock tables (or the same table with different keys) ensure no operation on dev can interfere with production state.

Variable Management Across Environments

terraform.tfvars per Environment

Each environment directory contains a terraform.tfvars with environment-specific values:

# environments/dev/terraform.tfvars
environment      = "dev"
aws_region       = "us-east-1"
app_version      = "2.4.1-dev"
alert_email      = "dev-alerts@your-org.com"

# environments/production/terraform.tfvars
environment      = "production"
aws_region       = "us-east-1"
app_version      = "2.4.1"
alert_email      = "pagerduty@your-org.com"

Sensitive Variables via Environment Variables

Secrets never appear in terraform.tfvars, inject them as TF_VAR_ prefixed environment variables at CI/CD runtime:

# In CI/CD pipeline — injected from secrets manager
export TF_VAR_db_password="$(vault kv get -field=password secret/production/rds)"
export TF_VAR_api_key="$(vault kv get -field=key secret/production/stripe)"

terraform apply -var-file=terraform.tfvars

# environments/production/variables.tf
variable "db_password" {
  type      = string
  sensitive = true   # prevents value from appearing in plan output or state
}

variable "api_key" {
  type      = string
  sensitive = true
}

The sensitive = true flag masks the variable value in terraform plan output, a critical safeguard when plans are posted to pull requests or logged in CI systems.

Choosing Between Workspaces and Directories

Neither approach is universally superior. Use this decision framework:

Factor	Workspaces	Directory-Based
Environments are structurally identical	Ideal	More boilerplate
Environments have significant config differences	Awkward branching	Clean separation
Different access controls per environment	Not supported natively	Enforce via CI/CD
Team size	Small	Medium to large
Number of environments	2–3	3+
Compliance requirement for state isolation	Soft isolation	Hard isolation
Risk of applying to wrong environment	Higher	Lower

Recommended default: directory-based layout for any environment where production carries business risk. The additional boilerplate is a worthwhile trade for explicit isolation, clear ownership, and CI/CD-enforced promotion workflows.

CI/CD Pipeline for Environment Promotion

The promotion workflow, change approved in dev, promoted to staging, promoted to production, is where directory-based layouts shine. Each environment is an independent pipeline target:

# .github/workflows/terraform-promote.yml
name: Terraform Environment Promotion

on:
  push:
    branches: [main]
    paths:
      - 'infrastructure/environments/**'
      - 'infrastructure/modules/**'

jobs:
  plan-dev:
    name: Plan — Dev
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan — Dev
        working-directory: infrastructure/environments/dev
        run: |
          terraform init
          terraform plan -var-file=terraform.tfvars -out=plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}

      - name: Upload plan artifact
        uses: actions/upload-artifact@v4
        with:
          name: dev-plan
          path: infrastructure/environments/dev/plan.tfplan

  apply-dev:
    name: Apply — Dev
    runs-on: ubuntu-latest
    needs: plan-dev
    environment: dev   # GitHub environment with optional approval
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: dev-plan
          path: infrastructure/environments/dev

      - name: Terraform Apply — Dev
        working-directory: infrastructure/environments/dev
        run: |
          terraform init
          terraform apply plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}

  plan-staging:
    name: Plan — Staging
    runs-on: ubuntu-latest
    needs: apply-dev   # staging only plans after dev applies successfully
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan — Staging
        working-directory: infrastructure/environments/staging
        run: |
          terraform init
          terraform plan -var-file=terraform.tfvars -out=plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}

      - name: Upload plan artifact
        uses: actions/upload-artifact@v4
        with:
          name: staging-plan
          path: infrastructure/environments/staging/plan.tfplan

  apply-staging:
    name: Apply — Staging
    runs-on: ubuntu-latest
    needs: plan-staging
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: staging-plan
          path: infrastructure/environments/staging

      - name: Terraform Apply — Staging
        working-directory: infrastructure/environments/staging
        run: |
          terraform init
          terraform apply plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}

  plan-production:
    name: Plan — Production
    runs-on: ubuntu-latest
    needs: apply-staging
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan — Production
        working-directory: infrastructure/environments/production
        run: |
          terraform init
          terraform plan -var-file=terraform.tfvars -out=plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}

      - name: Upload plan artifact
        uses: actions/upload-artifact@v4
        with:
          name: production-plan
          path: infrastructure/environments/production/plan.tfplan

  apply-production:
    name: Apply — Production
    runs-on: ubuntu-latest
    needs: plan-production
    environment: production   # requires mandatory human approval in GitHub Environments
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: production-plan
          path: infrastructure/environments/production

      - name: Terraform Apply — Production
        working-directory: infrastructure/environments/production
        run: |
          terraform init
          terraform apply plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}

Key pipeline design decisions:

Separate AWS credentials per environment. Dev, staging, and production each use their own IAM credentials, a misconfigured production run cannot accidentally use dev credentials, and vice versa. Scope each credential set to only the AWS resources that environment manages.

Plan before apply, always. Each environment plans first and saves the plan as an artifact. The apply step uses the saved plan, ensuring exactly what was reviewed is what gets applied. A new commit during the pipeline cannot change what production applies.

Production requires human approval. The environment: production GitHub Environment configuration requires designated reviewers to approve before the production apply runs. No automated path to production exists without human sign-off.

Production Safeguards

Layer these protections on your production environment to prevent accidental or unauthorized changes:

Deletion Protection on Critical Resources

# Never allow these to be destroyed via Terraform
resource "aws_db_instance" "production" {
  # ...
  deletion_protection = true
}

resource "aws_s3_bucket" "production_data" {
  # ...
}

resource "aws_s3_bucket_versioning" "production_data" {
  bucket = aws_s3_bucket.production_data.id
  versioning_configuration { status = "Enabled" }
}

Prevent Destroy with Lifecycle Rules

resource "aws_eks_cluster" "production" {
  # ...
  lifecycle {
    prevent_destroy = true   # plan fails if this resource would be destroyed
  }
}

S3 Backend with Object Locking

# Prevent state file deletion or tampering
resource "aws_s3_bucket_object_lock_configuration" "state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    default_retention {
      mode  = "COMPLIANCE"
      days  = 90
    }
  }
}

Terraform Sentinel Policies (Terraform Cloud/Enterprise)

For teams on Terraform Cloud or Enterprise, Sentinel policies enforce compliance rules before any plan can apply:

# sentinel/require-tags.sentinel
import "tfplan/v2" as tfplan

required_tags = ["Environment", "Team", "CostCenter", "ManagedBy"]

# Deny any resource missing required tags
deny_resources = filter tfplan.resource_changes as _, rc {
  rc.mode is "managed" and
  rc.change.actions contains "create" and
  any required_tags as tag {
    not rc.change.after.tags[tag] is defined
  }
}

main = rule { length(deny_resources) is 0 }

Module Versioning for Stability

As your module library grows, version-pin module references to prevent unexpected changes when modules are updated:

# environments/production/main.tf - pin module versions
module "rds" {
  source  = "git::https://github.com/your-org/infrastructure.git//modules/rds?ref=v2.3.1"
  # Pinned to v2.3.1, production won't break when v3.0.0 is released
  # ...
}

Or with the Terraform registry:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows patch updates, blocks major version changes
  # ...
}

Dev and staging can use looser version constraints or ref=main to test new module versions before pinning production to them, the directory structure makes this environment-specific pinning natural.

Common Pitfalls to Avoid

Shared state between environments. A single state file for all environments is not multi-environment Terraform, it is a disaster waiting to happen. Always use separate state backends per environment.

Hardcoded environment names in modules. Modules should accept environment as a variable, not reference it by name. A module that checks var.environment == "production" to enable deletion protection is testing the wrong thing, expose deletion_protection = bool and let the caller decide.

No plan review step for production. Applying Terraform to production without reviewing the plan is the infrastructure equivalent of merging code without reading the diff. The saved plan artifact pattern ensures what is reviewed is what is applied.

Skipping terraform validate and terraform fmt in CI. These run in seconds and catch formatting errors, type mismatches, and undefined variables before plan runs. Add them as the first step of every pipeline job.

Over-engineering with workspaces and directories simultaneously. Using both approaches in the same codebase, workspaces within directory-based environments, creates confusion about where state lives and which isolation mechanism is authoritative. Pick one approach per infrastructure boundary and be consistent.

Conclusion

Multi-environment Terraform is not primarily a technical problem, it is a discipline problem. The tooling makes isolation possible, the structure makes it maintainable, the CI/CD pipeline makes it safe. Directory-based layouts with isolated backends, environment-specific tfvars, separate CI credentials, and a plan-before-apply promotion workflow give you an infrastructure management system you can operate confidently at scale.

The directory-based approach requires more upfront structure than workspaces. It pays that cost back in clarity, every engineer knows exactly which directory controls which environment, exactly where the configuration differences live, and exactly what needs to be reviewed before any change reaches production.

Treat infrastructure environments the same way you treat code environments, isolated, explicitly promoted, never shared, always reviewed before production.

Terraform #InfrastructureAsCode #DevOps #MultiEnvironment #CloudInfrastructure

GitOps in Practice: How to Manage Kubernetes State with ArgoCD and Helm

ThankGod Chibugwum Obobo — Sun, 06 Sep 2026 17:00:00 +0000

Kubernetes configuration drift is a silent killer of operational confidence. A developer applies a quick fix directly with kubectl during an incident. A CI pipeline pushes an update that partially applies before timing out. An engineer edits a ConfigMap in production to test a theory and forgets to update the repository. Six weeks later, nobody is sure what is actually running, the cluster state has diverged from the repository that supposedly defines it.

GitOps solves this by making Git the single source of truth for cluster state. Every change, application version, configuration, replica count, ingress rule, is expressed as a commit. The cluster continuously reconciles toward that committed state. Unauthorized changes are detected and can be automatically reverted. The audit trail is your Git log.

ArgoCD is the most widely adopted GitOps controller for Kubernetes. Combined with Helm for templated application packaging, it provides a complete GitOps platform, continuous reconciliation, multi-environment promotion, RBAC-controlled access, and a UI that shows the real-time gap between desired and actual cluster state.

This guide covers how to implement a production-grade GitOps workflow with ArgoCD and Helm, from installation and Application setup through the App of Apps pattern, sync policies, secrets management, and progressive delivery.

The GitOps Principles in Practice

Before implementation, understand the four principles GitOps enforces:

Declarative: the entire system state is expressed declaratively in Git. Not scripts, not imperative commands, YAML manifests and Helm values that describe the desired end state.

Versioned and immutable: Git history is the audit trail. Every state change is a commit with author, timestamp, and diff. Rolling back is a revert commit, not a manual procedure.

Pulled automatically: the cluster pulls configuration from Git, rather than CI/CD pushing configuration to the cluster. This eliminates the need for CI systems to have cluster credentials, a significant security improvement.

Continuously reconciled: ArgoCD continuously compares desired state (Git) against actual state (cluster) and alerts on drift. With auto-sync enabled, it corrects drift automatically.

Step 1 - Repository Structure for GitOps

A clear repository structure is the foundation of a maintainable GitOps setup. The most common pattern separates application code from deployment configuration:

/gitops-repo                    ← dedicated GitOps repository
  /apps                         ← ArgoCD Application manifests
    /root-app.yaml              ← App of Apps root
    /platform/
      platform-apps.yaml        ← platform-level applications
    /teams/
      team-alpha-apps.yaml
      team-beta-apps.yaml
  /charts                       ← Helm charts for your services
    /orders-service/
      Chart.yaml
      values.yaml               ← default values
      templates/
        deployment.yaml
        service.yaml
        ingress.yaml
        hpa.yaml
  /envs                         ← environment-specific values
    /staging/
      orders-service.yaml       ← staging overrides
    /production/
      orders-service.yaml       ← production overrides
  /infrastructure               ← cluster-level resources
    /cert-manager/
    /ingress-nginx/
    /monitoring/

Key design decisions:
Separate application charts (/charts) from environment values (/envs). The chart defines the template, the environment values define the configuration. Promoting from staging to production is a values change, not a chart change.

Keep all ArgoCD Application manifests in Git (/apps). ArgoCD manages itself, the App of Apps pattern means ArgoCD's own configuration is version-controlled and reconciled like everything else.

Step 2 - Installing ArgoCD

# Create ArgoCD namespace and install
kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all components to be ready
kubectl wait --for=condition=available --timeout=300s \
  deployment/argocd-server -n argocd

# Retrieve initial admin password
kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath='{.data.password}' | base64 -d

# Port-forward the UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

For production installations, manage ArgoCD itself via Helm and the ArgoCD Helm chart, enabling version-controlled ArgoCD configuration and upgrades:

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --values argocd-values.yaml

Step 3 - Creating Your First ArgoCD Application

An ArgoCD Application is the core resource, it links a Git source to a Kubernetes destination and defines how reconciliation should behave:

# apps/orders-service-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-service-staging
  namespace: argocd
  labels:
    environment: staging
    team: backend
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade delete on app removal
spec:
  project: backend-team

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: charts/orders-service
    helm:
      valueFiles:
        - ../../envs/staging/orders-service.yaml  # relative to chart path

  destination:
    server: https://kubernetes.default.svc
    namespace: orders-staging

  syncPolicy:
    automated:
      prune: true         # delete resources removed from Git
      selfHeal: true      # revert unauthorized manual changes
      allowEmpty: false   # never sync an empty resource set (safety guard)
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - RespectIgnoreDifferences=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The selfHeal: true option is the GitOps enforcement mechanism, if someone runs kubectl edit deployment orders-service directly, ArgoCD detects the drift within 3 minutes and reverts it to match Git. This is what makes Git the actual source of truth rather than a suggestion.

Step 4 - Helm Values for Multi-Environment Configuration

Structure Helm values for environment-specific overrides:

# charts/orders-service/values.yaml — defaults
replicaCount: 1
image:
  repository: your-org/orders-service
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80
  targetPort: 3000

ingress:
  enabled: false
  host: ""

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70

env: {}

# envs/staging/orders-service.yaml — staging overrides
replicaCount: 1

image:
  tag: "2.4.1-staging"    # pinned to staging image tag

ingress:
  enabled: true
  host: orders.staging.your-domain.com

resources:
  requests:
    cpu: 50m
    memory: 64Mi

env:
  NODE_ENV: staging
  DATABASE_URL: postgresql://staging-db:5432/orders
  LOG_LEVEL: debug

# envs/production/orders-service.yaml — production overrides
replicaCount: 3

image:
  tag: "2.4.1"            # pinned to production image tag

ingress:
  enabled: true
  host: orders.your-domain.com

resources:
  requests:
    cpu: 100m
    memory: 224Mi
  limits:
    cpu: 500m
    memory: 360Mi

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20

env:
  NODE_ENV: production
  LOG_LEVEL: warn

Image tags are pinned, never latest and promotion from staging to production is a single-line diff in envs/production/orders-service.yaml. That diff is a pull request, code-reviewed, and merged. ArgoCD detects the merge and applies the change to the production cluster automatically.

Step 5 - The App of Apps Pattern

Managing dozens of ArgoCD Application resources individually doesn't scale. The App of Apps pattern treats the collection of applications as a managed set, a root Application that points to a directory of Application manifests, and ArgoCD manages them all:

# apps/root-app.yaml — the root of everything
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: apps            # ArgoCD watches this directory for Application manifests

  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply this single manifest manually once:

kubectl apply -f apps/root-app.yaml

ArgoCD discovers every Application manifest in the apps/ directory and creates/manages them all. Adding a new application is adding a new YAML file to the apps/ directory and committing. Removing an application is removing its manifest, with prune: true, ArgoCD cleans up the cluster resources automatically.

This is the GitOps pattern in its purest form, the cluster configuration, including its own management configuration, is fully defined in and controlled by Git.

Step 6 - Projects and RBAC for Multi-Team Environments

ArgoCD AppProject resources define boundaries between teams, controlling which repositories, clusters, and namespaces each team's Applications can target:

# infrastructure/argocd/projects/backend-team.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: backend-team
  namespace: argocd
spec:
  description: Backend engineering team applications

  # Which Git repositories this project can deploy from
  sourceRepos:
    - https://github.com/your-org/gitops-repo

  # Which clusters and namespaces this project can deploy to
  destinations:
    - server: https://kubernetes.default.svc
      namespace: orders-*     # wildcard — any namespace starting with orders-
    - server: https://kubernetes.default.svc
      namespace: users-*

  # Which Kubernetes resource types this project can manage
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace

  namespaceResourceWhitelist:
    - group: 'apps'
      kind: Deployment
    - group: ''
      kind: Service
    - group: 'networking.k8s.io'
      kind: Ingress
    - group: 'autoscaling'
      kind: HorizontalPodAutoscaler

  # Deny deploying privileged containers
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota

  roles:
    - name: developer
      description: Read-only access plus sync trigger
      policies:
        - p, proj:backend-team:developer, applications, get, backend-team/*, allow
        - p, proj:backend-team:developer, applications, sync, backend-team/*, allow
      groups:
        - backend-developers     # maps to your IdP group

    - name: lead
      description: Full access to backend team applications
      policies:
        - p, proj:backend-team:lead, applications, *, backend-team/*, allow
      groups:
        - backend-leads

RBAC is enforced at the ArgoCD level, developers can trigger syncs and view application state, but cannot modify Application resources, change sync policies, or deploy to namespaces outside their project's boundaries.

Step 7 - Secrets Management with ArgoCD

ArgoCD should never store secrets in Git, even encrypted. The two recommended approaches:

Sealed Secrets

# Install Sealed Secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Encrypt a secret for Git storage
kubectl create secret generic orders-db-credentials \
  --from-literal=DATABASE_URL=postgresql://user:password@db:5432/orders \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > charts/orders-service/templates/db-secret.sealed.yaml

The SealedSecret in Git can only be decrypted by the Sealed Secrets controller running in your specific cluster, safe to commit.

External Secrets Operator with Vault

For teams already using HashiCorp Vault (as covered in our secrets management guide):

# charts/orders-service/templates/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: orders-service-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: orders-service-env
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: secret/production/orders-service/database
        property: url
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: secret/production/orders-service/stripe
        property: api_key

ArgoCD deploys the ExternalSecret resource, the External Secrets Operator fetches the actual values from Vault and creates the Kubernetes Secret. Vault credentials never appear in Git.

Step 8 - Progressive Delivery with ArgoCD Rollouts

For production deployments where blue/green or canary strategies are required, Argo Rollouts extends ArgoCD with progressive delivery capabilities:

# charts/orders-service/templates/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: orders-service
spec:
  replicas: 5
  selector:
    matchLabels:
      app: orders-service
  template:
    spec:
      containers:
        - name: orders-service
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

  strategy:
    canary:
      steps:
        - setWeight: 10           # route 10% of traffic to new version
        - pause: { duration: 5m } # wait 5 minutes — observe error rate
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100          # full traffic if no issues detected

      # Automatically roll back if error rate exceeds threshold
      analysis:
        templates:
          - templateName: success-rate-check
        startingStep: 1
        args:
          - name: service-name
            value: orders-service

# infrastructure/argocd/analysis-templates/success-rate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.98
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus-server.monitoring.svc.cluster.local
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

The canary automatically rolls back if the success rate drops below 98% for three consecutive minutes, combining GitOps promotion (the image tag change in Git) with data-driven delivery (the rollback decision from Prometheus).

Step 9 - Handling Sync Waves and Dependencies

Some resources must exist before others, namespaces before deployments, CRDs before custom resources, Secrets before Deployments that mount them. ArgoCD sync waves control this ordering:

# Apply this first (wave -1) — before all other resources
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
# Apply this second (wave 0, default)
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Apply this last (wave 1) — after secrets and config are ready
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Use sync waves for:

Database migrations (wave: 0) before the application deployment (wave: 1)
CRD installation (wave: -1) before custom resources (wave: 0)
Secrets (wave: -1) before Deployments that reference them (wave: 0)

Common Pitfalls to Avoid

Using latest image tags. ArgoCD reconciles to the state in Git. If latest resolves to a different image without a Git commit, ArgoCD won't detect it as a change and won't reflect it in the UI. Always pin image tags and update them via Git commits.

Auto-sync without selfHeal in production. Auto-sync without selfHeal applies changes from Git but doesn't revert unauthorized manual changes. Enable both for full GitOps enforcement in production environments.

Storing unencrypted secrets in Git. Even in private repositories, plaintext secrets in Git are a security risk. Use Sealed Secrets or External Secrets Operator, there is no legitimate reason for plaintext credentials in a GitOps repository.

No sync windows for production. Configure sync windows to restrict when ArgoCD can apply changes to production clusters, preventing automated syncs during peak traffic hours or business-critical periods:

syncPolicy:
  syncWindows:
    - kind: allow
      schedule: '0 2-6 * * 1-5'  # weekdays 2am-6am only
      duration: 4h
      applications: ['*']

Ignoring ArgoCD health status. ArgoCD tracks application health (Healthy, Progressing, Degraded) based on Kubernetes resource conditions. A Degraded health status on a synced application means the desired state was applied but the application isn't healthy, a deployment with crashing pods, for example. Treat health status with the same urgency as a failed deployment.

Conclusion

GitOps with ArgoCD and Helm is not just a deployment mechanism, it is a complete operational model that eliminates an entire category of operational risk. Cluster drift, unclear ownership, undocumented changes, and failed rollbacks are structural problems that Git-driven reconciliation solves at the architecture level rather than the process level.

The App of Apps pattern scales the model across dozens of services and teams. Projects and RBAC enforce team boundaries without ticket-based access control. Sync policies enforce Git as the source of truth, not a guideline. Progressive delivery with Argo Rollouts brings data-driven safety to the final mile of deployment.

The investment is real, repository structure, RBAC design, secrets management, and sync policy tuning require deliberate engineering. But once the platform is in place, the operational discipline it enforces is automatic rather than cultural, which is the only kind of discipline that holds at scale.

Running multiple Kubernetes clusters across environments or cloud providers? ArgoCD's multi-cluster support with ApplicationSets scales the App of Apps pattern to fleet-level management without per-cluster configuration duplication.

GitOps #ArgoCD #Helm #Kubernetes #DevOps

Kubernetes Cost Optimization: How to Right-Size Workloads with Goldilocks and VPA

ThankGod Chibugwum Obobo — Sun, 30 Aug 2026 17:00:00 +0000

Kubernetes resource requests and limits are one of the most consequential configuration decisions in cloud-native infrastructure, and one of the least data-driven. Most teams set them once during initial deployment, based on estimates or copied from a Stack Overflow answer, and never revisit them. The result is predictable, workloads are systematically over-provisioned to avoid OOM kills, node utilization stays at 20–40%, and cloud bills grow faster than the business demands it.

The irony is that right-sizing is not guesswork, it's a measurement problem. Every Kubernetes cluster already collects the utilization data needed to set accurate resource requests. The gap is tooling that surfaces those measurements as actionable recommendations without requiring engineers to manually analyze Prometheus queries for every workload.

Goldilocks and the Vertical Pod Autoscaler (VPA) close that gap. VPA observes actual resource consumption and computes right-sized recommendations. Goldilocks surfaces those recommendations in a readable dashboard, organized by namespace, with diff-ready output that shows exactly what to change in your Helm values or Kubernetes manifests.

This guide covers how to deploy and configure both tools, interpret their recommendations, implement right-sizing changes safely, and integrate continuous cost optimization into your engineering workflow.

The Resource Request Problem

To understand why right-sizing matters, start with how Kubernetes uses resource requests:

Resource requests are what the Kubernetes scheduler uses to decide which node to place a pod on. A pod requesting 500m CPU and 512Mi memory will only be scheduled on a node with that much unallocated capacity.

Resource limits cap what a pod can actually consume. A pod that exceeds its CPU limit is throttled. A pod that exceeds its memory limit is OOM killed.

The consequence of over-provisioning requests is that capacity is reserved on nodes that the workload never uses. A pod requesting 2000m CPU that actually uses 200m is holding a slot for 1800m CPU that no other pod can use, you're paying for 10x the compute the workload needs.

The consequence of under-provisioning limits is that pods get OOM killed, generating false incidents. CPU throttling degrades latency without generating visible errors, a silent performance problem that many teams misdiagnose.

Accurate resource requests and limits require knowing what workloads actually consume under real load, which requires measurement, not estimation.

Architecture: How VPA and Goldilocks Work Together

Vertical Pod Autoscaler (VPA) is a Kubernetes controller that observes pod resource utilization over time and computes recommendation ranges:

Lower Bound - the minimum request that prevents resource starvation
Target - the recommended request for typical workload behavior
Upper Bound - the maximum consumed under peak conditions

VPA has three modes: Auto (applies recommendations automatically, with pod restarts), Recreate (applies on pod recreation only), and Off (computes recommendations but applies nothing). For most production environments, Off mode (recommendations without automatic application) is the right starting point.

Goldilocks wraps VPA's Off mode with a namespace-level dashboard that aggregates VPA recommendations across all deployments in a namespace. It creates a VerticalPodAutoscaler object in Off mode for every deployment it monitors, collects the resulting recommendations, and presents them in a UI showing current requests vs. recommended requests with quality-of-service (QoS) class implications.

The workflow is:

Workloads run → Metrics Server collects utilization →
VPA Recommender analyzes history →
Goldilocks reads VPA recommendations →
Dashboard shows what to change →
Engineers apply changes to Helm values / manifests →
Next cycle validates the new settings

Step 1 - Install Metrics Server

VPA depends on the Kubernetes Metrics Server for resource utilization data. Verify it is running:

kubectl top nodes
kubectl top pods -A

If kubectl top returns an error, install Metrics Server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Verify
kubectl get deployment metrics-server -n kube-system

Step 2 - Install the Vertical Pod Autoscaler

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Install VPA components
./hack/vpa-up.sh

# Verify all three VPA components are running
kubectl get pods -n kube-system | grep vpa
# vpa-admission-controller-xxx   1/1   Running
# vpa-recommender-xxx            1/1   Running
# vpa-updater-xxx                1/1   Running

Alternatively, install via Helm:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa \
  --namespace kube-system \
  --set recommender.enabled=true \
  --set updater.enabled=false \    # disable auto-updates - use Off mode
  --set admissionController.enabled=false

The VPA recommender is the only component required for Goldilocks, it computes recommendations. The updater and admission controller are needed only for automatic recommendation application, which we'll address separately.

Step 3 - Install Goldilocks

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace \
  --set dashboard.enabled=true \
  --set dashboard.service.type=ClusterIP

Verify Goldilocks is running:

kubectl get pods -n goldilocks
# goldilocks-controller-xxx    1/1   Running
# goldilocks-dashboard-xxx     1/1   Running

Step 4 - Enable Goldilocks Per Namespace

Goldilocks only monitors namespaces explicitly labeled for analysis. Label the namespaces you want to optimize:

# Enable Goldilocks for your application namespaces
kubectl label namespace orders-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace users-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace payments-service goldilocks.fairwinds.com/enabled=true

Once labeled, Goldilocks automatically creates a VerticalPodAutoscaler object in Off mode for every Deployment in the namespace. VPA begins observing resource utilization immediately.

Important: Allow at least 24–72 hours of observation before acting on recommendations. VPA's accuracy improves with more history, recommendations based on 15 minutes of data reflect startup behavior, not steady-state production utilization.

Step 5 - Accessing the Goldilocks Dashboard

# Port-forward the dashboard
kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80

# Open http://localhost:8080

The dashboard displays, per namespace and per deployment:

Current requests and limits - what your manifests currently specify
VPA recommendations - Lower Bound, Target, and Upper Bound for CPU and memory
QoS class impact - whether your current and recommended settings result in Guaranteed, Burstable, or BestEffort quality-of-service
Helm values diff - the exact resources block to paste into your Helm values file

A typical recommendation output looks like:

# Current (what you have)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "1Gi"

# Goldilocks recommended (VPA Target)
resources:
  requests:
    cpu: "87m"
    memory: "179Mi"
  limits:
    cpu: "262m"
    memory: "358Mi"

This workload is requesting 5.7x more CPU than it actually uses. Applying the recommendation would reduce its node footprint by ~83%, allowing the cluster to either schedule significantly more workloads on existing nodes or rightsize to fewer nodes.

Step 6 - Interpreting Recommendations Safely

Goldilocks recommendations are starting points, not gospel. Apply these interpretation rules before implementing changes:

Use the Upper Bound for Memory Limits

VPA's memory Target represents typical consumption. Set your memory limit to the Upper Bound, the peak observed usage, to avoid OOM kills during traffic spikes. Memory limits below observed peak are the primary cause of OOM kills.

resources:
  requests:
    memory: "179Mi"      # VPA Target → use as request
  limits:
    memory: "358Mi"      # VPA Upper Bound → use as limit (not Target)

Add a Safety Buffer for Memory Requests

For stateful or memory-sensitive workloads, add a 20–30% buffer above the VPA Target for memory requests:

# VPA Target: 179Mi → apply with 25% buffer
requests:
  memory: "224Mi"    # 179 × 1.25 = 223.75 → round up

Be Conservative with CPU Limits

Unlike memory, CPU throttling does not kill pods, it degrades latency. Setting CPU limits close to observed peak reduces burst headroom. Consider setting CPU limits to the Upper Bound and leaving headroom for traffic spikes:

requests:
  cpu: "87m"         # VPA Target - accurate for scheduling
limits:
  cpu: "500m"        # Upper Bound + buffer - prevents pathological consumption

Ignore Recommendations for Stateful Workloads

VPA recommendations for StatefulSets (databases, message brokers, caches) should be applied with extra caution, memory consumption patterns for stateful workloads are often highly variable and spike during compaction, reindexing, or failover operations. Treat VPA recommendations for stateful workloads as lower bounds, not targets.

Step 7 - Implementing Changes Safely

Never apply all VPA recommendations at once. A phased approach minimizes risk:

Phase 1 - Non-production environments first. Apply recommendations to staging and development. Observe for one week. Verify no OOM kills, no CPU throttling-related latency degradation, no unexpected behavior.

Phase 2 - Lowest-traffic production workloads. Apply to internal tools, admin services, and background workers, workloads with the lowest blast radius if a recommendation is wrong.

Phase 3 - Core application services. Apply to API services and user-facing workloads after observing Phase 2 outcomes. Monitor error rates, latency p95/p99, and OOM kill rates for 48 hours after each change.

For Helm-managed deployments, apply recommendations via values override:

# values/production/orders-service.yaml — after Goldilocks review
resources:
  requests:
    cpu: "100m"       # up from VPA's 87m — minor safety buffer
    memory: "224Mi"   # VPA Target (179Mi) + 25% buffer
  limits:
    cpu: "500m"       # VPA Upper Bound + margin
    memory: "360Mi"   # VPA Upper Bound

helm upgrade orders-service ./charts/orders-service \
  -f values/production/orders-service.yaml \
  --namespace orders-service

Step 8 - VPA Auto Mode for Non-Critical Workloads

For non-critical workloads (batch jobs, internal tools, development namespaces) where pod restarts are acceptable, VPA's Auto mode applies recommendations without manual intervention:

# vpa/orders-worker-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-worker-vpa
  namespace: orders-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-worker
  updatePolicy:
    updateMode: "Auto"   # applies recommendations automatically
  resourcePolicy:
    containerPolicies:
      - containerName: orders-worker
        minAllowed:
          cpu: "50m"
          memory: "64Mi"
        maxAllowed:
          cpu: "2000m"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]

The minAllowed and maxAllowed bounds prevent VPA from recommending values outside your operational comfort zone, a safeguard against runaway recommendations if a workload behaves unusually during the observation window.

Step 9 - CI/CD Integration for Continuous Right-Sizing

Make right-sizing a recurring engineering practice rather than a one-time event by integrating Goldilocks into your workflow:

# .github/workflows/cost-review.yml
name: Monthly Cost Optimization Review

on:
  schedule:
    - cron: '0 9 1 * *'   # first day of each month at 9am

jobs:
  generate-recommendations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Export Goldilocks recommendations
        run: |
          kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80 &
          sleep 5

          # Export recommendations via Goldilocks API
          curl -s http://localhost:8080/api/v1/namespaces \
            | jq '.[] | {namespace: .name, recommendations: .recommendations}' \
            > recommendations.json

      - name: Create GitHub Issue with recommendations
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const recs = JSON.parse(fs.readFileSync('recommendations.json'));

            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Monthly Cost Optimization Review — ${new Date().toISOString().slice(0, 7)}`,
              body: `## Goldilocks Recommendations\n\n\`\`\`json\n${JSON.stringify(recs, null, 2)}\n\`\`\`\n\nReview and apply relevant recommendations following the right-sizing runbook.`,
              labels: ['cost-optimization', 'infrastructure'],
              assignees: ['platform-team'],
            });

A monthly GitHub issue with current recommendations creates a lightweight, trackable cost optimization cadence, without requiring engineers to proactively access the dashboard.

Measuring the Impact

Track right-sizing outcomes with these Grafana/Prometheus metrics:

# Node CPU utilization — target 60-70% for cost efficiency
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (node) * 100

# Pod CPU request vs actual usage (request efficiency)
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, namespace)

# OOM kill rate — should be zero after right-sizing
sum(increase(kube_pod_container_status_restarts_total[24h])) by (pod, namespace)

# Memory request efficiency
sum(container_memory_working_set_bytes) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="memory"}) by (pod, namespace)

A request efficiency ratio below 0.3 (pods using less than 30% of requested resources) is a strong signal of over-provisioning. A ratio above 0.85 sustained means requests are too tight, burst traffic risks throttling or OOM.

Common Pitfalls to Avoid

Acting on fresh recommendations. VPA recommendations based on less than 24 hours of data reflect startup patterns, not steady-state behavior. Wait at least 72 hours before applying recommendations to production workloads.

Setting memory limits equal to requests. This creates Guaranteed QoS class pods, Kubernetes evicts Burstable pods before Guaranteed ones, which sounds good but means any memory spike above the limit results in an OOM kill with no burst headroom. Set limits meaningfully above requests.

Applying VPA Auto mode to HPA-managed deployments. VPA and HPA conflict when both target the same metric. If you use HPA for CPU-based scaling, configure VPA to manage only memory: controlledResources: ["memory"].

Ignoring the QoS class implications. Guaranteed QoS (requests = limits) gets the highest scheduling priority but no burst headroom. Burstable QoS (requests < limits) allows bursting but can be evicted under node pressure. Understand which class your workloads land in after right-sizing.

Right-sizing without load testing. VPA recommendations are based on historical traffic. If your staging environment doesn't reflect production load patterns, its recommendations don't reflect production requirements. Right-size production workloads using production observation data.

Conclusion

Kubernetes cost optimization is not a one-time project, it's an ongoing engineering discipline driven by measurement rather than estimation. VPA provides the utilization data. Goldilocks makes that data actionable. The right-sizing process, observe, recommend, review, apply, measure, is the feedback loop that keeps cluster efficiency high as workloads evolve.

The typical outcome of a systematic right-sizing initiative is a 30–50% reduction in resource requests, translating directly to fewer nodes needed to run the same workloads, and a proportional reduction in cloud spend. More importantly, accurate resource requests improve scheduling density, reduce node pressure evictions, and eliminate the silent CPU throttling that degrades latency without generating visible errors.

Start with one namespace. Label it for Goldilocks, wait 72 hours, review the dashboard, and apply the most conservative recommendations first. The data will tell you what to do, you just have to make time to look at it.

Using Karpenter for node provisioning alongside VPA for pod right-sizing? The combination is particularly effective, Karpenter provisions exactly the node types needed for right-sized pod requests, eliminating both over-provisioned pods and over-provisioned nodes simultaneously.

Kubernetes #CostOptimization #Goldilocks #VPA #CloudCost

Durable Execution with Temporal.io: Building Workflows That Never Lose State

ThankGod Chibugwum Obobo — Sun, 23 Aug 2026 17:00:00 +0000

Distributed systems fail in partial and unpredictable ways. A payment workflow that processes a charge, updates inventory, sends a confirmation email, and notifies a fulfilment service has seven places to fail, and no built-in mechanism to pick up where it left off. The traditional response is a combination of message queues, idempotency keys, retry logic, state machines stored in a database, and compensating transactions, each adding complexity, each requiring its own failure handling, and each representing a subtle opportunity for state to be lost or corrupted.

Temporal.io is built on a different premise, what if your workflow code simply continued executing after a failure, a restart, or a crash, exactly where it left off, with all local variables intact, all function calls remembered, and no manual state management required?

This is durable execution, the guarantee that a workflow's progress is preserved indefinitely, regardless of what happens to the infrastructure running it. Temporal achieves this by recording every workflow event to a persistent log and replaying the workflow's history to reconstruct state after any failure. From the developer's perspective, you write ordinary sequential code. Temporal makes it fault-tolerant.

This guide covers Temporal's core concepts and shows how to implement durable workflows in TypeScript, including activities, retries, signals, timers, and child workflows, for the kinds of long-running business processes that break traditional queue-based approaches.

How Temporal Works: The Core Model

Understanding Temporal's execution model upfront prevents confusion later.

Workflows are deterministic functions that orchestrate work. They define what happens and in what order. Workflow code must be deterministic, the same inputs must always produce the same sequence of events, because Temporal replays workflow history to reconstruct state after failures.

Activities are non-deterministic units of work, the things that actually talk to the outside world, database writes, API calls, file operations, sending emails. Activities are executed by workers, can be retried independently, and their results are recorded in the workflow history.

Workers are long-running processes that poll Temporal's task queues and execute workflow and activity code. You run workers, Temporal runs the orchestration.

The Temporal Server persists workflow state, manages task queues, handles timers, and routes signals. It is the durable backbone, workers are stateless and replaceable.

The key insight: workflow code never talks to the outside world directly. It schedules activities and waits for results. Temporal records the activity result to the event history. If the worker dies before the workflow advances, Temporal replays the history, skipping already-completed activities using their recorded results, and the workflow continues from where it stopped. The activity does not re-execute. No duplicate charges. No double emails.

Step 1 - Installation and Project Setup

# Create a new TypeScript project
mkdir temporal-demo && cd temporal-demo
npm init -y
npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity
npm install -D typescript ts-node @types/node

# Start Temporal development server (requires Docker)
docker run --rm -p 7233:7233 -p 8080:8080 temporalio/auto-setup:latest

Project structure:

/temporal-demo
  /src
    /workflows
      order-fulfillment.workflow.ts
      subscription-billing.workflow.ts
    /activities
      payment.activities.ts
      inventory.activities.ts
      notification.activities.ts
    worker.ts        ← registers and runs workflows + activities
    client.ts        ← starts workflows from your application

Step 2 - Defining Activities

Activities are where side effects live. Each activity is a plain async function, no Temporal-specific decorators required:

// src/activities/payment.activities.ts
import { ApplicationFailure } from '@temporalio/activity';

export interface ChargeResult {
  transactionId: string;
  amount: number;
  status: 'succeeded' | 'failed';
}

export async function chargeCustomer(
  customerId: string,
  amount: number,
  idempotencyKey: string,
): Promise<ChargeResult> {
  try {
    // Call your payment provider (Stripe, etc.)
    const charge = await stripeClient.charges.create({
      customer: customerId,
      amount,
      currency: 'usd',
      idempotency_key: idempotencyKey,   // prevent duplicate charges on retry
    });

    return {
      transactionId: charge.id,
      amount: charge.amount,
      status: 'succeeded',
    };
  } catch (error) {
    if (error.code === 'card_declined') {
      // Non-retryable failure — throw ApplicationFailure with retryable: false
      throw new ApplicationFailure(
        'Card declined',
        'CARD_DECLINED',
        false,            // do not retry this activity
        [],
        error,
      );
    }
    // Retryable error — Temporal will retry according to retry policy
    throw error;
  }
}

export async function refundCustomer(
  transactionId: string,
  amount: number,
): Promise<void> {
  await stripeClient.refunds.create({
    charge: transactionId,
    amount,
  });
}

// src/activities/inventory.activities.ts
export async function reserveInventory(
  items: OrderItem[],
  orderId: string,
): Promise<void> {
  await inventoryService.reserve(items, orderId);
}

export async function releaseInventory(
  orderId: string,
): Promise<void> {
  await inventoryService.release(orderId);
}

// src/activities/notification.activities.ts
export async function sendOrderConfirmation(
  customerId: string,
  orderId: string,
  items: OrderItem[],
): Promise<void> {
  await emailService.send({
    to: customerId,
    template: 'order-confirmation',
    data: { orderId, items },
  });
}

export async function sendOrderFailureNotification(
  customerId: string,
  reason: string,
): Promise<void> {
  await emailService.send({
    to: customerId,
    template: 'order-failed',
    data: { reason },
  });
}

Step 3 - Building the Order Fulfillment Workflow

This workflow orchestrates the full order lifecycle, charge, reserve inventory, confirm, with automatic compensation (refund + release inventory) if any step fails:

// src/workflows/order-fulfillment.workflow.ts
import { proxyActivities, sleep, defineSignal, setHandler, condition, ApplicationFailure } from '@temporalio/workflow';
import type * as PaymentActivities from '../activities/payment.activities';
import type * as InventoryActivities from '../activities/inventory.activities';
import type * as NotificationActivities from '../activities/notification.activities';

// Configure activity options - retry policy, timeouts
const { chargeCustomer, refundCustomer } = proxyActivities<typeof PaymentActivities>({
  startToCloseTimeout: '30 seconds',
  retry: {
    maximumAttempts: 3,
    initialInterval: '1 second',
    backoffCoefficient: 2,
    nonRetryableErrorTypes: ['CARD_DECLINED'],
  },
});

const { reserveInventory, releaseInventory } = proxyActivities<typeof InventoryActivities>({
  startToCloseTimeout: '10 seconds',
  retry: {
    maximumAttempts: 5,
    initialInterval: '500ms',
    backoffCoefficient: 1.5,
  },
});

const { sendOrderConfirmation, sendOrderFailureNotification } = proxyActivities<typeof NotificationActivities>({
  startToCloseTimeout: '15 seconds',
  retry: {
    maximumAttempts: 10,           // emails should retry aggressively
    initialInterval: '5 seconds',
  },
});

// Signals - external events that can influence workflow execution
export const cancelOrderSignal = defineSignal<[reason: string]>('cancelOrder');

export interface OrderInput {
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
}

export async function orderFulfillmentWorkflow(input: OrderInput): Promise<string> {
  const { orderId, customerId, items, totalAmount } = input;

  let cancelRequested = false;
  let cancelReason = '';

  // Register signal handler - cancellation can arrive at any time
  setHandler(cancelOrderSignal, (reason: string) => {
    cancelRequested = true;
    cancelReason = reason;
  });

  // Step 1 - Check for cancellation before starting
  if (cancelRequested) {
    await sendOrderFailureNotification(customerId, cancelReason);
    return 'CANCELLED';
  }

  // Step 2 - Charge the customer
  let chargeResult;
  try {
    chargeResult = await chargeCustomer(
      customerId,
      totalAmount,
      orderId,     // use orderId as idempotency key — safe to retry
    );
  } catch (error) {
    await sendOrderFailureNotification(customerId, 'Payment could not be processed.');
    throw error;  // fail the workflow — no compensation needed, no charge was made
  }

  // Check for cancellation after payment
  if (cancelRequested) {
    await refundCustomer(chargeResult.transactionId, totalAmount);
    await sendOrderFailureNotification(customerId, cancelReason);
    return 'CANCELLED_AND_REFUNDED';
  }

  // Step 3 — Reserve inventory
  try {
    await reserveInventory(items, orderId);
  } catch (error) {
    // Inventory failed — refund the charge (compensating transaction)
    await refundCustomer(chargeResult.transactionId, totalAmount);
    await sendOrderFailureNotification(customerId, 'Items are out of stock.');
    throw error;
  }

  // Step 4 — Send confirmation
  await sendOrderConfirmation(customerId, orderId, items);

  return 'FULFILLED';
}

The compensation pattern (refund on inventory failure) is expressed as ordinary sequential code, not a saga framework, not a choreography of events, not a state machine in a database. Temporal records every activity result. If the worker crashes during sendOrderConfirmation, Temporal replays the history, sees that chargeCustomer and reserveInventory already completed, and re-executes only from the sendOrderConfirmation call.

Step 4 - Long-Running Workflows with Timers

Temporal timers are durable, they survive server restarts and worker crashes, firing at the correct time regardless of what happened to the infrastructure while they were waiting:

// src/workflows/subscription-billing.workflow.ts
import { proxyActivities, sleep, continueAsNew } from '@temporalio/workflow';

const { chargeCustomer, sendInvoice, handlePaymentFailure } =
  proxyActivities<typeof BillingActivities>({
    startToCloseTimeout: '30 seconds',
    retry: { maximumAttempts: 3 },
  });

export interface SubscriptionInput {
  customerId: string;
  planId: string;
  monthlyAmount: number;
  billingCycleCount: number;   // how many cycles have run
}

export async function subscriptionBillingWorkflow(
  input: SubscriptionInput,
): Promise<void> {
  const { customerId, planId, monthlyAmount, billingCycleCount } = input;

  // Wait for next billing date — durable 30-day timer
  await sleep('30 days');

  // Attempt billing
  try {
    const result = await chargeCustomer(
      customerId,
      monthlyAmount,
      `${planId}-cycle-${billingCycleCount}`,
    );
    await sendInvoice(customerId, result.transactionId, monthlyAmount);
  } catch (error) {
    await handlePaymentFailure(customerId, billingCycleCount);
  }

  // Temporal event history grows with each cycle — use continueAsNew
  // to reset history while preserving workflow identity
  await continueAsNew<typeof subscriptionBillingWorkflow>({
    customerId,
    planId,
    monthlyAmount,
    billingCycleCount: billingCycleCount + 1,
  });
}

continueAsNew is critical for perpetual workflows, it closes the current workflow history and starts a new execution with fresh history, preserving the workflow ID. Without it, a subscription billing workflow running for years would accumulate an unbounded event history.

A durable sleep('30 days') in Temporal is not a 30-day setTimeout. If the worker is restarted, the server rebooted, or the entire Temporal cluster is upgraded during those 30 days, the timer fires correctly when the time comes. This is qualitatively different from any timer mechanism available in a standard application runtime.

Step 5 - Signals: Communicating with Running Workflows

Signals allow external systems to send events to a running workflow, without polling, without a database flag, without a queue:

// From your NestJS API — send a cancellation signal to a running workflow
import { Client } from '@temporalio/client';
import { cancelOrderSignal } from './workflows/order-fulfillment.workflow';

const client = new Client();

// Cancel a specific order — signal the running workflow directly
async function cancelOrder(orderId: string, reason: string): Promise<void> {
  const handle = client.workflow.getHandle(orderId);  // orderId is the workflow ID
  await handle.signal(cancelOrderSignal, reason);
}

// Query workflow state without interrupting it
async function getOrderStatus(orderId: string): Promise<string> {
  const handle = client.workflow.getHandle(orderId);
  return handle.query(orderStatusQuery);
}

The signal is delivered to the workflow exactly once, durably, even if the workflow is in the middle of a sleep, waiting for an activity, or processing another signal. The workflow receives it on the next evaluation cycle.

Step 6 - Child Workflows for Decomposition

Large workflows decompose into child workflows, independently observable, independently retryable units of orchestration:

// Parent workflow — orchestrates the overall order process
import { executeChild } from '@temporalio/workflow';
import { paymentWorkflow } from './payment.workflow';
import { fulfilmentWorkflow } from './fulfilment.workflow';

export async function orderOrchestrationWorkflow(
  order: Order,
): Promise<void> {
  // Execute payment as a child workflow — has its own retry policy and history
  const paymentResult = await executeChild(paymentWorkflow, {
    args: [{ orderId: order.id, customerId: order.customerId, amount: order.total }],
    workflowId: `payment-${order.id}`,
    taskQueue: 'payment-queue',
  });

  if (paymentResult.status !== 'succeeded') {
    throw new ApplicationFailure('Payment failed', 'PAYMENT_FAILED');
  }

  // Execute fulfilment as a child workflow in parallel for each warehouse
  await Promise.all(
    order.warehouses.map(warehouseId =>
      executeChild(fulfilmentWorkflow, {
        args: [{ orderId: order.id, warehouseId }],
        workflowId: `fulfilment-${order.id}-${warehouseId}`,
        taskQueue: 'fulfilment-queue',
      })
    )
  );
}

Child workflows are independently visible in the Temporal UI, you can observe the payment workflow and the fulfilment workflows as separate entities, each with their own event history, each independently retryable.

Step 7 - Running Workers

// src/worker.ts
import { Worker } from '@temporalio/worker';
import * as PaymentActivities from './activities/payment.activities';
import * as InventoryActivities from './activities/inventory.activities';
import * as NotificationActivities from './activities/notification.activities';

async function runWorker() {
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows/order-fulfillment.workflow'),
    activities: {
      ...PaymentActivities,
      ...InventoryActivities,
      ...NotificationActivities,
    },
    taskQueue: 'orders-queue',
    maxConcurrentActivityTaskExecutions: 20,
    maxConcurrentWorkflowTaskExecutions: 10,
  });

  await worker.run();
}

runWorker().catch(console.error);

Workers are stateless. Run as many as you need for throughput, Temporal distributes tasks across all available workers automatically. Scale workers horizontally without coordination.

Temporal vs. Traditional Approaches

Problem	Traditional Approach	Temporal
Retry failed steps	Manual retry logic + dead letter queues	Configurable retry policy per activity
Long-running timers	Cron jobs + database state	Durable `sleep()`, survives restarts
Parallel execution	Manual async coordination	`Promise.all()` over activity calls
Cancel in-flight process	Poll a database flag	Send a signal
Compensating transactions	Saga framework or manual rollback	Sequential code with try/catch
Observe running processes	Query database state	Temporal UI + queries
Workflow versioning	Schema migrations	Built-in versioning API

Common Pitfalls to Avoid

Non-deterministic workflow code. Calling Date.now(), Math.random(), or accessing external state directly in workflow code breaks replay. Use workflow.now() for timestamps, pass random values as workflow inputs, and read external state only through activities.

Blocking in workflows. Never use setTimeout, setInterval, or promise-based delays in workflow code. Use Temporal's sleep(), it is durable, the others are not.

Too much logic in activities. Activities should do one thing, one API call, one database write. Complex coordination logic belongs in the workflow, not the activity. An activity that does five things is an activity that partially fails and leaves unclear what to retry.

Missing idempotency keys on activities. Activities retry. If the activity makes a charge or sends an email, it must include an idempotency key derived from the workflow input, not a randomly generated UUID per attempt.

Unbounded workflow history. Perpetual workflows that never use continueAsNew accumulate event history indefinitely. This degrades performance and eventually hits Temporal's history size limits. Use continueAsNew for any workflow with recurring cycles.

Conclusion

Temporal.io solves a category of problem that traditional queues, cron jobs, and state machines handle poorly, long-running, multi-step business processes that must survive infrastructure failures without losing progress.

The programming model is its defining advantage. You write sequential TypeScript code, charge, reserve, confirm, and Temporal makes it fault-tolerant. No saga framework to configure. No compensating transaction registry to maintain. No polling loops to implement. The workflow code is the specification of the business process, readable by any engineer on the team.

The learning curve is real, the determinism constraint, replay semantics, and continueAsNew pattern require careful understanding. But once internalized, the model eliminates an entire category of distributed systems complexity that most engineering teams carry as permanent operational debt.

Migrating from a queue-based architecture (BullMQ, SQS, RabbitMQ) to Temporal? The migration is incremental, Temporal can consume from existing queues via activities while you migrate orchestration logic workflow by workflow.

Temporal #DurableExecution #WorkflowOrchestration #NodeJS #TypeScript

Goodbye YAML? My First Month with Pkl (Apple's Configuration Language)

ThankGod Chibugwum Obobo — Sun, 16 Aug 2026 17:00:00 +0000

YAML has been the configuration language of the cloud-native era by default, not by design. It is whitespace-sensitive in ways that cause real production incidents. It has no type system. It provides no mechanism for reuse, short of anchors and aliases that quickly become unreadable. It silently coerces values, the notorious NO becoming false, 1.0 becoming a float, and on becoming true, in ways that require memorization rather than reasoning.

Engineering teams have tolerated these limitations because the alternatives were worse, or at least less familiar. HCL is powerful but Terraform-specific. CUE is expressive but has a steep learning curve. Dhall is principled but alien to most developers. Jsonnet is underused and under-tooled.

Then Apple published Pkl (pronounced "pickle"), an open-source configuration language designed explicitly for the problems YAML fails to solve, type safety, reuse, abstraction, and validation built into the language rather than bolted on afterward.

After one month of using Pkl across real infrastructure configuration, Kubernetes manifests, application configs, and CI/CD templates, here is an honest account of what it does well, where it still struggles, and whether it is ready to replace YAML in a production environment.

What Is Pkl?

Pkl (Package Konfig Language) is a configuration language developed by Apple and open-sourced in February 2024. It compiles to JSON, YAML, XML, or plist, meaning it integrates with any toolchain that reads those formats, without requiring the entire ecosystem to adopt Pkl natively.

Its core design goals:

Type safety - every value has a type, enforced at evaluation time
Reuse through templates - configurations extend and override base templates
Validation built in - constraints are expressed in the language, not in external schema validators
Readability - syntax designed to be familiar to developers without YAML's whitespace traps
Evaluation, not just declaration - expressions, conditionals, and computed values are first-class

The compiler outputs are standard formats, so adopting Pkl is a source change, not an output change. Your Kubernetes cluster still receives YAML. Your application still reads JSON. Pkl lives in your repository and your CI pipeline, not in your runtime.

Syntax First: What Pkl Actually Looks Like

The fastest way to understand Pkl is to see it alongside the YAML it replaces.

Basic Value Declaration

# YAML
app:
  name: orders-service
  port: 3000
  debug: false
  replicas: 3

# Pkl
app {
  name = "orders-service"
  port = 3000
  debug = false
  replicas = 3
}

The structural difference is immediately visible: Pkl uses braces instead of indentation, = for assignment instead of :, and quoted strings that are unambiguously strings, "false" is a string, false is a boolean. There is no YAML parser needed to figure out which is which.

Type Annotations

// Pkl - explicit types on every field
class AppConfig {
  name: String
  port: Int(this > 1024 && this < 65535)   // inline constraint
  debug: Boolean
  replicas: Int(this >= 1 && this <= 10)
  environment: "staging" | "production" | "development"  // union type
}

app: AppConfig = new {
  name = "orders-service"
  port = 3000
  debug = false
  replicas = 3
  environment = "production"
}

Setting replicas = 0 against this class fails at evaluation time with a clear error message, not at runtime when Kubernetes tries to create zero pods. Setting environment = "prod" (a typo) fails immediately because the value is not in the union type. This is the type safety YAML schemas attempt to provide through external validators, Pkl builds it directly into the language.

YAML Coercion Problems - Solved

# YAML - notorious coercion traps
enabled: yes        # becomes true
version: 1.0        # becomes float in some parsers
country: NO         # becomes false (Norway's ISO code!)
port: 8080          # integer
tag: "8080"         # string — same value, different type, different behavior

# Pkl - unambiguous
enabled: Boolean = true
version: String = "1.0"
country: String = "NO"
port: Int = 8080
tag: String = "8080"

Every value's type is explicit. The Norway NO incident, a real class of YAML bug that has affected Kubernetes configurations in production, cannot happen in Pkl.

Templates and Inheritance: The Killer Feature

YAML's anchor-and-alias system for reuse looks like this:

# YAML anchors - functional but unreadable at scale
.defaults: &defaults
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"

deployments:
  orders-service:
    <<: *defaults
    replicas: 3
  users-service:
    <<: *defaults
    replicas: 2
    resources:
      limits:
        memory: "1Gi"    # override — but does this merge or replace?

The << merge key behavior is ambiguous in some parsers and was officially deprecated in YAML 1.2. Pkl's approach is unambiguous:

// Pkl - templates with explicit inheritance
class DeploymentDefaults {
  imagePullPolicy: String = "IfNotPresent"
  memoryRequest: String = "256Mi"
  cpuRequest: String = "100m"
  memoryLimit: String = "512Mi"
  cpuLimit: String = "500m"
}

class Deployment extends DeploymentDefaults {
  name: String
  replicas: Int = 1
}

ordersService: Deployment = new {
  name = "orders-service"
  replicas = 3
  // inherits all defaults
}

usersService: Deployment = new {
  name = "users-service"
  replicas = 2
  memoryLimit = "1Gi"    // explicitly override one field, all others inherited
}

The inheritance is explicit, the override is unambiguous, and the IDE can show you exactly which fields are inherited versus overridden, something YAML anchors fundamentally cannot do.

Real-World Usage: Kubernetes Manifests

Here is a production-representative Pkl template for a Kubernetes Deployment, showing how templating applies to the configuration I write most often:

// k8s/templates/deployment.pkl
module deployment

import "package://pkg.pkl-lang.org/pkl-k8s/k8s@1.0.0#/api/apps/v1/Deployment.pkl"

class AppDeployment {
  name: String
  namespace: String = "default"
  image: String
  tag: String
  replicas: Int(this >= 1) = 1
  port: Int = 8080
  cpuRequest: String = "100m"
  memoryRequest: String = "256Mi"
  cpuLimit: String = "500m"
  memoryLimit: String = "512Mi"
  env: Listing<EnvVar> = new {}
}

class EnvVar {
  name: String
  value: String
}

function toK8sDeployment(config: AppDeployment): Deployment = new {
  metadata {
    name = config.name
    namespace = config.namespace
    labels { app = config.name }
  }
  spec {
    replicas = config.replicas
    selector { matchLabels { app = config.name } }
    template {
      metadata { labels { app = config.name } }
      spec {
        containers {
          new {
            name = config.name
            image = "\(config.image):\(config.tag)"
            ports { new { containerPort = config.port } }
            resources {
              requests {
                cpu = config.cpuRequest
                memory = config.memoryRequest
              }
              limits {
                cpu = config.cpuLimit
                memory = config.memoryLimit
              }
            }
            envFrom = config.env
          }
        }
      }
    }
  }
}

// k8s/services/orders-service.pkl
import "../templates/deployment.pkl"

output {
  value = deployment.toK8sDeployment(new deployment.AppDeployment {
    name = "orders-service"
    namespace = "production"
    image = "your-org/orders-service"
    tag = "2.4.1"
    replicas = 3
    env {
      new { name = "DATABASE_URL"; value = "postgresql://db:5432/orders" }
      new { name = "NODE_ENV"; value = "production" }
    }
  })
}

Compile to YAML for Kubernetes:

pkl eval k8s/services/orders-service.pkl -f yaml > k8s/manifests/orders-service.yaml

The manifest is generated, not handwritten. The template enforces the constraint that replicas is at least 1. The image and tag are separate fields, no string concatenation errors. The output is standard Kubernetes YAML that kubectl apply consumes without modification.

CI/CD Integration

Pkl integrates into CI/CD pipelines anywhere you currently generate or validate YAML:

# .github/workflows/pkl-generate.yml
name: Generate and Validate Configs

on: [push, pull_request]

jobs:
  pkl-generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Pkl CLI
        run: |
          curl -L https://github.com/apple/pkl/releases/latest/download/pkl-linux-amd64 \
            -o /usr/local/bin/pkl
          chmod +x /usr/local/bin/pkl

      - name: Validate all Pkl configurations
        run: pkl eval --multiple-file-output /dev/null pkl/**/*.pkl

      - name: Generate Kubernetes manifests
        run: |
          for file in k8s/services/*.pkl; do
            service=$(basename "$file" .pkl)
            pkl eval "$file" -f yaml > "k8s/manifests/${service}.yaml"
          done

      - name: Commit generated manifests
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add k8s/manifests/
          git diff --cached --quiet || git commit -m "chore: regenerate manifests from Pkl"
          git push

Honest Assessment After One Month

What Pkl Gets Right

Type safety is genuinely useful. Finding configuration errors at compile time rather than at kubectl apply time, or worse, at runtime, is a real quality improvement. Inline constraints like Int(this > 0) replace entire schema validation workflows.

Templates and inheritance are the best solution I've seen. The YAML anchor problem is real. Pkl's class-based inheritance with explicit overrides is significantly cleaner than any YAML reuse mechanism.

String interpolation without escaping traps. "\(config.image):\(config.tag)" is readable and unambiguous. YAML multiline string handling is notoriously confusing.

Evaluation-time computation. Generating environment-specific configuration from a single template with conditional expressions reduces the number of files to maintain.

Where Pkl Still Struggles

Ecosystem maturity is early. The Kubernetes package library (pkl-k8s) covers most resources but not all. For anything not in the package, you're writing raw mappings to target types, which works but loses the ergonomic advantage.

IDE support is improving but incomplete. The VS Code extension provides syntax highlighting and basic completion. IntelliJ support is more mature. Neither yet matches the experience of working in a fully-supported language.

Team adoption friction. Pkl is not YAML. Developers familiar with YAML need to learn new syntax, new mental models for inheritance, and a new compilation step. This is a real cost that requires team buy-in.

Debugging compilation errors. When a constraint fails or a type mismatch occurs in a deeply nested template, the error messages are sometimes less precise than you'd want. This is improving with each release.

Not every toolchain accepts Pkl natively. Tools that consume configuration at runtime, some CI/CD platforms, older infrastructure tools, can't read Pkl directly. The compilation step to YAML/JSON is necessary, which adds pipeline complexity.

Should You Adopt Pkl?

Adopt Pkl today if:

You maintain large, complex YAML configurations with significant duplication
Type errors and coercion bugs have caused real incidents in your environment
Your team writes configuration as code and values IDE support
You're already compiling or generating YAML (Helm charts, Kustomize overlays), Pkl replaces the templating layer cleanly

Wait on Pkl if:

Your YAML configs are small and simple, the overhead isn't justified
Your team doesn't have bandwidth for adoption friction right now
You need native tool support without a compilation step
You're evaluating for a production migration within the next quarter, the ecosystem is maturing fast, but not yet complete

Conclusion

After one month, Pkl has earned a permanent place in my infrastructure toolchain, but not as a wholesale YAML replacement, yet. It is the right tool for complex, template-heavy configuration that benefits from type safety and reuse. It is not yet the obvious choice for simple, stable configuration that existing YAML handles without friction.

The trajectory is clear, Pkl is solving real problems that YAML ignores and doing so with a thoughtfulness that most configuration languages lack. The ecosystem will catch up. The IDE tooling will improve. Team familiarity will build.

YAML isn't dead. But for the first time in years, there's a credible successor that makes the case on engineering merit rather than novelty.

Have you tried Pkl in your infrastructure? What was your first-month experience, similar frustrations, different use cases? Share in the comments.

Pkl #YAML #ConfigurationAsCode #DevOps #InfrastructureAsCode

Green DevOps: How to Optimize Your Cloud Carbon Footprint Using Carbon-Aware SDKs

ThankGod Chibugwum Obobo — Sun, 09 Aug 2026 17:00:00 +0000

Cloud computing accounts for approximately 2–3% of global electricity consumption a figure that rivals the aviation industry and continues to grow as engineering teams scale infrastructure to meet demand. For most of that growth, carbon impact has been invisible, infrastructure decisions are made on cost, latency, and availability, with sustainability treated as someone else's problem.

Green DevOps changes the accounting. It integrates carbon awareness directly into the engineering decision loop, workload scheduling, region selection, CI/CD pipeline timing, and autoscaling policies, making the carbon cost of infrastructure decisions as visible and actionable as the financial cost.

The Carbon-Aware SDK, developed by the Green Software Foundation, provides the tooling layer that makes this practical. A standardized API for querying real-time and forecast carbon intensity data, enabling your applications and pipelines to make time and location-based decisions that minimize carbon emissions without sacrificing availability or performance.

This guide covers how to integrate carbon awareness into your cloud infrastructure and DevOps workflows, from SDK setup and workload time-shifting to Kubernetes scheduling policies and measurable sustainability metrics.

Understanding Carbon Intensity

Before implementing carbon-aware systems, understand what you're measuring:

Carbon intensity measures the amount of CO₂ equivalent (gCO₂eq) emitted per kilowatt-hour of electricity consumed. It varies dramatically by:

Location: A data center in Norway running on hydroelectric power has a carbon intensity of ~20 gCO₂eq/kWh. The same workload in a coal-heavy grid region might emit 700+ gCO₂eq/kWh.
Time: Renewable energy sources like solar and wind are intermittent. Carbon intensity on the same grid can vary by 3–5x between peak solar hours and overnight periods when fossil fuel plants pick up demand.

The two levers carbon-aware systems pull are:

Spatial shifting, running workloads in the region with the lowest current carbon intensity, where latency requirements permit.

Temporal shifting, delaying non-time-sensitive workloads (batch jobs, CI pipelines, data exports, model training) until carbon intensity in your region is lower, typically during high renewable generation periods.

The Carbon-Aware SDK

The Carbon-Aware SDK (Green Software Foundation) provides a unified interface to multiple carbon intensity data providers, Electricity Maps, WattTime, and the UK National Grid ESO, through a consistent REST API and CLI.

Installation and Setup

# Clone and run the Carbon-Aware SDK as a local API server
git clone https://github.com/Green-Software-Foundation/carbon-aware-sdk.git
cd carbon-aware-sdk/src/CarbonAware.WebApi

# Configure your data provider
cat > appsettings.json << EOF
{
  "DataSources": {
    "EmissionsDataSource": "ElectricityMaps",
    "ForecastDataSource": "ElectricityMaps"
  },
  "ElectricityMaps": {
    "APITokenHeader": "auth-token",
    "APIToken": "${ELECTRICITY_MAPS_API_TOKEN}",
    "BaseURL": "https://api.electricitymap.org/v3/"
  }
}
EOF

dotnet run
# API now running at http://localhost:5073

Alternatively, deploy as a container alongside your infrastructure:

# docker-compose.yml
services:
  carbon-aware-api:
    image: ghcr.io/green-software-foundation/carbon-aware-sdk:latest
    ports:
      - "5073:5073"
    environment:
      DataSources__EmissionsDataSource: "ElectricityMaps"
      DataSources__ForecastDataSource: "ElectricityMaps"
      ElectricityMaps__APIToken: ${ELECTRICITY_MAPS_API_TOKEN}
    restart: unless-stopped

Querying Carbon Intensity

// src/carbon/carbon-aware.client.ts
export class CarbonAwareClient {
  private readonly baseUrl: string;

  constructor(baseUrl = process.env.CARBON_AWARE_API_URL ?? 'http://localhost:5073') {
    this.baseUrl = baseUrl;
  }

  // Get current carbon intensity for a location
  async getCurrentIntensity(location: string): Promise<number> {
    const response = await fetch(
      `${this.baseUrl}/emissions/current?location=${location}`
    );
    const data = await response.json();
    return data[0]?.rating ?? 0; // gCO₂eq/kWh
  }

  // Get forecast to find optimal execution window
  async getForecast(location: string, windowMinutes: number = 360): Promise<CarbonForecast[]> {
    const now = new Date();
    const end = new Date(now.getTime() + windowMinutes * 60 * 1000);

    const response = await fetch(
      `${this.baseUrl}/emissions/forecasts/current?` +
      `location=${location}&` +
      `dataStartAt=${now.toISOString()}&` +
      `dataEndAt=${end.toISOString()}&` +
      `windowSize=60`
    );
    const data = await response.json();
    return data[0]?.optimalDataPoints ?? [];
  }

  // Find the lowest-carbon region from a set of candidates
  async getBestRegion(locations: string[]): Promise<string> {
    const params = locations.map(l => `location=${l}`).join('&');
    const response = await fetch(
      `${this.baseUrl}/emissions/current?${params}`
    );
    const data = await response.json();

    return data.reduce((best: any, current: any) =>
      current.rating < best.rating ? current : best
    ).location;
  }
}

Step 1 - Time-Shifting Batch Workloads

The highest-value application of carbon awareness is temporal shifting of batch workloads, jobs that must complete within a window but don't need to start immediately. Data exports, report generation, ML model training, database maintenance, and archive compression are all candidates.

// src/carbon/workload-scheduler.ts
import { CarbonAwareClient } from './carbon-aware.client';

interface WorkloadConfig {
  location: string;
  maxDelayMinutes: number;       // how long can we wait?
  carbonThreshold: number;       // only run if below this gCO₂eq/kWh
}

export class CarbonAwareScheduler {
  private readonly client: CarbonAwareClient;

  constructor() {
    this.client = new CarbonAwareClient();
  }

  async findOptimalExecutionTime(config: WorkloadConfig): Promise<Date> {
    const forecast = await this.client.getForecast(
      config.location,
      config.maxDelayMinutes
    );

    if (forecast.length === 0) {
      // No forecast data — run now rather than block indefinitely
      return new Date();
    }

    // Find the lowest carbon intensity window within our delay budget
    const optimal = forecast.reduce((best, current) =>
      current.value < best.value ? current : best
    );

    const optimalTime = new Date(optimal.timestamp);
    const maxWaitTime = new Date(Date.now() + config.maxDelayMinutes * 60 * 1000);

    // Don't wait longer than our deadline
    return optimalTime < maxWaitTime ? optimalTime : new Date();
  }

  async shouldRunNow(config: WorkloadConfig): Promise<boolean> {
    const current = await this.client.getCurrentIntensity(config.location);
    return current <= config.carbonThreshold;
  }
}

Integrate the scheduler into your batch job runner:

// src/jobs/data-export.job.ts
import { CarbonAwareScheduler } from '../carbon/workload-scheduler';
import { scheduleJob } from 'node-schedule';

export async function scheduleDataExport() {
  const scheduler = new CarbonAwareScheduler();

  const optimalTime = await scheduler.findOptimalExecutionTime({
    location: 'eastus',
    maxDelayMinutes: 360,     // can delay up to 6 hours
    carbonThreshold: 200,     // prefer below 200 gCO₂eq/kWh
  });

  console.log(`Data export scheduled for: ${optimalTime.toISOString()}`);
  console.log(`Delay: ${Math.round((optimalTime.getTime() - Date.now()) / 60000)} minutes`);

  scheduleJob(optimalTime, async () => {
    console.log('Executing carbon-optimized data export...');
    await runDataExport();
  });
}

Step 2 - Carbon-Aware CI/CD Pipelines

CI/CD pipelines are one of the most consistently overlooked sources of cloud carbon emissions, running on every commit, often around the clock, with flexible timing requirements. A test suite that takes 15 minutes can run at 2am during low-carbon hours just as effectively as at 2pm during peak demand.

# .github/workflows/carbon-aware-ci.yml
name: Carbon-Aware CI Pipeline

on:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 */6 * * *'  # Check every 6 hours for deferred runs

jobs:
  check-carbon:
    name: Carbon Intensity Check
    runs-on: ubuntu-latest
    outputs:
      should-run: ${{ steps.carbon-check.outputs.should-run }}
      intensity: ${{ steps.carbon-check.outputs.intensity }}

    steps:
      - name: Check current carbon intensity
        id: carbon-check
        run: |
          INTENSITY=$(curl -s \
            "https://api.electricitymap.org/v3/carbon-intensity/latest?zone=US-MIDA" \
            -H "auth-token: ${{ secrets.ELECTRICITY_MAPS_TOKEN }}" \
            | jq '.carbonIntensity')

          echo "intensity=$INTENSITY" >> $GITHUB_OUTPUT

          # Run immediately for pushes to main regardless of carbon
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "should-run=true" >> $GITHUB_OUTPUT
            echo "Main branch push — running regardless of carbon intensity ($INTENSITY gCO₂eq/kWh)"
          elif (( $(echo "$INTENSITY < 250" | bc -l) )); then
            echo "should-run=true" >> $GITHUB_OUTPUT
            echo "Carbon intensity acceptable: $INTENSITY gCO₂eq/kWh — running pipeline"
          else
            echo "should-run=false" >> $GITHUB_OUTPUT
            echo "Carbon intensity too high: $INTENSITY gCO₂eq/kWh — deferring pipeline"
          fi

  build-and-test:
    name: Build and Test
    runs-on: ubuntu-latest
    needs: check-carbon
    if: needs.check-carbon.outputs.should-run == 'true'

    steps:
      - uses: actions/checkout@v4

      - name: Log carbon context
        run: |
          echo "Running pipeline at carbon intensity: ${{ needs.check-carbon.outputs.intensity }} gCO₂eq/kWh"

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Build
        run: npm run build

The key design decision: main branch pushes always run, carbon awareness is applied to non-critical developer branch pipelines, not to production deployments where timing is dictated by business requirements, not grid carbon intensity.

Step 3 - Carbon-Aware Kubernetes Scheduling

For Kubernetes workloads with flexible placement, integrate carbon intensity into pod scheduling decisions using a custom scheduler plugin or node labels updated by a controller:

// carbon-node-labeler/src/labeler.ts
// A controller that labels Kubernetes nodes with current carbon intensity zones
import { KubeConfig, CoreV1Api } from '@kubernetes/client-node';
import { CarbonAwareClient } from './carbon-aware.client';

const kc = new KubeConfig();
kc.loadFromDefault();
const k8sApi = kc.makeApiClient(CoreV1Api);
const carbonClient = new CarbonAwareClient();

const REGION_TO_LOCATION: Record<string, string> = {
  'us-east-1':    'eastus',
  'eu-west-1':    'northeurope',
  'ap-southeast-1': 'southeastasia',
};

async function labelNodesWithCarbonZone(): Promise<void> {
  const nodes = await k8sApi.listNode();

  for (const node of nodes.body.items) {
    const region = node.metadata?.labels?.['topology.kubernetes.io/region'];
    if (!region || !REGION_TO_LOCATION[region]) continue;

    const intensity = await carbonClient.getCurrentIntensity(REGION_TO_LOCATION[region]);

    // Classify carbon zone
    const zone = intensity < 150 ? 'green'
               : intensity < 300 ? 'yellow'
               : 'red';

    // Label node with current carbon zone
    await k8sApi.patchNode(node.metadata!.name!, {
      metadata: {
        labels: {
          'carbon.fdn.dev/intensity-zone': zone,
          'carbon.fdn.dev/intensity-value': Math.round(intensity).toString(),
        }
      }
    }, undefined, undefined, undefined, undefined, undefined, {
      headers: { 'Content-Type': 'application/merge-patch+json' }
    });
  }
}

// Run every 15 minutes
setInterval(labelNodesWithCarbonZone, 15 * 60 * 1000);
labelNodesWithCarbonZone();

Use node affinity in batch job manifests to prefer green-zone nodes:

# k8s/batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing-job
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          # Prefer green nodes — schedule here if available
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: In
                    values: ["green"]
            - weight: 50
              preference:
                matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: In
                    values: ["yellow"]

          # Never schedule on red-zone nodes for non-critical jobs
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: NotIn
                    values: ["red"]

      containers:
        - name: data-processor
          image: your-org/data-processor:latest

Step 4 - Spatial Shifting for Multi-Region Workloads

For workloads without strict data residency requirements, route to the lowest-carbon region at scheduling time:

// src/carbon/region-selector.ts
export class CarbonAwareRegionSelector {
  private readonly client: CarbonAwareClient;

  // Candidate regions for this workload
  private readonly candidateRegions = [
    { id: 'eastus',         name: 'us-east-1' },
    { id: 'northeurope',    name: 'eu-west-1' },
    { id: 'westus2',        name: 'us-west-2' },
    { id: 'australiaeast',  name: 'ap-southeast-2' },
  ];

  async selectOptimalRegion(): Promise<string> {
    const bestLocation = await this.client.getBestRegion(
      this.candidateRegions.map(r => r.id)
    );

    const region = this.candidateRegions.find(r => r.id === bestLocation);
    console.log(`Carbon-optimal region selected: ${region?.name} (${bestLocation})`);
    return region?.name ?? 'us-east-1';
  }
}

// Use in your infrastructure provisioning or job dispatch
const selector = new CarbonAwareRegionSelector();
const region = await selector.selectOptimalRegion();
await dispatchBatchJob({ region, jobConfig });

Step 5 - Measuring and Reporting Carbon Impact

Carbon awareness only delivers accountability if emissions are measured and reported. The Cloud Carbon Footprint open-source tool aggregates billing data from AWS, GCP, and Azure into CO₂ estimates:

# Install Cloud Carbon Footprint
npx create-cloud-carbon-footprint-app@latest

# Configure AWS credentials
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_DEFAULT_REGION=us-east-1

# Run the estimation
npx ts-node packages/cli/src/CreateLookupTableAndEstimate.ts

Build carbon metrics into your observability stack:

// src/carbon/carbon-metrics.ts
import { MeterProvider } from '@opentelemetry/sdk-metrics';

export class CarbonMetricsReporter {
  private readonly meter = new MeterProvider().getMeter('carbon-metrics');

  private readonly intensityGauge = this.meter.createObservableGauge(
    'carbon.grid.intensity',
    { description: 'Current grid carbon intensity (gCO₂eq/kWh)', unit: 'gCO2eq/kWh' }
  );

  private readonly emissionsCounter = this.meter.createCounter(
    'carbon.workload.emissions',
    { description: 'Estimated carbon emitted by workloads (gCO₂eq)', unit: 'gCO2eq' }
  );

  async recordWorkloadEmissions(
    workloadName: string,
    durationHours: number,
    powerWatts: number,
    location: string,
  ): Promise<void> {
    const intensity = await new CarbonAwareClient().getCurrentIntensity(location);

    // Emissions (gCO₂eq) = Power (kW) × Duration (h) × Intensity (gCO₂eq/kWh)
    const emissionsGrams = (powerWatts / 1000) * durationHours * intensity;

    this.emissionsCounter.add(emissionsGrams, {
      workload: workloadName,
      location,
    });

    console.log(`${workloadName}: estimated ${emissionsGrams.toFixed(2)} gCO₂eq`);
  }
}

Surface carbon metrics in Grafana alongside your standard engineering dashboards:

# Average carbon intensity by region over the last 24 hours
avg_over_time(carbon_grid_intensity{location="eastus"}[24h])

# Total estimated emissions from CI pipelines this week
sum(increase(carbon_workload_emissions{workload="ci-pipeline"}[7d]))

# Carbon savings from time-shifted jobs vs immediate execution baseline
(carbon_baseline_emissions - carbon_actual_emissions) / carbon_baseline_emissions * 100

Step 6 - Green DevOps Principles Checklist

Beyond tooling, Green DevOps is a set of engineering practices:

Right-size before you optimize carbon. An oversized VM running at 5% CPU is wasting energy regardless of grid carbon intensity. Right-size your infrastructure first, carbon optimization on already-efficient infrastructure has a multiplied impact.

Prefer managed services. Cloud providers' managed services (RDS, Fargate, Cloud Functions) are typically more energy-efficient than self-managed equivalents at the same compute capacity, cloud hyperscalers achieve PUE (Power Usage Effectiveness) ratios that on-premise or self-managed rarely match.

Implement autoscaling aggressively. Scale to zero during off-hours for non-production environments. A staging environment running 24/7 at minimum capacity consumes energy around the clock, including during high-carbon grid periods.

Choose renewable-powered regions. AWS regions like eu-north-1 (Stockholm) and us-gov-west-1 run on high renewable percentages. GCP and Azure publish similar sustainability region data. When latency requirements permit region flexibility, factor carbon into the selection.

Add carbon to your architecture decision records. When making infrastructure decisions, region selection, instance type, deployment frequency, document the carbon trade-offs alongside cost and performance trade-offs. Visibility drives accountability.

Conclusion

Green DevOps is not a compliance exercise or a marketing initiative, it is an engineering discipline that optimizes cloud workloads against a metric that has been invisible in most infrastructure decision-making, carbon emissions.

The Carbon-Aware SDK makes the data available. Time-shifting batch jobs, deferring non-critical CI pipelines, routing spatially flexible workloads to lower-carbon regions, and scheduling Kubernetes batch jobs on green-zone nodes are all achievable with the patterns in this guide, with measurable impact on your infrastructure's carbon footprint.

Start with the highest-leverage intervention, identify your largest batch workloads and shift their execution timing by a few hours based on carbon intensity forecasts. Measure the before-and-after emissions. Once the pattern is proven, expand to CI pipeline scheduling, region selection policies, and Kubernetes node affinity rules.

Sustainable infrastructure is not slower infrastructure. Done right, it is simply infrastructure that knows what time it is, and chooses wisely.

What's your team's current approach to cloud sustainability — are you tracking emissions today? Share your stack and what you've measured in the comments.

The Flutter-to-DevOps Pipeline: How to Automate Play Store Deployments with GitHub Actions

ThankGod Chibugwum Obobo — Sun, 02 Aug 2026 17:00:00 +0000

Mobile app deployment is one of the last frontiers where many engineering teams still rely on manual processes. A developer builds a release locally, signs it on their machine, uploads it through the Play Console UI, and fills out a release form by hand. It works, until it doesn't. The signing keystore lives on one laptop. The Play Console credentials are shared over Slack. The versioning is managed in a spreadsheet.

This is not a deployment pipeline. It is a fragile ritual that couples your release process to a specific person, a specific machine, and a specific sequence of manual steps that nobody has fully documented.

GitHub Actions combined with Fastlane transforms Flutter's release process into a reproducible, auditable, automated pipeline, building, testing, signing, and deploying to the Google Play Store on every merge to your release branch, with no developer manual intervention required.

This guide covers the complete Flutter-to-Play Store pipeline, project configuration, automated testing, keystore management, build versioning, AAB generation, and staged Play Store deployment, all wired into a GitHub Actions workflow that runs without human input.

Pipeline Architecture

Before writing any YAML, understand what the complete pipeline does and in what order:

Push to release branch
           │
           ▼
┌─────────────────────┐
│   1. Setup          │  Install Flutter SDK, Java, dependencies
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   2. Test           │  Unit tests, widget tests, integration tests
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   3. Code Analysis  │  flutter analyze, dart format check
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   4. Version Bump   │  Auto-increment build number from Git
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   5. Build AAB      │  flutter build appbundle --release
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   6. Sign           │  Sign with keystore via Gradle
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   7. Deploy         │  Upload to Play Store internal track via Fastlane
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   8. Notify         │  Slack notification with release summary
└─────────────────────┘

Each stage gates the next, a failed test prevents a build, a failed build prevents a deployment. The pipeline is a sequence of quality gates, not just a delivery mechanism.

Step 1 - Flutter Project Configuration

Ensure your Flutter project is configured for automated builds before touching CI:

# pubspec.yaml - version format: major.minor.patch+buildNumber
name: your_app
version: 1.4.2+87       # +87 is the build number (versionCode in Play Store)

Configure Gradle to read signing credentials from environment variables rather than hardcoded file paths, critical for CI environments where keystores are injected at runtime:

// android/app/build.gradle
def keystoreProperties = new Properties()
def keystorePropertiesFile = rootProject.file('key.properties')

// Load from file if it exists (local development)
// Otherwise read from environment variables (CI)
if (keystorePropertiesFile.exists()) {
  keystoreProperties.load(new FileInputStream(keystorePropertiesFile))
}

android {
  signingConfigs {
    release {
      keyAlias     keystoreProperties['keyAlias']     ?: System.getenv('KEY_ALIAS')
      keyPassword  keystoreProperties['keyPassword']  ?: System.getenv('KEY_PASSWORD')
      storeFile    keystoreProperties['storeFile']    ? file(keystoreProperties['storeFile'])
                                                      : file(System.getenv('KEY_STORE_PATH') ?: 'keystore.jks')
      storePassword keystoreProperties['storePassword'] ?: System.getenv('KEY_STORE_PASSWORD')
    }
  }

  buildTypes {
    release {
      signingConfig     signingConfigs.release
      minifyEnabled     true
      shrinkResources   true
      proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }
  }
}

This pattern works identically in local development (reads from key.properties) and CI (reads from environment variables injected by GitHub Actions).

Step 2 - Secrets Setup in GitHub

Before writing the workflow, store all sensitive values as GitHub Actions secrets:

Repository → Settings → Secrets and Variables → Actions

KEYSTORE_BASE64          ← Base64-encoded keystore file
KEY_ALIAS                ← Key alias within the keystore
KEY_PASSWORD             ← Key password
KEY_STORE_PASSWORD       ← Keystore password
PLAY_STORE_JSON_KEY      ← Google Play service account JSON
SLACK_WEBHOOK_URL        ← Slack notification webhook (optional)

Encode your keystore for storage as a GitHub secret:

# On your local machine - encode keystore to base64
base64 -i your-release-keystore.jks | pbcopy   # macOS
base64 -w 0 your-release-keystore.jks | xclip  # Linux

# Paste the output as the KEYSTORE_BASE64 secret value

The base64 encoding is necessary because GitHub Secrets store string values, binary files must be encoded for storage and decoded at runtime.

Step 3 - Google Play Service Account

Fastlane's supply tool authenticates to the Play Store API using a Google service account, not your personal Google account. This is the secure, auditable approach, the service account has only the permissions it needs, and credentials can be rotated independently.

Set up the service account:

Go to Google Play Console → Setup → API access
Link to an existing Google Cloud project or create a new one
Create a service account in Google Cloud IAM with the Editor role on the linked project
Download the service account JSON key file
Grant the service account Release Manager permissions in Play Console
Paste the JSON key contents as the PLAY_STORE_JSON_KEY GitHub secret

Step 4 - Fastlane Configuration

Fastlane handles the Play Store upload. Initialize it in your Flutter project:

# From your Flutter project root
gem install fastlane
cd android
fastlane init

Configure the Fastfile with lanes for each deployment target:

# android/fastlane/Fastfile
default_platform(:android)

platform :android do

  desc "Run all tests"
  lane :test do
    gradle(task: "test")
  end

  desc "Deploy to Play Store internal testing track"
  lane :deploy_internal do
    upload_to_play_store(
      track:              "internal",
      aab:                "../build/app/outputs/bundle/release/app-release.aab",
      json_key_data:      ENV["PLAY_STORE_JSON_KEY"],
      release_status:     "draft",
      skip_upload_apk:    true,
      skip_upload_images: true,
      skip_upload_screenshots: true,
    )
  end

  desc "Promote internal build to beta"
  lane :promote_to_beta do
    upload_to_play_store(
      track:          "internal",
      track_promote_to: "beta",
      json_key_data:  ENV["PLAY_STORE_JSON_KEY"],
      skip_upload_apk: true,
      skip_upload_aab: true,
    )
  end

  desc "Promote beta to production with staged rollout"
  lane :promote_to_production do
    upload_to_play_store(
      track:              "beta",
      track_promote_to:   "production",
      rollout:            "0.1",   # start with 10% rollout
      json_key_data:      ENV["PLAY_STORE_JSON_KEY"],
      skip_upload_apk:    true,
      skip_upload_aab:    true,
    )
  end

end

The three-lane structure mirrors the Play Store track progression, internal -> beta -> production. Each lane is independently callable from GitHub Actions, giving you granular control over the promotion workflow.

Step 5 - The Complete GitHub Actions Workflow

# .github/workflows/flutter-deploy-android.yml
name: Flutter Android Deploy

on:
  push:
    branches: [release]

  # Manual trigger for promoting between tracks
  workflow_dispatch:
    inputs:
      action:
        description: "Deployment action"
        required: true
        type: choice
        options:
          - deploy-internal
          - promote-beta
          - promote-production

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.19.0'
          channel: 'stable'
          cache: true

      - name: Install dependencies
        run: flutter pub get

      - name: Verify formatting
        run: dart format --output=none --set-exit-if-changed .

      - name: Analyze code
        run: flutter analyze --fatal-infos

      - name: Run unit and widget tests
        run: flutter test --coverage

      - name: Upload coverage report
        uses: codecov/codecov-action@v4
        with:
          files: coverage/lcov.info

  build-and-deploy:
    name: Build & Deploy
    runs-on: ubuntu-latest
    needs: test                    # only runs if tests pass
    if: github.event_name == 'push' || github.event.inputs.action == 'deploy-internal'

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0           # full history for build number calculation

      - name: Set up Java
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Set up Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.19.0'
          channel: 'stable'
          cache: true

      - name: Install dependencies
        run: flutter pub get

      - name: Calculate build number from Git commit count
        id: versioning
        run: |
          BUILD_NUMBER=$(git rev-list --count HEAD)
          echo "BUILD_NUMBER=$BUILD_NUMBER" >> $GITHUB_OUTPUT
          echo "Build number: $BUILD_NUMBER"

      - name: Update pubspec version
        run: |
          CURRENT_VERSION=$(grep '^version:' pubspec.yaml | sed 's/version: //' | cut -d'+' -f1)
          NEW_VERSION="${CURRENT_VERSION}+${{ steps.versioning.outputs.BUILD_NUMBER }}"
          sed -i "s/^version: .*/version: ${NEW_VERSION}/" pubspec.yaml
          echo "Version set to: ${NEW_VERSION}"

      - name: Decode keystore from secret
        run: |
          echo "${{ secrets.KEYSTORE_BASE64 }}" | base64 --decode > android/app/keystore.jks

      - name: Build Android App Bundle
        run: |
          flutter build appbundle \
            --release \
            --build-number=${{ steps.versioning.outputs.BUILD_NUMBER }}
        env:
          KEY_ALIAS: ${{ secrets.KEY_ALIAS }}
          KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }}
          KEY_STORE_PASSWORD: ${{ secrets.KEY_STORE_PASSWORD }}
          KEY_STORE_PATH: keystore.jks

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Deploy to Play Store internal track
        working-directory: android
        run: bundle exec fastlane deploy_internal
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

      - name: Clean up keystore
        if: always()            # clean up even if build fails
        run: rm -f android/app/keystore.jks

      - name: Notify Slack on success
        if: success()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "✅ *Flutter Android Deploy* — Build ${{ steps.versioning.outputs.BUILD_NUMBER }} deployed to internal track.\nCommit: `${{ github.sha }}`\nAuthor: ${{ github.actor }}"
            }'

      - name: Notify Slack on failure
        if: failure()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "❌ *Flutter Android Deploy Failed* — Build ${{ steps.versioning.outputs.BUILD_NUMBER }}\nCommit: `${{ github.sha }}`\nAuthor: ${{ github.actor }}\nCheck: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }'

  promote-beta:
    name: Promote to Beta
    runs-on: ubuntu-latest
    if: github.event.inputs.action == 'promote-beta'
    environment: beta-promotion    # requires manual approval gate in GitHub environments

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Promote internal build to beta
        working-directory: android
        run: bundle exec fastlane promote_to_beta
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

  promote-production:
    name: Promote to Production
    runs-on: ubuntu-latest
    if: github.event.inputs.action == 'promote-production'
    environment: production-promotion   # requires mandatory reviewer approval

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Promote beta to production (10% rollout)
        working-directory: android
        run: bundle exec fastlane promote_to_production
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

Step 6 - Build Number Strategy

The git rev-list --count HEAD approach ties the build number to the Git commit count, it's monotonically increasing, reproducible from any clone, and doesn't require external state management:

# Build number = total number of commits on the branch
git rev-list --count HEAD
# Output: 847

# Reproducible - the same commit always produces the same build number
# Monotonically increasing - merges to main always increase the count
# No external state - no counter stored in a database or file

This eliminates the classic CI problem of "where is the build number stored?" the answer is always "in Git history."

Step 7 - Environment Protection Rules for Promotion

GitHub Environments add a required human approval gate before the promote-production job runs:

Repository → Settings → Environments → New Environment

Name: production-promotion
Required Reviewers: [release-managers team]
Wait Timer: 0 minutes
Deployment Branches: release

With this configuration, the promote-production workflow job pauses and sends a notification to the release-managers team requesting approval. The 10% staged rollout in Fastlane gives you a further safety buffer, a bad build affects 10% of users before you manually expand the rollout in Play Console.

Step 8 - Caching for Pipeline Speed

Flutter and Gradle dependencies are large, caching them dramatically reduces pipeline runtime:

# Cache Flutter SDK and pub packages
- name: Set up Flutter
  uses: subosito/flutter-action@v2
  with:
    flutter-version: '3.19.0'
    cache: true   # built-in Flutter cache

# Cache Gradle dependencies separately
- name: Cache Gradle
  uses: actions/cache@v4
  with:
    path: |
      ~/.gradle/caches
      ~/.gradle/wrapper
    key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
    restore-keys: |
      ${{ runner.os }}-gradle-

Common Pitfalls to Avoid

Keystore on disk after workflow completes. The if: always() cleanup step is non-negotiable. A keystore file left in the workspace of a GitHub Actions runner even ephemerally, is a security risk. Always delete it, even on failure.

Using personal Google accounts for Play Store API. Personal account credentials tied to a developer are revoked when that person leaves the organization. Service accounts are credential artifacts owned by the organization, rotate them independently of any individual.

Hardcoding the Flutter version. Pin the Flutter version in your workflow (flutter-version: '3.19.0') and update it deliberately. Floating to latest means a Flutter SDK update can silently break your build.

Skipping the test job on the release branch. The test job is the most important gate in the pipeline, removing it "to speed up releases" eliminates the quality guarantee the pipeline is built to provide.

No staged rollout for production. Deploying directly to 100% of production users with no staged rollout turns every release into a high-stakes gamble. Start at 10%, monitor crash rates and ANR rates in the Play Console, and expand manually when metrics confirm stability.

Conclusion

A Flutter deployment pipeline built on GitHub Actions and Fastlane transforms the release process from a fragile, person-dependent ritual into a reproducible, auditable, automated workflow. Tests gate builds. Builds gate deployments. Deployments require approval before reaching production users. Staged rollouts limit the blast radius of regressions.

Every element of the manual process, the keystore on a developer's laptop, the shared Play Console login, the manual version bump, the upload through the UI, has a secure, automated equivalent in this pipeline. The result is a release process your team can run confidently, repeatedly, and without heroics.

Building for iOS alongside Android? The same GitHub Actions workflow structure applies for App Store Connect deployment via Fastlane's deliver action, with Xcode cloud signing replacing the Android keystore step.

Platform Engineering with Spacelift: Building a Self-Service Infrastructure Platform

ThankGod Chibugwum Obobo — Sun, 26 Jul 2026 17:00:00 +0000

Platform engineering has emerged as the discipline that sits between DevOps tooling and the application teams that consume it. Its goal is deceptively simple, make it easy for developers to do the right thing, provision infrastructure, deploy services, manage environments, without requiring deep infrastructure expertise or opening a ticket to a platform team.

The gap between that goal and reality is often filled by a patchwork of CI/CD pipelines, manual approvals, Slack messages, and Terraform scripts that only two people understand. Developer experience suffers. Compliance and security controls exist only in documentation that nobody reads. Infrastructure sprawls across environments with no consistent enforcement.

Spacelift is a platform engineering tool designed specifically to close this gap, providing a collaborative, policy-driven, GitOps-native control plane for Terraform, OpenTofu, Pulumi, Ansible, and Kubernetes. It sits above your IaC tooling and gives platform teams the controls they need to enforce standards, while giving application teams the self-service access they need to move fast without breaking things.

This guide covers how to build a self-service infrastructure platform with Spacelift, from stack architecture and policy enforcement to drift detection, module registries, and the developer experience patterns that make platform engineering actually deliver on its promise.

What Makes Spacelift Different

Before diving into implementation, it helps to understand what Spacelift provides that raw CI/CD pipelines don't:

Capability	Raw CI/CD (GitHub Actions)	Spacelift
Terraform plan/apply	Manual pipeline config	Native, zero-config
Policy enforcement	Shell scripts or OPA sidecars	Built-in OPA policies (plan, access, trigger)
Drift detection	Custom cron jobs	Native, scheduled
Module registry	Terraform Registry or S3	Built-in, versioned
Stack dependencies	Manual pipeline ordering	Native dependency graph
Self-service provisioning	PR-based or ticket-based	Self-service catalog
Audit trail	CI logs	Dedicated, tamper-evident
Secret management	GitHub Secrets	Native Vault/cloud integration
Cost estimation	None	Infracost integration

The fundamental difference is intent, CI/CD pipelines are general-purpose automation. Spacelift is purpose-built infrastructure orchestration, with the safety, compliance, and developer experience features that platform engineering requires.

Core Concepts

Before building, understand Spacelift's key abstractions:

Stack, the fundamental unit in Spacelift. A stack wraps a single IaC configuration (a Terraform root module, a Pulumi program, an Ansible playbook) with its state backend, environment variables, policies, and deployment settings. One stack equals one deployable infrastructure unit.

Space, a hierarchical namespace for grouping and access-controlling stacks. Spaces mirror your organizational structure: root → platform → team-a, root → platform → team-b. Policies and access rules defined at a parent space are inherited by child spaces.

Policy, an OPA (Open Policy Agent) Rego rule that governs stack behavior. Spacelift ships four policy types: plan (approve or reject based on plan output), access (control who can do what), trigger (control when stacks run), and git push (control which commits trigger runs).

Module, a versioned, reusable Terraform module published to Spacelift's built-in registry. Modules are the mechanism through which platform teams publish approved, tested infrastructure patterns that application teams consume via self-service.

Run, a single execution of a stack's IaC code. Every run is tracked with full input context, plan output, policy decisions, approval state, and apply output, providing an immutable audit trail.

Step 1 - Repository and Stack Architecture

Design your repository structure before creating stacks. A common pattern for multi-team platform engineering:

/infrastructure
  /platform           ← platform team owns this
    /networking
      main.tf         ← VPCs, subnets, TGW
    /security
      main.tf         ← IAM roles, security groups, KMS keys
    /eks
      main.tf         ← shared EKS clusters
  /modules            ← approved module library
    /vpc
    /rds
    /s3-secure
    /eks-nodegroup
  /teams              ← application teams own this (self-service)
    /team-alpha
      /staging
        main.tf
      /production
        main.tf
    /team-beta
      /staging
        main.tf
      /production
        main.tf
  /policies           ← Spacelift policies as code
    plan-enforce-tagging.rego
    plan-cost-guard.rego
    access-team-boundaries.rego
    trigger-dependency-chain.rego

This structure enforces a clear ownership boundary, the platform team manages foundational infrastructure and approved modules. Application teams self-service within /teams/{team-name}, constrained by policies that prevent them from touching what they shouldn't.

Step 2 - Creating and Configuring Stacks

Create stacks programmatically using the Spacelift Terraform provider, treating your platform configuration as code:

# spacelift/stacks/platform-networking.tf
resource "spacelift_stack" "platform_networking" {
  name        = "platform-networking"
  description = "Core VPC and network infrastructure"
  space_id    = spacelift_space.platform.id

  repository   = "your-org/infrastructure"
  branch       = "main"
  project_root = "platform/networking"

  terraform_version = "1.6.0"

  autodeploy         = false    # require explicit approval for platform changes
  protect_from_deletion = true

  labels = ["platform", "networking", "critical"]
}

resource "spacelift_stack" "team_alpha_staging" {
  name        = "team-alpha-staging"
  description = "Team Alpha staging environment"
  space_id    = spacelift_space.team_alpha.id

  repository   = "your-org/infrastructure"
  branch       = "main"
  project_root = "teams/team-alpha/staging"

  terraform_version = "1.6.0"

  autodeploy = true    # staging auto-deploys on merge
  labels     = ["team-alpha", "staging"]
}

Configure environment variables and secret injection per stack:

# Environment variables, non-sensitive configuration
resource "spacelift_environment_variable" "team_alpha_region" {
  stack_id  = spacelift_stack.team_alpha_staging.id
  name      = "AWS_DEFAULT_REGION"
  value     = "us-east-1"
  write_only = false
}

# Secret injection from AWS Secrets Manager via Spacelift context
resource "spacelift_context" "aws_credentials" {
  name        = "aws-production-credentials"
  description = "AWS credentials for production deployments"
  space_id    = spacelift_space.platform.id
}

resource "spacelift_context_attachment" "team_alpha_prod" {
  context_id = spacelift_context.aws_credentials.id
  stack_id   = spacelift_stack.team_alpha_staging.id
  priority   = 1
}

Contexts are the Spacelift equivalent of shared environment variable sets, define credentials once at the platform level, attach to any number of stacks, and rotate in one place.

Step 3 - Policies as Code with OPA

Policies are where platform engineering leverage lives. A policy written once enforces a standard across every stack in your platform, no per-team configuration, no tribal knowledge, no exceptions that slip through code review.

Plan Policy - Enforce Resource Tagging

Reject any Terraform plan that creates resources without mandatory tags:

# policies/plan-enforce-tagging.rego
package spacelift

# Required tags for all resources in production
required_tags := {"Environment", "Team", "CostCenter", "ManagedBy"}

# Deny if any resource is missing required tags
deny[sprintf("Resource '%s' is missing required tags: %v", [resource.address, missing])] {
  resource := input.terraform.resource_changes[_]
  resource.change.actions[_] == "create"

  # Identify which required tags are missing
  provided_tags := {tag | resource.change.after.tags[tag]}
  missing := required_tags - provided_tags
  count(missing) > 0
}

# Warn if tags exist but Environment doesn't match a known value
warn[sprintf("Resource '%s' has unrecognized Environment tag: '%s'", [resource.address, env])] {
  resource := input.terraform.resource_changes[_]
  env := resource.change.after.tags.Environment
  not env in {"staging", "production", "development"}
}

Plan Policy - Cost Guard

Reject plans that would exceed a per-run cost threshold:

# policies/plan-cost-guard.rego
package spacelift

# Block plans with monthly cost increase above $500
deny["Monthly cost increase exceeds $500 threshold — escalate to platform team for approval"] {
  input.spacelift.run.cost_estimate.monthly_cost_increase > 500
}

warn["Monthly cost increase above $100 — ensure this is expected"] {
  input.spacelift.run.cost_estimate.monthly_cost_increase > 100
  input.spacelift.run.cost_estimate.monthly_cost_increase <= 500
}

Access Policy - Team Boundary Enforcement

Prevent teams from reading or modifying stacks they don't own:

# policies/access-team-boundaries.rego
package spacelift

# Extract team from the user's IdP groups
user_teams := {team |
  group := input.session.teams[_]
  team := trim_prefix(group, "team-")
}

# Allow access only to stacks labeled with the user's team
allow {
  stack_team := input.stack.labels[_]
  stack_team in user_teams
}

# Platform team members have access to all stacks
allow {
  "platform" in input.session.teams
}

# Block all other access
deny {
  not allow
}

Trigger Policy - Stack Dependency Chains

Automatically trigger downstream stacks when upstream infrastructure changes:

# policies/trigger-dependency-chain.rego
package spacelift

# When platform-networking applies successfully, trigger EKS stack
trigger["platform-eks"] {
  input.run.stack.id == "platform-networking"
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"
}

# When platform-eks applies successfully, trigger team stacks
trigger["team-alpha-staging"] {
  input.run.stack.id == "platform-eks"
  input.run.state == "FINISHED"
}

Attach policies to stacks or spaces:

resource "spacelift_policy" "enforce_tagging" {
  name = "enforce-resource-tagging"
  type = "PLAN"
  body = file("policies/plan-enforce-tagging.rego")
}

# Attach to the entire platform space — all child stacks inherit
resource "spacelift_policy_attachment" "enforce_tagging_platform" {
  policy_id = spacelift_policy.enforce_tagging.id
  stack_id  = "*"
  space_id  = spacelift_space.platform.id
}

Step 4 - Module Registry for Self-Service

The Spacelift module registry is the mechanism through which platform teams publish approved infrastructure patterns for self-service consumption. A module published to the registry is versioned, tested, and policy-compliant by definition, application teams consume it without needing to understand the underlying infrastructure design.

Publish a module to Spacelift's registry:

# spacelift/modules/rds-module.tf
resource "spacelift_module" "rds_postgres" {
  name           = "rds-postgres"
  space_id       = spacelift_space.platform.id
  repository     = "your-org/infrastructure"
  project_root   = "modules/rds"
  branch         = "main"
  description    = "Production-grade PostgreSQL RDS with encryption, backups, and parameter group"

  terraform_provider = "aws"

  labels = ["database", "postgres", "approved"]
}

Application teams consume it in their stacks like any Terraform module:

# teams/team-alpha/staging/main.tf
module "orders_database" {
  source  = "spacelift.io/your-org/rds-postgres/aws"
  version = "~> 2.0"

  identifier        = "orders-staging"
  instance_class    = "db.t3.medium"
  allocated_storage = 20
  environment       = "staging"
  team              = "team-alpha"
}

The version constraint (~> 2.0) means Team Alpha gets patch updates automatically but won't be silently upgraded to a breaking major version. The platform team controls what's published, application teams control what version they pin to.

Step 5 - Drift Detection

Infrastructure drift, the gap between what Terraform's state says exists and what actually exists in your cloud account, is one of the most persistent operational problems in IaC. Manual changes through the AWS console, emergency hotfixes applied directly, or race conditions between concurrent applies all cause drift.

Spacelift's built-in drift detection runs Terraform plan on a schedule and alerts when the live state diverges from the declared state:

resource "spacelift_stack" "platform_networking" {
  # ... other config ...

  # Run drift detection daily at 2am UTC
  drift_detection {
    enabled   = true
    reconcile = false    # alert on drift but don't auto-remediate — require human review
    schedule  = ["0 2 * * *"]
  }
}

For non-critical stacks where auto-remediation is safe:

drift_detection {
  enabled   = true
  reconcile = true     # automatically apply to reconcile drift
  schedule  = ["0 */6 * * *"]  # every 6 hours
}

Configure Slack or PagerDuty notifications for drift alerts:

resource "spacelift_webhook" "drift_alert" {
  stack_id = spacelift_stack.platform_networking.id
  endpoint = var.slack_webhook_url
  enabled  = true

  # Trigger on drift detection and failed reconciliation
  secret = var.webhook_secret
}

Step 6 - Self-Service Developer Experience

The final layer is the developer experience that makes self-service actually self-service. Spacelift's self-service catalog (Spaces + module registry + approval workflows) lets application teams provision infrastructure through a UI or API without opening tickets:

For application teams: a curated list of available modules (RDS, S3, SQS, ElastiCache) with a form-based provisioning interface, no Terraform knowledge required.

For platform teams: full visibility into what every team has provisioned, policy enforcement that fires automatically, drift detection that runs continuously, and an audit trail that captures every change with the identity of who made it and what the plan output was.

Connect Spacelift to your IdP (Okta, GitHub, Google) to enforce team boundaries automatically via SSO group membership, the access policies defined in Step 3 apply automatically as team membership changes.

Step 7 - CI/CD Integration

Spacelift integrates with GitHub, GitLab, Bitbucket, and Azure DevOps for Git-driven runs. A push to main in the infrastructure repo triggers plan runs on affected stacks automatically:

# .github/workflows/spacelift-preview.yml
name: Spacelift Plan Preview

on: [pull_request]

jobs:
  plan-preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Comment plan output on PR
        uses: spacelift-io/spacelift-pull-request-action@v1
        with:
          spacelift-api-key-id: ${{ secrets.SPACELIFT_API_KEY_ID }}
          spacelift-api-key-secret: ${{ secrets.SPACELIFT_API_KEY_SECRET }}

For stacks with autodeploy: false, Spacelift posts the plan output as a PR comment, requiring a human to review and approve before the apply runs. For stacks with autodeploy: true (staging environments), the apply runs automatically on merge to main.

Common Pitfalls to Avoid

Flat space hierarchy. A single space for all stacks gives no access boundary between teams. Design your space hierarchy to mirror your team structure from day one, retrofitting access controls onto a flat hierarchy is painful.

Policies without testing. A deny policy that fires incorrectly can block all deployments across your entire platform. Test policies against representative plan inputs using OPA's opa eval before attaching them to production stacks.

Auto-reconciling critical infrastructure. Drift auto-remediation is powerful but dangerous for foundational infrastructure. Networking, security groups, and IAM roles should alert on drift and require human review, not automatically apply. Reserve auto-reconciliation for stateless, idempotent resource sets.

No module versioning discipline. Publishing a breaking change to a module and expecting all consumers to upgrade simultaneously doesn't scale. Adopt semantic versioning for modules and maintain at least one prior major version while teams migrate.

Conclusion

Platform engineering is not about giving developers access to infrastructure tooling, it's about designing guardrails and self-service patterns so that developers can provision what they need safely, consistently, and without creating operational debt for the platform team.

Spacelift makes this achievable by providing a purpose-built control plane for IaC, policy enforcement that fires at plan time, a module registry that codifies approved patterns, drift detection that catches manual changes, and a space hierarchy that enforces team boundaries without manual intervention.

The platform team's job evolves from running infrastructure to building the platform that lets others run theirs safely, at scale, and with a complete audit trail.

Running OpenTofu instead of Terraform, or mixing Pulumi with Ansible? Spacelift's multi-IaC support handles heterogeneous toolchains in the same platform all under the same policy and audit model. Drop your stack in the comments.

PlatformEngineering #Spacelift #Terraform #InfrastructureAsCode #DevOps

Implementing mTLS in Kubernetes: How to Secure Service-to-Service Communication

ThankGod Chibugwum Obobo — Sun, 19 Jul 2026 17:00:00 +0000

Introduction

In a microservices architecture running on Kubernetes, services communicate constantly, the orders-service calls the users-service, the payments-service calls the fraud-detection-service, and the API gateway routes to a dozen downstream services. By default, most of this traffic travels unencrypted inside the cluster, authenticated only by the fact that the caller has network access.

That assumption that anything inside the cluster boundary is trustworthy, is the network-level equivalent of leaving every internal door unlocked because the building has a lock on the front entrance. A single compromised pod, a misconfigured network policy, or a lateral movement attack can reach any service in the cluster with no additional authentication required.

Mutual TLS (mTLS) closes this gap. Unlike standard TLS where only the server presents a certificate, mTLS requires both parties, client and server, to present and verify certificates before any data is exchanged. Every service-to-service connection is cryptographically authenticated. A compromised pod cannot impersonate payments-service to fraud-detection-service because it cannot produce a valid certificate for that identity.

This guide covers how to implement mTLS in Kubernetes using a service mesh, comparing Istio and Linkerd, configuring automatic certificate issuance, enforcing strict mTLS policies, and validating that your network is actually secured.

How mTLS Works in Kubernetes

Standard TLS in a web context works like this: the client (browser) verifies the server's certificate, establishing that the server is who it claims to be, but the server accepts any client. mTLS adds the reverse, the server also verifies the client's certificate, establishing mutual identity before the connection proceeds.

In Kubernetes, the mechanics are:

Each service is issued a certificate tied to its SPIFFE identity, a URI in the format spiffe://cluster.local/ns/{namespace}/sa/{service-account}.
When orders-service connects to users-service, both services present their certificates.
The receiving service verifies the caller's certificate against the cluster's trust anchor (a root CA).
If verification passes, the connection proceeds. If not, wrong identity, expired certificate, or untrusted issuer, the connection is rejected at the TLS layer, before any application code runs.

The critical operational property: certificates are automatically rotated by the service mesh control plane, typically every 24 hours. No manual certificate management, no expiry incidents, no long-lived credentials.

Choosing a Service Mesh: Istio vs. Linkerd

Both Istio and Linkerd implement mTLS via sidecar proxies injected into each pod. The proxies intercept all inbound and outbound traffic, transparently handling TLS without requiring application code changes.

Feature	Istio	Linkerd
mTLS implementation	Envoy sidecar	Linkerd2-proxy sidecar
Certificate authority	Istiod (built-in) or external	Built-in or cert-manager
mTLS enforcement	PeerAuthentication policy	Server policy
Observability	Rich (Kiali, Jaeger integration)	Lightweight (built-in dashboard)
Resource overhead	Higher (~200–350MB per sidecar)	Lower (~10–20MB per sidecar)
Configuration complexity	Higher	Lower
FIPS 140-2 compliance	Enterprise	Enterprise
Multi-cluster support	Native	Native

Choose Istio if you need advanced traffic management (canary deployments, fault injection, circuit breaking), rich observability, or WebAssembly-based policy extensibility.

Choose Linkerd if you prioritize simplicity, low resource overhead, and want mTLS with strong defaults out of the box with minimal configuration.

This guide covers both installation and policy configuration for each.

Approach 1 - mTLS with Istio

Step 1 - Install Istio

Install Istio using the istioctl CLI with the minimal production profile:

# Install istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-*/bin:$PATH

# Install Istio with production profile
istioctl install --set profile=default \
  --set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true \
  -y

# Verify installation
istioctl verify-install

Step 2 - Enable Automatic Sidecar Injection

Label namespaces where you want mTLS enforced. Istio automatically injects the Envoy sidecar proxy into every pod in labeled namespaces:

# Enable sidecar injection for your application namespaces
kubectl label namespace orders-service istio-injection=enabled
kubectl label namespace users-service istio-injection=enabled
kubectl label namespace payments-service istio-injection=enabled

# Verify injection is active
kubectl get namespace -L istio-injection

Rolling restart existing deployments to pick up sidecar injection:

kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service

Verify sidecars are injected, each pod should now show 2/2 containers (application + Envoy proxy):

kubectl get pods -n orders-service
# NAME                             READY   STATUS    RESTARTS   AGE
# orders-api-7d8b9f5c4-xk2pq      2/2     Running   0          45s

Step 3 - Configure PeerAuthentication for Strict mTLS

By default, Istio operates in permissive mode, it accepts both plaintext and mTLS traffic. This is safe for migration but provides no security guarantee. Switch to strict mode to enforce mTLS and reject all plaintext traffic:

# istio/peer-authentication-strict.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system   # mesh-wide policy
spec:
  mtls:
    mode: STRICT

kubectl apply -f istio/peer-authentication-strict.yaml

This single resource enforces mTLS for all service-to-service communication across the entire mesh. Any pod that attempts to connect to another pod without a valid certificate, including debug containers, legacy services without sidecars, or external probes, will have their connection rejected.

Step 4 - AuthorizationPolicy for Fine-Grained Access Control

mTLS establishes who a caller is. AuthorizationPolicy controls what authenticated identities are allowed to do, combining mTLS identity with RBAC-style rules:

# istio/authorization-payments.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payments-service-policy
  namespace: payments-service
spec:
  selector:
    matchLabels:
      app: payments-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              # Only orders-service can call payments-service
              - "cluster.local/ns/orders-service/sa/orders-service-sa"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/payments/charge", "/payments/refund"]

# Deny all traffic not explicitly allowed, defense in depth
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: payments-service
spec:
  {}  # Empty spec = deny all by default

Apply deny-all first, then layer specific ALLOW policies on top. This is the zero-trust principle applied at the service level: every inter-service call must be explicitly authorized.

Step 5 - Verify mTLS is Active with Kiali

Kiali, Istio's observability console, provides a visual confirmation of mTLS status:

# Install Kiali and supporting tools
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml

# Access Kiali dashboard
istioctl dashboard kiali

In Kiali's service graph, a padlock icon on each edge confirms mTLS is active on that connection. Missing padlocks indicate a service communicating without mTLS, a misconfiguration or missing sidecar.

Approach 2 - mTLS with Linkerd

Step 1 - Install Linkerd

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Validate cluster compatibility
linkerd check --pre

# Install Linkerd control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify installation
linkerd check

Step 2 - Inject Linkerd Proxy into Namespaces

# Annotate namespaces for automatic injection
kubectl annotate namespace orders-service linkerd.io/inject=enabled
kubectl annotate namespace users-service linkerd.io/inject=enabled
kubectl annotate namespace payments-service linkerd.io/inject=enabled

# Restart deployments to pick up proxy injection
kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service

Step 3 - Enforce mTLS with Server Policies

Linkerd enables mTLS by default in injected namespaces. To enforce authenticated traffic only (equivalent to Istio's strict mode), configure Server policies:

# linkerd/server-policy-payments.yaml
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: payments-server
  namespace: payments-service
spec:
  podSelector:
    matchLabels:
      app: payments-service
  port: 8080
  proxyProtocol: HTTP/2


apiVersion: policy.linkerd.io/v1beta2
kind: ServerAuthorization
metadata:
  name: payments-authz
  namespace: payments-service
spec:
  server:
    name: payments-server
  client:
    meshTLS:
      serviceAccounts:
        - name: orders-service-sa
          namespace: orders-service

kubectl apply -f linkerd/server-policy-payments.yaml

# Verify mTLS is active on connections
linkerd viz edges deployment -n orders-service
# Shows which connections are secured with mTLS (✓) vs plaintext (✗)

Certificate Management with cert-manager

Both Istio and Linkerd can integrate with cert-manager for certificate issuance, enabling integration with external CAs (HashiCorp Vault, AWS ACM PCA, Let's Encrypt) rather than relying solely on the mesh's built-in CA.

# cert-manager/cluster-issuer-vault.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.internal:8200
    path: pki/sign/kubernetes-mesh
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager
        secretRef:
          name: cert-manager-vault-token
          key: token

For Istio, plug the external CA into the control plane:

# Use cert-manager as Istio's certificate provider
istioctl install --set profile=default \
  --set values.pilot.env.EXTERNAL_CA=true \
  --set values.pilot.env.K8S_SIGNER=cert-manager.io/cluster-issuer/vault-issuer

External CA integration provides:

Centralized audit trail of every certificate issuance across your mesh
Cross-cluster trust using a shared root CA for multi-cluster mTLS
Compliance alignment with PKI policies already in use in your organization

Migrating to Strict mTLS Without Downtime

Flipping directly to strict mode on a production cluster breaks any service that doesn't yet have a sidecar. Use a phased migration:

Phase 1 - Permissive mode (baseline). Install the mesh with mTLS in permissive mode. All traffic is accepted, both plaintext and mTLS. Use Kiali or linkerd viz to identify which connections are NOT using mTLS.

Phase 2 - Namespace-by-namespace injection. Enable sidecar injection one namespace at a time. Verify each namespace's workloads after injection before proceeding.

Phase 3 - Strict mode per namespace. Apply PeerAuthentication or Server policies with strict enforcement to already-injected namespaces:

# Namespace-scoped strict policy (safer than mesh-wide during migration)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: payments-service   # scoped to one namespace
spec:
  mtls:
    mode: STRICT

Phase 4 - Mesh-wide strict enforcement. Once all namespaces are injected and verified, apply the mesh-wide PeerAuthentication in istio-system and remove namespace-scoped policies.

This phased approach takes longer but eliminates the risk of a mesh-wide strict enforcement breaking services that weren't ready.

Validating mTLS Enforcement

After enabling strict mode, validate it is actually enforced, don't assume:

# Attempt to connect to payments-service from a pod WITHOUT a sidecar
kubectl run debug-pod --image=curlimages/curl --restart=Never -- \
  curl -s http://payments-service.payments-service.svc.cluster.local:8080/health

# Expected result: connection refused or reset, mTLS is enforced
# Unexpected result: 200 OK, mTLS is NOT enforced, investigate

# Verify certificate details on an active connection
kubectl exec -n orders-service deployment/orders-api -c istio-proxy -- \
  openssl s_client \
    -connect payments-service.payments-service.svc.cluster.local:8080 \
    -showcerts 2>/dev/null | grep -A2 "subject="

# Expected output shows SPIFFE URI:
# subject=URI:spiffe://cluster.local/ns/payments-service/sa/payments-service-sa

# Check Istio's view of mTLS status per service
istioctl x describe service payments-service.payments-service

# Output confirms:
# mTLS: yes
# Policy: STRICT

Observability and Troubleshooting

mTLS adds authentication at the network layer, but it also adds debugging complexity. Keep these tools in your operational toolkit:

Istio proxy logs reveal TLS handshake failures with specific error codes:

# View Envoy access logs for a specific pod
kubectl logs -n orders-service deployment/orders-api -c istio-proxy | \
  grep -E "UF|UC|UH"  # Upstream connection failure codes

Linkerd tap provides real-time traffic inspection with mTLS status:

linkerd viz tap deployment/orders-api -n orders-service \
  --to deployment/payments-service -n payments-service

Certificate expiry monitoring even with automatic rotation, monitor certificate validity as a health signal:

# Prometheus alert for certificates expiring within 24 hours
- alert: MeshCertificateExpiringSoon
  expr: |
    (citadel_server_csr_count - citadel_server_success_cert_issuance_count) > 0
  for: 5m
  labels:
    severity: high
  annotations:
    summary: "Certificate issuance failures in Istio CA"

Common Pitfalls to Avoid

Enabling strict mode before all services have sidecars. Services without sidecars cannot participate in mTLS and will have their connections rejected in strict mode. Always verify 100% sidecar coverage before enabling mesh-wide strict enforcement.

Forgetting health check probes. Kubernetes liveness and readiness probes originate from the kubelet, outside the mesh. Configure Istio to rewrite probe URLs or exclude probe ports from mTLS enforcement:

# Exclude health check port from mTLS, kubelet doesn't have a sidecar
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    8081:          # health check port
      mode: DISABLE

Not testing AuthorizationPolicy before applying. A misconfigured deny-all policy can cut off legitimate service traffic in production. Test policies in a staging environment first, and use Istio's AUDIT action mode to log would-be denials before enforcing them.

Ignoring certificate rotation failures. The automatic certificate rotation is the operational backbone of mesh mTLS. Monitor certificate issuance metrics, a rotation failure that goes undetected leads to widespread connection failures when certificates expire.

Conclusion

Mutual TLS in Kubernetes transforms your cluster's internal network from an implicit trust zone into a zero-trust environment where every connection is cryptographically authenticated and explicitly authorized. A compromised pod cannot impersonate a legitimate service. A lateral movement attack cannot freely reach payment processing or user data services. Every inter-service call is auditable.

The operational cost, a service mesh, sidecar overhead, policy management, is real. But for systems where data sensitivity, regulatory requirements, or blast-radius concerns justify it, mTLS via a service mesh is the most complete implementation of zero-trust networking currently available for Kubernetes workloads.

Start with permissive mode, inject sidecars namespace by namespace, validate mTLS coverage with your mesh's observability tools, and graduate to strict enforcement once you're confident in coverage. The security guarantee at the end is worth the careful migration.

Running a multi-cluster Kubernetes setup? Extending mTLS across cluster boundaries with Istio's multi-cluster federation or Linkerd's cluster linking uses the same SPIFFE identity model, but adds a trust bundle exchange step between clusters.

DRY Infrastructure: How to Architect Modular Ansible Playbooks for Next.js

ThankGod Chibugwum Obobo — Sun, 12 Jul 2026 17:00:00 +0000

Infrastructure code suffers from the same antipatterns as application code, duplication, tight coupling, undocumented assumptions, and a growing resistance to change. A playbook written for one environment gets copy-pasted for the next, variables get hardcoded, and six months later you have five slightly different versions of the same deployment logic, each drifting independently and none of them trustworthy.

The DRY (Don't Repeat Yourself) principle applies to infrastructure as directly as it does to software. In Ansible, DRY infrastructure means roles for reusable task logic, variable hierarchies for environment-specific overrides, handlers for event-driven operations, and a playbook composition pattern that assembles production-grade deployments from interchangeable, independently testable units.

For Next.js, with its hybrid rendering modes, standalone server output, static asset optimization, and Node.js runtime requirements, a modular Ansible approach gives you a repeatable, environment-aware provisioning and deployment pipeline that works consistently from local staging to production at scale.

This guide covers how to design and implement a modular Ansible architecture for Next.js from the ground up.

Why Ansible for Next.js Infrastructure

Before diving into structure, it's worth addressing why Ansible fits Next.js provisioning well, especially alongside more modern IaC tools:

Ansible excels at configuration management and application deployment, the layer between raw infrastructure (where Terraform lives) and application orchestration (where Kubernetes lives). For teams running Next.js on VMs, bare metal, or alongside a lightweight container setup, Ansible bridges that gap cleanly.

Specifically for Next.js:

Node.js runtime management (version pinning, nvm, process managers)
Build artifact deployment (transferring .next/standalone output, managing static assets to CDN)
Nginx reverse proxy configuration (SSL termination, caching headers, compression)
Environment variable injection (.env files templated from Vault or group vars)
PM2 or systemd process management (zero-downtime restarts, health checks)
Log rotation and monitoring agent configuration

This is precisely where Ansible's agentless, SSH-based, idempotent task execution shines.

Step 1 - Project Structure for Modular Playbooks

Start with a structure that enforces separation of concerns from day one:

/ansible
  /inventories
    /staging
      hosts.yml
      group_vars/
        all.yml
        nextjs_servers.yml
    /production
      hosts.yml
      group_vars/
        all.yml
        nextjs_servers.yml
  /roles
    /common           ← base system configuration
    /nodejs           ← Node.js runtime installation and management
    /nextjs-app       ← Next.js application deployment
    /nginx            ← Nginx configuration and SSL
    /pm2              ← PM2 process management
    /monitoring       ← monitoring agent setup
    /logrotate        ← log rotation configuration
  /playbooks
    site.yml          ← master playbook (composes all roles)
    provision.yml     ← first-time server provisioning
    deploy.yml        ← application deployment only
    rollback.yml      ← rollback to previous release
  /group_vars
    all.yml           ← variables shared across all environments
  ansible.cfg

This structure reflects three key design decisions:

Separate provisioning from deployment. provision.yml runs once (or rarely) to set up a server. deploy.yml runs on every release. Conflating them means every deployment re-runs expensive provisioning tasks unnecessarily.

Separate inventories per environment. Staging and production are entirely isolated inventory trees, different hosts, different group vars, no risk of accidentally running a production playbook against staging or vice versa.

Roles are the unit of reuse. Every discrete concern is a role. Roles are independently versioned, testable with Molecule, and composable in any playbook.

Step 2 - Ansible Configuration

# ansible.cfg
[defaults]
inventory          = ./inventories
roles_path         = ./roles
host_key_checking  = False
retry_files_enabled = False
stdout_callback    = yaml
interpreter_python = auto_silent
forks              = 10

[ssh_connection]
ssh_args           = -o ControlMaster=auto -o ControlPersist=60s
pipelining         = True   # significant performance improvement

pipelining = True reduces the number of SSH connections per task, one of the most impactful performance improvements for large fleets. It requires requiretty to be disabled in sudoers on your target hosts.

Step 3 - Variable Hierarchy and Environment Overlays

Ansible's variable precedence is the mechanism for DRY environment-specific configuration. Define defaults at the broadest scope, override only what differs per environment:

# group_vars/all.yml - shared across ALL environments
nodejs_version: "20.11.0"
app_user: "nextjs"
app_group: "nextjs"
app_base_dir: "/opt/nextjs"
releases_to_keep: 5
pm2_instances: "max"
nginx_worker_processes: "auto"

# inventories/staging/group_vars/nextjs_servers.yml - staging overrides
app_environment: "staging"
app_port: 3000
next_public_api_url: "https://api.staging.your-domain.com"
pm2_instances: 2                  # fewer instances on smaller staging servers
nginx_ssl_certificate: "/etc/letsencrypt/live/staging.your-domain.com/fullchain.pem"
nginx_ssl_key: "/etc/letsencrypt/live/staging.your-domain.com/privkey.pem"
app_releases_dir: "{{ app_base_dir }}/releases"

# inventories/production/group_vars/nextjs_servers.yml - production overrides
app_environment: "production"
app_port: 3000
next_public_api_url: "https://api.your-domain.com"
pm2_instances: "max"              # use all available CPU cores
nginx_ssl_certificate: "/etc/letsencrypt/live/your-domain.com/fullchain.pem"
nginx_ssl_key: "/etc/letsencrypt/live/your-domain.com/privkey.pem"
app_releases_dir: "{{ app_base_dir }}/releases"

Sensitive values (database URLs, API keys, secrets) should never appear in group vars. Use Ansible Vault for sensitive variable encryption or integrate with HashiCorp Vault for dynamic secret injection, as covered in our guide on replacing hardcoded credentials with Vault and GitHub OIDC.

Step 4 - The Node.js Role

The Node.js role manages runtime installation via nvm, ensuring version pinning and easy upgrades without system package manager conflicts:

roles/nodejs/
  tasks/
    main.yml
  defaults/
    main.yml
  handlers/
    main.yml

# roles/nodejs/defaults/main.yml
nodejs_version: "20.11.0"
nvm_version: "0.39.7"
nvm_dir: "/home/{{ app_user }}/.nvm"
npm_global_packages:
  - pm2
  - typescript

# roles/nodejs/tasks/main.yml
---
- name: Create application user
  ansible.builtin.user:
    name: "{{ app_user }}"
    group: "{{ app_group }}"
    system: true
    shell: /bin/bash
    home: "/home/{{ app_user }}"
    create_home: true
  tags: [nodejs, user]

- name: Check if nvm is installed
  ansible.builtin.stat:
    path: "{{ nvm_dir }}/nvm.sh"
  register: nvm_stat
  tags: [nodejs, nvm]

- name: Install nvm
  ansible.builtin.shell:
    cmd: >
      curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v{{ nvm_version }}/install.sh | bash
  become: true
  become_user: "{{ app_user }}"
  when: not nvm_stat.stat.exists
  tags: [nodejs, nvm]

- name: Install Node.js via nvm
  ansible.builtin.shell:
    cmd: >
      source {{ nvm_dir }}/nvm.sh &&
      nvm install {{ nodejs_version }} &&
      nvm alias default {{ nodejs_version }}
  args:
    executable: /bin/bash
  become: true
  become_user: "{{ app_user }}"
  tags: [nodejs]

- name: Install global npm packages
  ansible.builtin.shell:
    cmd: >
      source {{ nvm_dir }}/nvm.sh &&
      npm install -g {{ item }}
  args:
    executable: /bin/bash
  become: true
  become_user: "{{ app_user }}"
  loop: "{{ npm_global_packages }}"
  tags: [nodejs, npm]

Step 5 - The Next.js Application Role

The Next.js role implements a release directory pattern - the key mechanism for zero-downtime deployments and fast rollbacks:

roles/nextjs-app/
  tasks/
    main.yml
    deploy.yml
    rollback.yml
    cleanup.yml
  templates/
    env.j2            ← .env.local template
    pm2.config.js.j2  ← PM2 ecosystem config template
  handlers/
    main.yml
  defaults/
    main.yml

# roles/nextjs-app/tasks/deploy.yml
- name: Generate release timestamp
  ansible.builtin.set_fact:
    release_timestamp: "{{ ansible_date_time.epoch }}"
  tags: [deploy]

- name: Create release directory
  ansible.builtin.file:
    path: "{{ app_releases_dir }}/{{ release_timestamp }}"
    state: directory
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
    mode: "0755"
  tags: [deploy]

- name: Upload Next.js standalone build
  ansible.builtin.synchronize:
    src: "{{ local_build_path }}/.next/standalone/"
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}"
    delete: true
    rsync_opts:
      - "--exclude=.git"
  tags: [deploy]

- name: Upload static assets
  ansible.builtin.synchronize:
    src: "{{ local_build_path }}/.next/static/"
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}/.next/static/"
  tags: [deploy]

- name: Template environment file
  ansible.builtin.template:
    src: env.j2
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}/.env.local"
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
    mode: "0640"           # restrict read to app user only
  tags: [deploy]

- name: Update current symlink atomically
  ansible.builtin.file:
    src: "{{ app_releases_dir }}/{{ release_timestamp }}"
    dest: "{{ app_base_dir }}/current"
    state: link
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
  notify: Reload PM2
  tags: [deploy]

The atomic symlink swap (current → new release) is the mechanism that makes zero-downtime deployments possible. PM2 reloads against the new current directory; if the reload fails, the symlink points back to the previous release.

# roles/nextjs-app/tasks/rollback.yml
- name: List available releases
  ansible.builtin.find:
    paths: "{{ app_releases_dir }}"
    file_type: directory
  register: available_releases
  tags: [rollback]

- name: Identify previous release
  ansible.builtin.set_fact:
    previous_release: >-
      {{
        available_releases.files
        | sort(attribute='mtime', reverse=True)
        | map(attribute='path')
        | list
        | nth(1)
      }}
  tags: [rollback]

- name: Rollback symlink to previous release
  ansible.builtin.file:
    src: "{{ previous_release }}"
    dest: "{{ app_base_dir }}/current"
    state: link
  notify: Reload PM2
  tags: [rollback]

- name: Report rollback target
  ansible.builtin.debug:
    msg: "Rolled back to: {{ previous_release }}"
  tags: [rollback]

Step 6 - The Nginx Role

# roles/nginx/templates/nextjs.conf.j2
upstream nextjs_upstream {
  server 127.0.0.1:{{ app_port }};
  keepalive 64;
}

server {
  listen 80;
  server_name {{ nginx_server_name }};
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  server_name {{ nginx_server_name }};

  ssl_certificate     {{ nginx_ssl_certificate }};
  ssl_certificate_key {{ nginx_ssl_key }};
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
  ssl_prefer_server_ciphers off;

  # Next.js static assets, long-lived cache with content hashing
  location /_next/static/ {
    alias {{ app_base_dir }}/current/.next/static/;
    expires 1y;
    add_header Cache-Control "public, immutable";
  }

  # Public assets
  location /public/ {
    alias {{ app_base_dir }}/current/public/;
    expires 30d;
    add_header Cache-Control "public";
  }

  # Next.js server
  location / {
    proxy_pass         http://nextjs_upstream;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection 'upgrade';
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_cache_bypass $http_upgrade;
  }
}

# roles/nginx/tasks/main.yml
- name: Install Nginx
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true
  tags: [nginx]

- name: Template Next.js site configuration
  ansible.builtin.template:
    src: nextjs.conf.j2
    dest: /etc/nginx/sites-available/nextjs
    owner: root
    group: root
    mode: "0644"
  notify: Reload Nginx
  tags: [nginx]

- name: Enable site
  ansible.builtin.file:
    src: /etc/nginx/sites-available/nextjs
    dest: /etc/nginx/sites-enabled/nextjs
    state: link
  notify: Reload Nginx
  tags: [nginx]

# roles/nginx/handlers/main.yml
- name: Reload Nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded
  listen: Reload Nginx

Step 7 - Composing Playbooks

With roles defined, playbooks become thin composition layers:

# playbooks/provision.yml - run once to set up new servers
- name: Provision Next.js servers
  hosts: nextjs_servers
  become: true
  roles:
    - role: common
      tags: [common]
    - role: nodejs
      tags: [nodejs]
    - role: nginx
      tags: [nginx]
    - role: pm2
      tags: [pm2]
    - role: logrotate
      tags: [logrotate]
    - role: monitoring
      tags: [monitoring]

# playbooks/deploy.yml - run on every release
- name: Deploy Next.js application
  hosts: nextjs_servers
  become: true
  vars:
    local_build_path: "{{ lookup('env', 'NEXT_BUILD_PATH') }}"

  pre_tasks:
    - name: Validate build artifact exists
      ansible.builtin.stat:
        path: "{{ local_build_path }}/.next/standalone"
      delegate_to: localhost
      register: build_stat
      failed_when: not build_stat.stat.exists

  roles:
    - role: nextjs-app
      tags: [deploy]

  post_tasks:
    - name: Verify application is healthy
      ansible.builtin.uri:
        url: "https://{{ nginx_server_name }}/api/health"
        status_code: 200
        timeout: 30
      retries: 5
      delay: 10
      register: health_check
      until: health_check.status == 200

# playbooks/rollback.yml
- name: Rollback Next.js to previous release
  hosts: nextjs_servers
  become: true
  tasks:
    - name: Execute rollback
      ansible.builtin.include_role:
        name: nextjs-app
        tasks_from: rollback

Step 8 - CI/CD Integration

# .github/workflows/deploy-nextjs.yml
name: Deploy Next.js

on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install and build
        run: |
          npm ci
          npm run build

      - name: Install Ansible
        run: pip install ansible ansible-lint --break-system-packages

      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
          chmod 600 ~/.ssh/deploy_key
          echo "StrictHostKeyChecking no" >> ~/.ssh/config

      - name: Deploy to production
        run: |
          ansible-playbook playbooks/deploy.yml \
            -i inventories/production/hosts.yml \
            --private-key ~/.ssh/deploy_key \
            -e "NEXT_BUILD_PATH=${{ github.workspace }}"
        env:
          ANSIBLE_HOST_KEY_CHECKING: "False"

Common Pitfalls to Avoid

Using shell tasks where native modules exist. ansible.builtin.shell bypasses idempotency, it runs every time regardless of whether the state has changed. Always prefer native modules (apt, copy, template, service) and fall back to shell only when no native module covers the task.

Hardcoding environment-specific values in roles. Role defaults should be genuinely sensible defaults, not production values masquerading as defaults. Production specifics belong in inventory group_vars.

Not cleaning up old releases. The release directory pattern accumulates historical deployments. Add a cleanup task to retain only {{ releases_to_keep }} releases, the rest is disk usage with no operational value.

Skipping the health check post-task. A deploy playbook that doesn't verify the application is healthy after deployment is incomplete. The health check is the difference between a successful deploy and an undetected outage.

Conclusion

Modular Ansible playbooks are not just a style preference, they are an operational necessity at any scale beyond a single server. Roles as units of reuse, variable hierarchies for environment overlays, the release directory pattern for zero-downtime deployments, and thin playbooks for composition give you an infrastructure codebase that is maintainable, testable, and genuinely DRY.

For Next.js specifically, this architecture handles the full lifecycle, provisioning the Node.js runtime, deploying standalone build artifacts, configuring Nginx for optimal caching of static assets, managing PM2 for process reliability, and providing a one-command rollback when things go wrong.

Infrastructure code that can't be understood, changed, and trusted by the team that operates it is a liability. Infrastructure code built to the same quality standards as your application code is a competitive advantage.