DEV Community: ThankGod Chibugwum Obobo

Platform Engineering with Spacelift: Building a Self-Service Infrastructure Platform

ThankGod Chibugwum Obobo — Sun, 26 Jul 2026 17:00:00 +0000

Platform engineering has emerged as the discipline that sits between DevOps tooling and the application teams that consume it. Its goal is deceptively simple, make it easy for developers to do the right thing, provision infrastructure, deploy services, manage environments, without requiring deep infrastructure expertise or opening a ticket to a platform team.

The gap between that goal and reality is often filled by a patchwork of CI/CD pipelines, manual approvals, Slack messages, and Terraform scripts that only two people understand. Developer experience suffers. Compliance and security controls exist only in documentation that nobody reads. Infrastructure sprawls across environments with no consistent enforcement.

Spacelift is a platform engineering tool designed specifically to close this gap, providing a collaborative, policy-driven, GitOps-native control plane for Terraform, OpenTofu, Pulumi, Ansible, and Kubernetes. It sits above your IaC tooling and gives platform teams the controls they need to enforce standards, while giving application teams the self-service access they need to move fast without breaking things.

This guide covers how to build a self-service infrastructure platform with Spacelift, from stack architecture and policy enforcement to drift detection, module registries, and the developer experience patterns that make platform engineering actually deliver on its promise.

What Makes Spacelift Different

Before diving into implementation, it helps to understand what Spacelift provides that raw CI/CD pipelines don't:

Capability	Raw CI/CD (GitHub Actions)	Spacelift
Terraform plan/apply	Manual pipeline config	Native, zero-config
Policy enforcement	Shell scripts or OPA sidecars	Built-in OPA policies (plan, access, trigger)
Drift detection	Custom cron jobs	Native, scheduled
Module registry	Terraform Registry or S3	Built-in, versioned
Stack dependencies	Manual pipeline ordering	Native dependency graph
Self-service provisioning	PR-based or ticket-based	Self-service catalog
Audit trail	CI logs	Dedicated, tamper-evident
Secret management	GitHub Secrets	Native Vault/cloud integration
Cost estimation	None	Infracost integration

The fundamental difference is intent, CI/CD pipelines are general-purpose automation. Spacelift is purpose-built infrastructure orchestration, with the safety, compliance, and developer experience features that platform engineering requires.

Core Concepts

Before building, understand Spacelift's key abstractions:

Stack, the fundamental unit in Spacelift. A stack wraps a single IaC configuration (a Terraform root module, a Pulumi program, an Ansible playbook) with its state backend, environment variables, policies, and deployment settings. One stack equals one deployable infrastructure unit.

Space, a hierarchical namespace for grouping and access-controlling stacks. Spaces mirror your organizational structure: root → platform → team-a, root → platform → team-b. Policies and access rules defined at a parent space are inherited by child spaces.

Policy, an OPA (Open Policy Agent) Rego rule that governs stack behavior. Spacelift ships four policy types: plan (approve or reject based on plan output), access (control who can do what), trigger (control when stacks run), and git push (control which commits trigger runs).

Module, a versioned, reusable Terraform module published to Spacelift's built-in registry. Modules are the mechanism through which platform teams publish approved, tested infrastructure patterns that application teams consume via self-service.

Run, a single execution of a stack's IaC code. Every run is tracked with full input context, plan output, policy decisions, approval state, and apply output, providing an immutable audit trail.

Step 1 - Repository and Stack Architecture

Design your repository structure before creating stacks. A common pattern for multi-team platform engineering:

/infrastructure
  /platform           ← platform team owns this
    /networking
      main.tf         ← VPCs, subnets, TGW
    /security
      main.tf         ← IAM roles, security groups, KMS keys
    /eks
      main.tf         ← shared EKS clusters
  /modules            ← approved module library
    /vpc
    /rds
    /s3-secure
    /eks-nodegroup
  /teams              ← application teams own this (self-service)
    /team-alpha
      /staging
        main.tf
      /production
        main.tf
    /team-beta
      /staging
        main.tf
      /production
        main.tf
  /policies           ← Spacelift policies as code
    plan-enforce-tagging.rego
    plan-cost-guard.rego
    access-team-boundaries.rego
    trigger-dependency-chain.rego

This structure enforces a clear ownership boundary, the platform team manages foundational infrastructure and approved modules. Application teams self-service within /teams/{team-name}, constrained by policies that prevent them from touching what they shouldn't.

Step 2 - Creating and Configuring Stacks

Create stacks programmatically using the Spacelift Terraform provider, treating your platform configuration as code:

# spacelift/stacks/platform-networking.tf
resource "spacelift_stack" "platform_networking" {
  name        = "platform-networking"
  description = "Core VPC and network infrastructure"
  space_id    = spacelift_space.platform.id

  repository   = "your-org/infrastructure"
  branch       = "main"
  project_root = "platform/networking"

  terraform_version = "1.6.0"

  autodeploy         = false    # require explicit approval for platform changes
  protect_from_deletion = true

  labels = ["platform", "networking", "critical"]
}

resource "spacelift_stack" "team_alpha_staging" {
  name        = "team-alpha-staging"
  description = "Team Alpha staging environment"
  space_id    = spacelift_space.team_alpha.id

  repository   = "your-org/infrastructure"
  branch       = "main"
  project_root = "teams/team-alpha/staging"

  terraform_version = "1.6.0"

  autodeploy = true    # staging auto-deploys on merge
  labels     = ["team-alpha", "staging"]
}

Configure environment variables and secret injection per stack:

# Environment variables, non-sensitive configuration
resource "spacelift_environment_variable" "team_alpha_region" {
  stack_id  = spacelift_stack.team_alpha_staging.id
  name      = "AWS_DEFAULT_REGION"
  value     = "us-east-1"
  write_only = false
}

# Secret injection from AWS Secrets Manager via Spacelift context
resource "spacelift_context" "aws_credentials" {
  name        = "aws-production-credentials"
  description = "AWS credentials for production deployments"
  space_id    = spacelift_space.platform.id
}

resource "spacelift_context_attachment" "team_alpha_prod" {
  context_id = spacelift_context.aws_credentials.id
  stack_id   = spacelift_stack.team_alpha_staging.id
  priority   = 1
}

Contexts are the Spacelift equivalent of shared environment variable sets, define credentials once at the platform level, attach to any number of stacks, and rotate in one place.

Step 3 - Policies as Code with OPA

Policies are where platform engineering leverage lives. A policy written once enforces a standard across every stack in your platform, no per-team configuration, no tribal knowledge, no exceptions that slip through code review.

Plan Policy - Enforce Resource Tagging

Reject any Terraform plan that creates resources without mandatory tags:

# policies/plan-enforce-tagging.rego
package spacelift

# Required tags for all resources in production
required_tags := {"Environment", "Team", "CostCenter", "ManagedBy"}

# Deny if any resource is missing required tags
deny[sprintf("Resource '%s' is missing required tags: %v", [resource.address, missing])] {
  resource := input.terraform.resource_changes[_]
  resource.change.actions[_] == "create"

  # Identify which required tags are missing
  provided_tags := {tag | resource.change.after.tags[tag]}
  missing := required_tags - provided_tags
  count(missing) > 0
}

# Warn if tags exist but Environment doesn't match a known value
warn[sprintf("Resource '%s' has unrecognized Environment tag: '%s'", [resource.address, env])] {
  resource := input.terraform.resource_changes[_]
  env := resource.change.after.tags.Environment
  not env in {"staging", "production", "development"}
}

Plan Policy - Cost Guard

Reject plans that would exceed a per-run cost threshold:

# policies/plan-cost-guard.rego
package spacelift

# Block plans with monthly cost increase above $500
deny["Monthly cost increase exceeds $500 threshold — escalate to platform team for approval"] {
  input.spacelift.run.cost_estimate.monthly_cost_increase > 500
}

warn["Monthly cost increase above $100 — ensure this is expected"] {
  input.spacelift.run.cost_estimate.monthly_cost_increase > 100
  input.spacelift.run.cost_estimate.monthly_cost_increase <= 500
}

Access Policy - Team Boundary Enforcement

Prevent teams from reading or modifying stacks they don't own:

# policies/access-team-boundaries.rego
package spacelift

# Extract team from the user's IdP groups
user_teams := {team |
  group := input.session.teams[_]
  team := trim_prefix(group, "team-")
}

# Allow access only to stacks labeled with the user's team
allow {
  stack_team := input.stack.labels[_]
  stack_team in user_teams
}

# Platform team members have access to all stacks
allow {
  "platform" in input.session.teams
}

# Block all other access
deny {
  not allow
}

Trigger Policy - Stack Dependency Chains

Automatically trigger downstream stacks when upstream infrastructure changes:

# policies/trigger-dependency-chain.rego
package spacelift

# When platform-networking applies successfully, trigger EKS stack
trigger["platform-eks"] {
  input.run.stack.id == "platform-networking"
  input.run.state == "FINISHED"
  input.run.type == "TRACKED"
}

# When platform-eks applies successfully, trigger team stacks
trigger["team-alpha-staging"] {
  input.run.stack.id == "platform-eks"
  input.run.state == "FINISHED"
}

Attach policies to stacks or spaces:

resource "spacelift_policy" "enforce_tagging" {
  name = "enforce-resource-tagging"
  type = "PLAN"
  body = file("policies/plan-enforce-tagging.rego")
}

# Attach to the entire platform space — all child stacks inherit
resource "spacelift_policy_attachment" "enforce_tagging_platform" {
  policy_id = spacelift_policy.enforce_tagging.id
  stack_id  = "*"
  space_id  = spacelift_space.platform.id
}

Step 4 - Module Registry for Self-Service

The Spacelift module registry is the mechanism through which platform teams publish approved infrastructure patterns for self-service consumption. A module published to the registry is versioned, tested, and policy-compliant by definition, application teams consume it without needing to understand the underlying infrastructure design.

Publish a module to Spacelift's registry:

# spacelift/modules/rds-module.tf
resource "spacelift_module" "rds_postgres" {
  name           = "rds-postgres"
  space_id       = spacelift_space.platform.id
  repository     = "your-org/infrastructure"
  project_root   = "modules/rds"
  branch         = "main"
  description    = "Production-grade PostgreSQL RDS with encryption, backups, and parameter group"

  terraform_provider = "aws"

  labels = ["database", "postgres", "approved"]
}

Application teams consume it in their stacks like any Terraform module:

# teams/team-alpha/staging/main.tf
module "orders_database" {
  source  = "spacelift.io/your-org/rds-postgres/aws"
  version = "~> 2.0"

  identifier        = "orders-staging"
  instance_class    = "db.t3.medium"
  allocated_storage = 20
  environment       = "staging"
  team              = "team-alpha"
}

The version constraint (~> 2.0) means Team Alpha gets patch updates automatically but won't be silently upgraded to a breaking major version. The platform team controls what's published, application teams control what version they pin to.

Step 5 - Drift Detection

Infrastructure drift, the gap between what Terraform's state says exists and what actually exists in your cloud account, is one of the most persistent operational problems in IaC. Manual changes through the AWS console, emergency hotfixes applied directly, or race conditions between concurrent applies all cause drift.

Spacelift's built-in drift detection runs Terraform plan on a schedule and alerts when the live state diverges from the declared state:

resource "spacelift_stack" "platform_networking" {
  # ... other config ...

  # Run drift detection daily at 2am UTC
  drift_detection {
    enabled   = true
    reconcile = false    # alert on drift but don't auto-remediate — require human review
    schedule  = ["0 2 * * *"]
  }
}

For non-critical stacks where auto-remediation is safe:

drift_detection {
  enabled   = true
  reconcile = true     # automatically apply to reconcile drift
  schedule  = ["0 */6 * * *"]  # every 6 hours
}

Configure Slack or PagerDuty notifications for drift alerts:

resource "spacelift_webhook" "drift_alert" {
  stack_id = spacelift_stack.platform_networking.id
  endpoint = var.slack_webhook_url
  enabled  = true

  # Trigger on drift detection and failed reconciliation
  secret = var.webhook_secret
}

Step 6 - Self-Service Developer Experience

The final layer is the developer experience that makes self-service actually self-service. Spacelift's self-service catalog (Spaces + module registry + approval workflows) lets application teams provision infrastructure through a UI or API without opening tickets:

For application teams: a curated list of available modules (RDS, S3, SQS, ElastiCache) with a form-based provisioning interface, no Terraform knowledge required.

For platform teams: full visibility into what every team has provisioned, policy enforcement that fires automatically, drift detection that runs continuously, and an audit trail that captures every change with the identity of who made it and what the plan output was.

Connect Spacelift to your IdP (Okta, GitHub, Google) to enforce team boundaries automatically via SSO group membership, the access policies defined in Step 3 apply automatically as team membership changes.

Step 7 - CI/CD Integration

Spacelift integrates with GitHub, GitLab, Bitbucket, and Azure DevOps for Git-driven runs. A push to main in the infrastructure repo triggers plan runs on affected stacks automatically:

# .github/workflows/spacelift-preview.yml
name: Spacelift Plan Preview

on: [pull_request]

jobs:
  plan-preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Comment plan output on PR
        uses: spacelift-io/spacelift-pull-request-action@v1
        with:
          spacelift-api-key-id: ${{ secrets.SPACELIFT_API_KEY_ID }}
          spacelift-api-key-secret: ${{ secrets.SPACELIFT_API_KEY_SECRET }}

For stacks with autodeploy: false, Spacelift posts the plan output as a PR comment, requiring a human to review and approve before the apply runs. For stacks with autodeploy: true (staging environments), the apply runs automatically on merge to main.

Common Pitfalls to Avoid

Flat space hierarchy. A single space for all stacks gives no access boundary between teams. Design your space hierarchy to mirror your team structure from day one, retrofitting access controls onto a flat hierarchy is painful.

Policies without testing. A deny policy that fires incorrectly can block all deployments across your entire platform. Test policies against representative plan inputs using OPA's opa eval before attaching them to production stacks.

Auto-reconciling critical infrastructure. Drift auto-remediation is powerful but dangerous for foundational infrastructure. Networking, security groups, and IAM roles should alert on drift and require human review, not automatically apply. Reserve auto-reconciliation for stateless, idempotent resource sets.

No module versioning discipline. Publishing a breaking change to a module and expecting all consumers to upgrade simultaneously doesn't scale. Adopt semantic versioning for modules and maintain at least one prior major version while teams migrate.

Conclusion

Platform engineering is not about giving developers access to infrastructure tooling, it's about designing guardrails and self-service patterns so that developers can provision what they need safely, consistently, and without creating operational debt for the platform team.

Spacelift makes this achievable by providing a purpose-built control plane for IaC, policy enforcement that fires at plan time, a module registry that codifies approved patterns, drift detection that catches manual changes, and a space hierarchy that enforces team boundaries without manual intervention.

The platform team's job evolves from running infrastructure to building the platform that lets others run theirs safely, at scale, and with a complete audit trail.

Running OpenTofu instead of Terraform, or mixing Pulumi with Ansible? Spacelift's multi-IaC support handles heterogeneous toolchains in the same platform all under the same policy and audit model. Drop your stack in the comments.

PlatformEngineering #Spacelift #Terraform #InfrastructureAsCode #DevOps

Implementing mTLS in Kubernetes: How to Secure Service-to-Service Communication

ThankGod Chibugwum Obobo — Sun, 19 Jul 2026 17:00:00 +0000

Introduction

In a microservices architecture running on Kubernetes, services communicate constantly, the orders-service calls the users-service, the payments-service calls the fraud-detection-service, and the API gateway routes to a dozen downstream services. By default, most of this traffic travels unencrypted inside the cluster, authenticated only by the fact that the caller has network access.

That assumption that anything inside the cluster boundary is trustworthy, is the network-level equivalent of leaving every internal door unlocked because the building has a lock on the front entrance. A single compromised pod, a misconfigured network policy, or a lateral movement attack can reach any service in the cluster with no additional authentication required.

Mutual TLS (mTLS) closes this gap. Unlike standard TLS where only the server presents a certificate, mTLS requires both parties, client and server, to present and verify certificates before any data is exchanged. Every service-to-service connection is cryptographically authenticated. A compromised pod cannot impersonate payments-service to fraud-detection-service because it cannot produce a valid certificate for that identity.

This guide covers how to implement mTLS in Kubernetes using a service mesh, comparing Istio and Linkerd, configuring automatic certificate issuance, enforcing strict mTLS policies, and validating that your network is actually secured.

How mTLS Works in Kubernetes

Standard TLS in a web context works like this: the client (browser) verifies the server's certificate, establishing that the server is who it claims to be, but the server accepts any client. mTLS adds the reverse, the server also verifies the client's certificate, establishing mutual identity before the connection proceeds.

In Kubernetes, the mechanics are:

Each service is issued a certificate tied to its SPIFFE identity, a URI in the format spiffe://cluster.local/ns/{namespace}/sa/{service-account}.
When orders-service connects to users-service, both services present their certificates.
The receiving service verifies the caller's certificate against the cluster's trust anchor (a root CA).
If verification passes, the connection proceeds. If not, wrong identity, expired certificate, or untrusted issuer, the connection is rejected at the TLS layer, before any application code runs.

The critical operational property: certificates are automatically rotated by the service mesh control plane, typically every 24 hours. No manual certificate management, no expiry incidents, no long-lived credentials.

Choosing a Service Mesh: Istio vs. Linkerd

Both Istio and Linkerd implement mTLS via sidecar proxies injected into each pod. The proxies intercept all inbound and outbound traffic, transparently handling TLS without requiring application code changes.

Feature	Istio	Linkerd
mTLS implementation	Envoy sidecar	Linkerd2-proxy sidecar
Certificate authority	Istiod (built-in) or external	Built-in or cert-manager
mTLS enforcement	PeerAuthentication policy	Server policy
Observability	Rich (Kiali, Jaeger integration)	Lightweight (built-in dashboard)
Resource overhead	Higher (~200–350MB per sidecar)	Lower (~10–20MB per sidecar)
Configuration complexity	Higher	Lower
FIPS 140-2 compliance	Enterprise	Enterprise
Multi-cluster support	Native	Native

Choose Istio if you need advanced traffic management (canary deployments, fault injection, circuit breaking), rich observability, or WebAssembly-based policy extensibility.

Choose Linkerd if you prioritize simplicity, low resource overhead, and want mTLS with strong defaults out of the box with minimal configuration.

This guide covers both installation and policy configuration for each.

Approach 1 - mTLS with Istio

Step 1 - Install Istio

Install Istio using the istioctl CLI with the minimal production profile:

# Install istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-*/bin:$PATH

# Install Istio with production profile
istioctl install --set profile=default \
  --set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true \
  -y

# Verify installation
istioctl verify-install

Step 2 - Enable Automatic Sidecar Injection

Label namespaces where you want mTLS enforced. Istio automatically injects the Envoy sidecar proxy into every pod in labeled namespaces:

# Enable sidecar injection for your application namespaces
kubectl label namespace orders-service istio-injection=enabled
kubectl label namespace users-service istio-injection=enabled
kubectl label namespace payments-service istio-injection=enabled

# Verify injection is active
kubectl get namespace -L istio-injection

Rolling restart existing deployments to pick up sidecar injection:

kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service

Verify sidecars are injected, each pod should now show 2/2 containers (application + Envoy proxy):

kubectl get pods -n orders-service
# NAME                             READY   STATUS    RESTARTS   AGE
# orders-api-7d8b9f5c4-xk2pq      2/2     Running   0          45s

Step 3 - Configure PeerAuthentication for Strict mTLS

By default, Istio operates in permissive mode, it accepts both plaintext and mTLS traffic. This is safe for migration but provides no security guarantee. Switch to strict mode to enforce mTLS and reject all plaintext traffic:

# istio/peer-authentication-strict.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system   # mesh-wide policy
spec:
  mtls:
    mode: STRICT

kubectl apply -f istio/peer-authentication-strict.yaml

This single resource enforces mTLS for all service-to-service communication across the entire mesh. Any pod that attempts to connect to another pod without a valid certificate, including debug containers, legacy services without sidecars, or external probes, will have their connection rejected.

Step 4 - AuthorizationPolicy for Fine-Grained Access Control

mTLS establishes who a caller is. AuthorizationPolicy controls what authenticated identities are allowed to do, combining mTLS identity with RBAC-style rules:

# istio/authorization-payments.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payments-service-policy
  namespace: payments-service
spec:
  selector:
    matchLabels:
      app: payments-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              # Only orders-service can call payments-service
              - "cluster.local/ns/orders-service/sa/orders-service-sa"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/payments/charge", "/payments/refund"]

# Deny all traffic not explicitly allowed, defense in depth
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: payments-service
spec:
  {}  # Empty spec = deny all by default

Apply deny-all first, then layer specific ALLOW policies on top. This is the zero-trust principle applied at the service level: every inter-service call must be explicitly authorized.

Step 5 - Verify mTLS is Active with Kiali

Kiali, Istio's observability console, provides a visual confirmation of mTLS status:

# Install Kiali and supporting tools
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml

# Access Kiali dashboard
istioctl dashboard kiali

In Kiali's service graph, a padlock icon on each edge confirms mTLS is active on that connection. Missing padlocks indicate a service communicating without mTLS, a misconfiguration or missing sidecar.

Approach 2 - mTLS with Linkerd

Step 1 - Install Linkerd

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Validate cluster compatibility
linkerd check --pre

# Install Linkerd control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify installation
linkerd check

Step 2 - Inject Linkerd Proxy into Namespaces

# Annotate namespaces for automatic injection
kubectl annotate namespace orders-service linkerd.io/inject=enabled
kubectl annotate namespace users-service linkerd.io/inject=enabled
kubectl annotate namespace payments-service linkerd.io/inject=enabled

# Restart deployments to pick up proxy injection
kubectl rollout restart deployment -n orders-service
kubectl rollout restart deployment -n users-service
kubectl rollout restart deployment -n payments-service

Step 3 - Enforce mTLS with Server Policies

Linkerd enables mTLS by default in injected namespaces. To enforce authenticated traffic only (equivalent to Istio's strict mode), configure Server policies:

# linkerd/server-policy-payments.yaml
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: payments-server
  namespace: payments-service
spec:
  podSelector:
    matchLabels:
      app: payments-service
  port: 8080
  proxyProtocol: HTTP/2


apiVersion: policy.linkerd.io/v1beta2
kind: ServerAuthorization
metadata:
  name: payments-authz
  namespace: payments-service
spec:
  server:
    name: payments-server
  client:
    meshTLS:
      serviceAccounts:
        - name: orders-service-sa
          namespace: orders-service

kubectl apply -f linkerd/server-policy-payments.yaml

# Verify mTLS is active on connections
linkerd viz edges deployment -n orders-service
# Shows which connections are secured with mTLS (✓) vs plaintext (✗)

Certificate Management with cert-manager

Both Istio and Linkerd can integrate with cert-manager for certificate issuance, enabling integration with external CAs (HashiCorp Vault, AWS ACM PCA, Let's Encrypt) rather than relying solely on the mesh's built-in CA.

# cert-manager/cluster-issuer-vault.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.internal:8200
    path: pki/sign/kubernetes-mesh
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager
        secretRef:
          name: cert-manager-vault-token
          key: token

For Istio, plug the external CA into the control plane:

# Use cert-manager as Istio's certificate provider
istioctl install --set profile=default \
  --set values.pilot.env.EXTERNAL_CA=true \
  --set values.pilot.env.K8S_SIGNER=cert-manager.io/cluster-issuer/vault-issuer

External CA integration provides:

Centralized audit trail of every certificate issuance across your mesh
Cross-cluster trust using a shared root CA for multi-cluster mTLS
Compliance alignment with PKI policies already in use in your organization

Migrating to Strict mTLS Without Downtime

Flipping directly to strict mode on a production cluster breaks any service that doesn't yet have a sidecar. Use a phased migration:

Phase 1 - Permissive mode (baseline). Install the mesh with mTLS in permissive mode. All traffic is accepted, both plaintext and mTLS. Use Kiali or linkerd viz to identify which connections are NOT using mTLS.

Phase 2 - Namespace-by-namespace injection. Enable sidecar injection one namespace at a time. Verify each namespace's workloads after injection before proceeding.

Phase 3 - Strict mode per namespace. Apply PeerAuthentication or Server policies with strict enforcement to already-injected namespaces:

# Namespace-scoped strict policy (safer than mesh-wide during migration)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: payments-service   # scoped to one namespace
spec:
  mtls:
    mode: STRICT

Phase 4 - Mesh-wide strict enforcement. Once all namespaces are injected and verified, apply the mesh-wide PeerAuthentication in istio-system and remove namespace-scoped policies.

This phased approach takes longer but eliminates the risk of a mesh-wide strict enforcement breaking services that weren't ready.

Validating mTLS Enforcement

After enabling strict mode, validate it is actually enforced, don't assume:

# Attempt to connect to payments-service from a pod WITHOUT a sidecar
kubectl run debug-pod --image=curlimages/curl --restart=Never -- \
  curl -s http://payments-service.payments-service.svc.cluster.local:8080/health

# Expected result: connection refused or reset, mTLS is enforced
# Unexpected result: 200 OK, mTLS is NOT enforced, investigate

# Verify certificate details on an active connection
kubectl exec -n orders-service deployment/orders-api -c istio-proxy -- \
  openssl s_client \
    -connect payments-service.payments-service.svc.cluster.local:8080 \
    -showcerts 2>/dev/null | grep -A2 "subject="

# Expected output shows SPIFFE URI:
# subject=URI:spiffe://cluster.local/ns/payments-service/sa/payments-service-sa

# Check Istio's view of mTLS status per service
istioctl x describe service payments-service.payments-service

# Output confirms:
# mTLS: yes
# Policy: STRICT

Observability and Troubleshooting

mTLS adds authentication at the network layer, but it also adds debugging complexity. Keep these tools in your operational toolkit:

Istio proxy logs reveal TLS handshake failures with specific error codes:

# View Envoy access logs for a specific pod
kubectl logs -n orders-service deployment/orders-api -c istio-proxy | \
  grep -E "UF|UC|UH"  # Upstream connection failure codes

Linkerd tap provides real-time traffic inspection with mTLS status:

linkerd viz tap deployment/orders-api -n orders-service \
  --to deployment/payments-service -n payments-service

Certificate expiry monitoring even with automatic rotation, monitor certificate validity as a health signal:

# Prometheus alert for certificates expiring within 24 hours
- alert: MeshCertificateExpiringSoon
  expr: |
    (citadel_server_csr_count - citadel_server_success_cert_issuance_count) > 0
  for: 5m
  labels:
    severity: high
  annotations:
    summary: "Certificate issuance failures in Istio CA"

Common Pitfalls to Avoid

Enabling strict mode before all services have sidecars. Services without sidecars cannot participate in mTLS and will have their connections rejected in strict mode. Always verify 100% sidecar coverage before enabling mesh-wide strict enforcement.

Forgetting health check probes. Kubernetes liveness and readiness probes originate from the kubelet, outside the mesh. Configure Istio to rewrite probe URLs or exclude probe ports from mTLS enforcement:

# Exclude health check port from mTLS, kubelet doesn't have a sidecar
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    8081:          # health check port
      mode: DISABLE

Not testing AuthorizationPolicy before applying. A misconfigured deny-all policy can cut off legitimate service traffic in production. Test policies in a staging environment first, and use Istio's AUDIT action mode to log would-be denials before enforcing them.

Ignoring certificate rotation failures. The automatic certificate rotation is the operational backbone of mesh mTLS. Monitor certificate issuance metrics, a rotation failure that goes undetected leads to widespread connection failures when certificates expire.

Conclusion

Mutual TLS in Kubernetes transforms your cluster's internal network from an implicit trust zone into a zero-trust environment where every connection is cryptographically authenticated and explicitly authorized. A compromised pod cannot impersonate a legitimate service. A lateral movement attack cannot freely reach payment processing or user data services. Every inter-service call is auditable.

The operational cost, a service mesh, sidecar overhead, policy management, is real. But for systems where data sensitivity, regulatory requirements, or blast-radius concerns justify it, mTLS via a service mesh is the most complete implementation of zero-trust networking currently available for Kubernetes workloads.

Start with permissive mode, inject sidecars namespace by namespace, validate mTLS coverage with your mesh's observability tools, and graduate to strict enforcement once you're confident in coverage. The security guarantee at the end is worth the careful migration.

Running a multi-cluster Kubernetes setup? Extending mTLS across cluster boundaries with Istio's multi-cluster federation or Linkerd's cluster linking uses the same SPIFFE identity model, but adds a trust bundle exchange step between clusters.

DRY Infrastructure: How to Architect Modular Ansible Playbooks for Next.js

ThankGod Chibugwum Obobo — Sun, 12 Jul 2026 17:00:00 +0000

Infrastructure code suffers from the same antipatterns as application code, duplication, tight coupling, undocumented assumptions, and a growing resistance to change. A playbook written for one environment gets copy-pasted for the next, variables get hardcoded, and six months later you have five slightly different versions of the same deployment logic, each drifting independently and none of them trustworthy.

The DRY (Don't Repeat Yourself) principle applies to infrastructure as directly as it does to software. In Ansible, DRY infrastructure means roles for reusable task logic, variable hierarchies for environment-specific overrides, handlers for event-driven operations, and a playbook composition pattern that assembles production-grade deployments from interchangeable, independently testable units.

For Next.js, with its hybrid rendering modes, standalone server output, static asset optimization, and Node.js runtime requirements, a modular Ansible approach gives you a repeatable, environment-aware provisioning and deployment pipeline that works consistently from local staging to production at scale.

This guide covers how to design and implement a modular Ansible architecture for Next.js from the ground up.

Why Ansible for Next.js Infrastructure

Before diving into structure, it's worth addressing why Ansible fits Next.js provisioning well, especially alongside more modern IaC tools:

Ansible excels at configuration management and application deployment, the layer between raw infrastructure (where Terraform lives) and application orchestration (where Kubernetes lives). For teams running Next.js on VMs, bare metal, or alongside a lightweight container setup, Ansible bridges that gap cleanly.

Specifically for Next.js:

Node.js runtime management (version pinning, nvm, process managers)
Build artifact deployment (transferring .next/standalone output, managing static assets to CDN)
Nginx reverse proxy configuration (SSL termination, caching headers, compression)
Environment variable injection (.env files templated from Vault or group vars)
PM2 or systemd process management (zero-downtime restarts, health checks)
Log rotation and monitoring agent configuration

This is precisely where Ansible's agentless, SSH-based, idempotent task execution shines.

Step 1 - Project Structure for Modular Playbooks

Start with a structure that enforces separation of concerns from day one:

/ansible
  /inventories
    /staging
      hosts.yml
      group_vars/
        all.yml
        nextjs_servers.yml
    /production
      hosts.yml
      group_vars/
        all.yml
        nextjs_servers.yml
  /roles
    /common           ← base system configuration
    /nodejs           ← Node.js runtime installation and management
    /nextjs-app       ← Next.js application deployment
    /nginx            ← Nginx configuration and SSL
    /pm2              ← PM2 process management
    /monitoring       ← monitoring agent setup
    /logrotate        ← log rotation configuration
  /playbooks
    site.yml          ← master playbook (composes all roles)
    provision.yml     ← first-time server provisioning
    deploy.yml        ← application deployment only
    rollback.yml      ← rollback to previous release
  /group_vars
    all.yml           ← variables shared across all environments
  ansible.cfg

This structure reflects three key design decisions:

Separate provisioning from deployment. provision.yml runs once (or rarely) to set up a server. deploy.yml runs on every release. Conflating them means every deployment re-runs expensive provisioning tasks unnecessarily.

Separate inventories per environment. Staging and production are entirely isolated inventory trees, different hosts, different group vars, no risk of accidentally running a production playbook against staging or vice versa.

Roles are the unit of reuse. Every discrete concern is a role. Roles are independently versioned, testable with Molecule, and composable in any playbook.

Step 2 - Ansible Configuration

# ansible.cfg
[defaults]
inventory          = ./inventories
roles_path         = ./roles
host_key_checking  = False
retry_files_enabled = False
stdout_callback    = yaml
interpreter_python = auto_silent
forks              = 10

[ssh_connection]
ssh_args           = -o ControlMaster=auto -o ControlPersist=60s
pipelining         = True   # significant performance improvement

pipelining = True reduces the number of SSH connections per task, one of the most impactful performance improvements for large fleets. It requires requiretty to be disabled in sudoers on your target hosts.

Step 3 - Variable Hierarchy and Environment Overlays

Ansible's variable precedence is the mechanism for DRY environment-specific configuration. Define defaults at the broadest scope, override only what differs per environment:

# group_vars/all.yml - shared across ALL environments
nodejs_version: "20.11.0"
app_user: "nextjs"
app_group: "nextjs"
app_base_dir: "/opt/nextjs"
releases_to_keep: 5
pm2_instances: "max"
nginx_worker_processes: "auto"

# inventories/staging/group_vars/nextjs_servers.yml - staging overrides
app_environment: "staging"
app_port: 3000
next_public_api_url: "https://api.staging.your-domain.com"
pm2_instances: 2                  # fewer instances on smaller staging servers
nginx_ssl_certificate: "/etc/letsencrypt/live/staging.your-domain.com/fullchain.pem"
nginx_ssl_key: "/etc/letsencrypt/live/staging.your-domain.com/privkey.pem"
app_releases_dir: "{{ app_base_dir }}/releases"

# inventories/production/group_vars/nextjs_servers.yml - production overrides
app_environment: "production"
app_port: 3000
next_public_api_url: "https://api.your-domain.com"
pm2_instances: "max"              # use all available CPU cores
nginx_ssl_certificate: "/etc/letsencrypt/live/your-domain.com/fullchain.pem"
nginx_ssl_key: "/etc/letsencrypt/live/your-domain.com/privkey.pem"
app_releases_dir: "{{ app_base_dir }}/releases"

Sensitive values (database URLs, API keys, secrets) should never appear in group vars. Use Ansible Vault for sensitive variable encryption or integrate with HashiCorp Vault for dynamic secret injection, as covered in our guide on replacing hardcoded credentials with Vault and GitHub OIDC.

Step 4 - The Node.js Role

The Node.js role manages runtime installation via nvm, ensuring version pinning and easy upgrades without system package manager conflicts:

roles/nodejs/
  tasks/
    main.yml
  defaults/
    main.yml
  handlers/
    main.yml

# roles/nodejs/defaults/main.yml
nodejs_version: "20.11.0"
nvm_version: "0.39.7"
nvm_dir: "/home/{{ app_user }}/.nvm"
npm_global_packages:
  - pm2
  - typescript

# roles/nodejs/tasks/main.yml
---
- name: Create application user
  ansible.builtin.user:
    name: "{{ app_user }}"
    group: "{{ app_group }}"
    system: true
    shell: /bin/bash
    home: "/home/{{ app_user }}"
    create_home: true
  tags: [nodejs, user]

- name: Check if nvm is installed
  ansible.builtin.stat:
    path: "{{ nvm_dir }}/nvm.sh"
  register: nvm_stat
  tags: [nodejs, nvm]

- name: Install nvm
  ansible.builtin.shell:
    cmd: >
      curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v{{ nvm_version }}/install.sh | bash
  become: true
  become_user: "{{ app_user }}"
  when: not nvm_stat.stat.exists
  tags: [nodejs, nvm]

- name: Install Node.js via nvm
  ansible.builtin.shell:
    cmd: >
      source {{ nvm_dir }}/nvm.sh &&
      nvm install {{ nodejs_version }} &&
      nvm alias default {{ nodejs_version }}
  args:
    executable: /bin/bash
  become: true
  become_user: "{{ app_user }}"
  tags: [nodejs]

- name: Install global npm packages
  ansible.builtin.shell:
    cmd: >
      source {{ nvm_dir }}/nvm.sh &&
      npm install -g {{ item }}
  args:
    executable: /bin/bash
  become: true
  become_user: "{{ app_user }}"
  loop: "{{ npm_global_packages }}"
  tags: [nodejs, npm]

Step 5 - The Next.js Application Role

The Next.js role implements a release directory pattern - the key mechanism for zero-downtime deployments and fast rollbacks:

roles/nextjs-app/
  tasks/
    main.yml
    deploy.yml
    rollback.yml
    cleanup.yml
  templates/
    env.j2            ← .env.local template
    pm2.config.js.j2  ← PM2 ecosystem config template
  handlers/
    main.yml
  defaults/
    main.yml

# roles/nextjs-app/tasks/deploy.yml
- name: Generate release timestamp
  ansible.builtin.set_fact:
    release_timestamp: "{{ ansible_date_time.epoch }}"
  tags: [deploy]

- name: Create release directory
  ansible.builtin.file:
    path: "{{ app_releases_dir }}/{{ release_timestamp }}"
    state: directory
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
    mode: "0755"
  tags: [deploy]

- name: Upload Next.js standalone build
  ansible.builtin.synchronize:
    src: "{{ local_build_path }}/.next/standalone/"
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}"
    delete: true
    rsync_opts:
      - "--exclude=.git"
  tags: [deploy]

- name: Upload static assets
  ansible.builtin.synchronize:
    src: "{{ local_build_path }}/.next/static/"
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}/.next/static/"
  tags: [deploy]

- name: Template environment file
  ansible.builtin.template:
    src: env.j2
    dest: "{{ app_releases_dir }}/{{ release_timestamp }}/.env.local"
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
    mode: "0640"           # restrict read to app user only
  tags: [deploy]

- name: Update current symlink atomically
  ansible.builtin.file:
    src: "{{ app_releases_dir }}/{{ release_timestamp }}"
    dest: "{{ app_base_dir }}/current"
    state: link
    owner: "{{ app_user }}"
    group: "{{ app_group }}"
  notify: Reload PM2
  tags: [deploy]

The atomic symlink swap (current → new release) is the mechanism that makes zero-downtime deployments possible. PM2 reloads against the new current directory; if the reload fails, the symlink points back to the previous release.

# roles/nextjs-app/tasks/rollback.yml
- name: List available releases
  ansible.builtin.find:
    paths: "{{ app_releases_dir }}"
    file_type: directory
  register: available_releases
  tags: [rollback]

- name: Identify previous release
  ansible.builtin.set_fact:
    previous_release: >-
      {{
        available_releases.files
        | sort(attribute='mtime', reverse=True)
        | map(attribute='path')
        | list
        | nth(1)
      }}
  tags: [rollback]

- name: Rollback symlink to previous release
  ansible.builtin.file:
    src: "{{ previous_release }}"
    dest: "{{ app_base_dir }}/current"
    state: link
  notify: Reload PM2
  tags: [rollback]

- name: Report rollback target
  ansible.builtin.debug:
    msg: "Rolled back to: {{ previous_release }}"
  tags: [rollback]

Step 6 - The Nginx Role

# roles/nginx/templates/nextjs.conf.j2
upstream nextjs_upstream {
  server 127.0.0.1:{{ app_port }};
  keepalive 64;
}

server {
  listen 80;
  server_name {{ nginx_server_name }};
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  server_name {{ nginx_server_name }};

  ssl_certificate     {{ nginx_ssl_certificate }};
  ssl_certificate_key {{ nginx_ssl_key }};
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
  ssl_prefer_server_ciphers off;

  # Next.js static assets, long-lived cache with content hashing
  location /_next/static/ {
    alias {{ app_base_dir }}/current/.next/static/;
    expires 1y;
    add_header Cache-Control "public, immutable";
  }

  # Public assets
  location /public/ {
    alias {{ app_base_dir }}/current/public/;
    expires 30d;
    add_header Cache-Control "public";
  }

  # Next.js server
  location / {
    proxy_pass         http://nextjs_upstream;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection 'upgrade';
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_cache_bypass $http_upgrade;
  }
}

# roles/nginx/tasks/main.yml
- name: Install Nginx
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true
  tags: [nginx]

- name: Template Next.js site configuration
  ansible.builtin.template:
    src: nextjs.conf.j2
    dest: /etc/nginx/sites-available/nextjs
    owner: root
    group: root
    mode: "0644"
  notify: Reload Nginx
  tags: [nginx]

- name: Enable site
  ansible.builtin.file:
    src: /etc/nginx/sites-available/nextjs
    dest: /etc/nginx/sites-enabled/nextjs
    state: link
  notify: Reload Nginx
  tags: [nginx]

# roles/nginx/handlers/main.yml
- name: Reload Nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded
  listen: Reload Nginx

Step 7 - Composing Playbooks

With roles defined, playbooks become thin composition layers:

# playbooks/provision.yml - run once to set up new servers
- name: Provision Next.js servers
  hosts: nextjs_servers
  become: true
  roles:
    - role: common
      tags: [common]
    - role: nodejs
      tags: [nodejs]
    - role: nginx
      tags: [nginx]
    - role: pm2
      tags: [pm2]
    - role: logrotate
      tags: [logrotate]
    - role: monitoring
      tags: [monitoring]

# playbooks/deploy.yml - run on every release
- name: Deploy Next.js application
  hosts: nextjs_servers
  become: true
  vars:
    local_build_path: "{{ lookup('env', 'NEXT_BUILD_PATH') }}"

  pre_tasks:
    - name: Validate build artifact exists
      ansible.builtin.stat:
        path: "{{ local_build_path }}/.next/standalone"
      delegate_to: localhost
      register: build_stat
      failed_when: not build_stat.stat.exists

  roles:
    - role: nextjs-app
      tags: [deploy]

  post_tasks:
    - name: Verify application is healthy
      ansible.builtin.uri:
        url: "https://{{ nginx_server_name }}/api/health"
        status_code: 200
        timeout: 30
      retries: 5
      delay: 10
      register: health_check
      until: health_check.status == 200

# playbooks/rollback.yml
- name: Rollback Next.js to previous release
  hosts: nextjs_servers
  become: true
  tasks:
    - name: Execute rollback
      ansible.builtin.include_role:
        name: nextjs-app
        tasks_from: rollback

Step 8 - CI/CD Integration

# .github/workflows/deploy-nextjs.yml
name: Deploy Next.js

on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install and build
        run: |
          npm ci
          npm run build

      - name: Install Ansible
        run: pip install ansible ansible-lint --break-system-packages

      - name: Configure SSH
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
          chmod 600 ~/.ssh/deploy_key
          echo "StrictHostKeyChecking no" >> ~/.ssh/config

      - name: Deploy to production
        run: |
          ansible-playbook playbooks/deploy.yml \
            -i inventories/production/hosts.yml \
            --private-key ~/.ssh/deploy_key \
            -e "NEXT_BUILD_PATH=${{ github.workspace }}"
        env:
          ANSIBLE_HOST_KEY_CHECKING: "False"

Common Pitfalls to Avoid

Using shell tasks where native modules exist. ansible.builtin.shell bypasses idempotency, it runs every time regardless of whether the state has changed. Always prefer native modules (apt, copy, template, service) and fall back to shell only when no native module covers the task.

Hardcoding environment-specific values in roles. Role defaults should be genuinely sensible defaults, not production values masquerading as defaults. Production specifics belong in inventory group_vars.

Not cleaning up old releases. The release directory pattern accumulates historical deployments. Add a cleanup task to retain only {{ releases_to_keep }} releases, the rest is disk usage with no operational value.

Skipping the health check post-task. A deploy playbook that doesn't verify the application is healthy after deployment is incomplete. The health check is the difference between a successful deploy and an undetected outage.

Conclusion

Modular Ansible playbooks are not just a style preference, they are an operational necessity at any scale beyond a single server. Roles as units of reuse, variable hierarchies for environment overlays, the release directory pattern for zero-downtime deployments, and thin playbooks for composition give you an infrastructure codebase that is maintainable, testable, and genuinely DRY.

For Next.js specifically, this architecture handles the full lifecycle, provisioning the Node.js runtime, deploying standalone build artifacts, configuring Nginx for optimal caching of static assets, managing PM2 for process reliability, and providing a one-command rollback when things go wrong.

Infrastructure code that can't be understood, changed, and trusted by the team that operates it is a liability. Infrastructure code built to the same quality standards as your application code is a competitive advantage.

Beyond the Basics: How to Build a Multi-Region EKS Cluster with Terraform

ThankGod Chibugwum Obobo — Sun, 05 Jul 2026 17:00:00 +0000

A single-region Kubernetes cluster is a liability in disguise. When AWS experiences a regional outage, and every major cloud provider does periodically, your entire application goes with it. For systems where availability is a business-critical requirement, a single region is not a deployment strategy, it's a single point of failure at planetary scale.

Multi-region EKS distributes your workloads across two or more AWS regions, giving you geographic redundancy, reduced latency for globally distributed users, and the ability to survive a regional failure without a production incident. But the operational complexity is real, managing two clusters, synchronizing deployments, routing traffic intelligently, and designing for state consistency across regions requires deliberate architecture.

Terraform makes this complexity manageable. With its provider aliasing, module composition, and remote state capabilities, you can define your entire multi-region infrastructure as version-controlled, reviewable, reproducible code.

This guide covers how to architect and implement a multi-region EKS cluster with Terraform, from VPC design and cluster provisioning to cross-region traffic routing, state management, and disaster recovery patterns.

Architectural Overview

Before writing configuration, establish the target architecture. A production multi-region EKS setup typically follows one of two patterns:

Active-Active: Both regions serve production traffic simultaneously. Traffic is load-balanced across regions via a global load balancer (AWS Global Accelerator or Route 53 latency routing). Each region is a full deployment of your application stack. This maximizes availability and minimizes latency but requires careful state synchronization between regions.

Active-Passive (Warm Standby): One region handles all production traffic, the second region runs a scaled-down but fully deployed replica, ready to receive traffic within minutes of a failover. Simpler to operate than active-active, at the cost of idle capacity in the passive region.

For this guide, we'll implement active-passive with warm standby, the more achievable starting point for most teams, with a clear upgrade path to active-active.

┌─────────────────────────────────────────────────────────────┐
│                    Route 53 / Global Accelerator            │
│              (Primary: eu-east-1 | Failover: eu-west-1)     │
└────────────────────┬───────────────────────────┬────────────┘
                     │                           │
          ┌──────────▼──────────┐    ┌──────────▼──────────┐
          │    EKS Cluster      │    │    EKS Cluster      │
          │    eu-east-1        │    │    eu-west-1        │
          │    (PRIMARY)        │    │    (WARM STANDBY)   │
          │                     │    │                     │
          │  ┌───────────────┐  │    │  ┌───────────────┐  │
          │  │ App Workloads │  │    │  │ App Workloads │  │
          │  │ (Full Scale)  │  │    │  │ (Min Scale)   │  │
          │  └───────────────┘  │    │  └───────────────┘  │
          │  ┌───────────────┐  │    │  ┌───────────────┐  │
          │  │  RDS Primary  │◄─┼────┼──│  RDS Replica  │  │
          │  └───────────────┘  │    │  └───────────────┘  │
          └─────────────────────┘    └─────────────────────┘

Step 1 - Repository and Terraform Structure

Organize your Terraform repository to support multi-region deployments without duplication:

/infrastructure
  /modules
    /vpc              ← reusable VPC module
    /eks              ← reusable EKS module
    /rds              ← reusable RDS module
    /alb              ← reusable ALB module
  /regions
    /eu-east-1        ← primary region configuration
      main.tf
      variables.tf
      outputs.tf
      backend.tf
    /eu-west-1        ← standby region configuration
      main.tf
      variables.tf
      outputs.tf
      backend.tf
  /global             ← cross-region resources (Route 53, Global Accelerator, IAM)
    main.tf
    variables.tf
    outputs.tf

Each region directory consumes shared modules with region-specific variable overrides. The global directory manages resources that span regions and depends on outputs from both regional deployments.

Step 2 - Configure Terraform Providers with Aliases

Multi-region Terraform requires provider aliasing, defining multiple AWS provider configurations, one per region:

# infrastructure/global/providers.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
  }
}

hcl

infrastructure/regions/eu-east-1/providers.tf

provider "aws" {
alias = "primary"
region = "eu-east-1"

default_tags {
tags = {
ManagedBy = "terraform"
Environment = var.environment
Region = "eu-east-1"
Role = "primary"
}
}
}

hcl

infrastructure/regions/eu-west-1/providers.tf

provider "aws" {
alias = "standby"
region = "eu-west-1"

default_tags {
tags = {
ManagedBy = "terraform"
Environment = var.environment
Region = "eu-west-1"
Role = "standby"
}
}
}

hcl

The default_tags block applies consistent tagging to every resource in each region, critical for cost attribution and operational clarity when managing multi-region infrastructure.

Step 3 - VPC Module Design

The VPC module provisions the network foundation. Both regions use the same module with non-overlapping CIDR ranges, a requirement for VPC peering or Transit Gateway connectivity:

# infrastructure/modules/vpc/main.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = var.vpc_cidr

  azs             = var.availability_zones
  private_subnets = var.private_subnet_cidrs
  public_subnets  = var.public_subnet_cidrs

  enable_nat_gateway     = true
  single_nat_gateway     = false       # one NAT gateway per AZ for HA
  enable_dns_hostnames   = true
  enable_dns_support     = true

  # Required tags for EKS to discover subnets
  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"             = "1"
    "kubernetes.io/cluster/${var.cluster_name}"   = "owned"
  }

  public_subnet_tags = {
    "kubernetes.io/role/elb"                      = "1"
    "kubernetes.io/cluster/${var.cluster_name}"   = "owned"
  }
}

# infrastructure/regions/eu-east-1/main.tf
module "vpc_primary" {
  source = "../../modules/vpc"
  providers = { aws = aws.primary }

  cluster_name         = "prod-eu-east-1"
  vpc_cidr             = "10.0.0.0/16"       # primary CIDR range
  availability_zones   = ["eu-east-1a", "eu-east-1b", "eu-east-1c"]
  private_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnet_cidrs  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
}

# infrastructure/regions/eu-west-1/main.tf
module "vpc_standby" {
  source = "../../modules/vpc"
  providers = { aws = aws.standby }

  cluster_name         = "prod-eu-west-1"
  vpc_cidr             = "10.1.0.0/16"       # non-overlapping CIDR range
  availability_zones   = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnet_cidrs = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
  public_subnet_cidrs  = ["10.1.101.0/24", "10.1.102.0/24", "10.1.103.0/24"]
}

Non-overlapping CIDRs (10.0.x.x for primary, 10.1.x.x for standby) are mandatory. Overlapping ranges prevent VPC peering and Transit Gateway routing between regions.

Step 4 - EKS Cluster Module

The EKS module provisions the control plane and managed node groups:

# infrastructure/modules/eks/main.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.kubernetes_version

  vpc_id                         = var.vpc_id
  subnet_ids                     = var.private_subnet_ids
  cluster_endpoint_public_access = false          # private endpoint only
  cluster_endpoint_private_access = true

  cluster_addons = {
    coredns    = { most_recent = true }
    kube-proxy = { most_recent = true }
    vpc-cni    = { most_recent = true }
    aws-ebs-csi-driver = { most_recent = true }
  }

  eks_managed_node_groups = {
    system = {
      name           = "${var.cluster_name}-system"
      instance_types = ["m6i.large"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2

      labels = { role = "system" }
      taints = [{
        key    = "CriticalAddonsOnly"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]
    }

    application = {
      name           = "${var.cluster_name}-application"
      instance_types = ["m6i.xlarge", "m6i.2xlarge"]
      min_size       = var.min_app_nodes
      max_size       = var.max_app_nodes
      desired_size   = var.desired_app_nodes

      labels = { role = "application" }
    }
  }

  # Enable IRSA for pod-level IAM roles
  enable_irsa = true
}

In the standby region, pass reduced node counts to minimize idle capacity costs:

# infrastructure/regions/eu-west-1/main.tf - standby node counts
module "eks_standby" {
  source = "../../modules/eks"
  providers = { aws = aws.standby }

  cluster_name        = "prod-eu-west-1"
  kubernetes_version  = "1.29"
  vpc_id              = module.vpc_standby.vpc_id
  private_subnet_ids  = module.vpc_standby.private_subnets

  # Warm standby — minimum viable footprint, scales up on failover
  min_app_nodes     = 1
  max_app_nodes     = 20      # headroom for failover traffic burst
  desired_app_nodes = 2
}

Step 5 - Remote State Configuration and Cross-Region References

Each region's Terraform state is stored independently in S3. The global configuration reads outputs from both regional states:

# infrastructure/regions/eu-east-1/backend.tf

hcl
terraform {
backend "s3" {
bucket = "your-org-terraform-state"
key = "eks/eu-east-1/terraform.tfstate"
region = "eu-east-1"
encrypt = true
dynamodb_table = "terraform-state-lock"
}
}

hcl

infrastructure/global/main.tf - reading cross-region outputs

data "terraform_remote_state" "primary" {
backend = "s3"
config = {
bucket = "your-org-terraform-state"
key = "eks/eu-east-1/terraform.tfstate"
region = "eu-east-1"
}
}

data "terraform_remote_state" "standby" {
backend = "s3"
config = {
bucket = "your-org-terraform-state"
key = "eks/eu-west-1/terraform.tfstate"
region = "eu-east-1" # state bucket is in primary region; key points to standby state
}
}


Remote state data sources allow your global configuration to consume the ALB DNS names, cluster endpoint ARNs, and VPC IDs from both regions, enabling Route 53 health-check-based failover configuration without manual copy-pasting.

## Step 6 - Cross-Region Traffic Routing with Route 53
Configure Route 53 health-check-based failover routing, automatically switching traffic to the standby region when the primary region's health check fails:

hcl

infrastructure/global/route53.tf

Health check against primary region ALB

resource "aws_route53_health_check" "primary" {
fqdn = data.terraform_remote_state.primary.outputs.alb_dns_name
port = 443
type = "HTTPS"
resource_path = "/health"
failure_threshold = 3
request_interval = 30

tags = { Name = "primary-health-check" }
}

Primary record - active under normal conditions

resource "aws_route53_record" "primary" {
zone_id = var.route53_zone_id
name = "api.your-domain.com"
type = "A"

failover_routing_policy {
type = "PRIMARY"
}

set_identifier = "primary"
health_check_id = aws_route53_health_check.primary.id

alias {
name = data.terraform_remote_state.primary.outputs.alb_dns_name
zone_id = data.terraform_remote_state.primary.outputs.alb_zone_id
evaluate_target_health = true
}
}

Standby record - receives traffic when primary health check fails

resource "aws_route53_record" "standby" {
zone_id = var.route53_zone_id
name = "api.your-domain.com"
type = "A"

failover_routing_policy {
type = "SECONDARY"
}

set_identifier = "standby"

alias {
name = data.terraform_remote_state.standby.outputs.alb_dns_name
zone_id = data.terraform_remote_state.standby.outputs.alb_zone_id
evaluate_target_health = true
}
}


With this configuration, Route 53 continuously polls the primary region's health check endpoint. If three consecutive checks fail (90 seconds), DNS failover to the standby region initiates automatically, no human intervention required.

## Step 7 - GitOps Deployment with Flux Across Regions
Managing application deployments across two clusters requires a GitOps controller in each region pulling from the same source of truth. **Flux CD** is purpose-built for this:

yaml

gitops/clusters/eu-east-1/flux-system/kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 5m
path: ./gitops/apps/production
prune: true
sourceRef:
kind: GitRepository
name: fleet-repository

yaml

gitops/clusters/eu-west-1/flux-system/kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 5m
path: ./gitops/apps/production # same app manifests, different cluster
prune: true
sourceRef:
kind: GitRepository
name: fleet-repository


Both Flux instances reconcile against the same `gitops/apps/production` path in your Git repository. A single push to main deploys to both regions, maintaining configuration parity automatically without manual multi-cluster kubectl commands.

## Step 8 - Database Strategy for Multi-Region
Application state is the hardest problem in multi-region architecture. For PostgreSQL:

hcl

infrastructure/modules/rds/main.tf - primary RDS instance

resource "aws_db_instance" "primary" {
provider = aws.primary
identifier = "${var.cluster_name}-primary"
engine = "postgres"
engine_version = "16.1"
instance_class = "db.r6g.xlarge"
allocated_storage = 100
storage_encrypted = true

backup_retention_period = 7
backup_window = "03:00-04:00"

# Enable automated backups to S3 for cross-region restore
enabled_cloudwatch_logs_exports = ["postgresql"]
}

Cross-region read replica in standby region

resource "aws_db_instance" "standby_replica" {
provider = aws.standby
identifier = "${var.cluster_name}-standby-replica"
replicate_source_db = aws_db_instance.primary.arn # cross-region replication
instance_class = "db.r6g.large" # smaller read-only in standby
storage_encrypted = true

# Replica can be promoted to primary during failover
# Promotion is a manual or automated operational step, not instantaneous
}


The read replica in the standby region receives continuous replication from the primary, typically with 1–5 seconds of replication lag. During a failover, promoting the replica to a standalone primary takes several minutes, during which write operations are unavailable. Design your application's failover behavior (queuing writes, surfacing a maintenance page) around this window.

## CI/CD Integration

yaml

.github/workflows/terraform-deploy.yml

name: Terraform Multi-Region Deploy

on:
push:
branches: [main]
paths: ['infrastructure/**']

jobs:
deploy-primary:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3

  - name: Terraform Apply - eu-east-1
    working-directory: infrastructure/regions/eu-east-1
    run: |
      terraform init
      terraform apply -auto-approve
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

deploy-standby:
runs-on: ubuntu-latest
needs: deploy-primary # apply standby after primary succeeds
environment: production
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3

  - name: Terraform Apply — eu-west-1
    working-directory: infrastructure/regions/eu-west-1
    run: |
      terraform init
      terraform apply -auto-approve
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

deploy-global:
runs-on: ubuntu-latest
needs: [deploy-primary, deploy-standby] # global depends on both regions
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3

  - name: Terraform Apply — global
    working-directory: infrastructure/global
    run: |
      terraform init
      terraform apply -auto-approve




The `needs` dependency chain ensures the global configuration (Route 53 failover) is applied only after both regional clusters are provisioned and their outputs are available in remote state.

## Common Pitfalls to Avoid
**Overlapping CIDR ranges.** The most common setup mistake, and the one that forces a complete VPC rebuild to fix. Plan your IP address space before writing any Terraform.

**Single NAT gateway per region.** Using one NAT gateway per region (instead of one per AZ) is cheaper but creates an availability zone as a single point of failure. Use `single_nat_gateway = false` in production.

**Not testing failover.** A failover procedure that has never been practiced will fail under pressure. Schedule a quarterly failover drill, redirect traffic to the standby region, validate application behavior, then cut back.

**Treating the standby as truly passive.** A warm standby that receives no real traffic is also receiving no validation that it works. Consider routing a small percentage of read traffic (5–10%) to the standby region to continuously validate its health.

**State drift between regions.** Without GitOps enforcing parity, application deployments to the primary region can drift from the standby. Flux or ArgoCD reconciliation against a single Git source is the most reliable prevention.

## Conclusion
Building a multi-region EKS cluster with Terraform is one of the highest-leverage investments a platform team can make in system reliability. It eliminates the single largest uncontrolled failure mode in cloud-native infrastructure, regional availability, and does so in a way that is reproducible, version-controlled, and operationally predictable.

Start with active-passive. Get the Terraform modules right, validate the Route 53 failover, practice the database promotion procedure, and establish GitOps parity across both clusters. Once that foundation is solid, the path to active-active is an evolution of what you've already built, not a rewrite.

The blueprint takes time to draw. The construction, as always, is where automation earns its keep.

*Using GKE or AKS instead of EKS? The multi-region architecture patterns and GitOps strategy in this guide apply directly, the Terraform provider and cluster module differ, but the design is cloud-agnostic. Drop your platform in the comments.*

The Blueprint for 2026: Why Architecture Must Precede Automation

ThankGod Chibugwum Obobo — Sun, 28 Jun 2026 17:01:00 +0000

2026 has brought no shortage of automation tooling, AI coding agents that scaffold entire features, CI/CD pipelines that deploy on every commit, and infrastructure-as-code platforms that provision environments in minutes. The temptation is to treat automation as a substitute for architectural discipline: if the tooling is fast enough, do the underlying design decisions even matter?

They matter more than ever. Automation amplifies whatever you point it at, including bad architecture. A poorly bounded service automated into a CI/CD pipeline doesn't become well-bounded, it becomes a poorly bounded service that ships faster and breaks more often. An AI coding agent given a vague, undocumented module doesn't infer good structure from chaos, it replicates the chaos at speed.

This article makes the case for a principle that's easy to state and hard to practice: architecture must precede automation. We'll look at why this ordering matters, what happens when teams get it backward, and how to build the discipline to get it right.

Automation Is a Force Multiplier

The core insight is simple, automation doesn't fix structural problems. It multiplies the rate at which those problems manifest.

Consider a backend system with tangled service boundaries, where orders-service reaches directly into users-service's database, where business logic is duplicated across three modules, where there's no clear ownership of shared data models. Without automation, these problems surface slowly, a developer hits friction, works around it, ships anyway. Painful, but contained.

Now introduce aggressive CI/CD automation, AI-assisted code generation, and infrastructure-as-code into that same system. Deployments that used to take a day now take ten minutes. AI agents generate new endpoints against the tangled data model in seconds. The same structural problems that used to surface occasionally now surface constantly and faster than your team can absorb the friction.

This is the central danger of automating before architecting: you don't remove the bottleneck, you relocate it, usually to incident response, on-call burden, or a production outage that automation made possible to ship faster but no easier to recover from.

What "Architecture First" Actually Means

"Architecture first" is not a call to return to multi-month design documents before writing a line of code. In 2026's engineering environment, that approach is its own form of dysfunction, over-planning that delays value delivery without guaranteeing better outcomes. Architecture-first means something more specific and more practical:

Define boundaries before you automate across them. If you don't know where one service's responsibility ends and another's begins, automating deployment, testing, or infrastructure across that ambiguous boundary only makes the ambiguity move faster. Service boundaries, data ownership, and API contracts are architectural decisions that must be made deliberately, not discovered accidentally through automation tooling.

Establish your domain model before generating code against it. AI coding agents are remarkably effective at extending an existing, well-structured codebase. They are equally effective at extending a poorly structured one, producing more code that follows the same bad patterns, at a volume no human reviewer can reasonably keep up with.

Decide your failure model before automating recovery. Self-healing systems, automated rollbacks, and circuit breakers are powerful, but only if you've already decided what failure looks like, what's recoverable, and what requires human judgment. Automating recovery for a system whose failure modes haven't been mapped is automating guesswork.

Choose your data architecture before automating data pipelines. Once an ETL pipeline, a caching layer, or an event stream is automated and running, the underlying data model becomes extremely expensive to change, every downstream automated process depends on it remaining stable.

The 2026 Pressure Toward Automation-First

It's worth naming why this principle is harder to follow now than it was five years ago.

AI coding agents lower the cost of generating code dramatically, which makes "just generate it and see" feel like a reasonable substitute for upfront design. The cost of writing code has dropped, the cost of living with the wrong architecture has not.

CI/CD maturity has made deployment nearly frictionless, which removes a natural pause point that used to force teams to think before shipping. When deploying took an afternoon of manual steps, teams were implicitly forced to batch decisions and consider them more carefully. When deploying takes ninety seconds, that forcing function disappears.

Competitive pressure rewards visible velocity over invisible architectural soundness. Shipping ten features this quarter is visible to leadership. The clean service boundary that made shipping the eleventh feature possible without a rewrite is invisible, until it's missing.

None of this means automation is the problem. It means automation has gotten powerful enough that it no longer provides natural friction against bad architectural decisions, that friction now has to be deliberately engineered back in, through team practices and review discipline.

What Architecture-First Looks like in Practice

This doesn't require slowing down, it requires sequencing.

Before automating CI/CD for a new service: define its API contract, its data ownership boundary, and its failure isolation strategy. This can take an afternoon of focused design discussion, not a quarter of documentation. The output is a lightweight decision record, not a 40-page architecture document.

Before pointing an AI coding agent at a codebase: ensure the codebase has clear module boundaries, consistent naming conventions, and a small number of canonical patterns the agent can learn from and extend. An agent given a clean example to follow produces dramatically better output than one given a blank slate or a tangled mess.

Before automating infrastructure provisioning: decide your environment topology, your network boundaries, and your secrets management approach (see our guide on replacing hardcoded credentials with Vault and GitHub OIDC). Terraform modules generated against an undefined topology become technical debt the moment they're applied.

Before automating test generation: define what correctness actually means for the system under test, the contracts, the invariants, the edge cases that matter to the business. AI-generated tests against an undefined specification produce tests that pass without verifying anything meaningful.

The pattern across all of these is, a small deliberate architectural decision made before automation gives the automation something solid to amplify. Skipping that decision gives automation chaos to amplify instead.

The Cost of Getting the Order Wrong

Teams that automate before architecting tend to discover the cost in a predictable sequence:

First, velocity feels great. Automation is shipping fast. Dashboards show high deployment frequency, high PR throughput, high AI-assisted code volume.

Then, defects compound faster than they can be fixed. Because the underlying structure was never sound, every new automated change has a wider blast radius than expected. Bugs surface in unexpected places because boundaries were never enforced.

Then, the team builds automation to manage the automation. More alerting to catch what the velocity is breaking. More rollback automation to recover from what the velocity is shipping. This is a sign the team is now spending its automation investment compensating for an architectural debt it never paid down, not multiplying genuine engineering leverage.

Finally, a rewrite becomes the only path forward. At this point, the team faces the choice no one wants, a costly, risky architectural rewrite, executed under the same time pressure that caused the original problem.

This sequence is avoidable, not by rejecting automation, but by insisting on the right order of operations.

A Practical Framework: The Architecture Gate

For teams who want a concrete practice rather than just a principle, introduce an architecture gate before any significant automation investment:

Before automating: [CI/CD pipeline | AI code generation | infra provisioning | test generation]

Ask:

Are the relevant service/module boundaries explicitly defined?
Is data ownership for this domain unambiguous?
Is there a documented contract (API, schema, interface) the automation will rely on?
Have failure modes for this component been identified and categorized?
Is there a clear owner for decisions this automation will encode at scale?

If any answer is "no", pause and resolve it first.
The fix is usually a focused half-day design discussion, not a lengthy process.

This isn't bureaucracy for its own sake. It's a five-question checklist that takes less time than debugging the first production incident caused by skipping it.

Architecture and Automation Are Not in Tension

The framing of "architecture vs. automation" is itself a false choice. The most effective engineering organizations in 2026 don't choose one over the other, they sequence them correctly. Sound architecture is what makes aggressive automation safe. Aggressive automation is what makes sound architecture deliver value at speed.

A well-bounded microservices architecture (see our guide on migrating from monolith to microservices with NestJS and Postgres) becomes dramatically more valuable once CI/CD, contract testing, and observability are automated around it, because the boundaries hold under the increased velocity rather than buckling.

The lesson for 2026 is not "slow down." It's "decide deliberately, then move fast." Architecture is the blueprint. Automation is the construction crew. No amount of construction speed compensates for a blueprint that was never drawn.

Conclusion

As AI tooling and automation platforms continue to compress the time between idea and deployed code, the temptation to skip architectural thinking will only grow stronger, because skipping it no longer feels like it costs anything, in the moment. It does. It simply defers the cost to a less convenient time, at a larger scale, with less context available to fix it.

The teams that will build the most reliable, adaptable systems in 2026 are not the ones who automate the most. They're the ones who know precisely what to automate, because they took the time to architect it first.

Draw the blueprint. Then build fast.

What's your team's process for architectural review before adopting new automation tooling? Share your approach in the comments.

Dependency Auditing at Scale: How to Automate Supply Chain Security with Dependabot and Snyk

ThankGod Chibugwum Obobo — Sun, 21 Jun 2026 17:00:00 +0000

The average modern web application depends on hundreds, sometimes thousands, of transitive npm packages. Each one is a potential attack vector, a vulnerability waiting to be disclosed, a maintainer account waiting to be compromised, or a malicious update waiting to be published. Manually tracking the security posture of a dependency tree this large is not just impractical at any meaningful scale, it is impossible.

Supply chain security has moved from a niche compliance concern to a top-tier engineering priority, driven by high-profile incidents like the event-stream backdoor, the ua-parser-js compromise, and the XZ Utils backdoor that nearly compromised SSH across major Linux distributions. Attackers have learned that compromising one widely-used package is more efficient than attacking a thousand individual applications.

This guide covers how to build an automated dependency auditing pipeline using GitHub Dependabot and Snyk, two complementary tools that, used together, give you continuous vulnerability detection, automated remediation, and CI/CD gates that stop vulnerable code before it ships.

Why Manual Dependency Management Doesn't Scale

Consider a mid-sized application with 40 direct dependencies. Each of those typically pulls in 10–30 transitive dependencies of its own, meaning your actual dependency tree can easily exceed 800–1200 packages. Each package:

Has its own maintainers, release cadence, and security practices
Can be updated at any time, potentially introducing a vulnerability or malicious code
May depend on other packages with their own vulnerabilities
Often includes postinstall scripts that execute arbitrary code on npm install

Manually monitoring CVE databases, cross-referencing them against your package-lock.json, and tracking remediation status across dozens of services is not a sustainable practice. It requires automation, both for detection and for response.

Understanding the Two-Tool Strategy

Dependabot (native to GitHub) and Snyk (a dedicated security platform) overlap in purpose but differ in depth and capability. Understanding the distinction helps you use both effectively rather than redundantly.

Capability	Dependabot	Snyk
Vulnerability database	GitHub Advisory Database	Snyk's proprietary database (broader, faster disclosure)
Automated PRs for fixes	Yes	Yes
License compliance scanning	No	Yes
Container image scanning	Limited	Yes
Infrastructure as Code scanning	No	Yes
Code-level vulnerability scanning (SAST)	No	Yes (Snyk Code)
Native GitHub integration	Built-in, free	Requires app installation
Fix prioritization by exploitability	Basic	Advanced risk scoring
Cost	Free for public & private repos	Free tier is limited, pay for scale

Recommended approach: Use Dependabot as your baseline, always-on dependency update mechanism, it's free, native, and requires minimal setup. Layer Snyk on top for deeper vulnerability intelligence, license compliance, and broader scanning coverage (containers, IaC, code-level issues) across your most critical services.

Step 1 - Configure Dependabot

Dependabot is configured via a single YAML file per repository. Start with version updates and security updates enabled:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "monday"
      time: "09:00"
    open-pull-requests-limit: 10
    groups:
      production-dependencies:
        dependency-type: "production"
        update-types: ["minor", "patch"]
      development-dependencies:
        dependency-type: "development"
        update-types: ["minor", "patch", "major"]
    ignore:
      - dependency-name: "legacy-package"
        update-types: ["version-update:semver-major"]
    labels:
      - "dependencies"
      - "automated"
    reviewers:
      - "platform-team"
    commit-message:
      prefix: "chore(deps)"

  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"

  - package-ecosystem: "github-actions"
    directory: "/.github/workflows"
    schedule:
      interval: "weekly"

Key configuration decisions:

groups consolidates multiple minor/patch updates into a single PR, dramatically reducing PR noise for large dependency trees.
open-pull-requests-limit prevents Dependabot from overwhelming your review queue.
Separate ecosystems (docker, github-actions) ensure your container base images and CI workflow actions are also kept current, both common and overlooked attack surfaces.

Enabling Security-Only Updates

Security updates are enabled by default once Dependabot alerts are turned on for the repository, and they bypass the schedule, firing immediately when a new vulnerability is disclosed for a dependency you use:

Repository → Settings → Code security and analysis
Dependabot alerts
Dependabot security updates
Dependabot version updates

Step 2 - Configure Snyk for Deeper Scanning

Install the Snyk GitHub integration and add a .snyk policy file to your repository:

npm install -g snyk
snyk auth
snyk test

Generate a baseline .snyk policy file to manage known, accepted-risk vulnerabilities with documented justification:

# .snyk
version: v1.5.0
ignore:
  SNYK-JS-LODASH-1040724:
    - '*':
        reason: >
          Vulnerability is in a dev-only build tool, not present in
          production bundle. Tracked in JIRA-4521 for upgrade in Q3.
        expires: 2026-09-30T00:00:00.000Z
patch: {}

Never use .snyk to silently suppress vulnerabilities. Every ignored entry requires a documented reason and an expiration date, forcing periodic re-evaluation rather than permanent suppression.

Snyk CI Integration

# .github/workflows/snyk-security.yml
name: Snyk Security Scan

on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  snyk-dependencies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Snyk to check for vulnerabilities
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high --fail-on=upgradable

      - name: Run Snyk Code (SAST)
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          command: code test
          args: --severity-threshold=high

  snyk-container:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build container image
        run: docker build -t app:${{ github.sha }} .

      - name: Run Snyk Container scan
        uses: snyk/actions/docker@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          image: app:${{ github.sha }}
          args: --severity-threshold=high

The --fail-on=upgradable flag is important, it only fails the build for vulnerabilities that have an available fix, preventing your pipeline from blocking on issues you currently have no way to remediate.

Step 3 - License Compliance Scanning

Open source licenses carry legal obligations, some permissive (MIT, Apache 2.0), others restrictive (GPL, AGPL) in ways that can create compliance risk for commercial products. Snyk's license scanning catches these automatically:

# Add license policy enforcement
- name: Snyk License Compliance
  uses: snyk/actions/node@master
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
  with:
    command: test
    args: --severity-threshold=high --fail-on=all
            --policy-path=.snyk-license-policy.json

Configure license policies in the Snyk dashboard to flag or block specific license types:

Disallow:  GPL-3.0, AGPL-3.0, SSPL-1.0
Warn:      LGPL-2.1, MPL-2.0
Allow:     MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC

This is particularly critical for companies building commercial or closed-source products, an AGPL-licensed transitive dependency can create unexpected obligations to release your own source code.

Step 4 - Prioritizing Remediation with Risk Scoring

Not every vulnerability deserves the same urgency. A critical CVE in a dependency that's only used in your test suite is lower risk than a medium-severity CVE in a package that processes user input in production.

Snyk's risk scoring considers exploit maturity, whether the vulnerable code path is actually reachable, and the dependency's position in your tree (direct vs. transitive) to prioritize remediation:

# View prioritized vulnerabilities sorted by risk score
snyk test --json | snyk-to-html -o report.html

# Or query via API for custom dashboards
curl -H "Authorization: token $SNYK_TOKEN" \
  "https://api.snyk.io/v1/org/$ORG_ID/projects/$PROJECT_ID/issues"

Establish a remediation SLA based on severity and reachability:

Severity	Reachable in Production Code	Remediation SLA
Critical	Yes	24–48 hours
Critical	No (dev/test only)	1 week
High	Yes	1 week
High	No	2 weeks
Medium	Yes	2 weeks
Medium	No	Next regular update cycle
Low	Any	Tracked, no SLA

Step 5 - Automating Patch PRs and Auto-Merge

For low-risk updates, patch and minor version bumps with passing tests, automate the entire remediation loop:

# .github/workflows/automerge-dependabot.yml
name: Auto-merge Dependabot PRs

on: pull_request

permissions:
  pull-requests: write
  contents: write

jobs:
  automerge:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'
    steps:
      - name: Fetch Dependabot metadata
        id: metadata
        uses: dependabot/fetch-metadata@v2
        with:
          github-token: "${{ secrets.GITHUB_TOKEN }}"

      - name: Auto-approve patch and minor updates
        if: steps.metadata.outputs.update-type == 'version-update:semver-patch' || steps.metadata.outputs.update-type == 'version-update:semver-minor'
        run: gh pr review --approve "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Enable auto-merge
        if: steps.metadata.outputs.update-type == 'version-update:semver-patch' || steps.metadata.outputs.update-type == 'version-update:semver-minor'
        run: gh pr merge --auto --squash "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Auto-merge should only apply to patch/minor updates and require your full CI suite (tests, linting, security scans) to pass first. gh pr merge --auto waits for all required status checks before merging. Major version updates should always require human review, since they're more likely to introduce breaking changes.

Step 6 - Software Bill of Materials (SBOM) Generation

Beyond vulnerability scanning, regulatory and enterprise procurement requirements increasingly demand a Software Bill of Materials, a complete inventory of every component in your software supply chain:

# Generate an SBOM in CycloneDX format using Snyk
snyk sbom --format=cyclonedx1.5+json --org=$ORG_ID > sbom.json

# .github/workflows/sbom.yml
- name: Generate SBOM
  run: snyk sbom --format=cyclonedx1.5+json > sbom.json
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

- name: Upload SBOM as release artifact
  uses: actions/upload-artifact@v4
  with:
    name: sbom
    path: sbom.json

Store SBOMs alongside each release artifact, they provide auditable proof of exactly what shipped, which is increasingly required for enterprise customers and government contracts (e.g., U.S. Executive Order 14028 compliance).

Step 7 - Blocking Malicious Packages Proactively

Beyond known CVEs, supply chain attacks often involve packages with no disclosed vulnerability, because the malicious behavior was intentionally introduced by a compromised maintainer account. Snyk and GitHub both maintain databases of known-malicious packages distinct from standard CVE tracking:

# Block installation of known-malicious packages at install time
- name: Check for malicious packages
  run: |
    npx socket-security audit  # cross-references known malicious package database

Combine this with npm ci --ignore-scripts in CI to prevent postinstall scripts from executing arbitrary code during dependency installation, a common vector for credential exfiltration:

npm ci --ignore-scripts
# Run necessary build scripts explicitly and individually instead of trusting postinstall hooks
npm run build

Common Pitfalls to Avoid

Treating Dependabot PR volume as noise to ignore. A repository with 60 open Dependabot PRs signals the automation is not being acted on, defeating its purpose. Use grouping and auto-merge for low-risk updates to keep the queue manageable.

Scanning only direct dependencies. The majority of real-world vulnerabilities live in transitive dependencies your team has never directly chosen. Ensure your scanning tools traverse the full dependency tree, not just package.json.

No reachability analysis. Flagging every CVE in every dependency, even ones whose vulnerable code path is never executed, creates alert fatigue. Use tools with reachability analysis (Snyk Code, Socket) to prioritize what's actually exploitable in your application.

Ignoring container and IaC layers. Application dependencies are only one layer of your supply chain. Base container images and Terraform providers carry their own vulnerabilities and deserve the same scanning rigor.

Conclusion

Supply chain security at scale cannot be a manual process, the sheer size of modern dependency trees makes that mathematically impossible to sustain. Dependabot provides a free, always-on baseline for dependency updates and security patches. Snyk adds depth: license compliance, container and IaC scanning reachability-aware risk scoring, and code-level SAST.

Used together, with auto-merge for low-risk updates, documented exception policies for accepted risk, and SBOM generation for auditability, you transform dependency management from a reactive scramble after each disclosed CVE into a continuous, automated engineering practice.

The goal isn't zero vulnerabilities, that's not achievable in any non-trivial dependency tree. The goal is continuous visibility, fast remediation for what's exploitable, and an audit trail for everything else.

Running a polyglot stack with Python, Go, and Java services alongside Node.js? Both Dependabot and Snyk support multi-ecosystem scanning natively — the configuration patterns in this guide apply with ecosystem-specific adjustments. Drop your stack in the comments.

Secrets Management in CI/CD: How to Replace Hardcoded Credentials with Vault and GitHub OIDC

ThankGod Chibugwum Obobo — Sun, 14 Jun 2026 17:00:00 +0000

Hardcoded credentials are one of the most persistent and dangerous security vulnerabilities in modern software delivery. Database passwords committed to .env files, AWS access keys stored as long-lived GitHub repository secrets, API tokens embedded in CI configuration, these are not edge cases. They are standard practice at the majority of engineering teams, and they represent a ticking clock.

When a long-lived credential leaks, through a misconfigured repository, a disgruntled employee, or a supply chain compromise, the window of exposure is measured in the lifetime of the credential, often months or years. The blast radius includes every system that credential has access to.

The solution is not better secret hygiene on static credentials. It is eliminating static credentials entirely, replacing them with dynamic, short-lived tokens issued on demand and automatically expired. HashiCorp Vault and GitHub OIDC make this achievable without rewriting your CI/CD pipelines from scratch.

This guide covers how to design and implement a secrets management architecture that removes hardcoded credentials from your CI/CD workflows permanently.

Why Static Secrets Are a Structural Problem

Static credentials, long-lived API keys, database passwords, and cloud access keys, have three fundamental weaknesses that secret rotation and better storage cannot fully address:

Broad access scope. A static AWS access key issued to a CI pipeline typically has fixed permissions. Over time, those permissions accumulate as new capabilities are needed and old ones are never revoked, a textbook case of privilege creep.

Long exposure windows. A leaked static credential remains valid until it is manually rotated. Discovery of a leak is rarely immediate, the average time to detect a credential compromise is measured in days to weeks, during which an attacker has unrestricted access.

No audit trail per workflow run. When ten pipeline runs use the same static credential, your cloud provider's access logs show ten identical requests from the same key, making it impossible to attribute access to a specific pipeline, branch, or commit.

Dynamic secrets solve all three: each pipeline run gets a unique, scoped, time-limited credential, automatically expired when the job completes.

The Architecture: GitHub OIDC + HashiCorp Vault

The zero-static-credentials architecture has two components working in concert:

GitHub OIDC (OpenID Connect): GitHub Actions can generate a signed JWT token, an OIDC token, that cryptographically proves the identity of the workflow that requested it. This token includes claims about the repository, branch, environment, and workflow that generated it. Critically, it requires no pre-shared secret to verify, Vault validates it directly against GitHub's public JWKS endpoint.

HashiCorp Vault: Vault receives the OIDC token from the GitHub Actions runner, validates it, maps it to a role with defined permissions, and issues a short-lived Vault token in return. The pipeline then uses that Vault token to fetch exactly the secrets it needs, no more, no less.

The request flow looks like this:

GitHub Actions Runner
        │
        │ 1. Request OIDC JWT from GitHub
        ▼
GitHub OIDC Provider
        │
        │ 2. Returns signed JWT (claims: repo, branch, workflow, environment)
        ▼
GitHub Actions Runner
        │
        │ 3. Exchange JWT for Vault token (POST /auth/jwt/login)
        ▼
HashiCorp Vault
        │
        │ 4. Validate JWT against GitHub JWKS, map to role, issue Vault token
        ▼
GitHub Actions Runner
        │
        │ 5. Use Vault token to read secrets (GET /secret/data/...)
        ▼
HashiCorp Vault
        │
        │ 6. Return scoped, time-limited secret
        ▼
GitHub Actions Runner
        │
        │ 7. Use secret in pipeline step (expires when job completes)
        ▼
     Pipeline Step

No hardcoded credential appears anywhere in this flow. The GitHub runner never holds a long-lived secret, only a short-lived Vault token scoped to the secrets needed for that specific job.

Step 1 - Configure Vault JWT Authentication

Enable and configure the JWT authentication backend in Vault to trust GitHub's OIDC provider:

# Enable JWT auth backend
vault auth enable jwt

# Configure GitHub as the trusted OIDC provider
vault write auth/jwt/config \
  oidc_discovery_url="https://token.actions.githubusercontent.com" \
  bound_issuer="https://token.actions.githubusercontent.com"

Vault will automatically fetch and cache GitHub's public keys from the discovery URL, no manual key management required.

Step 2 - Create Vault Policies

Vault policies define what secrets a role can access. Apply the principle of least privilege, each role gets access only to the secrets its corresponding workflow actually needs:

# vault/policies/orders-service-deploy.hcl
path "secret/data/production/orders-service/*" {
  capabilities = ["read"]
}

path "secret/data/production/shared/database" {
  capabilities = ["read"]
}

# Explicitly deny access to other services' secrets
path "secret/data/production/payments-service/*" {
  capabilities = ["deny"]
}

# vault/policies/terraform-plan.hcl
# Read-only access to infrastructure credentials for planning
path "aws/creds/terraform-plan-role" {
  capabilities = ["read"]
}

# Apply policies to Vault
vault policy write orders-service-deploy vault/policies/orders-service-deploy.hcl
vault policy write terraform-plan vault/policies/terraform-plan.hcl

Step 3 - Create Vault Roles Bound to GitHub Claims

Vault roles map GitHub OIDC token claims to Vault policies. This is where you define which GitHub workflows can access which secrets, using the claims in the OIDC token as the binding condition:

# Role for deploying orders-service from the main branch only
vault write auth/jwt/role/orders-service-deploy \
  role_type="jwt" \
  bound_audiences="https://github.com/your-org" \
  bound_claims_type="glob" \
  bound_claims='{
    "repository": "your-org/orders-service",
    "ref": "refs/heads/main",
    "environment": "production"
  }' \
  user_claim="actor" \
  policies="orders-service-deploy" \
  ttl="15m"                           # Vault token expires in 15 minutes

# Role for Terraform planning — any branch, read-only AWS credentials
vault write auth/jwt/role/terraform-plan \
  role_type="jwt" \
  bound_audiences="https://github.com/your-org" \
  bound_claims='{
    "repository": "your-org/infrastructure",
    "workflow": ".github/workflows/terraform-plan.yml"
  }' \
  user_claim="actor" \
  policies="terraform-plan" \
  ttl="30m"

The bound_claims binding is the critical security control. A workflow from your-org/payments-service requesting the orders-service-deploy role will be rejected, even if it has a valid GitHub OIDC token, because the repository claim doesn't match.

Step 4 - Configure GitHub Actions Workflows

With Vault configured, update your GitHub Actions workflows to fetch secrets dynamically:

# .github/workflows/deploy-orders-service.yml
name: Deploy Orders Service

on:
  push:
    branches: [main]

permissions:
  id-token: write    # required to request GitHub OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production  # must match Vault role's bound_claims

    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to Vault via GitHub OIDC
        uses: hashicorp/vault-action@v3
        id: vault
        with:
          url: ${{ secrets.VAULT_ADDR }}
          method: jwt
          role: orders-service-deploy
          secrets: |
            secret/data/production/orders-service/database  DB_PASSWORD | DATABASE_PASSWORD ;
            secret/data/production/orders-service/database  DB_HOST     | DATABASE_HOST ;
            secret/data/production/shared/stripe            API_KEY     | STRIPE_API_KEY ;

      - name: Run database migrations
        run: npm run migration:run
        env:
          DATABASE_URL: postgresql://${{ env.DATABASE_HOST }}:5432/orders?password=${{ env.DATABASE_PASSWORD }}

      - name: Deploy to Kubernetes
        run: kubectl apply -f k8s/
        env:
          STRIPE_API_KEY: ${{ env.STRIPE_API_KEY }}

The hashicorp/vault-action step handles the full OIDC exchange automatically, requesting the GitHub token, authenticating to Vault, and injecting the fetched secrets as environment variables for subsequent steps. The secrets are masked in logs and scoped to the job, they do not persist beyond the workflow run.

Step 5 - Dynamic Database Credentials with Vault

For databases, Vault can go further than storing static passwords, it can generate unique, short-lived database credentials for each pipeline run using the Vault database secrets engine:

# Enable the database secrets engine
vault secrets enable database

# Configure the PostgreSQL connection
vault write database/config/orders-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="orders-service-ci" \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/orders" \
  username="vault-admin" \
  password="$DB_ADMIN_PASSWORD"

# Define a role that generates credentials on demand
vault write database/roles/orders-service-ci \
  db_name="orders-db" \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
    GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="30m" \
  max_ttl="1h"

Now update the Vault policy to allow reading dynamic credentials:

# vault/policies/orders-service-ci.hcl
path "database/creds/orders-service-ci" {
  capabilities = ["read"]
}

Each CI pipeline run that requests database credentials receives a unique PostgreSQL role, created just for that run and automatically dropped when the TTL expires. No two pipeline runs share credentials. A leaked credential from a pipeline run is valid for a maximum of 30 minutes and grants access only to the tables that role needs.

Step 6 - Auditing and Monitoring

Every Vault operation generates an audit log entry. Enable the file audit backend and ship logs to your SIEM:

# Enable Vault audit logging
vault audit enable file file_path=/var/log/vault/audit.log

Each entry records the requesting token, the path accessed, the timestamp, and the response status, giving you a complete, tamper-evident trail of every secret access across all pipeline runs.

Alert on these patterns in your log aggregator:

Any access to secret/data/production/* outside of expected workflow hours
Failed authentication attempts against Vault JWT roles
Policy violations (denied path access attempts)
Unusually high secret read volume from a single token

Step 7 - Migrating from GitHub Repository Secrets

Migrating an existing CI/CD system from static GitHub repository secrets to Vault is a multi-step process. Approach it incrementally:

Phase 1 - Audit existing secrets. Inventory every secret in your GitHub organization's repository and environment settings. Classify each by service, sensitivity, and which workflows consume it.

Phase 2 - Replicate in Vault. Write all existing static secrets to Vault under a structured path hierarchy (secret/data/{environment}/{service}/{name}).

Phase 3 - Migrate non-critical workflows first. Start with staging and development workflows, lower risk, full learning opportunity. Validate that secret fetching works correctly before touching production.

Phase 4 - Migrate production workflows. Roll out Vault integration to production workflows one service at a time, with a rollback plan for each.

Phase 5 - Retire static secrets. Once all workflows are migrated and validated, delete the corresponding GitHub repository secrets. Validate that no workflow fails. Revoke access for any remaining long-lived credentials that were part of the old system.

Common Pitfalls to Avoid

Overly permissive Vault policies. Granting secret/data/production/* with ["read"] to a deployment role means that role can read every production secret, not just the ones it needs. Always scope policies to specific paths.

Skipping the environment claim binding. Without binding roles to a specific GitHub environment, any workflow in your repository could request production secrets. Always require the environment claim for production roles and protect GitHub environments with required reviewers.

Not setting TTLs. A Vault token without a TTL is effectively a long-lived credential, defeating the purpose of the architecture. Always set ttl and max_ttl on roles.

Logging secret values. The vault-action step masks secrets in GitHub Actions logs, but custom scripts that echo environment variables will expose them. Audit every workflow step that uses fetched secrets.

Conclusion

Hardcoded credentials are a solved problem, not by better storage of static secrets, but by eliminating the need for static secrets entirely. The combination of GitHub OIDC for cryptographic workflow identity and HashiCorp Vault for dynamic, scoped, time-limited secret issuance removes the structural weaknesses of static credentials at their root.

Every CI/CD pipeline that runs with this architecture gets credentials that are unique to that run, scoped to exactly what it needs, and automatically expired when the job completes. A leaked credential compromises a single pipeline run, not your entire production environment.

The migration requires upfront investment. But so does recovering from a credential compromise. The difference is that the migration is planned, controlled, and bounded, and the recovery rarely is.

Running workloads on AWS or GCP instead of relying on Vault for cloud credentials? AWS IAM Roles for GitHub Actions and GCP Workload Identity Federation follow the same OIDC pattern, no Vault required for cloud-native secrets.

Zero-trust React: defending against 2026's AI-powered phishing attacks

ThankGod Chibugwum Obobo — Sun, 07 Jun 2026 17:00:00 +0000

Phishing has graduated from clumsy grammar mistakes and suspicious domains to fully automated, AI-synthesized campaigns that clone your login UI pixel-for-pixel, intercept OAuth flows mid-flight, and exfiltrate session tokens before your monitoring pipeline fires its first alert. By mid-2026, the attack surface has expanded dramatically, large language models generate convincing spear-phishing lures at scale, adversarial browser extensions manipulate React's virtual DOM, and compromised npm packages inject silent keyloggers into your production bundle.

The perimeter-based security model, "everything inside our domain is trusted", is dead. Zero-trust architecture, for a long time, a backend and network-layer concern, must now be applied all the way to the browser. This article walks through a concrete zero-trust frontend model for React applications, maps it against the most prevalent 2026 threat vectors, and gives you production-ready patterns you can ship this sprint.

The 2026 phishing threat landscape

AI-synthesized clone attacks

Generative vision models can now scrape a login page, reproduce its layout with sub-pixel accuracy, register a homograph domain, and serve the clone from a CDN within minutes. The resulting page passes casual visual inspection and, crucially, passes many automated screenshot-diff detectors because the HTML structure is semantically identical to the original.

What makes React apps particularly vulnerable: Client-side rendering delays the fully-painted UI by 200–800ms. Attackers exploit that window to perform a "bait-and-switch", serving a static pre-rendered clone while quietly redirecting the real fetch calls to their collection server.

Adversarial browser extension injection

Extensions with broad tabs and scripting permissions can patch window.fetch, override React's synthetic event system, or redefine Element.prototype.addEventListener before your app boots. A compromised extension installed by a single engineer on a shared staging environment has compromised every credential entered on that machine.

Supply-chain poisoning via npm

The 2025 polyfill.io successor incidents demonstrated that a single transitive dependency update can silently inject credential-harvesting code into millions of React bundles. Attackers now specifically target packages that process form inputs, validation libraries, date-pickers, rich-text editors, because they sit closest to sensitive user data.

Adversarial session token exfiltration

Modern phishing no longer needs your password. Stealing a valid session cookie or an OAuth access token is sufficient. Techniques include MITM via malicious service workers, cross-origin XHR abuse through misconfigured CORS, and postMessage eavesdropping on misconfigured micro-frontend shells.

Zero-trust principles for the frontend

Zero-trust in a React context means applying three core axioms continuously:

Never trust implicitly. Treat every script, every iframe, every API response, and every user input as potentially hostile, regardless of origin.
Verify explicitly. Authenticate and validate at every boundary: script load, DOM mutation, API call, and user interaction.
Use least-privilege access. Grant capabilities only for the duration and scope needed. Revoke eagerly.

Practical React patterns

1. Content Security Policy with strict-dynamic

A well-crafted CSP is your first and strongest defence. The strict-dynamic keyword removes the need for domain whitelists (which grow stale and get abused) by trusting only scripts explicitly granted a nonce by your server.

Content-Security-Policy:
  default-src 'none';
  script-src 'nonce-{SERVER_GENERATED_NONCE}' 'strict-dynamic';
  style-src 'nonce-{SERVER_GENERATED_NONCE}';
  connect-src https://api.yourdomain.com;
  img-src 'self' data:;
  frame-ancestors 'none';
  upgrade-insecure-requests;
  report-uri /csp-violations;

For React apps using webpack or Vite, the csp-html-webpack-plugin and vite-plugin-csp packages inject nonces automatically at build time. The critical discipline is never using 'unsafe-inline', inline event handlers and style attributes should be refactored into external files or CSS Modules.

2. Subresource Integrity for every external asset

Any third-party script loaded without SRI (Subresource Integrity) is an unconditional trust grant to that CDN. One compromised CDN edge node silently delivers malicious JavaScript to every visitor.

<script
  src="https://cdn.example.com/analytics.js"
  integrity="sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC"
  crossorigin="anonymous"
></script>

In your CI pipeline, generate and pin SRI hashes for all third-party resources at build time. Rotate them on every dependency update. Fail the build if any external resource lacks an integrity attribute.

3. Trusted Types API

The Trusted Types API, now baseline across all major browsers, eliminates DOM XSS sinks at the platform level. Enabling it means no string can be assigned to innerHTML, outerHTML, document.write, or script src without first passing through a registered policy.

// vite.config.js / webpack config — add to CSP header
// require-trusted-types-for 'script'

// In your React app bootstrap
if (window.trustedTypes && window.trustedTypes.createPolicy) {
  const sanitizePolicy = window.trustedTypes.createPolicy('sanitize-html', {
    createHTML: (input) => DOMPurify.sanitize(input, { RETURN_TRUSTED_TYPE: true }),
    createScriptURL: (url) => {
      const allowed = ['https://api.yourdomain.com'];
      if (!allowed.some(origin => url.startsWith(origin))) {
        throw new TypeError(`Blocked script URL: ${url}`);
      }
      return url;
    }
  });
}

With Trusted Types enforced, injecting arbitrary HTML via a compromised third-party library throws a TypeError rather than silently executing. Combined with DOMPurify, this eliminates the entire class of XSS-via-React-dangerouslySetInnerHTML vulnerabilities.

4. Cookie and token hardening

HTTP-only, SameSite=Strict, Secure cookies are table stakes in 2026. What separates resilient apps is token binding and short-lived access tokens with silent refresh.

// Short-lived tokens: 5-minute access token, 24-hour refresh token
// Silent refresh in a sandboxed Web Worker — keeps tokens out of the main thread

// auth.worker.js
self.addEventListener('message', async (event) => {
  if (event.data.type === 'REFRESH_TOKEN') {
    const response = await fetch('/api/auth/refresh', {
      method: 'POST',
      credentials: 'include', // HttpOnly refresh cookie handled by browser
    });
    const { accessToken, expiresIn } = await response.json();
    self.postMessage({ type: 'TOKEN_REFRESHED', accessToken, expiresIn });
  }
});

Moving token refresh logic into a dedicated Web Worker isolates it from the main-thread JavaScript context that extension scripts can patch. The access token is never stored in localStorage, it lives in memory inside the Worker and is passed to fetch interceptors as needed.

5. Integrity checks on your own bundle

Supply-chain attacks often manifest as subtle mutations to your own production bundle, a minified analytics script with an extra fetch() call. Implement a runtime integrity monitor that hashes critical module exports on boot and compares them to known-good values baked in at build time.

// build-time: generate hashes of critical exports
// runtime: verify on app boot

async function verifyBundleIntegrity(expectedHashes) {
  for (const [moduleId, expectedHash] of Object.entries(expectedHashes)) {
    const moduleSource = getModuleSource(moduleId); // your module registry
    const buffer = await crypto.subtle.digest(
      'SHA-256',
      new TextEncoder().encode(moduleSource)
    );
    const actualHash = btoa(String.fromCharCode(...new Uint8Array(buffer)));
    if (actualHash !== expectedHash) {
      reportViolation({ moduleId, expectedHash, actualHash });
      throw new Error(`Bundle integrity violation: ${moduleId}`);
    }
  }
}

6. Sandboxed iframes for third-party widgets

Any third-party widget, chat bubbles, payment forms, social embeds, should be isolated in a maximally sandboxed iframe. The allow attribute now supports a fine-grained Permission Policy that can revoke access to the camera, clipboard, and geolocation APIs independently.

<iframe
  src="https://widget.trusted-partner.com/chat"
  sandbox="allow-scripts allow-same-origin allow-forms"
  allow="camera 'none'; microphone 'none'; clipboard-read 'none'; geolocation 'none'"
  referrerpolicy="no-referrer"
  loading="lazy"
></iframe>

Never grant allow-top-navigation to a third-party iframe, it is sufficient for a compromised widget to redirect the parent window to a phishing page.

7. Runtime anomaly detection

Ship a lightweight, privacy-preserving behavioural monitor that detects tell-tale phishing indicators at runtime:

class FrontendAnomalyDetector {
  constructor() {
    this.baseline = this.captureBaseline();
    this.monitor();
  }

  captureBaseline() {
    return {
      fetchDescriptor: Object.getOwnPropertyDescriptor(window, 'fetch'),
      addEventListenerDescriptor: Object.getOwnPropertyDescriptor(
        EventTarget.prototype, 'addEventListener'
      ),
      documentReferrer: document.referrer,
    };
  }

  monitor() {
    // Detect fetch monkey-patching (common in extension-based MitM)
    const currentFetch = Object.getOwnPropertyDescriptor(window, 'fetch');
    if (currentFetch?.value !== this.baseline.fetchDescriptor?.value) {
      this.report('fetch_tampered');
    }

    // Detect unexpected referrers (clone-site navigation)
    const trustedReferrers = ['https://yourdomain.com', ''];
    if (!trustedReferrers.some(r => document.referrer.startsWith(r))) {
      this.report('suspicious_referrer', { referrer: document.referrer });
    }
  }

  report(type, meta = {}) {
    navigator.sendBeacon('/api/security/report', JSON.stringify({ type, meta, ts: Date.now() }));
  }
}

Zero-trust React: implementation checklist

Use this checklist as a PR gate for any authentication or payment-adjacent feature:

Transport and headers

[x] Content-Security-Policy with strict-dynamic and per-request nonces, no unsafe-inline
[x] Strict-Transport-Security with includeSubDomains and preload
[x] X-Frame-Options: DENY and frame-ancestors 'none' in CSP
[x] Permissions-Policy disabling unused browser APIs

Scripts and assets

[x] SRI hashes on every third-party <script> and <link>
[x] require-trusted-types-for 'script' enforced in CSP
[x] DOMPurify with RETURN_TRUSTED_TYPE on all HTML rendering paths
[x] No dangerouslySetInnerHTML without an explicit Trusted Types policy

Authentication

[x] Access tokens live in memory (Web Worker), never in localStorage
[x] Refresh tokens in HttpOnly, SameSite=Strict, Secure cookies only
[x] Access token lifetime ≤ 5 minutes with silent refresh
[x] PKCE enforced for all OAuth flows

Third-party and supply chain

[x] All npm dependencies pinned to exact versions with package-lock.json integrity
[x] Automated dependency audit in CI (npm audit --audit-level=high)
[x] Third-party widgets isolated in maximally sandboxed iframes
[x] Runtime monkey-patch detection for fetch and addEventListener

Monitoring

[x] CSP violation reports collected and alerted on in real time
[x] Behavioural anomaly detector deployed and reporting to SIEM
[x] Bundle integrity hashes verified on every boot

Conclusion

The 2026 phishing playbook is sophisticated, automated, and specifically engineered to exploit the trust assumptions baked into most React applications. Defending against it requires treating the browser as an untrusted execution environment, because, from a zero-trust perspective, it is.

None of the patterns above require exotic infrastructure. CSP nonces, Trusted Types, and Web Worker token isolation are all production-ready today. The real investment is discipline: making these controls non-negotiable in code review, enforcing them in CI, and instrumenting violations so your security team learns from near-misses before they become incidents.

Apply zero-trust at every layer, network, API, and now frontend, and you transform the browser from your weakest link into an active participant in your defence.

Have a question or a pattern to add? Drop it in the comments below.

SLOs, SLAs, and Error Budgets: A Backend Developer's Guide to Reliability Engineering

ThankGod Chibugwum Obobo — Sun, 31 May 2026 17:04:28 +0000

Reliability engineering used to be the exclusive domain of Site Reliability Engineers and infrastructure teams. But as backend developers take on more ownership of the services they build, from deployment to on-call, understanding Service Level Objectives (SLOs), Service Level Agreements (SLAs), and error budgets has become an essential skill, not an optional one.

These concepts are not bureaucratic paperwork. They are the engineering framework that answers some of the most important questions your team faces: How reliable does this service actually need to be? How do we know when we're spending too much engineering effort on reliability versus new features? And when something breaks, how do we decide whether to drop everything and fix it or continue shipping?

This guide breaks down SLOs, SLAs, and error budgets from first principles, and shows you how to define, implement, and operationalize them in a real backend system.

The Reliability Vocabulary: SLI, SLO, and SLA

These three terms are closely related but play distinct roles. Understanding the difference is the starting point for everything else.

Service Level Indicator (SLI)

An SLI is a quantitative measurement of a specific aspect of your service's behavior. It is the raw signal, the metric you actually measure.

Common SLIs for backend services:
| SLI Type | Example Measurement |
|----------|-------------------|
| Availability | Percentage of HTTP requests returning non-5xx responses |
| Latency | Percentage of requests completing in under 300ms |
| Error Rate | Percentage of requests resulting in application errors |
| Throughput | Number of successful requests processed per second |
| Freshness | Percentage of data reads returning results updated within 60 seconds |

SLIs must be measurable, meaningful to the user, and directly observable from your infrastructure. "The service feels fast" is not an SLI. "95% of requests complete in under 200ms" is.

Service Level Objective (SLO)

An SLO is a target for your SLI, the threshold your service commits to meeting over a defined time window.

SLO = SLI + Target + Time Window

Example: 99.5% of HTTP requests return a non-5xx response measured over a rolling 28-day window

SLOs are internal commitments, they exist within your engineering organization to guide prioritization and define what "good enough" looks like. They are not customer-facing contracts.

Service Level Agreement (SLA)

An SLA is a formal, contractual commitment made to external customers or stakeholders, typically with financial consequences for violation (refunds, service credits, contract termination).

SLA = SLO with commercial consequences

Example: We guarantee 99.9% uptime per calendar month. If we fall below 99.9%, customers receive a 25% service credit.

The critical rule: your SLA must always be weaker than your SLO. If your SLO is 99.5% availability and your SLA promises 99.9%, you have no buffer, the moment you breach your internal target, you've already violated your customer contract. Set your SLA conservatively below your SLO to create an operational safety margin.

Choosing the Right SLIs

The most common mistake in SLO design is measuring the wrong things. SLIs should reflect user experience, not system internals.

User-facing SLIs (what users actually experience):

Request success rate (non-error responses)
Request latency at the p95 or p99 percentile
Data freshness for read-heavy services
End-to-end transaction completion rate

Avoid these as primary SLIs:

CPU utilization
Memory usage
Disk I/O

These are useful for capacity planning and root cause analysis, but they don't directly measure user experience. High CPU doesn't always mean users are suffering. High error rate always does.

For most backend HTTP services, start with two SLIs:

SLI 1 (Availability): Proportion of successful requests = (requests with status < 500) / (total requests)

SLI 2 (Latency): Proportion of requests meeting the latency target = (requests completing in < 300ms) / (total requests)

Setting Realistic SLO Targets

Setting targets too high creates unnecessary toil. Setting them too low fails your users. The right target is derived from your historical data, not aspirational thinking.

The Four-Step SLO Target Process

Step 1 - Measure your current performance. Query your metrics backend for your chosen SLI over the last 90 days. This is your baseline.

# Current 28-day availability SLI
(
  sum(rate(http_requests_total{status!~"5.."}[28d]))
  /
  sum(rate(http_requests_total[28d]))
) * 100

Step 2 - Identify your natural performance floor. Look at your worst performing week in the last 90 days. Your SLO target should be below your average performance but above your worst week's performance.

Step 3 - Apply the reliability tax. Subtract a buffer for planned maintenance, deployments, and anticipated incidents. A service running at 99.7% average availability might set an SLO of 99.5% to account for this variance.

Step 4 - Validate against user expectations. Does the target actually reflect an acceptable user experience? A 99.0% availability SLO means users encounter roughly 7 hours of downtime per month, acceptable for an internal admin tool, unacceptable for a payment API.

Error Budgets: Turning Reliability Into a Product Decision

An error budget is the inverse of your SLO, it defines how much unreliability your service is allowed to have within a given window.

Error Budget = 1 - SLO Target

99.5% availability SLO over 28 days:
Error Budget = 0.5% of requests
= 0.005 × total requests in 28 days

The error budget is what transforms SLOs from a compliance metric into an engineering tool. It answers the question: how much reliability risk can we afford to take right now?

How Error Budgets Drive Engineering Decisions

When you have plenty of error budget remaining, you can afford to:

Ship risky features and new deployments
Run load tests and chaos experiments in production
Prioritize feature velocity over reliability work

When your error budget is nearly exhausted, the calculus changes:

Slow down or halt risky deployments
Prioritize reliability and bug fixes over new features
Conduct a postmortem to understand what consumed the budget
Implement additional safeguards before resuming normal velocity

This is the core value of error budgets: they make reliability a shared engineering responsibility, not just an SRE concern, because they directly constrain the team's ability to ship features.

Implementing SLO Tracking in Grafana and Prometheus

Step 1 - Define Recording Rules

Recording rules pre-compute SLI metrics at query time, making SLO dashboards fast and burn rate calculations efficient:

# prometheus/rules/slo-rules.yaml
groups:
  - name: slo_recording_rules
    rules:
      # Availability SLI — 5-minute window
      - record: slo:http_availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status!~"5..",service="orders-service"}[5m]))
          /
          sum(rate(http_requests_total{service="orders-service"}[5m]))

      # Availability SLI — 1-hour window
      - record: slo:http_availability:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{status!~"5..",service="orders-service"}[1h]))
          /
          sum(rate(http_requests_total{service="orders-service"}[1h]))

      # Latency SLI — proportion of requests under 300ms
      - record: slo:http_latency_300ms:ratio_rate5m
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.3",service="orders-service"}[5m]))
          /
          sum(rate(http_request_duration_seconds_count{service="orders-service"}[5m]))

Step 2 - Calculate Error Budget Consumption

Track remaining error budget as a percentage of the total allowed:

# Remaining availability error budget (28-day window)
# SLO target: 99.5% → error budget: 0.5%
(
  sum(rate(http_requests_total{status=~"5..",service="orders-service"}[28d]))
  /
  sum(rate(http_requests_total{service="orders-service"}[28d]))
)
/
0.005  # error budget = 1 - 0.995

A value of 0.3 means 30% of the error budget has been consumed. A value above 1.0 means the budget is exhausted, the SLO is being violated.

Step 3 - Burn Rate Alerts

Burn rate alerting is more sophisticated than simple threshold alerting, it detects how fast you're consuming your error budget, allowing you to catch slow burns before the budget is exhausted.

# Fast burn rate alert, consuming budget 14x faster than sustainable
# If sustained, will exhaust 28-day budget in 48 hours
(
  1 - slo:http_availability:ratio_rate1h{service="orders-service"}
)
/
(1 - 0.995)  # SLO target
> 14.4

The burn rate multiplier of 14.4 means the service is failing at 14.4 times the sustainable rate for a 99.5% SLO. At this rate, the entire 28-day error budget would be consumed in approximately 48 hours, warranting an immediate page.

Pair fast burn rate alerts (short windows, high multipliers) with slow burn rate alerts (long windows, lower multipliers) for complete coverage:

Alert	Window	Burn Rate	Budget Consumed	Severity
Fast burn	1h	> 14.4x	~2% in 1h	Critical
Fast burn	6h	> 6x	~5% in 6h	High
Slow burn	3d	> 3x	~10% in 3d	Medium
Slow burn	3d	> 1x	Budget at risk	Low

The Error Budget Policy

The most important artifact in your SLO implementation is the error budget policy, a written document that defines what happens when budget thresholds are crossed. Without it, the error budget is just a number.

A minimal error budget policy covers:

Error Budget Policy - orders-service

SLO: 99.5% availability over a rolling 28-day window
Error Budget: 0.5% (approximately 21.6 minutes equivalent downtime)

When budget consumption reaches 25%:

Review recent deployments for contributing factors
Ensure on-call runbooks are up to date

When budget consumption reaches 50%:

Engineering lead notified
Reliability review scheduled within 1 week
No new risky feature deployments without explicit approval

When budget consumption reaches 75%:

Deployment freeze on non-critical changes
Engineering team shifts focus to reliability improvements
Daily sync between engineering lead and on-call

When budget is exhausted (100%):

Full deployment freeze
Incident postmortem required before resuming feature work
SLO target reviewed, may need adjustment

The policy transforms the error budget from a metric into a decision-making framework that the entire team, engineering, product, and leadership, can align around.

Common SLO Pitfalls to Avoid

Too many SLOs. Start with one or two SLIs per service. Ten SLOs per service are impossible to operationalize and dilute focus. Add more only when you have proven the process works.

SLOs without ownership. An SLO with no named owner and no budget policy is decoration. Every SLO needs a team or individual responsible for it.

Ignoring dependencies. If your service calls five downstream services, your availability is bounded by theirs. Account for dependency reliability in your SLO targets, or you'll constantly violate your SLO due to factors outside your control.

Chasing 100% reliability. The marginal cost of reliability increases exponentially as you approach 100%. Going from 99.9% to 99.99% availability is dramatically more expensive than going from 99.0% to 99.9%. Always ask: does the user experience actually require this level of reliability?

Conclusion

SLOs, SLAs, and error budgets are not compliance bureaucracy, they are an engineering framework for making deliberate trade-offs between reliability and velocity. When implemented correctly, they give your team a shared, quantitative language for discussing risk, a principled basis for prioritization decisions, and a feedback loop that continuously improves the reliability of the systems you build.

Start simple: pick two SLIs for your most critical service, set targets based on historical data, implement error budget tracking in Grafana, and write a one-page budget policy. Run it for a quarter, observe how it changes your team's conversations, and expand from there.

Reliability is not a feature you ship once. It is an ongoing engineering practice, and error budgets are how you keep score.

Already using Google's SRE Workbook methodology or OpenSLO for cross-platform SLO definitions? Drop your approach in the comments.

Alerting That Doesn't Cry Wolf: How to Design Meaningful Thresholds in Grafana

ThankGod Chibugwum Obobo — Sun, 24 May 2026 17:00:00 +0000

An alert that fires too often stops being an alert. When on-call engineers are conditioned to dismiss notifications because most of them resolve on their own, the one real incident that demands immediate attention gets lost in the noise. This is alert fatigue, one of the most common and costly problems in production engineering.

The antidote is not fewer alerts. It's better alerts, thresholds designed around the behavior of your actual system, routed to the right people, at the right time, with enough context to act immediately.

Grafana Alerting provides a flexible, powerful platform for building exactly this. But the tooling is only as good as the threshold design behind it. This guide covers how to think about alert thresholds, how to implement them correctly in Grafana, and how to build an alerting strategy that your on-call team will trust, not mute.

Why Most Alerting Systems Fail

Before designing better thresholds, it helps to understand the failure modes of typical alerting setups:
Static thresholds on dynamic systems. Setting CPU > 80% as a hard alert ignores the fact that your system's normal baseline changes across time of day, day of week, and traffic patterns. An 80% CPU reading at 3am during off-peak hours is a crisis. The same reading at noon on Black Friday is expected.

Alerting on symptoms instead of impact. High memory usage, elevated CPU, and increased error rates are symptoms. The question your alert should answer is: are users being impacted right now? An alert on HTTP 500 rate > 1% is more actionable than an alert on heap memory > 70%.

Missing evaluation windows. A single spike that recovers in 10 seconds should not page anyone. Alerts that fire on instantaneous values rather than sustained conditions generate constant noise for transient anomalies.

No severity differentiation. Treating every alert with the same urgency trains engineers to treat all alerts as low urgency. A P1 production outage and a P4 warning about disk space should trigger completely different responses.

The Four Properties of a Meaningful Alert

Every alert in your system should satisfy all four of these properties before it pages anyone:
Actionable - the person receiving the alert knows what to do. If the response to an alert is "check the dashboard and see if it's a real problem," the alert is not actionable. Every alert should link directly to a runbook.

Accurate - the alert fires when something is genuinely wrong, not when the system is behaving within acceptable variance. False positives are as damaging as missing real incidents.

Timely - the alert fires early enough to allow intervention before users are significantly impacted, but not so sensitive that it fires on noise.

Contextual - the alert carries enough information to begin diagnosis without opening five dashboards. Service name, environment, current value, threshold, and a runbook link should be included in every notification.

Step 1 - Establish Baselines Before Setting Thresholds

The most common mistake is setting thresholds without data. Before writing a single alert rule, spend time understanding your system's normal behavior:

What is the p50, p95, and p99 latency for your critical endpoints on a normal day?
What does error rate look like across different times of day and days of week?
What is the normal range for CPU, memory, and connection pool utilization under typical load?

In Grafana, use the Explore view with a 30-day time range to visualize historical metric distributions. Look for:

# p95 latency over the last 30 days, understand your normal range
histogram_quantile(0.95,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

If your p95 latency normally fluctuates between 120ms and 280ms, an alert threshold of > 500ms sustained for 5 minutes is meaningful. A threshold of > 150ms will fire constantly during normal operation.

Step 2 - Use Rate-Based Thresholds, Not Absolute Values

Absolute value thresholds (error count > 50) are fragile, they don't account for traffic volume. A system handling 10,000 requests per minute with 50 errors (0.05% error rate) is healthy. A system handling 100 requests per minute with 50 errors (50% error rate) is on fire.

Always alert on rates and ratios, not raw counts:

# Error rate as a percentage of total requests, traffic-aware
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

This expression returns the percentage of requests resulting in 5xx errors over the last 5 minutes, meaningful regardless of whether your system is handling 100 or 100,000 requests per minute.

Similarly for latency, alert on percentiles rather than averages:

# p99 latency, catches tail latency issues that averages hide
histogram_quantile(0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{service="orders-service"}[5m])
  )
)

Averages mask outliers. A p99 of 8 seconds with an average of 200ms means 1% of your users are waiting 8 seconds, and the average would never alert on it.

Step 3 - Configure Evaluation Windows and Pending Periods

Grafana Alerting evaluates rules on a configurable interval and supports a pending period, the duration a condition must be continuously true before the alert fires. This single setting eliminates the majority of false positives from transient spikes:

# Grafana Alert Rule configuration
evaluateEvery: 1m          # evaluate the query every minute
for: 5m                    # condition must hold for 5 consecutive minutes before firing

Guidelines for pending periods by alert severity:
| Severity | Pending Period | Rationale |
|----------|---------------|-----------|
| P1 - Critical | 1–2 minutes | Production down, act fast, accept occasional false positives |
| P2 - High | 3–5 minutes | Significant degradation, needs confirmation before paging |
| P3 - Medium | 10–15 minutes | Trend-based concern, sustained issue, not a spike |
| P4 - Low | 30–60 minutes | Capacity planning signal, no urgency |

A 5-minute pending period on a P2 alert means the condition must be true for five consecutive 1-minute evaluations. A 30-second CPU spike at minute 3 resets the clock, preventing the alert from firing on transient load.

Step 4 - Implement Multi-Condition Alerts

Some failure modes only become meaningful when multiple conditions are true simultaneously. Grafana supports multi-condition alert rules that combine signals for higher-precision alerting:

# Condition A - error rate elevated
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) > 0.02

# Condition B - latency degraded simultaneously
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
) > 1.0

Requiring both conditions to be true simultaneously, elevated error rate AND elevated latency, dramatically reduces false positives compared to alerting on either signal alone. A brief error rate spike during a deployment might not coincide with latency degradation; genuine service degradation almost always shows both.

Step 5 - Define Alert Severity and Routing

Not all alerts should wake someone up at 3am. Define a clear severity taxonomy and route accordingly:

# Grafana Alerting - labels for routing
labels:
  severity: critical   # critical | high | medium | low
  team: backend        # backend | frontend | platform | data
  service: orders-service
  environment: production

Map severities to notification channels in your Contact Points and Notification Policies:

Severity	Notification Channel	Response Expectation
Critical	PagerDuty (immediate page)	Respond within 5 minutes
High	PagerDuty (page after 10 min)	Respond within 30 minutes
Medium	Slack `#alerts-medium`	Review within business hours
Low	Slack `#alerts-low`	Review in weekly ops meeting

Grafana's Notification Policy tree allows routing based on label matchers, so severity=critical AND environment=production pages on-call, while severity=low posts silently to a Slack channel that gets reviewed on Monday.

Step 6 - Write Alerts That Explain Themselves

An alert notification that says "High Error Rate" forces the on-call engineer to open dashboards before they can even begin to understand the situation. Grafana's alert annotations and templated messages fix this:

{{ define "alert-summary" }}
{{ .Labels.severity | toUpper }} — {{ .Labels.service }}
Environment: {{ .Labels.environment }}

Condition: Error rate has exceeded 2% for 5 consecutive minutes
Current Value: {{ $value | printf "%.2f" }}%
Threshold: 2.00%

Service: {{ .Labels.service }}
Started: {{ .StartsAt | date "2006-01-02 15:04:05 UTC" }}

Runbook: https://runbooks.internal/{{ .Labels.service }}/high-error-rate
Dashboard: https://grafana.internal/d/service-overview?var-service={{ .Labels.service }}
{{ end }}

Every alert notification should answer:

What is wrong (metric name and current value)
How wrong it is (comparison to threshold)
How long it has been wrong
Where to look next (dashboard link)
What to do (runbook link)

Step 7 - Maintain Alerts as Code with Grafana Provisioning

Alerts defined manually in the Grafana UI are fragile, they live only in the database, can't be code-reviewed, and can't be rolled back. Use Grafana's provisioning system to manage alerts as version-controlled YAML:

# provisioning/alerting/orders-service.yaml
apiVersion: 1
groups:
  - name: orders-service
    folder: Backend Services
    interval: 1m
    rules:
      - uid: orders-error-rate-p2
        title: "orders-service — High Error Rate"
        condition: C
        for: 5m
        labels:
          severity: high
          team: backend
          service: orders-service
          environment: production
        annotations:
          summary: "Error rate above 2% for 5 minutes"
          runbook_url: "https://runbooks.internal/orders-service/high-error-rate"
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: |
                (
                  sum(rate(http_requests_total{service="orders-service",status=~"5.."}[5m]))
                  /
                  sum(rate(http_requests_total{service="orders-service"}[5m]))
                ) * 100
          - refId: C
            datasourceUid: "__expr__"
            model:
              type: threshold
              conditions:
                - evaluator:
                    type: gt
                    params: [2.0]
                  query:
                    params: [A]

Store alert provisioning files in your infrastructure repository, apply them via Helm or Terraform, and treat every alert change as a pull request requiring review.

Step 8 - Audit and Prune Regularly

An alerting system accumulates debt over time. Schedule a monthly or quarterly alert audit:

Identify never-firing alerts. An alert that hasn't fired in 90 days is either covering a scenario that never happens or has a threshold set so high it would never catch real issues.
Identify always-firing alerts. Any alert with a sustained firing rate above 20% is background noise, raise the threshold or add a pending period.
Review resolved-without-action alerts. If alerts are regularly acknowledged and resolved without any remediation, the threshold is too sensitive.
Retire obsolete alerts. Services get deprecated, features get removed. Orphaned alert rules are noise that erodes trust in the system.

Track alert quality metrics, false positive rate, mean time to acknowledgement, and alert-to-incident conversion rate, and treat them as engineering KPIs alongside uptime and latency.

Conclusion

Alerting is not a set-and-forget configuration, it's an ongoing engineering discipline. The difference between an alerting system your team trusts and one they mute is threshold design: understanding your system's baseline, alerting on rates rather than absolutes, requiring sustained conditions before paging, differentiating severity, and ensuring every notification carries enough context to act immediately.

Start with your three most critical services. Establish baselines from 30 days of historical data, set rate-based thresholds with appropriate pending periods, define severity routing, and write runbooks before the alerts go live. An alert without a runbook is a fire alarm without an evacuation plan.

Build alert quality into your engineering culture, review and prune regularly, track false positive rates, and treat every unnecessary 3am page as a bug worth fixing.

Distributed Tracing in NestJS: End-to-End Request Visibility with OpenTelemetry

ThankGod Chibugwum Obobo — Sun, 17 May 2026 17:00:00 +0000

In a monolithic application, debugging a slow or failing request is straightforward, you have one codebase, one log stream, and one execution context to reason about. In a microservices architecture, a single user request can touch a dozen services, three databases, and two external APIs before a response is returned. When something goes wrong, where do you look?

This is the problem distributed tracing solves. By attaching a unique trace identifier to every request and propagating it across every service boundary, distributed tracing gives you a complete, chronological map of exactly what happened, which services were called, in what order, how long each took, and where failures occurred.

OpenTelemetry is the open-source observability standard that makes this possible across any language and infrastructure. And NestJS, with its modular architecture and middleware system, is exceptionally well-suited for clean OpenTelemetry integration.

This guide walks through setting up distributed tracing in NestJS from scratch, auto-instrumenting HTTP and database calls, creating custom spans for business logic, propagating context across service boundaries, and visualizing traces in Grafana Tempo.

How Distributed Tracing Works

Before writing code, it helps to understand the core concepts:

Trace: the complete journey of a single request across all services. Identified by a traceId.
Span: a single unit of work within a trace (e.g., an HTTP handler, a database query, an external API call). Each span has a start time, duration, and status.
Context Propagation: the mechanism by which trace and span identifiers are passed between services, typically via HTTP headers (traceparent).
Exporter: the component that sends collected spans to a backend (Tempo, Jaeger, Zipkin, Datadog).

A fully traced request looks like this:

Trace: usr_checkout_8f3a2c
├── [0ms]   API Gateway          → POST /checkout            (12ms)
├── [12ms]  orders-service       → createOrder()             (45ms)
│   ├── [14ms]  PostgreSQL       → INSERT orders             (18ms)
│   └── [33ms]  payments-service → chargeCard()              (24ms)
│       └── [35ms]  Stripe API   → POST /charges             (21ms)
└── [57ms]  notifications-service → sendConfirmation()       (8ms)

Every box is a span. Every span shares the same traceId. The entire tree is the trace.

Step 1 - Install OpenTelemetry Packages

NestJS's OpenTelemetry setup requires the core SDK plus instrumentation libraries for Node.js HTTP, Express (which NestJS runs on), and any databases or HTTP clients your services use:

npm install \
  @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions \
  @opentelemetry/api

Step 2 - Create the Tracing Bootstrap File

OpenTelemetry must be initialized before any other application code, before NestJS bootstraps, before TypeORM connects, before any modules load. Create a dedicated file for this:

// src/tracing.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION, SEMRESATTRS_DEPLOYMENT_ENVIRONMENT } from '@opentelemetry/semantic-conventions';

const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME ?? 'unknown-service',
    [SEMRESATTRS_SERVICE_VERSION]: process.env.APP_VERSION ?? '1.0.0',
    [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV ?? 'development',
  }),
  traceExporter: exporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },      // PostgreSQL
      '@opentelemetry/instrumentation-redis': { enabled: true },   // Redis
      '@opentelemetry/instrumentation-dns': { enabled: false },    // too noisy
      '@opentelemetry/instrumentation-fs': { enabled: false },     // too noisy
    }),
  ],
});

sdk.start();

process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0));
});

Then import it as the very first line of your entry point:

// src/main.ts
import './tracing';  // ← must be first, before all other imports
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();

With this in place, every inbound HTTP request, outbound HTTP call, and PostgreSQL query is automatically traced, zero additional code required in your controllers or services.

Step 3 - Adding Custom Spans for Business Logic

Auto-instrumentation covers infrastructure-level operations. Custom spans capture business logic that matters to your domain, the operations auto-instrumentation doesn't know about:

// src/orders/orders.service.ts
import { Injectable } from '@nestjs/common';
import { trace, SpanStatusCode, context } from '@opentelemetry/api';

const tracer = trace.getTracer('orders-service');

@Injectable()
export class OrdersService {

  async createOrder(dto: CreateOrderDto): Promise<Order> {
    return tracer.startActiveSpan('orders.createOrder', async (span) => {
      try {
        // Add semantic attributes to the span
        span.setAttributes({
          'order.customerId': dto.customerId,
          'order.itemCount': dto.items.length,
          'order.currency': dto.currency,
        });

        const order = await this.processOrder(dto);

        span.setAttributes({ 'order.id': order.id });
        span.setStatus({ code: SpanStatusCode.OK });

        return order;
      } catch (error) {
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: error instanceof Error ? error.message : 'Unknown error',
        });
        span.recordException(error as Error);
        throw error;
      } finally {
        span.end(); // always end the span
      }
    });
  }

  private async processOrder(dto: CreateOrderDto): Promise<Order> {
    return tracer.startActiveSpan('orders.processOrder', async (span) => {
      try {
        // Nested span — appears as a child in the trace tree
        const inventory = await this.checkInventory(dto.items);
        span.addEvent('inventory.checked', { available: inventory.allAvailable });

        const order = await this.orderRepo.create(dto);
        span.addEvent('order.persisted', { orderId: order.id });

        return order;
      } finally {
        span.end();
      }
    });
  }
}

Key practices for custom spans:

Always call span.end(), an unclosed span leaks memory and never exports. Use try/finally to guarantee it.
Use semantic attribute names. Prefix with your domain (order., user., payment.) for consistent querying.
Record exceptions with span.recordException(), this attaches the full stack trace to the span in Tempo.
Add events for significant moments within a span (span.addEvent()), they appear as timestamped annotations on the span timeline.

Step 4 - Context Propagation Across Services

A trace is only useful if it spans service boundaries. When orders-service calls users-service, the trace context must travel with the request via HTTP headers.

Good news: if both services use OpenTelemetry with HTTP auto-instrumentation enabled, context propagation happens automatically. The outbound HTTP call from orders-service injects traceparent and tracestate headers, and users-service extracts them, linking the spans into the same trace tree.

For manual HTTP clients (e.g., Axios without auto-instrumentation), inject headers explicitly:

import { propagation, context } from '@opentelemetry/api';
import axios from 'axios';

async function callUsersService(userId: string) {
  const headers: Record<string, string> = {};

  // Inject current trace context into headers
  propagation.inject(context.active(), headers);

  const response = await axios.get(`${USERS_SERVICE_URL}/users/${userId}`, {
    headers,
  });

  return response.data;
}

For NestJS microservices using TCP or message brokers (NATS, RabbitMQ, Kafka), inject trace context into the message payload or metadata:

// Injecting context into a NestJS microservice message
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);

this.client.emit('order_placed', {
  ...orderPayload,
  _traceContext: carrier, // carry trace headers in the message
});

On the consumer side, extract and activate the context before processing:

@EventPattern('order_placed')
async handleOrderPlaced(data: OrderPlacedEvent) {
  const parentContext = propagation.extract(context.active(), data._traceContext ?? {});

  await context.with(parentContext, async () => {
    // All spans created here are children of the originating trace
    await this.processOrder(data);
  });
}

Step 5 - Trace-Log Correlation

Distributed tracing and structured logging are most powerful when correlated. By injecting the current traceId and spanId into every log entry, you can jump from a log line directly to the corresponding trace in Tempo, and vice versa.

// src/common/middleware/trace-context.middleware.ts
import { Injectable, NestMiddleware } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';
import { trace, context } from '@opentelemetry/api';
import { PinoLogger } from 'nestjs-pino';

@Injectable()
export class TraceContextMiddleware implements NestMiddleware {
  constructor(private readonly logger: PinoLogger) {}

  use(req: Request, res: Response, next: NextFunction) {
    const span = trace.getActiveSpan();
    const spanContext = span?.spanContext();

    if (spanContext) {
      // Pino's child logger binds these fields to every subsequent log in the request
      this.logger.assign({
        traceId: spanContext.traceId,
        spanId: spanContext.spanId,
        traceFlags: spanContext.traceFlags,
      });
    }

    next();
  }
}

In Grafana, this enables Loki-to-Tempo linking, click a traceId in a Loki log query and jump directly to the full distributed trace in Tempo. This is the observability trifecta, logs, traces, and metrics unified in a single investigation workflow.

Step 6 - Visualizing Traces in Grafana Tempo

Grafana Tempo is the recommended trace backend for teams already on the Grafana stack (and pairs naturally with Grafana Faro for frontend-to-backend tracing, as covered in the previous article).

Send spans to Tempo via the OpenTelemetry Collector:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

Once traces are flowing into Tempo, configure Grafana dashboards to:

Search traces by service name, operation name, duration, and status
View flame graphs showing the time breakdown of every span in a trace
Set alerts on p95/p99 latency for critical operations
Link traces to logs via traceId in Loki datasource configuration

Step 7 - CI/CD and Environment Configuration

Manage OpenTelemetry configuration via environment variables, the standard supported by all OTEL SDKs:

# .env.production
SERVICE_NAME=orders-service
APP_VERSION=2.4.1
NODE_ENV=production
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318/v1/traces
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1   # sample 10% of traces in production

Sampling is critical in production. Tracing 100% of requests at scale generates enormous volume and cost. Use parent-based ratio sampling, if the upstream service sampled the request, all downstream services honor that decision, keeping traces complete while reducing overall volume.

Common Pitfalls to Avoid

Initializing OpenTelemetry after NestJS bootstraps: If tracing.ts is not the first import in main.ts, auto-instrumentation patches won't apply to already-loaded modules. The SDK must load before any instrumented libraries.

Forgetting span.end(): Every span that isn't explicitly ended leaks memory and never exports. Always use try/finally blocks around custom spans.

Over-instrumenting: Not every function needs a custom span. Focus on operations with meaningful duration variance, external calls, database queries, and complex business logic. Instrumenting trivial utility functions adds noise without insight.

Ignoring sampling in production: Tracing 100% of requests is expensive. Configure ratio-based sampling early, retrofitting it later requires changes across every service.

Not propagating context through message brokers: Context propagation via HTTP headers is automatic, but message broker propagation requires manual injection and extraction. Skipping it breaks the trace tree at every async boundary.

Conclusion

Distributed tracing with OpenTelemetry and NestJS transforms debugging in microservices from guesswork into a precise, evidence-based workflow. A single traceId gives you the complete story of any request, every service it touched, every database query it triggered, every external call it made, and exactly where time was spent or errors occurred.

Set up auto-instrumentation first for immediate value with zero code changes, layer in custom spans for your critical business operations, enforce context propagation across all service boundaries, and correlate traces with your structured logs. Combined with Grafana Faro for frontend traces, you achieve the observability holy grail: end-to-end request visibility from the user's click to the database and back.

Running NestJS microservices on Kubernetes? OpenTelemetry's Kubernetes operator can inject the collector as a sidecar automatically, no per-pod configuration required. More on that in a future post.

Frontend Observability with Grafana Faro: Real User Monitoring for Production Web Apps

ThankGod Chibugwum Obobo — Sun, 10 May 2026 17:00:00 +0000

Backend observability is a solved problem for most mature engineering teams. Structured logs, distributed traces, and metrics dashboards are standard practice. But the moment a user opens your web app, you lose visibility. JavaScript exceptions, slow renders, failed API calls, and broken user flows happen entirely in the browser, invisible to your backend monitoring stack.

Frontend observability closes this gap. And Grafana Faro is one of the most capable open-source tools for doing it. Purpose-built to collect Real User Monitoring (RUM) data, JavaScript errors, performance metrics, and custom events directly from the browser, and feed them into the same Grafana stack your backend already uses.

This guide covers what Grafana Faro is, how to set it up in a React application, what to instrument, and how to connect frontend signals to your existing Grafana Cloud dashboards for end-to-end observability.

What Is Frontend Observability?

Frontend observability is the practice of collecting, correlating, and analyzing signals from real users' browsers in production. It covers four primary signal types:

Signal	What It Captures
Errors	JavaScript exceptions, unhandled promise rejections, network errors
Performance	Core Web Vitals, page load times, resource timing, long tasks
Traces	User session flows, distributed traces linking frontend calls to backend spans
Logs	Custom application events, user interactions, feature flag evaluations

Traditional Application Performance Monitoring (APM) tools cover backend services. Real User Monitoring (RUM) covers the actual experience of real users on real devices and networks, capturing variance that synthetic testing in CI simply cannot reproduce.

What Is Grafana Faro?

Grafana Faro is an open-source Web SDK and collector designed specifically for frontend observability. Released by Grafana Labs, it integrates natively with the broader Grafana observability stack:

Grafana Faro Web SDK - the browser-side agent that collects signals
Grafana Agent / Faro Collector - receives signals from the browser and forwards them to Grafana Cloud
Grafana Cloud - stores and visualizes the data in Loki (logs), Tempo (traces), and Prometheus (metrics)

The key advantage over standalone RUM tools like Sentry or Datadog RUM is unified observability, a single frontend error in Faro can be correlated with the corresponding backend trace in Tempo, giving you a complete picture of what the user experienced and why.

Step 1 - Installation and Basic Setup

Install the Faro Web SDK:

npm install @grafana/faro-web-sdk @grafana/faro-web-tracing

Initialize Faro as early as possible in your application, before any other imports, to capture errors that occur during startup:

// src/instrumentation.ts  ← import this first in main.tsx/index.tsx
import { initializeFaro, getWebInstrumentations } from '@grafana/faro-web-sdk';
import { TracingInstrumentation } from '@grafana/faro-web-tracing';

export const faro = initializeFaro({
  url: process.env.VITE_FARO_COLLECTOR_URL!, // your Grafana Agent endpoint
  app: {
    name: 'my-web-app',
    version: process.env.VITE_APP_VERSION ?? '1.0.0',
    environment: process.env.NODE_ENV,
  },
  instrumentations: [
    ...getWebInstrumentations({
      captureConsole: true,          // capture console.error and console.warn
      captureConsoleDisabledLevels: ['log', 'debug'], // skip noisy levels
    }),
    new TracingInstrumentation(),    // distributed tracing for fetch/XHR
  ],
});

Then in your entry point:

// src/main.tsx
import './instrumentation';         // must be first
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';

ReactDOM.createRoot(document.getElementById('root')!).render(<App />);

With this alone, Faro will automatically collect:

Unhandled JavaScript errors and promise rejections
Browser console errors and warnings
Core Web Vitals (LCP, CLS, INP)
Navigation and resource timing
Distributed trace headers on all fetch and XHR requests

Step 2 - Setting Up the Grafana Agent Collector

The Faro SDK sends signals to a collector endpoint, not directly to Grafana Cloud. The collector receives, batches, and forwards signals to the appropriate backends.

For local development, run the Grafana Agent via Docker:

# docker-compose.yml
services:
  grafana-agent:
    image: grafana/agent:latest
    ports:
      - "12347:12347"   # Faro receiver port
    volumes:
      - ./grafana-agent.yaml:/etc/agent/agent.yaml
    command: -config.file=/etc/agent/agent.yaml

Configure the agent to receive Faro signals and forward to Grafana Cloud:

# grafana-agent.yaml
faro:
  server:
    listen_address: 0.0.0.0:12347
    cors_allowed_origins:
      - "http://localhost:5173"
      - "https://your-production-domain.com"

logs:
  configs:
    - name: faro-logs
      clients:
        - url: https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push
          basic_auth:
            username: YOUR_GRAFANA_CLOUD_USER_ID
            password: YOUR_GRAFANA_CLOUD_API_KEY

traces:
  configs:
    - name: faro-traces
      receivers:
        otlp:
          protocols:
            grpc:
      remote_write:
        - endpoint: tempo-prod-eu-west-0.grafana.net:443
          basic_auth:
            username: YOUR_TEMPO_USER_ID
            password: YOUR_GRAFANA_CLOUD_API_KEY

Store all credentials as environment variables or secrets, never hardcode them in configuration files committed to version control.

Step 3 - Identifying Users and Sessions

Raw error logs are difficult to act on without user context. Attach user identity and session metadata to every Faro signal:

// After user authentication succeeds
import { faro } from './instrumentation';

function onUserAuthenticated(user: { id: string; email: string; plan: string }) {
  faro.api.setUser({
    id: user.id,
    email: user.email,          // only if your privacy policy permits
    attributes: {
      plan: user.plan,
      region: user.region,
    },
  });
}

function onUserLoggedOut() {
  faro.api.resetUser();
}

Faro automatically generates a session ID per browser session. Combined with the user ID, you can reconstruct the complete sequence of events that led to an error, page navigations, API calls, user interactions, all directly in Grafana.

Step 4 - Custom Events and Manual Instrumentation

Automatic instrumentation captures infrastructure-level signals. Custom events capture business-level signals, the actions that matter to your product:

import { faro } from './instrumentation';

// Track a significant user action
function onCheckoutCompleted(orderId: string, total: number) {
  faro.api.pushEvent('checkout_completed', {
    orderId,
    total: total.toString(),
    currency: 'USD',
  });
}

// Track feature flag evaluations
function onFeatureFlagEvaluated(flagKey: string, value: boolean) {
  faro.api.pushEvent('feature_flag_evaluated', {
    flagKey,
    value: value.toString(),
  });
}

// Push a custom log entry
function onPaymentRetry(attempt: number, reason: string) {
  faro.api.pushLog([`Payment retry attempt ${attempt}: ${reason}`], {
    level: LogLevel.WARN,
    context: { attempt: attempt.toString(), reason },
  });
}

Custom events are queryable in Grafana Loki using LogQL, you can build dashboards showing checkout conversion rates, feature flag adoption, or payment retry frequency directly from frontend signals.

Step 5 - Error Boundary Integration

React's Error Boundaries catch rendering errors that Faro's automatic instrumentation misses. Integrate Faro into your error boundary to capture them with full component context:

// src/components/ErrorBoundary.tsx
import React, { Component, ErrorInfo, ReactNode } from 'react';
import { faro } from '../instrumentation';

interface Props {
  children: ReactNode;
  fallback: ReactNode;
}

interface State {
  hasError: boolean;
}

export class ErrorBoundary extends Component<Props, State> {
  state: State = { hasError: false };

  static getDerivedStateFromError(): State {
    return { hasError: true };
  }

  componentDidCatch(error: Error, info: ErrorInfo) {
    faro.api.pushError(error, {
      type: 'ReactRenderError',
      context: {
        componentStack: info.componentStack ?? '',
      },
    });
  }

  render() {
    if (this.state.hasError) {
      return this.props.fallback;
    }
    return this.props.children;
  }
}

Wrap critical sections of your UI dashboards, checkout flows, data-heavy views, with dedicated error boundaries so rendering failures are captured with component stack context, not just as generic JavaScript errors.

Step 6 - Distributed Tracing: Connecting Frontend to Backend

The most powerful feature of Grafana Faro is distributed tracing. When TracingInstrumentation is enabled, Faro automatically injects OpenTelemetry trace headers (traceparent) into every fetch and XMLHttpRequest call.

If your backend is also instrumented with OpenTelemetry, the frontend and backend spans are linked. A single user action that triggers an API call produces a trace that spans both the browser and the server.

To ensure your backend accepts and propagates these headers, configure CORS to allow traceparent and tracestate:

// NestJS main.ts
app.enableCors({
  origin: process.env.FRONTEND_URL,
  allowedHeaders: ['Content-Type', 'Authorization', 'traceparent', 'tracestate'],
});

In Grafana Tempo, you can then view a single trace that shows the full lifecycle of a user request: the browser fetch initiating at 0ms, the API gateway receiving at 12ms, the database query completing at 45ms, and the response rendering in the browser at 67ms, in one unified view.

Step 7 - Performance Monitoring and Core Web Vitals

Faro automatically collects Core Web Vitals, the metrics Google uses to evaluate page experience:

Metric	Measures	Good Threshold
LCP (Largest Contentful Paint)	Loading performance	≤ 2.5s
INP (Interaction to Next Paint)	Interactivity	≤ 200ms
CLS (Cumulative Layout Shift)	Visual stability	≤ 0.1

These metrics are collected from real users across real devices and network conditions, far more representative than Lighthouse scores in CI. Build Grafana dashboards that segment Web Vitals by:

Device type (mobile vs. desktop)
Geographic region
Page route (which pages are slowest)
App version (did the latest deployment improve or degrade performance?)

Performance regressions caught by real user data are the ones that actually affect your users, not synthetic benchmarks.

Privacy and Data Minimization

Frontend observability collects data from real users, which means privacy is non-negotiable:

Redact PII before it reaches Faro. Never push user email addresses, payment details, or health data as custom event attributes.

Respect consent. Initialize Faro only after the user has accepted analytics cookies if your jurisdiction requires it (GDPR, CCPA).

Configure sampling rates. For high-traffic applications, collecting 100% of traces is unnecessary and costly. Configure Faro to sample a representative percentage:

initializeFaro({
  // ...
  sessionTracking: {
    samplingRate: 0.1, // collect sessions from 10% of users
  },
});

Use data retention policies. Configure Loki and Tempo retention to match your compliance requirements, frontend traces rarely need to be kept longer than 30–90 days.

Conclusion

Backend monitoring tells you when your services are struggling. Frontend observability with Grafana Faro tells you when your users are struggling and in production, those are often very different things. A backend that's perfectly healthy can still deliver a broken experience if a JavaScript error blocks checkout, a slow render degrades a dashboard, or a failed fetch leaves a user staring at a spinner.

By integrating Faro into your React application, connecting it to Grafana Cloud, and correlating frontend traces with backend spans, you achieve true end-to-end observability, a single pane of glass from the user's click to the database query and back.

Start with automatic instrumentation, add user context and custom events for your critical flows, and build dashboards around the metrics that reflect real user experience. Observability is only complete when it includes the user.

Using Vue, Angular, or a meta-framework like Next.js? Grafana Faro's Web SDK is framework-agnostic, the setup differs slightly but the instrumentation API is identical. Drop your stack in the comments.