DEV Community: InstaDevOps

PostgreSQL High Availability: A Practical Guide with Patroni and pgBouncer

InstaDevOps — Sun, 19 Apr 2026 13:46:56 +0000

Introduction

Your database is the last thing that should go down. Yet setting up PostgreSQL for high availability remains one of the most under-documented areas in the DevOps world. Most teams run a single Postgres instance until the day it dies, then scramble to set up replication while their users stare at error pages.

This guide walks through a production-ready PostgreSQL HA setup using the tools that have become the industry standard: Patroni for automated failover, pgBouncer for connection pooling, streaming replication for data redundancy, and pgBackRest for backups. We'll cover the architecture, actual configuration files, failure scenarios, and the operational runbooks you'll need.

If you're running Postgres in production and don't have automated failover, this article is for you.

Architecture Overview

A production HA Postgres setup has four layers:

                    ┌─────────────┐
                    │   HAProxy   │  (Virtual IP / DNS)
                    │  Port 5432  │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
        ┌─────┴─────┐ ┌───┴───┐ ┌─────┴─────┐
        │ pgBouncer  │ │pgBouncer│ │ pgBouncer  │
        │  Node 1    │ │ Node 2 │ │  Node 3    │
        └─────┬──────┘ └───┬───┘ └─────┬──────┘
              │            │            │
        ┌─────┴─────┐ ┌───┴───┐ ┌─────┴─────┐
        │ Patroni +  │ │Patroni│ │ Patroni +  │
        │ Postgres   │ │  + PG  │ │ Postgres   │
        │ (Primary)  │ │(Replica)│ │ (Replica)  │
        └────────────┘ └────────┘ └────────────┘
              │            │            │
        ┌─────────────────────────────────────┐
        │           etcd Cluster (3 nodes)     │
        └─────────────────────────────────────┘

Patroni manages the Postgres instances and handles leader election via a distributed consensus store (etcd, ZooKeeper, or Consul). When the primary fails, Patroni automatically promotes a replica and reconfigures the others to follow the new primary.

pgBouncer sits between your application and Postgres, pooling connections to avoid the expensive fork-per-connection model. Each Postgres node runs its own pgBouncer instance.

HAProxy (or a DNS-based solution) routes traffic to the current primary. Patroni exposes a REST API that HAProxy uses for health checks.

Setting Up Patroni

Install Patroni on each Postgres node:

pip3 install patroni[etcd]

Here's a production Patroni configuration (/etc/patroni/patroni.yml):

scope: postgres-cluster
name: pg-node-1

restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.0.1.10:8008

etcd3:
  hosts: 10.0.1.20:2379,10.0.1.21:2379,10.0.1.22:2379

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576  # 1MB
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        wal_level: replica
        hot_standby: "on"
        max_wal_senders: 10
        max_replication_slots: 10
        wal_log_hints: "on"
        archive_mode: "on"
        archive_command: >
          pgbackrest --stanza=main archive-push %p

  initdb:
    - encoding: UTF8
    - data-checksums

  pg_hba:
    - host replication replicator 10.0.1.0/24 md5
    - host all all 10.0.1.0/24 md5

postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.0.1.10:5432
  data_dir: /var/lib/postgresql/16/main
  bin_dir: /usr/lib/postgresql/16/bin

  authentication:
    superuser:
      username: postgres
      password: "your-secure-password"
    replication:
      username: replicator
      password: "your-replication-password"

  parameters:
    shared_buffers: 4GB
    effective_cache_size: 12GB
    work_mem: 64MB
    maintenance_work_mem: 512MB
    max_connections: 200
    checkpoint_completion_target: 0.9
    wal_buffers: 64MB
    random_page_cost: 1.1  # SSD storage

Key settings to understand:

maximum_lag_on_failover: A replica won't be promoted if it's more than 1MB behind the primary. This prevents data loss but means failover might not happen if all replicas are lagging heavily.
use_pg_rewind: Allows the old primary to rejoin the cluster as a replica after failover without a full base backup. This is critical for fast recovery.
use_slots: Replication slots prevent the primary from removing WAL segments that replicas still need, avoiding replication breakage during network issues.

Start Patroni as a systemd service:

sudo systemctl enable patroni
sudo systemctl start patroni

Check cluster status:

patronictl -c /etc/patroni/patroni.yml list

Output:

+ Cluster: postgres-cluster ----+---------+---------+----+-----------+
| Member    | Host       | Role    | State   | TL | Lag in MB |
+-----------+------------+---------+---------+----+-----------+
| pg-node-1 | 10.0.1.10  | Leader  | running |  1 |           |
| pg-node-2 | 10.0.1.11  | Replica | running |  1 |       0.0 |
| pg-node-3 | 10.0.1.12  | Replica | running |  1 |       0.0 |
+-----------+------------+---------+---------+----+-----------+

Configuring pgBouncer for Connection Pooling

Without connection pooling, each application connection forks a new Postgres backend process (~10MB RAM each). With 500 application connections, that's 5GB just for connection overhead. pgBouncer solves this by multiplexing thousands of application connections over a small pool of actual database connections.

; /etc/pgbouncer/pgbouncer.ini

[databases]
myapp = host=127.0.0.1 port=5432 dbname=myapp

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

; Pool settings
pool_mode = transaction
default_pool_size = 25
min_pool_size = 5
reserve_pool_size = 5
reserve_pool_timeout = 3

; Connection limits
max_client_conn = 1000
max_db_connections = 100

; Timeouts
server_idle_timeout = 300
client_idle_timeout = 0
query_timeout = 60
query_wait_timeout = 30

; Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60

Critical decision: pool_mode. Transaction mode is the right default for most applications - it assigns a server connection for the duration of a transaction, then returns it to the pool. Session mode holds the connection for the entire client session (defeats the purpose of pooling). Statement mode is the most aggressive but breaks multi-statement transactions and prepared statements.

Warning: Transaction pooling breaks SET commands, prepared statements that span transactions, and LISTEN/NOTIFY. If your application uses these, you'll need session pooling for those specific connections.

Backup Strategy with pgBackRest

pgBackRest is the gold standard for Postgres backups. It supports full, differential, and incremental backups with parallel compression and encryption.

; /etc/pgbackrest/pgbackrest.conf

[global]
repo1-type=s3
repo1-s3-bucket=your-pg-backups
repo1-s3-region=us-east-1
repo1-s3-endpoint=s3.amazonaws.com
repo1-path=/backups
repo1-retention-full=4
repo1-retention-diff=14
repo1-cipher-type=aes-256-cbc
repo1-cipher-pass=your-encryption-passphrase

process-max=4
compress-type=zst
compress-level=3

[main]
pg1-path=/var/lib/postgresql/16/main
pg1-port=5432

Create the stanza and run your first backup:

# Initialize the backup repository
pgbackrest --stanza=main stanza-create

# Full backup
pgbackrest --stanza=main --type=full backup

# Differential backup (only changed since last full)
pgbackrest --stanza=main --type=diff backup

# Incremental backup (only changed since last backup of any type)
pgbackrest --stanza=main --type=incr backup

Set up a cron schedule:

# Full backup every Sunday at 1 AM
0 1 * * 0 pgbackrest --stanza=main --type=full backup

# Differential backup every day at 1 AM (except Sunday)
0 1 * * 1-6 pgbackrest --stanza=main --type=diff backup

Point-in-time recovery is where pgBackRest shines. Because Patroni is already archiving WAL segments to pgBackRest, you can restore to any point in time:

pgbackrest --stanza=main --type=time \
  --target="2026-04-09 14:30:00" \
  --target-action=promote restore

Failover Testing and Operational Runbooks

Setting up HA means nothing if you don't test it. Here are the failure scenarios you must validate:

1. Primary node crash:

# Simulate primary failure
sudo systemctl stop patroni  # on the primary node

Expected: Within 30 seconds (TTL), Patroni promotes a replica. HAProxy health checks detect the change and route traffic to the new primary. Application sees a brief connection error, retries, and reconnects.

2. Network partition:

# Isolate primary from etcd
iptables -A OUTPUT -p tcp --dport 2379 -j DROP

Expected: Primary can't reach etcd, Patroni demotes it to read-only. A replica with etcd access gets promoted. When network heals, the old primary uses pg_rewind to rejoin as a replica.

3. Switchover (planned maintenance):

# Graceful switchover to a specific replica
patronictl -c /etc/patroni/patroni.yml switchover \
  --master pg-node-1 --candidate pg-node-2 --force

Expected: Zero or near-zero downtime. The primary finishes in-flight transactions, transfers leadership, and becomes a replica.

4. Full cluster restore from backup:

# Stop all Patroni instances
# On the new primary node:
pgbackrest --stanza=main --type=time \
  --target="2026-04-09 10:00:00" restore

# Start Patroni - it will bootstrap from the restored data
sudo systemctl start patroni

Run these tests quarterly. Document the results. Your future self at 3 AM will thank you.

Monitoring Your HA Cluster

You need visibility into replication lag, connection pool utilization, and failover events. Key metrics to track:

-- Replication lag (run on primary)
SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn,
  pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

-- Connection counts
SELECT state, count(*) FROM pg_stat_activity GROUP BY state;

-- Long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state != 'idle' AND now() - pg_stat_activity.query_start > interval '30 seconds';

For Prometheus monitoring, use postgres_exporter and set up alerts:

# Prometheus alerting rules
groups:
  - name: postgres
    rules:
      - alert: PostgresReplicationLag
        expr: pg_replication_lag_seconds > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replication lag is {{ $value }}s on {{ $labels.instance }}"

      - alert: PostgresConnectionPoolExhausted
        expr: pgbouncer_pools_server_active / pgbouncer_pools_server_max > 0.9
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "pgBouncer pool >90% utilized on {{ $labels.instance }}"

      - alert: PatroniNoLeader
        expr: patroni_master == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "No Patroni leader detected in cluster"

Need Help with Your DevOps?

Building a production-grade PostgreSQL HA cluster takes time and expertise - and maintaining it takes even more. At InstaDevOps, we design, deploy, and manage database infrastructure alongside your entire DevOps stack, starting at $2,999/month.

Book a free 15-minute consultation to discuss your database reliability needs: https://calendly.com/instadevops/15min

Nginx vs Traefik vs Caddy: Which Reverse Proxy Should You Pick?

InstaDevOps — Sat, 18 Apr 2026 13:46:53 +0000

Introduction

Choosing a reverse proxy is one of those decisions that seems trivial until you're debugging TLS certificate renewals at 2 AM or trying to figure out why your service discovery isn't propagating. If you're running microservices, your reverse proxy is the front door to everything - and picking the wrong one can cost you weeks of configuration headaches.

Here, we'll do a hands-on comparison of the three most popular reverse proxies in the modern DevOps stack: Nginx, Traefik, and Caddy. We'll look at configuration complexity, automatic TLS, service discovery, performance characteristics, and when each one makes sense.

No fluff. Just real configs, real trade-offs, and a clear recommendation based on your use case.

The Contenders at a Glance

Before we dive in, here's a quick snapshot:

Feature	Nginx	Traefik	Caddy
Language	C	Go	Go
Config Format	Custom DSL	YAML/TOML/Labels	Caddyfile/JSON
Auto TLS	Via Certbot	Built-in (Let's Encrypt)	Built-in (Let's Encrypt + ZeroSSL)
Service Discovery	Manual/scripted	Native (Docker, K8s, Consul)	Limited (on_demand_tls)
Hot Reload	`nginx -s reload`	Automatic	Automatic
Dashboard	Nginx Plus only	Built-in	Via API
Memory Footprint	~2-5 MB	~30-50 MB	~20-40 MB

Nginx has been the default choice for over a decade. Traefik was built specifically for dynamic microservice environments. Caddy focuses on simplicity and security-by-default.

Configuration Complexity

This is where the differences hit you first. Let's set up a simple reverse proxy with TLS for two services: an API on port 3000 and a frontend on port 8080.

Nginx:

server {
    listen 80;
    server_name api.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Plus you need Certbot installed, cron jobs for renewal, and you'll reload Nginx manually after any change.

Traefik (docker-compose labels):

services:
  traefik:
    image: traefik:v3.1
    command:
      - "--providers.docker=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.email=you@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/acme/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./acme:/acme

  api:
    image: my-api:latest
    labels:
      - "traefik.http.routers.api.rule=Host(`api.example.com`)"
      - "traefik.http.routers.api.tls.certresolver=letsencrypt"
      - "traefik.http.services.api.loadbalancer.server.port=3000"

  frontend:
    image: my-frontend:latest
    labels:
      - "traefik.http.routers.frontend.rule=Host(`app.example.com`)"
      - "traefik.http.routers.frontend.tls.certresolver=letsencrypt"
      - "traefik.http.services.frontend.loadbalancer.server.port=8080"

Caddy (Caddyfile):

api.example.com {
    reverse_proxy localhost:3000
}

app.example.com {
    reverse_proxy localhost:8080
}

That's it. Caddy automatically provisions TLS certificates, redirects HTTP to HTTPS, and enables HTTP/2. Two lines per service.

Automatic TLS and Certificate Management

This is Caddy's killer feature and Traefik's strong suit. With Nginx, you're on your own.

Caddy obtains certificates from Let's Encrypt and ZeroSSL automatically. It handles renewal, OCSP stapling, and even falls back to a secondary CA if the primary is down. You literally do nothing - just put a domain name in the Caddyfile and it works.

Traefik supports Let's Encrypt via ACME with HTTP-01 and DNS-01 challenges. DNS-01 is crucial if you're behind a load balancer or need wildcard certificates. Configuration looks like:

certificatesResolvers:
  letsencrypt:
    acme:
      email: ops@example.com
      storage: /acme/acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"

Nginx requires Certbot or a similar ACME client as a separate process. You need to configure it, set up renewal hooks, and make sure the reload happens after certificate updates. It works, but it's another moving part to maintain.

Verdict: If automatic TLS matters to you (it should), Caddy wins on simplicity and Traefik wins on flexibility. Nginx is manageable but requires more operational overhead.

Service Discovery and Dynamic Configuration

This is where Traefik pulls ahead for container-heavy environments.

Traefik natively watches Docker labels, Kubernetes Ingress resources, Consul catalogs, etcd, and more. When a new container spins up with the right labels, Traefik picks it up immediately - no restart, no reload, no config file change. This is incredibly powerful for dynamic environments where services scale up and down frequently.

Caddy has an API for dynamic configuration but doesn't natively integrate with container orchestrators the way Traefik does. You can use caddy-docker-proxy (community plugin) to get Docker label support, but it's not first-party.

Nginx is static by default. You change a file, you reload. Tools like consul-template or nginx-proxy (jwilder) exist to generate configs dynamically, but they're external tools bolted on.

For Kubernetes specifically, all three have Ingress controller implementations, but Traefik's is the most mature for dynamic environments and ships as the default ingress controller in K3s.

Performance Benchmarks

Raw performance matters less than you think for most workloads, but here are realistic numbers from common benchmarking setups:

Requests per second (simple HTTP proxy, wrk with 100 connections):

Proxy	RPS (HTTP)	RPS (HTTPS)	p99 Latency
Nginx	~45,000	~32,000	2.1 ms
Traefik	~28,000	~22,000	3.8 ms
Caddy	~30,000	~24,000	3.2 ms

Nginx is faster, period. It's written in C with an event-driven architecture that has been optimized for 20+ years. If you're proxying tens of thousands of requests per second and every millisecond counts, Nginx is the right choice.

But for most applications - handling hundreds to low thousands of RPS - the difference between 3ms and 2ms latency at p99 is irrelevant. You'll hit bottlenecks in your application code, database, or network long before the proxy becomes the issue.

Middleware and Advanced Features

All three support common middleware patterns, but the implementation differs.

Rate limiting:

# Nginx
http {
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    server {
        location /api/ {
            limit_req zone=api burst=20 nodelay;
            proxy_pass http://backend;
        }
    }
}

# Traefik (middleware via labels)
labels:
  - "traefik.http.middlewares.ratelimit.ratelimit.average=10"
  - "traefik.http.middlewares.ratelimit.ratelimit.burst=20"
  - "traefik.http.routers.api.middlewares=ratelimit"

# Caddy
api.example.com {
    rate_limit {remote.ip} 10r/s
    reverse_proxy localhost:3000
}

Authentication, compression, headers, CORS - all three handle these well. Traefik has the concept of middleware chains that you can compose and reuse across routers, which is elegant for complex setups. Nginx has the most extensive third-party module ecosystem. Caddy has a growing plugin system but fewer options overall.

When to Use Each

Choose Nginx when:

You need maximum raw performance (high-traffic APIs, CDN edge nodes)
You have a static infrastructure that doesn't change often
Your team already knows Nginx configuration well
You need advanced features like TCP/UDP proxying, custom Lua scripting, or GeoIP
You're running on resource-constrained hardware

Choose Traefik when:

You're running Docker or Kubernetes with frequently changing services
You want automatic service discovery without config file management
You need the built-in dashboard for visibility into routing
You're using multiple orchestration platforms (Docker + Consul, for example)
Your microservices scale up and down dynamically

Choose Caddy when:

You want the simplest possible setup with automatic HTTPS
You're a small team that doesn't want to maintain proxy configuration
You're running a handful of services that don't change often
You want a single binary with no external dependencies
You value security defaults (HTTPS everywhere, modern TLS settings out of the box)

Migration Tips

If you're switching from Nginx to Traefik or Caddy, here's what to watch for:

Upstream health checks - Nginx does passive health checks by default; Traefik and Caddy have active health check options that behave differently. Test failover behavior.
Websocket proxying - Nginx needs explicit proxy_set_header Upgrade and Connection headers. Caddy and Traefik handle websockets transparently.
Custom error pages - Each proxy handles error responses differently. Test your 502/503 pages after migration.
Logging format - If you're parsing access logs with Loki/ELK, your log format will change. Update your parsers before switching.
Certificate storage - If migrating to Traefik or Caddy, your existing Let's Encrypt certificates from Certbot won't transfer. The new proxy will obtain fresh certificates automatically, but be aware of Let's Encrypt rate limits (50 certificates per registered domain per week).

Need Help with Your DevOps?

Setting up and maintaining reverse proxies, load balancers, and TLS infrastructure is just one piece of the DevOps puzzle. At InstaDevOps, we handle your entire infrastructure - from proxy configuration to CI/CD pipelines, monitoring, and Kubernetes management - starting at $2,999/month.

Book a free 15-minute consultation to discuss your infrastructure needs: https://calendly.com/instadevops/15min

Cloud Cost FinOps: Cut Your AWS Bill by 40% Without Sacrificing Performance

InstaDevOps — Fri, 17 Apr 2026 13:46:50 +0000

Introduction

Cloud spending has a way of creeping up silently. You start with a few EC2 instances and an RDS database, and eighteen months later your AWS bill is five figures and nobody can explain exactly where the money goes. Engineering says they need all the resources. Finance says the cloud was supposed to be cheaper than on-premise. Both are frustrated.

FinOps - short for Cloud Financial Operations - is the practice of bringing financial accountability to cloud spending. It is not about cutting costs to the bone. It is about making sure every dollar you spend on cloud infrastructure delivers value, and that your team can make informed trade-off decisions between cost, speed, and quality.

This guide covers the practical FinOps strategies that consistently deliver the biggest savings on AWS: commitment discounts, spot instances, right-sizing, cost allocation, and the organizational practices that make cost optimization sustainable.

Understanding Your Current Spend

Before optimizing anything, you need visibility into where your money goes. Without data, you are guessing.

Enable Cost Allocation Tags

Tags are the foundation of cloud cost management. Without them, your bill is one opaque number. With them, you can attribute costs to teams, services, environments, and projects.

At minimum, enforce these tags on all resources:

# Terraform - enforce tagging via default_tags
provider "aws" {
  region = "eu-west-1"

  default_tags {
    tags = {
      Environment = var.environment      # production, staging, development
      Team        = var.team_name        # payments, platform, data
      Service     = var.service_name     # payment-api, user-service
      CostCenter  = var.cost_center      # maps to finance department codes
      ManagedBy   = "terraform"
    }
  }
}

Activate tags in the AWS Billing console under Cost Allocation Tags. It takes 24 hours for newly activated tags to appear in Cost Explorer.

AWS Cost Explorer and CUR

Cost Explorer gives you quick visibility. For deeper analysis, enable the Cost and Usage Report (CUR) which exports detailed billing data to S3:

resource "aws_cur_report_definition" "cost_report" {
  report_name                = "daily-cost-report"
  time_unit                  = "DAILY"
  format                     = "Parquet"
  compression                = "Parquet"
  additional_schema_elements = ["RESOURCES"]

  s3_bucket = aws_s3_bucket.cost_reports.id
  s3_region = "eu-west-1"
  s3_prefix = "cur"

  report_versioning = "OVERWRITE_REPORT"
}

Query the CUR data with Athena to answer questions like "How much did the payments team spend on RDS last quarter?"

Set Up Budget Alerts

Never let a bill surprise you. Create budgets with alerts at 50%, 80%, and 100% thresholds:

resource "aws_budgets_budget" "monthly" {
  name         = "monthly-total"
  budget_type  = "COST"
  limit_amount = "5000"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator       = "GREATER_THAN"
    threshold                 = 80
    threshold_type            = "PERCENTAGE"
    notification_type         = "ACTUAL"
    subscriber_email_addresses = ["engineering@company.com"]
  }

  notification {
    comparison_operator       = "GREATER_THAN"
    threshold                 = 100
    threshold_type            = "PERCENTAGE"
    notification_type         = "FORECASTED"
    subscriber_email_addresses = ["engineering@company.com", "finance@company.com"]
  }
}

Reserved Instances vs Savings Plans

Commitment discounts are the single largest cost lever for most AWS accounts, typically saving 30-60% over on-demand pricing.

Savings Plans (Recommended for Most Teams)

Savings Plans offer flexibility that Reserved Instances lack. You commit to a dollar amount per hour of compute usage, and AWS applies the discount automatically.

Compute Savings Plans cover EC2, Fargate, and Lambda across all regions and instance families. They offer the most flexibility:

On-demand: $0.0416/hour (t3.medium, eu-west-1)
1-year Compute SP (no upfront): $0.0270/hour - 35% savings
3-year Compute SP (all upfront): $0.0166/hour - 60% savings

EC2 Instance Savings Plans are locked to a specific instance family and region but offer slightly deeper discounts.

How to Calculate Your Commitment

Open Cost Explorer and view the last 3 months of EC2/Fargate spend
Identify your baseline (the minimum spend that is consistent every day)
Commit to 70-80% of that baseline with Savings Plans
Cover the remaining variable usage with on-demand and spot

Example:
  Average daily compute spend: $200/day
  Minimum daily spend (baseline): $160/day
  Recommended SP commitment: $160 * 0.75 = $120/day = $5.00/hour

  Annual savings: $120/day * 365 * 0.35 = $15,330/year

Common Mistake: Over-Committing

Do not commit to 100% of current usage. Your architecture will change, services will be rewritten, and traffic patterns will shift. Commit conservatively and re-evaluate quarterly.

Spot Instances: Up to 90% Savings

Spot instances use AWS's spare capacity at steep discounts (60-90% off on-demand), but can be interrupted with 2 minutes notice.

Where Spot Works Well

CI/CD build runners
Batch processing and data pipelines
Dev/staging environments
Stateless web workers behind a load balancer (with enough capacity diversity)
Machine learning training jobs with checkpointing

Where Spot Does Not Work

Single-instance databases
Stateful services without replication
Anything where a 2-minute shutdown causes data loss

ECS Spot with Capacity Providers

resource "aws_ecs_capacity_provider" "spot" {
  name = "spot-provider"

  auto_scaling_group_provider {
    auto_scaling_group_arn         = aws_autoscaling_group.spot.arn
    managed_termination_protection = "ENABLED"

    managed_scaling {
      maximum_scaling_step_size = 5
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 100
    }
  }
}

resource "aws_ecs_cluster_capacity_providers" "main" {
  cluster_name       = aws_ecs_cluster.main.name
  capacity_providers = [
    aws_ecs_capacity_provider.spot.name,
    "FARGATE",
    "FARGATE_SPOT"
  ]

  default_capacity_provider_strategy {
    base              = 2          # 2 tasks always on regular Fargate
    weight            = 1
    capacity_provider = "FARGATE"
  }

  default_capacity_provider_strategy {
    weight            = 3          # 3x weight on Fargate Spot
    capacity_provider = "FARGATE_SPOT"
  }
}

This runs a baseline of 2 tasks on regular Fargate and scales additional tasks on Fargate Spot at roughly 70% discount.

Right-Sizing: Stop Paying for Idle Resources

Most organizations over-provision by 30-50%. Right-sizing means matching resource allocation to actual usage.

Identifying Oversized Instances

Use AWS Compute Optimizer (free) or query CloudWatch directly:

# Check average CPU utilization for an instance over the last 2 weeks
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time $(date -u -v-14d +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 3600 \
  --statistics Average Maximum \
  --output table

Rules of thumb:

Average CPU under 20%: downsize by one tier
Average CPU under 10%: downsize by two tiers or consider Fargate
Max CPU consistently under 40%: safe to downsize one tier

Graviton Instances: 20% Better Price-Performance

AWS Graviton (ARM) instances offer 20% better price-performance than equivalent x86 instances. If your application runs on Linux and does not depend on x86-specific binaries, switching is straightforward:

t3.medium  (x86): $0.0416/hour
t4g.medium (ARM): $0.0336/hour - 19% cheaper, 20% faster

For containerized workloads, build multi-arch images:

docker buildx build --platform linux/amd64,linux/arm64 -t myapp:latest --push .

Storage and Data Transfer Optimization

S3 Lifecycle Policies

Most S3 data is accessed frequently for a short period and then rarely again:

resource "aws_s3_bucket_lifecycle_configuration" "logs" {
  bucket = aws_s3_bucket.logs.id

  rule {
    id     = "archive-old-logs"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"        # 45% cheaper
    }

    transition {
      days          = 90
      storage_class = "GLACIER_IR"         # 68% cheaper, millisecond retrieval
    }

    transition {
      days          = 365
      storage_class = "DEEP_ARCHIVE"       # 95% cheaper
    }

    expiration {
      days = 730  # Delete after 2 years
    }
  }
}

EBS Volume Optimization

Switch gp2 volumes to gp3: same performance, 20% cheaper, and you can independently scale IOPS
Delete unattached EBS volumes (check monthly)
Use EBS snapshots for infrequently accessed data instead of keeping volumes running

# Find unattached EBS volumes
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[*].{ID:VolumeId,Size:Size,Created:CreateTime}' \
  --output table

NAT Gateway Costs

NAT Gateways charge $0.045/hour ($32.85/month) plus $0.045/GB of data processed. For high-traffic workloads, this adds up fast.

Alternatives:

VPC Endpoints for AWS services (S3, DynamoDB, ECR) - eliminates NAT data charges for AWS API calls
NAT instances on t4g.nano ($3.07/month) for dev/staging environments

# S3 Gateway Endpoint (free, eliminates NAT charges for S3)
resource "aws_vpc_endpoint" "s3" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.eu-west-1.s3"
  route_table_ids = [aws_route_table.private.id]
}

# ECR Interface Endpoints
resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.eu-west-1.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  private_dns_enabled = true
}

Building a FinOps Culture

Tools alone do not reduce cloud costs. You need organizational practices that create accountability.

Weekly Cost Reviews

Run a 15-minute weekly meeting reviewing:

Total spend vs budget
Week-over-week change and top 3 drivers of increase
Top 5 most expensive services
Any anomalies or unexpected charges
Action items from last week

Team-Level Cost Dashboards

Give each team visibility into their own spending (this is where tags become critical). When engineers can see that their service costs $2,400/month, they start caring about optimization.

Architecture Decision Records for Cost

For significant infrastructure decisions, document the cost implications:

# ADR-015: Switch payment-api from t3.xlarge to t4g.large

## Context
Payment API currently runs on 3x t3.xlarge ($0.1664/hr each).
Average CPU: 22%. Average memory: 35%.

## Decision
Migrate to 3x t4g.large (ARM, $0.0672/hr each).
CPU and memory are sufficient based on 30-day CloudWatch data.

## Cost Impact
Before: 3 * $0.1664 * 730 = $364.42/month
After:  3 * $0.0672 * 730 = $147.17/month
Savings: $217.25/month ($2,607/year)

Quick Wins Checklist

If you are just starting with FinOps, tackle these in order. Each one can be done in a day or less:

Delete unused resources - unattached EBS volumes, old snapshots, idle load balancers, stopped instances with EBS volumes
Enable S3 Intelligent-Tiering on buckets with unpredictable access patterns
Switch gp2 to gp3 EBS volumes (20% savings, no performance loss)
Add VPC endpoints for S3 and DynamoDB (eliminates NAT data charges)
Purchase Savings Plans covering 70% of your steady-state compute
Set up budget alerts so you know before the bill arrives
Enable Compute Optimizer recommendations (free)
Review and clean up old ECR images, CloudWatch log groups, and Lambda versions

Need Help with Your DevOps?

Cloud cost optimization is an ongoing practice, not a one-time project. At InstaDevOps, we help startups and growing companies implement FinOps practices that reduce cloud spending by 30-50% while maintaining the performance and reliability your users expect.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to get a cloud cost assessment.

Building a DevSecOps Pipeline: Shift Security Left Without Slowing Down

InstaDevOps — Thu, 16 Apr 2026 13:46:46 +0000

Introduction

Every week brings news of another data breach, supply chain attack, or misconfigured cloud resource exposing millions of records. The traditional approach of bolting security on at the end of the development cycle no longer works. By the time a penetration tester finds a vulnerability in staging, the code has been through weeks of development, and fixing it means expensive rework.

DevSecOps integrates security testing directly into your CI/CD pipeline so vulnerabilities are caught in minutes, not months. The goal is not to slow down development - it is to catch security issues at the same speed you catch bugs: automatically, on every commit.

This guide walks through building a practical DevSecOps pipeline with real tool configurations, covering static analysis, dependency scanning, container image scanning, infrastructure security, and policy-as-code.

The DevSecOps Pipeline Stages

A mature DevSecOps pipeline runs security checks at every stage:

Code Commit → SAST → Dependency Scan → Build → Container Scan → IaC Scan → DAST → Deploy → Runtime Protection

You do not need all of these on day one. Start with the highest-impact, lowest-friction stages and layer on additional checks as your team matures.

Stage 1: Static Application Security Testing (SAST)

SAST tools analyze your source code for vulnerabilities without executing it. They catch issues like SQL injection, cross-site scripting, hardcoded secrets, and insecure cryptographic patterns.

Semgrep for Custom and Community Rules

Semgrep is fast, supports 30+ languages, and lets you write custom rules in a simple YAML syntax:

# .github/workflows/sast.yml
name: SAST Scan
on: [pull_request]

jobs:
  semgrep:
    runs-on: ubuntu-latest
    container:
      image: semgrep/semgrep
    steps:
      - uses: actions/checkout@v4
      - run: semgrep scan --config auto --error --sarif --output semgrep.sarif .
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: semgrep.sarif
        if: always()

The --config auto flag uses Semgrep's curated community ruleset. For stricter scanning, add specific rule packs:

semgrep scan \
  --config p/security-audit \
  --config p/secrets \
  --config p/owasp-top-ten \
  --error .

Writing Custom Security Rules

Create organization-specific rules for patterns your team should avoid:

# .semgrep/custom-rules.yml
rules:
  - id: no-raw-sql-queries
    patterns:
      - pattern: |
          db.query($SQL, ...)
      - pattern-not: |
          db.query($SQL, [...])
    message: "Use parameterized queries to prevent SQL injection"
    severity: ERROR
    languages: [javascript, typescript]

  - id: no-console-log-in-production
    pattern: console.log(...)
    message: "Remove console.log before merging to main"
    severity: WARNING
    languages: [javascript, typescript]
    paths:
      include:
        - src/

Stage 2: Dependency and Supply Chain Scanning

Your application's dependencies are a massive attack surface. A single vulnerable transitive dependency can compromise your entire system.

Scanning with Trivy

Trivy scans not just containers but also filesystems for vulnerable dependencies:

# .github/workflows/dependency-scan.yml
jobs:
  scan-dependencies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Trivy filesystem scan
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          scan-ref: .
          severity: HIGH,CRITICAL
          exit-code: 1
          format: sarif
          output: trivy-fs.sarif

      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy-fs.sarif
        if: always()

Software Bill of Materials (SBOM)

Generate an SBOM for compliance and vulnerability tracking:

# Generate SBOM with Syft
syft packages dir:. -o spdx-json > sbom.spdx.json

# Scan the SBOM for vulnerabilities with Grype
grype sbom:sbom.spdx.json --fail-on high

Automated Dependency Updates

Use Renovate or Dependabot to keep dependencies current:

// renovate.json
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "vulnerabilityAlerts": {
    "enabled": true,
    "labels": ["security"]
  },
  "packageRules": [
    {
      "matchUpdateTypes": ["patch"],
      "automerge": true,
      "automergeType": "branch"
    },
    {
      "matchUpdateTypes": ["major"],
      "labels": ["breaking-change"],
      "assignees": ["team-lead"]
    }
  ]
}

Stage 3: Container Image Scanning

Container images often contain vulnerable OS packages, misconfigured permissions, and unnecessary tools that increase the attack surface.

Scanning During Build

jobs:
  build-and-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: 1

      - name: Check for root user
        run: |
          USER=$(docker inspect --format='{{.Config.User}}' myapp:${{ github.sha }})
          if [ -z "$USER" ] || [ "$USER" = "root" ]; then
            echo "ERROR: Container runs as root. Add a USER directive to your Dockerfile."
            exit 1
          fi

Hardening Dockerfiles

A secure Dockerfile follows these patterns:

# Use specific version, not :latest
FROM node:20.11-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Multi-stage build - no build tools in final image
FROM node:20.11-alpine

# Create non-root user
RUN addgroup -g 1001 -S appuser && \
    adduser -S appuser -u 1001 -G appuser

WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --chown=appuser:appuser . .

# Drop all capabilities
USER appuser

# Use dumb-init to handle PID 1 properly
RUN apk add --no-cache dumb-init
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "server.js"]

Stage 4: Infrastructure as Code Security

Misconfigured infrastructure is the leading cause of cloud breaches. Scan your Terraform, CloudFormation, and Kubernetes manifests before they reach production.

Checkov for IaC Scanning

jobs:
  iac-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: terraform/
          framework: terraform
          soft_fail: false
          output_format: sarif
          output_file_path: checkov.sarif

      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: checkov.sarif
        if: always()

Checkov catches misconfigurations like:

S3 buckets without encryption
Security groups open to 0.0.0.0/0
RDS instances without encryption at rest
IAM policies with wildcard permissions
EBS volumes without encryption

tfsec for Terraform-Specific Scanning

# Run tfsec on your Terraform directory
tfsec terraform/ --format sarif --out tfsec.sarif

# Common findings:
# - aws-ec2-no-public-ingress-sgr
# - aws-s3-enable-bucket-encryption
# - aws-iam-no-policy-wildcards
# - aws-rds-encrypt-instance-storage-data

Stage 5: Dynamic Application Security Testing (DAST)

DAST tools test your running application by sending requests and analyzing responses for vulnerabilities. Run these against a staging environment after deployment.

OWASP ZAP Baseline Scan

jobs:
  dast:
    runs-on: ubuntu-latest
    needs: deploy-staging
    steps:
      - name: OWASP ZAP Baseline Scan
        uses: zaproxy/action-baseline@v0.12.0
        with:
          target: https://staging.example.com
          rules_file_name: zap-rules.tsv
          fail_action: true
          cmd_options: '-a -j'

DAST scans are slower (minutes to hours) and noisier than SAST, so run them post-deployment rather than on every commit.

Stage 6: Policy-as-Code with Open Policy Agent

Policy-as-Code enforces organizational rules programmatically. Open Policy Agent (OPA) lets you write policies in Rego that gate deployments:

# policy/kubernetes.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Deployment"
  container := input.request.object.spec.template.spec.containers[_]
  not container.resources.limits.memory
  msg := sprintf("Container '%v' must have memory limits set", [container.name])
}

deny[msg] {
  input.request.kind.kind == "Deployment"
  container := input.request.object.spec.template.spec.containers[_]
  container.image
  not contains(container.image, "@sha256:")
  not regex.match("^.*:[0-9]+\\.[0-9]+\\.[0-9]+$", container.image)
  msg := sprintf("Container '%v' must use a specific version tag or digest, not :latest", [container.name])
}

deny[msg] {
  input.request.kind.kind == "Deployment"
  container := input.request.object.spec.template.spec.containers[_]
  container.securityContext.privileged == true
  msg := sprintf("Container '%v' must not run in privileged mode", [container.name])
}

Conftest for CI Integration

Use Conftest to run OPA policies against Kubernetes manifests, Terraform plans, and Dockerfiles in CI:

# Test Kubernetes manifests
conftest test k8s/deployment.yaml --policy policy/

# Test Terraform plans
terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json
conftest test tfplan.json --policy policy/terraform/

# Test Dockerfiles
conftest test Dockerfile --policy policy/docker/

Putting It All Together

Here is a complete pipeline that ties all stages together:

name: DevSecOps Pipeline
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  # Fast checks first (< 2 minutes)
  secrets-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: trufflesecurity/trufflehog@main
        with:
          extra_args: --only-verified

  sast:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          docker run --rm -v "${PWD}:/src" semgrep/semgrep \
            semgrep scan --config auto --error /src

  # Medium checks (2-5 minutes)
  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          severity: HIGH,CRITICAL
          exit-code: 1

  iac-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bridgecrewio/checkov-action@v12
        with:
          directory: terraform/

  # Build and container scan
  build:
    needs: [secrets-scan, sast, dependency-scan, iac-scan]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t app:${{ github.sha }} .
      - uses: aquasecurity/trivy-action@master
        with:
          image-ref: app:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: 1

  # Deploy to staging, then DAST
  deploy-staging:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: echo "Deploy to staging"

  dast:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: zaproxy/action-baseline@v0.12.0
        with:
          target: https://staging.example.com

Managing False Positives

Every security scanning tool produces false positives. If you do not manage them, your team will start ignoring security alerts entirely.

Create a baseline. On initial integration, capture existing findings as a baseline and only alert on new issues.

Use inline suppressions with justification. Every suppressed finding should include a reason:

# nosemgrep: python.lang.security.audit.insecure-hash.insecure-hash
# Justification: MD5 used for non-security cache key, not for cryptographic purposes
cache_key = hashlib.md5(data).hexdigest()

Triage weekly. Assign someone to review and triage findings weekly. Close false positives, create tickets for real issues, and tune rules that produce too much noise.

Security Metrics That Matter

Tracking the right metrics tells you whether your DevSecOps pipeline is actually improving security or just generating noise.

Mean Time to Remediate (MTTR)

How long does it take from when a vulnerability is detected to when it is fixed in production? Track this by severity:

Critical: Target under 24 hours
High: Target under 7 days
Medium: Target under 30 days
Low: Target under 90 days

Vulnerability Escape Rate

What percentage of vulnerabilities make it past your CI/CD pipeline into production? This is your pipeline's effectiveness metric. Track it by comparing CI findings against production security scans and penetration test results.

Coverage Percentage

What percentage of your repositories have security scanning enabled? In most organizations, the answer is disturbingly low. Aim for 100% of production services covered by at minimum dependency scanning and SAST.

Fix Rate vs Find Rate

If you are finding vulnerabilities faster than you are fixing them, your backlog will grow indefinitely. This signals that your pipeline is too noisy (tune it) or your team needs more security training.

Getting Started: A Phased Approach

Do not try to implement everything at once. Here is a realistic rollout plan:

Week 1-2: Secrets scanning and dependency scanning. These have the highest signal-to-noise ratio and catch the most impactful issues. Enable Dependabot or Renovate for automated updates.

Week 3-4: SAST with Semgrep. Start with the default auto config. Do not write custom rules yet. Let the team get comfortable with the findings.

Month 2: Container image scanning. Add Trivy to your Docker build pipeline. Fix the critical and high findings, suppress the false positives with documentation.

Month 3: IaC scanning with Checkov. Scan your Terraform and Kubernetes manifests. This catches misconfigurations before they reach production.

Month 4+: DAST and policy-as-code. These require more setup and tuning but provide the deepest security coverage.

Need Help with Your DevOps?

Building a DevSecOps pipeline that catches real vulnerabilities without drowning your team in false positives takes careful tuning. At InstaDevOps, we implement security-hardened CI/CD pipelines for startups and growing teams - baking security in from day one.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to discuss your security posture.

AWS ECS vs EKS: A Practical Decision Framework for Startups

InstaDevOps — Wed, 15 Apr 2026 13:46:43 +0000

Introduction

You have containerized your application and now you need to run it in production on AWS. The two main options stare back at you from the AWS console: Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS). Both are fully managed, both run containers, and both integrate deeply with the AWS ecosystem.

So which one should you pick?

The answer is not "EKS because Kubernetes is the industry standard" and it is not "ECS because it is simpler." The right choice depends on your team size, operational maturity, workload characteristics, and growth trajectory. This article gives you a practical framework for making this decision, with real cost numbers and migration considerations.

ECS: The AWS-Native Path

ECS is AWS's proprietary container orchestration service. It is deeply integrated with the AWS ecosystem and was purpose-built to make running containers on AWS as straightforward as possible.

ECS Architecture

ECS has two launch types:

EC2 Launch Type: You manage the underlying EC2 instances. ECS handles container placement and scheduling.

Fargate Launch Type: Serverless containers. AWS manages the infrastructure entirely. You just define CPU, memory, and your container image.

{
  "family": "my-api",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/my-api:latest",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/my-api",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

ECS Strengths

Zero Kubernetes knowledge required. Your team interacts with familiar AWS concepts: task definitions, services, target groups, IAM roles. There is no kubectl, no YAML manifests with 15 indentation levels, no CRDs.

Deep AWS integration. ECS natively integrates with ALB/NLB, CloudWatch, IAM task roles, Secrets Manager, Parameter Store, App Mesh, and CloudMap for service discovery. These integrations work out of the box with minimal configuration.

Lower operational overhead. Especially with Fargate, there are no nodes to patch, no cluster upgrades to manage, no etcd to worry about. AWS handles all of it.

Simpler networking. The awsvpc network mode gives each task its own ENI and security group. No overlay networks, no kube-proxy, no CNI plugin decisions.

ECS Limitations

Vendor lock-in to AWS
Smaller community and ecosystem compared to Kubernetes
Limited scheduling customization
No equivalent to Kubernetes operators or CRDs for extending the platform
Fewer third-party tools (no Helm, no ArgoCD, no Istio)

EKS: The Kubernetes Path

EKS is AWS's managed Kubernetes service. AWS manages the control plane (API server, etcd, scheduler), and you manage the worker nodes or use Fargate for serverless pods.

EKS Architecture

# Sample Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      serviceAccountName: my-api
      containers:
        - name: api
          image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/my-api:latest
          ports:
            - containerPort: 3000
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  selector:
    app: my-api
  ports:
    - port: 80
      targetPort: 3000

EKS Strengths

Portability. Kubernetes manifests work on any cloud provider or on-premises cluster. If multi-cloud or hybrid-cloud is in your future, EKS keeps your options open.

Rich ecosystem. Helm charts, ArgoCD for GitOps, Istio or Linkerd for service mesh, Prometheus and Grafana for monitoring, cert-manager for TLS, external-dns for DNS automation. The Kubernetes ecosystem is enormous.

Advanced scheduling. Node affinities, taints and tolerations, pod topology spread constraints, priority classes, and preemption give you fine-grained control over workload placement.

Extensibility. Custom Resource Definitions and operators let you extend Kubernetes to manage databases, message queues, ML pipelines, and virtually anything else as native Kubernetes resources.

EKS Limitations

$0.10/hour ($73/month) control plane cost before any compute
Steep learning curve (expect 3-6 months for a team new to Kubernetes)
Cluster upgrades require planning and testing (new version every 4 months)
More moving parts: CNI plugins, ingress controllers, cluster autoscaler, DNS
Significantly higher operational burden

Cost Comparison

Let us compare the cost of running a typical startup workload: 3 services, each running 2 replicas with 0.5 vCPU and 1 GB RAM.

ECS Fargate

Per task: 0.5 vCPU + 1 GB RAM
Fargate pricing (eu-west-1):
  vCPU: $0.04048/hour
  Memory: $0.004445/GB/hour

Per task/hour: (0.5 * $0.04048) + (1 * $0.004445) = $0.02469
6 tasks total: $0.02469 * 6 = $0.14814/hour

Monthly: $0.14814 * 730 = $108.14/month

ECS EC2 (with Reserved Instances)

2x t3.medium (2 vCPU, 4 GB) with 1-year RI:
  On-demand: $0.0416/hour each
  1-year RI (no upfront): $0.027/hour each

Monthly: 2 * $0.027 * 730 = $39.42/month

EKS with Managed Node Group

EKS control plane: $0.10/hour = $73/month
2x t3.medium nodes (same as ECS EC2):
  1-year RI: 2 * $0.027 * 730 = $39.42/month

Monthly total: $73 + $39.42 = $112.42/month

EKS Fargate

EKS control plane: $73/month
Fargate pods (same as ECS Fargate): $108.14/month
Plus CoreDNS pods on Fargate: ~$15/month

Monthly total: $73 + $108.14 + $15 = $196.14/month

At small scale, ECS on EC2 is the cheapest option by far. EKS adds a fixed $73/month control plane cost that only becomes negligible at larger scales.

The Decision Framework

Choose ECS When

Your team is small (under 10 engineers). The operational overhead of Kubernetes is not justified when a few people can manage ECS with part-time attention.

You are all-in on AWS. If you have no plans for multi-cloud and are already using ALB, CloudWatch, and IAM extensively, ECS slots in naturally.

You want Fargate's simplicity. ECS Fargate is the easiest path from container image to running workload. No nodes, no cluster management, no upgrades.

Your workloads are straightforward. Web APIs, background workers, and scheduled tasks that do not need advanced scheduling or custom operators.

You need to ship fast. A team new to containers can have ECS running in production within a week. Kubernetes takes months to operate confidently.

Choose EKS When

Multi-cloud or hybrid is a real requirement. Not a hypothetical future possibility, but an actual business constraint.

You have Kubernetes experience on the team. If your engineers already know Kubernetes, EKS employs that knowledge. Forcing Kubernetes-experienced engineers to learn ECS is a lateral move with no payoff.

You need the ecosystem. If your architecture requires service mesh, GitOps (ArgoCD), advanced traffic management, or custom operators, the Kubernetes ecosystem is unmatched.

You are running 20+ services. At scale, Kubernetes' sophisticated scheduling, resource management, and namespace isolation become genuine advantages.

You are building a platform team. If part of your strategy is building an internal developer platform, Kubernetes provides the extensibility to build abstractions on top.

Migration Paths

ECS to EKS

If you outgrow ECS, migration is manageable:

Set up EKS cluster alongside existing ECS
Convert task definitions to Kubernetes deployments (the container config is similar)
Use AWS ALB Ingress Controller to maintain the same load balancer
Migrate services one at a time, shifting traffic gradually
Decommission ECS services after verification

EKS to ECS

Rarer, but it happens when teams realize they over-invested in Kubernetes:

Convert Kubernetes deployments to ECS task definitions
Replace Kubernetes services with ECS services + ALB target groups
Replace Helm/ArgoCD with AWS CodePipeline or similar
Migrate ConfigMaps/Secrets to AWS Parameter Store/Secrets Manager

A Pragmatic Recommendation

For most startups and SMBs running fewer than 15 services on AWS, start with ECS. Here is why:

You avoid 3-6 months of Kubernetes learning curve
You save $73/month in control plane costs (trivial individually, but symbolic of broader overhead)
You eliminate cluster upgrade maintenance windows
Your engineers focus on product features instead of infrastructure plumbing
If you outgrow ECS, the migration to EKS is well-understood

The exception: if you already have a team member who is deeply experienced with Kubernetes and can own the cluster operations, EKS is fine from day one.

Operational Overhead: What Nobody Tells You

The cost comparison above only covers compute. The real difference between ECS and EKS is operational overhead - the time your engineers spend managing the orchestration platform itself rather than building product features.

ECS Day-2 Operations

With ECS (especially Fargate), your ongoing operational tasks are minimal:

Task definition updates when you change container config (straightforward JSON/Terraform changes)
Service scaling policies adjustments based on traffic patterns
ALB health check tuning to match your application's startup time
Security group and IAM role updates as your network requirements change

Most of this is standard AWS work that any cloud engineer can handle.

EKS Day-2 Operations

EKS adds a significant layer of Kubernetes-specific operational work:

Cluster version upgrades every 4 months (Kubernetes drops support for old versions aggressively). Each upgrade requires testing all workloads, updating add-ons, and often updating manifests for deprecated APIs.
Add-on management: CoreDNS, kube-proxy, VPC CNI, cluster autoscaler, ingress controller, cert-manager, external-dns - each has its own version matrix and upgrade path.
Node group rotation when you change AMIs or instance types. Cordon, drain, replace.
RBAC policy management as teams and services grow.
etcd monitoring for the control plane (managed by AWS, but you still need to understand its implications for API server performance).
Debugging Kubernetes networking when services cannot reach each other (is it a NetworkPolicy? A CNI issue? A security group? A missing service account?).

A reasonable estimate: EKS adds 10-20 hours per month of platform engineering work compared to ECS. At a senior engineer's loaded cost, that is $3,000-6,000/month in operational overhead that does not show up on your AWS bill.

Real-World Scenarios

Scenario 1: 4-Person Startup, 2 Services

Choose ECS Fargate. You do not have the engineering bandwidth to learn and operate Kubernetes. ECS Fargate lets you deploy containers in an afternoon and forget about infrastructure management for the next 12 months.

Scenario 2: 25-Person Scale-Up, 12 Services, AWS Only

Choose ECS EC2. You are big enough that Fargate costs add up, but your services are straightforward enough that Kubernetes' advanced features are not needed. ECS on EC2 with Reserved Instances gives you the best cost efficiency.

Scenario 3: 50-Person Company, 30 Services, Multi-Cloud Mandate

Choose EKS. At this scale, you likely have or can hire a dedicated platform engineer. Kubernetes' portability and ecosystem justify the operational investment, and you will benefit from tools like ArgoCD, Istio, and custom operators.

Scenario 4: Regulated Industry, Compliance Requirements

EKS edges ahead. Kubernetes' namespace isolation, network policies, pod security standards, and OPA Gatekeeper provide a richer set of compliance controls. Combined with tools like Falco for runtime security, you can build a compliance story that auditors understand and respect.

Need Help with Your DevOps?

Choosing the right container orchestration platform is just the first decision. At InstaDevOps, we help startups set up production-grade container infrastructure on AWS - whether that is ECS, EKS, or a migration between the two.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to discuss your container strategy.

Advanced GitHub Actions: Matrix Builds, Reusable Workflows, and Self-Hosted Runners

InstaDevOps — Tue, 14 Apr 2026 13:46:40 +0000

Introduction

GitHub Actions has matured from a simple CI tool into a full-featured automation platform. Most teams start with a basic build-and-test workflow, but GitHub Actions offers far more powerful patterns that can dramatically reduce pipeline duplication, speed up builds, and cut CI costs.

This article covers advanced patterns that separate hobby-level GitHub Actions usage from production-grade CI/CD: matrix builds for testing across multiple environments, reusable workflows for eliminating duplication, self-hosted runners for cost and performance, and secrets management strategies that keep your pipelines secure.

If you are already comfortable writing basic GitHub Actions workflows, this guide will take you to the next level.

Matrix Builds: Test Everything in Parallel

Matrix builds let you run the same job across multiple combinations of variables - OS versions, language versions, dependency versions - without duplicating workflow code.

Basic Matrix Strategy

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18, 20, 22]
        os: [ubuntu-latest, macos-latest]
      fail-fast: false
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Setting fail-fast: false ensures all matrix combinations run to completion even if one fails. This is critical for understanding the full scope of compatibility issues.

Dynamic Matrix Generation

For more complex scenarios, generate the matrix dynamically from a previous job:

jobs:
  detect-services:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.find.outputs.services }}
    steps:
      - uses: actions/checkout@v4
      - id: find
        run: |
          # Find all directories with a Dockerfile
          SERVICES=$(find services/ -name Dockerfile -exec dirname {} \; | jq -R -s -c 'split("\n")[:-1]')
          echo "services=$SERVICES" >> $GITHUB_OUTPUT

  build:
    needs: detect-services
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: ${{ fromJson(needs.detect-services.outputs.services) }}
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t ${{ matrix.service }}:latest ${{ matrix.service }}

This pattern is invaluable for monorepos: only build and test services that have changed, and automatically pick up new services without modifying the workflow.

Matrix Include and Exclude

Fine-tune your matrix with include and exclude rules:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]
    node: [18, 20]
    include:
      # Add an experimental Node 22 build on Ubuntu only
      - os: ubuntu-latest
        node: 22
        experimental: true
    exclude:
      # Skip Node 18 on Windows (known incompatibility)
      - os: windows-latest
        node: 18

Reusable Workflows: Eliminate Duplication

If you have 10 microservices with nearly identical CI workflows, reusable workflows let you define the pipeline once and call it from each repository.

Creating a Reusable Workflow

Define a workflow with workflow_call trigger in a shared repository:

# .github/workflows/build-and-deploy.yml in org/shared-workflows repo
name: Build and Deploy Service
on:
  workflow_call:
    inputs:
      service-name:
        required: true
        type: string
      dockerfile-path:
        required: false
        type: string
        default: './Dockerfile'
      environment:
        required: true
        type: string
    secrets:
      AWS_ACCESS_KEY_ID:
        required: true
      AWS_SECRET_ACCESS_KEY:
        required: true
      ECR_REGISTRY:
        required: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-west-1

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push image
        run: |
          IMAGE="${{ secrets.ECR_REGISTRY }}/${{ inputs.service-name }}:${{ github.sha }}"
          docker build -f ${{ inputs.dockerfile-path }} -t $IMAGE .
          docker push $IMAGE

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster ${{ inputs.environment }}-cluster \
            --service ${{ inputs.service-name }} \
            --force-new-deployment

Calling a Reusable Workflow

Each service repository calls the shared workflow:

# .github/workflows/ci.yml in service repo
name: CI/CD
on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    uses: org/shared-workflows/.github/workflows/build-and-deploy.yml@v2.1.0
    with:
      service-name: payment-api
      environment: staging
    secrets:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      ECR_REGISTRY: ${{ secrets.ECR_REGISTRY }}

  deploy-production:
    needs: deploy-staging
    uses: org/shared-workflows/.github/workflows/build-and-deploy.yml@v2.1.0
    with:
      service-name: payment-api
      environment: production
    secrets:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      ECR_REGISTRY: ${{ secrets.ECR_REGISTRY }}

Pin the reusable workflow to a specific tag (@v2.1.0) so that upstream changes do not break your pipelines unexpectedly.

Self-Hosted Runners: Performance and Cost Control

GitHub-hosted runners are convenient but come with limitations: fixed hardware specs, cold start times, no persistent caches, and costs that scale linearly with usage. Self-hosted runners solve these problems.

When Self-Hosted Runners Make Sense

Large Docker builds that benefit from persistent layer caches
GPU workloads for ML model training or testing
Network-restricted environments where jobs need access to internal resources
High CI volume where GitHub-hosted runner costs exceed $1,000/month
Specialized hardware requirements

Setting Up Ephemeral Self-Hosted Runners on AWS

Use the actions-runner-controller (ARC) to autoscale runners on Kubernetes, or use EC2 with a simpler setup:

# Terraform for self-hosted runner ASG
resource "aws_launch_template" "runner" {
  name_prefix   = "github-runner-"
  image_id      = data.aws_ami.ubuntu.id
  instance_type = "c6i.2xlarge"

  user_data = base64encode(templatefile("runner-init.sh", {
    github_org    = var.github_org
    runner_token  = var.runner_registration_token
    runner_labels = "self-hosted,linux,x64,large"
  }))

  block_device_mappings {
    device_name = "/dev/sda1"
    ebs {
      volume_size = 100
      volume_type = "gp3"
    }
  }
}

resource "aws_autoscaling_group" "runners" {
  desired_capacity = 2
  max_size         = 10
  min_size         = 0

  launch_template {
    id      = aws_launch_template.runner.id
    version = "$Latest"
  }

  tag {
    key                 = "Purpose"
    value               = "github-actions-runner"
    propagate_at_launch = true
  }
}

Using Runner Labels for Job Routing

Target specific runners using labels:

jobs:
  unit-tests:
    # Fast tests on GitHub-hosted runners
    runs-on: ubuntu-latest
    steps:
      - run: npm test

  docker-build:
    # Heavy builds on self-hosted with Docker cache
    runs-on: [self-hosted, linux, large]
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t myapp:latest .

  gpu-tests:
    # ML tests on GPU runners
    runs-on: [self-hosted, gpu]
    steps:
      - run: python -m pytest tests/ml/

Secrets Management Patterns

Secrets handling in CI/CD is a top security concern. GitHub Actions provides several mechanisms, but you need to layer them correctly.

Environment-Scoped Secrets

Use GitHub environments to scope secrets to specific deployment targets:

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production  # Only secrets defined in "production" environment are available
    steps:
      - name: Deploy
        env:
          DB_PASSWORD: ${{ secrets.DB_PASSWORD }}  # Production-specific secret
        run: ./deploy.sh

Configure required reviewers on the production environment so deployments require manual approval.

OIDC Authentication (No Long-Lived Keys)

Eliminate static AWS credentials entirely using OpenID Connect:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
          aws-region: eu-west-1
          # No access key or secret needed - uses OIDC token exchange

The IAM role trust policy restricts which repositories and branches can assume it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      },
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:yourorg/yourrepo:ref:refs/heads/main"
      }
    }
  }]
}

Secrets Scanning and Rotation

Add a workflow that scans for accidentally committed secrets:

jobs:
  secrets-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: trufflesecurity/trufflehog@main
        with:
          extra_args: --only-verified

Caching Strategies for Faster Builds

Effective caching can cut build times by 50-80%.

Dependency Caching

- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      node_modules
    key: deps-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      deps-${{ runner.os }}-

Docker Layer Caching

- uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

Artifact Passing Between Jobs

Use artifacts to pass build outputs between jobs without rebuilding:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/
          retention-days: 1

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/
      - run: ./deploy.sh dist/

Workflow Organization at Scale

Path-Based Triggers

Only run workflows when relevant files change:

on:
  push:
    branches: [main]
    paths:
      - 'services/payment-api/**'
      - '.github/workflows/payment-api.yml'

Concurrency Control

Prevent redundant workflow runs when commits are pushed rapidly:

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

This cancels the previous run when a new commit is pushed to the same branch, saving runner minutes.

Composite Actions for Shared Steps

When full reusable workflows are overkill, composite actions bundle common steps:

# .github/actions/setup-project/action.yml
name: Setup Project
description: Install dependencies and configure environment
inputs:
  node-version:
    required: false
    default: '20'
runs:
  using: composite
  steps:
    - uses: actions/setup-node@v4
      with:
        node-version: ${{ inputs.node-version }}
    - uses: actions/cache@v4
      with:
        path: node_modules
        key: deps-${{ hashFiles('package-lock.json') }}
    - run: npm ci
      shell: bash

Use it in any workflow with a single line:

- uses: ./.github/actions/setup-project
  with:
    node-version: '20'

Monitoring and Debugging Workflows

When workflows fail in production, you need fast diagnosis. GitHub provides several tools for this.

Step-Level Timing Analysis

Every workflow run shows timing per step. If your pipeline is slow, look for steps that take disproportionate time. Common culprits:

Checkout with full history (fetch-depth: 0) on large repos - use shallow clones unless you need full history
Docker builds without caching - always use BuildKit cache with cache-from and cache-to
NPM/pip installs without cache - dependency caching alone can save 30-60 seconds per run

Workflow Run Logs and Annotations

Use ::warning and ::error workflow commands to surface issues directly in PR annotations:

- name: Check bundle size
  run: |
    SIZE=$(stat -f%z dist/bundle.js)
    if [ "$SIZE" -gt 500000 ]; then
      echo "::warning file=dist/bundle.js::Bundle size is ${SIZE} bytes (over 500KB limit)"
    fi

Reusable Debug Workflow

Create a debug composite action that dumps useful context when things go wrong:

- name: Debug info
  if: failure()
  run: |
    echo "## Environment" >> $GITHUB_STEP_SUMMARY
    echo "- Runner: ${{ runner.os }} ${{ runner.arch }}" >> $GITHUB_STEP_SUMMARY
    echo "- Node: $(node --version)" >> $GITHUB_STEP_SUMMARY
    echo "- Disk: $(df -h / | tail -1)" >> $GITHUB_STEP_SUMMARY
    echo "- Memory: $(free -h 2>/dev/null || vm_stat)" >> $GITHUB_STEP_SUMMARY

Common Pitfalls to Avoid

Not pinning action versions. Using actions/checkout@main means your workflow can break without any change on your side. Always pin to a specific version tag or commit SHA for third-party actions.

Leaking secrets in logs. GitHub automatically masks secrets in logs, but derived values (like a URL containing a token) are not masked. Use ::add-mask:: for any sensitive derived values.

Ignoring workflow permissions. The default GITHUB_TOKEN has broad permissions. Use the permissions key to restrict to only what each job needs:

permissions:
  contents: read
  pull-requests: write

Running everything on push and pull_request. This causes duplicate runs. Use push for deploys (main branch only) and pull_request for checks.

Not using job summaries. The $GITHUB_STEP_SUMMARY file lets you write rich Markdown summaries that appear in the workflow run UI - much better than scrolling through raw logs.

Need Help with Your DevOps?

Setting up production-grade CI/CD pipelines with proper security, caching, and scaling takes real-world experience. At InstaDevOps, we build and maintain CI/CD infrastructure for startups and growing teams so your developers can focus on shipping features.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to optimize your GitHub Actions workflows.

Terraform Modules Done Right: Mono-Repo, Versioning, and Registry Patterns

InstaDevOps — Mon, 13 Apr 2026 13:46:36 +0000

Introduction

As your Terraform codebase grows beyond a handful of resources, you inevitably face a structural decision: how do you organize reusable infrastructure components so that multiple teams can collaborate without stepping on each other's toes?

The answer is Terraform modules. But writing a module is easy - organizing, versioning, and distributing them at scale is where most teams struggle. Poorly structured modules lead to copy-paste drift, version conflicts, and infrastructure that nobody fully understands.

In this guide, we will walk through battle-tested patterns for organizing Terraform modules, choosing between mono-repo and multi-repo strategies, leveraging module registries, and implementing versioning that actually works in production.

What Makes a Good Terraform Module

Before discussing organization, let us establish what a well-designed module looks like. A good Terraform module follows these principles:

Single responsibility. Each module should manage one logical piece of infrastructure. A VPC module should not also create RDS instances.

Sensible defaults with full override capability. Provide defaults that work for 80% of use cases, but allow every significant parameter to be overridden:

variable "instance_type" {
  description = "EC2 instance type for the application servers"
  type        = string
  default     = "t3.medium"
}

variable "enable_enhanced_monitoring" {
  description = "Enable enhanced monitoring with 60-second granularity"
  type        = bool
  default     = true
}

variable "tags" {
  description = "Additional tags to apply to all resources"
  type        = map(string)
  default     = {}
}

Clear input/output contracts. Every variable should have a description, type constraint, and validation where appropriate. Every output that downstream consumers need should be explicitly exported:

variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type        = string

  validation {
    condition     = can(cidrhost(var.vpc_cidr, 0))
    error_message = "Must be a valid CIDR block."
  }
}

output "vpc_id" {
  description = "ID of the created VPC"
  value       = aws_vpc.main.id
}

output "private_subnet_ids" {
  description = "List of private subnet IDs"
  value       = aws_subnet.private[*].id
}

Minimal provider assumptions. Do not hardcode provider configurations inside modules. Let the caller configure the provider:

# Bad - hardcoded provider in module
provider "aws" {
  region = "us-east-1"
}

# Good - module relies on caller's provider configuration
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
  }
}

Mono-Repo vs Multi-Repo: Choosing the Right Strategy

This is the most debated structural decision in Terraform module management. Both approaches have legitimate trade-offs.

Mono-Repo Pattern

All modules live in a single repository with a directory structure like:

terraform-modules/
├── modules/
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── ecs-service/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── rds-postgres/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   └── s3-bucket/
│       ├── main.tf
│       ├── variables.tf
│       ├── outputs.tf
│       └── README.md
├── examples/
│   ├── complete-vpc/
│   └── ecs-with-rds/
├── tests/
│   ├── vpc_test.go
│   └── ecs_test.go
└── .github/
    └── workflows/
        └── test.yml

Advantages:

Atomic cross-module changes in a single PR
Unified CI/CD pipeline for testing
Easier to maintain consistency across modules
Simpler onboarding for new team members
One place to search for all infrastructure patterns

Disadvantages:

Git tags version the entire repo, not individual modules
Large repos can slow down CI runs
Permission boundaries are harder (everyone can see everything)

When to use: Teams under 20 engineers, organizations with fewer than 30 modules, or when most modules are tightly coupled.

Multi-Repo Pattern

Each module gets its own repository:

terraform-module-vpc/          (v2.3.1)
terraform-module-ecs-service/  (v1.8.0)
terraform-module-rds-postgres/ (v3.1.0)
terraform-module-s3-bucket/    (v1.2.4)

Advantages:

Independent versioning per module using Git tags
Fine-grained access control per repository
Smaller, focused CI/CD pipelines
Clear ownership boundaries

Disadvantages:

Cross-module changes require multiple PRs
Dependency management becomes complex
More repositories to maintain
Harder to ensure consistency

When to use: Organizations with 50+ modules, multiple platform teams owning different modules, or strict compliance requirements needing audit trails per component.

The Hybrid Approach

Many mature organizations use a hybrid: a mono-repo for foundational modules maintained by the platform team, with separate repos for domain-specific modules owned by product teams:

platform-terraform-modules/     # Platform team owns VPC, IAM, networking
├── modules/vpc/
├── modules/iam-roles/
└── modules/cloudfront/

team-payments-terraform/        # Payments team owns their service modules
├── modules/payment-service/
└── modules/fraud-detection/

Module Versioning Strategies

Versioning is where most Terraform setups break down. Without proper versioning, a module change can silently break every environment that references it.

Semantic Versioning

Follow semver strictly for modules:

MAJOR (v2.0.0): Breaking changes (removed variables, renamed resources that force replacement)
MINOR (v1.3.0): New features, new optional variables with defaults
PATCH (v1.2.1): Bug fixes, documentation updates

Pinning Module Versions

Always pin module versions in your root configurations:

# Good - pinned to exact version
module "vpc" {
  source  = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc?ref=v2.3.1"
}

# Acceptable - pinned to minor version range
module "vpc" {
  source  = "app.terraform.io/yourorg/vpc/aws"
  version = "~> 2.3"
}

# Bad - no version pin, uses latest
module "vpc" {
  source = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc"
}

Automated Version Bumping

Use a CI workflow that automatically creates releases when module directories change:

# .github/workflows/release.yml
name: Release Modules
on:
  push:
    branches: [main]

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      changed_modules: ${{ steps.changes.outputs.modules }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - id: changes
        run: |
          CHANGED=$(git diff --name-only HEAD~1 HEAD | grep '^modules/' | cut -d'/' -f2 | sort -u | jq -R -s -c 'split("\n")[:-1]')
          echo "modules=$CHANGED" >> $GITHUB_OUTPUT

  release:
    needs: detect-changes
    if: needs.detect-changes.outputs.changed_modules != '[]'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        module: ${{ fromJson(needs.detect-changes.outputs.changed_modules) }}
    steps:
      - uses: actions/checkout@v4

      - name: Determine version bump
        id: version
        run: |
          CURRENT=$(git tag -l "modules/${{ matrix.module }}/v*" | sort -V | tail -1)
          # Parse commit messages for bump type
          if git log --oneline HEAD~1..HEAD | grep -q "BREAKING"; then
            BUMP="major"
          elif git log --oneline HEAD~1..HEAD | grep -q "feat"; then
            BUMP="minor"
          else
            BUMP="patch"
          fi
          echo "bump=$BUMP" >> $GITHUB_OUTPUT

      - name: Create release tag
        run: |
          # Bump version and create tag
          git tag "modules/${{ matrix.module }}/v${NEW_VERSION}"
          git push --tags

Using a Private Module Registry

A module registry provides a discoverable, versioned catalog of your organization's modules. You have several options.

Terraform Cloud / HCP Terraform Registry

The simplest option if you are already using Terraform Cloud:

module "vpc" {
  source  = "app.terraform.io/yourorg/vpc/aws"
  version = "2.3.1"

  cidr_block  = "10.0.0.0/16"
  environment = "production"
}

Publishing is automatic when you connect your VCS repository to the registry.

Self-Hosted with Artifactory or S3

For air-gapped or highly regulated environments, you can host modules on S3:

module "vpc" {
  source = "s3::https://my-terraform-modules.s3.amazonaws.com/vpc/v2.3.1.zip"
}

Pair this with a CI pipeline that packages and uploads module archives on release:

#!/bin/bash
MODULE=$1
VERSION=$2

cd modules/$MODULE
zip -r "/tmp/${MODULE}-${VERSION}.zip" .
aws s3 cp "/tmp/${MODULE}-${VERSION}.zip" \
  "s3://my-terraform-modules/${MODULE}/${VERSION}.zip"

GitHub Releases as a Registry

A lightweight approach using Git tags directly:

module "vpc" {
  source = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc?ref=v2.3.1"
}

This works well for smaller organizations and avoids the overhead of a dedicated registry.

Module Testing and Validation

Modules without tests are modules you cannot trust. Here are the layers of testing you should implement.

Static Analysis

Run these on every PR:

# Format check
terraform fmt -check -recursive modules/

# Validation
for dir in modules/*/; do
  cd "$dir"
  terraform init -backend=false
  terraform validate
  cd ../..
done

# Security scanning with tfsec
tfsec modules/

# Linting with tflint
tflint --recursive

Integration Testing with Terratest

Write Go tests that actually provision and destroy infrastructure:

package test

import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)

func TestVpcModule(t *testing.T) {
    t.Parallel()

    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "cidr_block":  "10.99.0.0/16",
            "environment": "test",
            "name":        "terratest-vpc",
        },
    })

    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)

    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcId)

    privateSubnets := terraform.OutputList(t, terraformOptions, "private_subnet_ids")
    assert.Equal(t, 3, len(privateSubnets))
}

Example Configurations

Every module should ship with a working example in an examples/ directory:

modules/vpc/
├── main.tf
├── variables.tf
├── outputs.tf
├── README.md
└── examples/
    ├── simple/
    │   └── main.tf       # Minimal usage
    └── complete/
        └── main.tf       # All features enabled

Module Composition Patterns

Real infrastructure is built by composing modules together. Here are patterns that work well at scale.

The Root Module Pattern

Create environment-specific root modules that compose shared modules:

# environments/production/main.tf

module "network" {
  source  = "app.terraform.io/yourorg/vpc/aws"
  version = "2.3.1"

  cidr_block         = "10.0.0.0/16"
  availability_zones = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  environment        = "production"
}

module "database" {
  source  = "app.terraform.io/yourorg/rds-postgres/aws"
  version = "3.1.0"

  vpc_id             = module.network.vpc_id
  subnet_ids         = module.network.private_subnet_ids
  instance_class     = "db.r6g.xlarge"
  multi_az           = true
  environment        = "production"
}

module "application" {
  source  = "app.terraform.io/yourorg/ecs-service/aws"
  version = "1.8.0"

  vpc_id             = module.network.vpc_id
  subnet_ids         = module.network.private_subnet_ids
  lb_target_group    = module.network.alb_target_group_arn
  database_endpoint  = module.database.endpoint
  desired_count      = 6
  environment        = "production"
}

The Terragrunt DRY Pattern

For organizations managing many environments, Terragrunt eliminates repetition:

# terragrunt.hcl (root)
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "myorg-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

# environments/production/vpc/terragrunt.hcl
terraform {
  source = "git::https://github.com/yourorg/terraform-modules.git//modules/vpc?ref=v2.3.1"
}

inputs = {
  cidr_block  = "10.0.0.0/16"
  environment = "production"
}

Common Pitfalls and How to Avoid Them

Over-abstracting too early. Do not create a module until you have written the same Terraform code at least twice. Premature abstraction creates rigid modules that fight against real requirements.

Nested modules more than two levels deep. Module A calls module B which calls module C is already hard to debug. Keep your module hierarchy shallow.

Not using moved blocks during refactors. When restructuring modules, use moved blocks to prevent Terraform from destroying and recreating resources:

moved {
  from = aws_instance.web
  to   = module.application.aws_instance.web
}

Ignoring module documentation. Every module should have a README generated by terraform-docs:

# Generate docs automatically
terraform-docs markdown table modules/vpc/ > modules/vpc/README.md

Storing secrets in tfvars files. Use a secrets manager and data sources instead of committing sensitive values.

Need Help with Your DevOps?

Building and maintaining a well-structured Terraform module library takes experience. At InstaDevOps, we help startups and growing teams implement production-grade Infrastructure as Code from day one - so you can ship infrastructure changes with the same confidence as application code.

Plans start at $2,999/mo for a dedicated fractional DevOps engineer.

Book a free 15-minute consultation to discuss your Terraform architecture.

Infrastructure Testing: Terratest, Checkov, and Validating Your IaC

InstaDevOps — Sun, 12 Apr 2026 13:46:59 +0000

Introduction

Infrastructure as Code has transformed cloud management, but when a single Terraform apply can spin up hundreds of resources, testing becomes essential.

This article explores Terratest for functional testing and Checkov for security scanning.

Static Analysis with Checkov

pip install checkov
checkov -d ./terraform

Custom Policies

from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
from checkov.common.models.enums import CheckCategories, CheckResult

class EC2HasEnvironmentTag(BaseResourceCheck):
    def __init__(self):
        name = "Ensure EC2 instances have environment tag"
        id = "CKV_CUSTOM_1"
        supported_resources = ['aws_instance']
        super().__init__(name=name, id=id,
                        supported_resources=supported_resources)

    def scan_resource_conf(self, conf):
        tags = conf.get('tags', [{}])[0]
        if isinstance(tags, dict) and 'Environment' in tags:
            return CheckResult.PASSED
        return CheckResult.FAILED

Functional Testing with Terratest

func TestS3Bucket(t *testing.T) {
    t.Parallel()

    terraformOptions := terraform.WithDefaultRetryableErrors(t,
        &terraform.Options{
            TerraformDir: "../modules/s3-bucket",
            Vars: map[string]interface{}{
                "bucket_name": "test-bucket-" + random.UniqueId(),
            },
        })

    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)

    bucketID := terraform.Output(t, terraformOptions, "bucket_id")
    actualStatus := aws.GetS3BucketVersioning(t, "us-east-1", bucketID)
    assert.Equal(t, "Enabled", actualStatus)
}

When to Use Each Tool

Aspect	Checkov	Terratest
Type	Static analysis	Functional testing
Speed	Seconds	Minutes to hours
Cost	Free	Incurs cloud costs
Coverage	Security, compliance	Actual functionality

CI/CD Integration

jobs:
  static-analysis:
    steps:
      - uses: bridgecrewio/checkov-action@master
        with:
          directory: terraform/

  terratest:
    needs: static-analysis
    steps:
      - run: |
          cd test
          go test -v -timeout 30m ./...

Conclusion

Layer your testing: Start with Checkov for fast feedback, add Terratest for critical modules, and run full integration tests before production.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Kubernetes Networking Demystified: Services, Ingress, and Network Policies

InstaDevOps — Sat, 11 Apr 2026 13:46:57 +0000

Introduction

Kubernetes networking is often cited as one of the most challenging aspects of container orchestration. This guide covers the three pillars: Services, Ingress, and Network Policies.

Services: Stable Endpoints for Dynamic Pods

ClusterIP Services

apiVersion: v1
kind: Service
metadata:
  name: backend-api
spec:
  type: ClusterIP
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 8080

LoadBalancer Services

apiVersion: v1
kind: Service
metadata:
  name: public-api
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
  - port: 443
    targetPort: 8443

Ingress: HTTP/HTTPS Traffic Management

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - app.example.com
    secretName: app-tls-secret
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: backend-api
            port:
              number: 80

Network Policies: Securing Pod Communication

Default Deny All

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Allow Specific Ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Troubleshooting

# Check endpoints
kubectl get endpoints my-service

# Test DNS
kubectl exec -it my-pod -- nslookup kubernetes.default

# Debug with netshoot
kubectl run netshoot --image=nicolaka/netshoot -it --rm -- bash

Conclusion

Master Services, Ingress, and Network Policies, and you'll have a solid foundation for building secure, scalable applications on Kubernetes.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Container Registry Best Practices: ECR, Docker Hub, and Self-Hosted Options

InstaDevOps — Fri, 10 Apr 2026 13:46:54 +0000

Introduction

Container registries are the backbone of containerized application deployment. Choosing the right registry and implementing proper practices can mean the difference between smooth deployments and security nightmares.

Amazon ECR: AWS-Native Registry

# Create a repository
aws ecr create-repository \
    --repository-name my-app \
    --image-scanning-configuration scanOnPush=true

# Push an image
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

ECR Lifecycle Policies

{
  "rules": [{
    "rulePriority": 1,
    "description": "Keep last 10 production images",
    "selection": {
      "tagStatus": "tagged",
      "tagPrefixList": ["prod-"],
      "countType": "imageCountMoreThan",
      "countNumber": 10
    },
    "action": { "type": "expire" }
  }]
}

Docker Hub

Docker Hub remains the most widely used registry, hosting millions of public images.

Rate Limits: Anonymous pulls limited to 100 per 6 hours; authenticated free users get 200.

Self-Hosted: Harbor

helm install harbor harbor/harbor \
  --set expose.type=ingress \
  --set expose.ingress.hosts.core=registry.example.com \
  --set trivy.enabled=true

Security Best Practices

Enable image scanning
Implement least-privilege access
Sign your images with Cosign
Use immutable tags
Scan base images regularly

Image Tagging Strategies

VERSION="1.2.3"
GIT_SHA=$(git rev-parse --short HEAD)

docker build \
    -t my-app:${VERSION} \
    -t my-app:${VERSION}-${GIT_SHA} \
    -t my-app:${GIT_SHA} \
    .

Conclusion

Whether you choose ECR for AWS integration, Docker Hub for ubiquity, or Harbor for control, applying security best practices will keep your container infrastructure secure.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Incident Management: Building Effective On-Call Rotations and Runbooks

InstaDevOps — Thu, 09 Apr 2026 20:46:50 +0000

Introduction

Every engineering team dreads the 3 AM page. How your team handles these moments defines your reliability as a service provider.

This guide covers on-call rotation design, runbook creation, incident response processes, and blameless post-mortems.

On-Call Best Practices

Designing Sustainable Rotations

Rotation Length: Weekly rotations work best
Minimum Team Size: Never fewer than 4 engineers
Compensation: Fair pay for on-call work

Reducing Alert Fatigue

Actionable alerts only
Consolidate related alerts
Regular alert reviews

Runbook Template Example

# Runbook: Database Connection Pool Exhausted

## Impact Assessment
- User Impact: New requests may fail
- Affected Services: All services using primary database

## Immediate Actions
1. Check current connections:
   SELECT count(*), state FROM pg_stat_activity GROUP BY state;

2. Identify connection hogs:
   SELECT usename, application_name, count(*)
   FROM pg_stat_activity GROUP BY usename, application_name;

## Resolution Steps
- If long-running query: pg_terminate_backend(<pid>)
- If traffic spike: Scale up read replicas
- If connection leak: Restart affected service

Incident Response Process

Phase 1: Detection and Triage (0-5 minutes)

Acknowledge the alert
Assess severity
Open relevant runbook

Phase 2: Response Coordination

Incident Commander: Coordinates response
Technical Lead: Hands-on debugging
Communications Lead: Updates stakeholders

Blameless Post-Mortems

Focus on systems and processes, not individuals. The goal is learning and improvement.

Conclusion

Effective incident management is built on preparation, not heroics. Well-designed rotations, comprehensive runbooks, and blameless post-mortems transform incident response from stress into a competitive advantage.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

SRE Fundamentals: Defining SLOs, SLIs, and Error Budgets That Actually Work

InstaDevOps — Thu, 09 Apr 2026 20:33:35 +0000

Introduction

Site Reliability Engineering (SRE) has transformed how organizations think about system reliability. Central to this framework are Service Level Objectives (SLOs), Service Level Indicators (SLIs), and Error Budgets.

This guide will walk you through defining SLOs, SLIs, and Error Budgets that actually drive meaningful improvements.

Understanding the Reliability Hierarchy

SLAs (Service Level Agreements): External contracts with customers specifying consequences for failures.

SLOs (Service Level Objectives): Internal targets your team commits to, stricter than SLAs.

SLIs (Service Level Indicators): The actual measurements determining whether you're meeting SLOs.

Error Budgets: How much unreliability you can tolerate while meeting your SLO.

Defining Meaningful SLIs

The Four Golden Signals

Latency: How long requests take
Traffic: Request volume
Errors: Rate of failed requests
Saturation: How "full" your service is

# Availability SLI
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Latency SLI
sum(rate(http_request_duration_seconds_bucket{le="0.2"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Setting Realistic SLOs

Each additional "nine" dramatically reduces your error budget:

Availability	Monthly Downtime
99%	7.2 hours
99.9%	43.8 minutes
99.99%	4.38 minutes

Error Budgets: The Key to Balance

Error Budget = 1 - SLO

For a 99.9% SLO over 30 days:
Error Budget = 0.1% = 43.2 minutes of downtime

Conclusion

SLOs, SLIs, and Error Budgets aren't just metrics—they're a framework for making better decisions about reliability. The goal is appropriate reliability—enough to keep users happy while maintaining velocity.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups build production-ready infrastructure.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com