DEV Community: Bartłomiej Danek

Slack Webhooks in Production: Durable Queues, Distributed Locking, and the Testing Epiphany

Bartłomiej Danek — Tue, 22 Sep 2026 18:35:56 +0000

TL;DR: The POC from Part 1 used a go func() and hoped for the best. The production version has a Redis-backed event queue with sharded workers, distributed locking for multi-replica safety, an idempotency layer (because Slack will retry), and integration tests against real Redis containers. This is what it took to make it reliable enough to run at scale.

The `go func()` Betrayal

The original handler acknowledged Slack's webhook with 200 OK, then spawned a goroutine:

go func() {
    h.metrics.RecordNewThread(ctx, event.Channel, event.Ts)
}()

Simple. Elegant. Wrong.

Three things broke in production within the first week:

Process restart during processing - the goroutine dies, the event is lost, Slack never retries (it already got 200 OK).
Replica contention - with two pods, both process the same event, doubling every counter.
Out-of-order processing - a reaction_removed arrives before its reaction_added because the goroutine scheduler interleaved them differently than the Event API delivered them.

The fix was a durable event queue, but the implementation taught me more about Redis than I expected.

Architecture: What Changed

Every event goes through a Redis list as a durable buffer. The HTTP handler acknowledges Slack immediately, then enqueues. Background workers dequeue and process. If the process dies mid-processing, the event stays on the queue. If another replica picks it up, the idempotency key prevents double-counting.

The Sharding Problem

A single Redis list with multiple workers is a race: two workers both BRPop the same list and process sibling events concurrently. A reply and its parent message, or a reaction_added / reaction_removed pair, lose ordering guarantees.

The solution was routing keys + sharded queues:

// routingKey identifies the "thread of conversation" an event belongs to
func routingKey(event *SlackEventData) string {
    switch event.Type {
    case "message":
        if event.ThreadTS != "" {
            return channel + ":" + event.ThreadTS  // reply → thread
        }
        return channel + ":" + event.TS             // top-level → itself
    case "reaction_added", "reaction_removed":
        return channel + ":" + event.Item.TS        // reaction → target message
    }
    return channel
}

A FNV-32a hash of the routing key maps to one of N shards (swh:queue:events:0 through swh:queue:events:N-1). Every event for the same thread lands on the same shard, consumed by the same worker, in order.

Distributed Locking: The Shard Lock

With multiple replicas (EKS, two+ pods), two workers from different pods can pop from the same shard simultaneously - defeating the ordering guarantee. Each shard can only have one consumer cluster-wide at a time.

Enter the shard lock: a Redis key (swh:lock:shard:0) with a TTL-based lease. The worker acquires it via SETNX, then periodically renews it via a Lua script that atomically checks ownership before extending TTL:

-- renewShardLockScript: only renew if still the owner
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("PEXPIRE", KEYS[1], ARGV[2])
else
    return 0
end

The conditional renewal is critical - without it, a worker that lost its lock (e.g. network partition) would blindly extend the lock held by another replica, breaking the mutual-exclusion guarantee.

Lock Lifecycle

Phase	What Happens	TTL
Acquire	`SETNX lock:shard:0 instanceID`	15s
Renew (every 5s)	Lua: if GET == instanceID, PEXPIRE	15s
Release (shutdown)	Lua: if GET == instanceID, DEL	-
Orphan (crash)	TTL expires naturally	15s

The 5s renew interval with a 15s TTL gives three missed renewals before the lease expires - enough to tolerate a GC pause without losing ownership.

Idempotency: Slack Retries Are the Least of Your Worries

The original article covered Slack's 3-second retry. But there's a subtler case: what happens when the worker crashes after dequeueing but before storing to Redis?

The event is gone from the queue (BRPop removed it) but never processed. Slack doesn't retry because the HTTP handler already returned 200 OK.

The fix is two-layered:

Layer 1: Event ID deduplication. Before enqueueing, the handler tries SETNX swh:processed:{event_id} with a 1-hour TTL. If the key already exists, the event was already accepted (by this pod or another) - return 200 OK without enqueueing again.

func (h *Handler) markProcessed(ctx context.Context, eventID string) (duplicate bool, err error) {
    if eventID == "" {
        return false, nil  // no event_id = can't dedupe
    }
    key := constants.ProcessedEventKeyPrefix + eventID
    ok, err := h.redisClient.SetNX(ctx, key, 1, constants.IdempotencyTTL).Result()
    if err != nil {
        return false, err
    }
    return !ok, nil  // false = first time, true = already seen
}

Layer 2: Event backlog. The queue itself (Redis list) provides durability against pod restarts during processing. An event stays in the queue until a worker successfully processes it. The only gap is the DEAD-letter scenario - event dequeued, worker crashes mid-dispatch. That's a known limitation we accept (rare enough that manual replay from Slack's API is acceptable).

The Metrics Security Wake-Up Call

The /metrics endpoint exposes Prometheus counters with labels like channel, user_id, thread_id, and reaction. Those labels come straight from Slack payloads - and they leak sensitive information.

In the POC, metrics were wide open on the same port as the webhook handler. In production, /metrics is gated by an IP allowlist:

metricsGroup := r.Group("/")
metricsGroup.Use(middleware.IPAllowlist(cfg.MetricsAllowedCIDRs))
middleware.MountMetrics(metricsGroup)

The allowlist defaults to RFC1918 private ranges + loopback - safe for an EKS pod whose Prometheus collector scrapes via the VPC. But there's a trap: this only works if SetTrustedProxies is configured correctly.

Without trusted proxy configuration, Gin blindly trusts X-Forwarded-For headers. Any client on the internet can forge their source IP to match the allowlist and access /metrics. The fix is explicit:

trustedProxies := cfg.TrustedProxyCIDRs
if len(trustedProxies) == 0 {
    trustedProxies = middleware.DefaultInternalCIDRs
}
r.SetTrustedProxies(trustedProxies)

This tells Gin: only respect X-Forwarded-For from known proxies (ALBs, service mesh sidecars). Everything else gets c.ClientIP() from the actual TCP connection.

The Testing Epiphany

The POC had zero tests. The production code has ~5,000 lines of tests. The biggest lesson was which kind of test catches which kind of bug.

What I Learned About Test Levels

Layer	Tool	Catches
Unit (routing, validation)	stdlib `testing`	Logic errors, edge cases
Integration (HTTP → Redis)	testcontainers-go + real Redis 7-alpine	Wire bugs, Redis protocol differences
Worker (queue + locking)	miniredis + real Redis	Lock race conditions, Lua script errors
Middleware (rate limit, IP allowlist)	`httptest.NewRecorder`	Security bypasses

The most valuable tests were the integration tests that spin up a real Redis 7-alpine container per test via testcontainers-go:

type suite struct {
    t      *testing.T
    srv    *httptest.Server
    redis  *redis.Client
    rc     *testredis.Instance
    appURL string
}

func newSuite(t *testing.T) *suite {
    ti := testredis.Start(t)  // real container via testcontainers
    rc := redis.NewClient(&redis.Options{Addr: ti.Addr})
    srv := httptest.NewServer(newRouter(ctx, cfg, rc, rl))
    // ...
}

The comment in the codebase explains the decision:

miniredis is a faithful reimplementation for simple cases, but it doesn't implement BRPop with the same blocking semantics as real Redis. One worker test was consistently passing against miniredis and failing in staging. After eight hours of debugging, the root cause was a subtle difference in how miniredis handles concurrent blocking pops from the same list key. Real containers or nothing.

Kill Your Container Mid-Test

The test framework supports a Kill() method that terminates the Redis container mid-test, allowing tests for behavior during Redis outages:

func (i *Instance) Kill(t *testing.T) {
    i.mu.Lock()
    defer i.mu.Unlock()
    if err := i.container.Terminate(context.Background()); err != nil {
        t.Fatalf("testredis: kill container: %v", err)
    }
}

Tests that exercise the worker's reconnection loop with a live-and-then-dead Redis caught the "infinite retry spam" bug - when Redis goes away, the worker's BRPop error loop logs once per second forever. The fix was a backoff: sleepOrDone(ctx, time.Second) on error before retrying.

Graceful Shutdown: The Final Mile

The original handler had no shutdown logic. SIGTERM would kill the process mid-webhook, mid-goroutine, mid-Redis-write. The production version uses Go's signal.NotifyContext with proper context propagation:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()

// workers stop when ctx is done
h.StartWorkers(ctx, cfg.WorkerCount)

// server shutdown with 15s timeout
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
srv.Shutdown(shutdownCtx)

The workers check ctx.Done() at the top of every loop iteration. On shutdown, they finish their current event (up to redisOpTimeout), then exit. No half-processed events, no abandoned locks (the lock TTL handles orphaned locks if a node hard-crashes).

Config Structure: From Hardcoded to 12-Factor

The POC had constants scattered across files. The maintenance commit centralizes everything:

type Config struct {
    Port                int
    RedisURL            string
    RedisTLS            bool
    RedisCluster        bool
    RedisClusterAddrs   []string
    RedisOperationTimeout time.Duration
    ReadTimeout, WriteTimeout, IdleTimeout time.Duration
    WorkerCount         int
    MetricsAllowedCIDRs []string
    TrustedProxyCIDRs   []string
    SlackSigningSecret  string
    GinMode             string
    // ...
}

Every value comes from an environment variable with sensible defaults. Production mode (Gin release) requires SLACK_SIGNING_SECRET - the app refuses to start without it:

func (c *Config) IsProduction() bool {
    return c.GinMode == "release"
}

func (c *Config) Validate() error {
    if c.IsProduction() && c.SlackSigningSecret == "" {
        return errors.New("slack signing secret is required in production mode")
    }
    // ...
}

This caught a near-miss in staging: someone forgot to set the signing secret in the Helm values, and the app refused to start rather than silently accepting unauthenticated webhooks.

What's Still Missing

The queue gives durability against process restarts, but there's still a dead-letter gap: if a worker dequeues an event and crashes before fully processing it, that event is lost. The fix would be a Redis-backed scheduled retry queue with a visibility timeout (like SQS), but that's a future iteration.

Redis Cluster routing is also untested in CI - each integration test gets a single-node container. The shard-key hashing and lock-key prefixing might behave differently under CLUSTER with actual shard slots. We validated this manually in staging, but it deserves automated coverage.

Wrapping Up

The journey from go func() to sharded workers with distributed locks was driven by concrete production failures, not premature optimization:

Event loss → durable Redis queues
Duplicate metrics → event ID deduplication + idempotency keys
Out-of-order reactions → routing keys + sharded queues
Replica contention → distributed shard locks with Lua-based lease management
Security leaks → IP allowlist + trusted proxy configuration
Silent startup failures → config validation with production-mode guards
Redis outage chaos → graceful error handling with backoff
Undefined shutdown behavior → signal handling + context propagation

Each layer adds complexity but eliminates a real failure mode. The test suite grew 10:1 vs. production code, but every test justifies its line count by catching a bug that would otherwise need an incident post-mortem to discover.

The full codebase is at internal/handler/ in the repository - about 800 lines of production code and 3,000 lines of tests. The most important file is worker.go (290 lines): it contains the queue, the locks, the routing, and the graceful-shutdown logic. Everything else is plumbing around it.

Originally published at https://bard.sh/posts/slack-metrics-pt2/

Kill the Castle: How AWS IAM Replaces Network Perimeters

Bartłomiej Danek — Tue, 22 Sep 2026 18:35:50 +0000

The Premise: Network Location Is Not an Identity

In traditional "castle-and-moat" architectures, being inside a VPC or private subnet granted baseline trust. Requests from 10.0.0.0/8 were implicitly "safe." A compromised EC2 instance in the same subnet as an RDS database could query it freely because network adjacency was the authorization model.

Zero Trust inverts this completely:

Network location provides zero authorization. Every request-external, internal, same-node, same-pod-must be explicitly authenticated, authorized, and encrypted.

This is not a product you install. It's an architectural property that emerges from the disciplined combination of identity-centric policies, cryptographic attestation, microsegmented networking, layered data gates, and continuous verification. On AWS, implementing it requires working across five interlocking pillars.

Pillar 1: Identity as the Primary Perimeter

AWS IAM is the root policy engine. In a Zero Trust model, the enforcement boundary shifts from IP CIDRs to identity tokens and cryptographic attestation. The corollary: if it has long-lived credentials, it's a trust anchor you haven't eliminated yet.

Eliminating Long-Lived Credentials

Every static credential-IAM user access key, hardcoded database password, SSH private key checked into a repo-is a lateral movement vector. The Zero Trust mandate is absolute: no long-lived credentials anywhere in the system.

Workload Type	Anti-Pattern	Zero Trust Pattern
EC2 instances	Access keys in `~/.aws/credentials`	Instance profiles with IAM roles
ECS tasks	Shared task execution role	Per-task IAM roles via `taskRoleArn`
EKS pods	Node-level instance profile	EKS Pod Identity or IRSA (projected service account tokens)
CI/CD pipelines	Static "deployer" IAM user keys	OIDC federation directly to IAM roles
Human operators	IAM users with console passwords	IAM Identity Center (SSO) + external IdP + FIDO2/WebAuthn
Cross-account access	Shared credentials in Secrets Manager	`sts:AssumeRole` with external ID and session policies

OIDC Federation for CI/CD

This eliminates the single most common credential leak vector in modern AWS deployments-the CI/CD deployer key:

# Terraform: GitHub Actions OIDC provider
resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "github_deploy" {
  name = "github-actions-deploy"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          # Pin to specific repo AND branch - not org-wide
          "token.actions.githubusercontent.com:sub" = "repo:your-org/your-repo:ref:refs/heads/main"
        }
      }
    }]
  })
}

The sub claim pinning is critical. Without it, any repository in the GitHub organization can assume the role-a common misconfiguration that defeats the purpose entirely.

Context-Aware Authorization (ABAC)

Static IAM group membership (Developers, Admins) is a coarse authorization model. Attribute-Based Access Control (ABAC) evaluates request context dynamically.

Consider a fintech company where data engineers, ML engineers, and analysts all need S3 access-but to different buckets with different sensitivity levels. Instead of maintaining separate policies per team, one ABAC policy handles it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "ABACTeamAndClassificationMatch",
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::acme-datalake-*/*",
    "Condition": {
      "StringEquals": {
        "aws:PrincipalTag/team": "${s3:ResourceTag/team}",
        "aws:PrincipalTag/data-classification": "${s3:ResourceTag/data-classification}",
        "aws:PrincipalTag/cost-center": "${s3:ResourceTag/cost-center}"
      },
      "StringEqualsIfExists": {
        "aws:PrincipalTag/compliance-scope": "${s3:ResourceTag/compliance-scope}"
      },
      "Bool": {
        "aws:MultiFactorAuthPresent": "true"
      }
    }
  }]
}

Tag your IAM roles and S3 objects like this:

Principal (IAM Role)	`team`	`data-classification`	`cost-center`	`compliance-scope`
`role/ml-training-pipeline`	`ml-platform`	`confidential`	`cc-4200`	`soc2`
`role/analytics-dashboard`	`analytics`	`internal`	`cc-3100`	-
`role/fraud-detection-svc`	`ml-platform`	`restricted`	`cc-4200`	`pci-dss`

S3 Object Prefix	`team`	`data-classification`	`cost-center`	`compliance-scope`
`acme-datalake-features/`	`ml-platform`	`confidential`	`cc-4200`	`soc2`
`acme-datalake-reports/`	`analytics`	`internal`	`cc-3100`	-
`acme-datalake-txn-raw/`	`ml-platform`	`restricted`	`cc-4200`	`pci-dss`

The fraud-detection-svc role (tagged team=ml-platform, data-classification=restricted, compliance-scope=pci-dss) can access acme-datalake-txn-raw/ but not acme-datalake-reports/-wrong team, wrong classification, wrong cost center. The analytics-dashboard role cannot touch PCI-scoped data even if someone accidentally grants s3:GetObject on *, because the tag conditions will fail.

This single policy replaces potentially dozens of static resource-level policies. No MFA in the session? Denied regardless of tag match.

Session Policies: Runtime Permission Scoping

When assuming a role via STS, you can attach a session policy that further constrains the role's permissions for that specific session:

sts = boto3.client('sts')
response = sts.assume_role(
    RoleArn='arn:aws:iam::123456789012:role/ml-training-pipeline',
    RoleSessionName='feature-extraction-daily-2026-09-22',
    Policy=json.dumps({
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::acme-datalake-txn-raw/2026/09/22/*",
            "Condition": {
                "StringEquals": {
                    "s3:ResourceTag/data-classification": "restricted",
                    "s3:ResourceTag/compliance-scope": "pci-dss"
                }
            }
        }]
    }),
    Tags=[
        {"Key": "triggered-by", "Value": "airflow-dag-feature-extract"},
        {"Key": "run-id", "Value": "run-2026-09-22-0400"},
    ],
    DurationSeconds=900  # 15 minutes - just enough for the ETL window
)

The resulting session can only read today's PCI-scoped transaction partition, even though the underlying ml-training-pipeline role has broader S3 access. The session tags (triggered-by, run-id) flow into CloudTrail, giving auditors a direct link from data access back to the specific Airflow DAG run that triggered it. This is least privilege applied at the session level-each job gets exactly the permissions it needs for exactly the duration it needs them.

Pillar 2: Network Disintermediation & Microsegmentation

Zero Trust does not mean "networks don't matter." It means the network is hostile by default. Network controls act as guardrails and blast-radius containment, not trust anchors.

Eliminating Bastions and Open Ingress Ports

Every open inbound port is an attack surface. SSH (22) and RDP (3389) have been the entry point for countless breaches. AWS Systems Manager Session Manager replaces them entirely:

# Security group: ZERO inbound rules
resource "aws_security_group" "workload" {
  name_prefix = "zero-trust-workload-"
  vpc_id      = var.vpc_id

  # No ingress rules at all

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # HTTPS to AWS APIs only
    description = "SSM, CloudWatch, S3 (gateway endpoint preferred)"
  }
}

# SSM access is authenticated via IAM, logged to CloudTrail
resource "aws_iam_role_policy_attachment" "ssm" {
  role       = aws_iam_role.workload.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

Traffic tunnels over TLS through the AWS API plane. Authentication is IAM. Every session is logged to CloudTrail and optionally streamed to S3/CloudWatch. There is no SSH key to rotate, no bastion to patch, no port to scan.

Port forwarding through SSM handles the "but I need to reach the database" objection:

# Forward local port 5432 to RDS through SSM - no inbound SG rules needed
aws ssm start-session \
  --target i-0abc123def456 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["mydb.cluster-xyz.us-east-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["5432"]}'

VPC Lattice: Layer 7 Service Networking

VPC Lattice replaces the complexity of Transit Gateway routing tables, VPC peering, and perimeter firewalls with service-to-service Layer 7 proxying that enforces IAM authorization per HTTP request:

┌─────────────────────────────────────────────────────────┐
│                    VPC Lattice                          │
│                                                         │
│  ┌──────────┐    SigV4 Auth    ┌──────────┐            │
│  │ Service A ├────────────────►│ Service B │            │
│  │ (VPC-1)  │  IAM Policy      │ (VPC-2)  │            │
│  └──────────┘  Per-Request     └──────────┘            │
│                                                         │
│  Auth Policy:                                           │
│  ─ Principal matches role/service-a-prod                │
│  ─ HTTP method is GET or POST                           │
│  ─ Path matches /api/v2/*                               │
│  ─ Header X-Request-Source equals "internal"            │
└─────────────────────────────────────────────────────────┘

This is fundamentally different from network-level controls. A Security Group can allow TCP/443 between two CIDR ranges. VPC Lattice can allow GET /api/v2/users from role/service-a-prod while denying DELETE /api/v2/users from the same role. Authorization granularity moves from Layer 4 to Layer 7.

AWS PrivateLink: Eliminating Internet Transit

Every AWS API call that traverses the public internet is an interception opportunity. PrivateLink creates VPC interface endpoints that keep traffic entirely within the AWS backbone:

resource "aws_vpc_endpoint" "s3_interface" {
  vpc_id             = var.vpc_id
  service_name       = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type  = "Interface"
  subnet_ids         = var.private_subnet_ids

  # Endpoint policy: only this account's buckets, only from this VPC
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = "*"
      Action    = ["s3:GetObject", "s3:PutObject"]
      Resource  = "arn:aws:s3:::acme-datalake-*/*"
      Condition = {
        StringEquals = {
          "aws:PrincipalOrgID" = var.org_id
        }
      }
    }]
  })
}

Combined with a VPC endpoint policy, you enforce that even traffic flowing through the endpoint is scoped to specific buckets and organizational principals. This prevents a compromised workload from exfiltrating data to an attacker-controlled S3 bucket.

Egress Filtering: The Forgotten Half

Most Zero Trust discussions focus on ingress. Egress is where data actually leaves. A compromised container that can reach 0.0.0.0/0 on port 443 can exfiltrate to any HTTPS endpoint on the internet.

# Egress-only security group - whitelist, not blacklist
resource "aws_security_group" "strict_egress" {
  name_prefix = "strict-egress-"
  vpc_id      = var.vpc_id

  # AWS APIs via VPC endpoints only
  egress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.vpc_endpoints.id]
    description     = "AWS APIs via PrivateLink"
  }

  # Internal service mesh
  egress {
    from_port       = 8443
    to_port         = 8443
    protocol        = "tcp"
    security_groups = [aws_security_group.mesh_services.id]
    description     = "mTLS mesh traffic"
  }

  # DNS
  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "udp"
    cidr_blocks = [var.vpc_cidr]
    description = "VPC DNS resolver"
  }

  # Everything else: DENIED by default
}

Combine this with AWS Network Firewall or DNS Firewall for domain-level egress filtering when HTTPS inspection is required.

Pillar 3: Workload Identity & East-West Traffic

Within Kubernetes (EKS) or containerized architectures, east-west traffic inside the same VPC, same node, or even the same pod network namespace is untrusted. A compromised sidecar container must not be able to impersonate the primary application container.

Mutual TLS (mTLS)

Unencrypted east-west traffic is a passive eavesdropping vector. mTLS provides both wire encryption and cryptographic identity verification for every service-to-service hop:

┌──────────────┐                          ┌──────────────┐
│   Service A  │                          │   Service B  │
│              │     mTLS Handshake       │              │
│  ┌────────┐  │◄────────────────────────►│  ┌────────┐  │
│  │Envoy   │  │  1. Present client cert  │  │Envoy   │  │
│  │Sidecar │  │  2. Verify server cert   │  │Sidecar │  │
│  │        │  │  3. Check SPIFFE ID      │  │        │  │
│  └────────┘  │  4. Encrypted channel    │  └────────┘  │
└──────────────┘                          └──────────────┘
         │                                         │
         └──────── Certificates from ──────────────┘
                   SPIRE / Istio CA / 
                   ACM Private CA

Implementation choices and their trade-offs:

Approach	Pros	Cons
Istio (Envoy-based mesh)	Full L7 policy, traffic management, observability	High resource overhead (~100-200MB per sidecar), latency added per hop
Linkerd	Lighter weight, Rust-based proxy	Smaller ecosystem, fewer L7 features
VPC Lattice	AWS-native, no sidecar overhead	Limited to AWS, less granular than mesh L7 policies
Cilium (eBPF-based)	Kernel-level, minimal latency impact	Requires newer kernels, steeper learning curve

EKS Pod Identity: Per-Pod IAM Roles

The anti-pattern: all pods on a node share the node's EC2 instance profile. If any pod is compromised, the attacker inherits the union of all IAM permissions that node needs.

EKS Pod Identity (the successor to IRSA) provides distinct IAM roles per Kubernetes service account:

# Kubernetes ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-processor
  namespace: payments

---
# EKS Pod Identity Association (via Terraform)
# This binds the K8s SA to a specific IAM role

resource "aws_eks_pod_identity_association" "payment_processor" {
  cluster_name    = aws_eks_cluster.main.name
  namespace       = "payments"
  service_account = "payment-processor"
  role_arn        = aws_iam_role.payment_processor.arn
}

resource "aws_iam_role" "payment_processor" {
  name = "eks-payment-processor"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "pods.eks.amazonaws.com"
      }
      Action = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
}

# This role can ONLY access the payments DynamoDB table
resource "aws_iam_role_policy" "payment_processor" {
  role = aws_iam_role.payment_processor.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:Query"]
      Resource = "arn:aws:dynamodb:*:*:table/payments-*"
    }]
  })
}

Now the payment-processor pod has only DynamoDB access to payment tables. A compromised logging sidecar on the same node cannot access payment data because it runs under a different service account with a different IAM role.

SPIFFE/SPIRE: Cross-Cloud Workload Identity

For hybrid or multi-cloud deployments, cloud-provider-specific identity (IRSA, Pod Identity, GCP Workload Identity) creates lock-in and doesn't work across boundaries. SPIFFE (Secure Production Identity Framework for Everyone) defines a universal workload identity standard:

SPIFFE ID: spiffe://production.example.com/ns/payments/sa/payment-processor

┌────────────────┐        ┌──────────┐        ┌────────────────┐
│  EKS Workload  │        │  SPIRE   │        │  GCP Workload  │
│                │◄──────►│  Server  │◄──────►│                │
│  SVID Cert:    │        │          │        │  SVID Cert:    │
│  5min TTL      │        │  Central │        │  5min TTL      │
│                │        │  Attestor│        │                │
└────────────────┘        └──────────┘        └────────────────┘

SPIRE issues short-lived X.509 certificates (SVIDs) with 5-minute TTLs. Workloads present these for mTLS authentication. The certificate's SPIFFE ID is the identity, not the IP address or cloud IAM role.

Signed Artifact Verification

Zero Trust extends to the software supply chain. An admission controller must verify that container images are:

Built by your CI/CD pipeline (not pushed manually)
Cryptographically signed (Sigstore/Cosign, AWS Signer)
Scanned and clean (no critical CVEs above threshold)

# Kyverno ClusterPolicy: Enforce image signatures
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "123456789012.dkr.ecr.*.amazonaws.com/*"
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----
          attestations:
            - type: https://cosign.sigstore.dev/attestation/vuln/v1
              conditions:
                - all:
                    - key: "{{ scanner.result.criticalCount }}"
                      operator: LessThanOrEquals
                      value: "0"

An unsigned image or one with critical vulnerabilities is rejected at admission time-before it ever runs.

Pillar 4: Dual-Gate Data Protection

This is where Zero Trust gets architecturally interesting. A single compromised IAM role should not be sufficient to access sensitive data. You need orthogonal policy gates that require independent authorization:

Request ──► [Gate 1: Network]    ──► [Gate 2: IAM Policy]    ──► [Gate 3: Resource Policy] ──► [Gate 4: KMS Key Policy] ──► Data
             VPC Endpoint             sts:AssumeRole              S3 Bucket Policy                CMK Decrypt
             aws:sourceVpce           Principal perms             Explicit Allow                  Explicit Allow

All four gates must independently allow the request. Compromise of any single gate is insufficient.

Implementation: S3 Bucket with Triple-Gate Protection

# Gate 1: S3 bucket policy - requires specific VPC endpoint AND org
resource "aws_s3_bucket_policy" "sensitive_data" {
  bucket = aws_s3_bucket.sensitive.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "DenyNonVPCE"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:*"
        Resource = [
          aws_s3_bucket.sensitive.arn,
          "${aws_s3_bucket.sensitive.arn}/*"
        ]
        Condition = {
          StringNotEquals = {
            "aws:sourceVpce" = aws_vpc_endpoint.s3.id
          }
        }
      },
      {
        Sid       = "DenyOutsideOrg"
        Effect    = "Deny"
        Principal = "*"
        Action    = "s3:*"
        Resource = [
          aws_s3_bucket.sensitive.arn,
          "${aws_s3_bucket.sensitive.arn}/*"
        ]
        Condition = {
          StringNotEquals = {
            "aws:PrincipalOrgID" = var.org_id
          }
        }
      },
      {
        Sid       = "AllowAuthorizedRoles"
        Effect    = "Allow"
        Principal = {
          AWS = var.authorized_role_arns
        }
        Action = ["s3:GetObject"]
        Resource = "${aws_s3_bucket.sensitive.arn}/*"
      }
    ]
  })
}

# Gate 2: KMS key policy - independent authorization
resource "aws_kms_key" "data_key" {
  description             = "CMK for sensitive data encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "KeyAdministration"
        Effect    = "Allow"
        Principal = { AWS = "arn:aws:iam::${var.account_id}:role/SecurityAdmin" }
        Action    = ["kms:Create*", "kms:Describe*", "kms:Enable*", "kms:List*",
                     "kms:Put*", "kms:Update*", "kms:Revoke*", "kms:Disable*",
                     "kms:Get*", "kms:Delete*", "kms:ScheduleKeyDeletion"]
        Resource  = "*"
      },
      {
        Sid       = "KeyUsage"
        Effect    = "Allow"
        Principal = { AWS = var.authorized_role_arns }
        Action    = ["kms:Decrypt", "kms:GenerateDataKey"]
        Resource  = "*"
        Condition = {
          StringEquals = {
            "kms:ViaService"     = "s3.${var.region}.amazonaws.com"
            "aws:PrincipalOrgID" = var.org_id
          }
        }
      }
    ]
  })
}

# Gate 3: Enforce SSE-KMS on all objects
resource "aws_s3_bucket_server_side_encryption_configuration" "sensitive" {
  bucket = aws_s3_bucket.sensitive.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.data_key.arn
    }
    bucket_key_enabled = true
  }
}

Why this matters: An attacker who compromises an IAM role with s3:GetObject permissions still cannot read the data unless:

They are calling from within the designated VPC endpoint (network gate)
The S3 bucket policy explicitly allows their principal (resource policy gate)
The KMS key policy grants them kms:Decrypt (encryption gate)
Their principal belongs to the correct AWS Organization (organizational gate)

Compromise of credentials alone is insufficient. The attacker must also be in the right network position, which dramatically raises the difficulty.

Pillar 5: Continuous Verification & Blast Radius Containment

Trust is never binary or permanent. It is continuously evaluated and immediately revoked when anomalies are detected.

Automated Credential Revocation Pipeline

GuardDuty Finding ──► EventBridge Rule ──► Lambda ──► Revoke STS Sessions
(Anomalous API call)   (Pattern match)    (Automated)  (Immediate)

# Lambda: Revoke all active sessions for a compromised IAM role
import boto3
import json
from datetime import datetime, timezone

iam = boto3.client('iam')

def handler(event, context):
    detail = event['detail']
    finding_type = detail['type']

    # GuardDuty findings indicating credential compromise
    compromise_indicators = [
        'UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration',
        'UnauthorizedAccess:IAMUser/TorIPCaller',
        'Discovery:S3/MaliciousIPCaller',
        'UnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom',
    ]

    if finding_type not in compromise_indicators:
        return

    # Extract the compromised principal
    resource = detail['resource']
    access_key_id = resource.get('accessKeyDetails', {}).get('accessKeyId')
    principal_id = resource.get('accessKeyDetails', {}).get('principalId')
    user_name = resource.get('accessKeyDetails', {}).get('userName')

    # Revoke all sessions by applying an inline deny-all policy
    # with a time condition that invalidates tokens issued before now
    revocation_policy = {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*",
            "Condition": {
                "DateLessThan": {
                    "aws:TokenIssueTime": datetime.now(timezone.utc).isoformat()
                }
            }
        }]
    }

    # Apply to the role (for assumed-role principals)
    role_name = extract_role_name(principal_id)
    if role_name:
        iam.put_role_policy(
            RoleName=role_name,
            PolicyName='EmergencySessionRevocation',
            PolicyDocument=json.dumps(revocation_policy)
        )
        print(f"Revoked all sessions for role: {role_name}")

    # Notify security team
    sns = boto3.client('sns')
    sns.publish(
        TopicArn='arn:aws:sns:us-east-1:123456789012:security-alerts',
        Subject=f'CREDENTIAL COMPROMISE: {role_name or user_name}',
        Message=json.dumps(detail, indent=2, default=str)
    )

def extract_role_name(principal_id):
    """Extract role name from principal ID (AROA...)"""
    if not principal_id:
        return None
    # For assumed roles, principal_id format: AROAEXAMPLE:session-name
    # Look up the role via the access key or other detail
    # Implementation depends on your naming conventions
    pass

The aws:TokenIssueTime condition is the critical mechanism. STS tokens cannot be individually revoked. Instead, you attach an inline policy that denies all actions for tokens issued before the current timestamp. All existing sessions are immediately invalidated. New sessions (post-revocation) are unaffected.

Service Control Policies: Immutable Guardrails

SCPs are non-negotiable boundaries that even account administrators cannot override:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PreventCloudTrailDisable",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/OrganizationSecurityAudit"
        }
      }
    },
    {
      "Sid": "PreventKMSKeyDeletion",
      "Effect": "Deny",
      "Action": [
        "kms:ScheduleKeyDeletion",
        "kms:DisableKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/KMSAdministrator"
        }
      }
    },
    {
      "Sid": "EnforceIMDSv2",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringNotEquals": {
          "ec2:MetadataHttpTokens": "required"
        }
      }
    },
    {
      "Sid": "DenyPublicS3",
      "Effect": "Deny",
      "Action": [
        "s3:PutBucketPublicAccessBlock",
        "s3:PutAccountPublicAccessBlock"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "s3:PublicAccessBlockConfiguration/BlockPublicAcls": "true"
        }
      }
    }
  ]
}

Key SCPs for Zero Trust:

Prevent CloudTrail tampering - an attacker who disables logging can operate undetected
Enforce IMDSv2 - IMDSv1 is an SSRF-to-credential-theft vector
Prevent KMS key deletion - destroying encryption keys is an irreversible data destruction attack
Block public S3 - no bucket should ever be publicly accessible in a Zero Trust environment

IAM Access Analyzer: Continuous Permission Auditing

IAM Access Analyzer identifies two classes of risk:

External access: Resources (S3 buckets, KMS keys, SQS queues, Lambda functions) with policies granting access to principals outside your AWS Organization
Unused access: IAM roles and users with permissions they haven't exercised in 90+ days

resource "aws_accessanalyzer_analyzer" "org_level" {
  analyzer_name = "organization-analyzer"
  type          = "ORGANIZATION"

  # Generates findings for any resource accessible outside the org
}

resource "aws_accessanalyzer_analyzer" "unused_access" {
  analyzer_name = "unused-access-analyzer"
  type          = "ACCOUNT_UNUSED_ACCESS"

  configuration {
    unused_access {
      unused_access_age = 90
    }
  }
}

Feed findings into Security Hub, and set up automated remediation for high-severity findings.

The Reality Check: Common Failure Modes

1. The "Network Doesn't Matter" Trap

An overcorrection where engineers ignore VPC routing, Security Group discipline, and egress filtering because "IAM and TLS handle security."

Why this fails: If an application-layer token leaks (JWT in a log, STS credential in a stack trace), network controls are your last line of defense. A VPC endpoint condition in the S3 bucket policy means stolen credentials are useless outside your VPC. Egress filtering prevents a compromised container from reaching a C2 server. Defense-in-depth requires both identity and network controls.

2. Policy Complexity Explosion

A 200-microservice architecture where each service has its own IAM role, KMS key access, S3 bucket policy entry, VPC Lattice auth policy, and Kubernetes NetworkPolicy generates thousands of policy documents. Without automation, this becomes:

Unauditable: No human can review 2,000 IAM policies for correctness
Brittle: A single typo in a resource ARN causes a production outage
Slow: Adding a new service requires changes in 5+ policy systems

Mitigation: Policy-as-code with automated generation. Use Terraform modules, OPA/Rego policy validation, and CI/CD pipelines that lint and test IAM policies before deployment.

3. The Shared Role Shortcut

The most common Zero Trust failure in practice. Teams adopt Zero Trust on paper but run all pods under one or two broad IAM roles because managing per-service roles is operationally expensive.

# What the architecture diagram shows:
Pod A → Role A (scoped to DynamoDB)
Pod B → Role B (scoped to S3)
Pod C → Role C (scoped to SQS)

# What actually runs in production:
Pod A → shared-role (DynamoDB + S3 + SQS + KMS + SNS + ...)
Pod B → shared-role
Pod C → shared-role

Mitigation: Automated IAM policy generation from application code analysis. Tools like iamlive can record actual AWS API calls during development and generate least-privilege policies. Pair with IAM Access Analyzer's unused access findings to continuously right-size permissions.

4. Performance Impact of Per-Request Auth

Enforcing SigV4 authentication on high-throughput east-west microservice calls adds CPU overhead (HMAC-SHA256 computation) and latency (additional auth lookups). At 10,000 RPS between two services, this is measurable.

Mitigation:

Connection reuse: mTLS amortizes the TLS handshake cost over many requests on a persistent connection
Token caching: STS credentials are cached for their lifetime (typically 1 hour); not fetched per-request
Benchmarking: VPC Lattice and mesh proxies must be load-tested against your latency SLOs before production deployment
Selective enforcement: Not all east-west traffic requires the same level of verification. Health checks and metrics scraping may use simpler auth

Maturity Model: Where to Start

Zero Trust is a spectrum, not a binary state. A practical adoption path:

Level	Focus	Key Actions
L0: Foundations	Credential hygiene	Eliminate IAM users, enforce MFA, enable CloudTrail, IMDSv2 everywhere
L1: Perimeter Hardening	Ingress/egress control	SSM Session Manager, VPC endpoints for AWS services, strict SG egress
L2: Workload Identity	Per-workload IAM	EKS Pod Identity, OIDC for CI/CD, per-service IAM roles
L3: Data Protection	Dual-gate access	S3 bucket policies + KMS key policies with VPC endpoint conditions
L4: East-West Encryption	mTLS mesh	Service mesh or VPC Lattice for all inter-service traffic
L5: Continuous Verification	Automated response	GuardDuty → automated revocation, IAM Access Analyzer, Config rules

Most organizations find substantial security improvement at L2-L3 before tackling the operational complexity of L4-L5.

Closing Thought

Zero Trust on AWS is not a product, a checkbox, or a one-time migration. It's the sustained engineering discipline of treating every component-including your own infrastructure-as potentially compromised. The five pillars reinforce each other: identity without network controls is insufficient, network controls without identity are bypassable, and both without continuous verification give you a false sense of security.

The hardest part is not the technology. AWS provides all the primitives. The hardest part is the organizational discipline to maintain fine-grained policies at scale without collapsing back to shared roles and broad permissions under delivery pressure.

Originally published at https://bard.sh/posts/kill-the-castle/

The 5W Method for DevOps and Platform Engineering

Bartłomiej Danek — Fri, 29 May 2026 13:45:11 +0000

The 5W Method for DevOps and Platform Engineering

Why most DevOps communication fails

Every engineer has seen it: a Slack message that says "API is broken" a Jira ticket that says "upgrade Kubernetes," or a pull request description that says "fix infra." These statements feel productive but carry almost no useful information. They force everyone else to ask follow-up questions, guess context, or worse - act on incomplete understanding.

The 5W method fixes this. It is a structured thinking tool borrowed from journalism and root cause analysis that forces clarity before action. In DevOps, where unclear communication leads directly to outages, failed changes, and wasted engineering time, it is one of the simplest high-impact habits you can adopt.

What is the 5W method?

Five questions. Answer them before you write an incident report, file a change request, plan a task, or escalate a problem.

Question	What it means in DevOps
What?	What happened, or what needs to change?
Why?	Why is it needed? Why did it happen? What is the business impact?
Who?	Who is affected? Who owns it? Who needs to approve?
Where?	Which environment, service, cluster, region, account, repository, or pipeline?
When?	When did it start? When should it be done? How urgent is it?

Some teams add a sixth - How? - making it 5W1H. The "How" covers implementation approach, rollback plan, and verification steps.

The order depends on the situation

The 5W is not a rigid sequence. The entry point changes depending on whether you are reacting or planning:

During an incident, you start with What happened? and work toward the fix. During planning, you start with Why do we need this? and work toward execution. The questions are the same - the flow is not.

Example 1: Incident analysis

Without 5W:

"API is broken."

This tells you nothing. You don't know what is broken, who is affected, or how urgent it is.

With 5W:

What?

Users cannot log in. /auth/login returns HTTP 500.

Why?

Initial suspicion: database connection errors following the latest deployment.

Who?

Affected: all users in production
Owner: backend team
Support impact: customer support is receiving tickets

Where?

Production EKS cluster
Service: auth-api
Region: eu-central-1
Database: production PostgreSQL

When?

Started around 09:40
First alert fired at 09:43
Last deployment happened at 09:35

Now anyone reading this - an on-call engineer, a manager, a support agent - immediately understands the situation and can act.

Example 2: Change request

Without 5W:

"We need to upgrade Kubernetes."

This will sit in a backlog forever because nobody knows the scope, urgency, or ownership.

With 5W:

What?

Upgrade EKS clusters from version 1.33 to 1.34.

Why?

Current version is approaching end of support. We need access to newer security patches and compatibility fixes.

Who?

Platform team owns the upgrade
Application teams need to validate their workloads
Security team should be informed

Where?

All non-production clusters first
Production clusters after validation

When?

Dev clusters: this week
Staging: next week
Production: after successful validation and rollback plan approval

This change request is now reviewable and approvable without a single follow-up question.

Example 3: Platform engineering task

Without 5W:

"CI is too slow, let's fix it."

With 5W + How:

What?

Reduce average CI pipeline duration from 18 minutes to below 10 minutes.

Why?

Slow feedback delays developers and increases context switching. It affects every engineer on every push.

Who?

Platform team owns the pipeline
Backend and frontend teams are affected daily

Where?

GitHub Actions workflows across backend and frontend repositories.

When?

Target: before the next release cycle. Priority: high - it affects all engineers.

How?

Add dependency caching
Split slow jobs into parallel stages
Run tests concurrently
Avoid rebuilding unchanged Docker layers
Measure pipeline duration before and after to validate improvement

The "How" turns a vague improvement idea into a concrete, measurable plan.

Where to apply the 5W method

Use it anywhere DevOps work needs to be communicated clearly:

Incident reports - so responders have full context immediately
Postmortems - so root causes are described precisely
Change requests - so reviewers can approve without guessing
Architecture decisions - so trade-offs are explicit
Deployment plans - so rollout scope and rollback are clear
RFCs and design docs - so proposals are complete
Debugging summaries - so handoffs between shifts work
Production readiness reviews - so nothing is assumed
Prioritization discussions - so urgency is justified, not felt

A reusable template

Copy this into your wiki, your incident template, or your change request form:

## 5W DevOps Analysis

### What?
What happened or what do we want to change?

### Why?
Why does it matter? What problem are we solving? What is the impact if we do nothing?

### Who?
Who is affected? Who owns it? Who needs to approve?

### Where?
Which service, environment, cluster, region, account, repository, or pipeline?

### When?
When did it happen? When should it be fixed? How urgent is it?

### How? (optional)
How will we solve it? What is the rollback plan? How will we verify success?

The practical rule

Use 5W before implementation. Every time.

Bad DevOps flow:

"Let's change Terraform and see what happens."

Better DevOps flow:

"What are we changing, why, who is affected, where does it apply, when does it need to happen, and how do we verify it worked?"

The first approach leads to random fixes, unclear ownership, and risky production changes. The second takes 5 minutes of thinking and prevents hours of cleanup.

Final thought

The 5W method is not a process framework or a methodology. It is a thinking habit. It costs nothing to adopt, requires no tooling, and works whether you are a junior engineer writing your first incident report or a CTO reviewing a platform migration plan.

The best DevOps teams are not the ones with the most sophisticated tools. They are the ones that communicate clearly under pressure. Five questions is all it takes.

Originally published at https://bard.sh/posts/5w-method-devops/

[Boost]

Bartłomiej Danek — Mon, 04 May 2026 18:41:00 +0000

Bartłomiej Danek

May 4

hcl-linter 0.0.1-alpha is out

#terraform #hcl #terragrunt #devops

3 min read

hcl-linter 0.0.1-alpha is out

Bartłomiej Danek — Mon, 04 May 2026 18:32:55 +0000

hcl-linter 0.0.1-alpha is out

terraform fmt only touches whitespace. In a codebase with dozens of contributors and hundreds of HCL files, block order drifts, naming conventions split, and required fields go missing quietly. I wanted something that could catch and fix that automatically - so I built hcl-linter.

The first run fixed over 1000 terragrunt.hcl files.

getting started

Two commands to go from nothing to a first pass:

hcl-linter init             # scaffold .hcl-linter/ from your existing files
hcl-linter fix ./ --dry-run # preview what would change

init walks the project, groups files by basename, and writes a .hcl-linter/default.hcl plus one extends = "default" override per unique filename. No config needed for a first formatting pass either - --format applies opinionated defaults without any config:

hcl-linter fix ./ --format --dry-run

block order

Wrong block order is the most common drift. block_order catches and auto-fixes it:

# violation - terraform before include
terraform {
  source = "git::https://github.com/example/vpc.git"
}

include "root" {
  path = find_in_parent_folders()
}

dependency "vpc" {
  config_path = "../vpc"
}

After hcl-linter fix:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://github.com/example/vpc.git"
}

dependency "vpc" {
  config_path = "../vpc"
}

Config:

rules {
  block_order {
    enabled = true
    order   = ["include", "locals", "terraform", "dependency", "inputs"]
  }
}

naming

Hyphens in block labels break dependency.my-vpc.outputs.id references in locals. name_validation flags and auto-fixes them - hyphens replaced with underscores, references in locals updated too:

# violation
dependency "my-vpc" {
  config_path = "../vpc"
}

dependency "db-primary" {
  config_path = "../database"
}

rules {
  name_validation {
    enabled = true
    pattern = "^[a-z][a-z0-9_]*$"
    blocks  = ["dependency", "include"]
  }
}

required fields

required_fields catches missing attributes when a block is present and auto-adds them:

# violation - expose missing on include blocks
include "root" {
  path = find_in_parent_folders()
}

include "vpc" {
  path = "../vpc"
}

After fix:

include "root" {
  path   = find_in_parent_folders()
  expose = true
}

include "vpc" {
  path   = "../vpc"
  expose = true
}

rules {
  required_fields {
    include {
      expose = true
    }
  }
}

count and for_each

count_for_each warns on patterns that silently disable resources - count = 0 or for_each = {} left behind after feature flags, and count + for_each used on the same block (Terraform rejects this at plan time):

resource "aws_instance" "web" {
  count = 0         # WARNING: resource will not be created
  ami   = "ami-12345"
}

resource "aws_db_instance" "db" {
  count    = 1
  for_each = var.db_configs  # ERROR: cannot use both count and for_each
  engine   = "postgres"
}

Not fixable - count = 0 is sometimes intentional. The rule flags it so the intent is explicit.

rules {
  count_for_each {
    enabled                = true
    warn_on_count_zero     = true
    warn_on_empty_for_each = true
    warn_on_conflict       = true
  }
}

get_env without default

hcl_functions warns on get_env() calls with no fallback - those fail silently in CI if the variable is unset. It also validates that find_in_parent_folders() targets exist on disk:

locals {
  env  = get_env("MY_SECRET")                          # WARNING: no default
  root = find_in_parent_folders("nonexistent.hcl")     # ERROR: file not found
  port = get_env("PORT", "8080")                       # ok
}

rules {
  hcl_functions {
    enabled                       = true
    find_in_parent_folders_exists = true
    get_env_has_default           = true
  }
}

deprecated terraform hooks

Terragrunt renamed before_hook / after_hook to before_hooks / after_hooks a while back. terraform_block catches the old names still in use:

terraform {
  source = "git::https://github.com/example/vpc.git"

  before_hook "run_fmt" {      # WARNING: deprecated, use before_hooks
    commands = ["terragrunt"]
    execute  = ["terraform", "fmt"]
  }
}

rules {
  terraform_block {
    enabled              = true
    no_deprecated_fields = true
  }
}

dependency outputs

dependency_outputs statically validates that dependency.x.outputs.y references exist in the target module - no plan needed:

dependency "vpc" {
  config_path = "./mock-vpc"
}

inputs = {
  vpc_id     = dependency.vpc.outputs.vpc_id       # ok
  bad_output = dependency.vpc.outputs.nonexistent  # WARNING: not declared
}

Walks the dependency chain, parses output blocks from .tf files. Supports .mock-outputs.json for modules where upstream state is not available locally.

rules {
  dependency_outputs {
    enabled = true
  }
}

CI

check fails the build on any rule violation:

hcl-linter check ./infra

fix --dry-run catches everything check does, plus byte-level drift from fixable rules - if someone committed without running fix:

hcl-linter fix ./infra --dry-run

Both are composable: run check for semantic issues and fix --dry-run for formatting drift as separate steps.

install

go install github.com/bard-works/hcl-linter@latest

Or grab a binary from releases:

curl -L https://github.com/bard-works/hcl-linter/releases/latest/download/hcl-linter_linux_amd64.tar.gz | tar xz

Source and full docs: bard-works/hcl-linter

Originally published at https://bard.sh/posts/hcl-linter-release/

Beyond Slack Analytics: Building Custom Engagement Metrics with Webhooks, Prometheus, and Grafana

Bartłomiej Danek — Tue, 28 Apr 2026 13:01:24 +0000

TL;DR: Slack's native analytics won't tell you which threads die, which requests get ignored, or how yyour team's collaboration patterns evolve. I built a Go webhook handler that captures every reaction, reply, and thread via Slack's event API, exports it to Prometheus, and visualizes it in Grafana - giving me metrics Slack never will.

The Wake-Up Call

It started with a post-incident review. We'd just recovered from a 3-hour outage, and I was asked the usual questions: How fast did we respond? Which channels were involved? Were there warning signs in our internal Slack discussions before things broke?

I opened Slack's analytics dashboard. Member counts. Message volume. Channel growth. Vanity metrics that told me nothing about the incident or our response patterns.

I couldn't answer basic questions: How many infrastructure requests were sitting unresolved in our #platform-requests channel? Who was drowning in context-switching across too many active threads? Had we missed early warning signs buried in reaction patterns on code review requests?

Worse, I realized I was guessing about team health. I'd see a busy Slack day and assume we were productive. I'd see green checkmark reactions and assume requests were getting fulfilled. But I had no data - just gut feel and fragmented chat history.

Then came the compliance gap. A terraform apply failed in staging, and someone replied with "skipping, will fix later" - then applied the change manually to unblock themselves. It showed up as a casual thread reply with a 😱 reaction from a teammate, buried under 30 messages. No alert, no metric, no visibility - until I went looking for it during the post-mortem and found the octopus reaction someone had wisely added to flag the IaC violation.

That's when I realized: Slack is where my team actually works, but Slack's analytics treat it like a broadcast channel instead of a workflow engine. Every reaction, every thread, every reply is a data point. I just wasn't collecting them.

So I built something to fix that.

Why Slack Won't Tell You What You Need to Know

Slack gives you member counts, message volume, and channel growth. That's it. If you're running a platform engineering team, a DevOps org, or an internal support channel, those numbers are vanity metrics.

What you actually need to know:

Which threads get resolved vs. abandoned?
How fast does the team respond to infrastructure requests?
Which code review requests get ignored (and why)?
Are people actually engaging with your announcements?

Slack doesn't expose this. But Slack's event subscriptions do - if you know how to catch them.

Architecture: How I Capture What Slack Hides

The system has three layers: Slack events → Go webhook handler → Prometheus + Grafana.

The Webhook Handler

Built in Go 1.26 with Gin, the handler listens at POST /api/slack/events. Every event goes through this middleware stack:

Middleware	Purpose	Notes
Recovery	Panic recovery
RequestID	Unique ID per request
GinLogger	Structured JSON logging
RequestSizeLimit	Reject >1MB
RateLimiter	10 req/s per IP, burst 20	Handles Slack's event batching
SlackVerification	HMAC-SHA256 signature check

The burst 20 setting is critical - Slack's Event API sometimes batches multiple events into a short window, so a single IP can legitimately spike above 10 req/s. The burst allows those spikes without dropping legitimate events.

The three event types we care about:

1. message events - catch new threads and replies:

// internal/handler/slack.go (simplified)
if event.ThreadTs == "" || event.ThreadTs == event.Ts {
    // Top-level message → new thread
    metrics.RecordNewThread(ctx, channel, event.Ts)
} else {
    // Reply in thread
    metrics.RecordThreadReply(ctx, channel, event.ThreadTs, event.User)
}

2. reaction_added / reaction_removed - the signal layer:

// Reactions are stored in Redis + recorded as metrics
metrics.RecordThreadReaction(ctx, channel, event.Item.Ts, event.User, event.Reaction)
metrics.RecordUserEngagement(ctx, channel, event.Item.Ts, event.User)

3. url_verification - Slack's challenge-response during app setup.

The Slack Retry Problem

Slack expects a 200 OK within 3 seconds. If your handler is busy writing to Redis or the Prometheus exporter is slow, Slack will retry the event - and you'll get duplicate counts in your metrics.

The fix: acknowledge first, process asynchronously. Return 200 OK immediately, then handle the event in a goroutine:

// internal/handler/slack.go (simplified)
func (h *Handler) HandleEvent(c *gin.Context) {
    var event SlackEvent
    if err := c.ShouldBindJSON(&event); err != nil {
        c.JSON(400, gin.H{"error": err.Error()})
        return
    }

    // Acknowledge immediately - don't block on Redis/Prometheus
    c.JSON(200, gin.H{"ok": true})

    // Process in background
    go func() {
        ctx := context.Background()
        if event.ThreadTs == "" || event.ThreadTs == event.Ts {
            h.metrics.RecordNewThread(ctx, event.Channel, event.Ts)
        } else {
            h.metrics.RecordThreadReply(ctx, event.Channel, event.ThreadTs, event.User)
        }
    }()
}

This prevents Slack's retry mechanism from creating duplicate metric increments.

Why Reactions Are the Secret Signal

I don't just count messages. I track reactions as semantic signals:

Emoji	Reaction Name	Meaning	Metric Label
✅	white_check_mark	Request completed	`reaction="white_check_mark"`
😱	scream	Multiple active threads per person (overload)	`reaction="scream"`
🚫	no_entry	Irrelevant code review request	`reaction="no_entry"`
🔁	repeat	Repeating / duplicate request	`reaction="repeat"`
❌	x	Irrelevant request	`reaction="x"`
✔️	done	Completed (no PR review needed)	`reaction="done"`
🐙	octopus	Infrastructure-as-Code violation	`reaction="octopus"`

This turns Slack into a structured workflow tool - reactions become queryable metrics.

Getting the Team on Board (Social Engineering)

This only works if people actually use the reactions. I didn't mandate it - instead, I led by example: I started reacting to threads with octopus when I saw IaC violations, and with scream when I was drowning in threads. Within a week, the team adopted it naturally. The key is making it feel like a helpful shorthand, not a tracking mechanism. If the team doesn't use the emoji, your metrics are useless - so make it useful for them first.

Metric Definitions and Component Mapping

I define 5 OpenTelemetry counter metrics in internal/metrics/slack.go. The Prometheus exporter appends _total, giving the double suffix you see in PromQL queries - you might want to fix it in your OTEL Agent.

Metric-to-Component Mapping

Code Name	Labels	Triggered By
`webhook_requests_total`	`channel`, `status`	Every incoming webhook
`threads_total`	`channel`	New top-level message
`thread_replies_total`	`channel`, `user_id`	Reply in thread
`thread_reactions_total`	`channel`, `user_id`, `reaction`	Reaction added
`user_engagement_total`	`channel`, `user_id`	Reply OR reaction

The Prometheus Cardinality Bomb

If you're tempted to add thread_id as a label - don't. In my original POC, I made this mistake. In Prometheus, every unique combination of label values creates a new time series. With thousands of Slack threads (each with a unique thread_id), you're creating thousands of time series. This is a cardinality bomb that will explode your Prometheus memory usage and can crash the instance.

The fix: Keep Prometheus for aggregate channel-level counters only. Thread-level detail belongs in Redis (which I'm already using) or a SQL database, not in Prometheus labels. If you want the "Longest Thread" feature, query Redis directly - don't pollute your metrics with high-cardinality labels. Prometheus is for metrics, not event logging.

Key Performance Indicators

The dashboard is organized into four insight layers, each answering a different question about how your Slack channels actually operate:

Volume & Throughput - Are we drowning or cruising?
Total Threads shows raw request load per day, making it obvious which days are release days, incident spikes, or quiet periods. Active Threads (threads with at least one reply) reveals engagement depth - a high thread count with low reply rate means people are posting but not discussing, a signal of announcements drowning out conversation.

Resolution Signals - Are requests actually getting done?
The green panels tell the completion story. white_check_mark reactions track formal completions - infrastructure requests fulfilled, reviews done. done reactions capture quick wins that never needed a PR. Meanwhile, x reactions surface canceled or irrelevant requests, helping you spot scope creep or misrouted asks. Together, they give you a resolution rate without ever asking anyone to fill out a form.

Quality & Friction - Where is the process breaking?
The red and yellow panels are your early warning system. A spike in scream reactions means people are drowning in threads - too many active requests per person. no_entry on code reviews exposes recurring low-quality submissions from specific contributors. repeat reactions highlight documentation gaps - the same question asked three times means the answer isn't findable. octopus flags manual infrastructure changes that bypass your IaC pipeline, a compliance risk you'd otherwise miss.

Time-Based Insights - When is the team really working?
The blue panels surface patterns hidden by aggregate daily stats. After-Hours Work quantifies overtime - if 30% of threads start after 17:00, you have an on-call problem, not a "dedicated team" problem. Longest Thread with a clickable Slack link lets you jump directly to the most contentious discussion - usually a design debate or a stuck incident. Busiest Day aggregates threads by date to reveal your actual rhythm: Tuesday releases? Friday incident spikes? The data tells the story.

Business-Hour Awareness

One panel tracks requests outside 10:00-17:00 (browser time):

sum_over_time(
  (
    sum(
      increase(threads_total{channel=~"$channel_filter"}[5m])
      and on()
        ((hour() < 10) or (hour() >= 17))
    )
  )[$__range:5m]
)

This tells us: is the team working after hours? If so, why?

Note: The hour() function in Prometheus uses UTC. If your team is in a different timezone (e.g., CET/Poland), adjust the offsets: 10:00 CET = 08:00 UTC, 17:00 CET = 15:00 UTC. Adjust the < 10 / >= 17 values based on where your Prometheus server is running.

Longest Thread Detection

Since I removed thread_id from Prometheus labels (see The Prometheus Cardinality Bomb), the "Longest Thread" feature works differently: I pull thread IDs directly from Redis, where high-cardinality data lives safely. The Grafana dashboard queries Redis for the top threads by reply count, then overlays Prometheus aggregate trends for context.

To implement this, use a Grafana Infinity datasource or Redis datasource plugin to query:

LRANGE replies:{channel}:* 0 -1

Then match the longest list to its channel and thread_ts. The clickable Slack link is built from these Redis keys:

https://yourworkspace.slack.com/archives/{channel}/p{thread_ts_without_dot}

Alternatively, if you keep thread_id in Prometheus for a small team (few threads), the PromQL would be:

topk(
  1,
  sum by (thread_id, channel) (
    max_over_time(
      thread_replies_total_total{
        channel=~"$channel_filter"
      }[$__range]
    )
  )
)

With a Data link override pointing to https://yourworkspace.slack.com/archives/${channel}/p${__cell}.

Busiest Day Detection

sum(
  increase(threads_total{channel=~"$channel_filter"}[1d])
)

Grafana transformations convert the timestamp to YYYY-MM-DD format and sort by value.

Dependencies: What We Chose and Why

go.mod (direct dependencies)
├── github.com/gin-gonic/gin v1.11.0         # HTTP routing
├── go.opentelemetry.io/otel v1.40.0         # Metrics SDK
├── go.opentelemetry.io/otel/exporters/prometheus v0.62.0  # Prometheus export
├── go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0
├── github.com/prometheus/client_golang v1.23.2  # Prometheus client
├── github.com/redis/go-redis/v9 v9.17.3     # Redis client (cluster support)
├── golang.org/x/time v0.14.0                # Rate limiting
└── github.com/alicebob/miniredis/v2 v2.36.1 # In-memory Redis for tests

Key Trade-offs

Choice	Why	Trade-off
OpenTelemetry over raw Prometheus client	Future-proof, OTLP export option	More boilerplate setup
Redis for thread storage	Fast lookups, TTL support	Extra dependency, network hop
Gin over stdlib	Fast dev, built-in middleware	Framework lock-in
Counter metrics only	Simple, append-only	No histograms for latency (yet)
Fly.io deployment	Simple, cheap, EU region (waw)	Less control than K8s

What I Skipped

Histograms: I don't track webhook processing latency. The handler is fast enough (<10ms) that it hasn't mattered yet.
Gauge metrics: No "current active threads" gauge. I rely on increase() queries instead.
Thread cancellation: No separate metric needed - canceled threads are tracked via thread_reactions_total{reaction="x"} (the ❌ emoji). The Grafana dashboard queries reactions with reaction="x" to surface irrelevant/canceled requests.

Redis Data Structures

The handler stores thread context in Redis for enrichment (not just metrics):

Key Pattern	Type	Content
`reactions:{channel}:{ts}`	Hash	Field: `{reaction}:{user}`, Value: JSON with timestamps
`messages:{channel}:{ts}`	String	JSON: user, text, ts, channel
`replies:{channel}:{thread_ts}`	List	JSON entries for each reply
`processed_events:{event_id}`	String	TTL 5min - idempotency key for Slack retries

Event Deduplication

Slack retries events if it doesn't get a 200 OK fast enough. To prevent double-counting metrics, the handler uses the event_id as an idempotency key with SETNX:

// Before processing, check if we already handled this event
key := fmt.Sprintf("processed_events:%s", slackEvent.EventID)
// SetNX returns true if key was SET (new event), false if key already existed
processed, _ := rdb.SetNX(ctx, key, "1", 5*time.Minute).Result()
if !processed {
    // Event already processed - skip to prevent double-counting
    return
}

This ensures that even if Slack sends the same event twice during a network blip, your metrics are only incremented once.

Lessons Learned

1. Reactions Are Better Than Thread Replies for Signals

A reply requires typing. A reaction is one click. I get 3× more signal from reactions than replies because the friction is lower.

2. Grafana Transformations Are Powerful But Fragile

The "Longest Thread" panel uses 4 transformations: sortBy, limit, convertFieldType, formatTime, and renameByRegex. One breaks, the panel dies. Keep transformations minimal where possible.

3. Rate Limiting Is Essential

Slack's event replay will flood you if your handler was down. Our rate limiter (10 req/s per IP) prevents thundering herd on restart.

Wrapping Up: What You Get Out of This

Building custom Slack metrics isn't about dashboards for the sake of dashboards. It's about making the invisible visible. Here's what this setup gives you after a few weeks of data:

You stop guessing about team health. A quick glance at the KPI row tells you: are we keeping up (green), drowning in context-switching (scream spikes), or fielding the same question repeatedly (repeat reactions)? These aren't vanity metrics - they're leading indicators of burnout, documentation gaps, and process breakdowns.

You catch compliance and quality issues early. That octopus reaction on a thread? It's someone manually changing infrastructure instead of using Terraform. The no_entry on a code review? A recurring contributor quality problem you can address directly. Without this signal, those issues stay hidden in chat history until they become outages.

You understand your real working patterns. The after-hours panel quantifies overtime without timesheets. The longest thread view surfaces your actual bottlenecks - the design debates and stuck incidents that deserve retrospectives. The busiest day chart reveals whether your release rhythm is working or just creating spikes.

You get all of this without asking anyone to change behavior. No new forms, no status updates, no "please label your threads." People just use Slack the way they always have - the reactions and replies they'd make anyway become the data. That's the real win: observability that emerges from existing workflows, not another process to maintain.

The stack is lightweight: a Go handler (~200 lines), Redis for context, Prometheus for storage, Grafana for visualization. But the output is a window into your team's actual collaboration patterns - the stuff Slack's analytics won't show you, and the stuff you need if you want to lead a team effectively.

One final note: the working POC took me a few hours to build - without AI assistance. Not because I'm unusually fast, but because this isn't actually hard when you understand how the pieces fit together. Slack gives you webhooks. Prometheus takes counters. Grafana draws panels. The real skill isn't writing the code - it's recognizing that the data you need is already flowing through your tools, you just haven't built the bridge between them yet. Sometimes the best observability projects are the ones where you stop waiting for a vendor feature and wire up the integration yourself.

Originally published at https://bard.sh/posts/slack-metrics/

Terraform Modules: Composition Over Abstraction

Bartłomiej Danek — Mon, 27 Apr 2026 13:19:36 +0000

Terraform Modules: Composition Over Abstraction

Terraform makes it easy to build large, highly abstracted modules that try to solve everything in one place. At first glance, this feels efficient: fewer modules, fewer calls, less wiring.

In practice, that approach often creates more problems than it solves.

A better pattern-especially as infrastructure grows-is to design small, focused, “atomic” modules and combine them using composition.

What We Mean by Composition

In this context, composition means:

Building infrastructure by combining smaller, independent modules instead of hiding everything behind a single, all-in-one module.

Instead of:

one module doing everything internally

you have:

multiple modules wired together explicitly

module "role" { ... }

module "policy" { ... }

module "attachment" {
  role   = module.role.name
  policy = module.policy.arn
}

This is not just a stylistic choice-it directly impacts maintainability, safety, and clarity.

The Problem with “Do-It-All” Modules

A common anti-pattern is a module that:

creates multiple IAM roles
generates multiple policies
attaches them
conditionally enables features via flags
supports multiple unrelated use cases

Example:

module "irsa" {
  source = "./modules/irsa"

  roles = {
    service_a = {
      policies = ["s3", "dynamodb"]
    }
    service_b = {
      policies = ["sqs"]
    }
  }
}

This looks convenient, but introduces several issues.

Hidden Coupling

All resources share one lifecycle.

A small change:

can affect unrelated roles
may trigger unnecessary diffs
increases risk during apply

Poor Reusability

“Generic” modules often become:

too opinionated for some use cases
too complex for others

Consumers either:

fight the interface
or reimplement logic elsewhere

Hard-to-Review Changes

A simple modification may:

touch multiple resources
impact different logical paths

This makes it difficult to answer:

What will this change actually do?

Large Blast Radius

Terraform operates at the module/state level.

Large modules lead to:

bigger plans
slower applies
harder rollbacks
increased risk

Atomic Modules: A Better Approach

“Atomic” does not mean artificially small.

It means:

A module should represent a single logical responsibility.

Examples:

iam-role
iam-policy
iam-role-policy-attachment

Each module:

has a clear purpose
exposes a minimal interface
can be reused independently

Example: IRSA - Monolith vs Composition

Monolithic Approach

module "irsa" {
  source = "./modules/irsa"

  roles = {
    service_a = {
      policies = ["s3", "dynamodb"]
    }
  }
}

Problems:

tightly coupled lifecycle
unclear ownership
difficult to modify safely

Composed Approach

Create role

module "service_a_role" {
  source = "./modules/iam-role"

  name = "service-a"
  assume_role_policy = data.aws_iam_policy_document.irsa.json
}

Create policy

module "service_a_s3_policy" {
  source = "./modules/iam-policy"

  name   = "service-a-s3"
  policy = data.aws_iam_policy_document.s3.json
}

Attach policy

module "service_a_attach_s3" {
  source = "./modules/iam-role-policy-attachment"

  role_name  = module.service_a_role.name
  policy_arn = module.service_a_s3_policy.arn
}

Why Composition Works Better

Clear Ownership

Each resource:

is defined explicitly
belongs to a specific consumer

Safer Changes

Updating a policy:

affects only that policy
does not impact unrelated roles

Better Reusability

You can:

reuse policies across roles
attach policies flexibly
compose behavior without modifying modules

Easier Debugging

Failures are easier to trace because:

modules are small
responsibilities are clear

Composition vs Abstraction

These are often confused.

Approach	Focus	Trade-off
Abstraction	Simplicity of usage	Hidden complexity, rigidity
Composition	Flexibility, clarity	More explicit wiring

Monolithic modules favor abstraction.

Atomic modules favor composition.

Trade-offs of Composition

This approach is not free.

More Module Calls

You will write more blocks.

This increases verbosity, but also improves clarity.

More Explicit Wiring

You pass outputs between modules.

This is intentional:

dependencies are visible
behavior is predictable

Risk of Over-Fragmentation

Splitting everything blindly leads to:

unnecessary complexity
modules that are never used independently

Example of going too far:

separating tightly coupled resources that must always exist together

Practical Rule of Thumb

If a resource can be safely changed, reused, or destroyed independently, it should likely be its own module.

If not, keep it together.

When Larger Modules Still Make Sense

There are valid cases for bigger modules:

tightly coupled infrastructure (e.g., VPC with subnets and routing)
opinionated platform layers
internal “productized” infrastructure

Even then:

avoid “god modules”
keep boundaries clear
prefer internal composition

Key Insight

The real shift is this:

Move complexity from inside modules to between modules.

Monolith → implicit complexity
Composition → explicit complexity

In infrastructure systems, explicit complexity is easier to manage over time.

Summary

Prefer composition over monolithic abstraction
Design modules with single responsibilities
Keep dependencies explicit and visible
Accept some verbosity in exchange for safety and clarity

The goal is not smaller modules.

The goal is predictable, composable, and low-risk infrastructure.

Originally published at https://bard.sh/posts/terraform-modules-composition-over-abstraction/

Smart DNS: Auto-Detecting Route53 Records in Terraform

Bartłomiej Danek — Mon, 27 Apr 2026 12:15:06 +0000

Smart DNS: Auto-Detecting Route53 Records in Terraform

Every Route53 record in plain Terraform means a zone lookup, an explicit type, and for load balancers - a separate data source just to wire the alias.

data "aws_route53_zone" "this" {
  name = "api.example.com."
}

data "aws_lb" "api" {
  name = "api-nlb"
}

resource "aws_route53_record" "api" {
  zone_id = data.aws_route53_zone.this.zone_id
  name    = "api.example.com"
  type    = "A"

  alias {
    name                   = data.aws_lb.api.dns_name
    zone_id                = data.aws_lb.api.zone_id
    evaluate_target_health = true
  }
}

With a module that infers everything, the same record becomes:

inputs = {
  name    = "api.example.com"
  records = "api-nlb-1234567890.elb.eu-west-1.amazonaws.com"
}

zone auto-detection

name = "api.example.com"
        │
        └─ strip first label
                │
                ▼
        "example.com"  ──► aws_route53_zone lookup

The module splits the FQDN, drops the first label, queries Route53 for the hosted zone. No zone_id from the caller.

For apex records or non-standard cases, zone_override skips this.

type inference

records value
      │
      ├─ type_override set? ───────────────► use that type (NS/MX/TXT/etc.)
      │
      ├─ is map?
      │     ├─ has "name" key? ────────────► alias A  (lookup LB by name)
      │     └─ has "tags" key? ────────────► alias A  (lookup LB by tags)
      │
      ├─ is string or list?
      │     ├─ all values are IPv4? ───────► A
      │     ├─ ends with .amazonaws.com? ──► alias A  (lookup LB by hostname)
      │     └─ else ───────────────────────► CNAME
      │
      └─ (nothing created if no branch matches)

Each branch maps to a separate aws_route53_record resource using for_each over a conditional set - only one fires per invocation.

lb alias lookup modes

Mode	Input	How
By hostname	`records = "my-nlb-xxx.amazonaws.com"`	parse LB name from hostname prefix
By name	`records = { nlb = { name = "my-nlb" } }`	`aws_lb` data source by name
By tags	`records = { alb = { tags = {...} } }`	`aws_lb` data source by tag filter

All three produce an alias block with evaluate_target_health = true.

escape hatches

Scenario	Solution
NS delegation	`type_override = "NS"`
MX at zone apex	`type_override = "MX"` + `zone_override = "example.com"`
TXT verification	`type_override = "TXT"`
Private hosted zone	`private_zone = true`
Custom TTL	`ttl = 300`

the idea

DNS configuration is mostly mechanical - you know the hostname or IP address(es), you know what it points to. The type, the zone, whether it needs an alias block - those are derivable. This module pushes that derivation into Terraform so callers don't repeat it.

The tradeoff is explicitness. Plain Terraform is verbose, but every field is visible and validated. The module is terse, but inference can surprise you - especially the records = "..." string that silently becomes an alias lookup against AWS instead of a literal CNAME value.

It works best when:

you manage many records across consistent naming conventions
all your LBs follow predictable hostname patterns (*.<region>.elb.amazonaws.com)
your zones map cleanly to second-level domains (one zone per domain, no split-horizon edge cases)

It gets awkward when:

you need fine-grained TTL control per record
you have private hosted zones with the same name as public ones (need private_zone = true consistently)
your LB names are not stable - tag-based lookup is more resilient there

trade-off

records is type = any. Terraform cannot validate at plan time. A typo in a map key (naam instead of name) silently falls through to CNAME instead of alias. Review plan output before apply - the wrong record type shows up as a different resource being created.

full example

# CNAME
inputs = {
  name    = "api.example.com"
  records = "old-api.example.com"
}

# A record
inputs = {
  name    = "api.example.com"
  records = ["203.0.113.10", "203.0.113.11"]
}

# Alias to NLB by hostname
inputs = {
  name    = "api.example.com"
  records = "api-nlb-1234567890.elb.eu-west-1.amazonaws.com"
}

# Alias to ALB by tags
inputs = {
  name    = "api.example.com"
  records = {
    alb = {
      tags = {
        Name = "api-alb"
        Env  = "prod"
      }
    }
  }
}

# NS delegation with zone override
inputs = {
  name          = "example.com"
  records       = ["ns1.example.net", "ns2.example.net"]
  type_override = "NS"
  zone_override = "example.com"
}

Originally published at https://bard.sh/posts/route53-auto-detecting-terraform/

What is Terragrunt and why you need it

Bartłomiej Danek — Fri, 24 Apr 2026 09:43:55 +0000

What is Terragrunt and why you need it

the multi-env problem

With plain Terraform, managing multiple environments means duplicating config - same modules, different variable files, same backend boilerplate repeated everywhere.

environments/
├── dev/
│   ├── main.tf        # copy
│   ├── variables.tf   # copy
│   └── backend.tf     # copy with different key
├── staging/
│   ├── main.tf        # copy
│   ├── variables.tf   # copy
│   └── backend.tf     # copy with different key
└── prod/
    ├── main.tf        # copy
    ├── variables.tf   # copy
    └── backend.tf     # copy with different key

Any change to the module requires updating all three.

what Terragrunt does

Terragrunt is a thin wrapper around Terraform that adds:

DRY backend config - define once, inherit everywhere
dependency blocks - wire modules together without a monolith
run-all - apply/plan/destroy across multiple modules in one command
generate blocks - auto-generate files (provider configs, backend configs)
mock_outputs - unblock plan in CI when dependencies haven't been applied yet

terraform vs terragrunt

	Terraform	Terragrunt
Backend config	per-module	inherited from root
Multi-module apply	manual	`run-all`
Module wiring	`terraform_remote_state`	`dependency` block
DRY config	variables only	full config inheritance

installation

asdf plugin-add terragrunt
asdf install terragrunt 0.67.0
asdf local terragrunt 0.67.0

or developer.hashicorp.com/terraform/tutorials/automation/terragrunt

what the structure looks like

Instead of duplicated .tf files, you get one root.hcl and small terragrunt.hcl files per unit that only define what's different:

root.hcl
environments/
├── dev/
│   ├── vpc/
│   │   └── terragrunt.hcl   # 5 lines: include + inputs
│   ├── rds/
│   │   └── terragrunt.hcl
│   └── eks/
│       └── terragrunt.hcl
└── prod/
    ├── vpc/
    │   └── terragrunt.hcl
    ├── rds/
    │   └── terragrunt.hcl
    └── eks/
        └── terragrunt.hcl

A typical leaf terragrunt.hcl:

include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "git::https://github.com/org/tf-modules.git//vpc?ref=v1.2.0"
}

inputs = {
  name       = "prod-vpc"
  cidr_block = "10.0.0.0/16"
}

The backend, provider, and locking config all come from root.hcl - no repetition.

See folder structure and include for the full root.hcl setup.

Originally published at https://bard.sh/posts/what-is-terragrunt/

Terragrunt run-all

Bartłomiej Danek — Fri, 24 Apr 2026 09:43:14 +0000

Terragrunt run-all

Runs a Terraform command across all units in the current directory tree, respecting dependency order.

cd environments/prod
terragrunt run-all plan
terragrunt run-all apply
terragrunt run-all destroy   # reverse order

dependency-aware ordering

environments/prod/
├── vpc/
├── rds/          # depends on vpc
└── eks/          # depends on vpc

run-all apply applies vpc first, then rds and eks in parallel.

real CI pipeline

# .github/workflows/deploy.yml
- name: Plan all
  run: terragrunt run-all plan --terragrunt-non-interactive
  working-directory: environments/prod

- name: Apply all
  run: terragrunt run-all apply --terragrunt-non-interactive -auto-approve
  working-directory: environments/prod
  if: github.ref == 'refs/heads/main'

Detect drift in CI without applying:

# exit code 2 = changes detected, 0 = no changes, 1 = error
terragrunt run-all plan \
  --terragrunt-non-interactive \
  -detailed-exitcode

filtering

# skip a unit
terragrunt run-all apply --terragrunt-exclude-dir environments/prod/rds

# target a single unit and its dependencies
terragrunt run-all apply --terragrunt-include-dir environments/prod/eks

parallelism

# default is unlimited parallel - cap it if you hit API rate limits
terragrunt run-all apply --terragrunt-parallelism 4

output

Each unit prefixes its own output:

[environments/prod/vpc] Apply complete! Resources: 12 added.
[environments/prod/rds] Apply complete! Resources: 4 added.
[environments/prod/eks] Apply complete! Resources: 31 added.

Originally published at https://bard.sh/posts/terragrunt-run-all/

Terragrunt remote_state and generate blocks

Bartłomiej Danek — Fri, 24 Apr 2026 09:43:09 +0000

Terragrunt remote_state and generate blocks

`remote_state`

Defines the backend once in root.hcl - Terragrunt auto-generates the backend config for every unit that includes it.

# root.hcl
remote_state {
  backend = "s3"
  config = {
    bucket         = "my-tf-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

path_relative_to_include() resolves to the path of the calling terragrunt.hcl relative to root.hcl - e.g. environments/prod/vpc - giving each unit a unique state key automatically.

`generate` block

Writes any file before running Terraform - typically used for provider.tf and backend.tf so they don't need to be duplicated in every module.

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "eu-west-1"

  default_tags {
    tags = {
      ManagedBy   = "terraform"
      Environment = "prod"
    }
  }
}
EOF
}

`if_exists` options

overwrite_terragrunt - overwrite only if Terragrunt generated the file (safe default)
overwrite - always overwrite
skip - skip if file exists
error - error if file exists

generating multiple files

generate "versions" {
  path      = "versions.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
EOF
}

Originally published at https://bard.sh/posts/terragrunt-remote-state/

Terragrunt folder structure and include

Bartłomiej Danek — Fri, 24 Apr 2026 09:42:28 +0000

Terragrunt folder structure and include

Terragrunt projects split config into two layers: a root hcl file with shared settings, and per-unit terragrunt.hcl files that inherit from it.

root.hcl
environments/
├── dev/
│   ├── vpc/
│   │   └── terragrunt.hcl
│   ├── rds/
│   │   └── terragrunt.hcl
│   └── eks/
│       └── terragrunt.hcl
└── prod/
    ├── vpc/
    │   └── terragrunt.hcl
    ├── rds/
    │   └── terragrunt.hcl
    └── eks/
        └── terragrunt.hcl

Each leaf terragrunt.hcl is one Terraform unit - its own state file, its own apply.

`include` - inherit from root

# environments/prod/vpc/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "git::https://github.com/org/tf-modules.git//vpc?ref=v1.2.0"
}

inputs = {
  name       = "prod-vpc"
  cidr_block = "10.0.0.0/16"
}

root.hcl

# root.hcl
locals {
  env    = basename(dirname(get_terragrunt_dir()))
  region = "eu-west-1"
}

remote_state {
  backend = "s3"
  config = {
    bucket         = "my-tf-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = local.region
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.region}"
}
EOF
}

`find_in_parent_folders`

Walks up the directory tree until it finds the file - no hardcoded paths needed.

include "root" {
  path = find_in_parent_folders()          # finds root.hcl by default
}

include "root" {
  path = find_in_parent_folders("root.hcl") # explicit name
}

Originally published at https://bard.sh/posts/terragrunt-folder-structure/

DEV Community: Bartłomiej Danek

Slack Webhooks in Production: Durable Queues, Distributed Locking, and the Testing Epiphany

The go func() Betrayal

Architecture: What Changed

The Sharding Problem

Distributed Locking: The Shard Lock

Lock Lifecycle

Idempotency: Slack Retries Are the Least of Your Worries

The Metrics Security Wake-Up Call

The Testing Epiphany

What I Learned About Test Levels

Kill Your Container Mid-Test

Graceful Shutdown: The Final Mile

Config Structure: From Hardcoded to 12-Factor

What's Still Missing

Wrapping Up

Kill the Castle: How AWS IAM Replaces Network Perimeters

The Premise: Network Location Is Not an Identity

Pillar 1: Identity as the Primary Perimeter

Eliminating Long-Lived Credentials

OIDC Federation for CI/CD

Context-Aware Authorization (ABAC)

Session Policies: Runtime Permission Scoping

Pillar 2: Network Disintermediation & Microsegmentation

Eliminating Bastions and Open Ingress Ports

VPC Lattice: Layer 7 Service Networking

AWS PrivateLink: Eliminating Internet Transit

Egress Filtering: The Forgotten Half

Pillar 3: Workload Identity & East-West Traffic

Mutual TLS (mTLS)

EKS Pod Identity: Per-Pod IAM Roles

SPIFFE/SPIRE: Cross-Cloud Workload Identity

Signed Artifact Verification

Pillar 4: Dual-Gate Data Protection

Implementation: S3 Bucket with Triple-Gate Protection

Pillar 5: Continuous Verification & Blast Radius Containment

Automated Credential Revocation Pipeline

Service Control Policies: Immutable Guardrails

IAM Access Analyzer: Continuous Permission Auditing

The Reality Check: Common Failure Modes

1. The "Network Doesn't Matter" Trap

2. Policy Complexity Explosion

3. The Shared Role Shortcut

4. Performance Impact of Per-Request Auth

Maturity Model: Where to Start

Closing Thought

The 5W Method for DevOps and Platform Engineering

The 5W Method for DevOps and Platform Engineering

Why most DevOps communication fails

What is the 5W method?

The order depends on the situation

Example 1: Incident analysis

What?

Why?

Who?

Where?

When?

Example 2: Change request

What?

Why?

Who?

Where?

When?

Example 3: Platform engineering task

What?

Why?

Who?

Where?

When?

How?

Where to apply the 5W method

A reusable template

The practical rule

Final thought

[Boost]

hcl-linter 0.0.1-alpha is out

hcl-linter 0.0.1-alpha is out

hcl-linter 0.0.1-alpha is out

getting started

block order

The `go func()` Betrayal