DEV Community: Mohammad Jawad (Kasir) Barati

Monorepo vs. Polyrepo

Mohammad Jawad (Kasir) Barati — Thu, 10 Sep 2026 01:12:57 +0000

I had an technical debate about when one should "just put it all in one monorepo", and honestly at that time I was on the side of monorepo for the project in question but did not know how to reason about it. I mean I was not sure how it pays off. So that is why I did a little bit of thinking and realized we do not need to answer "are these projects related?". Almost everything in a product is related 😉.

Rather the question is: do these projects share a toolchain, and does that sharing actually reduce friction?

Here I decided to use my two repos from the same product to make a useful A/B test:

smart-novel is managed by Nx monorepo and it houses the frontend and backend
smart-novel-beatrice is a standalone Python service (Beatrice), split out on purpose.

Why Frontend and Backend Live Together

Both apps in smart-novel are TypeScript. That single fact cascades into a lot of shared infrastructure:

One dependency graph, one lockfile, one version of TypeScript/ESLint/Prettier across both apps.
Nx can build, test, and lint only what changed, across app boundaries, in a single command.
Shared types and utilities can be imported directly instead of published as packages.
One CI pipeline, one set of environment conventions, one place to look for config.

None of this requires the frontend and backend to be conceptually simple or tightly coupled, it requires them to speak the same toolchain. Nx's whole value proposition is coordinating a graph of packages that already share a runtime and build system. Put differently: the monorepo isn't paying for "frontend and backend are part of the same product", it's paying for "frontend and backend are both TypeScript projects that Nx can reason about together."

Why Beatrice Doesn't

Beatrice is Python, it comes with pyproject.toml, uv.lock. It has its own linters, test runner, CI/CD. None of that has anything in common with Nx's build graph. Folding it into the Nx repo wouldn't add coordination, it would add friction:

Nx has nothing to orchestrate for a Python package. It's dead weight in the workspace config.
A Python contributor now has to understand an Nx workspace just to find pyproject.toml.
CI has to be able to work with a Python project inside a workspace tuned for another language's tooling.
Versioning and release cadence for a Python service don't necessarily track the frontend/backend release cadence anyway.

So the separation isn't a judgment that Beatrice is "less important" or unrelated to the product, it's that gluing it to a toolchain built for a different language fights the tooling rather than helping iteration speed.

The Actual Criteria

Boiled down, the questions that decide "same repo or separate repo" are:

Do they share a build toolchain and language runtime? If yes, a monorepo tool like Nx can add real value; shared dependency graphs, affected-only builds, one lint/format config. If no, the tool has nothing to coordinate and just adds overhead.
Would merging them require one project to route around the other's tooling? If a Python service has to sit inside a Node build graph (or vice versa), you've added a foreign-language exception to every CI/CD script, editor config, and onboarding doc, that's cost, not synergy.
Do they release and version together in practice? Two apps deployed in lockstep benefit from atomic commits across both. Two services with independent release cadences don't need that coupling, and a shared repo can make it harder to see which one actually changed.
Does "related product" already imply "related tooling"? It often doesn't. Two parts of the same product can legitimately be written in different languages for good reasons (ML tooling in Python, application layer in TypeScript), that's a reason to keep them separate, not a coincidence to design around.

Observability & Telemetry Retention Policy

Mohammad Jawad (Kasir) Barati — Thu, 10 Sep 2026 00:11:16 +0000

tl;dr

Use OpenTelemetry (OTel) for application observability and start with Grafana Cloud Free as the backend. I love both of them. They offer everything you will be needing when you have a bug ticket.

Grafana Cloud Free currently provides 14-day retention for metrics, logs, traces, profiles, and k6 performance tests, with 50 GB each of logs and traces included. It is explicitly intended for personal projects and early-stage startups.

14 days is sufficient for most applications in beta/MVP. Incidents usually are investigated within hours or days, not months. Increase retention only when there is a demonstrated operational, business, security, or compliance need.

Think About Telemetry in Layers

Not all telemetry needs the same retention period.

                 ┌──────────────────────────┐
                 │      Long-lived data     │
                 │                          │
                 │ Metrics / trends         │
                 │ Audit & security events  │
                 │ Business-critical events │
                 └──────────────────────────┘
                              ▲
                              │
                       retain longer
                              │
                 ┌──────────────────────────┐
                 │     Short-lived data     │
                 │                          │
                 │ Application logs         │
                 │ Traces                   │
                 │ Debug information        │
                 └──────────────────────────┘

Initial policy

Telemetry	Retention	Rationale
Logs	14 days	Enough for normal debugging and incident investigation
Traces	14 days	Primarily useful for investigating recent requests/errors
Metrics	14 days initially	Sufficient during beta/MVP; increase later for long-term trends
Debug logs	As short as practical	High volume and usually low long-term value
Security/audit events	Separate policy	May require significantly longer retention
Business-critical events	Separate storage	Should not depend on observability retention

The goal is not to keep everything forever. The goal is to retain the information for as long as it is useful.

When Should Retention Increase

Increase retention when we have a concrete reason, for example:

A bug occurs less frequently than the current retention window.
We need to investigate incidents discovered weeks later.
We need historical performance/capacity trends.
Security or compliance requirements require longer retention.
The application becomes business-critical and historical investigation becomes important.

For example:

Beta:
  Logs/Traces ─────────────── 14 days

Growing production:
  Logs/Traces ─────────────── 30–90 days
  Metrics ─────────────────── 6–13+ months
  Audit/Security ──────────── separate policy

Compliance/security:
  Audit data ──────────────── potentially years

Do not automatically increase raw-log retention just because the application grows. Long-term trends are often better represented by metrics, while important audit/business events can be archived separately.

Avoid Vendor Lock-in

The application should never depend directly on a vendor-specific observability SDK or API.

Use:

Application
    │
    │ OpenTelemetry
    ▼
OTel Collector
    │
    ├──────────► Grafana Cloud
    │
    ├──────────► Honeycomb
    │
    └──────────► Other OTel backend

The OpenTelemetry Collector is specifically designed to receive, process, and export telemetry to one or more backends. This means changing providers should primarily be a Collector configuration/deployment change, rather than an application rewrite. Use OTLP, the standard OpenTelemetry protocol, for the application → Collector boundary.

Also avoid making provider-specific dashboards, alerts, queries, and metadata a critical part of the application architecture until there is a reason to commit to them.

Why OpenTelemetry?

OTel is a vendor-neutral, open-source observability framework for generating, collecting, and exporting logs, metrics, and traces. It is supported by a broad ecosystem of observability vendors.

Adopting OTel gives us:

Vendor portability.
Consistent telemetry semantics.
Logs ↔ traces ↔ metrics correlation.
Centralized sampling/filtering.
The ability to change observability backends later.
The option to send telemetry to multiple backends.

Principle:

Instrument once with OpenTelemetry. Choose the observability backend independently.

Some title

Mohammad Jawad (Kasir) Barati — Wed, 09 Sep 2026 18:14:44 +0000

We released XYZ feature

Watch

Watch on YouTube

A Database Design Mistake Which Shows itself in the API

Mohammad Jawad (Kasir) Barati — Mon, 07 Sep 2026 22:13:09 +0000

Imagine you have a ChapterContent table in PostgreSQL:

model ChapterContent {
  // ...
  content            String  @db.Text
  ttsFriendlyContent String? @map("tts_friendly_content") @db.Text

  contentHash String  @map("content_hash")
  ttsHash     String? @map("tts_hash")
  // ...
}

I initially though what an ingenious thing I did, two String columns, sitting next to each other, where the ttsFriendlyContent only makes sense as a function of content and that invariant lives entirely in application code, spread across codebase, instead of being enforced by the data model.

[!NOTE]

When I say "function" I mean it in the mathematical sense: ttsFriendlyContent = f(content), its value should be entirely determined by content (plus a bunch of rules we have for it, e.g. not using ~ for elongations). Given the same content, there's only one correct ttsFriendlyContent; it's not meant to be an independent value someone can set arbitrarily.

That's why storing it as its own free-standing column (rather than as a computed/derived artifact) is a design smell, the schema lets you set it to anything, even though semantically it should only ever be "whatever content maps to". And another relevant issue is maintaining it.

The Invariant Nobody can See

The rule I actually need enforced is simple to state and hard to find:

You must generate ttsFriendlyContent from content before you can generate narration audio from ttsFriendlyContent. You cannot set ttsFriendlyContent without content already existing. If content changes, ttsFriendlyContent becomes stale and narration built from it is now wrong too.

None of that is expressed by the schema. It's enforced (partially) by scattered checks:

ChapterService#updateContent#updateContent accepts content and ttsFriendlyContent as two independent, client-supplied strings in the same call, with a TODO admitting the obvious hole 😅:

  // TODO: validate the ttsFriendlyContent make sense (the content should match the TTS-friendly version)!

sequenceDiagram
    actor Client
    participant API as GraphQL API
    participant Service as ChapterService#updateContent
    participant Repo as ChapterContentRepository

    Client->>API: mutation (cnt: "He", ttsFriendlyCnt: "asd")
    Note over Client: Cannot use equality checks!

    API->>Service: updateContent(chapterId, content, ttsFriendlyContent)
    Note over Service: Already too late to validate

    Service->>Repo: upsertByChapterId(chapterId, content, ttsFriendlyContent)
    Note over Repo: Does NOT validates ttsFriendlyContent either

    Repo-->>Service: stored ✅ (silently wrong)
    Service-->>API: chapter
    API-->>Client: 😵 200 OK

You ask why we cannot use equality checks? Because one is suppose to the the text user sees while the other is a a machine-generated version of the content.

And just in case you are wondering why I wrote "too late to validate", I meant that validation should have been done when we received the request (DTO layer).

Right now there's exactly one door into updating ChapterContent's content and ttsFirendlyContent fields, the good thing is that client must send both fields every time, so the "set ttsFriendlyContent before content exists" scenario can't happen. It only becomes a real ordering problem if we add createChapter mutation: then a chapter could exist with content but no ttsFriendlyContent, and a later updateContent call would need to know whether it's being asked to set ttsFriendlyContent for the first time, edit content, or both! Drawing a line between them is costly and very much error-prone.

sequenceDiagram
    actor Client
    participant API as GraphQL API
    participant Svc as ChapterService

    Note over Client,Svc: Today - updateContent is the only mutation touching chapter_contents table, both args required together
    Client->>API: updateContent(id, content, ttsFriendlyContent)
    API->>Svc: writes both fields in one call - no way to send just one

    Note over Client,Svc: Hypothetical - if createChapter ships as its own mutation
    Client->>API: createChapter(content)
    API->>Svc: chapter row created, ttsFriendlyContent still null
    Client->>API: updateContent(id, content, ttsFriendlyContent)
    Note over Svc: this second call could also silently change content again

PrismaChapterContentRepository#upsertByChapterId recomputes contentHash from content on every write, but never touches ttsHash. The column exists, it's indexed (schema.prisma#L89), it's read back in the repository mapper, but nothing ever writes or compares it.

It's a staleness-detection mechanism that was designed and I did not get to finish it, but the point is that the two-fields-as-siblings shape doesn't naturally produce a "check if stale" step; you have to bolt it on and remember to call it everywhere.

ChapterNarrationService#startGeneration(chapterId, forceRegenerate) layers on Redis locking, in-flight HTTP-request cancellation, and a double-checked-locking re-read of the DB, all to answer one question safely: is the narration we're about to build actually derived from the current content? This is a lot of concurrency machinery compensating for the fact that the data model can't say "this narration is derived from commit X of this content" on its own.

Why this is Hard to Maintain

None of these four pieces is complicated in isolation. The problem is that the actual business rule, content → tts-friendly text → narration, each derived and invalidated by the previous step is nowhere written down as a single flow and they are scattered across multiple mutations. Miss any one piece when adding a new mutation path or changing one, and you get either a rejected valid request or worse narration audio generated from stale text, with no error at all.

Here's the actual authoring workflow that demonstrates the crux of the matter:

flowchart TD
    A["author writes content"] --> B["updateContent - content stored, contentHash recomputed"]
    B --> C["generateTtsFriendlyText - preview only, not persisted"]
    C --> D["updateContent again - ttsFriendlyContent persisted"]
    D --> E["startGeneration - narration built from ttsFriendlyContent"]

The flow is of mutations is far apart in time and they all work on the same content and ttsFriendlyContent fields. It is very easy to break the flow and introduce bugs when developing new features, fixing a bug, or refactoring.

The Potential Fixes

The textbook answer here is derived/generated columns or a version/lineage table which stores content, and content_version, then the tts_friendly_content (and narration) fields reference the exact version they were derived from, so staleness is a join/comparison instead of a hash which is maintained at the application level (imagine we had implemented it and use the ttsHash) prone to human-errors or application-level bugs.

This is the same shape as event sourcing/CQRS read-model invalidation or a plain content-addressable cache: never overwrite a derived artifact in place, key it by the hash of its source, and treat a hash mismatch as "recompute," not as a manual flow someone else has to remember to do.

Stop maintaining ttsFriendlyContent as hand-sanitized text all together. Instead of a regex pass + LLM normalization step that produces a separate stored string someone has to keep in sync, I'm going to hand the raw content straight to Qwen3-TTS with instructions in the prompt for how to handle markdown, elongated words, ALL-CAPS, bracketed sound cues, etc. Instead of normalizing the content first at generation time, not persistence time.

This doesn't fix the schema design mistake in a principled way. I am sidestepping it by deleting the second field that had to stay in sync. There's only one source string now (content), and "TTS-friendly" becomes an instruction send to Beatrice's generateAudio mutation. The instructions can be as rigid as a hardcoded string we can embed in the mutation itself, or as flexible as enabling authors to customize how Beatrice handles their content when it wanted to generate audio. Fewer fields, fewer hashes to forget to update, fewer maintenance taxes.

Long story short I decided to go with the second option.

The proper fix, content-addressable derived rows with explicit lineage is still the right answer if a project needs auditable, cacheable, multi-version derived content. For a single-author, single-version chapter model, removing the second field is cheaper than making the sync mechanism correct. Also keep in mind here we are talking about novels and chapters. So it should not be that big of a deal.

[!CAUTION]

I had a nasty experience with dev.to itself when I was using their platform to update my posts and then they just mixed it and my post was half old content and half new content. That was a disaster and why I decided to move to a more stable, local, and disciplined approach which was using a Version Control System. Currently I write my posts locally on my lovely Zed IDE and push them to GitHub, then there is a GitHub Actions which automatically publishes the posts to dev.to or schedule it.

I guess I am very much fond of the same pattern for smart-novel. AKA authors can write on their own local machines and push them to some sort of VSC. But the only issues is that they are most of the times not into tech. But I believe with LLMs and right amount of tooling I can make it easier for them to write their novels. I am thinking of having a MCP server which would exposes a couple of skills helping them register at GitHub, create a repository, add secrets and necessary env vars. So the nitty-gritty details are hidden away from the non-tech authors.

And of course they can still use the beautiful UI to update their posts. But I guess this demands a bit more brainstorming since I already had tried a couple of times to update my dev.to posts in their UI since it was a single line change. But then I remembered that would corrupt my post in my GitHub repository since now I have a change which does not exists in the VCS. If by any chance I update the markdown file in my repo and push it, it will override the changes I made in the dev.to UI.

GitHub Repos:

Node Sizing, Resource Alarms

Mohammad Jawad (Kasir) Barati — Sun, 06 Sep 2026 17:45:11 +0000

Maybe you also are curious as to why we picked t3.medium for the cluster and asked the obvious question: is this cluster actually sized for what we're running on it, and how would we know if it stopped being sized for it?

Is t3.medium (the instance type variables.tf defaults to) actually enough for 11 pods across 2 nodes?

How would you get paged the moment you're provisioned cluster runs out of resources instead of finding out from a CrashLoopBackOff at 2am?

[!NOTE]

JFYI, you might see ContainerCreating when running kubectl get pods, it basically means the kubelet has a node assigned and is doing setup work before the container process starts, and the single biggest contributor to that phase, on a brand new node, is pulling the image.

Is `t3.medium` Actually Enough for These 11 Pods

Short answer: yes, comfortably so for the current setup 😉.

What's actually running: voting (3 replicas), result-deployment (3), worker-deployment (3), plus the standalone db and redis, 11 application pods in total, against node_desired_size = 2, node_min_size = 1, node_max_size = 3 Kubernetes worker nodes.

It is important to understand that pods with the current setup are allocated per node, not CPU/RAM. EKS caps pods per node by ENI/IP capacity, not raw compute, because of how the VPC CNI assigns a real VPC IP to every pod:

max pods = (number of ENIs) × (IPv4 addresses per ENI − 1) + 2

For t3.medium that's 3 ENIs × (6 − 1) + 2 = 17 pods per node.

Subtract system pods that live on every node (kube-proxy, aws-node/vpc-cni, and coredns typically spread across 1-2 nodes), let's say we have 3-5 slots already reserved. With that in mind we roughly have 12-14 app-pod slots per node, ~24-28 across the 2 desired nodes. 11 pods fits with room to spare. This ceiling might bite quietly by running out of slots the moment you scale replicas up (kubectl scale --replicas=10 deployment/voting-deployment would eat most of that headroom by itself, does not matter if it was auto-scaled or not).

The other issue is that I did NOT declare resources.requests/limits. Check voting-deployment.yaml, result-deployment.yaml, worker-deployment.yaml, db-pod.yaml, and redis-pod.yaml.

None of them tell Kubernetes its CPU or memory requirements! That means the scheduler treats every pod as needing zero resources for placement purposes, so it will keep packing pods onto a node with no real headroom check at all. That means we won't be seeing a clean 0/2 nodes are available: Insufficient memory message when you're actually out of capacity.

Instead, actual memory usage creeps up under real traffic until the kubelet hits an eviction threshold and starts killing pods with OOMKilled or Evicted.

t3.medium = 2 vCPU / 4 GiB RAM. So we have the following:

Component	Replicas	Role
PostgreSQL 17 Alpine	1	Database
Redis 7 Alpine	1	Queue
`example-voting-app-result`	3	NodeJS result web app
`example-voting-app-vote`	3	Python voting web app
`example-voting-app-worker`	3	.NET worker
Total application containers	11

And you do not get to treat those 4 GiB and 2 CPUs as application capacity:

┌───────────────────────────────┐
│ EC2 t3.medium                 │
│                               │
│ 2 vCPU                        │
│ 4 GiB RAM                     │
│                               │
│ ┌───────────────────────────┐ │
│ │ Linux / kernel            │ │
│ │ kubelet                   │ │
│ │ container runtime         │ │
│ │ AWS/EKS components        │ │
│ └───────────────────────────┘ │
│                               │
│ ┌───────────────────────────┐ │
│ │ Your Pods                 │ │
│ │                           │ │
│ │ postgres                  │ │
│ │ redis                     │ │
│ │ result ×3                 │ │
│ │ vote ×3                   │ │
│ │ worker ×3                 │ │
│ └───────────────────────────┘ │
└───────────────────────────────┘

Kubernetes explicitly distinguishes node capacity from node allocatable. Allocatable is the amount available to Pods after accounting for resources reserved for Kubernetes/OS components. AWS likewise documents reserving resources for kubelet/system daemons because otherwise resource starvation can cause the node to become unhealthy.

Another factor which I deliberately did not touch is the fact that max_size = 3 doesn't mean autoscaling. node_max_size only sets the ceiling on the managed node group's underlying ASG. It's a permission slip, not a trigger. Nothing in our Terraform config actually watches for unschedulable pods and raises the desired count toward that ceiling. Without Cluster Autoscaler or Karpenter installed in the cluster, you can be pod-starved with two idle nodes sitting at desired_size = 2 and a third one it's allowed to create but has no reason to 😅.

Do NOT Calculate it Like this

Postgres needs 512 MB, Redis needs 100 MB, each app needs 100 MB, therefore we're good.

That's not gonna work!

Also it worth mentioning that currently we need to wire up alerting to detect and respond to resource pressures. The signal we need is unschedulable pods (Pending with a FailedScheduling event citing insufficient CPU/memory/pod-IPs) and node resource reservation approaching 100%. AWS's own path for this is CloudWatch Container Insights, which we can enable on the cluster. It publishes metrics like node_cpu_reserved_capacity, node_memory_reserved_capacity, and cluster_failed_node_count into CloudWatch, on top of which you set a plain CloudWatch Alarm:

resource "aws_cloudwatch_metric_alarm" "node_memory_pressure" {
  alarm_name          = "${var.cluster_name}-node-memory-reserved-high"
  namespace           = "ContainerInsights"
  metric_name         = "node_memory_utilization"
  dimensions          = { ClusterName = var.cluster_name }
  statistic           = "Average"
  period              = 300
  evaluation_periods  = 2
  threshold           = 80
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

Container Insights will run an agent/Fluent Bit DaemonSet, there is a cheaper and more direct alternative: alerting on kube_pod_status_phase{phase="Pending"} via kube-state-metrics + Prometheus + Alertmanager or we can go for the AWS-managed equivalent, Amazon Managed Service for Prometheus. That metric goes non-zero the instant a pod can't be placed.

We also must set resources.requests/limits in the manifests. Without them, Kubernetes has nothing to compare "current usage" against, so there's no signal to alarm on beyond raw node metrics. With requests set, you get two things for free: the scheduler stops overpacking nodes, and you can receive alarms on kube_pod_container_resource_limits, i.e. "this pod is at 90% of its memory limit" for example well before it gets OOMKilled. That's a config change to the *-deployment.yaml Kubernetes manifest files.

These are 4 different concepts to estimate how much CPU/RAM a pod will be needing:

flowchart TD
    A["📦 IMAGE SIZE: How much disk space the image consumes? postgres:17-alpine Image = 118.16 MB"]

    B["📊 ACTUAL RUNTIME USAGE: What the process actually consumes while running (workload shape)? E.g. Postgres needs at least 1GB of RAM, 1 CPU core"]

    C["⎈ What Kubernetes reserves for scheduling purposes."]

    D["🚧 KUBERNETES LIMIT: Maximum resource the container is allowed to consume."]

    A -->|"Image artifact != runtime resources"| B
    B -->|"Observe & measure actual behavior"| C
    C -->|"Set an upper boundary for runtime"| D

Some good to know notes:

Workload shape, not habit. E.g. this app is CPU-light, and memory-modest, but I/O intensive. A CPU-bound workload (video transcoding, heavy compute) wants compute-optimized (c-family); a memory-heavy workload (large caches, in-memory analytics) wants r-family.
A production node group under steady load usually wants m5/m6i (fixed, non-burstable performance) instead, specifically to avoid throttling under sustained load (t3.* throttles under sustained load).
If you're running many small pods (a common pattern with microservices), you can be pod-slot constrained well before you're resource-constrained. This is AWS-specific (the VPC CNI's IP-per-ENI model), not a general Kubernetes fact, and it's easy to miss because kubectl describe node reports it as pods: 17.
ARM64 Graviton instances are typically 10-20% cheaper for equivalent performance on workloads that don't depend on x86-only binaries. So if you can swap ami_type = "AL2023_x86_64_STANDARD" for AL2023_ARM_64_STANDARD and the containers will be able to run then you can save on costs.
Add something like this to your Kubernetes manifest files:

  resources:
    requests:
      cpu: 500m
      memory: 512Mi

Once traffic is real, you can collect metrics from Kubelet /metrics/resource endpoint directly. Or Container Insights tells you actual CPU/memory usage versus allocatable. Resize the node group only after you have that data.

Imagine you measure a container's RAM usage 100 times (same is true for CPU):

50, 51, 52, 53, ... 98, 100, 120 MiB

Then you need to know:

median: roughly the point where 50% of measurements are below it and 50% are above it. So if median RAM = 70 MiB, the container typically uses around 70 MiB.
p95: 95% of your measurements are at or below this value. If it is 120 MiB, then 95% of your measurements are below 120 MiB. This is the answer to the "How much does it use during almost all normal situations?"
p99: is exactly the same as p95. And in layman's terms: "How high does usage get during almost all situations, including occasional spikes?"
maximum: The single highest measurement you observed. But be careful about this one, you might have measurements like this:
```
70
72
68
75
71
69
350  ← weird spike, investigate what happened at that time!
```

behavior under load: What happens to CPU, memory, latency, errors, etc. when you give the application more work? For example you could have:

Users/requests       CPU       RAM       Response time
──────────────────────────────────────────────────────
10                    10%      80 MiB       20 ms
100                   25%      90 MiB       25 ms
500                   60%     120 MiB       40 ms
1000                  95%     180 MiB      200 ms
2000                 100%     300 MiB     2000 ms  ← struggling

The simple mental model:

  Median  → "What's typical?"
  P95     → "What happens most of the time, including busy periods?"
  P99     → "What happens during nearly all but the rarest spikes?"
  Maximum → "What's the biggest thing I actually observed?"
  Load    → "What happens when I give it more work?"

So Why `t3.medium` & Takeaway

Honestly I just picked something that we can get over with in this tutorial, not a sized decision based on the workload 😅.

What's actually missing isn't node capacity, it's visibility: no resources.requests/limits on any pod, and no alerting layer watching either node pressure or pod-level usage. So before adopting similar setup for production traffic you must first add visibility.

[!TIP]

Requests aren't actual usage. This distinction causes enormous confusion in Kubernetes, so if you have:
resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "2Gi"
And the application actually uses only 500MiB, the node will still be overprovisioned. You can visualize it like this:
Node:
    allocatable RAM = 2Gi

Pods:
    requests = 1Gi

Actual usage:
    500MiB
This is why I said you MUST add visibility to your infrastructure first.

Setup Infra, Deploy the App, Verify, and Tear It All Down

Mohammad Jawad (Kasir) Barati — Sun, 06 Sep 2026 11:08:24 +0000

I hope now you have a fundamental grasp on the what Terraform will do. This part is about setting up the infrastructure, deploying the app, verifying it works, and tearing it all down.

Prerequisites Checklist

`terraform init` & `terraform plan`

git clone git@github.com:kasir-barati/docker.git
cd k8s/voting-microservice-architecture/deployment/terraform
cp terraform.tfvars.example terraform.tfvars   # optional: edit values here

terraform init

init downloads the aws provider and the vpc/eks modules into a local .terraform/ folder, and writes/checks .terraform.lock.hcl (pins exact provider versions. This file should be committed to Git, unlike state (in the repo it is already present, but I just wanted to be clear about it).

terraform plan

You'll get a long list ending in something like:

Plan: 47 to add, 0 to change, 0 to destroy.

47-ish resources for "one VPC and one EKS cluster" sounds like a lot until you remember a module expands into subnets × 2 AZs, route tables, IAM roles and policy attachments, the node group's launch template, security groups, etc. Most of the times you wanna skim through it. BTW at this stage nothing has touched AWS yet.

`terraform apply`

Same plan, followed by a yes confirmation prompt. This step genuinely takes a while, EKS control planes take roughly 10-15 minutes to provision, plus a few more minutes for the node group's EC2 instances to launch and join. This is normal; it's AWS provisioning managed infrastructure, not a Terraform slowness so feel free to grab a coffee.

When it finishes, you'll see the outputs from outputs.tf:

Outputs:

cluster_endpoint = "https://ABCDEF1234167890.gr7.eu-central-1.eks.amazonaws.com"
cluster_name = "voting-app-cluster"
configure_kubectl = "aws eks update-kubeconfig --region eu-central-1 --name voting-app-cluster"
region = "eu-central-1"
vpc_id = "vpc-0123456789abcdef0"
...

Point `kubectl` at the new Cluster

Run the exact command from configure_kubectl above:

aws eks update-kubeconfig --region eu-central-1 --name voting-app-cluster
kubectl get nodes

NAME                                       STATUS   ROLES    AGE   VERSION
ip-10-0-0-123.ec2.internal                 Ready    <none>   2m    v1.33.x-eks-...
ip-10-0-1-45.ec2.internal                  Ready    <none>   2m    v1.33.x-eks-...

Two Ready nodes, one per AZ, this is the point where a real EKS cluster exists and you could deploy literally anything to it, not just this app.

Deploy the Voting App

cd k8s/voting-microservice-architecture/deployment
kubectl apply -f .
kubectl get pods

This applies the plain Kubernetes manifests; Postgres, Redis, and the replicated vote/result/worker app.

Get a Public URL

voting-service and result-service ship as type: LoadBalancer. On minikube, type: LoadBalancer is a superset of NodePort so minikube service list will show you the URLs of the voting and result apps. On EKS the worker nodes live in private subnets with no public IP, so there'd be no node address to even try hitting without it. kubectl apply -f . provisions the load balancers along with everything else.

kubectl get svc voting-service result-service

Within 1-3 minutes:

NAME             TYPE           EXTERNAL-IP                                                     PORT(S)
voting-service   LoadBalancer   b1b2c3d4e5f6-1234567895.elb.eu-central-1.amazonaws.com               80:31234/TCP
result-service   LoadBalancer   q6e5d4c3b2a1-0987654322.elb.eu-central-1.amazonaws.com               80:31567/TCP

That EXTERNAL-IP column is an AWS-issued DNS hostname, the default behavior for a Network Load Balancer, it's free, requires no domain of your own, and is publicly resolvable the moment it appears. Open them directly in a browser.

sequenceDiagram
    participant You as Your browser
    participant DNS as AWS DNS (*.elb.amazonaws.com)
    participant NLB as Network Load Balancer
    participant Node as EKS worker node
    participant Pod as voting-app pod

    You->>DNS: resolve a1b2c3...elb.eu-central-1.amazonaws.com
    DNS-->>You: NLB's public IP(s)
    You->>NLB: HTTP GET :80
    NLB->>Node: forward to a healthy node
    Node->>Pod: kube-proxy routes to a voting pod
    Pod-->>You: response

6. Tear it All Down

This is the part that trips people up, and it's worth understanding why, not just memorizing the commands.

The rule: delete anything Kubernetes created before running terraform destroy. The Load Balancers were created by Kubernetes' AWS integration reacting to type: LoadBalancer, not by a Terraform resource block. In other words, Terraform's state file has never heard of them.

And if you remember terraform destroy only deletes what's in state; it will delete the VPC's subnets and security groups regardless of an NLB still attached to them, which typically leaves that NLB orphaned or the terraform fails to delete the VPC. Have not verified it personally, but I suspect it does. Let me know in the comments if you know/experimented with it.

That said the cleanup.tf runs something like a kubectl delete -f . for you automatically as a destroy-time hook. So instead of:

# 1. Remove everything kubectl created, load balancers included
cd k8s/voting-microservice-architecture/deployment
kubectl delete -f .

# 2. Now tear down the infrastructure Terraform actually owns
cd k8s/voting-microservice-architecture/deployment/terraform
terraform destroy

You just run:

cd k8s/voting-microservice-architecture/deployment/terraform
terraform destroy

flowchart LR
    A["kubectl delete -f ."] --> B["NLBs actually deleted"]
    B --> C["terraform destroy"]
    C --> D["node group, EKS cluster, VPC, subnets, NAT gateway, IAM roles - all gone"]

terraform destroy reads terraform.tfstate, works out the reverse dependency order and deletes each resource via the AWS API.

Verifying nothing's left

Everything this project created is tagged Project = voting-microservice-architecture (look at providers.tf's default_tags). So go to the AWS Console → Resource Groups & Tag Editor → search that tag in all regions and in all resource types.

[!NOTE]

If terraform destroy failed to delete all resources or your state file was corrupted, then you can find and delete them manually in AWS Console.

That's the whole loop: infrastructure via Terraform, application via kubectl, kept deliberately separate so each tool's domain of responsibility is clear.

Walking Through the Terraform Config Files

Mohammad Jawad (Kasir) Barati — Sun, 06 Sep 2026 08:59:33 +0000

In this post I will walk through Terraform config files. Here's the map before we zoom in file by file:

File	Purpose
`versions.tf`	Pins Terraform and provider versions
`providers.tf`	Configures which AWS account/region Terraform talks to
`variables.tf`	Every configurable input, with sane defaults
`vpc.tf`	The network: VPC, subnets, NAT gateway
`eks.tf`	The cluster: control plane, IAM roles, worker nodes, and the read-only access entry for the Console-viewer user
`iam.tf`	The human, Console-only IAM user used to browse pods/logs in the AWS Console
`data.tf`	Read-only data sources (currently just the account ID)
`outputs.tf`	Values printed after `apply`
`cleanup.tf`	A destroy-time hook that removes Kubernetes-created LoadBalancers before the cluster/VPC teardown
`terraform.tfvars.example`	Template for overriding defaults

Terraform doesn't care about file names or how many files you split things across. It reads every .tf file in the directory and merges them into one configuration. The split above is purely for us to have an easier time comprehending what we have.

`versions.tf`

In this file we have required_version which stops anyone from running this with a Terraform version old enough to not understand some syntax used here. required_providers does the same for the AWS provider.

[!TIP]

~> 6.0 means "any 6.x, but not 7.0," so you get bugfixes automatically without silently picking up a breaking major version. This is what makes terraform init reproducible on a different laptop or in CI a year from now.
flowchart LR
    A["~> 1.5.0"] --> B["1.5.0 - Allowed"]
    A --> C["1.5.7 - Allowed"]
    A --> D["1.6.0 - NOT Allowed"]

    E["~> 1.5"] --> F["1.5.0 - Allowed"]
    E --> G["1.9.0 - Allowed"]
    E --> H["2.0.0 - NOT Allowed"]

`providers.tf`

Sets region, read from a variable rather than being hardcoded, so switching regions is a one-line change in terraform.tfvars, not a find-and-replace across every file.

[!TIP]

default_tags is applied to every single resource the AWS provider creates in this run. It's how every VPC, EC2 instance, IAM role, etc. ends up tagged Project = voting-microservice-architecture without repeating that tag block 20 times. Handy for finding everything this project owns in the AWS console later, and it's extremely useful when a module creates a resource which you did not specify in your own Terraform config files.

`variables.tf`

Every variable follows this shape: a type (string, number, bool, list(string), ...), a human-readable description, and usually a default so the project works out of the box. BTW this is variable precedence for overriding defaults:

flowchart TD
    L1["Lowest, edit default in variables.tf"]

    L1 --> L1note["⚠️ Not recommended. Mixes config with input"]

    L1note -.overridden by.-> L2["Medium, terraform.tfvars. node_instance_types = ['t3.small']"]
    L2 --> L2note["✅ Recommended. This is why terraform.tfvars.example exists as a template"]

    L2note -.overridden by.-> L3["Highest, env var / CLI flag"]
    L3 --> L3a["export TF_VAR_node_instance_types='[t3.small]'"]
    L3 --> L3b["-var node_instance_types='[t3.small]'"]

This project defines: aws_region, environment, cluster_name, kubernetes_version, vpc_cidr, azs, node sizing (node_instance_types/node_desired_size/node_min_size/node_max_size), cluster_endpoint_public_access, and console_viewer_username. Full list with descriptions can be seen in the variables.tf.

`vpc.tf`

A few things worth explaining if you haven't used Terraform's function syntax before:

module "vpc" { source = "...", version = "..." } is what we meant by "reuse someone's published .tf files" in action. source points at terraform-aws-modules/vpc on the Terraform Registry; instead of us hand-writing every aws_subnet, aws_route_table, aws_nat_gateway, etc. (which is a lot of boilerplate to get exactly right), we're using a battle-tested module that thousands of other projects already rely on.
Why CIDR exists: every resource in your VPC (nodes, load balancers, NAT gateway) needs its own private IP address, and something has to decide up front which addresses are "ours" to hand out. Before your VPC can exist, you have to reserve a block of addresses for it, like requesting a range of extension numbers for an office phone system before anyone can be assigned one. CIDR notation (10.0.0.0/16) is just the shorthand for "reserve this block".
cidrsubnet is a built-in Terraform function that carves a smaller CIDR block out of a bigger one.
We have a loop in there too so 2 AZs in the variable become 2 actual private subnets and 2 public subnets (offset by + 100 so public and private CIDRs never collide).
The *_subnet_tags blocks is the part that AWS integration of Kubernetes uses to discover which subnets to use by looking for these exact tag keys:
- kubernetes.io/role/elb on public subnets tells the cluster "put internet-facing Load Balancers here".
- kubernetes.io/role/internal-elb does the same for internal ones.
- kubernetes.io/cluster/<name> = shared marks a subnet as belonging to this cluster.

Skip these tags and type: LoadBalancer Kubernetes services simply won't provision correctly. Documented here: EKS VPC and subnet requirements.

`eks.tf`

vpc_id = module.vpc.vpc_id and subnet_ids = module.vpc.private_subnets is how Terraform wires two modules together, and it's also how Terraform knows the VPC must be created before the cluster. module.vpc.vpc_id is an output of the vpc module (every module can expose outputs the same way the root configuration does, from outputs.tf). You never had to write "create the VPC first"; Terraform derived that from the fact that eks.tf reads a value that only exists once vpc.tf has run.
Worker nodes go in module.vpc.private_subnets.
enable_cluster_creator_admin_permissions = true is what enables the IAM user who ran terraform apply to issue kubectl commands against. This flag registers your IAM identity as a cluster admin automatically via an EKS access entry, which is the modern replacement for the old, fiddlier aws-auth ConfigMap approach.
access_entries.console_viewer is the entry for IAM user I intend it to be allowed to look at the cluster. enable_cluster_creator_admin_permissions only ever creates one access entry, for the apply-time identity; a second human who wants to browse pods/logs in the AWS Console needs their own. This block grants exactly that: it maps aws_iam_user.console_viewer's ARN (iam.tf) to AWS's managed AmazonEKSViewPolicy (read-only, cluster-scoped) via a Kubernetes-RBAC access entry.
addons are AWS-managed installs of standard Kubernetes cluster components, so you don't kubectl apply them yourself: coredns (in-cluster DNS is how redis and db hostnames in the app's connection strings actually resolve), kube-proxy (routes service traffic to the right pod), vpc-cni (gives every pod a real VPC IP address, the AWS-specific way pod networking works on EKS), and eks-pod-identity-agent (lets pods assume IAM roles, a near-mandatory addon for anything that'll later talk to S3/DynamoDB/etc. from inside a pod). before_compute = true on two of them means "install this before the worker nodes even join" since nodes need working pod networking to function.
eks_managed_node_groups is the actual EC2 Auto Scaling group your pods run on. AL2023_x86_64_STANDARD is Amazon Linux 2023, the current default EKS-optimized AMI type. min_size/max_size/desired_size are Auto Scaling group settings, Kubernetes' own autoscaler (if you added one) would adjust desired_size between the min/max bounds; here they're fixed at whatever terraform.tfvars says.

[!TIP]

The Amazon Linux 2023, x86_64 architecture, "standard" (no GPU, no ARM), this is the image we used. Other options include things like AL2023_ARM_64_STANDARD (Graviton/ARM), or _NVIDIA/_GPU variants for GPU workloads.

Full module docs, including every other option available: terraform-aws-modules/eks/aws on the Registry.

`iam.tf`

enable_cluster_creator_admin_permissions covers the automation identity that runs terraform apply. It deliberately has no AWS Console access. So if you personally want to open the EKS Console and click through pods, clusters, etc, that's a second, separate IAM identity, created in iam.tf.

aws_iam_user.console_viewer: the IAM user itself, named by default voting-app-console-viewer.
aws_iam_user_policy: grants only enough to reach the EKS API: the account-level ListClusters/DescribeClusterVersions, and DescribeCluster/AccessKubernetesApi scoped to this cluster only (module.eks.cluster_arn). This policy grants permission to AWS Console access; the RBAC half is the access_entries block in eks.tf. We need both, and they answer different questions ("can this principal call the EKS API at all" vs. "what can it do once inside the cluster").
aws_iam_user_login_profile, with password_reset_required = true generates a one-time password and forces a change on first login. In my experience they usually do not do this, instead DevOps team grant permission to each developer appropriate access rights and it is tied to their role in the organization.
aws_iam_user_policy_attachment adds AWS's managed IAMUserChangePassword policy to enable users get self-service iam:ChangePassword permission to change their own password.

`data.tf`

A data block, unlike resource, doesn't create anything.
It just reads information that already exists, here the AWS account ID of whoever's credentials are active.
We need it when building the AWS Console sign-in URL in outputs.tf (https://<account_id>.signin.aws.amazon.com/console), which requires knowing the account ID.

`cleanup.tf`

Before we can torn down this cluster's VPC/subnets, we must first use kubectl and delete voting-service.yaml/result-service.yaml. This frees the AWS Load Balancers Kubernetes created for them. Those Load Balancers were never created by a resource block, so Terraform's state doesn't know they exist and terraform destroy alone can't remove them. Or at least this is what my lizard brain thinks considering all the hints I saw here and there. But still since I was not so sure I asked it here.

`outputs.tf`

Outputs are Terraform's way of handing you (or another Terraform config, or a CI pipeline) values it only knows after resources actually exist. Here we print stuff like AWS Console login URL, username, cluster endpoint, etc.

[!TIP]

When we dump something in the output you can mark it as sensitive so it won't show in the CI/CD pipeline logs in plain text. That is what I did for the temporary password. So if you need to see the value in plain text you must run terraform output -raw console_viewer_initial_password to actually read it.

How the pieces fit together

graph TB
    Root["Root module\n(this directory's .tf files)"]
    Root -->|passes vpc_cidr, azs| VPC["module: terraform-aws-modules/vpc/aws"]
    Root -->|passes cluster_name, node sizing| EKS["module: terraform-aws-modules/eks/aws"]
    VPC -->|"outputs: vpc_id, private_subnets"| EKS
    EKS -->|creates| CP[EKS control plane]
    EKS -->|creates| NG[Managed node group]
    EKS -->|creates| IAM[IAM roles for cluster + nodes]
    VPC -->|creates| Net[VPC, subnets, NAT gateway]
    Root -->|"aws_iam_user, policy, login profile"| ConsoleIAM["iam.tf: console_viewer"]
    ConsoleIAM -.ARN referenced by.-> EKS
    Root -->|"null_resource, depends_on module.eks"| Cleanup["cleanup.tf: cleans up LBs at before destroy starts"]

Looking at what we are Building

Mohammad Jawad (Kasir) Barati — Sat, 05 Sep 2026 21:25:59 +0000

So now that you have a basic understanding of how Terraform works, before you start running any terraform command against a real AWS account, two things need to happen: you need an identity Terraform can authenticate as, and you need a mental picture of what you're about to create, so the plan output in Part 4 isn't just a list of unfamiliar resource names.

Never Use Your AWS Root User

The root user (the email/password you signed up to AWS with) can do anything, including closing the account. It should basically never be used day-to-day. Instead, create a dedicated IAM user just for this project. In real life you would create a dedicated IAM user for your CI/CD pipeline to automate deployments:

AWS Console → IAM → Users → Create user (e.g. terraform-voting-app).
Do not enable AWS Console access, this user only needs programmatic access, i.e. an API key pair.
The AWS managed policy AdministratorAccess is the path of least friction, and is what you should use for the IAM user to test things out. but in real life you would go with least privilege approach, learn more about it in AWS EKS IAM policy examples.
On the user's Security credentials tab → Create access key → choose "Command Line Interface (CLI)". You'll get an Access Key ID and a Secret Access Key, store them somewhere safe, we will be needing them later.

Give Terraform those Credentials

The rule: credentials never go inside a .tf file, and never inside terraform.tfvars. So in your local machine or CI/CD pipeline you need to export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION as environment variables in the shell. Then you won't be needing aws configure or aws login step anywhere, the aws provider has no access_key/secret_key arguments of its own, so it falls back to the AWS SDK's standard credential chain, which checks these exact environment variables first.

The AWS CLI and, later kubectl read the same variables. In my local machine I do export an env variable files using a shell script. The export only lasts for the current shell session, re-run it in any new terminal before running terraform/aws/kubectl.

What You are About to Build

This is the part most beginner Terraform/EKS tutorials skip, and it's the part that makes the actual .tf files make sense at a glance instead of feeling like a wall of unfamiliar arguments.

flowchart TB
    subgraph AWS["AWS Account / Region"]
        subgraph VPC["VPC — 10.0.0.0/16"]
            IGW["Internet Gateway"]

            subgraph AZ1["Availability Zone A"]
                PubA["Public subnet\n10.0.100.0/24"]
                PrivA["Private subnet\n10.0.0.0/24"]
            end

            subgraph AZ2["Availability Zone B"]
                PubB["Public subnet\n10.0.101.0/24"]
                PrivB["Private subnet\n10.0.1.0/24"]
            end

            NAT["NAT Gateway\n(in a public subnet)"]

            CP["EKS Control Plane\n(managed by AWS,\nnot inside your subnets)"]

            PrivA --- Node1["EC2 worker node"]
            PrivB --- Node2["EC2 worker node"]
        end

        NLB["Network Load Balancer\n(public, one per exposed Service)"]
    end

    Internet(("Internet")) --> IGW
    IGW --> PubA
    IGW --> PubB
    PubA --> NAT
    NAT -.outbound only.-> Node1
    NAT -.outbound only.-> Node2
    CP <-. manages .-> Node1
    CP <-. manages .-> Node2
    Internet --> NLB
    NLB --> Node1
    NLB --> Node2

    Node1 -- runs --> Pods1["voting / result / worker\nredis / postgres pods"]
    Node2 -- runs --> Pods2["voting / result / worker\nredis / postgres pods"]

Reading it top to bottom:

VPC: a private network inside AWS, 10.0.0.0/16 here (65k addresses is just the conventional default and more than what we need).
Two Availability Zones: EKS requires subnets in at least 2 AZs, so a single AZ failure can't take the whole cluster down.
Public subnets have a route to the Internet Gateway. Their only job is hosting the NAT Gateway and, later, the public-facing Load Balancers.
Private subnets is where the actual EC2 worker nodes live. They have no direct route in from the internet, and no public IP at all.
NAT Gateway lets the private-subnet nodes reach out to the internet (to pull container images, talk to the EKS API, etc.) without allowing anything in from the internet. One-way door.
EKS Control Plane is a managed AWS service which is the Kubernetes API server, scheduler, etc. It doesn't live "in" your subnets the way an EC2 instance does, though it does attach elastic network interfaces into them to talk to your nodes.
Worker nodes are plain EC2 instances, sitting in the private subnets, that the EKS control plane schedules your pods onto. This is the "data plane", as opposed to the control plane above.
Network Load Balancer is created later, by Kubernetes (not Terraform) when you expose a Service as type: LoadBalancer. This is how traffic actually reaches your app from a browser.

EKS Costs

We have two separate charges, both starting the moment terraform apply finishes:

The EKS control plane itself has a flat hourly rate. Check current EKS pricing and this is regardless of whether any pods are running.
The EC2 worker nodes: ordinary EC2 billing, because they're ordinary EC2 instances. Two t3.medium nodes, in this project.

Plus a NAT Gateway (hourly + per-GB processed) and small EBS volumes for each node's disk. None of it is free-tier.

Reference links

Terraform Basics

Mohammad Jawad (Kasir) Barati — Sat, 05 Sep 2026 17:22:43 +0000

This is the first part of a real Terraform setup that provisions a brand new Amazon EKS cluster (VPC, control plane, worker nodes — everything, no pre-existing cluster assumed) and deploys a small multi-service voting app onto it. It assumes no prior Terraform knowledge, and I use Kubernetes manifests to deploy the app.

The actual Terraform code, and the plain Kubernetes manifests, live at k8s/voting-microservice-architecture/deployment in my docker repo. Feel free to give it a star 😉.

If you've ever created an AWS resource by clicking through the console, or run a pile of aws ec2 .../eksctl create cluster commands from a half-remembered README you've felt the problem Terraform solves: there is no reliable record of what you built, how to recreate it.

Terraform's job is to fix that.

What "Infrastructure as Code" Actually Means

You write down, in plain text files, what should exist: "a VPC with this CIDR," "an EKS cluster with 2 worker nodes", ...

You do not write the steps to create it. That distinction matters:

Imperative (a shell script, eksctl, clicking in the console): you describe the actions, create this, then that, then attach this to that. If you run it twice, you either get an error ("already exists") or a duplicate 🥲.
Declarative (Terraform, and Kubernetes too): you describe the end state. You run the same command as many times as you like; Terraform figures out the difference between "what exists" and "what you asked for", and only changes what's necessary.

If you've ever written a Kubernetes YAML manifest and run kubectl apply -f . more than once, you already understand declarative infrastructure. Terraform applies the identical idea one layer down, to the cloud resources underneath the cluster (or, as in this project, including the cluster itself).

The Building Blocks (HCL)

Terraform's config language is called HCL (HashiCorp Configuration Language). You'll see four kinds of blocks constantly:

# 1. A provider - which cloud/API Terraform should talk to
provider "aws" {
  region = "us-east-1"
}

# 2. A resource - one real thing to create
resource "aws_s3_bucket" "example" {
  bucket = "my-unique-bucket-name"
}

# 3. A variable - an input you can change without editing the file
variable "region" {
  type    = string
  default = "ap-east-1"
}

# 4. An output - a value Terraform prints after it's done
output "bucket_name" {
  value = aws_s3_bucket.example.bucket
}

That's genuinely most of the language. Everything else, loops (for_each/count), conditionals, functions like cidrsubnet() are sugar on top of "declare resources, wire them together with references".

One more block type you'll see a lot in this project is module. A module is just a folder of .tf files someone else already wrote and published, that you can reuse instead of writing hundreds of lines of resource blocks yourself:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"   # published on the Terraform Registry
  version = "~> 6.0"
  cidr    = "10.0.0.0/16"
}

Under the hood a module is exactly the same resource/variable/output blocks you just saw, someone packaged them up so you don't have to setup "a correct VPC" from scratch each time. More on the Terraform Registry.

The Workflow: 4 Commands

flowchart LR
    A["Write Terraform Config files\n(desired state)"] --> B["terraform init\ndownload providers/modules"]
    B --> C["terraform plan\npreview: create / change / destroy?"]
    C --> D["terraform apply\nactually call the AWS API"]
    D --> E[("terraform.tfstate\nrecord of what exists")]
    E -.compared against.-> C
    D --> F["terraform destroy\ntear it all back down"]
    F -.reads.-> E

terraform init downloads the providers and modules your config references. We run it once per directory, and again whenever we add/change a provider/module version.
terraform plan asks the AWS API "what exists right now that you're tracking?", diffs that against your .tf files, and prints what would change. Nothing is created or changed by plan. Always read the plan before applying, it's the single best safety net Terraform gives you.
terraform apply shows what it will change, asks you to type yes, then actually calls the API of AWS or any other provider. Terraform works out the correct order on its own (e.g. it won't try to create a subnet before the VPC that subnet belongs to) by following the references between your resources.
terraform destroy deletes everything Terraform currently has in its state, in dependency order.

State: The One File You Need to Ensure is Never Lost

Every time you apply, Terraform writes a file which by default is called terraform.tfstate. It records exactly what it created. E.g. when provisioning AWS services it keeps a track of the real AWS IDs of each resource (this VPC's actual vpc-0abc123..., that cluster's actual ARN, ...).

This file is the thing that makes Terraform work. Without it, Terraform has no way to know "I already made this VPC, don't make another one" or "if you ask me to destroy, here's exactly what to delete". A few rules that follow directly from that:

Never hand-edit it. Use terraform state subcommands if you truly need to intervene.
Never commit it to a public repo. It can contain sensitive values (e.g. database passwords set via a resource argument) in plain text.
Don't run apply from two places at once against the same state file. Two concurrent applies can corrupt it. (Real teams solve this with a remote backend, e.g. state is stored in a S3 bucket, or in Terraform Cloud. Learn more about Terraform backends.
If you delete it, Terraform "forgets" everything it made! It won't know how to clean those resources when you run terraform destroy anymore, and it'll try to create duplicates on the next terraform apply.

Where to go Deeper

Terraform's own "What is Terraform?" docs
HCL syntax reference
Terraform Registry — where providers and modules are published
Terraform state docs

The Pipeline Pattern

Mohammad Jawad (Kasir) Barati — Sun, 30 Aug 2026 13:55:51 +0000

I watched a nice video about a small multi-stage scraper: it was basically scraping S&P 500 tickers from Wikipedia, fetch a live price for each from Yahoo Finance, and write the results to Postgres. Each stage runs its own pool of worker threads, and stages talk to each other through queues. So I just decided to write about it since it is a nice distributed-systems design pattern.

Pipes & Filters (AKA The Pipeline Pattern)

This is a classic architectural pattern: a series of independent processing stages ("filters"), each connected to the next by a channel ("pipe"). Each stage:

Reads from an inbound queue.
Does one unit of work.
Writes to an outbound queue.
And knows nothing about the stages before or after it.

The first stage (no inbound queue) is the source or producer. The last stage (no outbound queue) is the sink or consumer. Everything in between is just a filter.

In the following example we have:

WikiWorker (source)
    -> symbol_queue
        -> YahooFinancePriceScheduler x4 (filter, fanned out)
            -> postgres_queue
                -> PostgresMasterScheduler x4 (sink, fanned out)

Another example which I believe is a strong candidate for this design pattern is when you have a pipeline for raw data, sending them to different LLM-powered apps to extract and generate useful insights while maintaining a clean architecture which is scalable and maintainable. LLMs are slow, I/O-bound, rate-limited, and failure-prone (timeouts, malformed JSON, rate limit errors), and you often want to run several different extraction tasks over the same raw data.

A natural shape:

Source: raw data ingestion (files, API, scrape, DB change stream)
    -> raw_queue
        -> Preprocessing stage (clean/chunk/normalize) x N workers
            -> llm_queue
                -> LLM extraction workers x N (fan-out: one pool per "app" -- summarizer, entity extractor, sentiment, classifier, etc.)
                    -> results_queue(s)
                        -> Sink: persistence / downstream index / notification

The fan-out step is the interesting part: a single normalized item can be pushed to multiple output queues, one per LLM-powered consumer, the same broadcast-to-all-output-queues trick our scraper already does. Each "app" (summarizer, extractor, classifier) becomes an independent filter stage with its own worker pool and its own scaling knob.

Benefits of this Design Pattern

Independent scaling per stage: LLM calls are the bottleneck, not ingestion. You can run 2 preprocessing workers and 20 LLM-call workers without touching the rest of the pipeline.
Isolation of failure and backpressure: A stalled or rate-limited LLM provider backs up its own queue instead of crashing ingestion or the other extraction apps running in parallel.
Easy horizontal fan-out to multiple "apps": Adding a new insight-extraction app is just: add a new output queue + a new worker pool. No changes to upstream stages.
Natural retry/DLQ boundaries: Since each stage only knows its own queue, you can wrap an LLM stage with its own retry logic or dead-letter queue without leaking that complexity into ingestion or persistence code.
Swappable transport: In-memory Queue today, Kafka/RabbitMQ/SQS tomorrow, the stage logic doesn't change, only what's plugged into the pipe. That's what "clean architecture" buys you here: stages depend on queue interfaces, not on each other.
Composability/testability. Each worker class is testable in isolation, feed it a fake input queue, assert what lands on the output queue. No need to spin up the whole pipeline to unit test one extraction stage.
Cost/throughput control point: Because concurrency is explicit per stage (pool size), it's a direct lever for controlling how hard you hit an LLM API's rate limits, independent of ingestion speed.

Note

This design pattern itself is architecture-agnostic. It's just stages + queues; whether those stages are threads in one process, services in a monorepo, or separate microservices is an implementation detail, not a change to the pattern. You do always need some orchestrator, though its job shrinks as you move outward:

Monolith/threads: you have a orchestrator (in our scraper example we have main.py), which owns queue creation, worker counts, and shutdown.

Monorepo/multi-process: orchestrator becomes a supervisor process or a broker's own routing (Kafka/RabbitMQ take over "the pipe").

Microservices + managed inference (SageMaker etc.): orchestrator is very thin, it just decides how many in-flight requests it allows.

Dual autoscaling concern: you now have concurrency control in two places, your orchestrator's queue-consumption concurrency, and SageMaker's endpoint autoscaling. And they aren't aware of each other. Imagine your orchestrator can burst requests faster than SageMaker's endpoint scales up (429s/throttling), or SageMaker scales up while your orchestrator is deliberately throttled, wasting provisioned capacity.

The fix is to make one the source of truth (SageMaker), and the orchestrator reacts to them:

Have your orchestrator's concurrency be a ceiling tuned to your endpoint's steady-state capacity, not an independent guess.

Let SageMaker's autoscaling handle burst absorption, and have your orchestrator back off on throttling responses (retry-with-backoff) rather than pre-guessing SageMaker's scale.

"Pre-guessing" = hardcoding a concurrency number based on an assumption like "SageMaker probably has 10 instances up, so I'll send 10x work."

Instead of polling "how many instances exist", react to the endpoint's actual signal.

Optionally drive orchestrator concurrency from an external signal (endpoint's current instance count / concurrency metric) instead of a static number.

The Pieces this Pattern is Made of

A few sub-patterns show up together whenever you build a pipeline like this:

Producer-consumer:
- A queue serves as a buffer between two parts of our app. One puts data in the queue, and the other takes them out.
- E.g. WikiWorker puts ticker symbols into symbol_queue. The Yahoo Finance workers take symbols out of it.
- WikiWorker doesn't need Yahoo to be ready, and Yahoo doesn't need Wiki to still be running
- The queue absorbs the timing mismatch.
Worker pool: Instead of having just one thread handle a stage, you run several identical threads that all pull from the same queue, so the work gets split among them automatically..
Fan-out / fan-in:
- Fan-out spreads work across workers:
- One queue.
- Many workers reading from it (this is the worker pool from above, "fanning out" work across threads).
- Fan-in collects workers' output back into one place:
- Many workers.
- All workers write into the same single downstream queue.
Poison pill (sentinel value): a special marker value, here it is 'DONE' string. Pushed onto a queue to tell every consumer "there's nothing more coming, exit your loop". It's the standard way to shut down a pipeline gracefully instead of killing threads abruptly.

Same Design Patterns in Other Programming Languages

Go, using goroutines and channels, their article on this is basically the canonical writeup: it describes a pipeline as a series of stages connected by channels, each stage a group of goroutines running the same function, with a done channel used for the poison-pill/cancellation (also read this).
Java, using BlockingQueue and thread pools (ExecutorService).
Unix shell pipelines (cmd1 | cmd2 | cmd3), the original inspiration for the name.
Message brokers like Kafka or RabbitMQ, where "queue" becomes a literal broker topic/queue instead of an in-memory object, and stages become separate services instead of threads.

The underlying idea; decouple stages, connect them with buffered channels, parallelize each stage independently is a general concurrency and distributed-systems pattern. Python's queue.Queue/multiprocessing.Queue is just one convenient implementation of "the pipe" part.

Scraper Example

I believe the database schema is more than obvious but just to leave no room to the imagination:

CREATE TABLE IF NOT EXISTS prices (
    id           SERIAL PRIMARY KEY,
    symbol       TEXT,
    price        DOUBLE PRECISION,
    insert_time  TIMESTAMPTZ
);

⚠️ Important Note

I did NOT sanity check if the scaping still is correct since I wanted to talk about the design pattern and not the scraping itself.

`src/workers/wiki.py`

"""Scrapes the current S&P 500 constituent list from Wikipedia."""

from collections.abc import Iterator

import requests
from bs4 import BeautifulSoup


class WikiWorker:
    """Fetches and parses the S&P 500 constituents table from Wikipedia."""

    def __init__(self) -> None:
        self._url: str = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"

    @staticmethod
    def _extract_company_symbols(page_html: str) -> Iterator[str]:
        """Yield ticker symbols parsed out of the constituents table HTML."""
        soup = BeautifulSoup(page_html, "html.parser")
        table = soup.find(id="constituents")
        table_rows = table.find_all("tr")

        for table_row in table_rows[1:]:
            symbol: str = table_row.find("td").text.strip("\n")
            yield symbol

    def get_sp_500_companies(self) -> Iterator[str]:
        """Yield each S&P 500 ticker symbol, or nothing on request failure."""
        response = requests.get(self._url)
        if response.status_code != 200:
            print("Couldn't get entries")
            return

        yield from self._extract_company_symbols(response.text)

`src/workers/yahoo_finance_price.py`

"""Worker pool that fetches live prices from Yahoo Finance."""

from __future__ import annotations

import random
import threading
import time
from datetime import UTC, datetime
from queue import Empty, Queue
from typing import Literal

import requests
from lxml import html

Symbol = str
Price = float
PriceMessage = tuple[Symbol, Price, str]
Sentinel = Literal["DONE"]


class YahooFinancePriceScheduler(threading.Thread):
    """
    Pulls ticker symbols off an input queue and fans price results out
    to one or more output queues.
    """

    def __init__(
        self,
        input_queue: "Queue[Symbol | Sentinel]",
        output_queues: list["Queue[PriceMessage | Sentinel]"],
        **kwargs,
    ) -> None:
        super().__init__(**kwargs)
        self._input_queue = input_queue
        self._output_queues = output_queues
        self.start()

    def run(self) -> None:
        while True:
            # Defensive programming: protects this scheduler from hanging
            # indefinitely. Trade-off: if a message arrives right after the
            # timeout fires, it's dropped without ever being processed.
            try:
                val = self._input_queue.get(timeout=20)
            except Empty:
                print("Timeout reached in Yahoo Finance scheduler, stopping...")
                break

            if val == "DONE":
                for output_queue in self._output_queues:
                    output_queue.put("DONE")
                break

            worker = YahooFinancePriceWorker(symbol=val)
            price = worker.get_price()
            if price is None:
                continue

            message: PriceMessage = (val, price, str(datetime.now(UTC)))
            for output_queue in self._output_queues:
                output_queue.put(message)

            time.sleep(random.random())  # Cloudflare may block bursty requests


class YahooFinancePriceWorker:
    """Fetches the current price for a single ticker symbol."""

    def __init__(self, symbol: Symbol, **kwargs) -> None:
        self._symbol = symbol
        base_url = "https://finance.yahoo.com/quote/"
        self._url = f"{base_url}{self._symbol}"

    def get_price(self) -> Price | None:
        """Return the current price, or None if the request/parse fails."""
        response = requests.get(self._url)
        if response.status_code != 200:
            return None

        page_contents = html.fromstring(response.text)
        nodes = page_contents.xpath(
            '//*[@id="quote-header-info"]/div[3]/div[1]/div/span[1]'
        )
        if not nodes:
            return None

        raw_price = nodes[0].text
        return float(raw_price.replace(",", ""))

`src/workers/postgres.py`

"""Worker pool that persists price messages into Postgres."""

from __future__ import annotations

import os
import threading
from queue import Empty, Queue
from typing import Literal

from sqlalchemy import create_engine, text

Symbol = str
Price = float
PriceMessage = tuple[Symbol, Price, str]
Sentinel = Literal["DONE"]


class PostgresMasterScheduler(threading.Thread):
    """Consumes price messages from a queue and writes each one to Postgres."""

    def __init__(
        self, input_queue: "Queue[PriceMessage | Sentinel]", **kwargs
    ) -> None:
        super().__init__(**kwargs)
        self._input_queue = input_queue
        self.start()

    def run(self) -> None:
        while True:
            try:
                val = self._input_queue.get(timeout=20)
            except Empty:
                print("Timeout reached in Postgres scheduler, stopping...")
                break

            if val == "DONE":
                break

            symbol, price, extracted_time = val
            worker = PostgresWorker(symbol, price, extracted_time)
            worker.insert_into_db()


class PostgresWorker:
    """Inserts a single price observation into the `prices` table."""

    def __init__(self, symbol: Symbol, price: Price, extracted_time: str) -> None:
        self._symbol = symbol
        self._price = price
        self._extracted_time = extracted_time

        self._pg_user: str = os.environ.get("PG_USER", "")
        self._pg_pw: str = os.environ.get("PG_PW", "")
        self._pg_host: str = os.environ.get("PG_HOST", "localhost")
        self._pg_db: str = os.environ.get("PG_DB", "postgres")

        self._engine = create_engine(
            f"postgresql://{self._pg_user}:{self._pg_pw}@{self._pg_host}/{self._pg_db}"
        )

    def insert_into_db(self) -> None:
        insert_query = """
            INSERT INTO prices (symbol, price, insert_time)
            VALUES (:symbol, :price, CAST(:extracted_time AS TIMESTAMP))
        """
        with self._engine.connect() as conn:
            conn.execute(
                text(insert_query),
                {
                    "symbol": self._symbol,
                    "price": self._price,
                    "extracted_time": self._extracted_time,
                },
            )
            conn.commit()

`src/main.py`

"""Wires the wiki -> Yahoo Finance -> Postgres pipeline together and runs it."""

from __future__ import annotations

import time
from multiprocessing import Queue

from workers.postgres import PostgresMasterScheduler
from workers.wiki import WikiWorker
from workers.yahoo_finance_price import YahooFinancePriceScheduler


def main() -> None:
    symbol_queue: Queue = Queue()
    postgres_queue: Queue = Queue()
    scraper_start_time: float = time.time()
    wiki_worker = WikiWorker()

    num_yahoo_finance_price_workers: int = 4
    yahoo_finance_price_scheduler_threads: list[YahooFinancePriceScheduler] = [
        # Imagine multiple output queues here: one for Redis, one for
        # RabbitMQ, etc. Each Yahoo worker fans its result out to all of them.
        YahooFinancePriceScheduler(
            input_queue=symbol_queue, output_queues=[postgres_queue]
        )
        for _ in range(num_yahoo_finance_price_workers)
    ]

    num_postgres_workers: int = 4
    postgres_scheduler_threads: list[PostgresMasterScheduler] = [
        PostgresMasterScheduler(input_queue=postgres_queue)
        for _ in range(num_postgres_workers)
    ]

    for symbol in wiki_worker.get_sp_500_companies():
        symbol_queue.put(symbol)

    for _ in yahoo_finance_price_scheduler_threads:
        symbol_queue.put("DONE")

    for thread in yahoo_finance_price_scheduler_threads:
        thread.join()

    for thread in postgres_scheduler_threads:
        thread.join()

    elapsed = time.time() - scraper_start_time

    print(f"Finished in {elapsed:.2f}s")


if __name__ == "__main__":
    main()

More Mature Pipelines

So I guess you have seen it now that if you need to add a new stage/worker to the existing orchestrator you must change the code and write a bunch of code which really feels like boilerplates. So next what you can do is defining your pipeline in a YAML file and then each time you have a new stage/worker you can just need to add it there. So create a src/pipelines directory and inside it create a yaml file:

# src/pipelines/wiki_yahoo_scraper_pipeline.yaml
queues:
  - name: SymbolQueue
    description: Contains symbols/tickers to be scraped from Yahoo
  - name: PostgresUploading
    description: Contains data that needs to be uploaded to Postgres

workers:
  - name: WikiWorker
    description: This scraps Wikipedia page nad extracts symbols/tickets
    location: workers.wiki
    class: WikiWorkerMasterScheduler
    instances: 1
    input_values:
      - https://en.wikipedia.org/wiki/List_of_S%26P_500_companies
    output_queues:
      - SymbolQueue
  - name: YahooFinanceWorker
    description: Pulls the stock price of a given ticket from Yahoo Finance
    location: workers.yahoo_finance_price
    class: YahooFinancePriceScheduler
    instances: 4
    input_queue: SymbolQueue
    output_queues:
      - PostgresUploading
  - name: PostgresWorker
    description: Stores stock data in Postgres
    location: workers.postgres
    class: PostgresMasterScheduler
    instances: 4
    input_queue: PostgresUploading

So as you can see with this you are effectively moving the logic of your pipeline into a config file which is written in YAML. And this makes your orchestrator cleaner and you can easily reason about your pipeline without having to read a bunch of python code. Also it makes it easier to differentiate between when you wanna e.g. change the URL you use for scraping data from it, from implementation changes you had to make to the code itself.

Easier time to review and understand what was changed in a PR/commit. And now your src/main.py would look a lot nicer with less manual steps involved:

"""Wires the wiki -> Yahoo Finance -> Postgres pipeline together and runs it."""

from __future__ import annotations

import time
from pathlib import Path

from pipelines.reader import YamlPipelineExecutor


def main() -> None:
    scraper_start_time = time.time()
    pipeline_location = Path(__file__).parent / 'pipelines' / 'wiki_yahoo_scraper_pipeline.yaml'
    yaml_pipeline_executor = YamlPipelineExecutor(pipeline_location=pipeline_location)
    yaml_pipeline_executor.process_pipeline()

    elapsed = time.time() - scraper_start_time

    print(f"Finished in {elapsed:.2f}s")


if __name__ == "__main__":
    main()

I did NOT add YamlPipelineExecutor code here. But that was not the only change I had to make to make this pipeline work with the aforementioned yaml file. You can see the complete example here.

If you've built something similar, especially across process or machine boundaries with a real broker instead of an in-memory queues I'd love to hear how you handled backpressure and shutdown.

RabbitMQ Consumer as a Separate Worker Service

Mohammad Jawad (Kasir) Barati — Sun, 30 Aug 2026 09:04:59 +0000

I come from a NodeJS background and honestly there we usually consume messages in the same NestJS app. But 2 years ago (2024) when I started to develop Backend APIs in Python I realized there is a difference in programming language paradigms. In Python I had to either use asyncio/threads/processes.

Here is an "at scale" alternative to the multithreading write-up I did in particular (but I believe you can get some inspiration even if you have a subprocess) instead of a background thread living inside a GraphQL API process, the RabbitMQ consumer has its own container, and its own set of configuration. So the flow would look like this:

┌─────────────────────────┐
│ RabbitMQ                │
└─────────────────────────┘
        |
        ▼
┌─────────────────────────┐
| Worker process          |
└─────────────────────────┘
        |
        ▼
┌─────────────────────────┐
| Fetch/create user       |
└─────────────────────────┘
        |
        | "At least once" delivery guarantee
        ▼
┌─────────────────────────┐
| Redis pub/sub           |
└─────────────────────────┘
        |
        ▼
┌─────────────────────────┐
| Strawberry subscription |
└─────────────────────────┘
        |
        ▼
┌─────────────────────────┐
| GraphQL client          |
└─────────────────────────┘

Why Split It Out at All?

A thread, or subprocess, or asyncio tasks in the GraphQL API process consumer ties three things together that don't actually belong together:

Scaling dimension. HTTP replica count and RabbitMQ consumer count become the same number. If you need 10 API replicas for request load but only 2 consumers' worth of queue throughput, you either over-consume or under-provision HTTP, because one knob controls both. You can only twiddle one knob to adjust two different thing.
Failure domain. An unhandled exception, a slow leak, or an OOM in message processing takes the HTTP server down with it (or vice versa: redeploying the API for an unrelated change restarts the consumer too, with all the reconnect/redelivery churn that implies).
Resource shape. A CPU-heavy message handler competes with request handling for the same GIL when we create a thread to process RabbitMQ messages. And the same container's CPU/memory limits is shared between the two, instead of getting their own, independently sized deployment.

A standalone worker process decouples all 3: scale it on queue depth, deploy/restart it without touching the API, and size its container for the work it actually does.

💡 Takeaway

This more about provisioning, maintaining, horizontal scaling, and monitoring.

"But in NodeJS/Express/Fastify/NestJS we just run it alongside the app"

That's a fair observation, and it works for a while for the same reason the same approach would works for a while in Python: at low volume, none of the three couplings above are painful yet. A few things make it look more tenable in Node than it might seem in Python:

NodeJS's single-threaded, non-blocking I/O model means an async RabbitMQ handler naturally interleaves with request handling without needing an extra thread or process at all, so there's no GIL-contention story like there is with a blocking library such as pika running in a Python thread.
Frameworks like NestJS ship first-class support for bolting a message consumer onto the same app (@nestjs/microservices' hybrid application mode), so it is the path of least resistance, not something bolted on.

None of that removes the scaling/failure-domain coupling, though: a NodeJS/NestJS process consuming RabbitMQ inline still ties consumer count to HTTP replica count, and a handler that blocks the event loop (a CPU-bound computation, a synchronous call, a bad regex) stalls HTTP requests exactly like it would in any single-process design. NodeJS/NestJS teams that hit real throughput or correctness requirements make the same move shown here, usually via a distinct @MessagePattern microservice or a queue-specific worker deployment rather than the main HTTP app.

So this isn't a Python-specific lesson, it's what happens once "a background thing riding along in the API process" needs to be reasoned about as infrastructure rather than as an implementation detail.

The Rule of Shared Libraries

Set up a monorepo whenever you need to share models/services/repositories. For example you can have an api and a worker in the same monorepo. They are separately deployables, but they share one database and one set of SQLAlchemy models/repositories, defined once in shared and imported by both.

At Least Once Delivery Guarantee

In the linked example we have two separate hops, because they have very different guarantees.

Hop 1: RabbitMQ → Worker → Postgres

At-least-once delivery comes from RabbitMQ fundamentals: durable queue + persistent messages + manual ack. Nothing is acked until it's fully processed, so a crash mid-flight just means redelivery to whoever reconnects.
Idempotency comes from the domain itself, not a deduplication table: UserRepository.get_or_create is naturally idempotent because "does a user with this email exist" is a deterministic query independent of how many times you ask. That's "idempotent by construction", which is a lighter-weight cousin of the inbox pattern (a real inbox pattern would track message IDs in a deduplication table for handlers whose side effects aren't naturally idempotent. E.g. "increment balance by $10"). Here you don't need that machinery because the operation happens to collapse to the same result on replay.
No outbox table either. A classic transactional outbox would write "email to publish" into a DB table in the same transaction as the user upsert, then a separate relay process drains that table into RabbitMQ/Redis. Instead, this worker does DB commit → Redis publish → RabbitMQ ack, non-transactionally, and covers the gap by not acking until both steps succeed. If it crashes after commit but before ack, whole process (DB write + Redis publish) reruns. That's why the README's crash table shows the DB write as safe (idempotent) but explicitly flags Redis as possibly duplicated. The ack-gating is doing the job an outbox table would normally do, at the cost of allowing duplicate publishes instead of guaranteeing exactly-once.

So: at-least-once + idempotent handler, achieved via ack ordering, not via outbox/inbox infrastructure, a genuinely solid and simple implementation.

Hop 2: Worker → Redis pub/sub → GraphQL Subscription

This is the part that isn't gracefully handled, and your instinct is right.

Duplicates from hop 1 leak straight through. When the worker crashes after redis.publish but before basic_ack, the redelivery causes a second publish of the same email to any subscriber.
There's currently no way for a client to tell besides treating the email address as some sort of message ID, or sequence number, or more commonly known as idempotency key.

Could you add an idempotency key over the websocket?

Yes, nothing about GraphQL subscriptions or graphql-ws prevents it, it's just app-level payload shape. You'd:

Give each RabbitMQ message a stable ID, derive one deterministically (e.g. hash of the email, or client whom is initiating this whole pipeline can send one, or we could simply use the ID generated and returned by database engine) so that a crash-retry reproduces the same ID rather than minting a new one.

Carry that ID through: RabbitMQ message → Redis publish payload (as JSON: {"id": ..., "email": ...} instead of a bare string) → the Subscription.queue_messages yield → the GraphQL client.

The GraphQL client keeps a small set/LRU of recently-seen IDs and drops repeats — that's effectively an inbox pattern implemented at the client, since the server-side pub/sub layer has no persistence to build a server-side inbox against.

Whether it's worth doing depends on how much a duplicate matters to your subscribers. Since the downstream effect here is "a user row exists", a duplicate push is currently harmless if the client is also just doing an upsert-style process. It only becomes a real problem if a client does something non-idempotent in response to the subscription event (e.g., "send a welcome email every time this fires").

Coding Agents & Workflows

Mohammad Jawad (Kasir) Barati — Tue, 25 Aug 2026 15:58:03 +0000

If you're working with coding agents like Claude Code, GitHub Copilot, or any other AI assistant, you've probably noticed something: they can generate code faster than you can review it. This creates a new set of challenges that traditional development workflows weren't designed to handle.

Here is what I believe will help you. BTW before reading this post try to:

Clone a large open-source project which you feel comfortable working with.
Find a TODO or bug in the code.
Ask your coding agent to fix it.
Apply whatever principles you currently have when working with a coding agent (whatever that looks like today).
Save the results in a separate branch. If you like you can also track how much you spend on it (time and LLM bills).

Keep that branch around, once you've read the post, you'll go back and redo the same task with the principles below applied, so you can compare the two results yourself.

Know Your Tools, Know Yourself

Your choice of tools depends on your experience level, and that choice shapes the whole workflow you're about to build:

Seasoned pros: tend to gravitate toward CLI-based tools. And let's not forget that most of the times CLI offers more features.
Beginners: often find IDEs more comfortable and approachable.

Whichever you pick, you need to learn to be patient with LLMs. They usually tend to generate a ton of text, and rushing past it defeats the point of reviewing it at all.

Give Your Agent Context, Deliberately

When you code, you have a ton of context in your head. So if you are expecting a coding agent to deliver good results you need to give it enough context but beware of context dilution.

Plugins

Browse official plugins.
Pick the most popular or relevant ones for your project type.
Examples:
- feature-dev for building features efficiently.
- code-simplifier to clean up and refactor.
- frontend-design for professional UI/UX work.

⚠️ Warning

If you install superpowers plugin get ready for it to spin up subagents and burn a ton of tokens in the process. Personally I downloaded their brainstorming, and systematic-debugging skill and removed the plugin.

MCP Servers Worth Knowing About

Consider using MCP servers (like Contact7) if they offer exactly what you need. Some plugins may already cover the same functionality, so try to avoid overlap. Commit them to git so everyone has access.

💡 Pro Tip

Install Context7, it is so useful since LLMs do not have always the latest documentations for a library. You can enforce it by adding a .mcp.json:
{
  "mcpServers": {
    "context7": {
      "command": "npx",
      "args": [
        "-y",
        "@upstash/context7-mcp@latest"
      ]
    }
  }
}

Case Study: Why Skills Matter

Rather than relying on general knowledge, develop skills tailored to your project. Imagine you are building the backend checkout service for a massive e-commerce platform (think Shopify, Amazon, or a large retail enterprise). This service handles:

Cart finalization with 10+ items.
Inventory reservation (not just decrementing a number, but holding stock across multiple warehouses).
Tax calculation (different rules per US state, VAT for EU, GST for Australia).
Payment orchestration (Stripe/Paddle/Braintree with 3D Secure fallbacks).
Fraud scoring (an internal ML model that returns a risk score).
Promotions & gift cards (stacking rules).
Event emission (RabbitMQ events for shipping, analytics, and receipts).

Now an engineer on your team asks the coding agent to:

We are releasing a 'Flash Sale' feature tomorrow. We need a new endpoint that accepts a list of product IDs and quantities, bypasses the user's shopping cart, and goes straight to the checkout/payment page. Ensure it validates the flash sale time window."

What happens WITHOUT a skill: the agent reads the prompt, sees "bypass cart" and "straight to payment", and happily writes something like this (LLMs try to follow your instructions as much as possible):

@app.post("/flash-checkout")
def flash_checkout(products, user_id):
    total = 0
    for p in products:
        # 🚨 UNSAFE: Direct DB hit.
        # Inventory is immediately deducted, even if the user's credit card fails (loss of stock).
        db.inventory.update_one({"_id": p.id}, {"$inc": {"stock": -p.qty}}) 
        total += p.price * p.qty

    # 🚨 WRONG: Hardcodes US tax, ignores EU and other regions
    # This is a contrived example and most top LLMs would ask you about it
    total *= 1.08

    # 🚨 BREAKS STATE MACHINE: Direct status update
    # State machine is broken, so the shipping service never picks it up.
    db.orders.insert_one({"user": user_id, "status": "COMPLETED", "total": total})

    # 🚨 NO IDEMPOTENCY: User double-clicking charges them twice
    stripe.charge(amount=total)
    return {"success": True}

Now imagine what would have happened if you had a skill like this in .claude/skills/checkout-orchestration-skill/SKILL.md:

---
name: checkout-orchestration-skill
description: 'Enforces the distributed saga pattern for the e-commerce checkout pipeline. Use this skill whenever implementing or modifying order placement, payment flows, or cart finalization. It mandates idempotency (Redis), inventory reservation (never direct deduction), strict order state machine transitions (OrderStateMachine), tax aggregation (TaxJar/Vertex routing), async fraud handling (800ms timeout), and Kafka event emission (order_placed topic) for eventual consistency.'
---

## Core Architectural Rules

**NEVER** treat checkout as a single CRUD update. It is a **distributed saga**.

1. **Idempotency First**: Every checkout request must include an `idempotency_key`. If the key exists in Redis, return the cached result immediately. 
2. **Reserve, Do Not Deduct**: Never write directly to `inventory` table. Always call `InventoryReservationService.reserve()` which holds stock for 15 minutes (timeout period). Only finalize deduction after `PAYMENT_CONFIRMED` event.
3. **State Machine Only**: The `Order` entity has a strict state machine: `PENDING -> FRAUD_CHECK -> AUTHORIZED -> PAYMENT_CAPTURED -> FULFILLMENT_QUEUE` Never skip states. Never manually set `order.status = 'completed'`. Use `OrderStateMachine.transition('CAPTURE')`.
4. **Tax Agnosticism**: We use TaxJar for US/CA, and Vertex for EU. Always call `TaxAggregator.get_tax()` which handles the routing under the hood. Do NOT hardcode tax rates.
5. **Fraud Timeout**: The fraud check is async. If it takes > 800ms, we proceed with `FRAUD_PENDING` status and place a dead-letter retry, but we do NOT block the user.
6. **Eventual Consistency**: After capturing payment, push exactly one event to the `order_placed` Kafka topic. Must include the `correlation_id` for tracing.

And now if you send the same prompt to the coding agent, it generates real, production-grade, domain-aware code:

@app.post("/flash-checkout")
def flash_checkout(products, user_id, idempotency_key):
    # 1. Enforce Idempotency
    cached = redis.get(idempotency_key)
    if cached: return cached

    # 2. Validate Flash Sale window (domain logic)
    if not timezone.now() in flash_sale_window:
        raise HTTPException(400, "Flash sale not active")

    # 3. Build line items
    line_items = [LineItem(p.id, p.qty) for p in products]

    # 4. Reserve inventory using the domain service (DOES NOT deduct yet)
    reservation_ids = InventoryReservationService.reserve_bulk(line_items)

    # 5. Get tax through the aggregator (handles US/EU routing)
    tax = TaxAggregator.get_tax(line_items, user_id)

    # 6. Initiate state machine
    order = OrderStateMachine.create(
        user_id=user_id, 
        line_items=line_items, 
        tax=tax,
        status="PENDING"
    )

    # 7. Fraud check (async, non-blocking)
    fraud_score = FraudScorer.submit_async(order.id)

    # 8. Authorization (not capture, just auth to check funds)
    auth_token = PaymentGateway.authorize(order.total, user.payment_method)

    # 9. Transition to authorized state
    OrderStateMachine.transition(order.id, "AUTHORIZE", context={"auth_token": auth_token})

    # 10. Emit Kafka event for downstream processing
    event_bus.publish("order_placed", {
        "order_id": order.id, 
        "correlation_id": request.headers["X-Correlation-ID"]
    })

    response = {"order_id": order.id, "status": "AUTHORIZED"}
    redis.setex(idempotency_key, 3600, json.dumps(response))
    return response

Skills are part of the harness engineering.

Add Commands

You can e.g. create a command for labeling GitHub issues:

mkdir -p .claude/commands
cat > .claude/commands/label-github-issue.md << 'EOF'
---
description: Fetch a GitHub issue and apply appropriate labels using gh
argument-hint: <github-issue-url>
allowed-tools: Bash(gh issue view:*), Bash(gh label list:*), Bash(gh issue edit:*), Bash(gh repo view:*)
---

You are labeling a GitHub issue. The issue URL is: $ARGUMENTS

Steps:
1. Parse the URL to extract the owner, repo, and issue number.
2. Run `gh issue view <number> --repo <owner>/<repo> --json title,body,labels` to read the issue's current title, body, and existing labels.
3. Run `gh label list --repo <owner>/<repo>` to see which labels actually exist in this repo. Only use labels from this list, if you strongly believe we lack certain label just let the user know as a side note, do NOT stop here even if you believe certain labels are missing.
4. Based on the issue's title and body, decide which existing labels best apply (e.g. bug, enhancement, documentation, question, good first issue, priority levels, area/* labels, etc).
5. Apply the chosen labels with `gh issue edit <number> --repo <owner>/<repo> --add-label "label1,label2"`.
6. Report back to the user: which labels you applied and a one-line reason for each. Also report back the labels which you believe are good to add with a single line as to why.

If the issue already has labels that are still appropriate, leave them and only add what's missing. If no labels in the repo genuinely fit, say so instead of forcing one on.
EOF

And ensure you are committing them so others will be using the same commands when needed!

`AGENTS.md`: Your Agent's Navigation System

The most important investment you can make is in documenting a navigation system for your AI agents. You wanna commit and push this documentation to your VCS. And I believe you already know it but I usually just link to AGENTS.md in CLAUDE.md. This way I do not have to copypaste or maintain both.

Also you can have a AGENTS.local.md/CLAUDE.local.md for your local setup which is not committed to git. So in AGENTS.md we usually put stuff such as:

Bash commands.
Common MCP tools.
Style guides, this can be:
- Frontend UX.
- Best practices and design patterns.
- Testing strategies (although you can move this to a separate markdown file and just link it).
Architectural decisions.
A link to important files.

Basically anything you usually need to work on that codebase. But make sure to keep it short and concise since if it is too long then it just take up space in your context window with no real benefits.

❗ Note

You can have a AGENTS.md/CLAUDE.md in subdirectories of a project, and coding agents load them when reading and working with that directory's files. So there you can be more meticulous with your instructions.

I would also like to make it crystal clear that LLMs love to optimize for coverage percentages. They'll often write brittle tests that:

Overly mock dependencies.

Test every code path without regard to actual functionality.

Break when you refactor.

But what we want from our test suite cases are:

Good tests: Test the functionality you're building.

Refactoring-safe: Shouldn't break when you reimplement. NOTE, we are assuming the APIs remain the same.

Logic-catching: Should break when logic breaks.

Process Discipline

Embrace Trial and Error

Adopt an experimenter's mindset.
Start a task, see how it goes.
If it doesn't work, abandon it and try another approach.
This flexibility is critical, don't be afraid to throw away what doesn't serve you.

Manage Context Proactively

Check how much of the context is filled, e.g. in Cloud Code, you must use /context frequently.
Don't always wait for coding agent to compact the context. You can do it yourself too if for example you need a fresh start and just wanted to summarize the current state in a more controlled manner.
Clear your context with /clear and start fresh when needed.
Write summaries to markdown to preserve important info before clearing. This is where a PLAN.md or similar progress log earns its keep: treat it as your project memory and communication log, not just a one-off note.
Work in bite-sized chunks. So instead of:

  Hey agent, refactor this 50-person project's entire codebase.

Try to ask LLM:

  Break this large task into 10 small, specific steps. Where each step should be independently:
  - Specifiable.
  - Testable.
  - Reviewable by a human.

Explore, Plan, Confirm, Then Code

It is a good practice to start with writing a PLAN.md to brainstorm what you wanna do. In fact, that's why I downloaded the brainstorming skill from the superpowers plugin. So next time you wanna develop a feature which is big enough for the LLM to derail or misunderstand how it should work, first ask it to brainstorm and write a plan for you. Review and work on that plan, then start with implementing it. So if I wanted to visualize this:

[Explore] => [Plan] => [Confirm] => [Code] => [Commit]

The prompt would be:

Figure out the root cause for issue #983, then propose a few fixes. Let me choose an approach before you code. ultrathink

💡 Pro Tip

"ultrathink" is a special "magic keyword" you can add anywhere in your prompt to trigger a maximum reasoning depth mode for that specific request.

Give It Acceptance Criteria

If you give your coding agent a way to measure how good it did, the results would be closer to what you wanted since it will iterate over it. For this usually we can:

Write tests => commit => code => iterate => code: > Write tests for @utils/markdown.ts to make sure links render properly (note the tests won't pass yet, since links aren't yet implemented). Then commit. Then update the code to make the tests pass.
Write code => screenshot results => iterate: > Implement [mock.png]. Then screenshot it with Puppeteer and iterate until it looks like the mock.
Write acceptance criteria in a markdown files similar to what you usually get in a Jira ticket written by a product owner: > Take @specs/payment-flow.md as the source of truth. Write e2e integration tests that mirror each acceptance scenario exactly. Run the tests, and they should fail initially. Then implement the @api/payment/ module and keep iterating until every test passes. Let me know which ACs were tricky or ambiguous. Or you can e.g. write > Let's work on https://acmecorp.atlassian.net/browse/PAY-842. Treat each bullet point in acceptance criteria section as a hard pass/fail criterion. Build the @app/checkout feature, then simulate a full user journey through the UI (using Playwright) and check each block. If a step fails, pause, fix, and re-run until the entire ticket is green. Do not mark the ticket as done until all checks pass.

Ownership & the Safety Net

Use Git Heavily

Git is your safety net.
Commit often.
Branch freely.
Roll back with confidence.

You Own Every Line You Push

No AI slop, look out for:
- Long, rambling files that are hard to review.
- Overly defensive code (excessive error handling, validation).
- Generated code that's "technically correct" but architecturally poor.
Avoid:
- Unnecessary tests.
- Extra READMEs.
- Emojis.
Be ruthless about code quality.
Ensure everything you push is clean, purposeful, and well-written.

Remember I have already talked about it in my post on pragmatic agentic programming. This is code you stand behind.

Key Takeaway

Create a culture of rejecting mediocre AI-generated code.

Bonus: Use Your Agent to Onboard, Not Just to Code

It is always a good idea to have another engineer to onboard you. But nowadays you can simply try to utilize LLMs and coding agents to help you with that as well. Ask questions such as:

How is @RoutingController.py used?
How do I make a new @app/services/ValidationTemplateFactory?
Why does recoverFromException take so many arguments? Look through git history to answer.
I'm new to the codebase. Give me a mental model of the @core/ folder, what are the primary responsibilities of each subdirectory, and which one should I touch if I need to modify the logging behavior?

Or you can ask other questions outside of onboarding process:

Why did we fix issue #18363 by adding the if/else in @src/login.ts API?
In which version did we release the new @api/ext/PreHooks.php API?
Look at PR #9383, then carefully verify which app versions were impacted.
What did I ship last week?
If I deprecate the @utils/legacyParser.ts function, which modules across the entire monorepo are still relying on it? Show me the call stack in each case.
Why was the @api/middleware/AuthGuard implemented as a class instead of a factory function? Look at the original PR and the team's discussion history to explain the trade-offs.

So your coding agent, if it is smart enough, will be able to look at git history whenever needed, and let's imagine you have an MCP server to return your microservice architecture. Then it will use it to gain a deeper understanding of how service A interacts with service B. Sometimes you have to be specific so it knows it must use the MCP server. And there are times it can figure that out itself.

The Asymmetry Problem

It's becoming increasingly easy to generate tons of code, and the burden is shifting to humans to review it all. Everything above is really one answer to this same problem, combat it with:

Disciplined review processes.
Challenging agents to write succinct, clean code.
Making sure agents don't overwhelm reviewers.

Now go back to that first branch you made. Apply what you've read:

Write a detailed AGENTS.md.
Use plugins where they fit.
Break the work into small, reviewable steps.
Review the output critically.

Save the results in a separate branch, and compare it against your first attempt. That difference is the whole point of this post.

References

Mastering Claude Code in 30 minutes.