DEV Community: Vahid Aghajani

Docker Compose vs an Orchestrator — Compose Is a Command, an Orchestrator Is a Loop

Vahid Aghajani — Mon, 12 Oct 2026 15:13:58 +0000

Originally published on software-engineer-blog.com.

Docker Compose versus an orchestrator. Both start your containers from one file, so many people think the orchestrator is only a bigger version of Compose. It is not. Compose is a command. An orchestrator is a loop.

This article follows one service that wants three copies, and one machine that dies at three in the morning. It reads one Compose file, one small Python model and one Kubernetes file, line by line. At the end you know what the restart line protects, what it cannot protect, and what an orchestrator costs you.

The server dies at 03:00. Who starts your containers again?

You run one service as three containers on one server. A container is one packaged, running copy of your program.

One Compose file describes all three copies, and one command starts them. For months this works.

time	what happens
for months	3 copies run on one server
03:00	the server loses power
03:00	3 copies become 0
	the file still says 3

But a file cannot do anything. And the command that read the file finished months ago.

So here is the question:

When the machine dies, who starts your containers again?

What this article covers (and what it does not)

The subject is DevOps: the work of running software on servers. This article covers one decision inside it: Compose on one machine, or an orchestrator on many machines.

An orchestrator is a program that keeps your containers running across a group of machines.

Left alone on purpose: what Docker and Kubernetes are, as two tools, and container images. Both have their own episodes.

Where does this live? It is built into your tools. You will not write an orchestrator yourself. Kubernetes, Docker Swarm and Nomad are orchestrators, and they are already built. What you touch is small:

a file that says how many copies you want;
the number of machines;
the failure you must plan for.

This decision matters the first time one machine is not enough, or one machine is not safe enough.

A light switch and a thermostat

Think of two things on a wall.

The first thing is a light switch. You press it once, and the lamp turns on. After that, the switch does nothing. If the lamp breaks at night, the switch does not know, and the room stays dark until you come back.

The second thing is a thermostat. You give it a target: twenty-one degrees. It measures the room again and again. When a window opens and the room gets cold, the thermostat turns the heater on. Nobody told it to do that.

	a light switch	a thermostat
what you give it	one press	a target: 21 degrees
what it does after that	nothing	measures the room, again and again
when something breaks at night	the room stays dark until you come	it turns the heater on by itself
in this article	Compose: a command	an orchestrator: a loop

Compose is the light switch. An orchestrator is the thermostat.

The Compose file, line by line

Here is the Compose file. It is real, and it is short.

services:
  web:
    image: registry.example.com/web:1.4
    restart: unless-stopped
    deploy:
      replicas: 3

# then, on the one machine, you run:
#   docker compose up -d

The service is called web.
The image line says which packaged program to run.
The restart line tells the Docker engine to start a container again if that container stops. The Docker engine is the program on the machine that runs containers.
The replicas line says three. A replica is one running copy.

Now you run one command: docker compose up.

step	what the command does
1	reads this file, once
2	asks the engine on this one machine for three containers
3	is finished

The whole system, drawn once

Part	What it is	What it does
The file	one YAML file, one number	holds the number you want: 3
The command	`docker compose up`, one process	reads the file once, starts 3 copies, then exits
Node A	1 server, 1 container engine	runs the copies

A node is one machine that runs containers.

Two arrows join the parts. The file goes to the command: read once. The command goes to node A: start 3. Both arrows point one way only. Nothing flows back, and after the command exits nothing runs in the middle any more.

So remember one number for this whole article.


wanted	3
running	3
machines	1

The restart line: what it protects, and what it cannot

But the file has a restart line. Is that not enough? Test it.

The first failure: one container crashes. The engine on node A sees it, and starts that container again in a few seconds. Running: three again. Good. (The Linux processes episode shows the same idea with systemd.)

The second failure: node A itself dies. The engine was on node A, so the engine is dead too.

time	what happens	running
before	3 copies on node A	3
a crash	one container crashes, the engine restarts it	2, then 3
03:00	node A dies, and the engine dies with it	0
03:40	a person wakes up and fixes it	3

That is forty minutes with zero copies.

So the restart line protects a container. Here is the second question:

What protects you from a dead machine?

The answer: a loop that never exits

An orchestrator answers that question with a loop.

This code is a small model of what Kubernetes, Swarm and Nomad do inside. You do not copy it into your service. It only helps you reason about them.

DESIRED = 3          # copies of web we asked for
ROOM = 2             # copies one node can hold

def reconcile(nodes):
    alive = {n: c for n, c in nodes.items() if c is not None}   # keep only the nodes that are alive
    missing = DESIRED - sum(alive.values())                     # desired, minus the copies that run
    while missing > 0:
        free = [n for n in sorted(alive) if alive[n] < ROOM]    # nodes that still have room
        if not free:
            break                                               # no room: those copies stay pending
        target = min(free, key=lambda n: alive[n])              # the node with the fewest copies
        alive[target] += 1                                      # start one copy there
        missing -= 1
    return alive, missing

while True:
    state, pending = reconcile(read_nodes())
    time.sleep(5)                                               # again every five seconds, forever

DESIRED is three: the copies we asked for.
ROOM is two: one node can hold two copies.
The reconcile function first keeps only the nodes that are alive.
Then it counts what is missing: desired, minus the copies that run.
While something is missing, it lists the nodes that still have room.
If no node has room, it stops, and those copies stay pending.
Otherwise it picks the node with the fewest copies, and starts one copy there.
At the bottom, the function runs again every five seconds, forever.

The same picture, with one change

In the middle there is now a control loop, and it never exits.

Part	What it is	What it does
The file	one YAML file, one number	holds the number you want: 3
The control loop	the orchestrator, a service that never exits	reads the status of every node and starts what is missing
Nodes A, B, C	3 servers, 1 container engine each	run one copy each

You send the file once, and the loop keeps the number. The arrows between the loop and the nodes now go both ways: status comes up, start commands go down.

Now node A dies, the same machine as before.

step	what the loop does	node A	node B	node C	running
before		1	1	1	3
1	asks every node for its status; node A does not answer	dead	1	1	2
2	counts: desired 3, running 2, missing 1	dead	1	1	2
3	node B and node C each hold one copy, so it picks node B	dead	2	1	3

Running: three again.

No person woke up. And two copies answered users during the whole time.

What you write: a file, not the loop

So what do you write? Not the loop. You write a file, and it is very close to the Compose file.

This file is for Kubernetes, and it is called a Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      containers:
        - name: web
          image: registry.example.com/web:1.4
          resources:
            requests:
              memory: 2Gi

replicas: 3. That is the same number as before.
The image is the same too.
The memory line is new. It says that one copy needs two gigabytes. The orchestrator uses that line to decide which node has room.

You send this file once, and the control loop keeps it.

Now the cost in time. Kubernetes does not replace the copies at once. By default, it waits about five minutes for a silent node to come back.

The catch: the loop cannot create machines

And here is the catch.

The loop can start containers. It cannot create machines. So the room must exist before the failure.

The check is one line.

def survives(nodes, room, desired, failures): return (nodes - failures) * room >= desired

survives(3, 2, 3, failures=1)   # (3 - 1) * 2 = 4    4 >= 3    True
survives(3, 2, 3, failures=2)   # (3 - 2) * 2 = 2    2 <  3    False
# False: 2 copies run, 1 copy stays pending

Take the nodes, subtract the failures, and multiply by the room on each node. The result must be at least the desired number.

nodes	failures	room that is left	desired	result
3	1	2 nodes × 2 = 4	3	4 is at least 3: we survive
3	2	1 node × 2 = 2	3	2 is less than 3: 2 copies run, 1 stays pending

You pay for spare room, or the loop has nowhere to put a copy.

The bill, because it is real

So which one do you choose?

	Compose	an orchestrator
what it is	a command that runs once	a loop that never exits
machines for the work	1	3 in our example, with spare room
machines for the control loop	0	usually 3 of its own
a container crashes	the engine restarts it	the loop repairs it
a machine dies	0 copies until a person comes	no outage: 2 copies keep answering, and the third is started again
what you learn and run	one file, one command	more machines and more to learn

Why three machines for the control loop? Because the control loop must survive a dead machine too. Three can lose one, because two of three still agree.

And what you pay for it:

what you gained	what you pay
A dead machine causes no outage.	Three nodes with spare room, not one.
No person has to wake up at 03:00.	A control plane of three machines, and more to learn.
The number in the file is kept, not only started.	A default wait of about five minutes in Kubernetes before a silent node is replaced.

Compose is right for your laptop, for tests, and for a small product that can accept a short outage.

Choose the orchestrator when one machine is not enough, or when forty minutes of zero is not acceptable.

For your next interview

What is the difference between Docker Compose and an orchestrator?

Docker Compose is a command. It reads one file and starts your containers on one machine, one time. An orchestrator, such as Kubernetes, is a control loop over many machines. It compares what you want with what is running, again and again, and it repairs the difference.

Three words to remember:

Desired state: the number of copies you asked for.
Reconcile: compare desired with running, and start what is missing.
Headroom: the spare room on other nodes that a repair needs.

The one-line verdict

Compose is a command that reads your file once on one machine; an orchestrator is a loop that keeps the number, so stay on Compose while a short outage is acceptable, and move to an orchestrator when one machine is not enough or not safe enough, and pay for the spare room first.

One thing to go and look at tonight: open your Compose file and count the machines it runs on. If the answer is one, decide how many minutes of zero copies your product can accept.

References and further reading

On the Compose file and the command

Docker, Compose Deploy Specification (Docker documentation) — replicas specifies the number of containers that should be running at any given time: docs.docker.com — Compose Deploy Specification
Docker, docker compose up (Docker documentation) — the command builds, creates and starts the containers for a service, and with --detach it leaves them running in the background: docs.docker.com — docker compose up

On the restart line

Docker, Start containers automatically (Docker documentation) — restart policies control whether a container starts again automatically when it exits, and unless-stopped is one of them: docs.docker.com — Start containers automatically

On the control loop and the desired state

The Kubernetes Authors, Controllers (Kubernetes documentation) — a control loop is a loop that never ends and regulates the state of a system; the page uses the thermostat as its example, and each controller moves the current state closer to the desired state: kubernetes.io — Controllers
Burns, Grant, Oppenheimer, Brewer and Wilkes, Borg, Omega, and Kubernetes (ACM Queue, 2016) — the reconciliation controller loop, which compares a desired state with the observed state, is shared by all three systems and makes them robust to failures: research.google — Borg, Omega, and Kubernetes

On the Deployment file, the memory line and the five-minute wait

The Kubernetes Authors, Resource Management for Pods and Containers (Kubernetes documentation) — the scheduler uses the resource request of a container to decide which node to place it on, and a Pod that fits on no node stays unscheduled until a place is found: kubernetes.io — Resource Management for Pods and Containers
The Kubernetes Authors, Taints and Tolerations (Kubernetes documentation) — Kubernetes adds a toleration of 300 seconds for a node that is not ready or unreachable, so Pods stay bound to that node for 5 minutes after the problem is detected: kubernetes.io — Taints and Tolerations

On the three machines of the control loop

Docker, Raft consensus in swarm mode (Docker documentation) — Raft tolerates up to (N-1)/2 failures and needs a majority of (N/2)+1 members to agree, so three managers can lose one: docs.docker.com — Raft consensus in swarm mode

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Docker Compose vs an orchestrator — Explained in Detail on YouTube

Container Images, Layers & Registries — An Image Is Not One Big File: One Changed Line, 150 MB Moved, and the Dockerfile Order That Makes It 2 MB

Vahid Aghajani — Sat, 10 Oct 2026 09:52:26 +0000

Originally published on software-engineer-blog.com.

Container images, layers, and registries. Many people think that an image is one big file. It is not. It is a stack of smaller files, and each one is stored and sent on its own.

This article follows one image of 200 megabytes and one changed line of code. It reads two Dockerfiles, two small Python functions and one deploy file, line by line. At the end you know why the order of a Dockerfile decides how much is built and moved, and why a tag does not tell you what really runs.

You changed one line. Why did 150 MB move?

You run one service as three pods. A pod is one running copy of your service. Every deploy builds a new container image, and every server downloads it.

The image is two hundred megabytes. Today you change one line of code.

step	what happens
1	the build installs every dependency again
2	the build uploads 150 MB
3	every server downloads 150 MB
	your change was one line, in a file of a few kilobytes

So here is the question:

You changed one line. Why did one hundred and fifty megabytes move?

What this article covers (and what it does not)

The subject is DevOps, and one part of it: how a container image is built, stored and delivered.

An image is the packed file system that a container starts from.

This article covers three things:

layers, which are the pieces of an image;
the build cache, which decides what is built again;
the registry, which is the server that stores images.

Left alone on purpose: the running process and its signals. That has its own episode.

Where does this live? It is built into your tools. You will not write this machinery yourself. It is built into Docker, Podman, containerd and Kubernetes. What you write is the Dockerfile, and the image name in your deploy file. Those two decide your build time, and what really runs in production.

A reprinted textbook and a thin clear sheet

Here are two ways to publish a correction to a thick textbook.

The first way: print the whole book again for every correction. Three corrections give you three thick books, and every reader must carry all of that paper home.

The second way: keep one book. Write each correction on a thin clear sheet, and lay the sheet over the page. To publish a correction, you send only the sheet. Every reader already has the book.

	reprint the book	lay a sheet over the page
three corrections	three thick books	one book, three thin sheets
what you send	the whole book, every time	only the new sheet
what the reader already has	nothing that helps	the book
in an image	one big file, copied whole	a base layer, plus thin layers

A container image works the second way. The book is the base layer. Each sheet is one layer on top of it.

The whole system, drawn once

Part	What it is	What it does
The build machine	1 CI runner, with 1 layer cache	reads your Dockerfile and builds the image
The registry	1 HTTP server	stores every layer under its digest
The nodes	2 servers that run your pods	pull layers and start containers

A digest is a hash of the bytes, so the same bytes always get the same name. A node is one server that runs your pods.

Two arrows join the parts. The build machine pushes layers to the registry. Each node pulls layers from the registry. Layer bytes move in one direction only: a node never sends a layer back.

Now one image for the whole article. Its name is shop.

layer	size
the base layer	50 MB
the dependencies	148 MB
your code	2 MB
the image	200 MB

The Dockerfile that most people write first

A Dockerfile is the list of steps that builds an image.

FROM python:3.12-slim                      #  50 MB
COPY . /app                                #   2 MB
RUN pip install -r /app/requirements.txt   # 148 MB
CMD ["python", "/app/main.py"]             # no layer

Line one is FROM. It starts from a base image that already has Python. That is the 50 MB layer.
Line two is COPY. It copies your whole project into the image. That is the 2 MB layer.
Line three is RUN. It installs the dependencies. That is the 148 MB layer.
Line four is CMD. It only records which command to start, so it adds no files.

step	layer	layer size
`FROM`	base image	50 MB
`COPY . /app`	your code	2 MB
`RUN pip install`	dependencies	148 MB

Three steps made three layers.

Remember the order: the code layer comes before the dependency layer.

The build cache: one key for each step

Now a small model of what the build tool does inside. It is not code for your service. It helps you reason about the tool.

from hashlib import sha256

def key(prev, step, files):
    text = prev + step + files
    return sha256(text.encode()).hexdigest()

def build(steps, cache):
    prev, rebuilt = "", []
    for step, files in steps:
        prev = key(prev, step, files)
        if prev not in cache:
            cache.add(prev)
            rebuilt.append(step)
    return rebuilt

steps = [("FROM base", ""), ("COPY . .", "app.py"), ("RUN pip install", "")]

Every step gets a cache key. The key is a hash of three things: the key of the step before it, the text of the step, and the files that the step copies.
The build function walks the steps in order.
If the key is already in the cache, the tool reuses the old layer.
If the key is new, the tool runs the step again.

Now change one line of code.

step	its key	what the tool does	size
`FROM`	the same	reuses the layer	50 MB
`COPY . /app`	new, because a copied file changed	runs the step again	2 MB
`RUN pip install`	new, because its key includes the key before it	runs the step again	148 MB

The COPY step gets a new key. The next step includes that key, so its key changes too.

One changed step rebuilds every step after it: 2 + 148 = 150 MB.

The fix: change the order

Put the steps that change rarely first, and the steps that change often last.

FROM python:3.12-slim
COPY requirements.txt /app/
RUN pip install -r /app/requirements.txt
COPY . /app
CMD ["python", "/app/main.py"]

Line two now copies only one file: the list of dependencies.
Line three installs them.
Line four copies your code, last.

Now change one line of code again.

step	cache	layer size
`FROM`	reused	50 MB
`COPY requirements.txt`	reused	0 MB
`RUN pip install`	reused	148 MB
`COPY . /app`	REBUILT	2 MB

Line two copies the same file as before, so its key is the same. Line three has the same key too. The tool reuses both layers. Only line four runs again.

One line of code now rebuilds 2 MB, not 150.

The registry asks one question for every layer

The registry uses the same idea. Here is the model of a push.

def push(layers, registry):
    sent = 0
    for digest, size in layers:
        if digest not in registry:      # do you have this digest?
            registry[digest] = size     # no: upload this layer
            sent += size
    return sent

pull = push    # the same loop, against the disk of the node

For every layer, the tool first asks the registry one question: do you already have this digest?
If the answer is yes, the tool sends nothing.
If the answer is no, the tool uploads that layer.
A pull works the same way, against the disk of the node.

Now run both Dockerfiles after the one-line change.

order	rebuilt	pushed	pulled per node
code first	150 MB	150 MB	150 MB
code last	2 MB	2 MB	2 MB

Layers that did not change never move again.

The trap: three pods, one tag

Here is the same picture, one day later.

Your team pushes every build under the same tag: shop:prod. A tag is a readable name that points at one image.

day	what happens
Monday	nodes one and two pull `shop:prod`. They keep Monday's image on disk.
Tuesday	you push a new build under the same tag. The registry moves the tag to the new image.
Tuesday	the cluster adds node three. It has an empty disk, so it pulls the tag, and it gets Tuesday's image.
Tuesday	nodes one and two already have that tag on disk, so they pull nothing.

Here is the second question:

Three pods run the same tag. Are they running the same code?

The answer: a tag can move, a digest cannot

No. Two pods run Monday's code, and one pod runs Tuesday's code.

The cause is that a tag can move. A digest cannot. The digest of an image is a hash of its list of layers. If one byte changes, the digest changes.

So this is the guard. This is the deploy file that Kubernetes reads.

# by tag: a tag can move
image: shop:prod

# by digest: a hash of its list of layers, so it cannot move
image: shop@sha256:9f2c41d07b6e58a3c1f04e7d92ab35c68e10f7a4d3b925c06e8f1a7b4d2c93e5

The first line names the image by tag.
The second line names it by digest, after an at sign.

Now every node must run exactly those bytes. A node that holds a different image pulls the right one.

image named	node one	node two	node three
by tag	monday	monday	tuesday
by digest	tuesday	tuesday	tuesday

Give each build its own tag, and deploy by digest.

The catch: a layer only adds

There is one catch that surprises people.

A layer only adds changes. It never removes anything from the layers before it.

step	what it does	what the image holds
1	copies a password file into the image	a layer with the file
2	deletes that file	a new layer that says: this file is gone
	the final file system	does not show the file
	the earlier layer	still holds the file

Anyone who pulls the image can read that earlier layer. The image also did not get smaller.

So never copy a secret into an image.

The bill, because it is real

One deploy is one push and three pulls, one pull for each node. So four moves.

	code first	code last
one move	150 MB	2 MB
one deploy: 4 moves	600 MB	8 MB
ten deploys a day	6,000 MB	80 MB

6,000 divided by 80 is 75.

The same service, the same code, the same image: 75 times less to build and move.

And what you pay for it:

what you gained	what you pay
A one-line change rebuilds and moves 2 MB, not 150.	The order of the Dockerfile is now a rule that the team must keep.
Every node runs exactly the same bytes.	The deploy file names a long digest, so a tool must write it for each build.
Layers that did not change never move again.	A layer never forgets: one copied secret stays in the image.

For your next interview

What is a container image, and why does the order of a Dockerfile matter?

A container image is a stack of read-only layers, plus a small file that lists them. Tools like Docker and containerd store, push and pull each layer on its own.

Four words to remember:

A layer: the files that one build step added.
A digest: a hash of the bytes, which can never point at different bytes.
A tag: a readable name, which can be moved.
A registry: the server that stores layers by digest.

The order matters because one changed step rebuilds every step after it.

The one-line verdict

An image is a stack of layers that are cached, pushed and pulled one by one, so put the steps that change rarely first and your code last, and name the image by digest, because a tag can move.

One thing to go and look at tonight: open the Dockerfile of your main service. Find the line that copies your code, and the line that installs the dependencies. Check which line comes first.

References and further reading

On layers and digests

Open Container Initiative, OCI Image Format Specification: Image Manifest — an image is a manifest that lists its layers in order, and each layer is named by its digest: github.com — image-spec, manifest.md
Open Container Initiative, OCI Image Format Specification: Content Descriptors — a digest identifies content by a collision-resistant hash of its bytes, such as SHA-256: github.com — image-spec, descriptor.md

On the build cache and the order of the steps

Docker, Docker build cache (Docker documentation) — once a layer changes, all the layers after it must be rebuilt as well: docs.docker.com — Build cache
Docker, Optimize cache usage in builds (Docker documentation) — steps that change often belong near the end of the Dockerfile, with the dependency file copied and installed before the rest of the code: docs.docker.com — Optimize cache usage
Docker, Build cache invalidation (Docker documentation) — the real rules behind the cache key; for COPY and ADD the builder computes a checksum from the files involved: docs.docker.com — Cache invalidation

On the registry, push and pull

Open Container Initiative, OCI Distribution Specification — a client checks whether the registry already has a blob with a HEAD request on its digest, and a tag is a human-readable pointer to a manifest: github.com — distribution-spec, spec.md

On tags, digests and the pull policy

The Kubernetes Authors, Images (Kubernetes documentation) — with the pull policy IfNotPresent the kubelet skips the pull when the image is already on the node; if a tag changes you can end up with a mix of Pods on old and new code, and a digest names one specific version of the image: kubernetes.io — Images

On the catch: a layer only adds

Open Container Initiative, OCI Image Format Specification: Image Layer Filesystem Changeset — a removal is stored as a whiteout file in the new layer, so the earlier layer is not changed: github.com — image-spec, layer.md
Docker, Build secrets (Docker documentation) — build arguments and environment variables are the wrong place for a secret, because they persist in the final image; a secret mount gives the build access without storing it: docs.docker.com — Build secrets

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Container images, layers & registries — Explained in Detail on YouTube

async/await Explained: How One Thread Serves a Hundred Waiting Tasks

Vahid Aghajani — Tue, 15 Sep 2026 08:23:49 +0000

Originally published on software-engineer-blog.com.

Every async tutorial starts at the syntax. Start one level below it instead.

A hundred requests are in flight on your server. Every one of them is waiting on the database. Now take away their hundred threads, and a single question is left standing: where does each half-finished function keep its place?

That question is all async / await really answers. Everything else — the keywords, the coloured-function arguments, the library churn — is downstream of it.

A thread is a stack, and a stack is a bookmark

Ask what a thread actually buys you and the answer is disappointingly concrete: a stack. That stack exists to hold two things — your local variables, and the line you are on. That is it. When the CPU switches away from your thread and comes back, the stack is how it knows where "back" was.

Look at a perfectly ordinary handler:

def get_order(order_id):
    started = time.time()           # a local
    row = db.query(order_id)        # ← 200 ms of waiting
    elapsed = time.time() - started # a local
    return render(row, elapsed)

Time it honestly and it spends maybe two milliseconds computing and two hundred milliseconds waiting. For that entire 200 ms the thread is doing nothing at all. It is not slow; it is parked. And it is holding a full stack open the whole time, purely so that when db.query returns, something remembers that started was 1726... and that the next line to run is the one assigning elapsed.

That is the bill nobody itemises. On Linux a pthread reserves 8 MB of address space for its stack by default (ulimit -s), committed lazily page by page; runtimes that choose their own are usually in the ~1 MB neighbourhood. Round it to a megabyte and the arithmetic is brutal:

100 parked requests → ~100 MB of memory held open to remember three variables and a line number.

You are not paying for work. You are paying for bookkeeping, at the price of a data structure built for something far more general.

`await` is a suspension point, not "go faster"

Here is the misreading that survives most tutorials: await does not make the database answer sooner. It does not add a thread. It does not parallelise anything.

await marks a suspension point — the line your function is allowed to stop on.

async def get_order(order_id):
    started = time.time()
    row = await db.query(order_id)   # ← stops HERE
    elapsed = time.time() - started
    return render(row, elapsed)

When control reaches that await, three things happen in order:

The function stops, mid-body, with two lines still unexecuted.
It registers interest: wake me when the row is ready.
It hands control back to whoever is running it.

Step 3 is the one that pays. The thread is not parked on your function any more — it is free, immediately, to go run somebody else's.

But step 1 leaves a debt. A function that stopped halfway still has a started value and a resume point, and those have to live somewhere.

Where the state goes: a frame on the heap

This is the whole trick, and it is smaller than it sounds.

Instead of keeping a megabyte-sized stack alive to hold three values, the runtime copies the locals plus the resume point into a small object on the heap — a coroutine frame. A few hundred bytes. In CPython you can hold one in your hand:

async def get_order(order_id): ...

coro = get_order(42)      # nothing has run yet
print(type(coro))         # <class 'coroutine'>
print(coro.cr_frame.f_locals)  # the locals live here, on the heap

Nothing executed when you called it. Calling an async function does not run it — it builds the bookmark.

A stack is a room you rent to hold your place. A coroutine frame is a slip of paper with your place written on it. Both remember the same two facts. One costs a megabyte of reserved address space; one costs the size of the variables you actually have.

100 parked requests → ~30 KB of heap frames, rather than ~100 MB of stacks. Same information. Three orders of magnitude apart.

One thread, a ready queue of bookmarks

Now the loop. One thread holds a ready queue of those frames and does something almost boringly simple, forever:

Take the next ready frame off the queue.
Run it up to its next await.
Put it aside; take the next one.
When an I/O completes, push that frame back onto the ready queue.

Step 4 is where the operating system earns its keep. The loop is not spinning and checking each socket in turn — it blocks in one epoll_wait-style call that says wake me when any of these three hundred sockets is ready, and the kernel hands back exactly the ones that are.

async def main():
    # a hundred bookmarks, one thread
    await asyncio.gather(*(get_order(i) for i in range(100)))

gather does not start a hundred threads. It creates a hundred frames, and every one of them runs on the same thread, up to its first await, in turn.

Resuming: the line after the `await`

When your row finally arrives, the frame rejoins the ready queue, the loop reopens it, and execution continues on the line after the await — with started exactly as you left it.

It did not restart. It did not re-run time.time(). It was parked, with no thread held open to wait for it.

That is the entire mechanism. A suspension point, a bookmark on the heap, and a loop that reopens bookmarks in the order they become ready.

	Thread per request	async / await
What holds your place	A stack	A frame on the heap
Cost per waiting request	~1 MB (8 MB reserved on Linux)	Hundreds of bytes
Who decides to switch	The OS scheduler, at any instruction	Your code, only at an `await`
Cost of a switch	Kernel context switch	A function return, in user space
Realistic concurrent waiters	Thousands, then memory bites	Tens of thousands on one thread
Helps CPU-bound work	Yes, across cores	No — one thread, one core
Fails when	You need many more waiters than RAM allows	One task refuses to yield

The three catches, which are the whole interview

Everything above is the part tutorials cover. These three are the part they skip — and they are exactly what a good interviewer probes.

1. You bought concurrency on waiting, never on computing. There is still one thread. If two tasks both need to hash a password, they do it one after the other. async moved the waiting out of the way; it did not add a core. CPU-bound work needs processes, threads, or a native extension that releases the GIL — not await.

2. One blocking call freezes everyone. This is the failure mode that actually reaches production. A single synchronous call inside an async function — requests.get, time.sleep, a driver that is secretly synchronous, a 40 ms JSON parse — does not just stall that task. It holds the only thread, so every other task on the loop stops too, including ones whose data arrived ten milliseconds ago.

async def handler():
    row = await db.query(1)      # fine: yields
    blob = requests.get(url)     # 🔴 blocks the ENTIRE loop
    return blob

The fix is to push it off the loop rather than pretend it is async:

blob = await asyncio.to_thread(requests.get, url)

3. Nothing is pre-empted. The OS can interrupt a thread anywhere. The event loop cannot. Scheduling here is cooperative: a task keeps the thread until it chooses to yield, and it only ever chooses at an await. A task that never awaits never lets anyone else run — a while True: with no suspension point inside it is not a busy task, it is a stopped server.

The mental test for any async code you write is one question: between here and the next await, how long is this thread mine?

The same mechanism, one floor up: serving LLMs

This is not a backend-trivia pattern. It is the reason a single modest Python process can sit in front of a GPU cluster and hold thousands of open chat connections — and it is also the reason people misdiagnose their inference latency.

An LLM request is almost entirely waiting. A completion that streams for eight seconds involves a few milliseconds of your code and eight seconds of the model producing tokens somewhere else. Thread-per-request would mean one megabyte-class stack per open conversation, parked, doing nothing. The gateway in front of a model is the purest possible async workload: thousands of bookmarks, one loop, near-zero CPU.

Streaming makes the suspension point visible. When you iterate a token stream, every async for is an await — the loop hands the thread away between tokens, so one process can fan hundreds of concurrent streams out to hundreds of clients:

async def stream(prompt):
    async for token in client.completions(prompt, stream=True):
        yield token            # suspends between tokens; loop stays free

But catch #1 sets a hard ceiling on what async can fix. Async gives you concurrency on waiting, and the two numbers everyone tunes — TTFT (time to first token) and TPOT (time per output token) — are not waiting. They are GPU work. No amount of await shortens them. What actually raises throughput is continuous batching: the server merges newly arrived requests into the running batch at every decoding iteration, rather than making them queue behind a finished batch. Note the symmetry — that scheduler is doing the same thing the event loop does, at the level of a token instead of a syscall: keep a queue of half-finished work, advance each one step, never hold an expensive resource open for something that is merely waiting.

And catch #2 is where real inference gateways bleed. Tokenising a 100 KB prompt, a synchronous embedding call, a hefty json.loads of a tool-call payload — each is pure CPU on the loop thread, and each stalls every in-flight stream, not just its own. The symptom is maddening: the model is fast, the GPU is idle, and p99 latency is terrible. The cause is a blocking call inside an async handler.

The verdict

You never needed a hundred threads. You needed a hundred bookmarks and one thread to turn the pages.

A thread is a stack, and a stack's whole job is to remember your locals and your line.
await is where the runtime takes that memory and writes it on a slip of paper instead.
One loop runs the slips, each up to its next suspension point, and resumes each exactly where it stopped.
It buys you waiting, not computing — and it holds only as long as every task keeps yielding.

Reach for it when your service is I/O-bound: an API gateway, a scraper, a websocket fan-out, a model proxy. Reach for processes when the work is genuinely CPU-bound. And whichever you choose, the diagnostic question stays the same: who is holding this thread, and when do they give it back?

References and further reading

On what a thread actually costs: the stack

M. Kerrisk, The Linux Programming Interface (No Starch Press, 2010) — chapters 29–33 on POSIX threads: what a thread owns privately (its stack) versus what it shares, and why the per-thread stack is the dominant per-thread cost.
pthread_create(3) — Linux man-pages — states that a thread's stack size defaults to the RLIMIT_STACK soft limit (commonly 8 MB on Linux) and that the stack is allocated per thread: man7.org — pthread_create(3)
A. Silberschatz, P. Galvin, G. Gagne, Operating System Concepts (Wiley, 10th ed., 2018) — chapter 4 on threads and chapter 5 on CPU scheduling: the context switch, and the pre-emptive model that the event loop deliberately does not use.

On await as a suspension point, and where the state lives

Y. Selivanov, PEP 492 — Coroutines with async and await syntax (Python, 2015) — the language-level definition: await suspends execution of the coroutine and yields control, and calling a coroutine function returns a coroutine object rather than running it: peps.python.org — PEP 492
G. van Rossum, PEP 3156 — Asynchronous IO Support Rebooted: the "asyncio" Module (Python, 2012) — the design of the event loop itself: the ready queue, callbacks on I/O completion, and the single-threaded execution model: peps.python.org — PEP 3156

On the loop and non-blocking I/O

M. Kerrisk, The Linux Programming Interface (No Starch Press, 2010) — chapter 63 on alternative I/O models: select, poll and epoll, the syscall the loop actually blocks in while every task is suspended.
The Node.js Event Loop — Node.js docs — the same single-threaded loop described phase by phase, for the runtime where this model is the default rather than a choice: nodejs.org — event loop

On the three catches: blocking, cooperation, and what concurrency is not

Developing with asyncio — Python docs — the explicit warning that blocking CPU-bound or I/O-bound calls must never be called directly from a coroutine because they block the whole event loop, plus to_thread / run_in_executor as the escape hatch: docs.python.org — developing with asyncio
Don't Block the Event Loop — Node.js docs — the same hazard stated as an operational rule, with the latency consequences for every other connection: nodejs.org — don't block the event loop
R. Pike, Concurrency Is Not Parallelism (Heroku Waza talk, 2012) — the distinction catch #1 rests on: dealing with many things at once is a structural property, and it is not the same as doing many things at once: go.dev — talk

On the LLM-serving section: why async cannot shorten TTFT

G. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI, 2022) — introduces iteration-level scheduling, where the server revisits its batch at every decoding step instead of waiting for the slowest request to finish: usenix.org — Orca
W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP, 2023) — the vLLM paper: continuous batching plus paged KV-cache memory, and the throughput ceiling that request-level concurrency alone cannot lift: arxiv.org — 2309.06180

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the reel: async/await: how ONE thread serves a hundred waiting tasks

Design a notification system — a full mock system design interview (Ep 4)

Vahid Aghajani — Mon, 14 Sep 2026 13:56:28 +0000

Originally published on software-engineer-blog.com.

You walk into a system design interview and hear "Design a notification system." Your instinct is to draw a box labeled "Notification API," connect it to a push provider and an email provider, and call it done. That instinct will cost you the round.

The trap is that "notifications" is not one system—it is two systems running at completely different speeds, and you cannot see that difference until you ask before you draw.

Mental model: A notification system is not a delivery system; it is a speed-matcher. Services produce notifications at one rate (unpredictable spikes), providers consume them at wildly different rates (20,000/s push, 200/s SMS), and your job is to decouple those two curves so that a ten-million-message campaign does not leave a password reset waiting fourteen hours.

Four questions before the first box

A real candidate does not guess. They ask:

What notification types do we send? Push, email, SMS, in-app, Slack—each has its own delivery latency and throughput. Push is fast and fire-and-forget. SMS is slow and expensive. Email is somewhere in the middle. You cannot design one queue if the drain rates differ by a factor of a hundred.
What is the expected volume? The answer tells you whether to build or to buy. A hundred million notifications a day sounds large until you do the arithmetic: that is roughly 1,160 per second on average. But the average is useless. Peak hour for a social network might be ten times the average. Peak for a campaign is all ten million notifications arriving within minutes.
What is the latency requirement? A password reset should arrive in seconds. A marketing email can arrive in hours. A campaign notification can arrive whenever the queue drains it. Latency is not a number; it is a contract that shapes whether you use a queue at all.
What happens if delivery fails? Do we retry? How many times? Do we log it and move on? Do we notify the service that sent the notification? This answer determines whether your system is fire-and-forget or transactional, and it changes whether you need a database row per notification.

The number that decides the design

Once you have answers, you size. Assume:

100 million notifications per day
50% push, 30% email, 20% SMS (rough split)
24-hour delivery window

That gives you:

Average per second:
  Push:  100M * 0.5 / 86400 = 578/s
  Email: 100M * 0.3 / 86400 = 347/s
  SMS:   100M * 0.2 / 86400 = 231/s

Peak hour (assume 10x average):
  Push:  5,780/s
  Email: 3,470/s
  SMS:   2,310/s

Provider throughput (from SLAs, not your code):
  Push:  ~20,000/s (Apple, Google, Firebase handle this)
  Email: ~5,000/s (SendGrid, AWS SES have rate limits)
  SMS:   ~200/s (Twilio, carrier agreements are slow)

The SMS number is the bottleneck. It is not because your code is slow; it is because the provider is slow. Your system must accept notifications in milliseconds (202 Accepted, queue them, and drain to SMS at 200 per second without blocking services or letting a ten-million-message campaign starve a password reset.

Why one queue does not work

You cannot put all three notification types into a single FIFO queue. Here is why:

Imagine a campaign sends 10 million SMS notifications. They arrive and sit in one queue. A user requests a password reset and gets a notification pushed behind those 10 million campaign messages. At 200 SMS per second, the password reset waits:

10,000,000 / 200 = 50,000 seconds = 13.8 hours

The user forgets what they clicked for.

The fix: separate queues per notification type, each with its own sender that respects the provider's throughput.

Services → Notify API (202 Accepted)
           ↓
        [Notify Store]
           ↓
        ┌──────────────────┐
        │ Push Queue       │ → Push Sender → Push Provider (20,000/s)
        │ Email Queue      │ → Email Sender → Email Provider (5,000/s)
        │ SMS Queue        │ → SMS Sender → SMS Provider (200/s)
        └──────────────────┘

Now each notification type drains at the speed of its provider, and a password reset SMS no longer waits behind a campaign.

The API contract

Your /notify endpoint does not send. It accepts and queues:

POST /notify
Content-Type: application/json

{
  "user_id": "12345",
  "notification_type": "sms",
  "provider": "twilio",
  "template": "password_reset",
  "params": {
    "reset_link": "https://..."
  },
  "retry_policy": "exponential",
  "max_retries": 3
}

→ 202 Accepted
  Location: /notify/event-id-xyz

{
  "event_id": "event-id-xyz",
  "status": "queued",
  "created_at": "2025-01-15T10:23:45Z"
}

The service gets a 202 in milliseconds. It is decoupled. It does not care whether SMS is slow or push is fast. It submitted the notification and moved on.

Why you cannot send it inside the request

The interviewer will push back: "A password reset is one API call. Why not just send it inside the POST /login request? It is simpler."

This is the moment that separates a rehearsed answer from a real one. Do not say "because decoupling is good" or "because queues are scalable." Say this:

On latency: If you send SMS inline, the login request blocks until Twilio responds. Twilio is a network hop to their servers, possibly in another region, and SMS is slow—they might batch for throughput. The user waits 2–5 seconds for a success response. The password reset request blocking on SMS delivery is a blast radius violation: a third-party service outage brings down login.

On reliability: If the SMS provider times out, the request times out. The client retries. Twilio may have received the message even though the response did not come back (the connection dropped). You now send the same password reset twice. The user gets two SMS.

Queuing solves both: send a 202 in 5 milliseconds, store the request durably, and retry independently. The provider timeout is a sending problem, not a login problem.

The retention and idempotency problem

Once you queue, you need to store the notification somewhere. A row per notification:

CREATE TABLE notifications (
  id UUID PRIMARY KEY,
  user_id BIGINT NOT NULL,
  notification_type VARCHAR(50),
  provider VARCHAR(50),
  template_id VARCHAR(100),
  params JSONB,
  status VARCHAR(20),
  created_at TIMESTAMP,
  sent_at TIMESTAMP,
  retry_count INT DEFAULT 0,
  last_error TEXT,
  provider_message_id VARCHAR(200),
  expires_at TIMESTAMP,
  INDEX (user_id, created_at),
  INDEX (status, created_at),
  INDEX (provider, status)
);

Each row is roughly 200–400 bytes. A hundred million notifications a day is about 20–40 GB. Retention is not free, but it buys you:

Deduplication: If the sender crashes and restarts, it re-reads the queue from the database. It checks provider_message_id. If the SMS was already sent, it updates the row and moves on. Without this, a crash turns into duplicate notifications.
Audit trail: A customer can ask "did you already send my password reset?" You check the database. You answer yes, the sent_at was at 10:23 UTC, and the SMS went through to Twilio (we have the Twilio message ID).
Preferences and do-not-contact: You can query the user's notification preferences at send time. If the user unsubscribed from marketing emails between the time the notification was queued and the time you try to send it, you do not send it.

The provider timeout trap

Here is where candidates break: the SMS provider sends the message, the connection times out, and your code sees an error.

sender.py:

response = twilio.send_sms(to, message)
# Twilio sends the SMS to the carrier
# Connection drops
# Your code sees: TimeoutError
# Your code retries
# Twilio receives the request again
# Carrier receives the SMS twice

The fix is idempotency:

request_id = notification["id"]  # UUID, generated once at /notify
response = twilio.send_sms(
    to=to,
    message=message,
    idempotency_key=request_id  # Twilio deduplicates on this
)

if response.status == "success" or response.status == "already_sent":
    notification["status"] = "sent"
    notification["provider_message_id"] = response.id
    db.update(notification)
else:
    notification["retry_count"] += 1
    if notification["retry_count"] >= max_retries:
        notification["status"] = "failed"
        log(notification)
    else:
        queue.requeue(notification, delay=backoff(retry_count))

The idempotency key is the notification UUID, generated once at /notify. Twilio stores it, deduplicates on retry, and returns "already sent" instead of sending twice.

Putting it together: the data flow

With all pieces:

Service calls POST /notify with a notification request.
API generates a UUID, writes the row to the notifications table with status queued, and returns 202 Accepted in <5ms.
An async worker reads the queued rows, groups by notification type, and pushes each batch into its type-specific queue (Redis streams, Kafka, SQS, or a database-backed queue).
Type-specific sender workers poll their queues, check user preferences, and call the provider API with an idempotency key.
Provider responds with success or error.
Sender updates the row: status becomes sent (or failed after max retries).
If a timeout occurs, retry logic uses exponential backoff and the same idempotency key; the provider deduplicates.
A cleanup job deletes notifications older than the retention window (e.g., 30 days).

Comparison: when to queue vs. when to send inline

Scenario	Send Inline (Don't Queue)	Queue (Decouple)
Latency SLA	User must wait for delivery (rare; login, payment confirmation)	Delivery is best-effort or has a loose SLA (marketing, campaign, most push)
Volume per second	< 100 per second; provider handles sync calls	> 100 per second; one provider slower than the others
Failure mode	Client retries the whole request; acceptable	Client gets 202; retry is independent; provider timeout is not a blast radius
Idempotency	Client-side retry is acceptable; user may get duplicates	Server-side idempotency key prevents duplicates on timeout
User expectation	Synchronous: "did it work?" in the response	Asynchronous: "submitted" in the response; delivery status checked separately

LLM inference angle: batch vs. stream

If you are serving notifications to a large language model for personalization (e.g., generating a subject line or body from a template), the same throughput mismatch applies:

Your notification queue drains to an LLM API at 5,000–10,000 tokens per second (your rate limit or batch size).
The LLM API handles 50,000+ tokens per second (their scale).
You are not bottlenecked by the model; you are bottlenecked by your queue or your network.
A ten-million-message campaign with personalization will fan out to N parallel LLM requests to avoid the batching latency.
Caching the LLM output per template + user segment reduces the number of API calls; one password reset should not regenerate a subject line if a thousand identical resets came before it.

The architecture does not change: separate queues, separate senders, idempotency keys, and timeout handling. But the bottleneck moves from the SMS provider to your LLM API, and caching becomes a first-class concern.

Verdict

Reach for queuing and separate per-type senders when notification types have throughput that differs by more than 10× or when services produce notifications in spikes that exceed any single provider's SLA. Reach for inline sending when latency is critical (the client must see success in the response), volume is predictable and low (<100/s), and the provider is fast. In practice, almost all real systems queue; the inline case is rare.

Watch the 90-second reel for the full walkthrough: https://youtube.com/shorts/--GXhdPi1zA

Move the data to the compute, or the compute to the data?

Vahid Aghajani — Tue, 08 Sep 2026 08:43:40 +0000

Originally published on software-engineer-blog.com.

Most cost work arrives too late. A bill lands on someone's desk three months after launch, and the hunt begins: which server can we shrink? But the real number was locked in during a design review, in one sentence about where a computation happens.

Mental model: Your bill is not a list of servers. It is a list of decisions about where bytes move.

The concrete job

You have an orders table. Twelve million rows. One question: what was total revenue this month?

That single query lives at the exact fork where system design breaks into two paths. You must choose. And the choice you make determines whether you move 2.1 GB across a network boundary or 8 bytes.

Side A: pull the data back

Fetch the rows. Add them up in your code.

rows = db.query("SELECT amount FROM orders")
total = sum(row.amount for row in rows)

This is the version most teams write first, and it deserves real defence.

It is simple. The logic lives in a language you control and test every day. You can unit-test the aggregation function in isolation. You can reason about it. No query planner surprises. No hidden index choice breaking your assumptions next Thursday.

It is portable. Swap the database, swap the schema, swap the cloud. The Python function stays the same. You own the contract between your code and the data.

It is easy to test. Mock the result. Inject a list of rows. Verify the sum. Done.

But here is the cost: on every run, this code moves approximately 2.1 GB across a boundary. The boundary is where the meter sits. Your cloud provider counts bytes crossing that line. Your application needs heap space proportional to the data size—for 12 million rows, that is memory you pay for. Wall-clock time is dominated by transfer, not arithmetic. The addition itself takes microseconds. You are paying for 2.1 GB of I/O to do 2.1 GB ÷ 8 bytes = microseconds of work.

Side B: push the compute down

SELECT sum(amount) FROM orders

The computation runs where the data already lives. No row transfer. No memory bloat in your service. One number comes back.

2.1 GB on the wire becomes 8 bytes.

Identical answer. 288 runs a day. Every run pays for 8 bytes instead of 2.1 GB. The arithmetic happens in the database engine, which is already licensed for that work, already running, already paid for.

The trade-off table

Criterion	Pull the data back (A)	Push the compute down (B)
Network transfer per run	~2.1 GB	~8 bytes
Application memory	Proportional to data	Constant (one result)
Code testability	High—logic in app code	Lower—logic in SQL/query planner
Portability	High—language agnostic	Coupled to SQL dialect
Scaling bottleneck	Network, app memory	Database compute and I/O

This is not a SQL trick

This decision shows up everywhere, wearing different clothes.

A view versus a loop in Python
An aggregate in the database versus a scan in the application
A join in the database versus a join in your service layer
Compute co-located with storage versus compute pulling data remotely

Every time, the trade-off is identical: simplicity and testability and portability against transfer cost and coupling.

The honest bill for side B

Pushing computation down is not free. It costs you three things:

One: coupling. You are now coupled to that engine. SQL Server has different functions than PostgreSQL. Snowflake has different semantics than DuckDB. Your query must live somewhere, version-controlled and tested, but not in your application code. If you change databases, the query changes.

Two: logic out of code you can test. When the sum is in your Python function, you own the test. When it is in a SQL query, a planner owns the execution. You cannot easily mock a query planner's choice. You cannot unit-test a query in the same way you unit-test a function.

Three: load on the box hardest to scale sideways. Databases scale vertically far more easily than horizontally. A service scales to 100 instances. A database scales to 3 replicas, maybe. If you push computation down, you are adding load to the component that is hardest to scale. If you pull computation back, you scale the service layer, which is easy—add more instances.

Taken too far, this is how a team ends up with business rules buried in a 400-line query nobody dares change, because nobody can test it in isolation, and the query planner's choices have become a black box.

Side A is not wrong

Side A is a bet. It is a bet that the data is small.

Usually, that is a good bet. When the orders table has 50,000 rows instead of 12 million, pulling the data back makes sense. The network transfer is 87 MB. The memory footprint fits comfortably. You own the logic. You control the test.

The failure is not choosing side A.

The failure is never re-checking the bet after the data grew.

You write side A when you launch. You ship it. Months pass. The dataset grows. Orders accumulate. Suddenly, 50,000 rows becomes 12 million. That query now moves 2.1 GB. Your application crashes because it ran out of memory. Your bill explodes.

And nobody remembers why side A was chosen. The reasoning lived in someone's head, and that person is on a different team now.

One step further: caches and CDNs

A cache and a CDN are this same idea pushed one level further.

If moving data to compute is one step, then moving compute to data is the step after. But there is a step before that: stop moving the bytes at all.

A cache stores the result of a computation locally, near the consumer. When the next request comes in, the bytes are already there. No network. No wait. The cheapest byte is the one you never sent.

This is why CDNs work. This is why Redis clusters live inside your application network. This is why database read replicas live on the same LAN as your services.

The principle holds at every scale: bytes crossing a boundary cost money. Bytes that never cross a boundary cost nothing.

For LLM inference: the same decision under latency

If you are serving LLM inference, data movement is also latency.

Side A—pulling data back to your inference service—means serializing embeddings or tokens, sending them over the network, loading them into GPU memory, computing a result. Time-to-first-token (TTFT) is high. Your inference service is I/O-bound, not compute-bound.

Side B—pushing the model to the data—means running inference where the embeddings already live (in a vector database, in a cache). TTFT is lower. Time-per-output-token (TPOT) is more predictable.

The trade-off is identical: which component carries the load, and is it the one hardest to scale? The principle survives the domain shift.

One decision, different clothes

This is why your bill is not a list of servers.

It is a list of decisions about where bytes move. Every line item on your invoice traces back to a moment in a design review when someone wrote:

"Fetch the rows, sum them in the app"
"Run the aggregate in the database"
"Cache the result locally"
"Replicate the data to the edge"

Each sentence moved the meter. The server choices that follow are just plumbing. A bigger instance or a smaller one is not the decision. The decision was made when you chose where the computation happens.

Verdict

Reach for side A (pull the data back) when you are confident the dataset will stay small and you need the logic in application code you can test. Reach for side B (push the compute down) when the dataset is large, the query is simple, and the database is your scaling bottleneck anyway. Revisit the bet every time the data grows.

Watch the 90-second reel for the full visual breakdown.

Why your index is ignored: expression indexes explained

Vahid Aghajani — Mon, 07 Sep 2026 15:38:28 +0000

Originally published on software-engineer-blog.com.

You have an index on email. You write what looks like a perfectly ordinary case-insensitive lookup:

SELECT * FROM users
WHERE LOWER(email) = 'bob@site.com';

And it takes seconds. EXPLAIN says Seq Scan on users. The index is right there, and the database is refusing to use it.

This is not a planner bug, and the index is not "disabled". It is a value mismatch, and once you see it you cannot unsee it.

Mental model: A B-tree index is a sorted copy of raw column values; applying a function to the column creates a different value that the index does not contain, so the planner must scan every row instead.

What a B-tree index actually stores

A B-tree index on email is a sorted tree whose leaves hold the value exactly as you stored it:

Alice@Example.com
BOB@site.com
carol@Site.com

When your query asks for LOWER(email) = 'bob@site.com', the planner checks whether it can navigate the tree using that predicate. But the tree is keyed on the raw email values. The lowercase version—bob@site.com—is nowhere in the leaves. So the planner cannot use the index. It does the only thing left: read every row from disk, compute LOWER(email) for each one, and compare.

On a table with five million rows, that is five million function calls per query. On a write-heavy table, you pay this cost every read.

The fix: expression indexes

An expression index is not a new index type. It is the same B-tree, but keyed on the result of a function instead of a raw column:

CREATE INDEX idx_lower_email ON users (LOWER(email));

Now the leaves hold the pre-computed lowercase values:

alice@example.com
bob@site.com
carol@site.com

When your query asks for LOWER(email) = 'bob@site.com', the planner finds that exact value in the tree. One B-tree lookup. No sequential scan.

The expression is computed once, when the row is written—on insert or update. The cost is paid upfront, at write time, not amortized across millions of reads.

The one rule that decides if it works

The expression in the index must match the expression in the query exactly.

If you have:

CREATE INDEX idx_lower_email ON users (LOWER(email));

Then WHERE LOWER(email) = 'bob@site.com' uses it. But WHERE UPPER(email) = 'BOB@SITE.COM' does not—different function, different tree, back to a scan.

The same principle applies to any expression:

(data->>'email') — a JSONB key extraction
date_trunc('day', created_at) — a timestamp truncation
lower(trim(email)) — a chained function
extract(year from order_date) — date arithmetic

If your WHERE clause computes it, you can index it. If the index expression does not match the query expression, the planner will not use it.

The trade-off: work moves, not vanishes

Aspect	Regular Index	Expression Index
Read cost	Sequential scan + function on every row	One B-tree lookup
Write cost	Store raw value	Compute function, store result, maintain index
Storage	Column value only	Expression result + index overhead
When to use	Occasional lookups, write-heavy workload	Frequent lookups on the same expression, read-heavy workload

The work does not disappear. It moves from read time to write time. Every INSERT or UPDATE now runs the expression before writing to disk. In exchange, reads that match that expression become instant.

This trade-off is worth it when:

You have a read-heavy table.
You run the same expression in the WHERE clause frequently.
The cost of computing the expression is non-trivial (like a function call or JSON extraction).

It is not worth it when:

The table is write-heavy and rarely queried with that expression.
The expression is cheap (like a simple arithmetic operation).
The table is already under write pressure.

For LLM inference workloads

If you are serving embedding lookups or vector searches via a relational database (common in RAG and agent frameworks), expression indexes become critical. You often normalize text before embedding lookup:

WHERE LOWER(TRIM(query_text)) = model_input

Without an expression index, every inference request triggers a full table scan. With one, you move the normalization cost to index write time—a one-time hit per new document ingested, then negligible read latency per inference. On high-throughput inference services, this can be the difference between TTFT under 100ms and over 500ms.

Verdict

Reach for a regular index when lookups are rare or writes are frequent; reach for an expression index when you query the same computed value repeatedly and read traffic dominates. The key constraint: the index expression and the query WHERE clause must match exactly, character for character.

Watch the 90-second reel on Instagram or YouTube for the visual walkthrough: index leaves, the value mismatch, the sequential scan, and the expression index fix drawn in real time.

Design a Rate Limiter: The Four-Point Answer, and the Trap

Vahid Aghajani — Tue, 01 Sep 2026 07:53:27 +0000

Originally published on software-engineer-blog.com.

"Design a rate limiter."

It is one of the most common system design interview questions, and it has an unusually reliable failure mode. The candidate hears it as an algorithms question, names one — token bucket, leaky bucket, fixed window, sliding window log — sketches the mechanism, and stops.

Everything they said was correct. It also answered the easy half.

Naming the algorithm tells the interviewer you have read the same article everyone else read. What they are actually listening for starts one question later: where does the count live when you are running more than one server? The algorithm is a function. The counter is state, and state is where system design lives.

Here is the four-point answer.

1. Count against the API key or the user id, not the IP

Before any mechanism, decide what you are counting against. This is a choice with a wrong answer, not a definition.

The tempting answer is the IP address, because it is always there and it needs no authentication. The problem is that an IP address is not a person. One IP is routinely one office, one university campus, one coffee shop, one mobile carrier's NAT pool — hundreds or thousands of unrelated people sharing a single address. Rate limit on it and every one of them shares a single budget. The first heavy user of the morning exhausts the quota for the entire building.

Meanwhile, the abuser you built this for is the one party in the story for whom addresses are cheap and disposable. They rotate through a proxy pool and never touch your ceiling.

So you punish the innocent and miss the abuser — a limiter that is somehow both too strict and too loose, which is a strange thing to ship.

	IP address	API key / user id
What it identifies	A network path, at this moment	An account — the thing you actually meter
Who shares one	An office, a campus, a carrier NAT pool — thousands of strangers	One customer, by construction
Cost to rotate	Cents, via any proxy pool	A new signup, and you can see it happen
Available before auth	Yes	No
Stable across networks	No — wifi to cellular changes it	Yes
Failure mode	Throttles the innocent, misses the abuser	Needs an identity to exist first

Count against the identity: the API key, the user id, the tenant. That is stable across networks, it is the thing your product actually meters, and it is the thing a customer would recognise on an invoice.

There is one honest exception. Anonymous, unauthenticated traffic — a public signup endpoint, a login form, an unauthenticated search — has no identity to count against yet. There, the IP is the best signal available and an IP limit is the right call. Say that out loud in the interview; knowing when the weaker key is the correct key is worth more than rejecting it outright.

2. Enforce it at the gateway, not inside every service

Put the limiter at the front door — the API gateway, the edge proxy, the ingress — and not inside each service behind it.

Two reasons, and they are different in kind.

The operational one: one front door is one place where limits are configured, one place where they are changed, one place to look when a customer says they are being throttled. Implement the check in each service and you have N implementations of the same rule, which will drift, and a limit change becomes N deploys.

The economic one is stronger. A request rejected at the door never gets past the door. It does not open a database connection, does not enqueue a job, does not hit your billing code, does not consume a worker slot. That is the entire point of rate limiting under load: the traffic you refuse must be cheap to refuse. A limiter that sits deep in the stack — say, as a decorator on a service method — has already paid for connection setup, deserialisation and authentication before it decides the answer is no. Under the exact conditions you built it for, it costs you the most.

There is a second-order version of the same mistake: a request that fails the check in service C has already done real work in A and B.

3. The counter is shared state

This is the point most answers miss, and it is where the question is actually decided.

Say you keep the count in a dictionary in the API process. It is fast, it needs no extra infrastructure, and it works perfectly in development, where you run one process.

Then you run three.

You wrote "100 requests per minute" in the spec. What you have built is 100 requests per minute per instance. Three instances is 300. Ten is 1000. And the number is not even fixed — a load balancer spreading a user's requests across the fleet means each instance sees roughly a third of that user's traffic and never trips, so in practice the ceiling is higher still and depends on how the balancer happens to route.

Then the autoscaler joins in. Now the effective limit moves with your traffic: busy hour, more instances, higher real limit — precisely the opposite of what a limiter is for. The system is most permissive at the moment it should be strictest, and nothing in your monitoring will say so, because every individual instance is dutifully enforcing 100.

The fix is not a cleverer algorithm. It is one store, one atomic increment.

One store: a single place — Redis is the usual choice — that every instance talks to, so there is exactly one number per key per window.

One atomic increment: not read, then decide, then write. Those are three steps, and two requests arriving together can both read 99, both decide they are allowed, and both write 100. The check and the claim have to be a single indivisible operation. Whether that is INCR with an expiry, a conditional decrement, or a small server-side script is an implementation detail; the invariant is what earns the point.

That store is now on the hot path of every single request, so say the consequences out loud too: it needs to be fast, it needs to be close, and you need an answer for what happens when it is unreachable. Fail open and you have no limiter during exactly the incident that produced the outage. Fail closed and a limiter outage is a full outage. Choosing deliberately, and being able to say which way you chose and why, is the senior answer.

4. Answer with a 429, a Retry-After, and the quota left

The last point is the contract, and it is the one candidates skip because it feels like trivia.

Reject with 429 Too Many Requests. Include Retry-After with the number of seconds until the window resets. Include the remaining quota — X-RateLimit-Remaining, or whatever your API already uses — on the successful responses too, so a well-behaved client can see the wall coming before it hits it.

HTTP/1.1 429 Too Many Requests
Retry-After: 12
X-RateLimit-Remaining: 0

The reason this matters is not politeness. It is that the client now has a number to wait on instead of a guess.

A limiter that returns a bare 500, or a 403, or a 200 with an error body, teaches every client integrating against you to invent its own retry timing — and the timings they invent are uniformly worse than the one you already know, because you are the only party who knows when the window resets. You are handing out a fact you have and they do not. Withholding it costs you the retries.

The same question, in front of a model

If you have ever been handed an LLM API key, you have already met all four points wearing different names — and one extra twist that makes this the sharpest live example of the pattern.

Look at how a model provider states its limits. Not "100 requests per minute" but RPM and TPM — requests per minute and tokens per minute, enforced together. That second dimension breaks an assumption the classic answer quietly makes: that every request costs the same. Here they do not. One request is forty tokens; the next drags a 100,000-token document behind it.

So the count is no longer +1. It is +cost, and you do not know the cost until you have counted the prompt — and you do not know the full cost until the model has finished generating, because output tokens count too. Real limiters handle this by reserving an estimate up front and reconciling on completion: increment by prompt tokens plus a max-output allowance, then correct the counter down when the stream ends. That is still one store and one atomic operation. It is just an operation that adds a number rather than one.

The other three points transfer unchanged, and get more important rather than less:

What you count against — the tenant, not the IP, because one of your customers is a chat product with ten thousand users behind one egress address.
Where you enforce it — at the gateway in front of the model, because a rejected inference request must never reach the GPU. This is the economic argument at its most extreme: a request admitted by mistake does not cost you a database connection, it occupies a slot on a machine that costs dollars an hour, for seconds at a time.
The reply — 429 with a Retry-After. Every major provider sends one, and every serious client library reads it.

And if you are serving your own model rather than calling someone else's, the scarce resource shifts again: what actually runs out is not requests per minute but concurrent sequences in the batch and blocks in the KV cache. A limiter in front of an inference server is really admission control for GPU memory, and the honest version of it counts what the GPU is short of rather than what is easy to count.

Which is the same lesson as point 3, one level up: the interesting part was never the algorithm. It was working out what the number means and where it is kept.

The trap

Everyone answers "token bucket". The interviewer is waiting to hear where the count lives.

That is the whole question, compressed. The algorithm is a small, well-documented function that you could look up in a minute. The counter is distributed mutable state on the hot path of every request — which is a system design problem, which is why this is a system design interview.

If you have time for one more sentence after the four points, make it this one: the algorithm decides whether to allow a request, and the storage decides whether that decision is true.

The verdict — the whole answer in four lines

key → count per user or API key, not per IP; IP only for anonymous traffic
gateway → one front door, so a rejected request costs nothing
counter → one shared store, one atomic increment, because in-process counts multiply by your instance count
429 → plus Retry-After and the remaining quota, so the client waits on a number instead of a guess

Name an algorithm if you like — token bucket is a fine choice and takes five seconds to justify. Then spend the rest of your answer on the four points above, because that is the part the interviewer cannot look up on your behalf.

References and further reading

On the reply contract — point 4

IETF, RFC 6585 §4 — Additional HTTP Status Codes: 429 Too Many Requests (IETF, 2012) — the normative definition of the status code: the server indicates the user has sent too many requests in a given amount of time, and the response may include a Retry-After. Worth quoting in an interview because the spec explicitly declines to define how the server counts, which is exactly the storage decision this article argues you are being tested on: rfc-editor.org/rfc/rfc6585#section-4
IETF, RFC 9110 §10.2.3 — HTTP Semantics: the Retry-After header field (IETF, 2022) — the support for "a number to wait on, not a guess": Retry-After carries either a delay in seconds or an HTTP-date after which the client may retry, and nothing else: rfc-editor.org/rfc/rfc9110#field.retry-after
IETF, draft-ietf-httpapi-ratelimit-headers — RateLimit header fields for HTTP (IETF, in progress) — the ongoing standardisation of the "quota remaining" half of point 4, and the reason the X-RateLimit-* headers above are a de-facto convention rather than a standard. Note that it is a draft, not a published RFC: datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers

On the shared counter — point 3

Redis documentation, INCR — "Pattern: Rate limiter" — the concrete mechanism behind "one store, one atomic increment", from the store this design would actually use: a counter per key per window, incremented atomically and expired with the window. It also documents the exact race the wording above is chosen to avoid — the gap between the INCR and the EXPIRE: redis.io/docs/latest/commands/incr

On what to count and where to enforce it — points 1 and 2

Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering, chapter 21, "Handling Overload" (O'Reilly / Google, 2016) — Google's practice of metering per customer against a quota rather than per source address, and the argument that a rejected request is not free: it still costs CPU and memory, so it must be refused as early and as cheaply as possible. That is the economic case for the front door.
Paul Tarjan, Scaling your API with rate limiters (Stripe Engineering blog, 2017) — a production account that lines up with points 1, 2 and 4: limits applied per API key at the edge rather than inside application code, backed by a shared Redis counter, returning 429 with enough information for the caller to act: stripe.com/blog/rate-limiters

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the reel: Design a rate limiter — the four-point answer, and the trap

MCP vs API: Why Your Server Needs a /mcp Endpoint (and What You Actually Write)

Vahid Aghajani — Mon, 31 Aug 2026 15:25:57 +0000

Originally published on software-engineer-blog.com.

Same server. Same database. Same functions.

Your service already has an API. It has had one for years. It is documented, it is authenticated, it is in production, and it works.

So why is everyone suddenly standing up a /mcp endpoint right next to it?

Not because the old one is broken. Behind both doors is the same server, the same database, the same Python functions. Nothing underneath changes. The difference is one sentence long:

Everything else in this article is a consequence of that sentence — including the one part of the job that no library will do for you.

Why a plain endpoint is not enough

Think about how your API actually got called.

Somewhere, a human being opened your documentation. They read that the path is /v1/invoices, that it takes a customer_id and a from and a to, that the dates are ISO-8601 and that the response is paginated. Then they went back to their editor and hard-coded that knowledge into a client.

All of the understanding happened outside the wire. The endpoint itself never explained anything. Ask it what it is for and it has no answer — there is no request you can send it that means "describe yourself." It was never designed to have one, because the reader was always going to be a person with a browser.

A model does not have a browser, and it was not in the room when your docs were read. If the only thing on the wire is POST /v1/invoices, the model is guessing.

What MCP changes is the first move

Here is the actual shift, and it is smaller than it sounds:

With an API, the first thing that happens on the wire is a call.
With MCP, the first thing that happens on the wire is a question.

Before invoking anything, the client asks the server: what can you do? The server answers with a list. Only then does the model pick something from that list and call it.

Discovery first, invocation second. That ordering is the protocol's whole reason to exist.

1. client → server   tools/list      "what can you do?"
2. server → client   [ 8 tools, each with a description and a schema ]
3. client → server   tools/call      "do this one, with these arguments"
4. server → client   the result

Steps 1 and 2 are the part your REST API has never had.

The thing every REST developer notices first

Open the traffic and something looks wrong: there is no resource path.

Your API puts the noun in the URL and the verb in the HTTP method — GET /invoices/42, DELETE /invoices/42. MCP does neither. There is one path, /mcp, and every single message is a POST to it.

That is because MCP speaks JSON-RPC 2.0. The operation is not in the URL and not in the method — it is in the body, in a field called method:

POST /mcp
Authorization: Bearer <token>
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list",
  "params": {}
}

The reply is a menu — and this is the interesting part. Each entry carries a name, a plain-English description of what it is for, and a schema in which every parameter has its own description:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {
        "name": "find_invoices",
        "description": "Find a customer's invoices for a given year. Use this when someone asks about billing history, unpaid bills, or what a customer was charged.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "customer": {
              "type": "string",
              "description": "Customer name or account ID, e.g. 'ACME Ltd' or 'cus_8812'"
            },
            "year": {
              "type": "integer",
              "description": "Four-digit calendar year, e.g. 2026"
            }
          },
          "required": ["customer", "year"]
        }
      }
    ]
  }
}

That is documentation, delivered as data, over the same wire the call goes out on. Nobody had to read anything in a browser.

Calling goes through the same door. Only the method changes:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "find_invoices",
    "arguments": { "customer": "ACME Ltd", "year": 2026 }
  }
}

And it is authenticated exactly the way your API is — a bearer token on every request. This is not a new security model. It is your security model, on a new endpoint.

You never write that JSON by hand

Look at that schema again and the obvious objection is: nobody is maintaining that.

Correct. You do not write it. FastMCP derives it from code you were already writing:

from fastmcp import FastMCP
from pydantic import Field
from typing import Annotated

mcp = FastMCP("billing")

@mcp.tool
def find_invoices(
    customer: Annotated[str, Field(description="Customer name or account ID, e.g. 'ACME Ltd' or 'cus_8812'")],
    year: Annotated[int, Field(description="Four-digit calendar year, e.g. 2026")],
) -> list[dict]:
    """Find a customer's invoices for a given year.

    Use this when someone asks about billing history, unpaid bills,
    or what a customer was charged.
    """
    return billing.query_invoices(customer=customer, year=year)

Three mappings, and that is the whole trick:

What you write in Python	What the model receives
Function name `find_invoices`	The tool `name`
The docstring	The tool `description` — what it is for and when to reach for it
Type hints + `Field(description=...)`	The `inputSchema`, one described parameter at a time

Which has a consequence worth sitting with for a second.

It is no longer read by a teammate skimming the file. It is read by the thing deciding whether to call your function at all, and with what. A vague docstring is now a runtime failure mode, not a code-review nit.

The part that is not generated: the tools themselves

Here is where most first MCP servers go wrong, and it is the reason this article exists.

Because FastMCP generates the schema, it is tempting to conclude that the whole job is generated — point something at your OpenAPI spec, get an MCP server out, ship it. You will get a server. You will not get a usable one.

MCP is not a thin 1:1 layer over your existing endpoints. You write a new tool per capability, and the good ones are shaped like tasks, not like resources.

That distinction is concrete. Your REST API is decomposed the way a database is: /customers, /invoices, /line-items, each with filters, each returning a page. Answering "what did ACME pay us in 2026?" takes three calls and some glue — the human client author wrote that glue once and forgot about it.

A tool has no glue author. So the tool is the whole task:

	Auto-generated from the spec	Written by hand
Unit of design	One tool per endpoint / per resource	One tool per task a user actually asks for
Typical count	60+	8
Answering "what did ACME pay in 2026?"	3 chained calls, IDs threaded by the model	1 call: `find_invoices(customer, year)`
Descriptions	Inherited from a spec written for humans, or empty	Written for the decision the model has to make
Parameters	Every filter the endpoint supports, most of them noise	Only what the task needs
Failure mode	Model picks the wrong one of six similar tools, or gives up	Model picks correctly, because the choice is obvious

Point a generator at your OpenAPI spec and you get 60 tools nobody can use. Write 8 by hand, with real docstrings, and the model uses them correctly.

Why this bites harder than it looks (the AI-engineering angle)

If you have only built request/response systems, the cost of a bad tool surface is not obvious. It is not a latency problem. It is a context and decision problem, and it shows up in three places.

Every tool is permanently in the prompt. The tool list is not fetched when needed — it is serialised into the model's context on every single turn of the conversation. Sixty tools with full schemas is thousands of tokens spent before the user has said anything, on every message, forever. Eight task-shaped tools cost a fraction of that, and leave the budget for the actual work.

Every extra tool is another chance to choose wrong. Selection accuracy degrades as near-duplicates pile up. Six endpoints that all list something, with descriptions inherited from a spec written for a human who already knew which one they wanted, is a menu designed to be misread. This is the same reason a chained workflow is fragile: three dependent calls means three chances to thread the wrong ID, and one wrong ID is a confidently wrong answer rather than an error.

The description is the interface. In a normal API, the contract is the signature and the docs are advisory. Here the prose is the routing logic — a tool whose description does not say when to use it will be called at the wrong moment, no matter how correct its implementation is. Write descriptions that answer the model's actual question ("is this the one?"), not the ones that restate the function name.

Which gives you a useful design test. For each tool, ask: is this a thing a user would ask for in a sentence? find_invoices(customer, year) passes. list_line_items(invoice_id, page, per_page, sort) does not — it is an implementation detail that leaked onto the menu.

The verdict

Adding MCP is not rewriting your product, and it is not flipping a switch either.

Keep everything below the surface: your queries, your business logic, your auth, your database. None of it moves. The REST API stays exactly where it is, serving the clients it already serves — MCP does not replace it, it sits beside it.

What is genuinely new is a tool surface, and that surface is a design artifact you author. The protocol gives you discovery. FastMCP gives you the schema. The judgment about which capabilities exist and what shape they have is the part that is still yours, and it is the part that decides whether the thing works.

The short version, if you take one line away: the schema is generated for you; the tool design is not. Write eight tools shaped like the questions people ask, give each a docstring you would be happy to have read aloud as an instruction, and you are done.

References and further reading

On the protocol itself — discovery, and the JSON-RPC envelope

Model Context Protocol, Specification — the normative definition of tools/list and tools/call, the tool object with its description and inputSchema, and the transport and authorization sections behind the bearer-token point above: modelcontextprotocol.io/specification
JSON-RPC Working Group, JSON-RPC 2.0 Specification (2010) — the envelope MCP is built on: why the operation lives in a method field inside the body rather than in a URL or an HTTP verb, and how id, params, result and error fit together: jsonrpc.org/specification

On why a REST API looks so different — the noun in the URL

Roy T. Fielding, Architectural Styles and the Design of Network-based Software Architectures (PhD dissertation, UC Irvine, 2000), ch. 5 — the resource/identifier/uniform-interface model that puts the noun in the path and the verb in the method, which is exactly the convention MCP declines to use: ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
OpenAPI Initiative, OpenAPI Specification — what an endpoint-shaped description of a service actually contains (paths, operations, parameters), and therefore what a generator pointed at one can and cannot know about the tasks a user wants to perform: spec.openapis.org/oas/latest.html

On generating the schema from code

FastMCP documentation — the @mcp.tool decorator and the derivation rules used above: function name to tool name, docstring to description, type hints to inputSchema: gofastmcp.com
Pydantic documentation, Fields and JSON Schema — how Field(description=...) and annotated types become the per-parameter descriptions the model reads, which is where most of the usable signal in a tool schema comes from: docs.pydantic.dev/latest/concepts/fields/

On the part that is not generated — designing the tools

Anthropic, Writing effective tools for agents (Anthropic Engineering, 2025) — the case for consolidating several low-level endpoints into one task-shaped tool, for spending real effort on tool descriptions, and for evaluating a tool surface rather than assuming it: anthropic.com/engineering/writing-tools-for-agents

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the short: MCP vs API — Why Your Server Needs a /mcp Endpoint

Keep going: the rest of the free AI-engineering course lives at software-engineer-blog.com/ai.

RTO vs RPO: The Two Numbers That Decide What Disaster Recovery Costs

Vahid Aghajani — Mon, 31 Aug 2026 07:34:53 +0000

Originally published on software-engineer-blog.com.

Two letters apart. Both measured in minutes. Bought with completely different money.

That is why teams argue about disaster recovery for a week without noticing they are talking about two separate things. This article takes the two numbers apart, and then prices both of them against one small shop.

Tuesday, twelve minutes past three

The server that holds every order for an online shop is gone.

The alert fires. Somebody wakes up. By twenty past three they have found the most recent saved copy of the data. That copy was taken at three, on the hour. So they build a new server, and they put that copy onto it. At nineteen minutes past four, the shop can take orders again.

Now count what actually happened.

The shop was shut for sixty-seven minutes. And twelve minutes of orders — the ones taken between three o'clock and the moment it died — do not exist anywhere any more.

Two different things went wrong.

For the next week, the team argued about how to make sure it never happened again. And in that whole week, nobody ever said which of the two numbers they were trying to make smaller. So which one was it? The sixty-seven minutes the shop was shut, or the twelve minutes of orders that vanished?

What this article covers, and what it does not

Three things are covered:

What the two numbers actually are. One is how long you may be down. The other is how much data you may lose.
What each one is bought with, because they are not bought with the same money.
How the standard tiers are priced, and where that price stops being reasonable.

Three things are not covered. This is not how backups work on the inside — there are no restore steps here, and that is a subject for its own article. This is not about which site serves your traffic or how failover flips between two sites, which is covered elsewhere on this site. And this is not one product: no tool and no cloud provider is named anywhere below.

Two dials, not one

Here is the confusion this kills. People hear "disaster recovery" and think there is one dial, and that turning it up makes everything better.

There are two dials. A shop that catches fire is the easiest way to see them.

First question. How long is the shop shut? The fire is out, but the doors still do not open. Every hour shut is customers who go somewhere else. You make that shorter by renting a second shop, already fitted out and ready. And a second shop costs rent every single month, whether or not there is ever a fire.

Second question. How many pages of the order book burned? The shop can open again, but some orders simply do not exist. You lost every order taken since the last time somebody carried a page across the road to the safe. You make that shorter by copying more often. And copying after every single order slows down every single order you ever take.

Now notice how independent those two are. You could reopen in an hour with a whole day of orders missing. You could lose nothing at all and stay shut for a week. Both are possible, because these are two different numbers.

One line of time, two arrows

Draw the life of the failure as a line of time. Mark the moment it broke. Everything to the left already happened. Everything to the right has not happened yet.

Look backwards from the break, to the last moment your data was safely saved somewhere else. That distance is the recovery point objective, or RPO. It is the data you are willing to lose. On Tuesday it was twelve minutes.

Look forwards from the break, to the moment a customer could place an order again. That distance is the recovery time objective, or RTO. It is the time you are willing to be down. On Tuesday it was sixty-seven minutes.

Same instant. Two arrows. They point in opposite directions.

	RPO	RTO
Which way it points	backwards, into the data	forwards, into the clock
What it measures	work that vanished	time the doors were shut
What sets it	how often you copy	how fast you can be serving again
On Tuesday	12 minutes	67 minutes

Both are measured in minutes, and that is exactly why people mix them up. Every section below is this same line with the two marks moved.

RPO: the promise is the worst case

Suppose you save a copy of the data every fifteen minutes. What have you actually promised?

Here is the whole thing, in a few lines.

# rpo.py -- what does "we copy every fifteen minutes" really promise?
COPY_EVERY = 15 * 60      # seconds between one saved copy and the next
ORDERS_PER_MIN = 50       # this shop, on an ordinary day

def lost(failed_at, last_copy_at):
    return failed_at - last_copy_at    # everything in the gap is gone

# The failure does not check your schedule. It lands anywhere in the gap.
worst = lost(failed_at=899, last_copy_at=0)   # a second before the next copy
best  = lost(failed_at=1,   last_copy_at=0)   # a second after the last one

print(worst, best)                 # 899 1  -> the whole gap, or almost nothing
print(worst / 60 * ORDERS_PER_MIN) # 749.2  -> orders that never existed

Everything written between the last saved copy and the failure is gone. That is the definition, and it is the only thing that matters here.

Now, the failure does not check your schedule before it happens. It can land anywhere inside that gap. If it lands one second before the next copy was due, you lose fourteen minutes and fifty-nine seconds of work. If it lands one second after a copy, you lose one second. Both are possible on the same schedule.

And here is the trap. The number you promise the business is the worst case, never the average. A copy every fifteen minutes means fifteen minutes.

This shop takes fifty orders a minute, and an order is worth thirty-eight euros. So the same schedule, said honestly, looks like this:

How often you copy	Worst case	Orders that never existed	Value
Once a night	23 h 59 min 59 s	71,999	EUR 2,735,968
Every 15 minutes	14 min 59 s	749	EUR 28,468
As each order is taken, about 2 s behind	2 seconds	1.67	EUR 63

Look at what changed between those three rows. Not the quality of the copies. Not the storage. Not the restore. The only thing that changed is how often.

That is the only lever RPO has, and every step of it costs you something on the way in. Which is why arguing about backup software rarely moves this number at all.

RTO is a sum, and the restore is the small term

Now the other number, and it has a trap of its own.

Ask a team what their RTO is, and most of them answer with how long the restore takes. Watch what that leaves out. The clock does not start when you start working. It starts when it breaks.

# rto.py -- the clock starts when it breaks, not when you start working.
detect    =  4 * 60    # the alert fires, and a human actually sees it
decide    = 11 * 60    # somebody decides this is a disaster, not a blip
provision =  9 * 60    # new machines exist, and they are reachable
restore   = 26 * 60    # the data is put back onto them
verify    =  8 * 60    # somebody checks it is really the right data
cut_over  =  9 * 60    # traffic is pointed at the new place

RTO = detect + decide + provision + restore + verify + cut_over
print(RTO // 60)       # 67  -> the real number, in minutes

# the one term everybody budgets for, on its own:
print(restore // 60)   # 26  -> not even half of the sixty seven

Sixty-seven minutes in total. And the one term everybody budgets for, the restore, is twenty-six of them. That is 38.8 percent. Not even half.

Term	Minutes	Can money make it shorter?
Somebody actually sees the alert	4	no — people
Somebody decides this is a real disaster	11	no — people
New machines exist and are reachable	9	partly
The data is put back	26	yes
Somebody checks it is the right data	8	no — process
Traffic is pointed at the new place	9	partly
Total	67

Here is why that split matters. The twenty-six is the part you can buy your way out of, with faster disks and a bigger pipe. The other forty-one minutes are people and process. No hardware makes any of that shorter.

So a team that spends its whole recovery budget on a faster restore cannot get its RTO below forty-one minutes, no matter what else it spends. And it usually does not know that.

If you want the number below forty-one, you have to stop restoring, and start already having a second copy running.

The three tiers, and why you do not choose them

Once both numbers are written down, something useful happens. The strategy stops being a choice.

There are three tiers, and they sit at fixed points on those two numbers.

Tier	RTO	RPO	What it costs
A saved copy, and a rebuild	hours	however often you copy	storage, and almost nothing else
A second site, kept warm	minutes	however far behind the copying runs	a second bill, every month
Two live sites, every write waits for both	seconds	near zero	that second bill, plus something on every request

Cheaper and slower at the top. Faster and far more expensive at the bottom.

For this shop, at fifty orders a minute, the difference between the rows is orders that were never placed at all:

Tier	Time down	Orders never placed
A saved copy, and a rebuild	67 minutes	3,350
A second site, kept warm	5 minutes	250
Two live sites	20 seconds	17

You do not shop for these three. You write down your two numbers, and exactly one row is left.

What RTO is bought with: capacity that sits there

Moving RTO down means having capacity that is already running when the failure happens. Not capacity you can create quickly. Capacity that is already there.

A second site kept warm is machines that are running, patched, watched and paid for on every ordinary day.

Say the production environment costs four thousand two hundred euros a month. A full warm twin doubles that. Fifty thousand four hundred euros a year.

And here is the shape of that purchase. The bill arrives twelve times a year, every year. The outage it exists for might arrive once. Or never.

That is not an argument against buying it. It is an argument for knowing exactly what you are buying.

What RPO is bought with: waiting, on every write

Now the other mark. This is where it gets interesting, because RPO is not bought with money. It is bought with waiting.

Here is the same order, saved two ways.

# write.py -- the same order, saved two ways. Only the waiting differs.
RTT_MS = 12               # a round trip to the second place, far away
WRITES_PER_ORDER = 6      # one checkout touches six rows

def save_async(row):      # RPO is the lag: about two seconds of orders
    write_here(row)       # the customer waits for this, and nothing else
    send_to_other_place(row)   # this happens after the customer is gone

def save_sync(row):       # RPO is zero: nothing is ever lost
    write_here(row)
    wait_for_other_place(row)  # the customer waits for this too, every time

print(RTT_MS * WRITES_PER_ORDER)   # 72 -> milliseconds added to every order

In the first version, you write the order here, and send it to the other place afterwards. The customer waits for the first write and nothing else. RPO is whatever the sending falls behind by. Call it two seconds.

In the second version, you write it here, and then you wait for the other place to say yes before you tell the customer their order went through. Now RPO is zero. Nothing is ever lost.

But look at who is waiting. The customer is. Every time.

If the other place is far enough away that a round trip takes twelve milliseconds, and one checkout touches six rows, that is seventy-two milliseconds added to every order. Forever.

Seventy-two milliseconds does not sound like much. So multiply it out.

What	Amount
Orders a year, at 50 a minute	26,280,000
Added to each one	72 ms
Added up, per year	1,892,160 s = 525.6 hours = 21.9 days
What it protects	1.67 orders — EUR 63 — on the one bad day

Almost twenty-two whole days of human waiting, added up, every single year. To buy back sixty-three euros of orders on a day that may or may not ever come.

That is what driving RPO all the way to zero actually costs.

The step that is worth it, and the step right next to it

Now put the money side by side. This shop loses one thousand nine hundred euros for every minute it is down — fifty orders a minute, thirty-eight euros an order.

Step one. From sixty-seven minutes down to five.

That costs one more full environment: fifty thousand four hundred euros a year. It saves sixty-two minutes of downtime, which is one hundred and seventeen thousand, eight hundred euros per outage.

Break-even is 0.43 outages a year. Have one outage every two years, and it has already paid for itself. That is an easy yes.

Step two. From five minutes down to twenty seconds.

That costs another environment, and now every write waits for the far side. It saves four minutes and forty seconds, which is eight thousand, eight hundred and sixty-seven euros per outage.

Break-even is 5.7 outages a year. And you pay the five hundred and twenty-five hours of customer waiting either way.

	Step one: 67 min → 5 min	Step two: 5 min → 20 s
Saved per outage	EUR 117,800	EUR 8,867
Costs	EUR 50,400 a year	another EUR 50,400 a year
Plus	nothing on the request path	525.6 hours of waiting a year
Break-even	0.43 outages a year	5.7 outages a year
Verdict	an easy yes	almost certainly no

Same ladder. Same shop. Same arithmetic. The step that is obviously worth it and the step that is obviously not are right next to each other.

This is the shape to remember. Both marks move a long way for money you can put in a budget. It is the last small step — pushing either mark all the way onto the failure itself — where the price stops being money and starts being waiting on every request you will ever serve.

How you actually pick

You ask the business two questions, and you write the answers down before anybody says the word "backup".

How long may we be shut before it really hurts?
How much work may vanish before it really hurts?

Those two answers are your RTO and your RPO. Everything after that is arithmetic, and the arithmetic is genuinely this small.

# pick.py -- you do not pick a strategy. You pick two numbers.
# Ask the business first, and write both answers down before anything else.
rto_minutes = 5      # how long may we be shut before it really hurts?
rpo_seconds = 2      # how much work may vanish before it really hurts?

def tier(rto_minutes, rpo_seconds):
    if rto_minutes > 60 or rpo_seconds > 900:
        return "a saved copy, and a rebuild"        # cheapest. Hours.
    if rto_minutes > 1 or rpo_seconds > 0:
        return "a second site, kept warm"           # minutes
    return "two live sites, and every write waits"  # seconds, and it hurts

print(tier(rto_minutes, rpo_seconds))   # a second site, kept warm

The two numbers came first. The tier came out of them.

Three questions you can answer this afternoon

First. If the main copy of your data disappeared right now, how many minutes of work would be gone? Not on average. In the worst case. If nobody can say the number, you do not have an RPO — you have a schedule.

Second. Add up your six terms: seeing the alert, deciding it is real, getting machines, putting the data back, checking it, switching traffic over. What is your floor once you remove the restore? If that floor is already above what you promised, buying a faster restore cannot save you.

Third. Who agreed to these two numbers? If the answer is "the engineers", they are estimates. They become objectives when the business says them out loud.

The verdict

	RTO	RPO
The question	How long may we be down?	How much work may vanish?
Measured	forwards from the break	backwards from the break
Set by	a sum of six terms, only one of which is the restore	how often you copy — nothing else
Bought with	capacity running on every ordinary day	a wait added to every write, forever
You pay	whether or not anything ever breaks	on every request you will ever serve
Fails quietly when	the budget all goes to a faster restore	the schedule is quoted as an average

RTO is how long you may be down, and you buy it with machines that sit there costing money while nothing is wrong.

RPO is how much work may vanish, and you buy it with a wait added to every write, forever.

Write both numbers down before anybody says the words "backup", "standby" or "second region". After that, the tier is the only thing left to read off.

You do not pick a disaster recovery strategy. You pick two numbers, and the strategy falls out of them.

References and further reading

On the two definitions

NIST Special Publication 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems (NIST, 2010) — the formal definitions both terms come from: RTO as the maximum time a resource may be unavailable before the impact is unacceptable, and RPO as the point in time to which data can be recovered, which in practice is how much data loss the organisation can tolerate: nvlpubs.nist.gov

On writing the two numbers down with the business

Laine Campbell and Charity Majors, Database Reliability Engineering (O'Reilly, 2017), ch. 2 "Service-Level Management" and ch. 7 "Backup and Recovery" — the same two numbers, under the names availability and durability, treated as service-level indicators agreed in advance. Durability is defined there in exactly this shape: "in the event of a system failure, no more than the past two seconds of data can be lost." Chapter 7 then opens the recovery-strategy discussion by sending you back to those objectives first.
Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara and Stephen Thorne (eds.), The Site Reliability Workbook (O'Reilly, 2018), ch. 2 "Implementing SLOs" — how a reliability target is chosen with the people who own the product rather than by the team that will engineer to it, and why the rationale behind the number gets written down with it: sre.google/workbook/implementing-slos

On the restore being the small term

Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 26 "Data Integrity: What You Read Is What You Wrote" — "No one really wants to make backups; what people really want are restores", and the requirement that recovery finish "well within the uptime needs of a service", which is the RTO by another name: sre.google/sre-book/data-integrity

On what the last step of RPO costs

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), ch. 5 "Replication", section "Synchronous Versus Asynchronous Replication" — waiting for the second copy to acknowledge is what removes the loss window, and it is also what puts an unbounded wait on the write path, which is why almost all replication is asynchronous in practice.

On the last rung costing more than the outage

Betsy Beyer et al. (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 3 "Embracing Risk" — reliability is not maximised, it is targeted: "an incremental improvement in reliability may cost 100x more than the previous increment", and the target is treated as both a minimum and a maximum: sre.google/sre-book/embracing-risk

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: RTO vs RPO on YouTube

Cache Invalidation: The Delete Ran, and the Page Is Still Wrong

Vahid Aghajani — Sun, 30 Aug 2026 14:56:48 +0000

Originally published on software-engineer-blog.com.

The ticket that makes no sense

A user changes her email address. The form says saved.

She reloads the page. The old address is still there.

You go and look, because you wrote this code and you know it is fine. The database has the new value. You check the logs: the UPDATE returned ok. The cache delete right after it returned ok too. There is no exception, no timeout, no retry, no failed job sitting in a queue. Every single call in that request succeeded.

And the cache is still holding the old email, with five minutes left on its timer.

Nothing failed. Nobody forgot anything. So what actually happened?

What this article covers, and what it does not

Three things are covered:

Why an invalidation that provably ran can leave a stale value behind — and why the delete was not too late, it was too early.
The three moves that fix it, in order, each one measured rather than asserted.
The honest limit: the class of cached answers you cannot invalidate at all, and what the workaround costs.

Five things are deliberately not covered, because each one is its own subject and gets confused with this one:

This is not the dual-write problem. That is the case where the second write never happened — the database committed and the cache call died. Here the cache call ran and returned successfully. If your failure mode is a lost second write, you want an outbox, not this article.
This is not the write path. Write-through and write-back are about whether writes go through the cache at all. Everything below assumes cache-aside: your application code reads the cache, falls back to the database, and fills the cache itself.
This is not the stampede. Invalidating a hot key sends every waiting reader to the database at the same instant. That is a real and separate cost, and it is a real and separate article.
This is not eviction. LRU and LFU decide what to throw away when memory runs out. Eviction is about space. Invalidation is about truth. They both remove keys and that is the entire similarity.
This is not HTTP caching. No browser caches, no ETag, no max-age, no CDN purge. Everything here happens on your side of the wire, inside your own service.

One system, drawn once

There are exactly three things, and everything below is a mutation of this picture.

A service. Beside it a cache — say one Redis. Under it a database — say one Postgres primary. The service talks to both. The cache and the database never talk to each other. Nothing keeps them in step except the code you write.

The read path:

# every read goes through here. ex=300 is five minutes.
def get_user(user_id):
    key = "user:" + str(user_id)
    hit = cache.get(key)
    if hit is not None:
        return hit          # a hit. No database call.
    row = db.query_one(
        "SELECT * FROM users WHERE id = %s", (user_id,))
    cache.set(key, row, ex=300)  # a miss. Fill it back.
    return row

And the write path — the one in the ticket:

# the write path. Two calls, and both of them succeed.
def change_email(user_id, new_email):
    db.execute(
        "UPDATE users SET email = %s WHERE id = %s",
        (new_email, user_id))
    cache.delete("user:" + str(user_id))
    return "saved"

# Order matters. Database first, cache second.
# Delete before the commit and a reader can refill the
# key from the old row before the new one is even there.

Look at those two functions. They are correct. That is the point of the article.

The race: the delete was too early

Put a second person in the picture. Not a second writer — a reader, doing nothing unusual.

Follow one clock:

A read arrives. Cache miss.
That reader runs SELECT and gets the old row. It is now holding that value in a local variable, on its way to cache.set.
The writer commits UPDATE. The database now holds the new value.
The writer runs DELETE user:42.
The reader — a couple of milliseconds late — finally runs cache.set(key, old_row, ex=300).

The database says new. The cache says old. And the cache has a full five-minute timer ahead of it, because step 5 set a fresh one.

Here is the part that surprises people, measured on a real Redis 7.4.9 rather than reasoned about. In step 4, the delete returns:

4. writer DEL user:42 -> DEL returned 0
   (key existed before = 0, exists after = 0)

Zero. The delete removed nothing, and it removed nothing because there was nothing there yet. The stale value was still in flight inside another thread. You cannot delete a value that has not been written.

Say that out loud, because it is the whole idea: the invalidation was not too late. It was too early.

And it is not a fluke of the cold cache. A second run was built where a different reader had already cached the old value, so the delete had a real victim and returned 1. Same ending: database new, cache old, five minutes on the clock. A delete that provably removed a real key is still undone by one slow reader.

How often, honestly

The tempting thing to do here is put a percentage on the screen. It would be wrong, because there is no single rate. The rate is a ratio:

Timing every step of 5,000 runs, that predicate predicted the actual outcome 99.8–100% of the time — the mechanism is verified, not assumed. But the rate moved enormously with database commit latency:

Reader gap (row read → cache set)	With a ~3.3 ms fsync commit	With a ~0.02 ms commit
0 ms	0 / 500 (0.00%)	1 / 500 (0.20%)
2 ms	0 / 500 (0.00%)	56 / 500 (11.20%)
5 ms	31 / 500 (6.20%)	122 / 500 (24.40%)
10 ms	148 / 500 (29.60%)	264 / 500 (52.80%)
20 ms	400 / 500 (80.00%)	416 / 500 (83.20%)

Below a 3 ms gap with slow commits it is literally 0/500, because the writer physically cannot finish before the reader has already filled the cache. Make your database faster and the window opens. This is not a bug you can benchmark your way out of.

The case that actually hurts

Under continuous writes to the same row this heals itself — the next writer's delete clears the poisoned key. Measured under eight readers and one writer hammering the same row, the cache held a wrong value at 0.43% of sampled instants, and 0.48% of hits were wrong: about 68 wrong answers in six seconds, in short bursts.

The dangerous case is the opposite one. A row nobody writes again. A profile edited once. A price changed once. A feature flag flipped once. There is no next writer to heal it, so the wrong value sits there for the entire remaining TTL, answering every read.

That is the common case, and it is the expensive one.

Move 1: delete, do not update

The first instinct after seeing this is to make the write path more authoritative: I already know the new value, so let me put it straight into the cache.

# tempting, and wrong: it writes a value you computed
def change_email_bad(user_id, new_email):
    row = db.update_and_return(user_id, new_email)
    cache.set("user:" + str(user_id), row, ex=300)

# what you actually want: it writes no value at all
def change_email_ok(user_id, new_email):
    db.execute(
        "UPDATE users SET email = %s WHERE id = %s",
        (new_email, user_id))
    cache.delete("user:" + str(user_id))

Why the first one is worse: two writers can commit in the order A then B, and have their cache writes arrive in the order B then A. The database ends holding B, which is right. The cache ends holding A — a value that nobody wrote last — and it holds it for a full TTL.

Two concurrent writers, 1,500 runs each way:

Policy	Cache order ≠ commit order	Cache ≠ database at the end
write, then `SET`	590 / 1500 (39.33%)	589 / 1500 (39.27%)
write, then `DELETE`	573 / 1500 (38.20%)	0 / 1500 (0.00%)

The reorder happens just as often either way — 39% versus 38%, that is the same physics. Under SET, essentially every reorder produced a wrong cache. Under DELETE, none did, because a delete carries no value to get out of order. It says "whatever is there is now suspect", and the next reader re-reads the truth.

Two honest qualifications.

It costs you a miss. That miss is priced:

Setup	Hit	Miss (get + select + set)	Extra cost of the miss
Local database, no network	91.9 µs	176.4 µs	+84.6 µs
With a 5 ms database round trip	77.5 µs	6.91 ms	+6.84 ms, once

Seven milliseconds, one time. Compare that against what the SET policy buys you in the next section: five figures of wrong answers.

And it does not fix everything. DELETE removes the writer-versus-writer reorder completely. It does not remove the read-path race from the section above — that one is still there, and it is what versioned keys and a sane TTL are for.

Move 2: put a version in the name

The second problem is not correctness, it is bookkeeping. One user does not have one cached key. She has a profile, her settings, five pages of feed, her friends list, her badges, her unread count, her avatar. Eleven keys, in this example.

When her row changes, which of those do you delete?

You could hunt. Measured on a 200,000-key cache:

How you find them	Wall time	Round trips	Cost
All 11 in one variadic `DEL`	0.16 ms	1	you must know all 11 names
11 named `DEL`s, one at a time	1.93 ms	11	you must know all 11 names
`KEYS user:42:*`	12.4 ms	1	blocks the whole server
`SCAN MATCH user:42:* COUNT=1000`	58.0 ms	200	safe, but 200 trips
`KEYS *`	299.7 ms	1	blocks the whole server

The reason KEYS is disqualified is not its own runtime. Redis executes commands on a single thread, so while KEYS * runs, everyone else waits. Latency of an unrelated GET from a second client, measured during it:

Server state	p50	p99	max
idle	0.079 ms	0.177 ms	0.279 ms
while `KEYS *` runs	0.721 ms	246.9 ms	516.2 ms

An unrelated GET got 1,851× worse at the tail. SCAN exists precisely so you never do that.

But look again at the top row of the hunt table. If you genuinely know all eleven names, one variadic DEL does it in 0.16 ms — faster than any alternative here. So the cost of the hunt was never round trips.

The cost of the hunt is knowing the names. They are scattered across a codebase, built by string concatenation in six different modules, and the bug is always the twelfth key somebody added last month and forgot to add to the list. The list is the bug.

So stop keeping a list. Put a version in the name instead:

# the key carries a version number for that one user
def key_for(user_id, part):
    v = cache.get("user:" + str(user_id) + ":ver") or 1
    return "user:%s:v%s:%s" % (user_id, v, part)

# invalidating every derived key for one user
def invalidate_user(user_id):
    cache.incr("user:" + str(user_id) + ":ver")

# One call. Every old key is now unreachable, because
# nothing composes those names any more. You never had
# to know how many there were, or what they were called.

One INCR, 0.701 ms, one round trip. Keys a v8 reader can still reach: 0 of 11. Not because they were deleted — because nothing constructs those names any more.

The honest cost: orphans

Nothing about this is free, and the bill is memory.

After the bump, all 11 of 11 old keys are still resident, with 3,600 seconds of TTL left on them. Nothing reclaims them early. They held 16,552 bytes for one user — about 1,505 bytes per key. Scale that: 100,000 users bumping once is roughly 1.7 GB of unreachable data sitting in your cache until it expires.

That is only survivable under one of two conditions: every versioned key has a TTL, or your maxmemory-policy is an LRU/LFU variant that will evict them. Redis defaults to noeviction, which means an orphan pile eventually stops accepting writes rather than making room.

Versioning trades memory you cannot reach for invalidation you cannot get wrong. That is usually a good trade. It is not a free one.

Move 3: the timer is a backstop, not a plan

Every cache line you write has an ex=, and it is the most misunderstood argument in the file.

# the TTL is not the plan. It is the bound.
cache.set(key, row, ex=300)

# what that one argument actually promises:
#   it does NOT make the copy correct
#   it DOES cap how long a wrong copy can live
#   so pick it from what a stale answer costs you,
#   not from how much memory you happen to have

# and this line says: be wrong forever
cache.set(key, row)

A TTL has never once made a cached value correct. All it does is put a ceiling on how long a wrong one is allowed to keep answering. So price it that way. One incident, one poisoned key, fifty reads a second:

TTL	Wrong answers served	Self-heals?
5 s	251 (measured)	yes, after 5.02 s
10 s	500 (measured)	yes, after 10.00 s
30 s	1,500 (measured)	yes, after 30.00 s
5 min	~15,000 (extrapolated)	yes
1 hour	~180,000 (extrapolated)	yes
none	unbounded	no

The measured rate came out at 50.07 wrong answers per second of TTL at fifty reads a second — linear enough that the extrapolations are safe. And the rate is what makes the number mean anything: "fifteen thousand" is only true at fifty reads a second, on a row nobody writes again.

Which reframes the question you should be asking. Not "how long can I cache this for?" but "how many wrong answers is this row worth?" A stale marketing headline: thousands, fine. A stale account balance, permission check, or price: the honest answer might be zero, and that is a signal not to cache that read at all.

The last line in that snippet — cache.set(key, row) with no expiry — is a promise to be wrong forever if anything ever goes sideways. It is worth grepping for.

The limit: you cannot invalidate what you cannot name

Everything above assumes the cached value has a name derived from the thing that changed. user:42:profile obviously belongs to user 42.

Now cache a search result. The key is a hash of the query — search:results:<sha1> — and the value happens to contain twenty user records. Or cache leaderboard:top10, computed from every score in the system.

User 42 changes her name. Which keys do you delete?

There is no answer. The key name is a function of the query, not of user 42. leaderboard:top10 does not mention a user at all. Every classic move above is unavailable, because all of them start with a name.

The only way back is to write the mapping down as you fill the cache:

# when you cache a COMPUTED answer, write down which
# rows it was built from. That list is the only way
# back from a changed row to the keys holding it.
def cache_search(query_key, rows, result):
    cache.set(query_key, result, ex=300)
    for r in rows:
        cache.sadd("tag:user:" + str(r.id), query_key)

# when a row changes, read its list and delete them
def invalidate_row(user_id):
    tag = "tag:user:" + str(user_id)
    for key in cache.smembers(tag):
        cache.delete(key)
    cache.delete(tag)

Over 5,000 cached search pages, finding the 40 keys that embed user 42:

Approach	Found	Time	Round trips
Brute force: scan, fetch and parse all 5,000 values	40	40.8 ms	10 scans + 5,000 values parsed
Tag set: `SMEMBERS tag:user:42` then delete	40	0.860 ms	3

47× faster, and the gap widens as the keyspace grows, because the tag lookup does not care how many keys exist.

And here is what it costs, which is the part usually left out:

20× write amplification. Building the index for those 5,000 pages took 100,000 SADD calls — one per embedded entity, on every single cache fill.
+130% memory. 2.58 MB of cached pages became 5.93 MB. The reverse index more than doubled the workload.
Dangling members forever. After invalidating user 42, the neighbouring tag:user:43 set still listed two keys that no longer exist. Nothing cleans them. Tag sets grow without bound unless you also keep a key → tags reverse map (doubling the write amplification again) or put a TTL on the tag sets themselves.

Tags turn an impossible problem into a solvable but expensive one, and they introduce a second garbage-collection problem of their own. That is the trade. Make it deliberately, on the handful of computed keys that actually need it — not as a default for everything.

The same problem, wearing an AI hat

If you are serving an LLM application, you already have three caches and probably have not named them as caches.

The semantic cache is the unnameable-key problem, exactly. You store an answer under an embedding of the question so that a similar question can reuse it. Now a document changes, or a price changes, or a policy changes. Which cached answers were built from that fact? The key is a vector derived from the question — it says nothing about which sources produced the answer. There is no name to delete. This is the search-result case from the section above, and the way out is the same: record which retrieved chunks fed each cached answer, tag by source document, and accept the write amplification. If you cannot afford that bookkeeping, the honest fallback is a short TTL, priced by the table above — how many wrong answers is this document worth?

The RAG index is derived data with a version. Re-embedding after a chunker change, an embedding-model swap, or a bulk document update is the version bump from move 2 — a new index name, atomically switched, rather than a hunt for which vectors are stale. The orphan bill is the same too: the old index stays resident and costs money until something deletes it.

And the write-then-delete rule survives intact. Two pipelines that re-embed the same document concurrently can have their index writes land in the opposite order to their source commits, leaving the index holding a vector nobody wrote last. Invalidating the entry and letting the next read re-embed removes that class of bug for the price of one miss — and in this setting the miss is a real embedding call, so it is worth measuring rather than assuming.

The provider-side prompt cache is the one you do not control: it has its own timer and its own key derivation, so treat it as a bound you were given, not a plan you made.

The verdict

The delete ran. That was never the problem.

Invalidation fails because of ordering, not omission: a reader that read before the write and wrote after it, holding a stale value in a variable across the exact moment the delete fired. The delete was too early, not too late.

So, in order:

Delete, never update. 589 divergences in 1,500 runs the other way, zero this way. It costs one miss — about 7 ms once, against a five-figure count of wrong answers.
Put a version in the name when a row has many derived keys. Not because it is faster — a variadic DEL beats it — but because you never needed the list of names, and the list was always the bug. Budget for the orphans, and give every versioned key a TTL.
Treat the TTL as a bound, and choose it from what a wrong answer costs rather than from how much memory you have. A cache line with no ex= is a promise to be wrong forever.

And then the limit: you cannot invalidate what you cannot name. For computed answers — search results, leaderboards, semantically cached LLM responses — either write down the mapping and pay for it, or admit you are relying on a timer and price that timer honestly.

The two-hard-things joke gets repeated because it is funny. The real reason invalidation is hard is duller and more useful: a cache is a second copy of the truth, kept in step by nothing but your own code, on a clock that two independent parties are writing to at once.

References and further reading

On the pattern itself

Microsoft, Cache-Aside pattern, Azure Architecture Center — the pattern every code sample above assumes: the application, not the cache, is responsible for loading data on a miss and for removing it when the underlying data changes, with the consistency caveats stated plainly: learn.microsoft.com/azure/architecture/patterns/cache-aside

On deleting rather than updating, and on the read-path race

Rajesh Nishtala et al., Scaling Memcache at Facebook (USENIX NSDI, 2013) — the production argument for deleting cached data instead of updating it (deletes are idempotent, so concurrent invalidations cannot reorder into a wrong value), and the "stale set" race in which a reader holding an old value writes it back after the invalidation — the exact failure this article opens on, which is why the paper introduces leases to close it: usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf

On finding the keys, and why KEYS is disqualified

Redis, SCAN command reference — the cursor-based iteration guarantees, and the explicit warning that KEYS may block the server for a long time on a large keyspace, which is what the tail-latency measurement above shows in practice: redis.io/docs/latest/commands/scan

On what a TTL actually promises

Redis, EXPIRE command reference and key expiration semantics — how expiry is stored with the key, how it is cleared by a write that replaces the value, and the lazy plus active expiration model that decides when the key really goes away: redis.io/docs/latest/commands/expire

On the orphans left behind by versioned keys

Redis, Key eviction and maxmemory-policy — the available policies and the fact that the default is noeviction, which is why unreachable versioned keys are only reclaimed by their own TTL and can otherwise fill the instance until writes start failing: redis.io/docs/latest/develop/reference/eviction

On caches as derived data — the "cannot name it" limit

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), ch. 11 "Stream Processing", sections on change data capture and derived data — a cache is a derived view of a system of record, and keeping derived views correct by ad-hoc writes scattered through application code is precisely what produces the inconsistencies above; a single ordered stream of changes is the alternative framing.

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Cache Invalidation — Explained in Detail · the short version

Continuous Integration vs Continuous Delivery: The Two Halves Almost Nobody Separates

Vahid Aghajani — Sat, 29 Aug 2026 19:56:36 +0000

Originally published on software-engineer-blog.com.

Almost everybody writes these two words as one word. CI/CD. It gets said like the name of a product you install.

It is not one thing. It is two rules. They are about two different objects, and they point in opposite directions. Most teams follow the first one and stop there, and they do not notice they stopped, because the two words have been glued together for so long.

This article takes them apart.

Merge day

Start with the thing that made you look this up.

Six developers. One codebase. Six branches open at the same time. A branch is your own private copy of the code, where you work without disturbing anyone else. Yours opened on a Monday. You are building one feature, and it touches eighteen files.

While you work, the other five keep working too. Each of them lands one change on the shared trunk — the one main copy of the code that everybody shares — every day. You do not see any of it.

After one day, five changes have landed underneath you. That is fine.

After six weeks, one hundred and fifty changes have landed underneath you. Your branch has not moved. The ground it was standing on has.

Then comes merge day. Nobody put merge day in the plan. Nobody estimated it. But it is a week of work, it happens every single time, and the team has quietly decided that this is simply what software feels like.

What this article covers, and what it does not

Three things are covered:

What continuous integration actually is, and why it is a rule about people rather than a piece of software.
What continuous delivery actually is, and why it is a rule about one object, called the artifact.
The third word, deployment, which is not the same as delivery, and which is where most of the arguments come from.

Three things are not covered. No tool is named anywhere in this article. Not one. How a release is rolled out to users is a different subject, covered elsewhere on this site. And this is not advice about how to write tests.

The workshop

The nearest confusion is this one. People think CI/CD is a single thing. A file in the repository. A list of steps that a machine runs. So they install it, and then they say they have it.

Picture a workshop instead.

On the left there is one bench, with one wooden frame being built on it. Everybody who makes a piece brings it to that bench today, and fits it into the frame today. If two pieces do not fit each other, you find that out today, while each piece is still one cut away from fitting.

On the right there is a loading dock. There is one crate. It is packed, sealed, stamped, and standing at the door. A lorry is waiting outside. Whether the crate leaves is a separate decision, made by a person.

The bench is continuous integration. The crate is continuous delivery. Two different rules, about two different objects.

One instant, two halves

Here is the whole article in one picture.

Draw the life of one change as a line of time. Mark one instant on that line. That instant is the green build: the moment your change has joined everybody else's work, the machine has built it, the tests have run, and everything passed.

Everything to the left of that instant is continuous integration. It looks backwards, at the merge. Its question is: how old is the oldest piece of work that has not joined the trunk yet? A good answer is measured in hours.

Everything to the right of that instant is continuous delivery. It looks forwards, at the artifact. Its question is: how much stands between this build and a real user? A good answer is one button.

Two halves. One instant between them. Pointing in opposite directions. Every section below is this same picture with the marks moved.

What a long branch actually costs

The cost of a long branch is not the cost most people assume. It is worth making concrete, because the arithmetic says something surprising.

Here is a small model of the team.

# drift.py -- what a branch costs is not its size. It is its age.
TEAM          = 6      # developers on this one codebase
REPO_FILES    = 1200   # files that anybody might touch
FILES_I_TOUCH = 18     # what my one branch actually edits

def drift(days_open):
    # every OTHER developer lands one change a day while my branch sits still
    return (TEAM - 1) * days_open

def chance_of_a_conflict(days_open):
    miss = (REPO_FILES - FILES_I_TOUCH) / REPO_FILES   # one edit misses me
    return 1 - miss ** (2 * drift(days_open))          # each change edits 2 files

print(drift(1),  chance_of_a_conflict(1))     # 5 changes    -> 14 percent
print(drift(30), chance_of_a_conflict(30))    # 150 changes  -> 99 percent

Drift is the number of changes that landed on the shared trunk while my branch sat still. Five other developers, one change each per day, so drift is five changes per day.

The chance of a conflict is the chance that at least one of those landed changes touched at least one of my eighteen files. Each landed change edits two files, so I take the chance of one edit missing me, and raise it to that power.

Run it. Every figure below came out of that script, and each one was cross-checked with a 200,000-run simulation that shuffles the edits at random:

How long the branch stayed open	Changes that landed underneath it	Chance of a conflict
1 working day	5	14.0% (simulated: 14.1%)
1 working week	25	53.0%
6 working weeks	150	98.9% (simulated: 98.9%)

Read the first and last rows next to each other. A branch that is one working day old has a 14% chance of hitting a conflict when it merges. Most days, nothing happens. You merge, and you carry on with your afternoon.

A branch that is six weeks old has a 98.9% chance. Not most of the time. Essentially always.

And here is the honest third number. Nothing about the work changed between those two rows. The same eighteen files. The same feature. The same developer. The only thing that changed is that the branch was allowed to live thirty times longer.

Look at the script again and notice what is missing. FILES_I_TOUCH is fixed at eighteen in both runs. The size of the branch never appears in the calculation. Only its age. A branch is a debt, and the interest on it is time.

That is what continuous integration attacks. Not your tests, and not your tooling. The age of the oldest thing that has not been joined yet.

Continuous integration, written out

So here is continuous integration in full. Notice that nothing in it is a product you can buy.

# merge.py -- continuous integration, with nothing bought and nothing installed.

def integrate(my_change):
    trunk  = fetch("trunk")              # 1. what everyone else already landed
    joined = merge(trunk, my_change)     # 2. join my work to it -- TODAY

    if not builds(joined):               # 3. does the whole thing still compile?
        return stop("the build broke")

    if not tests_pass(joined):           # 4. does it still do what it did?
        return stop("the tests broke")

    return push("trunk", joined)         # 5. now everyone builds on my work too

Step by step. Take what everyone else has already landed on the trunk. Join my work to it — not next month, today. Build it: does the whole thing still compile, with my change inside it? If it does not, stop, and say so out loud. Run the tests: does the whole thing still do what it did before? If it does not, stop, and say so out loud.

And if both of those passed, push it back to the trunk. From this moment, everybody else is building on top of my work as well.

That is all of it. Really it is two words. Every change, and today.

A machine that runs your tests but lets branches live for six weeks is not doing this. It is just running your tests.

The part that gets skipped: CI is a rule about people

Let us say this plainly, because it is the part that gets skipped.

Continuous integration is a rule about how people work. The machine is only there to enforce it.

Which means a team can have every green tick in the world and still not be doing it. If the tests run on every proposed change, but those proposals stay open for three weeks, then the joining still happens once every three weeks. The machine is measuring something. It is not fixing anything.

A green tick on a three-week-old branch proves that the branch was fine three weeks ago, against a trunk that no longer exists.

And it is cheap to install, but expensive to keep. Installing it takes an afternoon. Keeping it means the test suite has to stay fast, and it has to stay honest.

The moment people stop believing a red result, you have made things worse than having no tests at all.

The arithmetic of a suite nobody believes

That last sentence sounds like an opinion. It is arithmetic.

A flaky test is one that sometimes fails even though the code is perfectly fine. Say you have four hundred tests. Say each one, on its own, fails wrongly about two times in every thousand runs.

Two in a thousand sounds like nothing at all. But a build is only green when all four hundred pass. Multiply it out:

Wrong failures per test, per 1,000 runs	Chance a perfectly good build shows red	In other words
0.5	18.1%	1 good build in every 5.5
1	33.0%	1 good build in every 3
2	55.1%	1 good build in every 1.8

At two in a thousand, more than half of your perfectly good builds show red. So more than half the time, red means nothing.

And then the number that actually costs you: zero. That is how many people will look carefully at a red build, once they have learned that red usually means nothing. They press the button that runs it again.

Now a real failure and a fake failure look exactly the same. Your suite has stopped being a signal. It is still running, it is still costing machine time, and it is no longer telling anybody anything.

This is why "we installed it" is not the same as "we have it". Continuous integration is a habit that a machine happens to enforce, and habits decay.

Crossing the line: the artifact

Now cross to the right hand side of that instant, and follow one green build all the way out.

Here is the green build. The change is on the trunk, it compiled, the tests passed. Continuous integration has done its whole job, and it stops right here.

Continuous delivery starts here. The first thing it does is turn that build into one object. One package, with one identity, built exactly once. That object is called the artifact, and from this moment it is the only thing that moves.

The same artifact is then pushed into real environments. Not a rebuild. The same bytes, with different configuration — addresses, secrets, sizes — handed to it from outside.

And at the end there is production, with one step in front of it. In continuous delivery, that step is a button, and a person presses it.

Build once, promote the same bytes

Here is what that means in code. This is the rule that carries the whole idea.

# build.py -- the artifact is built ONCE. Only the configuration changes.

def build_artifact(commit):
    # one build, one identity. This object is the only thing that ever ships.
    return package(commit, digest=sha256(commit))

def promote(artifact, environment):
    config = load_config(environment)    # address, secret, size -- from OUTSIDE
    return deploy(artifact, config)      # the same bytes. Nothing is rebuilt.

art = build_artifact("a91f4c")
for env in ("staging", "pre-production", "production"):
    promote(art, env)                    # 1 build, 3 deploys, identical bytes

# Rebuild once per environment instead and you get 3 builds -- which is
# 2 chances for what reaches production to differ from what you tested.

Build the artifact from one commit. Package it, and give it an identity — a digest, which is a short fingerprint of the contents, so you can point at exactly this thing and no other.

Promoting is a separate step, and this is the important line. Promoting takes the artifact and an environment. It loads that environment's configuration from outside the artifact. Then it deploys the artifact, unchanged.

So one build goes to staging, then to pre-production, then to production. Three deployments, one set of bytes. The thing your users get is the exact thing your tests ran against.

Now do it the other way. Build again for each environment. Three builds, which means two chances for what reaches production to be quietly different from what you tested. Nobody chooses that on purpose. It happens because rebuilding is the easy thing to write.

What continuous delivery demands

Continuous delivery is not free, and the price is not money. It is a list of things that have to already be true.

Required:

One artifact for every green build, and nothing else ever ships.
Configuration comes from outside that artifact, always.
Database changes that work with both the old and the new version of the code, because for a few minutes both of them are running.
A rollback you have actually used, on a real day. Not a paragraph in a document.

And here is what has to go:

Servers that somebody edits by hand.
A separate build for each environment.
A long-lived release branch.
A freeze week before every release.
A page of manual steps written down in a document.

Without that list, continuous delivery is only a faster way to ship the same bug.

One boundary, so you know what this is not

There is a whole subject sitting next to this one, and it is worth naming so you can see the edge.

How a release is rolled out is that other subject, and it is covered elsewhere on this site: two full production stacks with the traffic flipped between them, a small share of users getting the new version first, and requests already in flight being allowed to finish before the old version stops.

Shipping code switched off and turning it on later is a third subject, also covered elsewhere, and it is not the same as either half here.

All of them are real, and all of them matter. This article is about something before all of them: how often work is joined together, whether every green build becomes one artifact, and how far that artifact travels on its own.

The pipeline, and the artifact. Not the rollout. This article stops at the artifact.

The third D

And now the third word, which is where almost every argument about CI/CD actually comes from.

Continuous delivery means every green build is ready to go to production. It has been built once, it has been through the environments, and it is sitting at the door. A person decides when it goes. The team can ship ten times a day if they want to, and they can also decide not to ship anything for a week. Ready is the promise. Released is a choice.

Continuous deployment removes the person. Green means it goes. Nobody presses anything.

Both of them are written CD, which is exactly why people argue past each other. One person is defending a discipline, the other is defending an automation, and they both think the letters mean the same thing.

They are also not the same difficulty at all. Delivery needs discipline. Deployment needs everything delivery needs, plus a way to notice a bad release and undo it without a human — because there is no longer a human in the loop to notice.

The difference is often one line

Inside a real pipeline, the difference between those two is often exactly one setting.

# promote.py -- continuous DELIVERY and continuous DEPLOYMENT, in one file.

REQUIRE_A_HUMAN = True      # True: delivery.   False: deployment.

def on_green_build(artifact):
    promote(artifact, "staging")
    if not smoke_tests("staging"):
        return stop("staging is unhappy -- nothing goes anywhere")

    if REQUIRE_A_HUMAN:
        return wait_for_the_button(artifact)    # DELIVERY: ready, not released
    return promote(artifact, "production")      # DEPLOYMENT: it ships itself

At the top there is one setting: require a human, true or false.

On a green build, promote the artifact to staging, and run the quick checks there. If staging is unhappy, stop, and nothing goes anywhere.

And now the branch. If require-a-human is true, you wait for the button. The artifact is finished, tested, and standing still. That is continuous delivery: ready, but not released.

If require-a-human is false, the same artifact goes to production right there, on the same green build, with nobody watching. That is continuous deployment.

Continuous delivery ends at a button that nobody is obliged to press. Continuous deployment deletes the button.

Note what did not change between the two paths: the artifact, the promotion, the checks. Everything above that setting is the same work. That is why you cannot skip delivery on the way to deployment.

Why the size of a release matters, on a bad day

There is one more reason to care, and it only shows up when something goes wrong.

Suppose your six developers finish one change each per day, and you release once a quarter. That release contains 378 changes. Then something breaks in production. You now have 378 suspects. Finding the one, by repeatedly cutting the list in half, takes about 9 rounds.

Now release once a day instead. That release contains 6 changes. Six suspects. About 3 rounds.

Release rhythm	Changes in one release	Rounds of halving to find the culprit
Once a quarter	378	~9
Once a day	6	~3
Difference	63×	3×

The same team. The same code. The same bugs. The only thing that changed is how much work was allowed to pile up between one release and the next.

This is not just arithmetic on a page, either. The multi-year Accelerate research found the same direction in real organisations: teams that release in small batches, often, also recover faster and fail less. Small batches are not a nicety on top of the pipeline. They are most of what the pipeline buys you.

The same two halves, when part of what you ship is a model

If your system includes a trained model, both halves still apply, and one of them gets harder.

The left half does not change at all. Prompts, retrieval settings, evaluation scripts and serving code are all just code on a trunk, and a branch holding them is exactly as expensive with age as any other branch.

The right half is where people quietly break the rule. The artifact stops being one object. The code is packaged once, but the model weights are fetched at start-up from wherever the latest ones happen to live, and the prompt text is edited in a console by hand. Now three things ship on three different schedules, and none of them has one identity. When output quality drops on a Tuesday, you cannot say what changed, because "what changed" was never a single thing you could point at.

The fix is the same sentence as before, applied honestly: one artifact per green build. Pin the weights by digest, put the prompt in the package, and let configuration — endpoints, sizes, keys — come from outside. Then a bad Tuesday has one identity attached to it, and rolling back means going back to one object rather than reconstructing three.

Three questions about your own team

You can answer these this afternoon.

First. How old is the oldest piece of work that has not been joined to the trunk yet? If you answer in weeks, you do not have continuous integration, no matter what runs on your proposed changes.

Second. Does every green build produce exactly one artifact, which is never rebuilt on its way out? If not, you do not have continuous delivery yet — because there is nothing being delivered.

Third. What is the last step before production? If it is a button, that is delivery. If there is no step at all, that is deployment. If it is a person following instructions written in a document, then it is neither of them.

The verdict

	Continuous integration	Continuous delivery
It is a rule about	the merge	the artifact
It looks	backwards, at joining work	forwards, at shipping it
Its question	How old is the oldest unjoined work?	How much stands between this build and a user?
A good answer	hours	one button
What it costs	cheap to install, expensive to keep honest	discipline: config outside, two-way migrations, a used rollback
How it fails quietly	green ticks on three-week-old branches	a rebuild per environment

Continuous integration is a rule about the merge. Everybody's work joins the same trunk today, and every join is built and tested, so a break is found while the change is still small enough to understand.

Continuous delivery is a rule about the artifact. Every green build becomes one package, built once, that travels the environments unchanged, until it is standing one press away from production.

And the D that everybody argues about is a third thing. Delivery means always ready. Deployment means the person is gone as well.

You can have the first without the second, and almost everybody does.

References and further reading

On branch age and merge day

Paul Hammant and contributors, Trunk Based Development (trunkbaseddevelopment.com) — the short-lived-branch rule and why branch lifetime, not branch size, is the thing to manage: trunkbaseddevelopment.com

On continuous integration as a rule about people

Kent Beck, Extreme Programming Explained: Embrace Change (Addison-Wesley, 1999; 2nd ed. 2004) — the origin of "integrate continuously" as a team practice rather than a piece of infrastructure.
Martin Fowler, Continuous Integration (martinfowler.com, 2006) — the canonical definition, including "everyone commits to mainline every day": martinfowler.com/articles/continuousIntegration.html

On a suite people stop believing

Martin Fowler, Eradicating Non-Determinism in Tests (martinfowler.com, 2011) — why a suite that fails at random destroys the trust that makes it worth running at all: martinfowler.com/articles/nonDeterminism.html

On the artifact, and build-once-promote

Jez Humble and David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley, 2010) — the deployment pipeline, building binaries exactly once, and keeping configuration outside the artifact.

On the third D: delivery versus deployment

Martin Fowler, ContinuousDelivery and Continuous Delivery vs Continuous Deployment (martinfowler.com) — the ready-versus-released distinction stated directly: martinfowler.com/bliki/ContinuousDelivery.html

On batch size and release rhythm

Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (IT Revolution, 2018) — the measured link between small batches, deployment frequency, lead time and recovery.

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Continuous Integration vs Continuous Delivery on YouTube

Service Boundaries: Where Does the Line Between Two Services Actually Go?

Vahid Aghajani — Sat, 29 Aug 2026 07:10:11 +0000

Originally published on software-engineer-blog.com.

You have decided to split. That part is not in question here — this article takes it as given and asks only the next question, the one that actually costs money: where does the line go?

The split almost everyone draws first is by noun — one service per thing the business talks about. Order, user, product, pricing. Or by the org chart — one service per team. Both are visible on day one, on a whiteboard, before anyone has written a line of code. That is exactly why they get drawn, and it is also why neither of them is a boundary.

Six months later you have four services, and one page load needs all four.

The tell: a chain of synchronous calls

Here is the shape you are looking for. order_svc owns the orders table and nothing else on the page. So it asks:

# order_svc - GET /orders/{oid}
# it owns the orders table. it owns nothing else on this page.

@app.get("/orders/{oid}")
def get_order(oid: int):
    row = db.execute("SELECT * FROM orders WHERE id=?", [oid]).fetchone()

    # the customer name is not ours. ask the user service, and wait.
    user = httpx.get(f"{USER_SVC}/users/{row[1]}").json()

    # the title is not ours either - and this one runs PER LINE ITEM.
    for it in line_items(row):
        p = httpx.get(f"{PRODUCT_SVC}/products/{it[0]}").json()
        # ...and product_svc, to answer that, asks pricing_svc itself.

    return {"order_id": oid, "customer": user["full_name"],
            "items": priced, "total_cents": total}

Read that loop again. The fan-out is not a fixed number — it grows with the data on the page. product_svc is asked once per line item, and each of those asks pricing_svc. On the rig used for the episode, a one-line-item order produced 3 outbound calls; the two-line-item demo order produced 5. The formula is 1 + 2 × line_items.

The first version of this argument said "three services, three calls" and drew a single product fetch. Both were wrong. Any article that says three services means three calls is counting boxes on a diagram, not counting requests.

And 5 and 6 are both correct numbers that mean different things. The X-Hops header reports 5 outbound calls. The rig's total inbound count is 6, because the page load into order_svc is itself a request that can fail. Which one you use matters a great deal in a moment.

That is not four services. It is one distributed monolith: the latency of four, the failure modes of four, and the independence of none.

What the chain actually costs

Every figure below was measured on a real four-service rig, 2,000 requests per build, and the naive, collapsed, and modular builds return byte-for-byte identical responses (diff exit 0). Same feature, three architectures.

What was measured	Result
Naive chain (4 processes, HTTP) — mean / p95	10.12 ms / 12.21 ms
Cost of each extra network hop	~1.80 ms
Collapsed (1 process, 1 database) — mean / p95	1.11 ms / 1.18 ms
Speed-up, identical response bytes	9.09× (9.01 ms/req saved)
Actual work — every DB read, all four services	140 µs = 1.38% of the request
`order_svc` blocked on a socket	87.96% of the request
Outbound calls — 1-item order / 2-item order	3 / 5
Boundary in-process vs no boundary	+30 µs (+2.7%)
Boundary over the network vs no boundary	+9.01 ms (+809%)

Two lines in that table are the whole argument. The real work — every database read in all four services — is 1.38% of the request. order_svc spends 88% of its life blocked on a socket waiting for a service to answer a question it could have answered itself.

And a boundary is not intrinsically expensive. The same wall, drawn inside one process, costs 30 microseconds. Drawn across a network, it costs 9 milliseconds — three hundred times more, for the same separation of concerns. You are not paying for the boundary. You are paying for the transport you chose to put it on.

The availability bill: right rule, wrong exponent

Everyone knows the multiplication rule. Four services at 99.9% each, in a chain where all must succeed:

0.999⁴ = 99.6006% → 172.5 minutes of downtime per month

That figure is arithmetically correct — a 1,000,000-trial Monte Carlo matched it. It is also the wrong number, because the read path does not involve four things that must succeed. It involves six: the request into order_svc, plus the five calls it sets off.

0.999⁶ = 99.4015% → 258.6 minutes per month

That is an 86-minute-a-month gap, and it is hidden by counting boxes on a diagram instead of counting what has to answer. Availability is raised to the number of things that must succeed, not the number of boxes you drew.

One honest note, because the temptation to overclaim here is strong: the measured rig matched theory within one standard error once a rig artefact was removed (a sqlite3 connection shared across the threadpool produced phantom 404s and 500s — 36% of all observed failures, all of them pushing the rate down). Measurement did not beat the theory. The interesting result is the exponent, not a discrepancy.

The deploy story is worse than you have been told

Every article on this subject says: deploy them in the right order. Measured on a one-field rename across the boundary — name → full_name in user_svc, which order_svc reads — both orders fail:

Deploy user_svc first → order_svc raises KeyError: 'name' → HTTP 500
Deploy order_svc first → KeyError: 'full_name' → HTTP 500

There is no ordering that works, because the two versions are never simultaneously correct. The only thing that works is a three-deploy expand/contract sequence: emit both fields, migrate the reader, then drop the old field. The seeded history on the rig shows exactly that shape — a commit reading rename name -> full_name (expand: emit both) and, later, (contract: drop name).

So the real cost of that boundary is not "coordination". It is three releases to rename one field.

So what actually decides a boundary?

Two things, and neither of them is a noun.

1. Who owns the data

Exactly one service may write a given fact. Everyone else asks. The interesting move is not making the service smaller — it is deciding, per fact, whose fact it is.

# the fix is not a smaller service. it is deciding who owns each fact.

# WRONG - the order page asks the product service what the price is.
#         that is the price TODAY. the customer paid LAST MONTH.
price = httpx.get(f"{PRODUCT_SVC}/products/{pid}").json()["price"]

# RIGHT - the price charged is not a product fact at all.
#         it is a fact about THIS order, and it never changes again.
db.execute(
    "INSERT INTO orders (user_id, product_id, price_paid, title_at_sale)"
    " VALUES (?, ?, ?, ?)",
    [user_id, pid, price_now, title_now],
)

# reading that order back now touches one table, in one process.
# the call to pricing did not get faster. it stopped existing.

That last comment is the point. The price the customer paid is a fact about the order, not about the product. Once you see that, the call to pricing_svc on the read path is not slow — it is wrong, and it does not need optimising, it needs deleting. A correct ownership decision does not make a call faster. It removes the call.

This is also, incidentally, a correctness fix and not only a performance one: asking the product service for "the price" on an order page returns the price today, for an order placed last month.

2. What changes together

Two things that are always released in the same commit are one thing wearing two names. You do not have to guess at this — your version control has been recording the answer for years.

$ git log --name-only --pretty=format:%H | python3 cochange.py

  pair                            commits touching BOTH      share
  order_svc   + user_svc                    6               75.0%
  product_svc + pricing_svc                 1               25.0%
  order_svc   + product_svc                 0                0.0%
  order_svc   + pricing_svc                 0                0.0%

# 75%, and every user_svc commit touched order_svc too: one unit of change.
# 0% does NOT automatically mean the wall is earning its keep -
#   check whether that pair is on the READ PATH before you trust it.
#   both zero pairs here are on it, and they were resolved differently.

order_svc and user_svc co-change 75% of the time, and every single user_svc commit also touched order_svc. That is not two services. That is one unit of change with a network cable running through the middle of it.

The two mechanisms disagree, and you should say so

Look at the table again. Co-change says nothing at all about order_svc + product_svc — 0%. So the co-change test does not argue for merging order and product; the read-path fan-out does. Two mechanisms, pointing different ways, on the same pair.

Most writing on this topic presents the rules as if they always agree. They do not. When yours disagree, name which one you are following and why. Here: order and product get merged on fan-out grounds, and order and pricing stay apart despite an identical 0% because pricing comes off the read path entirely once price_paid is owned by the order.

The line, moved

Merge order + user + product into one deployable — three modules, one process, one database. Keep pricing a real service: different team, different release day, and crucially it is not on the read path at all any more, because the order stores price_paid and title_at_sale at the time of sale. Its only remaining edge is a one-way, nightly, asynchronous one.

Losing the network does not mean losing the wall:

# ONE process. ONE deployable. the boundary is still real.

# shop/users/api.py - the only way in. no table name ever leaves this file.
def get_user(uid: int) -> User:
    return _row_to_user(_db.execute(
        "SELECT id, name FROM users WHERE id=?", [uid]).fetchone())

# shop/orders/service.py
from shop.users.api import get_user       # a function, not a URL
def order_page(oid: int):
    o = _orders.by_id(oid)                # our own table
    return {"customer": get_user(o.user_id).name,
            "price": o.price_paid}        # our own column

# and the wall is enforced, not merely agreed:
#   a sqlite authorizer refuses any statement from shop.orders that
#   names the users table. a violation raises. it does not lint.

The last three lines are what separates this from wishful thinking. A module boundary that lives in a code-review convention is not a boundary; it is a hope. A SQLite authorizer that raises when shop.orders names the users table is a boundary, and it costs 30 microseconds.

Three architectures, one feature

	Four services (by noun)	One deployable, no wall	One deployable, modules
Mean latency	10.12 ms	1.11 ms	1.14 ms
Cost of the boundary	+9.01 ms (+809%)	—	+30 µs (+2.7%)
Things that must succeed	6	1	1
Availability at 99.9% each	99.4015%	99.9%	99.9%
Rename one field across the line	3 deploys (expand/contract)	1 commit	1 commit
Wall enforced by	the network	nothing	an authorizer that raises
Moving the line later	data migration	refactor	refactor

The costs, stated plainly

This is not free, and the honest version says so. You end up with fewer and larger services than the diagram wanted, which will feel like a regression to anyone who counts services. Moving a line later is a data migration, not a refactor — merging two databases is real work. And a big in-process module still needs someone to defend the wall, because the compiler will happily let you reach across it.

Which is exactly why the reversible move is to draw the line inside one deployable first: one module, one function as the only way in, and the wall enforced mechanically. If the line turns out to be in the wrong place, you move it with a refactor instead of a migration. If it turns out to be right, promoting it to a process later is a small, well-understood step — and you will have real co-change data by then to prove it.

Keeping two databases in step is its own problem, with its own name and its own article. Do not sign up for it by accident on a whiteboard.

The same two tests, on an LLM-serving stack

None of this is specific to shopping carts, and the AI stack is where the noun-split is currently being repeated most enthusiastically. A typical serving diagram has a gateway, an orchestrator, a retriever, a reranker, a model server, and an eval service — six boxes, one per noun, drawn on day one.

Run the two tests on it.

What changes together? In practice the prompt template, the few-shot examples, and the output parser change in the same commit, essentially always. Change the prompt and the parser breaks; tighten the schema and the prompt has to say so. That is a co-change rate near 100% — one unit of change. Split it into a "prompt service" and a "parsing service" and you have bought yourself three deploys to add a field, for exactly the reason name → full_name needed three.

The chunker and the embedder are the same story from the other side: change the embedding model and every vector in the index is invalid. They ship together or the index is quietly wrong.

Who owns the data? The vector index has exactly one legitimate writer — whichever service owns the ingest path. Everyone else reads. And the price_paid lesson transfers directly: when you log a trace, store the prompt text, the model version, and the retrieved chunk ids as they were at call time, on the trace row. They are facts about that call, not about the retriever's current state. A trace that has to ask the retrieval service what it would return today cannot be replayed, and your eval numbers will drift underneath you the same way a re-fetched price does.

What about the fan-out? A request that goes gateway → orchestrator → retriever → reranker → model server synchronously has the same six-exponent availability bill as the shop did, and every hop lands directly on time-to-first-token — the number your users actually feel. The 88%-blocked-on-a-socket figure is, if anything, kinder than what you will measure here, because a rerank hop is not 1.8 ms.

And which box genuinely earns its own process? The model server. Not because it is a different noun, but because it fails all three of the reasons the others failed: it scales on a completely different axis (GPU memory, batch size, KV cache), it releases on a different cadence, and it does not co-change with the prompt at all. That is what an earned boundary looks like — different data owner, different unit of change, different scaling axis. Not a different word on a whiteboard.

The verdict

A boundary you have to call synchronously on every request is not a boundary. It is a network hop you added to a function call.

The point of the line was never tidiness or a tidy diagram. It was independent deployability — can these two things ship apart, on different days, by different people, with nobody coordinating? If the answer is no, you do not have two services. You have one system, and the line you drew is a cost you are paying for nothing.

So before the next split, ask the two questions that do not appear on the whiteboard: who owns this fact, and what changes with it. Then draw the line inside one deployable, enforce it with something that raises rather than something that comments, and let it earn its process later.

References and further reading

The premise — you have decided to split, and now the line has to go somewhere

Sam Newman, Building Microservices (O'Reilly, 2nd ed. 2021) — independent deployability as the defining property of a service, and the case that a boundary you cannot ship apart is not one.
Martin Fowler, MicroservicePremium (martinfowler.com, 2015) — names the fixed cost every distributed boundary charges before it delivers any benefit, which is the ledger this whole article is keeping.

The tell — a synchronous chain across four services

Martin Fowler, First Law of Distributed Object Design (martinfowler.com, 2003) — "don't distribute your objects": the original statement that a remote call is not a local call with extra latency, it is a different thing.
Michael T. Nygard, Release It! (Pragmatic Bookshelf, 2nd ed. 2018) — integration points as the leading source of instability, and how a synchronous dependency turns one service's bad day into everyone's.

The availability arithmetic

Betsy Beyer et al., eds., Site Reliability Engineering (O'Reilly, 2016), ch. "Embracing Risk" — availability targets, error budgets, and why a service's own target has to account for everything it depends on to answer.

Decider 1 — who owns the data

Chris Richardson, Pattern: Database per Service (microservices.io) — one writer per fact, stated as a pattern with its trade-offs, including the queries it makes hard.
Eric Evans, Domain-Driven Design (Addison-Wesley, 2003) — bounded context: the same word means different things to different parts of the business, which is why "price" belongs to the order and not to the product.

Decider 2 — what changes together

David L. Parnas, On the Criteria To Be Used in Decomposing Systems into Modules (Communications of the ACM, 1972) — decompose by what is likely to change, not by processing steps; the ancestor of the co-change test.
Adam Tornhill, Your Code as a Crime Scene (Pragmatic Bookshelf, 2015) — mining version-control history for change coupling, i.e. how to get the git log table above out of your own repository.
Melvin E. Conway, How Do Committees Invent? (Datamation, 1968) — the org-chart split, and why it keeps getting drawn whether or not it is the right line.

The three-deploy rename

Danilo Sato, ParallelChange (martinfowler.com, 2014) — expand / migrate / contract, the only sequence that survives a rename across a boundary when neither deploy order works.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: the 2-minute version draws the four-service shop and names the two deciders; the full episode runs the ownership fix, the co-change analysis, and the in-process module wall on a rig that really executes.

DEV Community: Vahid Aghajani

Docker Compose vs an Orchestrator — Compose Is a Command, an Orchestrator Is a Loop

The server dies at 03:00. Who starts your containers again?

What this article covers (and what it does not)

A light switch and a thermostat

The Compose file, line by line

The whole system, drawn once

The restart line: what it protects, and what it cannot

The answer: a loop that never exits

The same picture, with one change

What you write: a file, not the loop

The catch: the loop cannot create machines

The bill, because it is real

For your next interview

The one-line verdict

References and further reading

Container Images, Layers & Registries — An Image Is Not One Big File: One Changed Line, 150 MB Moved, and the Dockerfile Order That Makes It 2 MB

You changed one line. Why did 150 MB move?

What this article covers (and what it does not)

A reprinted textbook and a thin clear sheet

The whole system, drawn once

The Dockerfile that most people write first

The build cache: one key for each step

The fix: change the order

The registry asks one question for every layer

The trap: three pods, one tag

The answer: a tag can move, a digest cannot

The catch: a layer only adds

The bill, because it is real

For your next interview

The one-line verdict

References and further reading

async/await Explained: How One Thread Serves a Hundred Waiting Tasks

A thread is a stack, and a stack is a bookmark

await is a suspension point, not "go faster"

Where the state goes: a frame on the heap

One thread, a ready queue of bookmarks

Resuming: the line after the await

The three catches, which are the whole interview

The same mechanism, one floor up: serving LLMs

The verdict

References and further reading

Design a notification system — a full mock system design interview (Ep 4)

Four questions before the first box

The number that decides the design

Why one queue does not work

The API contract

Why you cannot send it inside the request

The retention and idempotency problem

The provider timeout trap

Putting it together: the data flow

Comparison: when to queue vs. when to send inline

LLM inference angle: batch vs. stream

Verdict

Move the data to the compute, or the compute to the data?

The concrete job

Side A: pull the data back

Side B: push the compute down

The trade-off table

This is not a SQL trick

The honest bill for side B

Side A is not wrong

One step further: caches and CDNs

For LLM inference: the same decision under latency

One decision, different clothes

Verdict

Why your index is ignored: expression indexes explained

What a B-tree index actually stores

The fix: expression indexes

The one rule that decides if it works

The trade-off: work moves, not vanishes

For LLM inference workloads

Verdict

Design a Rate Limiter: The Four-Point Answer, and the Trap

1. Count against the API key or the user id, not the IP

2. Enforce it at the gateway, not inside every service

3. The counter is shared state

4. Answer with a 429, a Retry-After, and the quota left

The same question, in front of a model

The trap

`await` is a suspension point, not "go faster"

Resuming: the line after the `await`