DEV Community: Vahid Aghajani

Design a Rate Limiter: The Four-Point Answer, and the Trap

Vahid Aghajani — Tue, 01 Sep 2026 07:53:27 +0000

Originally published on software-engineer-blog.com.

"Design a rate limiter."

It is one of the most common system design interview questions, and it has an unusually reliable failure mode. The candidate hears it as an algorithms question, names one — token bucket, leaky bucket, fixed window, sliding window log — sketches the mechanism, and stops.

Everything they said was correct. It also answered the easy half.

Naming the algorithm tells the interviewer you have read the same article everyone else read. What they are actually listening for starts one question later: where does the count live when you are running more than one server? The algorithm is a function. The counter is state, and state is where system design lives.

Here is the four-point answer.

1. Count against the API key or the user id, not the IP

Before any mechanism, decide what you are counting against. This is a choice with a wrong answer, not a definition.

The tempting answer is the IP address, because it is always there and it needs no authentication. The problem is that an IP address is not a person. One IP is routinely one office, one university campus, one coffee shop, one mobile carrier's NAT pool — hundreds or thousands of unrelated people sharing a single address. Rate limit on it and every one of them shares a single budget. The first heavy user of the morning exhausts the quota for the entire building.

Meanwhile, the abuser you built this for is the one party in the story for whom addresses are cheap and disposable. They rotate through a proxy pool and never touch your ceiling.

So you punish the innocent and miss the abuser — a limiter that is somehow both too strict and too loose, which is a strange thing to ship.

	IP address	API key / user id
What it identifies	A network path, at this moment	An account — the thing you actually meter
Who shares one	An office, a campus, a carrier NAT pool — thousands of strangers	One customer, by construction
Cost to rotate	Cents, via any proxy pool	A new signup, and you can see it happen
Available before auth	Yes	No
Stable across networks	No — wifi to cellular changes it	Yes
Failure mode	Throttles the innocent, misses the abuser	Needs an identity to exist first

Count against the identity: the API key, the user id, the tenant. That is stable across networks, it is the thing your product actually meters, and it is the thing a customer would recognise on an invoice.

There is one honest exception. Anonymous, unauthenticated traffic — a public signup endpoint, a login form, an unauthenticated search — has no identity to count against yet. There, the IP is the best signal available and an IP limit is the right call. Say that out loud in the interview; knowing when the weaker key is the correct key is worth more than rejecting it outright.

2. Enforce it at the gateway, not inside every service

Put the limiter at the front door — the API gateway, the edge proxy, the ingress — and not inside each service behind it.

Two reasons, and they are different in kind.

The operational one: one front door is one place where limits are configured, one place where they are changed, one place to look when a customer says they are being throttled. Implement the check in each service and you have N implementations of the same rule, which will drift, and a limit change becomes N deploys.

The economic one is stronger. A request rejected at the door never gets past the door. It does not open a database connection, does not enqueue a job, does not hit your billing code, does not consume a worker slot. That is the entire point of rate limiting under load: the traffic you refuse must be cheap to refuse. A limiter that sits deep in the stack — say, as a decorator on a service method — has already paid for connection setup, deserialisation and authentication before it decides the answer is no. Under the exact conditions you built it for, it costs you the most.

There is a second-order version of the same mistake: a request that fails the check in service C has already done real work in A and B.

3. The counter is shared state

This is the point most answers miss, and it is where the question is actually decided.

Say you keep the count in a dictionary in the API process. It is fast, it needs no extra infrastructure, and it works perfectly in development, where you run one process.

Then you run three.

You wrote "100 requests per minute" in the spec. What you have built is 100 requests per minute per instance. Three instances is 300. Ten is 1000. And the number is not even fixed — a load balancer spreading a user's requests across the fleet means each instance sees roughly a third of that user's traffic and never trips, so in practice the ceiling is higher still and depends on how the balancer happens to route.

Then the autoscaler joins in. Now the effective limit moves with your traffic: busy hour, more instances, higher real limit — precisely the opposite of what a limiter is for. The system is most permissive at the moment it should be strictest, and nothing in your monitoring will say so, because every individual instance is dutifully enforcing 100.

The fix is not a cleverer algorithm. It is one store, one atomic increment.

One store: a single place — Redis is the usual choice — that every instance talks to, so there is exactly one number per key per window.

One atomic increment: not read, then decide, then write. Those are three steps, and two requests arriving together can both read 99, both decide they are allowed, and both write 100. The check and the claim have to be a single indivisible operation. Whether that is INCR with an expiry, a conditional decrement, or a small server-side script is an implementation detail; the invariant is what earns the point.

That store is now on the hot path of every single request, so say the consequences out loud too: it needs to be fast, it needs to be close, and you need an answer for what happens when it is unreachable. Fail open and you have no limiter during exactly the incident that produced the outage. Fail closed and a limiter outage is a full outage. Choosing deliberately, and being able to say which way you chose and why, is the senior answer.

4. Answer with a 429, a Retry-After, and the quota left

The last point is the contract, and it is the one candidates skip because it feels like trivia.

Reject with 429 Too Many Requests. Include Retry-After with the number of seconds until the window resets. Include the remaining quota — X-RateLimit-Remaining, or whatever your API already uses — on the successful responses too, so a well-behaved client can see the wall coming before it hits it.

HTTP/1.1 429 Too Many Requests
Retry-After: 12
X-RateLimit-Remaining: 0

The reason this matters is not politeness. It is that the client now has a number to wait on instead of a guess.

A limiter that returns a bare 500, or a 403, or a 200 with an error body, teaches every client integrating against you to invent its own retry timing — and the timings they invent are uniformly worse than the one you already know, because you are the only party who knows when the window resets. You are handing out a fact you have and they do not. Withholding it costs you the retries.

The same question, in front of a model

If you have ever been handed an LLM API key, you have already met all four points wearing different names — and one extra twist that makes this the sharpest live example of the pattern.

Look at how a model provider states its limits. Not "100 requests per minute" but RPM and TPM — requests per minute and tokens per minute, enforced together. That second dimension breaks an assumption the classic answer quietly makes: that every request costs the same. Here they do not. One request is forty tokens; the next drags a 100,000-token document behind it.

So the count is no longer +1. It is +cost, and you do not know the cost until you have counted the prompt — and you do not know the full cost until the model has finished generating, because output tokens count too. Real limiters handle this by reserving an estimate up front and reconciling on completion: increment by prompt tokens plus a max-output allowance, then correct the counter down when the stream ends. That is still one store and one atomic operation. It is just an operation that adds a number rather than one.

The other three points transfer unchanged, and get more important rather than less:

What you count against — the tenant, not the IP, because one of your customers is a chat product with ten thousand users behind one egress address.
Where you enforce it — at the gateway in front of the model, because a rejected inference request must never reach the GPU. This is the economic argument at its most extreme: a request admitted by mistake does not cost you a database connection, it occupies a slot on a machine that costs dollars an hour, for seconds at a time.
The reply — 429 with a Retry-After. Every major provider sends one, and every serious client library reads it.

And if you are serving your own model rather than calling someone else's, the scarce resource shifts again: what actually runs out is not requests per minute but concurrent sequences in the batch and blocks in the KV cache. A limiter in front of an inference server is really admission control for GPU memory, and the honest version of it counts what the GPU is short of rather than what is easy to count.

Which is the same lesson as point 3, one level up: the interesting part was never the algorithm. It was working out what the number means and where it is kept.

The trap

Everyone answers "token bucket". The interviewer is waiting to hear where the count lives.

That is the whole question, compressed. The algorithm is a small, well-documented function that you could look up in a minute. The counter is distributed mutable state on the hot path of every request — which is a system design problem, which is why this is a system design interview.

If you have time for one more sentence after the four points, make it this one: the algorithm decides whether to allow a request, and the storage decides whether that decision is true.

The verdict — the whole answer in four lines

key → count per user or API key, not per IP; IP only for anonymous traffic
gateway → one front door, so a rejected request costs nothing
counter → one shared store, one atomic increment, because in-process counts multiply by your instance count
429 → plus Retry-After and the remaining quota, so the client waits on a number instead of a guess

Name an algorithm if you like — token bucket is a fine choice and takes five seconds to justify. Then spend the rest of your answer on the four points above, because that is the part the interviewer cannot look up on your behalf.

References and further reading

On the reply contract — point 4

IETF, RFC 6585 §4 — Additional HTTP Status Codes: 429 Too Many Requests (IETF, 2012) — the normative definition of the status code: the server indicates the user has sent too many requests in a given amount of time, and the response may include a Retry-After. Worth quoting in an interview because the spec explicitly declines to define how the server counts, which is exactly the storage decision this article argues you are being tested on: rfc-editor.org/rfc/rfc6585#section-4
IETF, RFC 9110 §10.2.3 — HTTP Semantics: the Retry-After header field (IETF, 2022) — the support for "a number to wait on, not a guess": Retry-After carries either a delay in seconds or an HTTP-date after which the client may retry, and nothing else: rfc-editor.org/rfc/rfc9110#field.retry-after
IETF, draft-ietf-httpapi-ratelimit-headers — RateLimit header fields for HTTP (IETF, in progress) — the ongoing standardisation of the "quota remaining" half of point 4, and the reason the X-RateLimit-* headers above are a de-facto convention rather than a standard. Note that it is a draft, not a published RFC: datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers

On the shared counter — point 3

Redis documentation, INCR — "Pattern: Rate limiter" — the concrete mechanism behind "one store, one atomic increment", from the store this design would actually use: a counter per key per window, incremented atomically and expired with the window. It also documents the exact race the wording above is chosen to avoid — the gap between the INCR and the EXPIRE: redis.io/docs/latest/commands/incr

On what to count and where to enforce it — points 1 and 2

Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering, chapter 21, "Handling Overload" (O'Reilly / Google, 2016) — Google's practice of metering per customer against a quota rather than per source address, and the argument that a rejected request is not free: it still costs CPU and memory, so it must be refused as early and as cheaply as possible. That is the economic case for the front door.
Paul Tarjan, Scaling your API with rate limiters (Stripe Engineering blog, 2017) — a production account that lines up with points 1, 2 and 4: limits applied per API key at the edge rather than inside application code, backed by a shared Redis counter, returning 429 with enough information for the caller to act: stripe.com/blog/rate-limiters

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the reel: Design a rate limiter — the four-point answer, and the trap

MCP vs API: Why Your Server Needs a /mcp Endpoint (and What You Actually Write)

Vahid Aghajani — Mon, 31 Aug 2026 15:25:57 +0000

Originally published on software-engineer-blog.com.

Same server. Same database. Same functions.

Your service already has an API. It has had one for years. It is documented, it is authenticated, it is in production, and it works.

So why is everyone suddenly standing up a /mcp endpoint right next to it?

Not because the old one is broken. Behind both doors is the same server, the same database, the same Python functions. Nothing underneath changes. The difference is one sentence long:

Everything else in this article is a consequence of that sentence — including the one part of the job that no library will do for you.

Why a plain endpoint is not enough

Think about how your API actually got called.

Somewhere, a human being opened your documentation. They read that the path is /v1/invoices, that it takes a customer_id and a from and a to, that the dates are ISO-8601 and that the response is paginated. Then they went back to their editor and hard-coded that knowledge into a client.

All of the understanding happened outside the wire. The endpoint itself never explained anything. Ask it what it is for and it has no answer — there is no request you can send it that means "describe yourself." It was never designed to have one, because the reader was always going to be a person with a browser.

A model does not have a browser, and it was not in the room when your docs were read. If the only thing on the wire is POST /v1/invoices, the model is guessing.

What MCP changes is the first move

Here is the actual shift, and it is smaller than it sounds:

With an API, the first thing that happens on the wire is a call.
With MCP, the first thing that happens on the wire is a question.

Before invoking anything, the client asks the server: what can you do? The server answers with a list. Only then does the model pick something from that list and call it.

Discovery first, invocation second. That ordering is the protocol's whole reason to exist.

1. client → server   tools/list      "what can you do?"
2. server → client   [ 8 tools, each with a description and a schema ]
3. client → server   tools/call      "do this one, with these arguments"
4. server → client   the result

Steps 1 and 2 are the part your REST API has never had.

The thing every REST developer notices first

Open the traffic and something looks wrong: there is no resource path.

Your API puts the noun in the URL and the verb in the HTTP method — GET /invoices/42, DELETE /invoices/42. MCP does neither. There is one path, /mcp, and every single message is a POST to it.

That is because MCP speaks JSON-RPC 2.0. The operation is not in the URL and not in the method — it is in the body, in a field called method:

POST /mcp
Authorization: Bearer <token>
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list",
  "params": {}
}

The reply is a menu — and this is the interesting part. Each entry carries a name, a plain-English description of what it is for, and a schema in which every parameter has its own description:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {
        "name": "find_invoices",
        "description": "Find a customer's invoices for a given year. Use this when someone asks about billing history, unpaid bills, or what a customer was charged.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "customer": {
              "type": "string",
              "description": "Customer name or account ID, e.g. 'ACME Ltd' or 'cus_8812'"
            },
            "year": {
              "type": "integer",
              "description": "Four-digit calendar year, e.g. 2026"
            }
          },
          "required": ["customer", "year"]
        }
      }
    ]
  }
}

That is documentation, delivered as data, over the same wire the call goes out on. Nobody had to read anything in a browser.

Calling goes through the same door. Only the method changes:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "find_invoices",
    "arguments": { "customer": "ACME Ltd", "year": 2026 }
  }
}

And it is authenticated exactly the way your API is — a bearer token on every request. This is not a new security model. It is your security model, on a new endpoint.

You never write that JSON by hand

Look at that schema again and the obvious objection is: nobody is maintaining that.

Correct. You do not write it. FastMCP derives it from code you were already writing:

from fastmcp import FastMCP
from pydantic import Field
from typing import Annotated

mcp = FastMCP("billing")

@mcp.tool
def find_invoices(
    customer: Annotated[str, Field(description="Customer name or account ID, e.g. 'ACME Ltd' or 'cus_8812'")],
    year: Annotated[int, Field(description="Four-digit calendar year, e.g. 2026")],
) -> list[dict]:
    """Find a customer's invoices for a given year.

    Use this when someone asks about billing history, unpaid bills,
    or what a customer was charged.
    """
    return billing.query_invoices(customer=customer, year=year)

Three mappings, and that is the whole trick:

What you write in Python	What the model receives
Function name `find_invoices`	The tool `name`
The docstring	The tool `description` — what it is for and when to reach for it
Type hints + `Field(description=...)`	The `inputSchema`, one described parameter at a time

Which has a consequence worth sitting with for a second.

It is no longer read by a teammate skimming the file. It is read by the thing deciding whether to call your function at all, and with what. A vague docstring is now a runtime failure mode, not a code-review nit.

The part that is not generated: the tools themselves

Here is where most first MCP servers go wrong, and it is the reason this article exists.

Because FastMCP generates the schema, it is tempting to conclude that the whole job is generated — point something at your OpenAPI spec, get an MCP server out, ship it. You will get a server. You will not get a usable one.

MCP is not a thin 1:1 layer over your existing endpoints. You write a new tool per capability, and the good ones are shaped like tasks, not like resources.

That distinction is concrete. Your REST API is decomposed the way a database is: /customers, /invoices, /line-items, each with filters, each returning a page. Answering "what did ACME pay us in 2026?" takes three calls and some glue — the human client author wrote that glue once and forgot about it.

A tool has no glue author. So the tool is the whole task:

	Auto-generated from the spec	Written by hand
Unit of design	One tool per endpoint / per resource	One tool per task a user actually asks for
Typical count	60+	8
Answering "what did ACME pay in 2026?"	3 chained calls, IDs threaded by the model	1 call: `find_invoices(customer, year)`
Descriptions	Inherited from a spec written for humans, or empty	Written for the decision the model has to make
Parameters	Every filter the endpoint supports, most of them noise	Only what the task needs
Failure mode	Model picks the wrong one of six similar tools, or gives up	Model picks correctly, because the choice is obvious

Point a generator at your OpenAPI spec and you get 60 tools nobody can use. Write 8 by hand, with real docstrings, and the model uses them correctly.

Why this bites harder than it looks (the AI-engineering angle)

If you have only built request/response systems, the cost of a bad tool surface is not obvious. It is not a latency problem. It is a context and decision problem, and it shows up in three places.

Every tool is permanently in the prompt. The tool list is not fetched when needed — it is serialised into the model's context on every single turn of the conversation. Sixty tools with full schemas is thousands of tokens spent before the user has said anything, on every message, forever. Eight task-shaped tools cost a fraction of that, and leave the budget for the actual work.

Every extra tool is another chance to choose wrong. Selection accuracy degrades as near-duplicates pile up. Six endpoints that all list something, with descriptions inherited from a spec written for a human who already knew which one they wanted, is a menu designed to be misread. This is the same reason a chained workflow is fragile: three dependent calls means three chances to thread the wrong ID, and one wrong ID is a confidently wrong answer rather than an error.

The description is the interface. In a normal API, the contract is the signature and the docs are advisory. Here the prose is the routing logic — a tool whose description does not say when to use it will be called at the wrong moment, no matter how correct its implementation is. Write descriptions that answer the model's actual question ("is this the one?"), not the ones that restate the function name.

Which gives you a useful design test. For each tool, ask: is this a thing a user would ask for in a sentence? find_invoices(customer, year) passes. list_line_items(invoice_id, page, per_page, sort) does not — it is an implementation detail that leaked onto the menu.

The verdict

Adding MCP is not rewriting your product, and it is not flipping a switch either.

Keep everything below the surface: your queries, your business logic, your auth, your database. None of it moves. The REST API stays exactly where it is, serving the clients it already serves — MCP does not replace it, it sits beside it.

What is genuinely new is a tool surface, and that surface is a design artifact you author. The protocol gives you discovery. FastMCP gives you the schema. The judgment about which capabilities exist and what shape they have is the part that is still yours, and it is the part that decides whether the thing works.

The short version, if you take one line away: the schema is generated for you; the tool design is not. Write eight tools shaped like the questions people ask, give each a docstring you would be happy to have read aloud as an instruction, and you are done.

References and further reading

On the protocol itself — discovery, and the JSON-RPC envelope

Model Context Protocol, Specification — the normative definition of tools/list and tools/call, the tool object with its description and inputSchema, and the transport and authorization sections behind the bearer-token point above: modelcontextprotocol.io/specification
JSON-RPC Working Group, JSON-RPC 2.0 Specification (2010) — the envelope MCP is built on: why the operation lives in a method field inside the body rather than in a URL or an HTTP verb, and how id, params, result and error fit together: jsonrpc.org/specification

On why a REST API looks so different — the noun in the URL

Roy T. Fielding, Architectural Styles and the Design of Network-based Software Architectures (PhD dissertation, UC Irvine, 2000), ch. 5 — the resource/identifier/uniform-interface model that puts the noun in the path and the verb in the method, which is exactly the convention MCP declines to use: ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
OpenAPI Initiative, OpenAPI Specification — what an endpoint-shaped description of a service actually contains (paths, operations, parameters), and therefore what a generator pointed at one can and cannot know about the tasks a user wants to perform: spec.openapis.org/oas/latest.html

On generating the schema from code

FastMCP documentation — the @mcp.tool decorator and the derivation rules used above: function name to tool name, docstring to description, type hints to inputSchema: gofastmcp.com
Pydantic documentation, Fields and JSON Schema — how Field(description=...) and annotated types become the per-parameter descriptions the model reads, which is where most of the usable signal in a tool schema comes from: docs.pydantic.dev/latest/concepts/fields/

On the part that is not generated — designing the tools

Anthropic, Writing effective tools for agents (Anthropic Engineering, 2025) — the case for consolidating several low-level endpoints into one task-shaped tool, for spending real effort on tool descriptions, and for evaluating a tool surface rather than assuming it: anthropic.com/engineering/writing-tools-for-agents

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the short: MCP vs API — Why Your Server Needs a /mcp Endpoint

Keep going: the rest of the free AI-engineering course lives at software-engineer-blog.com/ai.

RTO vs RPO: The Two Numbers That Decide What Disaster Recovery Costs

Vahid Aghajani — Mon, 31 Aug 2026 07:34:53 +0000

Originally published on software-engineer-blog.com.

Two letters apart. Both measured in minutes. Bought with completely different money.

That is why teams argue about disaster recovery for a week without noticing they are talking about two separate things. This article takes the two numbers apart, and then prices both of them against one small shop.

Tuesday, twelve minutes past three

The server that holds every order for an online shop is gone.

The alert fires. Somebody wakes up. By twenty past three they have found the most recent saved copy of the data. That copy was taken at three, on the hour. So they build a new server, and they put that copy onto it. At nineteen minutes past four, the shop can take orders again.

Now count what actually happened.

The shop was shut for sixty-seven minutes. And twelve minutes of orders — the ones taken between three o'clock and the moment it died — do not exist anywhere any more.

Two different things went wrong.

For the next week, the team argued about how to make sure it never happened again. And in that whole week, nobody ever said which of the two numbers they were trying to make smaller. So which one was it? The sixty-seven minutes the shop was shut, or the twelve minutes of orders that vanished?

What this article covers, and what it does not

Three things are covered:

What the two numbers actually are. One is how long you may be down. The other is how much data you may lose.
What each one is bought with, because they are not bought with the same money.
How the standard tiers are priced, and where that price stops being reasonable.

Three things are not covered. This is not how backups work on the inside — there are no restore steps here, and that is a subject for its own article. This is not about which site serves your traffic or how failover flips between two sites, which is covered elsewhere on this site. And this is not one product: no tool and no cloud provider is named anywhere below.

Two dials, not one

Here is the confusion this kills. People hear "disaster recovery" and think there is one dial, and that turning it up makes everything better.

There are two dials. A shop that catches fire is the easiest way to see them.

First question. How long is the shop shut? The fire is out, but the doors still do not open. Every hour shut is customers who go somewhere else. You make that shorter by renting a second shop, already fitted out and ready. And a second shop costs rent every single month, whether or not there is ever a fire.

Second question. How many pages of the order book burned? The shop can open again, but some orders simply do not exist. You lost every order taken since the last time somebody carried a page across the road to the safe. You make that shorter by copying more often. And copying after every single order slows down every single order you ever take.

Now notice how independent those two are. You could reopen in an hour with a whole day of orders missing. You could lose nothing at all and stay shut for a week. Both are possible, because these are two different numbers.

One line of time, two arrows

Draw the life of the failure as a line of time. Mark the moment it broke. Everything to the left already happened. Everything to the right has not happened yet.

Look backwards from the break, to the last moment your data was safely saved somewhere else. That distance is the recovery point objective, or RPO. It is the data you are willing to lose. On Tuesday it was twelve minutes.

Look forwards from the break, to the moment a customer could place an order again. That distance is the recovery time objective, or RTO. It is the time you are willing to be down. On Tuesday it was sixty-seven minutes.

Same instant. Two arrows. They point in opposite directions.

	RPO	RTO
Which way it points	backwards, into the data	forwards, into the clock
What it measures	work that vanished	time the doors were shut
What sets it	how often you copy	how fast you can be serving again
On Tuesday	12 minutes	67 minutes

Both are measured in minutes, and that is exactly why people mix them up. Every section below is this same line with the two marks moved.

RPO: the promise is the worst case

Suppose you save a copy of the data every fifteen minutes. What have you actually promised?

Here is the whole thing, in a few lines.

# rpo.py -- what does "we copy every fifteen minutes" really promise?
COPY_EVERY = 15 * 60      # seconds between one saved copy and the next
ORDERS_PER_MIN = 50       # this shop, on an ordinary day

def lost(failed_at, last_copy_at):
    return failed_at - last_copy_at    # everything in the gap is gone

# The failure does not check your schedule. It lands anywhere in the gap.
worst = lost(failed_at=899, last_copy_at=0)   # a second before the next copy
best  = lost(failed_at=1,   last_copy_at=0)   # a second after the last one

print(worst, best)                 # 899 1  -> the whole gap, or almost nothing
print(worst / 60 * ORDERS_PER_MIN) # 749.2  -> orders that never existed

Everything written between the last saved copy and the failure is gone. That is the definition, and it is the only thing that matters here.

Now, the failure does not check your schedule before it happens. It can land anywhere inside that gap. If it lands one second before the next copy was due, you lose fourteen minutes and fifty-nine seconds of work. If it lands one second after a copy, you lose one second. Both are possible on the same schedule.

And here is the trap. The number you promise the business is the worst case, never the average. A copy every fifteen minutes means fifteen minutes.

This shop takes fifty orders a minute, and an order is worth thirty-eight euros. So the same schedule, said honestly, looks like this:

How often you copy	Worst case	Orders that never existed	Value
Once a night	23 h 59 min 59 s	71,999	EUR 2,735,968
Every 15 minutes	14 min 59 s	749	EUR 28,468
As each order is taken, about 2 s behind	2 seconds	1.67	EUR 63

Look at what changed between those three rows. Not the quality of the copies. Not the storage. Not the restore. The only thing that changed is how often.

That is the only lever RPO has, and every step of it costs you something on the way in. Which is why arguing about backup software rarely moves this number at all.

RTO is a sum, and the restore is the small term

Now the other number, and it has a trap of its own.

Ask a team what their RTO is, and most of them answer with how long the restore takes. Watch what that leaves out. The clock does not start when you start working. It starts when it breaks.

# rto.py -- the clock starts when it breaks, not when you start working.
detect    =  4 * 60    # the alert fires, and a human actually sees it
decide    = 11 * 60    # somebody decides this is a disaster, not a blip
provision =  9 * 60    # new machines exist, and they are reachable
restore   = 26 * 60    # the data is put back onto them
verify    =  8 * 60    # somebody checks it is really the right data
cut_over  =  9 * 60    # traffic is pointed at the new place

RTO = detect + decide + provision + restore + verify + cut_over
print(RTO // 60)       # 67  -> the real number, in minutes

# the one term everybody budgets for, on its own:
print(restore // 60)   # 26  -> not even half of the sixty seven

Sixty-seven minutes in total. And the one term everybody budgets for, the restore, is twenty-six of them. That is 38.8 percent. Not even half.

Term	Minutes	Can money make it shorter?
Somebody actually sees the alert	4	no — people
Somebody decides this is a real disaster	11	no — people
New machines exist and are reachable	9	partly
The data is put back	26	yes
Somebody checks it is the right data	8	no — process
Traffic is pointed at the new place	9	partly
Total	67

Here is why that split matters. The twenty-six is the part you can buy your way out of, with faster disks and a bigger pipe. The other forty-one minutes are people and process. No hardware makes any of that shorter.

So a team that spends its whole recovery budget on a faster restore cannot get its RTO below forty-one minutes, no matter what else it spends. And it usually does not know that.

If you want the number below forty-one, you have to stop restoring, and start already having a second copy running.

The three tiers, and why you do not choose them

Once both numbers are written down, something useful happens. The strategy stops being a choice.

There are three tiers, and they sit at fixed points on those two numbers.

Tier	RTO	RPO	What it costs
A saved copy, and a rebuild	hours	however often you copy	storage, and almost nothing else
A second site, kept warm	minutes	however far behind the copying runs	a second bill, every month
Two live sites, every write waits for both	seconds	near zero	that second bill, plus something on every request

Cheaper and slower at the top. Faster and far more expensive at the bottom.

For this shop, at fifty orders a minute, the difference between the rows is orders that were never placed at all:

Tier	Time down	Orders never placed
A saved copy, and a rebuild	67 minutes	3,350
A second site, kept warm	5 minutes	250
Two live sites	20 seconds	17

You do not shop for these three. You write down your two numbers, and exactly one row is left.

What RTO is bought with: capacity that sits there

Moving RTO down means having capacity that is already running when the failure happens. Not capacity you can create quickly. Capacity that is already there.

A second site kept warm is machines that are running, patched, watched and paid for on every ordinary day.

Say the production environment costs four thousand two hundred euros a month. A full warm twin doubles that. Fifty thousand four hundred euros a year.

And here is the shape of that purchase. The bill arrives twelve times a year, every year. The outage it exists for might arrive once. Or never.

That is not an argument against buying it. It is an argument for knowing exactly what you are buying.

What RPO is bought with: waiting, on every write

Now the other mark. This is where it gets interesting, because RPO is not bought with money. It is bought with waiting.

Here is the same order, saved two ways.

# write.py -- the same order, saved two ways. Only the waiting differs.
RTT_MS = 12               # a round trip to the second place, far away
WRITES_PER_ORDER = 6      # one checkout touches six rows

def save_async(row):      # RPO is the lag: about two seconds of orders
    write_here(row)       # the customer waits for this, and nothing else
    send_to_other_place(row)   # this happens after the customer is gone

def save_sync(row):       # RPO is zero: nothing is ever lost
    write_here(row)
    wait_for_other_place(row)  # the customer waits for this too, every time

print(RTT_MS * WRITES_PER_ORDER)   # 72 -> milliseconds added to every order

In the first version, you write the order here, and send it to the other place afterwards. The customer waits for the first write and nothing else. RPO is whatever the sending falls behind by. Call it two seconds.

In the second version, you write it here, and then you wait for the other place to say yes before you tell the customer their order went through. Now RPO is zero. Nothing is ever lost.

But look at who is waiting. The customer is. Every time.

If the other place is far enough away that a round trip takes twelve milliseconds, and one checkout touches six rows, that is seventy-two milliseconds added to every order. Forever.

Seventy-two milliseconds does not sound like much. So multiply it out.

What	Amount
Orders a year, at 50 a minute	26,280,000
Added to each one	72 ms
Added up, per year	1,892,160 s = 525.6 hours = 21.9 days
What it protects	1.67 orders — EUR 63 — on the one bad day

Almost twenty-two whole days of human waiting, added up, every single year. To buy back sixty-three euros of orders on a day that may or may not ever come.

That is what driving RPO all the way to zero actually costs.

The step that is worth it, and the step right next to it

Now put the money side by side. This shop loses one thousand nine hundred euros for every minute it is down — fifty orders a minute, thirty-eight euros an order.

Step one. From sixty-seven minutes down to five.

That costs one more full environment: fifty thousand four hundred euros a year. It saves sixty-two minutes of downtime, which is one hundred and seventeen thousand, eight hundred euros per outage.

Break-even is 0.43 outages a year. Have one outage every two years, and it has already paid for itself. That is an easy yes.

Step two. From five minutes down to twenty seconds.

That costs another environment, and now every write waits for the far side. It saves four minutes and forty seconds, which is eight thousand, eight hundred and sixty-seven euros per outage.

Break-even is 5.7 outages a year. And you pay the five hundred and twenty-five hours of customer waiting either way.

	Step one: 67 min → 5 min	Step two: 5 min → 20 s
Saved per outage	EUR 117,800	EUR 8,867
Costs	EUR 50,400 a year	another EUR 50,400 a year
Plus	nothing on the request path	525.6 hours of waiting a year
Break-even	0.43 outages a year	5.7 outages a year
Verdict	an easy yes	almost certainly no

Same ladder. Same shop. Same arithmetic. The step that is obviously worth it and the step that is obviously not are right next to each other.

This is the shape to remember. Both marks move a long way for money you can put in a budget. It is the last small step — pushing either mark all the way onto the failure itself — where the price stops being money and starts being waiting on every request you will ever serve.

How you actually pick

You ask the business two questions, and you write the answers down before anybody says the word "backup".

How long may we be shut before it really hurts?
How much work may vanish before it really hurts?

Those two answers are your RTO and your RPO. Everything after that is arithmetic, and the arithmetic is genuinely this small.

# pick.py -- you do not pick a strategy. You pick two numbers.
# Ask the business first, and write both answers down before anything else.
rto_minutes = 5      # how long may we be shut before it really hurts?
rpo_seconds = 2      # how much work may vanish before it really hurts?

def tier(rto_minutes, rpo_seconds):
    if rto_minutes > 60 or rpo_seconds > 900:
        return "a saved copy, and a rebuild"        # cheapest. Hours.
    if rto_minutes > 1 or rpo_seconds > 0:
        return "a second site, kept warm"           # minutes
    return "two live sites, and every write waits"  # seconds, and it hurts

print(tier(rto_minutes, rpo_seconds))   # a second site, kept warm

The two numbers came first. The tier came out of them.

Three questions you can answer this afternoon

First. If the main copy of your data disappeared right now, how many minutes of work would be gone? Not on average. In the worst case. If nobody can say the number, you do not have an RPO — you have a schedule.

Second. Add up your six terms: seeing the alert, deciding it is real, getting machines, putting the data back, checking it, switching traffic over. What is your floor once you remove the restore? If that floor is already above what you promised, buying a faster restore cannot save you.

Third. Who agreed to these two numbers? If the answer is "the engineers", they are estimates. They become objectives when the business says them out loud.

The verdict

	RTO	RPO
The question	How long may we be down?	How much work may vanish?
Measured	forwards from the break	backwards from the break
Set by	a sum of six terms, only one of which is the restore	how often you copy — nothing else
Bought with	capacity running on every ordinary day	a wait added to every write, forever
You pay	whether or not anything ever breaks	on every request you will ever serve
Fails quietly when	the budget all goes to a faster restore	the schedule is quoted as an average

RTO is how long you may be down, and you buy it with machines that sit there costing money while nothing is wrong.

RPO is how much work may vanish, and you buy it with a wait added to every write, forever.

Write both numbers down before anybody says the words "backup", "standby" or "second region". After that, the tier is the only thing left to read off.

You do not pick a disaster recovery strategy. You pick two numbers, and the strategy falls out of them.

References and further reading

On the two definitions

NIST Special Publication 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems (NIST, 2010) — the formal definitions both terms come from: RTO as the maximum time a resource may be unavailable before the impact is unacceptable, and RPO as the point in time to which data can be recovered, which in practice is how much data loss the organisation can tolerate: nvlpubs.nist.gov

On writing the two numbers down with the business

Laine Campbell and Charity Majors, Database Reliability Engineering (O'Reilly, 2017), ch. 2 "Service-Level Management" and ch. 7 "Backup and Recovery" — the same two numbers, under the names availability and durability, treated as service-level indicators agreed in advance. Durability is defined there in exactly this shape: "in the event of a system failure, no more than the past two seconds of data can be lost." Chapter 7 then opens the recovery-strategy discussion by sending you back to those objectives first.
Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara and Stephen Thorne (eds.), The Site Reliability Workbook (O'Reilly, 2018), ch. 2 "Implementing SLOs" — how a reliability target is chosen with the people who own the product rather than by the team that will engineer to it, and why the rationale behind the number gets written down with it: sre.google/workbook/implementing-slos

On the restore being the small term

Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 26 "Data Integrity: What You Read Is What You Wrote" — "No one really wants to make backups; what people really want are restores", and the requirement that recovery finish "well within the uptime needs of a service", which is the RTO by another name: sre.google/sre-book/data-integrity

On what the last step of RPO costs

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), ch. 5 "Replication", section "Synchronous Versus Asynchronous Replication" — waiting for the second copy to acknowledge is what removes the loss window, and it is also what puts an unbounded wait on the write path, which is why almost all replication is asynchronous in practice.

On the last rung costing more than the outage

Betsy Beyer et al. (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 3 "Embracing Risk" — reliability is not maximised, it is targeted: "an incremental improvement in reliability may cost 100x more than the previous increment", and the target is treated as both a minimum and a maximum: sre.google/sre-book/embracing-risk

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: RTO vs RPO on YouTube

Cache Invalidation: The Delete Ran, and the Page Is Still Wrong

Vahid Aghajani — Sun, 30 Aug 2026 14:56:48 +0000

Originally published on software-engineer-blog.com.

The ticket that makes no sense

A user changes her email address. The form says saved.

She reloads the page. The old address is still there.

You go and look, because you wrote this code and you know it is fine. The database has the new value. You check the logs: the UPDATE returned ok. The cache delete right after it returned ok too. There is no exception, no timeout, no retry, no failed job sitting in a queue. Every single call in that request succeeded.

And the cache is still holding the old email, with five minutes left on its timer.

Nothing failed. Nobody forgot anything. So what actually happened?

What this article covers, and what it does not

Three things are covered:

Why an invalidation that provably ran can leave a stale value behind — and why the delete was not too late, it was too early.
The three moves that fix it, in order, each one measured rather than asserted.
The honest limit: the class of cached answers you cannot invalidate at all, and what the workaround costs.

Five things are deliberately not covered, because each one is its own subject and gets confused with this one:

This is not the dual-write problem. That is the case where the second write never happened — the database committed and the cache call died. Here the cache call ran and returned successfully. If your failure mode is a lost second write, you want an outbox, not this article.
This is not the write path. Write-through and write-back are about whether writes go through the cache at all. Everything below assumes cache-aside: your application code reads the cache, falls back to the database, and fills the cache itself.
This is not the stampede. Invalidating a hot key sends every waiting reader to the database at the same instant. That is a real and separate cost, and it is a real and separate article.
This is not eviction. LRU and LFU decide what to throw away when memory runs out. Eviction is about space. Invalidation is about truth. They both remove keys and that is the entire similarity.
This is not HTTP caching. No browser caches, no ETag, no max-age, no CDN purge. Everything here happens on your side of the wire, inside your own service.

One system, drawn once

There are exactly three things, and everything below is a mutation of this picture.

A service. Beside it a cache — say one Redis. Under it a database — say one Postgres primary. The service talks to both. The cache and the database never talk to each other. Nothing keeps them in step except the code you write.

The read path:

# every read goes through here. ex=300 is five minutes.
def get_user(user_id):
    key = "user:" + str(user_id)
    hit = cache.get(key)
    if hit is not None:
        return hit          # a hit. No database call.
    row = db.query_one(
        "SELECT * FROM users WHERE id = %s", (user_id,))
    cache.set(key, row, ex=300)  # a miss. Fill it back.
    return row

And the write path — the one in the ticket:

# the write path. Two calls, and both of them succeed.
def change_email(user_id, new_email):
    db.execute(
        "UPDATE users SET email = %s WHERE id = %s",
        (new_email, user_id))
    cache.delete("user:" + str(user_id))
    return "saved"

# Order matters. Database first, cache second.
# Delete before the commit and a reader can refill the
# key from the old row before the new one is even there.

Look at those two functions. They are correct. That is the point of the article.

The race: the delete was too early

Put a second person in the picture. Not a second writer — a reader, doing nothing unusual.

Follow one clock:

A read arrives. Cache miss.
That reader runs SELECT and gets the old row. It is now holding that value in a local variable, on its way to cache.set.
The writer commits UPDATE. The database now holds the new value.
The writer runs DELETE user:42.
The reader — a couple of milliseconds late — finally runs cache.set(key, old_row, ex=300).

The database says new. The cache says old. And the cache has a full five-minute timer ahead of it, because step 5 set a fresh one.

Here is the part that surprises people, measured on a real Redis 7.4.9 rather than reasoned about. In step 4, the delete returns:

4. writer DEL user:42 -> DEL returned 0
   (key existed before = 0, exists after = 0)

Zero. The delete removed nothing, and it removed nothing because there was nothing there yet. The stale value was still in flight inside another thread. You cannot delete a value that has not been written.

Say that out loud, because it is the whole idea: the invalidation was not too late. It was too early.

And it is not a fluke of the cold cache. A second run was built where a different reader had already cached the old value, so the delete had a real victim and returned 1. Same ending: database new, cache old, five minutes on the clock. A delete that provably removed a real key is still undone by one slow reader.

How often, honestly

The tempting thing to do here is put a percentage on the screen. It would be wrong, because there is no single rate. The rate is a ratio:

Timing every step of 5,000 runs, that predicate predicted the actual outcome 99.8–100% of the time — the mechanism is verified, not assumed. But the rate moved enormously with database commit latency:

Reader gap (row read → cache set)	With a ~3.3 ms fsync commit	With a ~0.02 ms commit
0 ms	0 / 500 (0.00%)	1 / 500 (0.20%)
2 ms	0 / 500 (0.00%)	56 / 500 (11.20%)
5 ms	31 / 500 (6.20%)	122 / 500 (24.40%)
10 ms	148 / 500 (29.60%)	264 / 500 (52.80%)
20 ms	400 / 500 (80.00%)	416 / 500 (83.20%)

Below a 3 ms gap with slow commits it is literally 0/500, because the writer physically cannot finish before the reader has already filled the cache. Make your database faster and the window opens. This is not a bug you can benchmark your way out of.

The case that actually hurts

Under continuous writes to the same row this heals itself — the next writer's delete clears the poisoned key. Measured under eight readers and one writer hammering the same row, the cache held a wrong value at 0.43% of sampled instants, and 0.48% of hits were wrong: about 68 wrong answers in six seconds, in short bursts.

The dangerous case is the opposite one. A row nobody writes again. A profile edited once. A price changed once. A feature flag flipped once. There is no next writer to heal it, so the wrong value sits there for the entire remaining TTL, answering every read.

That is the common case, and it is the expensive one.

Move 1: delete, do not update

The first instinct after seeing this is to make the write path more authoritative: I already know the new value, so let me put it straight into the cache.

# tempting, and wrong: it writes a value you computed
def change_email_bad(user_id, new_email):
    row = db.update_and_return(user_id, new_email)
    cache.set("user:" + str(user_id), row, ex=300)

# what you actually want: it writes no value at all
def change_email_ok(user_id, new_email):
    db.execute(
        "UPDATE users SET email = %s WHERE id = %s",
        (new_email, user_id))
    cache.delete("user:" + str(user_id))

Why the first one is worse: two writers can commit in the order A then B, and have their cache writes arrive in the order B then A. The database ends holding B, which is right. The cache ends holding A — a value that nobody wrote last — and it holds it for a full TTL.

Two concurrent writers, 1,500 runs each way:

Policy	Cache order ≠ commit order	Cache ≠ database at the end
write, then `SET`	590 / 1500 (39.33%)	589 / 1500 (39.27%)
write, then `DELETE`	573 / 1500 (38.20%)	0 / 1500 (0.00%)

The reorder happens just as often either way — 39% versus 38%, that is the same physics. Under SET, essentially every reorder produced a wrong cache. Under DELETE, none did, because a delete carries no value to get out of order. It says "whatever is there is now suspect", and the next reader re-reads the truth.

Two honest qualifications.

It costs you a miss. That miss is priced:

Setup	Hit	Miss (get + select + set)	Extra cost of the miss
Local database, no network	91.9 µs	176.4 µs	+84.6 µs
With a 5 ms database round trip	77.5 µs	6.91 ms	+6.84 ms, once

Seven milliseconds, one time. Compare that against what the SET policy buys you in the next section: five figures of wrong answers.

And it does not fix everything. DELETE removes the writer-versus-writer reorder completely. It does not remove the read-path race from the section above — that one is still there, and it is what versioned keys and a sane TTL are for.

Move 2: put a version in the name

The second problem is not correctness, it is bookkeeping. One user does not have one cached key. She has a profile, her settings, five pages of feed, her friends list, her badges, her unread count, her avatar. Eleven keys, in this example.

When her row changes, which of those do you delete?

You could hunt. Measured on a 200,000-key cache:

How you find them	Wall time	Round trips	Cost
All 11 in one variadic `DEL`	0.16 ms	1	you must know all 11 names
11 named `DEL`s, one at a time	1.93 ms	11	you must know all 11 names
`KEYS user:42:*`	12.4 ms	1	blocks the whole server
`SCAN MATCH user:42:* COUNT=1000`	58.0 ms	200	safe, but 200 trips
`KEYS *`	299.7 ms	1	blocks the whole server

The reason KEYS is disqualified is not its own runtime. Redis executes commands on a single thread, so while KEYS * runs, everyone else waits. Latency of an unrelated GET from a second client, measured during it:

Server state	p50	p99	max
idle	0.079 ms	0.177 ms	0.279 ms
while `KEYS *` runs	0.721 ms	246.9 ms	516.2 ms

An unrelated GET got 1,851× worse at the tail. SCAN exists precisely so you never do that.

But look again at the top row of the hunt table. If you genuinely know all eleven names, one variadic DEL does it in 0.16 ms — faster than any alternative here. So the cost of the hunt was never round trips.

The cost of the hunt is knowing the names. They are scattered across a codebase, built by string concatenation in six different modules, and the bug is always the twelfth key somebody added last month and forgot to add to the list. The list is the bug.

So stop keeping a list. Put a version in the name instead:

# the key carries a version number for that one user
def key_for(user_id, part):
    v = cache.get("user:" + str(user_id) + ":ver") or 1
    return "user:%s:v%s:%s" % (user_id, v, part)

# invalidating every derived key for one user
def invalidate_user(user_id):
    cache.incr("user:" + str(user_id) + ":ver")

# One call. Every old key is now unreachable, because
# nothing composes those names any more. You never had
# to know how many there were, or what they were called.

One INCR, 0.701 ms, one round trip. Keys a v8 reader can still reach: 0 of 11. Not because they were deleted — because nothing constructs those names any more.

The honest cost: orphans

Nothing about this is free, and the bill is memory.

After the bump, all 11 of 11 old keys are still resident, with 3,600 seconds of TTL left on them. Nothing reclaims them early. They held 16,552 bytes for one user — about 1,505 bytes per key. Scale that: 100,000 users bumping once is roughly 1.7 GB of unreachable data sitting in your cache until it expires.

That is only survivable under one of two conditions: every versioned key has a TTL, or your maxmemory-policy is an LRU/LFU variant that will evict them. Redis defaults to noeviction, which means an orphan pile eventually stops accepting writes rather than making room.

Versioning trades memory you cannot reach for invalidation you cannot get wrong. That is usually a good trade. It is not a free one.

Move 3: the timer is a backstop, not a plan

Every cache line you write has an ex=, and it is the most misunderstood argument in the file.

# the TTL is not the plan. It is the bound.
cache.set(key, row, ex=300)

# what that one argument actually promises:
#   it does NOT make the copy correct
#   it DOES cap how long a wrong copy can live
#   so pick it from what a stale answer costs you,
#   not from how much memory you happen to have

# and this line says: be wrong forever
cache.set(key, row)

A TTL has never once made a cached value correct. All it does is put a ceiling on how long a wrong one is allowed to keep answering. So price it that way. One incident, one poisoned key, fifty reads a second:

TTL	Wrong answers served	Self-heals?
5 s	251 (measured)	yes, after 5.02 s
10 s	500 (measured)	yes, after 10.00 s
30 s	1,500 (measured)	yes, after 30.00 s
5 min	~15,000 (extrapolated)	yes
1 hour	~180,000 (extrapolated)	yes
none	unbounded	no

The measured rate came out at 50.07 wrong answers per second of TTL at fifty reads a second — linear enough that the extrapolations are safe. And the rate is what makes the number mean anything: "fifteen thousand" is only true at fifty reads a second, on a row nobody writes again.

Which reframes the question you should be asking. Not "how long can I cache this for?" but "how many wrong answers is this row worth?" A stale marketing headline: thousands, fine. A stale account balance, permission check, or price: the honest answer might be zero, and that is a signal not to cache that read at all.

The last line in that snippet — cache.set(key, row) with no expiry — is a promise to be wrong forever if anything ever goes sideways. It is worth grepping for.

The limit: you cannot invalidate what you cannot name

Everything above assumes the cached value has a name derived from the thing that changed. user:42:profile obviously belongs to user 42.

Now cache a search result. The key is a hash of the query — search:results:<sha1> — and the value happens to contain twenty user records. Or cache leaderboard:top10, computed from every score in the system.

User 42 changes her name. Which keys do you delete?

There is no answer. The key name is a function of the query, not of user 42. leaderboard:top10 does not mention a user at all. Every classic move above is unavailable, because all of them start with a name.

The only way back is to write the mapping down as you fill the cache:

# when you cache a COMPUTED answer, write down which
# rows it was built from. That list is the only way
# back from a changed row to the keys holding it.
def cache_search(query_key, rows, result):
    cache.set(query_key, result, ex=300)
    for r in rows:
        cache.sadd("tag:user:" + str(r.id), query_key)

# when a row changes, read its list and delete them
def invalidate_row(user_id):
    tag = "tag:user:" + str(user_id)
    for key in cache.smembers(tag):
        cache.delete(key)
    cache.delete(tag)

Over 5,000 cached search pages, finding the 40 keys that embed user 42:

Approach	Found	Time	Round trips
Brute force: scan, fetch and parse all 5,000 values	40	40.8 ms	10 scans + 5,000 values parsed
Tag set: `SMEMBERS tag:user:42` then delete	40	0.860 ms	3

47× faster, and the gap widens as the keyspace grows, because the tag lookup does not care how many keys exist.

And here is what it costs, which is the part usually left out:

20× write amplification. Building the index for those 5,000 pages took 100,000 SADD calls — one per embedded entity, on every single cache fill.
+130% memory. 2.58 MB of cached pages became 5.93 MB. The reverse index more than doubled the workload.
Dangling members forever. After invalidating user 42, the neighbouring tag:user:43 set still listed two keys that no longer exist. Nothing cleans them. Tag sets grow without bound unless you also keep a key → tags reverse map (doubling the write amplification again) or put a TTL on the tag sets themselves.

Tags turn an impossible problem into a solvable but expensive one, and they introduce a second garbage-collection problem of their own. That is the trade. Make it deliberately, on the handful of computed keys that actually need it — not as a default for everything.

The same problem, wearing an AI hat

If you are serving an LLM application, you already have three caches and probably have not named them as caches.

The semantic cache is the unnameable-key problem, exactly. You store an answer under an embedding of the question so that a similar question can reuse it. Now a document changes, or a price changes, or a policy changes. Which cached answers were built from that fact? The key is a vector derived from the question — it says nothing about which sources produced the answer. There is no name to delete. This is the search-result case from the section above, and the way out is the same: record which retrieved chunks fed each cached answer, tag by source document, and accept the write amplification. If you cannot afford that bookkeeping, the honest fallback is a short TTL, priced by the table above — how many wrong answers is this document worth?

The RAG index is derived data with a version. Re-embedding after a chunker change, an embedding-model swap, or a bulk document update is the version bump from move 2 — a new index name, atomically switched, rather than a hunt for which vectors are stale. The orphan bill is the same too: the old index stays resident and costs money until something deletes it.

And the write-then-delete rule survives intact. Two pipelines that re-embed the same document concurrently can have their index writes land in the opposite order to their source commits, leaving the index holding a vector nobody wrote last. Invalidating the entry and letting the next read re-embed removes that class of bug for the price of one miss — and in this setting the miss is a real embedding call, so it is worth measuring rather than assuming.

The provider-side prompt cache is the one you do not control: it has its own timer and its own key derivation, so treat it as a bound you were given, not a plan you made.

The verdict

The delete ran. That was never the problem.

Invalidation fails because of ordering, not omission: a reader that read before the write and wrote after it, holding a stale value in a variable across the exact moment the delete fired. The delete was too early, not too late.

So, in order:

Delete, never update. 589 divergences in 1,500 runs the other way, zero this way. It costs one miss — about 7 ms once, against a five-figure count of wrong answers.
Put a version in the name when a row has many derived keys. Not because it is faster — a variadic DEL beats it — but because you never needed the list of names, and the list was always the bug. Budget for the orphans, and give every versioned key a TTL.
Treat the TTL as a bound, and choose it from what a wrong answer costs rather than from how much memory you have. A cache line with no ex= is a promise to be wrong forever.

And then the limit: you cannot invalidate what you cannot name. For computed answers — search results, leaderboards, semantically cached LLM responses — either write down the mapping and pay for it, or admit you are relying on a timer and price that timer honestly.

The two-hard-things joke gets repeated because it is funny. The real reason invalidation is hard is duller and more useful: a cache is a second copy of the truth, kept in step by nothing but your own code, on a clock that two independent parties are writing to at once.

References and further reading

On the pattern itself

Microsoft, Cache-Aside pattern, Azure Architecture Center — the pattern every code sample above assumes: the application, not the cache, is responsible for loading data on a miss and for removing it when the underlying data changes, with the consistency caveats stated plainly: learn.microsoft.com/azure/architecture/patterns/cache-aside

On deleting rather than updating, and on the read-path race

Rajesh Nishtala et al., Scaling Memcache at Facebook (USENIX NSDI, 2013) — the production argument for deleting cached data instead of updating it (deletes are idempotent, so concurrent invalidations cannot reorder into a wrong value), and the "stale set" race in which a reader holding an old value writes it back after the invalidation — the exact failure this article opens on, which is why the paper introduces leases to close it: usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf

On finding the keys, and why KEYS is disqualified

Redis, SCAN command reference — the cursor-based iteration guarantees, and the explicit warning that KEYS may block the server for a long time on a large keyspace, which is what the tail-latency measurement above shows in practice: redis.io/docs/latest/commands/scan

On what a TTL actually promises

Redis, EXPIRE command reference and key expiration semantics — how expiry is stored with the key, how it is cleared by a write that replaces the value, and the lazy plus active expiration model that decides when the key really goes away: redis.io/docs/latest/commands/expire

On the orphans left behind by versioned keys

Redis, Key eviction and maxmemory-policy — the available policies and the fact that the default is noeviction, which is why unreachable versioned keys are only reclaimed by their own TTL and can otherwise fill the instance until writes start failing: redis.io/docs/latest/develop/reference/eviction

On caches as derived data — the "cannot name it" limit

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), ch. 11 "Stream Processing", sections on change data capture and derived data — a cache is a derived view of a system of record, and keeping derived views correct by ad-hoc writes scattered through application code is precisely what produces the inconsistencies above; a single ordered stream of changes is the alternative framing.

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Cache Invalidation — Explained in Detail · the short version

Continuous Integration vs Continuous Delivery: The Two Halves Almost Nobody Separates

Vahid Aghajani — Sat, 29 Aug 2026 19:56:36 +0000

Originally published on software-engineer-blog.com.

Almost everybody writes these two words as one word. CI/CD. It gets said like the name of a product you install.

It is not one thing. It is two rules. They are about two different objects, and they point in opposite directions. Most teams follow the first one and stop there, and they do not notice they stopped, because the two words have been glued together for so long.

This article takes them apart.

Merge day

Start with the thing that made you look this up.

Six developers. One codebase. Six branches open at the same time. A branch is your own private copy of the code, where you work without disturbing anyone else. Yours opened on a Monday. You are building one feature, and it touches eighteen files.

While you work, the other five keep working too. Each of them lands one change on the shared trunk — the one main copy of the code that everybody shares — every day. You do not see any of it.

After one day, five changes have landed underneath you. That is fine.

After six weeks, one hundred and fifty changes have landed underneath you. Your branch has not moved. The ground it was standing on has.

Then comes merge day. Nobody put merge day in the plan. Nobody estimated it. But it is a week of work, it happens every single time, and the team has quietly decided that this is simply what software feels like.

What this article covers, and what it does not

Three things are covered:

What continuous integration actually is, and why it is a rule about people rather than a piece of software.
What continuous delivery actually is, and why it is a rule about one object, called the artifact.
The third word, deployment, which is not the same as delivery, and which is where most of the arguments come from.

Three things are not covered. No tool is named anywhere in this article. Not one. How a release is rolled out to users is a different subject, covered elsewhere on this site. And this is not advice about how to write tests.

The workshop

The nearest confusion is this one. People think CI/CD is a single thing. A file in the repository. A list of steps that a machine runs. So they install it, and then they say they have it.

Picture a workshop instead.

On the left there is one bench, with one wooden frame being built on it. Everybody who makes a piece brings it to that bench today, and fits it into the frame today. If two pieces do not fit each other, you find that out today, while each piece is still one cut away from fitting.

On the right there is a loading dock. There is one crate. It is packed, sealed, stamped, and standing at the door. A lorry is waiting outside. Whether the crate leaves is a separate decision, made by a person.

The bench is continuous integration. The crate is continuous delivery. Two different rules, about two different objects.

One instant, two halves

Here is the whole article in one picture.

Draw the life of one change as a line of time. Mark one instant on that line. That instant is the green build: the moment your change has joined everybody else's work, the machine has built it, the tests have run, and everything passed.

Everything to the left of that instant is continuous integration. It looks backwards, at the merge. Its question is: how old is the oldest piece of work that has not joined the trunk yet? A good answer is measured in hours.

Everything to the right of that instant is continuous delivery. It looks forwards, at the artifact. Its question is: how much stands between this build and a real user? A good answer is one button.

Two halves. One instant between them. Pointing in opposite directions. Every section below is this same picture with the marks moved.

What a long branch actually costs

The cost of a long branch is not the cost most people assume. It is worth making concrete, because the arithmetic says something surprising.

Here is a small model of the team.

# drift.py -- what a branch costs is not its size. It is its age.
TEAM          = 6      # developers on this one codebase
REPO_FILES    = 1200   # files that anybody might touch
FILES_I_TOUCH = 18     # what my one branch actually edits

def drift(days_open):
    # every OTHER developer lands one change a day while my branch sits still
    return (TEAM - 1) * days_open

def chance_of_a_conflict(days_open):
    miss = (REPO_FILES - FILES_I_TOUCH) / REPO_FILES   # one edit misses me
    return 1 - miss ** (2 * drift(days_open))          # each change edits 2 files

print(drift(1),  chance_of_a_conflict(1))     # 5 changes    -> 14 percent
print(drift(30), chance_of_a_conflict(30))    # 150 changes  -> 99 percent

Drift is the number of changes that landed on the shared trunk while my branch sat still. Five other developers, one change each per day, so drift is five changes per day.

The chance of a conflict is the chance that at least one of those landed changes touched at least one of my eighteen files. Each landed change edits two files, so I take the chance of one edit missing me, and raise it to that power.

Run it. Every figure below came out of that script, and each one was cross-checked with a 200,000-run simulation that shuffles the edits at random:

How long the branch stayed open	Changes that landed underneath it	Chance of a conflict
1 working day	5	14.0% (simulated: 14.1%)
1 working week	25	53.0%
6 working weeks	150	98.9% (simulated: 98.9%)

Read the first and last rows next to each other. A branch that is one working day old has a 14% chance of hitting a conflict when it merges. Most days, nothing happens. You merge, and you carry on with your afternoon.

A branch that is six weeks old has a 98.9% chance. Not most of the time. Essentially always.

And here is the honest third number. Nothing about the work changed between those two rows. The same eighteen files. The same feature. The same developer. The only thing that changed is that the branch was allowed to live thirty times longer.

Look at the script again and notice what is missing. FILES_I_TOUCH is fixed at eighteen in both runs. The size of the branch never appears in the calculation. Only its age. A branch is a debt, and the interest on it is time.

That is what continuous integration attacks. Not your tests, and not your tooling. The age of the oldest thing that has not been joined yet.

Continuous integration, written out

So here is continuous integration in full. Notice that nothing in it is a product you can buy.

# merge.py -- continuous integration, with nothing bought and nothing installed.

def integrate(my_change):
    trunk  = fetch("trunk")              # 1. what everyone else already landed
    joined = merge(trunk, my_change)     # 2. join my work to it -- TODAY

    if not builds(joined):               # 3. does the whole thing still compile?
        return stop("the build broke")

    if not tests_pass(joined):           # 4. does it still do what it did?
        return stop("the tests broke")

    return push("trunk", joined)         # 5. now everyone builds on my work too

Step by step. Take what everyone else has already landed on the trunk. Join my work to it — not next month, today. Build it: does the whole thing still compile, with my change inside it? If it does not, stop, and say so out loud. Run the tests: does the whole thing still do what it did before? If it does not, stop, and say so out loud.

And if both of those passed, push it back to the trunk. From this moment, everybody else is building on top of my work as well.

That is all of it. Really it is two words. Every change, and today.

A machine that runs your tests but lets branches live for six weeks is not doing this. It is just running your tests.

The part that gets skipped: CI is a rule about people

Let us say this plainly, because it is the part that gets skipped.

Continuous integration is a rule about how people work. The machine is only there to enforce it.

Which means a team can have every green tick in the world and still not be doing it. If the tests run on every proposed change, but those proposals stay open for three weeks, then the joining still happens once every three weeks. The machine is measuring something. It is not fixing anything.

A green tick on a three-week-old branch proves that the branch was fine three weeks ago, against a trunk that no longer exists.

And it is cheap to install, but expensive to keep. Installing it takes an afternoon. Keeping it means the test suite has to stay fast, and it has to stay honest.

The moment people stop believing a red result, you have made things worse than having no tests at all.

The arithmetic of a suite nobody believes

That last sentence sounds like an opinion. It is arithmetic.

A flaky test is one that sometimes fails even though the code is perfectly fine. Say you have four hundred tests. Say each one, on its own, fails wrongly about two times in every thousand runs.

Two in a thousand sounds like nothing at all. But a build is only green when all four hundred pass. Multiply it out:

Wrong failures per test, per 1,000 runs	Chance a perfectly good build shows red	In other words
0.5	18.1%	1 good build in every 5.5
1	33.0%	1 good build in every 3
2	55.1%	1 good build in every 1.8

At two in a thousand, more than half of your perfectly good builds show red. So more than half the time, red means nothing.

And then the number that actually costs you: zero. That is how many people will look carefully at a red build, once they have learned that red usually means nothing. They press the button that runs it again.

Now a real failure and a fake failure look exactly the same. Your suite has stopped being a signal. It is still running, it is still costing machine time, and it is no longer telling anybody anything.

This is why "we installed it" is not the same as "we have it". Continuous integration is a habit that a machine happens to enforce, and habits decay.

Crossing the line: the artifact

Now cross to the right hand side of that instant, and follow one green build all the way out.

Here is the green build. The change is on the trunk, it compiled, the tests passed. Continuous integration has done its whole job, and it stops right here.

Continuous delivery starts here. The first thing it does is turn that build into one object. One package, with one identity, built exactly once. That object is called the artifact, and from this moment it is the only thing that moves.

The same artifact is then pushed into real environments. Not a rebuild. The same bytes, with different configuration — addresses, secrets, sizes — handed to it from outside.

And at the end there is production, with one step in front of it. In continuous delivery, that step is a button, and a person presses it.

Build once, promote the same bytes

Here is what that means in code. This is the rule that carries the whole idea.

# build.py -- the artifact is built ONCE. Only the configuration changes.

def build_artifact(commit):
    # one build, one identity. This object is the only thing that ever ships.
    return package(commit, digest=sha256(commit))

def promote(artifact, environment):
    config = load_config(environment)    # address, secret, size -- from OUTSIDE
    return deploy(artifact, config)      # the same bytes. Nothing is rebuilt.

art = build_artifact("a91f4c")
for env in ("staging", "pre-production", "production"):
    promote(art, env)                    # 1 build, 3 deploys, identical bytes

# Rebuild once per environment instead and you get 3 builds -- which is
# 2 chances for what reaches production to differ from what you tested.

Build the artifact from one commit. Package it, and give it an identity — a digest, which is a short fingerprint of the contents, so you can point at exactly this thing and no other.

Promoting is a separate step, and this is the important line. Promoting takes the artifact and an environment. It loads that environment's configuration from outside the artifact. Then it deploys the artifact, unchanged.

So one build goes to staging, then to pre-production, then to production. Three deployments, one set of bytes. The thing your users get is the exact thing your tests ran against.

Now do it the other way. Build again for each environment. Three builds, which means two chances for what reaches production to be quietly different from what you tested. Nobody chooses that on purpose. It happens because rebuilding is the easy thing to write.

What continuous delivery demands

Continuous delivery is not free, and the price is not money. It is a list of things that have to already be true.

Required:

One artifact for every green build, and nothing else ever ships.
Configuration comes from outside that artifact, always.
Database changes that work with both the old and the new version of the code, because for a few minutes both of them are running.
A rollback you have actually used, on a real day. Not a paragraph in a document.

And here is what has to go:

Servers that somebody edits by hand.
A separate build for each environment.
A long-lived release branch.
A freeze week before every release.
A page of manual steps written down in a document.

Without that list, continuous delivery is only a faster way to ship the same bug.

One boundary, so you know what this is not

There is a whole subject sitting next to this one, and it is worth naming so you can see the edge.

How a release is rolled out is that other subject, and it is covered elsewhere on this site: two full production stacks with the traffic flipped between them, a small share of users getting the new version first, and requests already in flight being allowed to finish before the old version stops.

Shipping code switched off and turning it on later is a third subject, also covered elsewhere, and it is not the same as either half here.

All of them are real, and all of them matter. This article is about something before all of them: how often work is joined together, whether every green build becomes one artifact, and how far that artifact travels on its own.

The pipeline, and the artifact. Not the rollout. This article stops at the artifact.

The third D

And now the third word, which is where almost every argument about CI/CD actually comes from.

Continuous delivery means every green build is ready to go to production. It has been built once, it has been through the environments, and it is sitting at the door. A person decides when it goes. The team can ship ten times a day if they want to, and they can also decide not to ship anything for a week. Ready is the promise. Released is a choice.

Continuous deployment removes the person. Green means it goes. Nobody presses anything.

Both of them are written CD, which is exactly why people argue past each other. One person is defending a discipline, the other is defending an automation, and they both think the letters mean the same thing.

They are also not the same difficulty at all. Delivery needs discipline. Deployment needs everything delivery needs, plus a way to notice a bad release and undo it without a human — because there is no longer a human in the loop to notice.

The difference is often one line

Inside a real pipeline, the difference between those two is often exactly one setting.

# promote.py -- continuous DELIVERY and continuous DEPLOYMENT, in one file.

REQUIRE_A_HUMAN = True      # True: delivery.   False: deployment.

def on_green_build(artifact):
    promote(artifact, "staging")
    if not smoke_tests("staging"):
        return stop("staging is unhappy -- nothing goes anywhere")

    if REQUIRE_A_HUMAN:
        return wait_for_the_button(artifact)    # DELIVERY: ready, not released
    return promote(artifact, "production")      # DEPLOYMENT: it ships itself

At the top there is one setting: require a human, true or false.

On a green build, promote the artifact to staging, and run the quick checks there. If staging is unhappy, stop, and nothing goes anywhere.

And now the branch. If require-a-human is true, you wait for the button. The artifact is finished, tested, and standing still. That is continuous delivery: ready, but not released.

If require-a-human is false, the same artifact goes to production right there, on the same green build, with nobody watching. That is continuous deployment.

Continuous delivery ends at a button that nobody is obliged to press. Continuous deployment deletes the button.

Note what did not change between the two paths: the artifact, the promotion, the checks. Everything above that setting is the same work. That is why you cannot skip delivery on the way to deployment.

Why the size of a release matters, on a bad day

There is one more reason to care, and it only shows up when something goes wrong.

Suppose your six developers finish one change each per day, and you release once a quarter. That release contains 378 changes. Then something breaks in production. You now have 378 suspects. Finding the one, by repeatedly cutting the list in half, takes about 9 rounds.

Now release once a day instead. That release contains 6 changes. Six suspects. About 3 rounds.

Release rhythm	Changes in one release	Rounds of halving to find the culprit
Once a quarter	378	~9
Once a day	6	~3
Difference	63×	3×

The same team. The same code. The same bugs. The only thing that changed is how much work was allowed to pile up between one release and the next.

This is not just arithmetic on a page, either. The multi-year Accelerate research found the same direction in real organisations: teams that release in small batches, often, also recover faster and fail less. Small batches are not a nicety on top of the pipeline. They are most of what the pipeline buys you.

The same two halves, when part of what you ship is a model

If your system includes a trained model, both halves still apply, and one of them gets harder.

The left half does not change at all. Prompts, retrieval settings, evaluation scripts and serving code are all just code on a trunk, and a branch holding them is exactly as expensive with age as any other branch.

The right half is where people quietly break the rule. The artifact stops being one object. The code is packaged once, but the model weights are fetched at start-up from wherever the latest ones happen to live, and the prompt text is edited in a console by hand. Now three things ship on three different schedules, and none of them has one identity. When output quality drops on a Tuesday, you cannot say what changed, because "what changed" was never a single thing you could point at.

The fix is the same sentence as before, applied honestly: one artifact per green build. Pin the weights by digest, put the prompt in the package, and let configuration — endpoints, sizes, keys — come from outside. Then a bad Tuesday has one identity attached to it, and rolling back means going back to one object rather than reconstructing three.

Three questions about your own team

You can answer these this afternoon.

First. How old is the oldest piece of work that has not been joined to the trunk yet? If you answer in weeks, you do not have continuous integration, no matter what runs on your proposed changes.

Second. Does every green build produce exactly one artifact, which is never rebuilt on its way out? If not, you do not have continuous delivery yet — because there is nothing being delivered.

Third. What is the last step before production? If it is a button, that is delivery. If there is no step at all, that is deployment. If it is a person following instructions written in a document, then it is neither of them.

The verdict

	Continuous integration	Continuous delivery
It is a rule about	the merge	the artifact
It looks	backwards, at joining work	forwards, at shipping it
Its question	How old is the oldest unjoined work?	How much stands between this build and a user?
A good answer	hours	one button
What it costs	cheap to install, expensive to keep honest	discipline: config outside, two-way migrations, a used rollback
How it fails quietly	green ticks on three-week-old branches	a rebuild per environment

Continuous integration is a rule about the merge. Everybody's work joins the same trunk today, and every join is built and tested, so a break is found while the change is still small enough to understand.

Continuous delivery is a rule about the artifact. Every green build becomes one package, built once, that travels the environments unchanged, until it is standing one press away from production.

And the D that everybody argues about is a third thing. Delivery means always ready. Deployment means the person is gone as well.

You can have the first without the second, and almost everybody does.

References and further reading

On branch age and merge day

Paul Hammant and contributors, Trunk Based Development (trunkbaseddevelopment.com) — the short-lived-branch rule and why branch lifetime, not branch size, is the thing to manage: trunkbaseddevelopment.com

On continuous integration as a rule about people

Kent Beck, Extreme Programming Explained: Embrace Change (Addison-Wesley, 1999; 2nd ed. 2004) — the origin of "integrate continuously" as a team practice rather than a piece of infrastructure.
Martin Fowler, Continuous Integration (martinfowler.com, 2006) — the canonical definition, including "everyone commits to mainline every day": martinfowler.com/articles/continuousIntegration.html

On a suite people stop believing

Martin Fowler, Eradicating Non-Determinism in Tests (martinfowler.com, 2011) — why a suite that fails at random destroys the trust that makes it worth running at all: martinfowler.com/articles/nonDeterminism.html

On the artifact, and build-once-promote

Jez Humble and David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley, 2010) — the deployment pipeline, building binaries exactly once, and keeping configuration outside the artifact.

On the third D: delivery versus deployment

Martin Fowler, ContinuousDelivery and Continuous Delivery vs Continuous Deployment (martinfowler.com) — the ready-versus-released distinction stated directly: martinfowler.com/bliki/ContinuousDelivery.html

On batch size and release rhythm

Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (IT Revolution, 2018) — the measured link between small batches, deployment frequency, lead time and recovery.

If a reference you would expect is missing, say so in the comments and I will add it.

Watch the full episode: Continuous Integration vs Continuous Delivery on YouTube

Service Boundaries: Where Does the Line Between Two Services Actually Go?

Vahid Aghajani — Sat, 29 Aug 2026 07:10:11 +0000

Originally published on software-engineer-blog.com.

You have decided to split. That part is not in question here — this article takes it as given and asks only the next question, the one that actually costs money: where does the line go?

The split almost everyone draws first is by noun — one service per thing the business talks about. Order, user, product, pricing. Or by the org chart — one service per team. Both are visible on day one, on a whiteboard, before anyone has written a line of code. That is exactly why they get drawn, and it is also why neither of them is a boundary.

Six months later you have four services, and one page load needs all four.

The tell: a chain of synchronous calls

Here is the shape you are looking for. order_svc owns the orders table and nothing else on the page. So it asks:

# order_svc - GET /orders/{oid}
# it owns the orders table. it owns nothing else on this page.

@app.get("/orders/{oid}")
def get_order(oid: int):
    row = db.execute("SELECT * FROM orders WHERE id=?", [oid]).fetchone()

    # the customer name is not ours. ask the user service, and wait.
    user = httpx.get(f"{USER_SVC}/users/{row[1]}").json()

    # the title is not ours either - and this one runs PER LINE ITEM.
    for it in line_items(row):
        p = httpx.get(f"{PRODUCT_SVC}/products/{it[0]}").json()
        # ...and product_svc, to answer that, asks pricing_svc itself.

    return {"order_id": oid, "customer": user["full_name"],
            "items": priced, "total_cents": total}

Read that loop again. The fan-out is not a fixed number — it grows with the data on the page. product_svc is asked once per line item, and each of those asks pricing_svc. On the rig used for the episode, a one-line-item order produced 3 outbound calls; the two-line-item demo order produced 5. The formula is 1 + 2 × line_items.

The first version of this argument said "three services, three calls" and drew a single product fetch. Both were wrong. Any article that says three services means three calls is counting boxes on a diagram, not counting requests.

And 5 and 6 are both correct numbers that mean different things. The X-Hops header reports 5 outbound calls. The rig's total inbound count is 6, because the page load into order_svc is itself a request that can fail. Which one you use matters a great deal in a moment.

That is not four services. It is one distributed monolith: the latency of four, the failure modes of four, and the independence of none.

What the chain actually costs

Every figure below was measured on a real four-service rig, 2,000 requests per build, and the naive, collapsed, and modular builds return byte-for-byte identical responses (diff exit 0). Same feature, three architectures.

What was measured	Result
Naive chain (4 processes, HTTP) — mean / p95	10.12 ms / 12.21 ms
Cost of each extra network hop	~1.80 ms
Collapsed (1 process, 1 database) — mean / p95	1.11 ms / 1.18 ms
Speed-up, identical response bytes	9.09× (9.01 ms/req saved)
Actual work — every DB read, all four services	140 µs = 1.38% of the request
`order_svc` blocked on a socket	87.96% of the request
Outbound calls — 1-item order / 2-item order	3 / 5
Boundary in-process vs no boundary	+30 µs (+2.7%)
Boundary over the network vs no boundary	+9.01 ms (+809%)

Two lines in that table are the whole argument. The real work — every database read in all four services — is 1.38% of the request. order_svc spends 88% of its life blocked on a socket waiting for a service to answer a question it could have answered itself.

And a boundary is not intrinsically expensive. The same wall, drawn inside one process, costs 30 microseconds. Drawn across a network, it costs 9 milliseconds — three hundred times more, for the same separation of concerns. You are not paying for the boundary. You are paying for the transport you chose to put it on.

The availability bill: right rule, wrong exponent

Everyone knows the multiplication rule. Four services at 99.9% each, in a chain where all must succeed:

0.999⁴ = 99.6006% → 172.5 minutes of downtime per month

That figure is arithmetically correct — a 1,000,000-trial Monte Carlo matched it. It is also the wrong number, because the read path does not involve four things that must succeed. It involves six: the request into order_svc, plus the five calls it sets off.

0.999⁶ = 99.4015% → 258.6 minutes per month

That is an 86-minute-a-month gap, and it is hidden by counting boxes on a diagram instead of counting what has to answer. Availability is raised to the number of things that must succeed, not the number of boxes you drew.

One honest note, because the temptation to overclaim here is strong: the measured rig matched theory within one standard error once a rig artefact was removed (a sqlite3 connection shared across the threadpool produced phantom 404s and 500s — 36% of all observed failures, all of them pushing the rate down). Measurement did not beat the theory. The interesting result is the exponent, not a discrepancy.

The deploy story is worse than you have been told

Every article on this subject says: deploy them in the right order. Measured on a one-field rename across the boundary — name → full_name in user_svc, which order_svc reads — both orders fail:

Deploy user_svc first → order_svc raises KeyError: 'name' → HTTP 500
Deploy order_svc first → KeyError: 'full_name' → HTTP 500

There is no ordering that works, because the two versions are never simultaneously correct. The only thing that works is a three-deploy expand/contract sequence: emit both fields, migrate the reader, then drop the old field. The seeded history on the rig shows exactly that shape — a commit reading rename name -> full_name (expand: emit both) and, later, (contract: drop name).

So the real cost of that boundary is not "coordination". It is three releases to rename one field.

So what actually decides a boundary?

Two things, and neither of them is a noun.

1. Who owns the data

Exactly one service may write a given fact. Everyone else asks. The interesting move is not making the service smaller — it is deciding, per fact, whose fact it is.

# the fix is not a smaller service. it is deciding who owns each fact.

# WRONG - the order page asks the product service what the price is.
#         that is the price TODAY. the customer paid LAST MONTH.
price = httpx.get(f"{PRODUCT_SVC}/products/{pid}").json()["price"]

# RIGHT - the price charged is not a product fact at all.
#         it is a fact about THIS order, and it never changes again.
db.execute(
    "INSERT INTO orders (user_id, product_id, price_paid, title_at_sale)"
    " VALUES (?, ?, ?, ?)",
    [user_id, pid, price_now, title_now],
)

# reading that order back now touches one table, in one process.
# the call to pricing did not get faster. it stopped existing.

That last comment is the point. The price the customer paid is a fact about the order, not about the product. Once you see that, the call to pricing_svc on the read path is not slow — it is wrong, and it does not need optimising, it needs deleting. A correct ownership decision does not make a call faster. It removes the call.

This is also, incidentally, a correctness fix and not only a performance one: asking the product service for "the price" on an order page returns the price today, for an order placed last month.

2. What changes together

Two things that are always released in the same commit are one thing wearing two names. You do not have to guess at this — your version control has been recording the answer for years.

$ git log --name-only --pretty=format:%H | python3 cochange.py

  pair                            commits touching BOTH      share
  order_svc   + user_svc                    6               75.0%
  product_svc + pricing_svc                 1               25.0%
  order_svc   + product_svc                 0                0.0%
  order_svc   + pricing_svc                 0                0.0%

# 75%, and every user_svc commit touched order_svc too: one unit of change.
# 0% does NOT automatically mean the wall is earning its keep -
#   check whether that pair is on the READ PATH before you trust it.
#   both zero pairs here are on it, and they were resolved differently.

order_svc and user_svc co-change 75% of the time, and every single user_svc commit also touched order_svc. That is not two services. That is one unit of change with a network cable running through the middle of it.

The two mechanisms disagree, and you should say so

Look at the table again. Co-change says nothing at all about order_svc + product_svc — 0%. So the co-change test does not argue for merging order and product; the read-path fan-out does. Two mechanisms, pointing different ways, on the same pair.

Most writing on this topic presents the rules as if they always agree. They do not. When yours disagree, name which one you are following and why. Here: order and product get merged on fan-out grounds, and order and pricing stay apart despite an identical 0% because pricing comes off the read path entirely once price_paid is owned by the order.

The line, moved

Merge order + user + product into one deployable — three modules, one process, one database. Keep pricing a real service: different team, different release day, and crucially it is not on the read path at all any more, because the order stores price_paid and title_at_sale at the time of sale. Its only remaining edge is a one-way, nightly, asynchronous one.

Losing the network does not mean losing the wall:

# ONE process. ONE deployable. the boundary is still real.

# shop/users/api.py - the only way in. no table name ever leaves this file.
def get_user(uid: int) -> User:
    return _row_to_user(_db.execute(
        "SELECT id, name FROM users WHERE id=?", [uid]).fetchone())

# shop/orders/service.py
from shop.users.api import get_user       # a function, not a URL
def order_page(oid: int):
    o = _orders.by_id(oid)                # our own table
    return {"customer": get_user(o.user_id).name,
            "price": o.price_paid}        # our own column

# and the wall is enforced, not merely agreed:
#   a sqlite authorizer refuses any statement from shop.orders that
#   names the users table. a violation raises. it does not lint.

The last three lines are what separates this from wishful thinking. A module boundary that lives in a code-review convention is not a boundary; it is a hope. A SQLite authorizer that raises when shop.orders names the users table is a boundary, and it costs 30 microseconds.

Three architectures, one feature

	Four services (by noun)	One deployable, no wall	One deployable, modules
Mean latency	10.12 ms	1.11 ms	1.14 ms
Cost of the boundary	+9.01 ms (+809%)	—	+30 µs (+2.7%)
Things that must succeed	6	1	1
Availability at 99.9% each	99.4015%	99.9%	99.9%
Rename one field across the line	3 deploys (expand/contract)	1 commit	1 commit
Wall enforced by	the network	nothing	an authorizer that raises
Moving the line later	data migration	refactor	refactor

The costs, stated plainly

This is not free, and the honest version says so. You end up with fewer and larger services than the diagram wanted, which will feel like a regression to anyone who counts services. Moving a line later is a data migration, not a refactor — merging two databases is real work. And a big in-process module still needs someone to defend the wall, because the compiler will happily let you reach across it.

Which is exactly why the reversible move is to draw the line inside one deployable first: one module, one function as the only way in, and the wall enforced mechanically. If the line turns out to be in the wrong place, you move it with a refactor instead of a migration. If it turns out to be right, promoting it to a process later is a small, well-understood step — and you will have real co-change data by then to prove it.

Keeping two databases in step is its own problem, with its own name and its own article. Do not sign up for it by accident on a whiteboard.

The same two tests, on an LLM-serving stack

None of this is specific to shopping carts, and the AI stack is where the noun-split is currently being repeated most enthusiastically. A typical serving diagram has a gateway, an orchestrator, a retriever, a reranker, a model server, and an eval service — six boxes, one per noun, drawn on day one.

Run the two tests on it.

What changes together? In practice the prompt template, the few-shot examples, and the output parser change in the same commit, essentially always. Change the prompt and the parser breaks; tighten the schema and the prompt has to say so. That is a co-change rate near 100% — one unit of change. Split it into a "prompt service" and a "parsing service" and you have bought yourself three deploys to add a field, for exactly the reason name → full_name needed three.

The chunker and the embedder are the same story from the other side: change the embedding model and every vector in the index is invalid. They ship together or the index is quietly wrong.

Who owns the data? The vector index has exactly one legitimate writer — whichever service owns the ingest path. Everyone else reads. And the price_paid lesson transfers directly: when you log a trace, store the prompt text, the model version, and the retrieved chunk ids as they were at call time, on the trace row. They are facts about that call, not about the retriever's current state. A trace that has to ask the retrieval service what it would return today cannot be replayed, and your eval numbers will drift underneath you the same way a re-fetched price does.

What about the fan-out? A request that goes gateway → orchestrator → retriever → reranker → model server synchronously has the same six-exponent availability bill as the shop did, and every hop lands directly on time-to-first-token — the number your users actually feel. The 88%-blocked-on-a-socket figure is, if anything, kinder than what you will measure here, because a rerank hop is not 1.8 ms.

And which box genuinely earns its own process? The model server. Not because it is a different noun, but because it fails all three of the reasons the others failed: it scales on a completely different axis (GPU memory, batch size, KV cache), it releases on a different cadence, and it does not co-change with the prompt at all. That is what an earned boundary looks like — different data owner, different unit of change, different scaling axis. Not a different word on a whiteboard.

The verdict

A boundary you have to call synchronously on every request is not a boundary. It is a network hop you added to a function call.

The point of the line was never tidiness or a tidy diagram. It was independent deployability — can these two things ship apart, on different days, by different people, with nobody coordinating? If the answer is no, you do not have two services. You have one system, and the line you drew is a cost you are paying for nothing.

So before the next split, ask the two questions that do not appear on the whiteboard: who owns this fact, and what changes with it. Then draw the line inside one deployable, enforce it with something that raises rather than something that comments, and let it earn its process later.

References and further reading

The premise — you have decided to split, and now the line has to go somewhere

Sam Newman, Building Microservices (O'Reilly, 2nd ed. 2021) — independent deployability as the defining property of a service, and the case that a boundary you cannot ship apart is not one.
Martin Fowler, MicroservicePremium (martinfowler.com, 2015) — names the fixed cost every distributed boundary charges before it delivers any benefit, which is the ledger this whole article is keeping.

The tell — a synchronous chain across four services

Martin Fowler, First Law of Distributed Object Design (martinfowler.com, 2003) — "don't distribute your objects": the original statement that a remote call is not a local call with extra latency, it is a different thing.
Michael T. Nygard, Release It! (Pragmatic Bookshelf, 2nd ed. 2018) — integration points as the leading source of instability, and how a synchronous dependency turns one service's bad day into everyone's.

The availability arithmetic

Betsy Beyer et al., eds., Site Reliability Engineering (O'Reilly, 2016), ch. "Embracing Risk" — availability targets, error budgets, and why a service's own target has to account for everything it depends on to answer.

Decider 1 — who owns the data

Chris Richardson, Pattern: Database per Service (microservices.io) — one writer per fact, stated as a pattern with its trade-offs, including the queries it makes hard.
Eric Evans, Domain-Driven Design (Addison-Wesley, 2003) — bounded context: the same word means different things to different parts of the business, which is why "price" belongs to the order and not to the product.

Decider 2 — what changes together

David L. Parnas, On the Criteria To Be Used in Decomposing Systems into Modules (Communications of the ACM, 1972) — decompose by what is likely to change, not by processing steps; the ancestor of the co-change test.
Adam Tornhill, Your Code as a Crime Scene (Pragmatic Bookshelf, 2015) — mining version-control history for change coupling, i.e. how to get the git log table above out of your own repository.
Melvin E. Conway, How Do Committees Invent? (Datamation, 1968) — the org-chart split, and why it keeps getting drawn whether or not it is the right line.

The three-deploy rename

Danilo Sato, ParallelChange (martinfowler.com, 2014) — expand / migrate / contract, the only sequence that survives a rename across a boundary when neither deploy order works.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: the 2-minute version draws the four-service shop and names the two deciders; the full episode runs the ownership fix, the co-change analysis, and the in-process module wall on a rig that really executes.

Validating Input at the Edge: Why Four Checks in Four Places Are Worse Than One Border

Vahid Aghajani — Fri, 28 Aug 2026 14:10:08 +0000

Originally published on software-engineer-blog.com.

Somebody signs up. Their browser sends one JSON body to your endpoint. And here is the thing almost every tutorial skips over:

Your program does not receive an object. It receives bytes.

raw = await request.body()
# type(raw) -> <class 'bytes'>
# repr(raw) -> b'{"email":"ada@example.com","age":"42","role":"admn"}'

doc = json.loads(raw)
# type(doc) -> <class 'dict'>
# doc -> {'email': 'ada@example.com', 'age': '42', 'role': 'admn'}

json.loads upgrades bytes to a dict. That is a real upgrade, and it is also the last one you get for free. A dict carries zero guarantees. age is still the string '42', not a number. role is still the typo 'admn'. Nothing between the socket and your handler has an opinion about whether any of this makes sense.

So the question is not whether something validates that data. Something always does. The question is how many things, and where.

The naive version: four places, four rules

Here is what happens by default, and it happens because every single step of it is reasonable in isolation. The request reaches your handler. The handler is careful, so it checks the email is not empty. It passes the dictionary on to a service function — which does not trust its caller, so it checks for an at-sign. The service calls the database layer, which checks the length before the write. And the template that renders the confirmation trims the whitespace, because whitespace looks bad.

# handler
if not body.get("email"):
    return {"detail": "invalid input"}, 400

# service
if "@" not in email:
    raise ValueError

# db layer
if len(email) > 320:
    raise ValueError

# template
email = email.strip()

Four places. Four slightly different rules for one field. Nobody wrote it that way on purpose — it accreted, one careful commit at a time.

And nothing in the system knows the four rules disagree. The day somebody loosens one of them, the other three say nothing.

What that actually costs

I ran it. Not as a thought experiment — as two real FastAPI apps with a SQLite database behind them, and a set of measurement scripts.

First result: a silent split-brain, on a request that succeeded. POST an email with spaces around it — " ada@example.com ":

naive status  : 200
naive response: {"email":"ada@example.com", ...}
naive DB row  : {'email': '  ada@example.com  ', ...}
response email == stored email ? False

HTTP 200. Nothing failed. Nothing was logged. And the row saved in the database is not the value returned to the user, because layer three and layer four disagree about what an email is. Two truths about the same user, created in the same request.

Second result: the crashes. Four bad bodies at the naive server:

{'emial': ..., 'age': '42'}          -> 400  {"detail":"invalid input"}
{'email': ..., 'age': '42'}          -> 500  TypeError: unsupported operand type(s) for //: 'str' and 'int'
{'email': ..., (no age)}             -> 500  KeyError: 'age'
{'email': ..., 'age': None}          -> 500  TypeError: ... 'NoneType' and 'int'

Three of the four returned a 500. And the fourth — the one with the typo'd field name — returned a 400 that says invalid input and names nothing, so the client has no idea which of the three fields it got wrong.

Worth being precise here, because it is tempting to tell this story with the typo body crashing: it does not. The handler's truthiness check fires first and short-circuits into the 400. The 500 needs a valid email plus a bad age. That is the honest version, and it is the stronger one — the naive app either falls over or answers uselessly, and which of the two you get depends on the order in which the checks happen to run.

Third result: the enum nobody enforced. LEGAL_ROLES is declared at the top of the naive app and never used. So:

POST {"email": "ada@example.com", "age": 42, "role": "admn"}
status: 200
rows now in the DB: [{'email': 'ada@example.com', 'role': 'admn', ...}]

HTTP 200, and 'admn' is now a row in your database. Forever. That constant sitting unused at the top of the file is exactly how it happens for real.

The mechanism: a border that converts

The fix is one move, and the important part is that it is not a check.

Validate once, where the data arrives, and nowhere after that. One model, at the door:

class SignupIn(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")

    email: EmailStr = Field(max_length=254)
    age: int = Field(ge=13, le=120)
    role: Literal["admin", "editor", "viewer", "billing", "support"]


@app.post("/signup")
def signup(body: SignupIn):      # not Request. SignupIn.
    ...

Here is the part people miss, and it is the whole idea:

The border does not hand you back True or False. It hands you back a typed value — an object that could not have been constructed if the data were wrong. If the body cannot become a SignupIn, the function body never runs at all.

That is the difference between a check and a border. A check leaves you holding the same untrusted dictionary you had before, plus a boolean you now have to remember to respect. A border changes what you are holding. After it, body.age is an int because there is no reachable universe in which it is not.

And the consequence shows up as a measurement. Counting defensive lines in the two apps with grep -cE 'is None|if not ':

	Naive app	Bordered app
Lines of code	73	37
Defensive `is None` / `if not` checks	13	0
Validation sites for `email`	6	1
5xx from the four bad bodies	3 of 4	0
Problems named per round trip	1	3

Thirteen defensive checks became zero. Not because anyone was braver about skipping them — because after the border there is nothing left for them to catch.

That gives you the diagnostic that makes this idea worth internalising: every if x is None deep in your business logic is evidence that the border leaked. It is not defensive programming, it is a bug report about your boundary.

The four rules the border follows

1. Say what you accept, not what you reject

An allow-list, not a block-list. A block-list is a list of the attacks and mistakes you already thought of, which by construction excludes the ones that will actually reach you. Literal["admin", "editor", ...] enumerates the five legal roles; everything else is not a role, and you never have to enumerate the infinite set of things that are not roles.

2. Reject, do not coerce

This is the rule most codebases get backwards, because the default is convenient. Same input, one config flag apart:

input: {'age': '42'}   type(input['age']): <class 'str'>

LAX    -> .age = 42, type <class 'int'>
STRICT -> ValidationError:
          Input should be a valid integer [type=int_type, input_value='42', input_type=str]

Lax mode silently turns the text "42" into the number 42. That feels helpful right up until the client that sent a string was sending a string because of a bug, and you have just laundered the bug into your database as a plausible-looking integer. Rejecting is what lets you find out.

One precision point the video states loosely and the code makes exact: strict=True controls type coercion, not value normalisation. A field type can still transform the value it accepts:

strict=True, EmailStr:
  '  ada@example.com  ' -> 'ada@example.com'
  'Ada@EXAMPLE.COM'     -> 'Ada@example.com'   # domain lowercased, local part not

Strict means it will not accept a str where you declared an int. It is not a promise that the bytes pass through untouched. And that normalisation is a feature here, not a leak: it is why the bordered app returned and stored the same trimmed address, while the naive app stored the padded one. The border normalised once, in one place.

3. Name the field that failed

Compare the two answers to one bad body, {"email": "not-an-email", "age": 7, "role": "wizard"}:

NAIVE    -> 400 {"detail":"invalid input"}

BORDERED -> 422
{"detail": [
  {"type": "value_error",        "loc": ["body","email"], "msg": "value is not a valid email address: An email address must have an @-sign."},
  {"type": "greater_than_equal", "loc": ["body","age"],   "msg": "Input should be greater than or equal to 13"},
  {"type": "literal_error",      "loc": ["body","role"],  "msg": "Input should be 'admin', 'editor', 'viewer', 'billing' or 'support'"}
]}

Three problems named in one round trip, instead of one. That is a client that fixes its request once instead of three times.

Be careful about how far you push this claim, though. The border reports every problem at that layer. Pydantic still short-circuits per field, and a model validator that runs after field validation does not run at all if a field already failed. "Three problems in one round trip instead of one" is measured and true. "All problems, always" is not.

4. HTTP is not the only edge

The most useful demonstration is the one where you cannot blame the internet. A producer in this same process puts a message on a queue.Queue; a consumer reads it back and revalidates with the same model:

producer drops a field   -> role    / Field required [type=missing]
producer changes a type  -> user_id / Input should be a valid integer [type=int_type]
producer adds a field    -> tier    / Extra inputs are not permitted [type=extra_forbidden]

Our own code wrote that message and our own code serialised it, and the consumer still caught a schema change, a type change, and a field nobody agreed to. Anywhere data crosses from a system you do not control right now — a queue, a webhook, a config file, a CSV, another team's service, last year's version of your own producer — is an edge.

What it costs you, honestly

A border is a trade, not a free win. Four costs, all of them measured.

The shape is now written in two places, and it drifts immediately. Your model is one definition of a user; your table is another. In the test schema they disagreed in both directions within minutes:

model: email max_length=254     vs  DB: CHECK(length(email) <= 320)
  -> a 272-char email: MODEL rejects (string_too_long), DB stores it happily

model: age ge=13                vs  DB: CHECK(age >= 18)
  -> age=15: MODEL accepts, then
     sqlite3.IntegrityError: CHECK constraint failed: age >= 18

model: role Literal[...]        vs  DB: no CHECK on role at all

So "one model at the edge means the shape is defined in one place" is not true. It is defined in fewer places, which is the actual win.

A strict border breaks callers — loudly, which is the point, but it still breaks them. Add a required field and every existing client fails:

V2 with a REQUIRED field: country / Field required [type=missing]
V2 with a DEFAULT       : accepted, country='CH'

One line of difference. And with extra="forbid", a purely additive producer change — adding an optional tier field nobody has to use — takes every consumer down. On a public API that is what you want. On an internal event bus it may not be, and you should choose deliberately rather than inherit the default.

Over-validating drags domain rules into the boundary. "Is this a well-formed email address" belongs at the border. "May this user publish" does not — that needs the database, the session, and the current state of the world, none of which the border has.

It is not a security or integrity boundary on its own. A perfectly well-formed duplicate sails straight through:

model accepted: email='wiz@example.com' role='admin' age=30
sqlite3.IntegrityError: UNIQUE constraint failed: users.email

The model has no idea what is already in the table. Uniqueness, authorization, balances, rate limits, quotas — still the database's job, still the service's job. Edge validation stops bad shapes, not bad facts.

The performance folklore is backwards

The usual objection is that validating every request costs you. Measured with timeit, N = 100,000, pydantic 2.12.5 on Python 3.10:

Operation	Per call
`json.loads(bytes)` — baseline, no validation	2.75 µs
`Small.model_validate(dict)`	1.30 µs
`Small.model_validate_json(bytes)`	1.31 µs
`WithEmailStr.model_validate(dict)`	92.9 µs

Validating a three-field model straight from JSON bytes is 1.31 µs — roughly twice as fast as the json.loads you were already paying for. pydantic-core parses the JSON itself in Rust, so you skip the stdlib parse entirely. On this model, validation is not a tax. It is a discount.

But do not then say validation is free, because one field blows that up: EmailStr costs 92.9 µs, about 68× the rest of the model put together. The email-validator library does real syntax and IDNA work in Python. If a hot endpoint validates an email on every call, that is your cost — and it is one field, not the idea. Which is a much more useful thing to know than either slogan, because you can act on it: keep the border, and decide consciously whether that endpoint needs full RFC email validation or a cheaper constrained string.

The same border, one layer up: LLM tool calls and structured output

If you are building anything agentic, you already have this problem in its purest form — and most codebases are currently solving it with the four-checks version.

When a model calls a tool, what your code receives is a JSON string generated by a probabilistic text generator. That is not a stricter edge than an HTTP request body; it is a looser one. An HTTP client is at worst buggy. A language model is sampling, and it will occasionally produce "42" where you documented an integer, omit a field it has filled correctly a thousand times, invent an enum value that reads perfectly plausibly, or hallucinate a parameter your function does not have. role: "admn" is exactly the failure mode of a model that is 99% right.

So the border is the same border:

class SearchArgs(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")
    query: str = Field(max_length=200)
    top_k: int = Field(ge=1, le=50)
    index: Literal["docs", "tickets", "code"]

args = SearchArgs.model_validate_json(tool_call.arguments)   # or it does not run

Three things transfer directly, and one flips.

The allow-list becomes the schema you hand the model. The same declaration that rejects bad arguments is also the JSON Schema you put in the tool definition — so the allow-list is doing double duty: it constrains generation and it verifies the result. With constrained decoding or strict structured-output modes, part of your border has moved into the decoder itself. That is genuinely better, and it is still not sufficient — the shape can be guaranteed while the values remain nonsense, so you still validate what comes back.

The 422 body becomes a repair prompt. This is the part that is nicer in the LLM case than in the HTTP one. Pydantic's error list is not just diagnostics — it is text a model can act on. Feed Input should be greater than or equal to 13 back to the model and it will usually fix the argument on the next turn. The "name the field that failed" rule stops being a courtesy to a human client and becomes the actual control loop.

Every edge still counts, and there are more of them. Model output into your tool, tool output back into the context window, retrieved documents into the prompt, another agent's message into this agent's inbox. Each of those is an untrusted dictionary crossing into code that assumes a shape.

And the flip: the cost argument disappears completely. That 92.9 µs EmailStr field, which was worth thinking about on a hot HTTP endpoint, sits next to an inference call measured in hundreds of milliseconds. It is four orders of magnitude cheaper than the thing it is guarding. There is no performance conversation to have.

One warning, and it is §8c again in a much more dangerous costume: validating tool arguments is not authorization. A model that asks to delete every row in a table will produce beautifully well-formed arguments. The border will hand you a perfectly valid DeleteArgs. Whether that call is allowed is a question about permissions and blast radius, and no schema in the world answers it.

The verdict

Validation is not a check you perform. It is a border you cross.

Do it once, at the place data enters your program, and let it convert rather than approve — so what comes out the other side is a typed value that could not have been constructed if the input were wrong. Everything inside then takes that type as a precondition, instead of asking the same question again in slightly different words.

The practical test is the one you can run on your own codebase this afternoon: go find the if x is None checks buried in your business logic. Every one of them is either a border you never built, or a border that leaked. In the app measured here there were thirteen of them, and building one border took all thirteen to zero — along with 36 lines of code and three of four 500s.

And then stay honest about the edges of the idea: the shape is still written in your database too, and it will drift. Strict borders break callers on schema change, which is a feature you must still opt into deliberately on an internal bus. And a well-formed duplicate is not the model's problem — that is what the UNIQUE constraint is for, and it stays.

References and further reading

The mechanism — validate once and convert

Pydantic documentation, Models — model_validate / model_validate_json and the "parse, don't validate" behaviour this whole article rests on: the model returns a typed instance or raises, never a boolean.
Alexis King, Parse, Don't Validate (2019) — the clearest statement of why a function that returns a type beats one that returns True, and where the "every if x is None is evidence the border leaked" diagnostic comes from.

Rule 1 — say what you accept

OWASP, Input Validation Cheat Sheet — the allow-list-over-block-list position, stated as a security control rather than a style preference.

Rule 2 — reject, do not coerce

Pydantic documentation, Conversion Table — exactly which inputs are accepted for each field type in lax versus strict mode; the '42' → 42 case is in here.
Pydantic documentation, Strict Mode — what strict=True does and does not cover, which is the source of the coercion-versus-normalisation distinction above.

Rule 3 — name the field that failed

FastAPI documentation, Handling Errors — RequestValidationError and the shape of the 422 body, including the loc / msg / type entries quoted here.
RFC 9457, Problem Details for HTTP APIs (IETF, 2023) — the standard way to return a machine-readable error that says which member of the request was wrong and why.

The costs — drift, breakage, and what the border does not cover

Pydantic documentation, Model Config — extra — the extra="forbid" setting and therefore the additive-producer-change trade-off.
SQLite documentation, CHECK constraints — the second definition of your data's shape, and why the UNIQUE violation in the duplicate case came from the database rather than the model.

The LLM edge

OpenAI, Structured Outputs — schema-constrained generation: the allow-list pushed into the decoder, and an explicit statement of what it does and does not guarantee.
Anthropic, Tool use — how tool arguments are returned and why they need validating on arrival like any other untrusted input.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: the 2-minute version draws the four-stage pipeline and shows the drift; the full episode proves all of it on two apps that really run.

Read Replicas Explained: How to Scale Database Reads Without Shipping a Time-Travel Bug

Vahid Aghajani — Fri, 28 Aug 2026 06:13:58 +0000

Originally published on software-engineer-blog.com.

Here is a system design question that comes up constantly, and that almost everyone answers half-correctly:

The reflex answer is "add read replicas." That's not wrong — it's the right answer to the read half of the question. But the half people skip is the half where the bugs live, and it's the half that separates someone who has read about replicas from someone who has run them.

Let's build it from the floor up.

Start below the concept: one box

DevTalk is a developer forum. Threads, comments, upvotes. It runs on one database box.

Every page view hits that machine. Every new comment hits that machine. One CPU, one disk, one queue for all of it. That works fine — until it doesn't.

Before reaching for a solution, measure what the traffic actually is. On a forum, it looks like this:

Thousands of people reading threads.
A handful of people posting.

Roughly 90% reads, 10% writes. That lopsided ratio isn't a detail. It's the entire opening — it's the reason a fix exists at all.

Why one box fails, specifically

The failure mode is more interesting than "it gets slow." As read traffic grows, the flood of reads crowds out the writes. Your database has finite CPU, finite disk IO, finite connection slots, and reads are consuming nearly all of them.

So the user who is contributing — writing a comment, the thing your product actually exists for — is queued behind 10,000 people who are just browsing. The cheapest, least valuable requests are starving the most valuable one.

The mechanism: copy the box

The fix is exactly as literal as it sounds. Copy the box.

The original machine becomes the primary. The copies are read replicas — full, live copies of the same data, on separate machines.

And then one rule does all the work:

Every WRITE goes to the primary, and only the primary.
Every READ fans out across the replicas.

One writer, many readers. That's the whole idea. Everything else is plumbing.

How the copies stay current

The primary already writes every change it makes to a log — Postgres calls it the WAL (write-ahead log), MySQL calls it the binlog. It exists for crash recovery, and it's a complete, ordered record of every change.

Replication reuses it. The primary streams that log to each replica, continuously. Each replica replays it, applying the same changes in the same order. The replica isn't running your queries again — it's replaying the outcome.

Add replicas, add read capacity, roughly linearly. Three replicas, three times the read throughput.

Routing, in practice

Most stacks give you two connection strings and let you pick per query:

# Two engines, two roles.
primary = create_engine("postgresql://primary.db.internal/devtalk")
replica = create_engine("postgresql://replica.db.internal/devtalk")  # usually behind a LB

def get_thread(thread_id: int):
    # A read: safe to be a few ms stale.
    with replica.connect() as conn:
        return conn.execute(SELECT_THREAD, {"id": thread_id}).fetchone()

def post_comment(thread_id: int, user_id: int, body: str):
    # A write: primary, always. There is no "sometimes" here.
    with primary.begin() as conn:
        conn.execute(INSERT_COMMENT, {"thread": thread_id, "user": user_id, "body": body})

The failure mode to watch for is a write that sneaks onto a replica connection — replicas are read-only, so it doesn't corrupt anything, it just throws. That's a good thing: the database enforces the rule you're trying to enforce in code.

The upper bound nobody mentions

Here's where "just add more replicas" stops being a strategy.

Every replica replays every write. All of them. A replica isn't doing less work than the primary on the write path — it's doing exactly the same write work, just without originating it.

Which means:

Replicas buy you zero write capacity. If your bottleneck is writes, replicas do nothing. Worse than nothing — you now have more machines applying the same write load.
Each replica costs money, and adds one more copy that can fall behind.

Replicas are a knob for one specific problem. Know which problem you have before you turn it.

Dimension	Primary	Read replica
Accepts writes	Yes — the only one that does	No, read-only
Serves reads	Yes, but you want to spare it	Yes — this is its whole job
Data freshness	Always current, by definition	Milliseconds behind, sometimes much worse
Write work performed	Originates every change	Replays every change anyway
What adding one buys you	n/a — there is exactly one	More read throughput, ~linearly
What adding one does NOT buy you	n/a	Any write capacity at all
Scaling limit	Vertical only — a bigger box	Cost, and lag pressure per replica

Replication lag: the reason this is a real topic

Now the part that turns this from a diagram into an incident.

That streaming is asynchronous. The primary commits your write, tells the client "done", and moves on. It does not wait for the replicas to catch up. That's deliberate — if it waited, every write would be as slow as your slowest replica, and you'd have traded a read problem for a much worse write problem.

The consequence: a replica is always a little behind. Usually single-digit milliseconds. During a bulk import, a long-running vacuum, or a network hiccup, it can be seconds or minutes.

The bug you will actually hit

Here is the sequence, and it is so ordinary that it ships to production constantly:

A user types a comment and hits Post.
The write lands on the primary. It commits. The API returns 200.
The page reloads and fetches the thread.
That read is a read, so it goes to a replica — one that happens to be 40ms behind.
The comment isn't there.

From the user's side, the app just ate their comment. So they post it again. Now you have duplicate comments, and a support ticket that says "the site is broken" with no error in any log, because nothing errored. Every component did exactly what it was told.

This is a consistency problem wearing a UI bug's clothing, and it is the single most common way read replicas hurt teams that adopted them successfully.

The fix: read-your-own-writes

The fix has a name, and knowing the name is most of the battle: read-your-own-writes consistency.

The rule: for a few seconds after a user writes something, route that one user's reads to the primary. Everyone else keeps hitting replicas.

RYOW_WINDOW = timedelta(seconds=5)

def engine_for_read(user_id: int):
    # After you write, you read from the primary — briefly, and only you.
    last_write = cache.get(f"lastwrite:{user_id}")
    if last_write and (now() - last_write) < RYOW_WINDOW:
        return primary
    return replica

def post_comment(thread_id: int, user_id: int, body: str):
    with primary.begin() as conn:
        conn.execute(INSERT_COMMENT, {"thread": thread_id, "user": user_id, "body": body})
    cache.set(f"lastwrite:{user_id}", now(), ttl=RYOW_WINDOW)

Note how narrow this is. It's not "turn off replicas." It's a per-user, time-boxed exception that costs the primary a trickle of extra reads — the reads of people who just wrote, which by the 90/10 ratio is a tiny slice of traffic.

There are stronger variants when you need them: pin reads to a replica that has confirmed it replayed at least the log position of your write (Postgres exposes this via LSN comparison), or run one synchronous replica for the reads that truly cannot be stale. Both cost latency. Start with the time-boxed version.

Watch the lag, or the lag will find you

Replication lag is a first-class metric, not a debugging afterthought. Alert on it:

-- Postgres, on a replica: how far behind is this copy, in seconds?
SELECT now() - pg_last_xact_replay_timestamp() AS replication_delay;

And have a rule for what happens when it spikes: a replica lagging 30 seconds should be pulled out of the read pool, not left quietly serving stale pages.

Replicas are not sharding

Keep this distinction clean, because interviews probe it and architectures die on it.

Replication copies all the data to more machines. It scales reads. Every machine has everything.
Sharding splits the data across machines by some key. It scales writes (and dataset size). Each machine has a slice.

If your writes are the bottleneck, replicas will not help you, no matter how many you add. That's sharding's problem, and sharding is a substantially harder tool — cross-shard queries, rebalancing, and a shard key you can never comfortably change.

They compose, too: a sharded system usually has replicas per shard. But reach for them for different reasons.

The AI reframe: replicas under a RAG stack

This isn't only a 2010s web-forum concern. It reappears, with the same shape and the same bug, the moment you put a retrieval layer in front of an LLM.

A typical RAG service has exactly the DevTalk asymmetry, only more extreme:

Reads: every user question triggers a vector similarity search, often several (query expansion, multi-hop retrieval, re-ranking a wide candidate set). One question can be five reads.
Writes: ingestion — documents chunked, embedded, upserted. Bursty, and comparatively rare.

If your embeddings live in Postgres with pgvector, you have a database serving 95%+ reads, and read replicas are the obvious lever: fan the similarity searches across replicas, keep ingestion on the primary. Vector search is CPU-hungry per query, so this scales unusually well.

And then you hit the same bug, in a form that's harder to diagnose:

The user doesn't see a stale row. They see a model that is confidently wrong, which is far more corrosive to trust than a missing comment. The fix is identical: read-your-own-writes on the retrieval path, scoped to the user (or tenant) who just ingested.

A second wrinkle worth knowing: an ANN index (HNSW, IVFFlat) has to be built, and replaying an index build on every replica is real, sustained work. Bulk ingestion is exactly the workload that pushes replication lag from milliseconds into minutes — which is to say, the moment you most want the new documents visible is the moment your replicas are furthest behind. Ingest in throttled batches, and watch the lag metric during them.

The same asymmetry logic also drives the serving tier around the model — read-heavy traffic gets fanned out and cached, the small write path stays authoritative — but the database layer is where the correctness bug actually bites.

The verdict

Read replicas scale reads by giving up "now."

That's the trade in one line. Your data on a replica is not wrong — it is correct, just slightly late. And the engineering judgment isn't whether to accept that; it's deciding, read by read, which ones can tolerate lateness and which absolutely cannot.

A working default:

Fan out to replicas: thread lists, search results, feeds, dashboards, analytics, recommendation lookups, anything a cache would already be serving stale.
Keep on the primary: the read immediately after that user's own write, balance and inventory checks, anything that gates a decision (auth, permissions, "can this user do X"), and any read whose result you're about to write back.

Add replicas when reads are your bottleneck. Reach for sharding when writes are. And whichever you add, ship the lag metric with it — because the failure mode isn't an error page, it's a user who quietly stops trusting your app.

References and further reading

The read/write asymmetry and the one-box floor

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — chapter 5, "Replication", is the single best treatment of everything in this article; it opens with exactly this argument for why a read-heavy workload has a replication-shaped answer.
Silvia Botros & Jeremy Tinley, High Performance MySQL, 4th ed. (O'Reilly, 2021) — the operational view: what actually saturates first on one box, and how replication topologies are run in practice.

The mechanism — log streaming and replay

PostgreSQL documentation, High Availability, Load Balancing, and Replication — streaming replication, hot standby, and the knobs (synchronous_commit, hot_standby_feedback) referenced above.
MySQL documentation, Replication — the binlog-based equivalent, including row- vs statement-based replication.
Alex Petrov, Database Internals (O'Reilly, 2019) — part II on distributed systems; useful for understanding why the log is the replication unit rather than the query.

Replication lag and read-your-own-writes

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — "Problems with Replication Lag" in chapter 5 names read-your-own-writes, monotonic reads, and consistent prefix reads. The comment-disappears bug is his "reading your own writes" example.
PostgreSQL documentation, Monitoring — replication views — pg_stat_replication and the LSN positions you need to alert on lag or to pin a read to a caught-up replica.
Amazon, Working with Amazon RDS read replicas — the managed-service version, and a clear statement of the asynchronous guarantee you are actually buying.

Replicas vs. sharding

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — chapter 6, "Partitioning", is the other tool; reading 5 and 6 back to back makes the distinction permanent.

The retrieval/AI angle

pgvector, project documentation — index types (HNSW, IVFFlat), build cost, and the read characteristics that make vector search a good replica candidate.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: the 3-minute version builds DevTalk from one box, adds the replicas on screen, and walks through the disappearing-comment bug and its fix.

MVCC and VACUUM: Why Deleting a Million Rows Made Your Table Bigger

Vahid Aghajani — Thu, 27 Aug 2026 14:07:30 +0000

Originally published on software-engineer-blog.com.

Someone deletes a million old rows to free up disk. They watch the table afterwards and it is bigger than before.

Nothing is broken. They just met the machinery that lets every read in the database run without ever waiting for a write — and got the invoice for it.

To see why, you have to start one level below the concept everybody quotes.

The floor: an UPDATE does not overwrite the row

Here is the thing that explains all of it.

When your database runs an UPDATE, it does not go to that row on disk and change the bytes. It writes a new version of the row somewhere else, and stamps the old one dead as of transaction 91.

Both versions are now sitting in the table.

Why go to that trouble? Because of a query that started a second ago and is still running. That query must keep seeing the world it started in. If the update had overwritten the bytes, the reader would either have to be blocked until the writer finished, or watch its own data change underneath it mid-scan.

So the database keeps both, and lets each reader figure out which one is theirs.

Every row therefore carries two hidden columns you never wrote:

xmin — the transaction that created this version.
xmax — the transaction that killed it (empty while it is still live).

You can see them:

SELECT xmin, xmax, id, status FROM orders WHERE id = 4711;

 xmin | xmax |  id  | status
------+------+------+---------
   87 |   91 | 4711 | pending
   91 |    0 | 4711 | shipped

One logical row. Two physical rows. This is the whole idea, and everything below is a consequence of it.

Snapshots: how a reader knows which version is its own

Each query — more precisely, each transaction, depending on your isolation level — takes a snapshot. A snapshot is essentially just a number plus a short list of transactions that were still in flight when it was taken.

The visibility rule is then almost embarrassingly simple. A row version is visible to you if:

its xmin committed before your snapshot, and
its xmax is empty, or belongs to a transaction that had not committed before your snapshot.

Run that against the two versions of order 4711 above. A reader with snapshot 89 sees version (87, 91) — created before it, killed by a transaction that had not happened yet — so it reads pending. A reader with snapshot 95 sees (91, 0) and reads shipped. Same table, same instant, two different truths, both correct.

Nothing locked. That is the payoff, and it is worth saying precisely, because the slogan gets repeated without its meaning:

A reader never waits for a writer, because the version the reader needs is still physically sitting there. A writer never waits for a reader, because it is appending a new version rather than fighting for the old one. Two writers touching the same row still serialise — MVCC does not make write conflicts disappear — but the read path, which is most of your traffic, stops queueing entirely.

That is a genuinely large win. Now the bill.

A DELETE frees nothing

If an UPDATE is "insert new, stamp old", what is a DELETE?

It is only the stamp. The database writes xmax = 104 onto the row and moves on. It does not free a byte. It cannot — some transaction older than 104 may still legitimately need to read that row.

So DELETE FROM orders WHERE created_at < '2024-01-01' on a million rows produces:

a million rows that are now dead tuples — invisible to every new query, still fully present on disk,
a WAL record for each one,
and, on a table that was also taking ordinary update churn, a total of something like 3,400,000 dead tuples.

Live rows: unchanged at 2,000,000. File on disk: 1.2 GB → 4.8 GB.

That is the moment people file the bug report.

Operation	What happens on disk	Space freed immediately
`INSERT`	New version written, `xmin` = your txn	—
`UPDATE`	New version written and old one stamped `xmax`	None — the table grows by one row
`DELETE`	Old version stamped `xmax`. Nothing else.	None
`VACUUM`	Dead tuples marked reusable inside the same file	None returned to the OS; space is reusable internally
`VACUUM FULL`	Whole table rewritten into a fresh file	All of it — under an `ACCESS EXCLUSIVE` lock

Bloat charges you twice

The disk number is the visible half. The other half is worse:

Scans read the corpses. A sequential scan has to walk 4.8 GB of pages to find 1.2 GB of live rows. Your buffer cache fills with dead tuples. The query that used to fit in memory now does not.
Indexes still point into them. Every index entry for a dead tuple is still an index entry. The index grows, gets deeper, and every lookup that lands on a dead pointer has to go to the heap to find out it was wasted work.

So the table did not merely get bigger. It got slower in two independent ways, and neither of them shows up as an error.

VACUUM: the janitor, and the part everyone misses

The cleanup job is VACUUM, usually running automatically as autovacuum. What it does is scan for tuples that nobody can see any more and mark that space as reusable.

Run it and the dead-tuple count drops to zero. Then you check the file size, and it is still 4.8 GB.

This is the single most misunderstood fact about the whole system:

Which is the right trade, once you see the alternative. Handing space back to the operating system means moving live rows down to fill the holes and truncating the file — that is VACUUM FULL, and it rewrites the entire table while holding an ACCESS EXCLUSIVE lock. Nothing reads it. Nothing writes it. On a 40 GB table that is an outage, not a maintenance task. (pg_repack exists to do the same job online, at the cost of temporarily needing room for a second copy.)

For a table with steady churn, the ordinary VACUUM is what you actually want anyway. That 3.6 GB of reusable space is where the next 3.6 GB of updates will be written, instead of extending the file further. The table reaches a steady state at some size larger than its live set. That is not a leak. That is the working set of a versioned store.

The failure that actually pages people

Everything so far is fine. Here is the one that ruins a night.

A dead tuple can only be cleaned once nobody can still see it. That condition is global, not local. The database computes an oldest visible transaction horizon across the whole system, and refuses to remove any version newer than it — because some transaction out there might still legally read it.

So a single forgotten open transaction pins every dead row in the database.

An idle BEGIN; in a colleague's psql window from Tuesday. A connection-pool client that opened a transaction, hit an exception, and never issued a COMMIT or ROLLBACK. A read replica with hot_standby_feedback = on running a long analytics query, holding the primary's horizon back from the other end of the network. An abandoned replication slot.

And the symptom is not an error. The symptom is autovacuum running all night and reclaiming nothing, while the disk fills. You find out from the pager.

The first thing to look at is therefore not the table. It is the horizon:

-- who is holding the horizon back?
SELECT pid, state, backend_xmin,
       now() - xact_start AS txn_age, query
  FROM pg_stat_activity
 WHERE backend_xmin IS NOT NULL
 ORDER BY age(backend_xmin) DESC
 LIMIT 5;

-- replication slots do it too, without any session attached
SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;

The oldest backend_xmin — usually a session sitting in idle in transaction — is your culprit far more often than any vacuum tuning parameter. Two defences worth having on by default: idle_in_transaction_session_timeout so a forgotten BEGIN cannot live forever, and an alert on the age of the oldest transaction rather than only on disk usage, so you hear about it hours before the disk does.

The same bill, in an AI stack

If your database is also your vector store — and for a lot of RAG systems it is, via pgvector — this stops being a Postgres trivia question and becomes a running cost, because embedding workloads are unusually good at generating dead tuples.

Re-embedding is an UPDATE storm. Swap the embedding model, or re-chunk a corpus, and you rewrite the embedding column on every row. Each one is insert-new-plus-stamp-old, on rows carrying a 1536-dimension vector — roughly 6 KB each before overhead. A million re-embedded chunks is gigabytes of dead tuples, generated in one batch job that looked like an in-place update.
The index pays the most. An HNSW or IVFFlat index is far more expensive per entry than a B-tree, and it inherits the same problem: entries for dead tuples are still entries, and a bloated vector index is both larger in RAM and slower to traverse. Recall stays correct; latency does not stay flat.
Long-running ingestion pins everything. A nightly job that opens one transaction and streams a corpus through it for two hours is, from the horizon's point of view, exactly the forgotten BEGIN. Batch the commits.
And the reason you wanted MVCC is still true. During a re-index, your retrieval traffic keeps reading the old versions without blocking. That is the feature. Just budget for the disk it costs, and monitor autovacuum on the embeddings table specifically — it is usually the one that needs a more aggressive autovacuum_vacuum_scale_factor than the default, because the default is tuned for small rows.

The general rule, whatever the workload: anything that rewrites a whole column across a whole table is a bloat event. Backfills, re-embeddings, schema migrations that touch every row, GDPR deletion sweeps. Plan the vacuum, not just the write.

The verdict

The alternative to MVCC is not "no cost". It is making writers wait for readers — a lock-based system, cheaper on disk and considerably more miserable to operate, where one slow report can stall your write path. MVCC is the better trade for very nearly everyone.

You just have to know what you are paying in, and it is three things: disk, index bloat, and a background job you are obliged to monitor.

So: do not fight the model. Do not reach for VACUUM FULL on a schedule, do not read a table that is larger than its live set as corruption, and do not tune autovacuum before you have checked whether anything can be vacuumed at all. Delete in batches with commits between them. Alert on the age of your oldest open transaction. Expect a churny table to settle at some multiple of its live size and treat that number as normal.

Your database never deletes anything at the moment you tell it to. It writes a tombstone and hires a janitor.

40 GB of table holding 2 GB of live rows is not corruption. It is the invoice for concurrency.

References and further reading

On the floor — row versions, xmin/xmax, and why an UPDATE is an insert

PostgreSQL Documentation, Ch. 13: Concurrency Control — the primary source for the visibility rules and what each isolation level actually promises; the definition this article's snapshot section is built on.
Bruce Momjian, MVCC Unmasked — the clearest walkthrough of the hidden system columns and how a row version is judged visible, with the on-disk picture drawn out.
Joseph M. Hellerstein, Michael Stonebraker & James Hamilton, Architecture of a Database System (Foundations and Trends in Databases, 2007) — where the storage layer and the transaction layer meet, and why version storage is a design axis rather than an implementation detail.

On snapshots and what they do and do not guarantee

Hal Berenson, Phil Bernstein, Jim Gray, Jim Melton, Elizabeth O'Neil & Patrick O'Neil, A Critique of ANSI SQL Isolation Levels (SIGMOD, 1995) — the paper that defined snapshot isolation formally and showed which anomalies it still permits. The right next read if this article left you wondering what a snapshot cannot protect you from.
Jim Gray & Andreas Reuter, Transaction Processing: Concepts and Techniques (Morgan Kaufmann, 1993) — the canonical treatment of the lock-based alternative, which is what makes the trade in the verdict concrete rather than rhetorical.

On the bill — dead tuples, bloat, and the janitor

PostgreSQL Documentation, §25.1: Routine Vacuuming — states directly that ordinary VACUUM makes space available for re-use rather than returning it to the operating system, and covers the autovacuum tuning knobs.
PostgreSQL Documentation: VACUUM — the FULL variant, its ACCESS EXCLUSIVE lock, and the disk-space requirement for the rewrite.
Yingjun Wu, Joy Arulraj, Jiexi Lin, Ran Xian & Andrew Pavlo, An Empirical Evaluation of In-Memory Multi-Version Concurrency Control (VLDB, 2017) — a systematic comparison of version storage, garbage collection and index-management schemes across MVCC implementations; the best single source for why garbage collection, not the versioning itself, is where these systems live or die.
pg_repack — the online alternative to VACUUM FULL, and its own trade-off: no long exclusive lock, but room for a second copy of the table.

On the AI/vector-store section

pgvector — index types, build and maintenance behaviour for HNSW and IVFFlat; the reference for why a re-embedding pass is an index event and not only a table event.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: MVCC and VACUUM — why deleting a million rows made your table bigger

What Is Middleware? The Onion Model, and Why the Order Is the Configuration

Vahid Aghajani — Thu, 27 Aug 2026 06:40:32 +0000

Originally published on software-engineer-blog.com.

One small service. One process, one port, six business endpoints plus a health check.

Three things had to happen on almost every one of them: check the caller's token, write a log line with a request id, start and stop a timer. Nine lines. They were written once, and then pasted by hand into endpoint after endpoint.

# the SAME nine lines, pasted by hand into endpoint after endpoint
async def ep_account(scope, receive, send):
    user = user_for(header(scope, b"authorization"))        # pasted
    if user is None:                                        # pasted
        return await send_json(send, 401, {"error": "no"})  # pasted
    started = clock()                                       # pasted
    log.info("GET /account", rid=rid(scope))                # pasted
    result = handlers.account(user)                         # the actual work
    metrics.observe("GET /account", clock() - started)      # pasted
    await send_json(send, 200, result)

Then the sixth endpoint was added. It was copied from the one next to it, and in the copy the two auth lines did not come along.

async def ep_create_review(scope, receive, send):
    started = clock()                                       # pasted
    log.info("POST /reviews", rid=rid(scope))               # pasted
    result = handlers.create_review(user_of(scope), body)   # the actual work
    metrics.observe("POST /reviews", clock() - started)     # pasted
    await send_json(send, 200, result)   # the auth lines are not here

Sweep the API with no Authorization header at all and you get this:

GET     /books            401  {"error": "unauthorized"}
GET     /books/1          401  {"error": "unauthorized"}
POST    /orders           401  {"error": "unauthorized"}
GET     /orders/1         401  {"error": "unauthorized"}
GET     /account          401  {"error": "unauthorized"}
POST    /reviews          200  {"review": {"user": "anonymous", "book": 1, "stars": 5}}
GET     /healthz          200  {"ok": true}

POST /reviews answered everyone. Nothing threw. No test failed. No error log line was ever written — the endpoint was working exactly as its code said. Counting the cross-cutting lines inside the endpoint functions gives 50 across the module: nine on each of five endpoints, and five on the sixth. The missing four are the entire incident.

That is the problem middleware solves. Not "less typing" — we will get to the honest bill later, and it is not smaller. One place.

What a middleware actually is

A middleware is a function that is handed two things: the request, and the next thing to call.

That second argument is the entire idea.

def timing(request, call_next):
    started = clock()                      # 1. before  — runs on the way IN
    response = call_next(request)          # 2. hand it on, and WAIT here
    took = clock() - started               # 3. after   — runs on the way OUT
    response.headers["x-took-ms"] = took
    return response

Look at line 2. It is not "finish my job and hand over." It blocks until everything deeper in the chain has run and come back. Which is precisely why line 3 is able to know how long the whole rest of the request took.

If you delete the call_next argument, you no longer have a middleware. You have a hook.

The chain is an onion, not a queue

Almost everyone's first mental picture of a middleware chain is a row of turnstiles: the request passes through gate one, then gate two, then gate three, then reaches your handler. It is a tidy picture and it is wrong.

The steps do not stand in a row. Each one wraps the next. So every step runs twice: once on the way in, and once on the way back out.

Here is a real trace, printed by a chain of five:

--> ERRORS
--> REQUEST ID
--> LOG + TIMER
--> AUTH
--> RATE LIMIT
    [ YOUR HANDLER RUNS ]
<-- RATE LIMIT
<-- AUTH
<-- LOG + TIMER
<-- REQUEST ID
<-- ERRORS

entered 5 steps, exited 5 steps
exits are the entries REVERSED : True

Read the two halves. Going in, the order is the list order. Coming out, it is the list reversed. That reversal is the whole shape, and it is not a stylistic detail — it is what makes several of the most common middlewares possible at all:

A timer on the outside can measure the whole request, because its "after" half runs last.
An error handler on the outside can catch an exception thrown by anything beneath it.
A response-header step can set a header only after the response exists.

Measured on a chain of four: the outermost layer's mean was 68.3 µs, the innermost (the handler alone) 33.9 µs. The handler was 49.7 % of what the outermost layer measured, and the outer figure was greater than or equal to the inner one on every one of 2,000 requests. That is the onion, in a number.

A step is allowed to refuse

The second consequence of "you are handed the next thing to call" is that you may decline to call it.

def auth(request, call_next):
    if not valid_token(request.headers.get("authorization")):
        return json({"detail": "unauthorized"}, status=401)   # no call_next
    request.state.user = user_for(request)
    return call_next(request)        # only a valid caller gets past this line

Fifty requests with no token, through a chain of auth → rate limit → database lookup → handler:

AUTH reached           50
RATE LIMIT reached      0   <- never ran
database lookup         0   <- never ran
your handler reached    0   <- never ran
responses              50 x 401

Your handler did not decide to refuse those. Your handler was never asked. A browser CORS preflight that never reaches your code is the same move, and so is a WAF rule, and so is an API gateway's quota check.

This is also the first hint of a real operational problem. If a request is rejected two layers above your handler, your handler's own metrics never see it. In one run, 160 requests arrived, the handler counted 50, and the gap of 110 reconciled exactly to 80 rejections plus 30 throttles. That is 68.8 % of all traffic invisible to the dashboard most teams actually look at.

The order is the configuration

Here is the part that makes middleware a design decision rather than a convenience.

# the whole ordering decision, in one list, outermost first.
CHAIN = [errors, request_id, logging, auth, rate_limit]   # A: auth, then limit
CHAIN = [errors, request_id, logging, rate_limit, auth]   # B: limit, then auth

Same two components. Same code inside them. One line moved. Now send the identical 90-request burst at both — 30 from one authenticated user, 30 from thirty different authenticated users, 30 with no token, limit 10 per key:

On the same 90 requests	ORDER A auth → limit	ORDER B limit → auth
reached the auth check	90	10
reached the rate limiter	60	90
200 OK	40	7
401 unauthorized	30	3
429 too many requests	20	80
rate-limiter keys used	31 (per user)	1 (per IP)
the one heavy user, throttled	20	26
thirty different users, throttled	0	27

40 successful responses versus 7. That is not a mild trade-off, it is two different products.

And neither order is wrong. Order A is a per-user quota; order B is a per-source flood gate. The limiter can only key by user if something above it has already identified one — placed first it has no user yet, so it falls back to keying by client address, and behind a shared address that means thirty innocent users share one bucket. That is not a bug. It is the consequence of the position.

The same principle bites a request-id step. Placed first, all four downstream log lines carry the id. Placed last, only 1 of 4 does:

request-id FIRST                      request-id LAST
[log:edge]    rid=ffc68931            [log:edge]    rid=-
[log:access]  rid=ffc68931            [log:access]  rid=-
[log:audit]   rid=ffc68931            [log:audit]   rid=-
[log:app]     rid=ffc68931            [log:app]     rid=fe9f57b1

Note that it is 3 of 4 lost, not all four — the step still tags whatever runs beneath it. And the error handler is the sharpest case of all. Outermost, with a failing layer below it: HTTP 500, body {"error": "internal server error"}, caught. Innermost, with the failing layer above it: no HTTP response at all — the exception escapes the application entirely.

The worst property of an ordering mistake is that it is silent. On that 90-request burst under the wrong order: exceptions escaped 0, exceptions caught 0, ERROR log lines 0, 5xx responses 0, log lines written 90 — all INFO, every status code documented and expected. The only visible symptom, and only if you already knew your intent, was that 27 requests from users who had sent one request each came back 429.

The honest bill

Middleware is usually sold as a cleanup. Measured, it is a trade, and it is worth being precise about what you are trading.

It does not mean fewer lines. The endpoint module shrank from 92 SLOC to 52. But the steps themselves are another 137 SLOC, plus an 8-line list to order them. At six endpoints, the repository is bigger. What you bought is not brevity — it is that the auth rule exists in one place, so the sixth endpoint cannot forget it. On the chain variant, the same unauthenticated sweep returns 401 on all six business endpoints, including the newest one, which nobody had to remember.

It costs something on every request. A chain of eight layers against a bare handler: bare 4.3 µs, full chain 30.7 µs — an overhead of 26.4 µs/request, about 7.1×, roughly 3.3 µs per layer. Quoted alone, that number sounds alarming. So here is the other one: measured end to end over a real socket, a request took 640.1 µs, and the chain was 4.1 % of it. Quote either figure by itself and you are misleading someone. The chain is expensive relative to a function call and cheap relative to a request.

Every request pays, including the ones that did not need to. One layer doing a small database read added +13.8 µs/request to /healthz — 2.02× the same chain without it — and performed 5,300 of 5,300 database reads for an endpoint that needed exactly none.

It is invisible from the handler, and the blast radius is the whole API. Changing if scheme != "Bearer": to if scheme == "Bearer": is a one-character diff in a shared layer. Result: 6 of 6 business endpoints reject a valid credential, 1 file changed, 0 endpoint files changed, and handlers.py byte-identical before and after. Nothing in the handler you are staring at explains the failure.

The same shape, in front of a model

If you work on LLM serving, you have this chain whether you called it middleware or not — it is what an inference gateway is. And the vocabulary maps cleanly:

Auth becomes API-key resolution to a tenant and a model allowlist.
Rate limiting becomes a token budget rather than a request budget, and the ordering lesson lands twice as hard: a limiter placed before authentication cannot key by tenant, so one noisy customer and thirty quiet ones share a bucket — the exact 27-out-of-30 failure above, except now it is a paying customer's SLA.
Request id becomes the trace id that has to survive a response lasting several seconds.
Guardrails — prompt-injection screening on the way in, PII or policy filtering on the way out — are the onion's two halves, and the reason they belong in a wrapper rather than in the handler is the same reason auth did: the next endpoint you add must not be able to skip them.
Cost accounting is a pure "way out" step: you cannot bill for tokens you have not generated yet.

But one thing genuinely does not transfer, and it catches people. The onion's "on the way back out" half assumes the response is a single object handed back up the stack. When you stream tokens over SSE, the response object returns almost immediately and the body arrives afterwards. A timing middleware written the ordinary way will therefore record time to first token, not total generation time — a number that can be twenty times smaller and looks perfectly healthy on a dashboard. The same applies to an output filter: by the time your "after" half runs, the first tokens are already on the client's screen. Streaming-aware guardrails have to wrap the iterator, not the response.

The shape holds. The assumption that a request is one atomic unit of work does not.

Copy-paste versus chain, side by side

	Pasted into every endpoint	One chain, registered once
cross-cutting lines inside endpoints	50	0
total code	92 SLOC	52 + 137 SLOC (bigger)
the newest endpoint	can silently omit auth	covered without being asked
changing the auth rule	edit 6 files, hope you got them all	edit 1 file
blast radius of a typo	one endpoint	every endpoint
where the behaviour is written	in front of you, in the handler	somewhere else, in a list
cost	~0	~3.3 µs per layer, on every request
ordering bugs	impossible	possible, and silent

The verdict

Use middleware for the things that are genuinely true of almost every request — authentication, request ids, error handling, logging, rate limiting, CORS. Those earn the wrapper, because their failure mode is omission, and a chain makes omission impossible.

Do not use it for things that are true of some requests. A layer that reads the database for every call so that three endpoints can avoid a lookup is a tax collected 5,300 times to be spent 3 times; that belongs in a dependency the three endpoints ask for.

And then treat the order as what it is. It is not registration boilerplate at the bottom of a file — it is a configuration file for your API's behaviour, written in the least obvious syntax imaginable, and it fails without raising anything. Put the list somewhere a reviewer will look at it, write down why each step sits where it does, and test the order the way you would test a feature: send a burst, count the status codes, and check you got the product you meant to build.

Your handler is not the first code that runs. Know what is in front of it.

References and further reading

On the problem — cross-cutting concerns and the pasted nine lines

Gregor Kiczales et al., Aspect-Oriented Programming (ECOOP, 1997) — the paper that named cross-cutting concerns and the scattering/tangling problem; the missing auth block in POST /reviews is a textbook instance of scattering.
Deepak Alur, John Crupi & Dan Malks, Core J2EE Patterns, 2nd ed. (Prentice Hall, 2003) — the Intercepting Filter pattern: a configurable, ordered chain of filters around a request handler. This is the pattern middleware implements, described before the word "middleware" was common in web frameworks.

On the shape — why each step wraps the next

PEP 3333 — Python Web Server Gateway Interface v1.0.1 — the primary source for the wrapping definition: middleware is itself an application that plays server to the application below it. The onion is written into the spec, not into a framework.
The ASGI Specification — the async successor, and the place to read how the model changes when a response is a stream of events rather than one return value.
Writing middleware for use in Express apps (Express documentation) — the shortest statement of the next() contract, including what happens when you decline to call it.
Roy T. Fielding, Architectural Styles and the Design of Network-based Software Architectures, ch. 5 (dissertation, 2000) — the layered system constraint: why intermediaries a client cannot see are a deliberate property of the web, not an accident.

On ordering, refusal, and what it costs you operationally

Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the case for guards that run before your code at all (bulkheads, circuit breakers, handshaking), and why a component that sheds load must sit where it can actually see the load.
Charity Majors, Liz Fong-Jones & George Miranda, Observability Engineering (O'Reilly, 2022) — instrumenting at the edge versus inside the handler; the direct answer to the 68.8 % of traffic the handler's own counters could not see.
Jeffrey Dean & Luiz André Barroso, The Tail at Scale (Communications of the ACM, 56(2), 2013) — why a small fixed per-request cost is usually the wrong thing to worry about, and variance is the right one.

On the model-serving section

Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP, 2023) — the vLLM paper; useful here for why a request to a model server is not one atomic unit of work, which is exactly the assumption a "way back out" middleware makes.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: Middleware — the code that runs before your handler · or the full episode.

Design Search Autocomplete: The Four-Point Answer, and the Trap

Vahid Aghajani — Tue, 25 Aug 2026 16:35:04 +0000

Originally published on software-engineer-blog.com.

"Design search autocomplete" — the suggestion dropdown under a search box — is one of the most
common system-design interview questions, and one of the easiest to answer badly. The obvious
answer is a search: the user types, you go and find the matching queries, you sort them, you
return the best ten.

That is the failure. It describes a search, and the answer is a read.

Everything below follows from one observation: the request fires on every keystroke, it has a
budget of roughly 100 milliseconds end to end, and reads outnumber writes by an enormous
margin. A design that does real work per keystroke cannot survive that ratio. So you do the work
in advance, once, for everybody.

1. A trie — a prefix tree

Start with the structure, because it is the thing that makes "what the user has typed so far" a
first-class object rather than a string you have to search for.

A trie stores a set of strings as a tree in which every prefix is a node on the path from the
root:

        (root)
          │
          d
          │
         da
          │
        dat
          │
       data
        /    \
  database   data structures

Two consequences worth saying out loud in the interview:

A keystroke is one step, not one query. The client already knows which node it was on for dat; typing a moves to a child. Lookup cost is a function of the length of the prefix, not of how many queries you have indexed. Ten queries or ten billion, data is four hops.
The prefix is a position. "What the user has typed" stops being a search key and becomes a pointer into a data structure. That reframing is the whole answer in miniature.

In production this is rarely a naïve pointer-per-character trie — it is usually compacted (a radix
tree) or compiled into a finite-state transducer so that the whole thing fits in memory and shares
suffixes. Worth naming, not worth dwelling on. The shape is what matters.

2. Precompute the top K completions at each node

Here is the trick the entire design turns on, and the point where most answers quietly go wrong.

The naïve version is: walk to the node for data, then traverse the subtree below it to
collect every completion, score them, and take the best ten. That is a graph traversal on the hot
path, executed once per keystroke — and it is slowest exactly where the traffic is heaviest,
because short popular prefixes (d, da) have enormous subtrees.

Instead, at build time, you store the finished list on the node itself:

node "data"   →   top 10, precomputed
                    data structures
                    database
                    data science
                    ...

Now the query path is:

walk d → da → dat → data
read the list stored there
return it

No traversal, no sort, no scoring, no ranking work of any kind. You land on the node and read
the answer that is already sitting there. That is a lookup, not a search — and it is the sentence
you want the interviewer to hear.

The cost you are trading against is space: every node carries a small list, so the index gets
bigger. That is a good trade, and it is worth volunteering. You are buying a bounded, predictable
read for a one-off increase in the size of an artifact you rebuild offline anyway.

3. Cache the hot prefixes at the edge

Short section, deliberately.

Prefix traffic is wildly skewed. A small number of prefixes — the first one or two characters of
whatever the world is currently searching for — account for most of the requests, while the long
tail is enormous and almost never hit:

 ▉
 ▉  ▊
 ▉  ▊  ▍
 ▉  ▊  ▍  ▁  ▁
 └── prefixes, by traffic ──►

So put the answers for the hot prefixes in a cache close to the user. A few thousand entries
absorb the bulk of the load and never reach your service at all. The response for a given prefix
is a small, stable blob, which is exactly the shape a cache likes. (I have written separately
about what a CDN actually is and how it differs from an application cache; here that is the only
property that matters.)

4. Rank from real query logs, rebuilt offline in batch

This is the point most answers miss, and the one that most clearly separates "I read a blog post"
from "I have thought about this".

The ranking is not alphabetical. Nobody wants the alphabetically-first completion; they want
the one people actually search for. So the score on each completion comes from real query
logs — how often that full query was issued, and often whether the suggestion was clicked when
it was shown.

And the ranking is not live. The pipeline is a batch job:

yesterday's query logs
        │
        ▼
  batch job: count + rank
        │
        ▼
  a fresh trie, shipped

Aggregate yesterday's query frequencies, recompute the top-K list for every node, build a whole
new trie, and ship it as an immutable artifact that the serving tier loads and swaps in.

The consequence is the important part: the serving path only reads. It never writes a query
back into the index it is currently serving from. The moment it does, you have put a write on the
hot path — locks, invalidation, tail latency — and given away the property you designed the whole
system to have.

Two follow-ups worth having ready, because a good interviewer will ask:

"What about a query that trends right now?" You run a second, much smaller, fast path — a near-real-time layer over the last few minutes of logs — and merge its handful of results with the batch answer at read time. It is a deliberate exception to the batch rule, and it is small precisely because the exception is expensive.
"How do you personalise it?" Same shape: the batch trie is the global answer, and a small per-user signal re-orders the top few. You do not build a trie per user.

The two designs, side by side

The difference is not an optimisation. It is a different system with a different failure mode:

	Search at query time (the wrong answer)	Read a precomputed answer (the right one)
What a keystroke does	Matches the prefix, collects candidates, scores and sorts them	Walks a few nodes and reads a stored list
Work per request	Grows with the size of the subtree under the prefix	Proportional to the length of the prefix — bounded, and tiny
Worst case	Short, popular prefixes: biggest subtree, highest traffic, same request	Short prefixes are the cheapest and the most cacheable
Where the ranking happens	On the hot path, per request, per user	Once, offline, in a batch job over query logs
What the serving tier does	Reads and writes the live index	Reads only; new data arrives as a swapped-in artifact
Cost you pay	CPU and tail latency, on every keystroke, forever	Storage, plus staleness measured in hours
Fails by	Falling over exactly when it is popular	Being slightly out of date

The right-hand column trades a property nobody notices (a few hours of staleness in a suggestion
list) for a property everybody notices (a dropdown that appears instantly). That is the trade the
interviewer is checking you can make.

The same trick, in LLM serving

If you work on model serving rather than search, you have met this design already — it is wearing
different vocabulary.

The prefix is still a position. When you send a prompt to an LLM, the expensive part of the
first token is processing the prompt itself. But most production traffic shares prefixes: the same
system prompt, the same few-shot examples, the same conversation replayed one turn longer. So
serving engines keep the computed attention state (the KV cache) keyed by token prefix, and a
request that shares a prefix with something already computed starts from where that left off
instead of from scratch. SGLang's RadixAttention does this with a literal radix tree over token
prefixes — the same structure as section 1, holding cached computation instead of cached
completions. "Prompt caching" in the commercial APIs is the same idea behind a billing line.

And the ranking is still offline. In a RAG system, nothing about the corpus is understood at
request time. The chunking, the embeddings, the index — all built in a batch job, ahead of time,
and shipped. The request embeds one short query and does a lookup. If your retrieval path is
embedding documents while the user waits, you have made the autocomplete mistake in a new costume.

The mapping is close enough to be worth carrying between the two domains:

Autocomplete	LLM serving
Trie node per prefix	Radix-tree node per token prefix (RadixAttention, prompt caching)
Top-K list precomputed on the node	KV cache: the attention state for that prefix, already computed
Hot prefixes cached at the edge	Hot system prompts pinned in cache; a shared prefix is a cache hit
Trie rebuilt nightly from query logs	Vector index rebuilt from the corpus, offline, and swapped in
Serving path never writes	Inference never rebuilds the index it is retrieving from

Same sentence in both worlds: do the expensive thing once, in advance, and make the request a
read.

The trap: it is a read, not a search

Here is the closing line, and it belongs at the start of your answer, not the end.

If you are running a search when the user types, you have already lost. Autocomplete answers
from something that was built hours ago. The trie was built offline. The top-K lists were computed
offline. The ranking came from yesterday's logs. All the request does on the hot path is walk a
few nodes and read a list.

Say the word precomputed in the first minute, and every other decision — why the top-K lives
on the node, why the ranking is a batch job, why the serving tier never writes — follows from it
without you having to argue for any of them separately.

The verdict

The whole answer in four lines:

a trie → one node per prefix; a keystroke is one step down, not one query
top K → precomputed and stored on the node, so the read is a lookup and not a traversal
the edge → a few thousand hot prefixes absorb most of the traffic
the logs → yesterday's real queries decide the ranking, rebuilt offline; serving never writes

And the one sentence underneath all four: the request does not compute the answer, it fetches
one that already exists. Every autocomplete design that is fast is fast for that reason, and every
one that is slow is slow because something on the hot path is still thinking.

References and further reading

The structure (§1)

Edward Fredkin, Trie Memory (Communications of the ACM 3(9), pp. 490–499, 1960) — the original paper introducing the trie, and the source of the property §1 rests on: the key is the path, so every prefix is a node and a lookup costs the length of the key. Covers the data structure only, nothing about ranking or serving.
Robert Sedgewick & Kevin Wayne, Algorithms, 4th ed. (Addison-Wesley, 2011) — §5.2 on R-way tries and ternary search tries, including keysWithPrefix. This is the textbook version of the "collect the completions by traversing the subtree" operation that §2 deliberately replaces.

Precomputing the completions (§2)

Elastic, Elasticsearch Reference — Suggesters (completion suggester) — a production system with exactly this shape: a dedicated in-memory FST built at index time, optimised for prefix lookups, rather than a query against the inverted index at request time.

Ranking from query logs (§4)

Ziv Bar-Yossef & Naama Kraus, Context-Sensitive Query Auto-Completion (WWW '11, pp. 107–116, 2011) — defines and evaluates MostPopularCompletion, ranking completions by their frequency in a query log rather than alphabetically. The direct support for §4's claim about where the ranking comes from; it does not address the batch/serving split.
Fei Cai & Maarten de Rijke, A Survey of Query Auto Completion in Information Retrieval (Foundations and Trends in Information Retrieval 10(4), pp. 273–363, 2016) — the broad survey: candidate generation, ranking signals (popularity, time-sensitivity, personalisation), and evaluation. The best single source if you want to go past the four points, and the place to read up on the trending-query follow-up.

The LLM-serving parallel

Lianmin Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (2024) — RadixAttention keeps the KV cache in a radix tree keyed on token prefixes so that requests sharing a prefix reuse the computation. The same structure as §1, holding cached work instead of cached completions.

If a reference you would expect to see here is missing, say so in the comments and I will add it.

▶ Watch the reel: Design search autocomplete — the four-point answer, and the trap

The Ring Buffer: Why a Queue With No Limit Is a Memory Leak (and What Happens When It Fills)

Vahid Aghajani — Tue, 25 Aug 2026 07:26:35 +0000

Originally published on software-engineer-blog.com.

Here is a system small enough to hold in your head. One Python process. No broker, no network, no database. A request handler measures how long each request took and writes the sample into a queue. A background thread reads samples back out and appends them to a file on disk.

# one CPython process. no broker, no network, no database.
# the request handler writes a sample; a background thread drains it to disk.

from collections import deque

q = deque()                                     # no maxlen
sample = {"path": "/checkout", "ms": 41.2}      # one sample, made by the request

This works perfectly, right up until the writer is faster than the reader. And then it does something worse than break: it keeps working.

The queue never complains. It never returns an error, never logs a warning, never blocks. It just gets longer.

The failure that never looks like a failure

I ran it. A producer pushing 4× faster than the consumer drains, three million samples produced, 750,000 consumed, 2.25 million left as backlog:

=== collections.deque()  (NO maxlen) ===
  start RSS                :       9.9 MB
  after   250,000 events : RSS     60.6 MB   grew     50.7 MB     202.8 MB/M events
  after 1,000,000 events : RSS    212.4 MB   grew    202.5 MB     202.5 MB/M events
  after 2,000,000 events : RSS    414.8 MB   grew    404.9 MB     202.5 MB/M events
  after 3,000,000 events : RSS    617.2 MB   grew    607.3 MB     202.4 MB/M events

9.9 MB to 617.4 MB. And look at the last column — 202.8 at the first checkpoint, 202.4 at the twelfth. The growth is not "roughly linear", it is linear to within a rounding error: 202.5 MB per million backlogged events, forever.

The accounting closes exactly, which is what makes it so mundane: 2.25 million dicts at 232 bytes each is 522 MB of dict objects, and the measured growth is 607 MB — the extra ~85 MB is the deque's own block storage plus the float objects inside each sample. Nothing is leaking in the C sense. Every byte is reachable, correct, and doing its job. There is simply no line of code anywhere that says stop.

This is why it is so hard to catch. A classic memory leak has a smoking gun — an object nobody freed. Here, every object has a legitimate owner. The queue is supposed to hold things. It is holding things. The bug is that nobody ever said how many.

Swap in queue.Queue() with no maxsize and you get the same curve — 617.4 MB peak, 196.4 MB per million — plus a lock on every operation that makes it 4.3× slower (4.60 s vs 1.08 s for the same work).

The fix is a decision, not a data structure

A ring buffer is a fixed-length array plus two integers. That's genuinely all it is:

CAPACITY = 8                     # decided once, at startup, and never again

buf  = [None] * CAPACITY         # the entire memory footprint is this line
head = 0                         # how many samples have ever been written
tail = 0                         # how many samples have ever been read

The trick is that head and tail never wrap. They count forever. Only the index wraps, at the moment you touch the array:

class Ring:
    def __init__(self, n):
        self.buf  = [None] * n     # allocated once, never again
        self.n    = n
        self.head = 0              # total writes, ever
        self.tail = 0              # total reads, ever

    def push(self, x):
        self.buf[self.head % self.n] = x
        self.head += 1

    def pop(self):
        x = self.buf[self.tail % self.n]
        self.tail += 1
        return x

Because the counters are monotonic, the three questions you actually want to ask become plain arithmetic, with no special cases:

how many are in there? head - tail
is it empty? head == tail
is it full? head - tail == n

Here is the same run against a 1,024-slot ring — identical producer, identical three million events:

  start RSS                  :     10.4 MB
  sys.getsizeof(buf) BEFORE  : 8,248 bytes
  after 1,000,000 events   : RSS     11.0 MB   grew   +0.64 MB   getsizeof(buf) 8,248
  after 3,000,000 events   : RSS     11.0 MB   grew   +0.64 MB   getsizeof(buf) 8,248

After three million writes the container is byte-for-byte the same size it was before the first one. RSS growth: +0.64 MB against +607.4 MB — 949× less. Measured with tracemalloc on a smaller run, peak traced memory is 79.28 MB unbounded versus 0.27 MB for the ring, a ratio of 293:1.

One honest correction, because it's the kind of thing that gets repeated wrongly: in Python, "nothing is allocated after startup" is true of the buffer and false of the events. Half a million dicts really were created in that loop — I counted them. What the ring changes is that at most n of them are ever reachable, so CPython recycles the same freed memory instead of asking the OS for more. Memory goes flat, not to zero. In C, where the ring holds raw structs rather than pointers to heap objects, the stronger claim does hold — you malloc once at startup and push/pop are index arithmetic and a store, with no allocator on the hot path at all. That is precisely why kernels, audio callbacks and embedded firmware use this structure: not because it's fast, but because it's predictable.

The part everybody gets wrong: full and empty look identical

If head and tail are stored wrapped — the way most tutorials write it — the structure has a genuine ambiguity, and it is not a subtle one.

  EMPTY ring : pushed 0, popped 0        -> (head, tail) = (0, 0)
  FULL  ring : pushed 8, popped 0        -> (head, tail) = (0, 0)

  contents of the FULL ring's storage    : [1, 2, 3, 4, 5, 6, 7, 8]
  contents of the EMPTY ring's storage   : [None, None, None, None, None, None, None, None]

  (head, tail) pairs IDENTICAL?          : True   <-- BYTE-IDENTICAL

A buffer holding eight unread events and a buffer holding nothing produce the byte-identical pair (0, 0). And it isn't an artifact of starting at zero — do three push/pop cycles first and you get (3, 3) for empty and (3, 3) for full. head == tail cheerfully reports EMPTY while eight events sit there unread.

The reason is a counting argument, and once you see it the whole family of fixes falls out:

So you either add one bit of information, or you delete a state. Every real implementation is one of those two moves:

Technique	The move	Usable capacity	Cost
Separate `count` field	add information	n of n	every push and pop must update a third variable
Waste one slot	delete a state	n − 1 of n	declare 8 slots, actually get 7
Monotonic counters (never wrap)	add information	n of n	counters grow unboundedly (a non-issue at 64 bits)
A single `full` boolean	add information	n of n	one bit, plus remembering to maintain it

You'll often read that there are only two options — a count or a wasted slot. There are at least four, and the third one is the formulation used above and the one the Linux kernel's kfifo uses. It costs nothing and wastes nothing.

The real question: what happens when it's full?

Here's the part that makes this a system-design topic rather than a data-structures exercise.

Bounding the queue doesn't make the overflow problem go away. It forces you to answer it, at the moment you write the code, instead of discovering the answer at 3am when the OOM killer picks a process. And there are exactly two answers:

    def push_overwrite(self, x):       # a log buffer, a metrics window
        if self.head - self.tail == self.n:
            self.tail += 1                 # give up the oldest sample
            self.dropped += 1              # and COUNT what you gave up
        self.buf[self.head % self.n] = x
        self.head += 1
        return True

    def push_or_refuse(self, x):       # a job queue: never lose work
        if self.head - self.tail == self.n:
            self.refused += 1
            return False                   # the producer has to handle this
        self.buf[self.head % self.n] = x
        self.head += 1
        return True

Same structure. Opposite guarantee. One branch of code apart.

Overwrite says: the newest data matters most, and I would rather lose history than lose the present. That's a metrics window, a log ring, an audio buffer, a flight recorder — you want the last N seconds before the crash, and the seconds before those are worthless.

Refuse says: every item is work that must not vanish, so if I can't take it, the producer needs to know. That's a job queue. Refusing is how backpressure gets created — the pain travels back up to whoever is producing too fast, which is the only place it can actually be fixed.

Choosing wrong is a data-loss bug wearing a performance costume. Silently overwriting a job queue loses paid work. Refusing writes to a metrics buffer takes down the request path to protect telemetry that nobody was going to read.

Python hands you both — but only one for free

from collections import deque
from queue import Queue

# POLICY ONE, handed to you: a bounded deque OVERWRITES, silently
d = deque([1, 2, 3], maxlen=3)
d.append(4)                  # d is now deque([2, 3, 4]) - the 1 is gone
                             # nothing raised, nothing logged, no return value

# POLICY TWO, and you have to ask for it by name
q = Queue(maxsize=3)
for i in (1, 2, 3): q.put_nowait(i)
q.put_nowait(4)              # raises queue.Full

Just how silent is the overwrite? I went looking for anything at all that signals the loss:

append() returns None — exactly as it does on a non-full deque.
No exception, no warning, no log line.
len() is 8 before and 8 after, so length can't tell you either.
There is no attribute anywhere on the object that counts discards. I checked dir().

The only way to know is to ask before you append: if len(d) == d.maxlen: dropped += 1. If you take one practical thing from this article, take that line.

The refusing side is loud but oddly terse: the exception is queue.Full, and str(e) is the empty string — the type is the message. After the refusal qsize() is still 8 and the contents are unchanged. Note which item died in each case: deque(maxlen) sacrifices the oldest data silently; Queue(maxsize) sacrifices the newest, loudly. Same bounded capacity, opposite victim.

One more distinction worth keeping straight: a deque without maxlen grows. A ring buffer refuses to — and refusing to grow is the entire feature, not a limitation you work around.

The loss isn't just bounded, it's predictable

"You'll drop data" sounds like a concession. It's actually the strongest thing about the design, because the drops obey a closed form:

  ring n=1,024  burst=  900 events   DROPPED        0 ( 0.00 %)
  ring n=1,024  burst=1,200 events   DROPPED   35,200 (14.67 %)
  ring n=1,024  burst=4,096 events   DROPPED  614,400 (75.00 %)
  ring n=4,096  burst=4,096 events   DROPPED        0 ( 0.00 %)

  predicted drops = bursts x max(0, burst - n)   -> EXACT MATCH, all four rows

drops = bursts × max(0, burst_size − n), and the measured count matched the prediction exactly in every configuration. Size the ring at or above the burst — the last row — and the drop count is exactly zero.

Meanwhile peak RSS stayed at ~11 MB across all of it. Push the overrun to its worst and peak memory does not move, because peak memory tracks the ring you chose, not the burst you got. That is the whole trade: you convert an unbounded, invisible, unpredictable memory risk into a bounded, counted, predictable data loss. One of those you can put on a dashboard and alert on. The other one pages you at 3am.

Where it's already running on your machine

You don't have to adopt this pattern; you're already surrounded by it. All of these are readable without root:

Ring	Size on my box	Policy
Kernel log ring (`printk`)	256 KiB (`CONFIG_LOG_BUF_SHIFT=18`, fixed at kernel build)	overwrite the oldest
A pipe	65,536 bytes = 16 pages of 4 KiB	refuse — blocks the writer
`perf` event ring	516 KiB per user	overwrite, and increments a lost-event counter
ALSA PCM preallocation	32,768 KiB max per substream	overwrite (audio can't wait)

The pipe is the best one, because you can watch it happen with no privileges at all. F_GETPIPE_SZ on a fresh pipe reports 65,536 bytes. Fill it non-blocking and it accepts exactly 65,536 and then the write refuses with EAGAIN. Read 8,192 bytes back out and the writer immediately accepts 4,096 more — the ring wrapped and reused the freed slots.

So the deque(maxlen)-versus-Queue(maxsize) decision isn't a Python quirk. It's the same fork, made inside the kernel: the log ring overwrites you, the pipe blocks you. Same structure, the other policy.

And note what perf does, because it's the design worth copying: it overwrites and counts what it lost. Overwriting is defensible. Overwriting silently is not.

The bit-trick that isn't (in Python)

While measuring, one piece of folklore fell over, and it's worth flagging because it's repeated everywhere: "use & (n - 1) instead of % n, it's much faster."

  bare arithmetic  head % N     :   24.66 ns
  bare arithmetic  head & MASK  :   34.27 ns     <- the mask is 39 % SLOWER
  inside a method  RingMod.push :  285.14 ns
  inside a method  RingMask.push:  283.00 ns     <- indistinguishable, it's noise

In CPython the mask is 39% slower on the bare arithmetic, and inside a real method call the difference vanishes into noise. Both are a single bytecode op, so it's a fair fight — CPython's long_mod has a fast path for single-digit ints while & goes through a more general path.

The trick is real in C, Rust and assembly, where % compiles to a hardware divide costing tens of cycles and & is one. Say it about compiled languages; don't claim it about the Python on your screen. Use whichever is clearer, and save the trick for the language where it pays.

While we're at it, here's what the operations actually cost:

Operation	ns/op
hand-written ring push (Python method)	~285
hand-written ring pop	~109
`deque(maxlen=n).append` on a full deque	~37
`Queue.put_nowait` + `get_nowait` pair	~1,867

The stdlib deque beats the hand-rolled ring by 7.8×, because deque is C and your ring pays a ~250 ns Python method call. Queue is 4.7× slower than the hand-written ring and 32.7× slower than deque — that gap is the lock, and it's the honest number for what thread-safety costs you.

So: write the ring by hand to understand it. In production Python, reach for deque(maxlen=n) — and add the drop counter yourself.

The reframe: this is the shape of LLM serving

If you work on model serving rather than web backends, you've met all of this under different names — and the failure mode is more expensive, because the memory in question is GPU memory.

The request queue in front of a model server is the exact push_or_refuse branch. An inference server can only hold so many concurrent sequences; the KV cache for each one is real, reserved VRAM. If the arrival rate exceeds the completion rate and the admission queue is unbounded, you don't get slow — you get a CUDA OOM that kills in-flight requests that were already 80% generated. This is why serving stacks admit a bounded number of sequences and reject the overflow with a 429 rather than buffering it. Refusing the write is the backpressure, and it's the difference between a degraded service and a dead one.

Sliding-window attention is push_overwrite, applied to context. A model with a fixed window keeps the last n tokens of KV cache and evicts the oldest as new ones arrive — a ring buffer over tokens, with the overwrite policy chosen deliberately because recent context is worth more than distant context. The fixed window is what makes memory per sequence a constant you chose at config time rather than a function of conversation length. Exactly the 617 MB → 11 MB trade, in VRAM.

Streaming token buffers are rings too. The generation loop produces tokens faster than a network client consumes them; a bounded buffer between them keeps a slow client from making the GPU-side loop hold unbounded state.

And the metrics case is the original one: every inference server tracks rolling TTFT and inter-token latency over a fixed window of recent requests. Not because old measurements are wrong, but because a p99 over "all of time" is useless and an unbounded list of samples is the 202.5 MB/million curve waiting to happen on your serving box.

The question is identical in every one of these, and it's the question the unbounded version never made anyone ask: when this fills up, does the oldest thing die quietly, or does the newest thing get refused loudly? In a token window, the oldest should die. In an admission queue, the newest must be refused. Getting that backwards costs you either a truncated context or a dropped payment.

The verdict

An unbounded queue is not a queue. It's an unbounded memory allocation with a friendly API, and it will happily grow at 202.5 MB per million events until the kernel picks a process and kills it. It never returns an error, because from its point of view nothing has gone wrong.

A ring buffer is a fixed number of slots decided once, at startup. That single constraint buys you three things: memory that is flat and known before you deploy (+0.64 MB versus +607.4 MB over the same three million events), loss that is predictable in closed form rather than catastrophic, and — most valuable of all — it forces the overflow decision into your source code, where you can read it, instead of leaving it to the OOM killer.

So the real content of this data structure isn't the modulo. It's the branch:

In Python, reach for deque(maxlen=n) when overwriting is right and Queue(maxsize=n) when refusing is right. Then add the one line the stdlib doesn't give you — the counter that tells you it's happening.

References and further reading

The structure itself, and the wrap arithmetic

Donald E. Knuth, The Art of Computer Programming, Volume 1: Fundamental Algorithms, 3rd ed. (Addison-Wesley, 1997) — §2.2.2 "Sequential Allocation" is the classical treatment of a queue in a circular array, including the full-versus-empty problem and the wasted-slot fix.
Linux kernel documentation, FIFO Buffer (kfifo) — the monotonic-counter formulation used throughout this article, in production C.

Choosing the policy: overwrite or refuse

Python documentation, collections.deque — the maxlen paragraph, which is where the silent-discard behaviour is specified rather than implied.
Python documentation, queue.Queue — maxsize, put_nowait, and the queue.Full exception.
Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — chapter 11 on message brokers frames "what happens when the queue fills" (drop, buffer, or apply backpressure) as a design decision with consequences, which is the argument this article makes at process scale.

Cost, locks, and lock-free rings

Martin Thompson, Dave Farley, Michael Barker, Patricia Gee & Andrew Stewart, Disruptor: High Performance Alternative to Bounded Queues for Exchanging Data Between Concurrent Threads (LMAX, 2011) — a production ring buffer built specifically to avoid the lock that costs queue.Queue its 32.7× penalty here.
Ross Bencina, Real-time audio programming 101: time waits for nothing — why "no allocator on the hot path" is a correctness requirement and not an optimisation, which is the strongest case for the fixed-at-startup array.

Where the rings already are

man 7 pipe — pipe capacity and F_GETPIPE_SZ; the 65,536-byte figure measured above, and the blocking semantics that make a pipe the refusing variant.
Brendan Gregg, Systems Performance: Enterprise and the Cloud, 2nd ed. (Addison-Wesley, 2020) — the perf and BPF ring buffers, including the lost-event counters that make overwriting observable rather than silent.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: the 2-minute version frames the failure, and the full episode builds the ring from scratch and runs every measurement on screen.