DEV Community: Vahid Aghajani

Background Jobs Without a Broker: Your Worker Is Holding a Database Connection for 40 Minutes

Vahid Aghajani — Fri, 14 Aug 2026 12:07:08 +0000

Originally published on software-engineer-blog.com.

Here is a real row from a real database.

pid  | application_name | state               | duration
-----+------------------+---------------------+-----------
2841 | worker-07        | idle in transaction | 00:40:12

One worker. One open connection. State: idle in transaction. Forty minutes and twelve seconds.

The job it was running finished long ago. Nothing in the code has crashed. There is no error in any log file. But two things are going wrong at the same time:

New connections are starting to be refused.
The cleanup job inside the database is running, and freeing nothing. Not in one table. In every table of that database.

This post is about how an ordinary background job ends up here, and how to write one that never can.

Two sentences carry the whole thing:

A job is a row in a table, not a function call.
A connection is not a transaction.

If you only remember those two, you will already write better workers than most teams.

Every number and every error string below was measured against a real PostgreSQL 16.14 and SQLAlchemy 2.0.52. One of them contradicts what almost everybody says. I will show you that one.

First, the picture that kills the confusion

Think of a bank with a long counter.

On the left, a person is sitting at it. The bank has a fixed number of seats. Taking one costs exactly one seat. It costs one seat whether you are writing on a form, waiting for a phone call, or finished ten minutes ago and simply stayed. When every seat is taken, the next person who walks in is turned away.

Now look at the right. Same counter, same seat. But this time a large ledger book is lying wide open. An open page means: not signed off yet. And while any page is open, the clerk behind cannot file old paper away, because your open page might still refer to it. One open page holds up the filing for the whole bank.

The seat is a connection: one open line between your program and the database.
The open page is a transaction: a group of changes the database treats as one unit.

Your worker takes both. Only one of them is cheap.

The system: four boxes

Four boxes, and nothing else appears for the rest of this post.

The browser — one person, one tab. They upload a large file, and later they ask: is it ready yet?
Your API server — one uvicorn process. It must answer in milliseconds, so it must not do slow work.
Postgres — one server, with one table called jobs, one row for each piece of work.
The worker — a separate Python process. Thirty of them, doing the slow work.

Now the lines. The browser talks to the API server and gets an answer back. The API server writes the job row into Postgres and reads it back later. The worker claims a row and updates it.

Now look at what is missing. There is no line between the browser and the worker. They never speak. Everything they share, they share through one row.

Where almost every codebase starts

It is only one line.

@router.post("/uploads")
def create_upload(file: UploadFile, bg: BackgroundTasks,
                  db: Session = Depends(get_db)):
    row = Upload(name=file.filename, status="converting")
    db.add(row)
    db.commit()

    bg.add_task(convert, row.id)   # one line. the request returns now.
    return {"id": row.id, "status": "converting"}

A file comes in. We insert a row that says converting, and we commit it. Then the line everybody writes: bg.add_task(convert, row.id). And we return straight away.

The user gets an answer in about forty milliseconds. The conversion carries on afterwards. It really does look like it solved the problem.

Now look again — not at what the code does, but at where it runs.

A BackgroundTask has no worker. The slow work is running inside the API server, the one box whose whole job is to answer fast. There is no arrow to the worker, because there is no worker.

And now three ordinary things end the story badly:

The process runs out of memory, and Linux kills it.
You deploy, and the old process is replaced in the middle of a file.
The machine simply reboots.

In all three cases the work is gone. What is left behind is a row that still says converting, forever, with nothing anywhere knowing that a job was lost.

The work lived inside a process whose whole purpose is to be replaceable.

A job is a row

Here is the sentence everything else follows from. A job is a row. Not a function call.

A function call lives inside one process. When that process ends, the call is gone, and there is nothing left to look at.

A row lives in the database:

It outlives the request that created it.
It outlives the worker, because if that worker dies, another one can pick the same row up again.
It outlives your deploy, because a deploy replaces processes. It does not touch rows.

So the rule is simple. Write the work down first. Only then do it.

The table is the whole queue

CREATE TABLE jobs (
    id         bigserial   PRIMARY KEY,
    kind       text        NOT NULL,
    payload    jsonb       NOT NULL,
    status     text        NOT NULL DEFAULT 'queued',
    attempts   int         NOT NULL DEFAULT 0,
    run_after  timestamptz NOT NULL DEFAULT now(),
    locked_by  text,
    heartbeat  timestamptz
);

CREATE INDEX jobs_ready ON jobs (run_after)
    WHERE status = 'queued';

Read the columns:

kind says what work this is.
payload is JSON, and carries the arguments.
status starts at queued.
attempts lets you stop a job that keeps failing.
run_after is a timestamp, so a job can be scheduled for later — and so a retry can wait before trying again.
locked_by says which worker owns this row right now.
heartbeat says when that worker last proved it was still alive.

Then one index. Notice the word WHERE at the end. That is a partial index: it only covers the rows that are still queued. That matters, because a jobs table fills up with finished rows very quickly, and you do not want the lookup to get slower every day.

The one thing a broker structurally cannot do

A broker is a separate server whose only job is to hold a list of work — RabbitMQ, or Celery with Redis behind it. We are not using one. Look at what that buys you.

def create_upload(db: Session, name: str) -> int:
    with db.begin():                 # ONE transaction opens here
        row = Upload(name=name, status="converting")
        db.add(row)
        db.flush()                   # the database gives us row.id

        db.add(Job(kind="convert",
                   payload={"upload_id": row.id}))
    # the commit happened on that line. both rows, or neither.
    return row.id

One transaction. We insert the upload row, we flush so the database gives us the id, then we insert the job row using that id. The commit happens when the block ends. Both rows, or neither.

Now think about what happens with a broker instead. You have two systems:

The database can commit while the message to the broker is lost. Now you have an upload nobody will ever convert.
Or the message can be sent while the database rolls back. Now a worker picks up a job for a row that does not exist.

Both of those are real bugs that real teams spend real weeks on. Put the job in the same table and the same transaction, and neither one can happen. It is not that they are unlikely. They are impossible.

Thirty workers, one table

How do thirty workers not collide? With one keyword.

-- app/worker/claim.py — handed to SQLAlchemy text()
UPDATE jobs SET status = 'running', locked_by = :me,
                heartbeat = now()
WHERE id = (
    SELECT id FROM jobs
    WHERE status = 'queued' AND run_after <= now()
    ORDER BY id
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
RETURNING id, kind, payload;

Read the sub-query from the bottom up:

LIMIT 1 — take a single row.
FOR UPDATE — lock that row, so no other worker can take it.
SKIP LOCKED — and this is the important part. Without it, a worker that finds a locked row waits for it. With it, the worker steps over that row and takes the next free one.

I ran this with two workers starting at the same moment on a real Postgres:

Setup	Worker A took	Worker B took	Worker B waited
`FOR UPDATE SKIP LOCKED`	ids 1, 2	ids 3, 4	1.12 ms
`FOR UPDATE` only	ids 1, 2	blocked until A committed	3.035 s

With SKIP LOCKED, worker B stepped straight over A's two rows and was working in about one millisecond.

Without it, worker B blocked for 3.035 seconds — exactly how long worker A held its transaction. While it waited, pg_stat_activity showed it as wait_event_type = 'Lock', wait_event = 'transactionid'. It was not doing anything useful. It was queueing behind a row it was never going to get, because once A committed status = 'running', READ COMMITTED re-checked the WHERE clause and those rows no longer matched.

In production that hold is not three seconds. It is the length of the job.

So do you even need a broker?

People argue about this with the wrong measurement. They ask how long the job runs. That is not the axis.

The real question is: who owns the status of this job?

Put it in a table when…	Reach for a broker when…
A user can ask "is my file ready yet?"	Nobody ever asks about one individual job
You need to show progress, or the reason something failed	You are sending one event out to many consumers
The job must survive the queue itself restarting	The volume is high and each message is tiny
Volume is thousands a day, not millions a minute	Fan-out and routing are the actual product

In the left column, the status is domain data. It belongs in a table you can query, join and index.

Both are correct answers — to different questions. And when the table genuinely runs out of room, the next step is usually not a broker. It is PgBouncer in transaction mode.

The trap: a connection is not a transaction

Now the reason this post exists.

A connection is the seat at the counter. Your worker holds one so it can talk to the database at all. A transaction is the page lying open on that counter, not signed off yet.

Here is the problem. Your worker takes both, at the same moment, without anybody asking it to. One of them costs a seat. The other one costs the whole building.

Take them one at a time. Expensive one first.

Child one: the invisible one

Here is what people usually say:

I measured it. That sentence is wrong.

VACUUM is the cleanup job inside Postgres. When you update a row, the old version is left behind as a dead tuple, and VACUUM is what reclaims that space. It can only reclaim a dead version that no open transaction could still need to see. The oldest thing any open transaction might still need is called the xmin horizon.

So the question is not "is a transaction open?" It is "is this transaction holding the horizon back?"

I built a table with 200,000 dead tuples, opened one blocking session in nine different states, and ran VACUUM (VERBOSE) on a table that blocker had never touched. Here is what actually happened:

The blocking session	Holds a snapshot / XID?	Dead tuples reclaimed?
Control — no long transaction open	no	✅ reclaimed
READ COMMITTED, bare `BEGIN;`, no statement	no	✅ reclaimed
READ COMMITTED, `SELECT` only, then idle	no	✅ reclaimed
READ COMMITTED, `UPDATE`, then idle (your worker)	XID	❌ nothing freed
READ COMMITTED, statement still in flight	snapshot	❌ nothing freed
READ COMMITTED, open cursor, then idle	snapshot	❌ nothing freed
REPEATABLE READ, `SELECT` only, then idle	snapshot	❌ nothing freed
SERIALIZABLE, `SELECT` only, then idle	snapshot	❌ nothing freed
REPEATABLE READ, but in a different database	snapshot	✅ reclaimed

Read row three again. A READ COMMITTED session that only read something and then sat idle for twenty seconds had all 200,000 dead tuples reclaimed normally. The folklore version of the rule is simply not true.

Two things actually pin the horizon:

Holding a snapshot — that means REPEATABLE READ, SERIALIZABLE, an open cursor, or being in the middle of a statement.
Holding an XID — a transaction id, which Postgres hands out the moment a transaction writes anything.

Now look at what your worker did to claim its job. It ran UPDATE jobs SET status = 'running'. That is a write. So it holds a transaction id for the entire length of the conversion.

-- the worker: it claimed a job, and the claim was a WRITE.
BEGIN;
UPDATE jobs SET status = 'running' WHERE id = 1;
-- ... and now it converts a file for forty minutes ...

-- meanwhile, on a table this worker has never touched:
VACUUM (VERBOSE) events;
-- tuples: 0 removed, 400000 remain, 200000 are dead
--         but not yet removable
-- removable cutoff: 783  <-- the transaction id of that worker

That output is copied from the real run. removable cutoff: 783 was exactly that worker's backend_xid. Nothing was freed — on a table it had never opened.

Two limits are worth being precise about:

It is database-wide, not cluster-wide. The last row of the table shows it: the same blocker sitting in a different database did not hold this one back.
The damage stops the moment the transaction commits. The horizon is released immediately; it is not a permanent debt.

Child two: connections multiply

This one is cheaper on its own. It multiplies.

Count the connections one busy worker holds:

one for the loop that claims jobs,
one for the heartbeat that says it is alive,
one inside the handler that is doing the work.

Three. Thirty workers, doing nothing clever, on a completely ordinary day: ninety.

And how many can you have? A stock Postgres sets max_connections = 100 and keeps superuser_reserved_connections = 3 back for the administrator. So an ordinary application role gets 97.

Your API server has a connection pool of its own on top of that. A pool is a small set of connections opened once and reused. SQLAlchemy's defaults are pool_size=5 and max_overflow=10 — that is 15 per process, so two API processes can want thirty.

Ninety plus thirty is one hundred and twenty. You have ninety-seven. Somebody is refused.

Nothing in that arithmetic is a bug or a leak. That is the default configuration, running exactly as designed.

And here is a detail almost nobody expects. The error your application sees is not the famous one:

FATAL:  remaining connection slots are reserved for roles
        with the SUPERUSER attribute

sorry, too many clients already is what a superuser sees. An ordinary application role hits the message above, three connections earlier.

The box that hears it first is the API server — because the workers already took their seats and are sitting on them, while the API server is still asking for new ones. The workers broke nothing. They just arrived first.

Two problems, one query

And the obvious version of that query is wrong.

Most people look for sessions where backend_xmin IS NOT NULL. That misses your worker completely, because a READ COMMITTED transaction that wrote something sets a backend_xid and leaves backend_xmin empty.

Combine the two columns and you catch every case:

SELECT pid, application_name, state,
       age(COALESCE(backend_xmin, backend_xid)) AS holding_back_by,
       now() - xact_start AS open_for
FROM   pg_stat_activity
WHERE  datname = current_database()
  AND  COALESCE(backend_xmin, backend_xid) IS NOT NULL
ORDER  BY holding_back_by DESC;

Read the results like this:

A session that only read, then went idle: no transaction id, nothing held back. Harmless.
A session that wrote a row and then went idle — exactly your worker: it has a transaction id, and it pins the cleanup.
A session using REPEATABLE READ, or holding an open cursor: it holds a snapshot, so it pins the cleanup too.

One query, and it tells you which of the two problems you actually have.

The fix is a shape, not a setting

def run_one() -> bool:
    with session() as s:              # session 1 — the claim
        job = claim(s)                # short. milliseconds.
        if job is None:
            return False
        job_id  = job.id              # snapshot the fields we need
        payload = dict(job.payload)   # BEFORE the session goes away

    # ---- nothing held here: no session, no connection, no transaction ----
    result = convert(payload)         # minutes. hours. it does not matter.

    with session() as s:              # session 2 — record the outcome
        s.execute(FINISH, {"id": job_id, "result": result})
    return True

Session one: claim the job. This part is short — a few milliseconds — and it commits and closes immediately.

Then, the important line: nothing is held here. No session. No connection. No transaction. The conversion can take five minutes or five hours, and the database does not care, because the database does not know it is happening.

Session two: write the result, and close.

That is the entire pattern. Claim fast, close, work holding nothing, then record.

And copying job.id and job.payload out before the block ends is not tidiness. SQLAlchemy's Session runs with expire_on_commit=True by default, which marks every attribute on that object as out of date when the transaction commits. Read job.payload after the block and you get a DetachedInstanceError — there is no session left to load it from. (Note the precise cause: it is expire_on_commit, not close() on its own. A session closed without committing leaves already-loaded attributes readable.)

One setting left

engine = create_engine(
    DATABASE_URL,
    pool_size=5,          # the default. PER PROCESS.
    max_overflow=10,      # the default. PER PROCESS.
    pool_pre_ping=True,   # NOT the default. turn it on.
    pool_recycle=1800,    # and retire a connection after 30 minutes
)

pool_size=5 and max_overflow=10 are the defaults, and both are per process. pool_recycle=1800 retires a connection after thirty minutes. And pool_pre_ping is off by default. Turn it on.

Here is why it is correctness, not tidiness.

A pooled connection can be closed underneath you while it sits idle — by a network device, by a database restart, by a timeout. Without pre-ping, the next time your code uses it you get a sqlalchemy.exc.OperationalError (SQLAlchemy error code e3q8, wrapping a psycopg2.OperationalError). It is not an InterfaceError, which is what most people write their except clause against.

Now think about when that happens. Your worker has just finished a forty-minute conversion and is about to write down that it succeeded. The write fails. The job is recorded as failed, and the sweeper runs the whole thing again.

With pool_pre_ping=True, the pool quietly tests the connection on checkout, throws it away, and opens a fresh one. I measured it: the backend process id simply changes, and your code never notices.

So, all of it

Because a job is a row, it survives the request, the worker and the deploy. Many workers share it with FOR UPDATE SKIP LOCKED. It commits together with the business row, so it can never be lost or invented. And a user can simply ask the table whether it is ready. You get a queue, and you did not add a service.

Because a connection is not a transaction, a worker that wrote something and then went idle stops the cleanup everywhere in that database. Connections are cheap one by one and expensive together. Use COALESCE(backend_xmin, backend_xid) to find them.

So: claim, close, do the work holding nothing, then record.

Two settings are worth turning on today:

pool_pre_ping=True on the engine.
idle_in_transaction_session_timeout on Postgres, which kills a session that has been sitting on an open transaction for too long.

And if that joined a few things up — go and look at pg_stat_activity on your own database right now.

The full walk-through, with every measurement on screen, is the 15-minute episode.

Heaps and Priority Queues: The Array That Is Secretly a Tree

Vahid Aghajani — Thu, 13 Aug 2026 06:10:37 +0000

Originally published on software-engineer-blog.com.

One warning before we start, because the word is overloaded. This is the heap data structure, not the heap *memory region* where your language allocates new objects. They share a name and nothing else — different concept, different field of study, zero overlap. If you came looking for the memory one (stack vs heap allocation), this is not that post, and it never touches it again.

The data structure does exactly one job: it always hands you the most urgent item next, and it does that cheaply.

Where this bites in real code

You run background jobs — nightly reports, welcome emails, invoice files, password resets. For a long time you process them in arrival order, and that is fine.

Then the queue gets long. One night there are 100,000 jobs waiting, and 12,000 of them are nightly reports. A password reset lands at the back. Someone is sitting there, staring at a "check your email" screen, behind twelve thousand PDFs.

The fix sounds trivial: stop taking the oldest job, take the most urgent one. That single change is the whole problem, because "find the smallest thing in a big pile, over and over" is startlingly expensive done naively.

The two obvious implementations

# jobs.py  —  100,000 queued jobs. always take the most urgent one next.

jobs = []                          # every job is a pair: (urgency, name)

# way 1 — a plain list. cheap to add. expensive to choose.
def take_most_urgent(jobs):
    best = min(jobs)               # look at all 100,000 to find one
    jobs.remove(best)              # then look at them all again to delete it
    return best                    # 72.0 seconds to drain the queue

# way 2 — keep it sorted by -urgency, so the most urgent one is LAST.
import bisect
def add(jobs, urgency):
    bisect.insort(jobs, -urgency)  # the search is fast. making room is not:
                                   # every item after that spot shifts by one.
def take_sorted(jobs):
    return -jobs.pop()             # free to take. 1.39 seconds for the queue.

Way 1 makes two full passes over the list for every single job you pull. Way 2 moves the cost to insertion: bisect finds the right slot in log n, but then every item after that slot shifts over by one, which is O(n) of memory movement.

Measured on one machine, CPython 3.10, draining all 100,000 jobs:

Implementation	Time to drain 100,000 jobs	Cost per operation
Plain list + `min()`	72.0 s	O(n) to choose, O(n) to remove
Sorted list (`bisect.insort`)	1.39 s	O(n) to insert, O(1) to take
Binary heap	54.8 ms	O(log n) both ways

Seventy-two seconds. Not milliseconds — seconds, to work through a queue of background jobs. The sorted list is a real improvement and still spends 25x what the heap does. The heap is roughly 1,300x faster than the naive version.

The bakery and the hospital

Before any more code, one picture, because there is a confusion worth killing early.

On the left, a bakery. You walk in, take a numbered ticket, and the counter calls the numbers in order. That is a plain FIFO queue. It never asks why you came. If you are in real trouble, you still wait behind everybody else.

On the right, a hospital. The same five people walk in, in the same order. But a nurse at the desk asks one question each and reorders them on the spot. Arrival order stops mattering; urgency decides.

A priority queue is that promise: items keep arriving, and whenever you ask, you get the best one out. It is a behaviour, not an implementation. A binary heap is the standard, cheap way to keep that promise — and it has only two moves in the entire structure: sift up, which runs when you add, and sift down, which runs when you take.

The one rule

Draw your items as a tree. Every circle holds one value. Now the rule, and there is only one:

That is it. That is a heap.

Read the rule again and notice what it does not say. It says nothing about two nodes sitting next to each other. [3, 5, 7, 9, 12, 8, 14] is a perfectly legal min-heap even though 12 sits before 8 — they are in different subtrees and the rule never compares them.

This is the single most important thing to internalise: a heap is not sorted. sorted([3, 5, 7, 9, 12, 8, 14]) gives you [3, 5, 7, 8, 9, 12, 14], a different list. That weakness is the reason it is fast. Sorting maintains a global ordering on every insert; a heap maintains a local one, only along each parent-child edge, and a local promise is enormously cheaper to keep.

There is no tree

Here is the part that makes it cheap, and it is smaller than most people expect.

There are no circles. There are no node objects. There is a plain list. Take the tree, read it level by level — top to bottom, left to right — and write those values into one flat array in that order. That array is the heap. The tree was only ever a picture we drew to explain it.

Because the layout is fixed, family relationships become arithmetic:

# heap.py  —  the whole trick: the tree is never built. it is computed.

def parent(i): return (i - 1) // 2      # who is above me
def left(i):   return 2 * i + 1         # my first child
def right(i):  return 2 * i + 2         # my second child

# there is no Node class here. no left pointer, no right pointer, no
# allocation per item. the tree is only a picture we draw of a flat list.

def is_heap(heap):                      # the one rule the list must obey
    return all(heap[parent(i)] <= heap[i] for i in range(1, len(heap)))

# $ python -i heap.py
# >>> is_heap([3, 5, 7, 9, 12, 8, 14])
# True                                  <- a legal heap
# >>> sorted([3, 5, 7, 9, 12, 8, 14])
# [3, 5, 7, 8, 9, 12, 14]               <- a different list. a heap is NOT sorted.

Three one-line functions replace an entire node class. To find who sits above index 7, you do not chase a pointer — you compute (7 - 1) // 2 = 3. No allocation per item, no pointer chasing, and the whole structure sits in one contiguous block of memory that the CPU cache actually likes.

Move one: push, and sift up

You add a job with urgency 2 — very urgent. It lands on the end of the list, at index 7, which is almost certainly the wrong place. So you fix it, one step at a time.

The parent of index 7 is (7-1)//2 = 3, which holds 9. Is 2 < 9? Yes — trade places. Now your value sits at index 3, and you repeat against its parent. You climb until the rule holds again.

# heap.py  —  push: drop it on the end, then walk it up

def sift_up(heap, i):
    while i > 0 and heap[i] < heap[parent(i)]:
        heap[i], heap[parent(i)] = heap[parent(i)], heap[i]    # trade places
        i = parent(i)                   # and keep climbing

def push(heap, item):
    heap.append(item)                   # the list just grows on the end
    sift_up(heap, len(heap) - 1)        # at most one swap per level

# the loop stops for one of two reasons: you reached index 0, or your
# parent is already smaller than you. both mean the rule holds again.

# $ python heap.py
#   before        [3, 5, 7, 9, 12, 8, 14]
#   push(2)       [2, 3, 7, 5, 12, 8, 14, 9]      <- 3 swaps, 8 items

The loop stops for exactly one of two reasons: you reached index 0 (you are the smallest thing in the heap), or your parent is already smaller than you (the rule holds again everywhere above).

Move two: pop, and sift down

Taking the most urgent job out is the mirror image. The answer itself is free — it is always at index 0.

The work is closing the hole. A heap has to stay a complete tree with no gaps, so you take the very last item in the list and drop it into the vacated root. It is almost certainly too big to be there, so you sink it: compare against both children, swap with the smaller of them, and repeat.

# heap.py  —  pop: the answer is index 0. the work is closing the hole.

def sift_down(heap, i):
    n = len(heap)
    while True:
        smallest, l, r = i, left(i), right(i)
        if l < n and heap[l] < heap[smallest]: smallest = l
        if r < n and heap[r] < heap[smallest]: smallest = r
        if smallest == i:               # nobody below me is smaller: stop
            return
        heap[i], heap[smallest] = heap[smallest], heap[i]
        i = smallest                    # follow the value down

def pop(heap):
    last = heap.pop()                   # take the last item off the end
    if not heap: return last
    top = heap[0]                       # this is the answer we owe the caller
    heap[0] = last                      # drop the last item into the hole
    sift_down(heap, 0)                  # and sink it to where it belongs
    return top

The two length checks are not decoration. A node near the bottom may have one child or none, and reading heap[left(i)] without checking is an IndexError waiting for your production queue.

Why both moves are cheap

Both moves walk one level at a time, so the obvious question is: how many levels are there?

A complete binary tree doubles at every level — 1 node, then 2, then 4, then 8. The number of levels is therefore log₂(n): how many times you can halve the count before reaching one.

1,000,000 items → 19 levels
1,000,000,000 items → 29 levels

A thousand-fold increase in data costs you ten extra comparisons. That is the whole performance story.

The move that surprises people: heapify

Say you already have a list of a million jobs and you want a heap out of it. The obvious answer is to push them in one at a time. That is the wrong answer.

Instead, start at the last node that actually has a child — halfway through the list — and walk backwards toward the root, sifting each node down.

# heap.py  —  you already HAVE the list. do not push it in one item at a time.

def heapify(values):
    heap = list(values)
    start = len(heap) // 2 - 1          # the last node that HAS a child
    for i in range(start, -1, -1):      # walk backwards, towards the root
        sift_down(heap, i)              # everything below i is already legal
    return heap

# the second half of any list is all leaves. a leaf has no children, so it
# is already a legal heap of one item. that is why the loop skips them.

Why is this allowed? Because the second half of any list is all leaves, and a leaf has no children, so it is already a legal heap of one item. Half the work is free before you start.

Measured, building a heap from 1,000,000 items on worst-case input:

Build strategy	Swaps	Swaps per item	Time
Bottom-up `heapify`	999,988	1.00	21.0 ms
Pushing one at a time	17,951,445	17.95	322.7 ms

One swap per item against eighteen — about 15x on the clock. The reason is a counting argument, and it is the source of the classic "heapify is O(n), not O(n log n)" result: when you push one at a time, every item can climb the full height. When you build bottom-up, half the nodes are leaves and cannot sink at all, a quarter can sink one level, and only the single root can fall the full height. Sum nodes × distance across the tree and the series converges to n.

The honest part: what a heap will not do

This is where people get hurt. A heap answers exactly one question: what is the smallest thing you are holding. Everything else is a trap.

It is not a sorted list, and you cannot read it in order without emptying it.
Finding a specific item by name means scanning the whole array — precisely the O(n) cost you adopted a heap to avoid.
There is no cheap delete-in-the-middle and no cheap "change this job's priority" after it is queued.
Even the second-smallest item is not simply at index 1. It is at index 1 or index 2, and you must check both. Across 20,000 random heaps, it sat at index 2 in 8,074 of them.

If you need any of those things, you need something beside the heap — typically a dictionary from job name to array position, maintained in step. Reaching for a heap and then searching through it is the mistake this section exists to prevent.

Three guards before this ships

# scheduler.py  —  the three guards you need before this ships

import heapq, itertools
pq, counter = [], itertools.count()

# 1. TIES. two jobs at urgency 5, so python compares the JOBS themselves:
heapq.heappush(pq, (5, job))                 # TypeError: Job vs Job
heapq.heappush(pq, (5, next(counter), job))  # a counter never ties, so equal
                                             # urgency means first in, first out.
# 2. YOU WANT THE LARGEST. negate the key. do not write a second heap.
heapq.heappush(pq, (-score, next(counter), item))

# 3. TOP 10 OF A STREAM you cannot hold in memory. keep exactly 10.
for value in stream:
    if len(top) < 10:
        heapq.heappush(top, value)
    elif value > top[0]:                     # top[0] is the worst of the best
        heapq.heapreplace(top, value)        # one pop and one push, in one pass

Ties crash. You push (urgency, job). Two jobs at urgency 5 — Python compares the urgencies, finds them equal, then falls through to comparing the job objects. If your job class defines no ordering, that is a TypeError in production at 3 a.m. The fix is a monotonic counter in the middle: it can never tie, so equal urgency degrades gracefully to first-in-first-out.
You often want the largest. Negate the key on the way in and again on the way out. Do not maintain a second heap.
Top-K of an unbounded stream. Keep a heap of exactly K. Streaming 5,000,000 values for the top 10: sorting everything took 2,018 ms and held all 5M in memory; the bounded heap took 1,078 ms and held 10. Twice as fast and half a million times smaller.

The AI angle: this is your inference scheduler

If you serve models rather than web pages, you have not escaped this structure — you have just met it under different names.

Continuous batching. An LLM server holding hundreds of in-flight requests must decide, every iteration, which sequences join the next forward pass. That is a priority queue keyed on some mix of arrival time, sequence length and SLA class — and it is popped thousands of times per second, which is exactly the workload where 72 s versus 54.8 ms stops being academic.
Deadline and preemption tracking. Which request breaches its time-to-first-token budget soonest? That is heap[0], in O(1).
KV-cache eviction. When GPU memory fills, something must be evicted — least-recently-used, lowest-priority, or furthest-from-completion. "What dies next" is the canonical heap question.
Top-k sampling and beam search. Selecting the k highest-probability tokens from a 100,000-entry vocabulary is a bounded-heap problem, guard 3 above, run once per generated token.
Vector search. Approximate-nearest-neighbour indexes (HNSW and friends) maintain candidate sets as bounded priority queues throughout the graph traversal.

The lesson generalises: when a system must repeatedly answer "which one next?" under pressure, a heap is nearly always underneath. Recognising it is what lets you read a serving engine's scheduler and understand it in one pass.

Where you have already used one

You have almost certainly used a heap under a different name. Task schedulers use one to decide which job runs next. Timer systems use one to find the deadline expiring soonest. Dijkstra's algorithm and A*, the shortest-path algorithms behind map routing and game pathfinding, are a priority queue with a graph attached.

And here is where you will write one yourself: top-K over a stream, cache expiry, retry queues. Anything at all where the words are "handle the most important one first."

Python calls it heapq. Java calls it PriorityQueue. Go has container/heap. C++ has priority_queue. Same structure, same two moves.

The verdict

Question	Answer
What is it?	A flat array obeying one rule: every parent ≤ both children.
What is it for?	Repeatedly answering "what is the most urgent item?" in O(log n).
Push / pop	O(log n) — one swap per level, 19 levels at a million items.
Peek the minimum	O(1) — it is index 0, always.
Build from an existing list	O(n) bottom-up, not O(n log n). 1.00 vs 17.95 swaps per item.
Find an arbitrary item	O(n). Do not do this. Keep a side index.
Read everything in order	O(n log n) and it destroys the heap. Use `sorted()`.
Reach for it when…	"Handle the most important one first" — schedulers, timers, top-K, Dijkstra, cache expiry.
Reach for something else when…	You need ordering, lookup by key, mid-queue deletion, or priority updates.

The rule of thumb: if you catch yourself calling min() inside a loop over a collection that keeps changing, you have hand-written the 72-second version. That is the moment to reach for heapq.

Three interview questions

If this comes up in an interview, these are the three that get asked — and all three answers come from one picture: the tree above, and the same flat list below.

Why is a heap not just a sorted array? Because a sorted array pays O(n) on every insert — everything after the new item shifts over. A heap only ever orders a parent against its own children, so it pays O(log n).
Why is building a heap O(n) and not O(n log n)? Because half the nodes are leaves and cannot sink at all, and only the root can fall the full height. Sum nodes × distance and the series converges to n.
How do you get the largest instead of the smallest? Negate the key, or wrap items in a class with reversed comparison. Never keep a second heap in sync with the first.

Want the whole thing drawn, with sift up and sift down walked one swap at a time and every number above measured live? Watch the full episode on YouTube — or the 2-minute version if you just want the shape of it.

Arrays vs Linked Lists: Why the Textbook Winner Loses on Real Hardware

Vahid Aghajani — Wed, 12 Aug 2026 06:16:49 +0000

Originally published on software-engineer-blog.com.

Every data-structures course teaches the same table. Array: insert O(n), because you have to shift everything. Linked list: insert O(1), because you just repoint two pointers. Conclusion: if you insert a lot, use a linked list.

Then you write both, measure them, and the array wins anyway.

Here is the measurement that starts the whole story. Ten million integers, held as one contiguous array and as ten million linked nodes. Walk each one, summing as you go — the same operation count, the same complexity class, O(n) both:

contiguous array: 0.53 seconds (52.6 ns per element)
linked nodes: 1.53 seconds (153.3 ns per element)

2.9× apart, and nothing in the code explains it. Both loops do one add per element. Big-O says they're identical. The machine disagrees, and the reason is not in the code at all — it's in where the data physically sits.

Four ideas explain the whole gap: contiguity, the 64-byte cache line, the prefetcher, and pointer chasing.

The library, before the hardware

Imagine you need to read a hundred books.

In the array version, all hundred sit in order on one shelf. You walk up once, and you can grab an armful at a time. Your arms hold eight books, so a hundred books is about thirteen trips.

In the linked list version, each book sits on its own stand, somewhere in the building. Inside each book is a slip of paper telling you where the next one is. You cannot grab an armful, because you don't know where book two is until you've opened book one. A hundred books is a hundred separate walks — and you can't even start walking to the next stand until you've finished reading the current book.

That's the entire performance story. The rest is just naming the hardware that plays the role of "arms" and "walk."

The two structures, in code

# layout.py — the same 10,000,000 numbers, held two ways

data = list(range(10_000_000))    # ONE block: slot i sits beside slot i+1

class Node:
    __slots__ = ('val', 'next')   # 48 bytes, and no spare __dict__

def sum_array(data):              # walk the row
    total = 0
    for i in range(len(data)):
        total += data[i]
    return total

def sum_linked(head):             # follow next
    total, node = 0, head
    while node is not None:
        total += node.val
        node = node.next          # the next address lives INSIDE
    return total                  # the node you just finished reading

Look at the last line of sum_linked. The address of the next node is stored inside the node you are currently reading. You cannot know where to go next until the current fetch has completed. That single property is what costs you 2.9×, and it has a name: pointer chasing.

Where the data actually sits

The CPU never reads one integer from memory. It reads a cache line — 64 bytes, always, minimum. That's the "armful."

For a contiguous array of 8-byte values, one 64-byte fetch brings back eight useful values. You pay one trip to memory and get seven more elements for free. Walking a million elements costs you roughly 125,000 trips, not a million.

For scattered linked nodes, each node is its own allocation, sitting wherever the allocator happened to put it. One 64-byte fetch brings back one useful value — the rest of the line is that node's other fields and whatever unrelated bytes happen to be adjacent. Eight values cost eight full trips.

And the price of a trip is not a rounding error:

follow one pointer, already in L1 cache: 1.26 ns
follow one pointer, out in main memory: 81.5 ns

That's 65×. Not the folklore "a cache miss costs ~100×" — measured on this machine, it's 65. (The only ~100×-shaped real number here is 132×, and that's random pointer chase versus sequential streaming, which is a different comparison — keep that framing attached whenever you quote it.)

Then there's the prefetcher, which is the part most people never account for. When the CPU notices you walking memory in a predictable forward pattern, it starts fetching lines before you ask for them. Walking a contiguous 1 GiB array in order costs 0.62 ns per element — faster than a single L1 pointer hop, because the memory traffic is happening in the background while you compute. The prefetcher cannot help a linked list at all: it can't guess an address that hasn't been loaded yet.

What the hardware does	Contiguous array	Scattered linked nodes
Values per 64-byte fetch	8	1
Can the prefetcher help?	Yes — address is predictable	No — next address is unknown until the current load lands
Memory per 1,000,000 elements	8.2 MB (`array.array`)	80 MB
Walk 10,000,000, measured	0.53 s	1.53 s

The proof: an experiment with no linked list in it

Everything above is a story about why the linked list is slower. It could be wrong. Maybe the gap is really about node objects, or attribute lookup, or allocation count — plenty of things differ between those two loops.

So here's the experiment that isolates the cause. Take the same array. Do the same additions. Same class, same code, same object count. Change exactly one thing: the order in which you touch the elements.

walking in order: 44.4 ns per operation
walking the identical data in shuffled order: 265.8 ns per operation

~6× slower, with no linked list anywhere in the experiment. Nothing changed but the access pattern. That's the cache effect, measured on its own, with every other variable held down.

There's a matching version on the allocation side: keep the linked list, keep the class, keep the code, and shuffle only the order the nodes were allocated in. That alone is 2.1× slower — pure layout, no algorithmic difference at all.

This is the section to remember. The linked list isn't slow because it's a linked list. It's slow because it is the data structure most likely to scatter your data, and scattered data defeats every memory optimization your CPU has.

What Big-O counts, and what it never counts

Big-O is not wrong here. It's answering a different question.

Big-O counts operations, and it deliberately throws away constant factors — that's the whole point of the abstraction, and it's why it survives across machines and decades. What it throws away includes: how far the data travelled, whether the fetch hit L1 or DRAM, and whether the prefetcher was able to work ahead.

Two O(n) walks can be 2.9× apart. Two O(1) operations can be 65× apart. Big-O tells you how the cost grows; it says nothing about what one unit of that cost actually is on real silicon. For large asymptotic gaps — O(n) vs O(log n) — the growth term dominates and Big-O decides. For comparisons within the same class, it's silent, and the memory hierarchy does the deciding.

The honest other half: where linked structures genuinely win

Now the fair fight, because there is a real case here and it deserves real numbers.

Front insertion. Inserting at the head of an array means shifting every element. Inserting at the head of a linked list means writing one pointer:

array, N = 1,000,000: 572,255 ns
linked, N = 1,000,000: 243 ns

That's a genuine, enormous, structural win. It is not a rounding error and it doesn't go away on better hardware.

But that O(1) has a precondition that almost every textbook drops: you must already hold the node. Splicing after a node you have in hand is genuinely O(1). Finding that node first is not.

# insert_middle.py — the O(1) everyone quotes, with its precondition put back

def splice_after(node, value):     # O(1) — TRUE, but only if you HOLD node
    node.next = Node(value, node.next)

def insert_at(head, k, value):     # what you actually have to write
    node = head
    for _ in range(k):             # and this is not a memmove.
        node = node.next           # it is k dependent cache misses, one at a time.
    splice_after(node, value)

Measured at N = 1,000,000, inserting in the middle:

list.insert(mid, x): 255,486 ns
linked: traverse + splice: 63,488,013 ns — 249× slower

The array's O(n) shift is a memmove: one tight, prefetcher-friendly, sequential sweep that the hardware is exceptionally good at. The linked list's O(k) traversal is k dependent cache misses in a row, each one waiting on the last. Same complexity class, wildly different machine.

So the rule is not "linked lists are slow." It's: a linked structure wins when you already hold the position.

Operation, N = 1,000,000	Array / list	Linked	Winner
Walk everything	0.53 s	1.53 s	Array, 2.9×
Insert at front	572,255 ns	243 ns	Linked, huge
Insert in middle, position not held	255,486 ns	63,488,013 ns	Array, 249×
`deque.appendleft` vs `list.insert(0,x)`	25,434 ns	53.3 ns	Linked, 477×
Remove a node you already hold vs `del lst[i]`	13,692 ns	335 ns	Linked, 41×

Linked structures also give you stable references: a node's address doesn't move when its neighbours change, so pointers held elsewhere stay valid. Arrays give you no such guarantee. That property — not raw speed — is why linked structures show up inside allocator free lists, LRU caches, and intrusive kernel lists.

What your language already chose for you

Most of this decision was made for you, and it's worth knowing what you actually have:

A Python list is a contiguous array — but of pointers. The integers themselves are separate heap objects scattered elsewhere, which is why a million elements costs ~40 MB rather than 8.2 MB. You get contiguity of the references, not of the values.
array.array('q') is the true contiguous case: the 8-byte integers really do sit side by side. That's the 8.2 MB row.
collections.deque is a doubly-linked list of blocks, not of single elements. Each block holds many items contiguously, so it gets cache-friendly iteration and O(1) ends. That hybrid is why appendleft beats list.insert(0, x) by 477×, and it's the shape most "linked list" wins in production actually take.
Java's ArrayList vs LinkedList: the same story, and the reason LinkedList is near-universally discouraged in modern Java style guides.
NumPy arrays are genuinely contiguous typed memory — the reason numeric Python is fast at all.

Notice the pattern: the winning structures are blocked — contiguous runs, linked at a coarse granularity. You get sequential access inside a block and cheap restructuring between blocks.

The same rule, one layer up: why this shows up in LLM serving

If you work on inference rather than data structures, you have met this exact tradeoff wearing a different hat.

Serving an LLM means holding a KV cache — the keys and values for every token in every active sequence. Naively you allocate one contiguous buffer per sequence, sized to the maximum possible length. That's the array choice: perfectly sequential, prefetcher-friendly reads during attention, and enormous waste, because most sequences never reach the maximum and you cannot hand the slack to anyone else.

The obvious fix is to allocate per token and link them. That's the linked-list choice, and it fails for exactly the reason above: attention would then walk a pointer chain per token, turning the hottest loop in the system into dependent cache misses.

What production engines actually do — PagedAttention in vLLM being the well-known case — is the deque answer: a linked list of fixed-size blocks. Each block holds many tokens contiguously, so attention reads run sequentially inside a block, while a per-sequence block table lets memory be allocated, freed, and even shared between sequences at block granularity. Contiguity where you iterate, indirection where you restructure.

The batching story is the same shape. Continuous batching wins partly because it keeps the tensors that the GPU streams through contiguous and predictable; every gather or scatter you introduce is the shuffled-access experiment from earlier, running on far more expensive hardware. Data locality is not a CPU-era detail you've outgrown — the memory wall is wider on a GPU, not narrower.

Five questions worth being able to answer

Two O(n) loops over the same data are 3× apart. Why? Memory layout: cache lines and prefetching. Big-O counts operations, not the cost of a fetch.
When is a linked list's O(1) insert actually O(1)? Only when you already hold the node. Reaching position k first costs k dependent cache misses.
Why is an array's O(n) shift often faster than a linked list's O(k) traversal? The shift is a sequential memmove the hardware loves; the traversal is serialized misses it can't predict.
Why is deque.appendleft fast without being slow to iterate? It's a linked list of blocks — contiguous within, linked between.
When would you genuinely choose a linked structure? When you hold positions and need stable references under mutation: LRU caches, allocator free lists, intrusive kernel lists.

The verdict

Default to the contiguous array — a Python list, a Java ArrayList, a Vec, a NumPy array. It's what your hardware was built for, and the prefetcher works for free.

Reach for a linked structure when you already hold the position you're mutating and need references to stay stable while the structure changes around them.

When you need both, use the blocked hybrid — a deque, a rope, a chunked list. That's the answer nearly every production system converges on, from collections.deque to a paged KV cache.

And when a measurement disagrees with the complexity table, the measurement is not wrong. Big-O just wasn't answering that question.

Every number in this post was executed on the recording machine before it was drawn: Intel Core i5-9400F @ 2.90 GHz, L1d 32 KB/core, L2 256 KB/core, L3 9 MB shared, 64-byte cache line, CPython 3.10.12.

Watch the full 14-minute episode for the memory-row diagrams, all four code walkthroughs, and the isolation experiment step by step — or the 2-minute version if you just want the shape of it.

Processes vs Threads

Vahid Aghajani — Tue, 11 Aug 2026 15:39:46 +0000

Originally published on software-engineer-blog.com.

You run code concurrently all the time. But "concurrent" hides a critical choice: are you spawning separate processes or threads inside the same process? That choice decides whether one crash takes down your entire system or stays contained, and whether you're copying data between isolated worlds or racing to read the same memory.

Mental model: A process is its own house; threads are roommates sharing one.

Processes: Isolation at the Cost of Weight

When you start a process, the operating system hands it its own private address space. That address space is walled off. Your process can't touch another process's memory—the OS enforces it at the CPU level. If your process crashes, it corrupts only its own memory. The kernel cleans it up. Every other process keeps running untouched.

This is why browsers put each tab in its own process. One tab runs malicious JavaScript, spins into an infinite loop, or has a memory leak—that tab's process dies. The rest of your browser lives. You close the dead tab and open a new one. Your other tabs don't even hiccup.

But isolation isn't free. Each process carries:

Its own copy of the heap, stack, and memory pages
Its own file descriptor table, open sockets, and kernel resources
OS overhead to track and protect it

Spawning a process is expensive—milliseconds on modern hardware, but measurably heavier than a thread. And if two processes need to share data, they can't just read the same memory. One process must copy data into a pipe or socket, send it across, and the other process must copy it out and into its own memory. That's overhead on every exchange.

Threads: Speed and Sharing, With a Trap

Threads live inside a single process and share that process's entire memory. The kernel doesn't wall them off from each other. When you spawn a thread, you're not duplicating the heap, the file descriptors, or the kernel state—you're just creating a new stack and registering it with the scheduler.

Spawning a thread is orders of magnitude cheaper than spawning a process. And sharing data is free: both threads read and write the same variables. No copying, no pipes. If you have multi-core hardware and a job that threads can split (parsing a large file, computing in parallel), threads let you put all your cores to work on that one job instantly.

But shared memory is the trap.

When two threads touch the same data without coordination, they collide. One thread reads a 64-bit integer while another thread is writing it—the reader sees a torn write (half the old value, half the new value). Two threads increment a counter simultaneously—both read 5, both write 6, and you lost an increment. One thread modifies a linked list while another traverses it—the traverser hits a dangling pointer and crashes.

To avoid collisions, you add locks (mutexes, semaphores). Thread A locks a resource, does its work, unlocks it. Thread B waits for the lock. But now you've traded one problem for another:

Deadlock: Thread A holds lock X and waits for lock Y. Thread B holds lock Y and waits for lock X. Both freeze forever.
Lock contention: Many threads fighting for the same lock spend CPU time spinning, waiting, and context-switching instead of working.
Fragile logic: A lock protects a section of code, but a developer forgets to acquire it in one place, and you have a race condition that shows up in production only under load.

And crucially: if one thread crashes (null pointer dereference, stack overflow, segmentation fault), it brings down the entire process. No isolation. All threads die with it.

Head-to-Head Comparison

Property	Process	Thread
Memory space	Own isolated address space	Shared within one process
Startup cost	Heavy (copy memory, kernel state)	Light (just a stack)
Data sharing	Copy via pipe/socket	Direct read/write (free but risky)
One crash	Stays contained	Kills entire process
Synchronization	Message passing (inherently safe)	Locks, mutexes (race conditions, deadlock)
Scaling to cores	Hard (separate heaps)	Easy (shared memory)

A Concrete Example: Parsing a Large File

Using threads:

import threading

results = []  # Shared across threads
lock = threading.Lock()

def parse_chunk(chunk_data, chunk_id):
    parsed = parse(chunk_data)
    with lock:
        results.append((chunk_id, parsed))

threads = []
for i, chunk in enumerate(chunks):
    t = threading.Thread(target=parse_chunk, args=(chunk, i))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print(results)

Each thread parses its chunk and locks before appending to the shared results list. Cheap to spawn, instant sharing—but you must remember the lock or race.

Using processes:

from multiprocessing import Pool

def parse_chunk(chunk_data):
    return parse(chunk_data)

with Pool(4) as pool:
    results = pool.map(parse_chunk, chunks)

print(results)

Each worker process parses independently. The Pool framework collects results and sends them back. No locks, no races—each process is isolated. But the framework must serialize the chunks, send them over, deserialize, parse, serialize the results, send them back, and deserialize. That copying is the cost.

Why Python Threads Are Different: The GIL

Python's Global Interpreter Lock (GIL) means only one thread can execute Python bytecode at a time. If you spawn 10 threads in Python, they do take turns, but they don't run in true parallel on multi-core systems—the GIL ensures only one holds the interpreter at once. CPU-bound threads don't scale; I/O-bound threads do (one thread blocks on a socket read, another grabs the GIL and runs).

For CPU-bound parallel work in Python, processes (via multiprocessing) sidestep the GIL entirely. Each process has its own interpreter and its own GIL, so they run in true parallel. The cost is process creation overhead and data serialization, but you get real parallelism.

When LLM Inference Matters: Batching Over Threads

If you're serving LLM inference (or any I/O-heavy workload), concurrency is critical. Many requests come in, and you want to batch them and serve them in parallel. At first glance, threads look ideal—cheap, share memory for the model weights.

But shared memory on a model is a trap. If multiple threads try to read from the same model weights while one is loading new weights, you have data races. Most production inference engines (vLLM, Ray, TensorFlow Serving) use process isolation per request batch or per GPU, not threads, because:

Process isolation prevents one corrupted inference from poisoning the weights.
Modern inference is I/O-heavy (reading from disk, network requests)—threads excel at this but aren't the bottleneck.
GPUs naturally parallelize batches, so multi-threading inside a single process adds complexity without proportional gain.

The pattern is: use async I/O (coroutines) to queue many requests cheaply, batch them, and hand them off to a process or GPU worker. Threads between request and batch are overkill; processes between batch and inference are safety.

The Verdict

Reach for threads when: you have fast, in-memory work on shared data and need to use all your cores (numeric computation, graph traversal)—and your language doesn't have a GIL, or your work is I/O-bound. Keep lock sections tiny.

Reach for processes when: you need isolation (crash safety, multi-tenancy), you're doing CPU-bound work in Python, or you're running independent jobs (worker pools, load-balanced services).

Most real systems use both: a main process with a thread pool for I/O, or a process pool running I/O-async inside each worker, or a browser spawning tabs as processes and using threads inside each for UI rendering and JavaScript execution.

The choice isn't one or the other—it's knowing the tradeoff and picking the right tool for the layer you're building.

Watch the 90-second reel for a quick visual breakdown of this exact tradeoff.

Active-Active vs Active-Passive: You Had a Second Server, So Why Were You Down for 41 Minutes?

Vahid Aghajani — Tue, 11 Aug 2026 07:32:01 +0000

Originally published on software-engineer-blog.com.

Two in the morning. A payments service, one database, and a second identical machine sitting right next to it, copying every change as it happens. Then the first machine's disk controller dies — no warning, no slow decline, it just stops answering.

And you had planned for this. There is a second machine, right there, fully in sync. It has been in sync for ten months.

The service is down for forty-one minutes.

Not because the standby was missing. Not because the data was gone. The standby was fine the whole time. Every one of those forty-one minutes was spent switching over to it.

That's the question this article answers: you had a second server — so why were you down for forty-one minutes?

The subject: redundancy, and its two shapes

"We have a second one" is the same sentence in both designs. It means two completely different things.

Active-passive — the second machine waits. It's kept up to date, and it gets promoted when the first one dies.
Active-active — both machines take live traffic, all the time.

Two things this is not. It isn't backups: a backup is a copy of yesterday, this is a copy of right now. And it isn't horizontal scaling — that's about adding capacity, and it's a different problem. This is the shape that sits underneath every cloud provider's version of it.

One picture first

Think of a building with an emergency generator in the basement. It's fuelled. It's serviced. Once a month somebody starts it for ten minutes with nothing plugged into it, ticks a box, and switches it off. That building is active-passive.

Now think of a building fed by two power lines from two different substations, each carrying half the load, all day. One line fails and the other picks up the rest. Nothing starts up. Nothing gets promoted. The lights don't even flicker. That building is active-active.

Here's the part that matters. In the second building you already knew the second line worked — because the lights it was powering were on. The proof was the service itself. In the first building, the generator has proved it can start. It has never once proved it can carry the building.

Active-passive, in code

Two machines in a config, one named primary and one named standby. Now look at the routing function — the thing that decides where an incoming request goes:

# config.py — addresses from RFC 5737 (documentation range)
NODES = {
    "primary": {"host": "192.0.2.10", "role": "primary"},
    "standby": {"host": "192.0.2.11", "role": "standby"},
}

def route(request):
    return NODES["primary"]          # ...that is the entire function

def health_check():
    return ping(NODES["primary"])    # nobody ever asks the standby

That's it. There is no branch in it. There is no condition. The word standby appears in the config, and then it never appears again anywhere on the path a request takes.

Look at the health check too — it only ever asks about the primary. Nobody is asking the standby whether it could serve a request, because nobody ever intends to send it one. That is what passive means: the second machine is invisible to production until the day you need it.

Failover is not an instant

People say "failover" as if it were a single moment. It's a state machine, and naming the states is worth doing:

Serving — everything normal, all traffic to A.
Suspect — health checks are failing, but you're not sure yet. Was that the machine, or the network between you and the machine?
Promoted — you've committed. B is the primary now.

Then there's the transition nobody plans for: the old machine comes back. It still has your data. It still believes it's the primary. If it starts accepting writes again, you now have two primaries and two versions of the truth. That last arrow is where most real failover incidents actually go wrong.

The four steps, and what they cost

def failover():
    # 1. DETECT — three failed checks, ten seconds apart.
    #    Three, not one: a single missed check is usually a network hiccup,
    #    and promoting on a hiccup is worse than waiting.
    wait_for(consecutive_failures=3, interval=10)      # ~30 s

    # 2. FENCE — make certain the old machine cannot accept another write.
    #    Cut its network, revoke its credentials, power it off.
    #    Skipping this is exactly how you get two primaries.
    fence(NODES["primary"])                            # ~2 s

    # 3. PROMOTE — standby applies what it hadn't caught up on, opens for writes.
    promote(NODES["standby"])                          # ~15 s

    # 4. REPOINT — every client still holds the old address.
    #    Until they let go of it, the promotion changed nothing they can see.
    update_dns(ttl=60)                                 # ~60 s

Add it up, for a failover where everything is automated and nothing goes wrong: 30 + 2 + 15 + 60 = 107 seconds. Just under two minutes, and that's the good number — the one you can defend in a design review.

The number in the story at the top was forty-one minutes. And the difference between them is not technology. The difference is that two of those four steps had never been run before, so somebody had to work out what they were while the site was down.

An idle standby is an untested standby

Your standby has been sitting there, in sync, for three hundred days. In that time your service handled about two and a half billion requests, and the standby served zero of them. What has it actually proved?

A few real things, and they're worth monitoring: the process is running, replication is keeping up, there's disk space.

Now look at what it has not proved:

That the promote command works while the machine is under load.
That the address change actually reaches your clients.
That your application reconnects, instead of sitting there holding a pool of dead connections.
That the alert wakes up a human being.

Every single one of those runs for the first time during your worst hour.

Active-active, in code

Same two machines. Now both are in the pool, both have a weight, and the routing function asks which ones are healthy:

POOL = [
    {"host": "192.0.2.10", "healthy": True, "weight": 1},
    {"host": "192.0.2.11", "healthy": True, "weight": 1},
]

def route(request):
    live = [n for n in POOL if n["healthy"]]      # asks about BOTH, every second
    return weighted_choice(live)

def on_health_failure(node):
    node["healthy"] = False                       # ...that is the whole handler

Watch what the failure handler does. It sets a flag to false. There's no promote. There's no fence. There's no address change — because the address was never pointing at a machine, it was pointing at the pool.

When node B dies at 02:14, three seconds later the healthy list has one entry in it, and you're running at 50% capacity and 100% availability.

And here's the quiet part: that same line runs every single time you deploy. The code that handled your outage had already run a thousand times this month.

That's the real argument for active-active, and it isn't the one people usually give. People say it's about capacity, or being closer to users. Both are true, and both are secondary. The actual argument is that active-active deletes the failover procedure instead of trying to improve it — there's no promotion step to get wrong because nothing is promoted, no fencing problem because there's no single writer to fence, and the second machine is continuously proven to work for the simple reason that it is working. If it were broken, you'd have found out on an ordinary Tuesday afternoon, not at two in the morning.

The bill for active-active

So why does anybody still run active-passive? Because active-active isn't free redundancy. It's a trade, and you should hear the bill before you sign it.

Off your plate: no promotion, no fencing, no address change, no path that has never been executed.

Handed to you instead:

1. Two live writers, so you own conflicts

One product, ten units in stock. Two customers each buy one, forty milliseconds apart, landing on different machines:

A: read stock=10 → 10-1 → write 9
B: read stock=10 → 10-1 → write 9      # last write wins
                              stock = 9   ← two sales, one decrement

You didn't get an error. You didn't get a conflict warning. You got data that is quietly, plausibly wrong.

-- the fix: every row carries a version, and a write must say
-- which version it believes it is replacing
UPDATE inventory
   SET stock = 9, version = 8
 WHERE id = 42 AND version = 7;   -- 0 rows updated → somebody moved first → raise a conflict

That's one extra field, plus a real decision about what your application does when it fires.

2. Shared session state

Anything you were keeping in memory on one machine has to move somewhere both of them can see it.

3. Capacity planning changes

With two sites, each has to be able to carry the whole load alone — which means each one normally runs at about half.

4. Split brain

The network between your two sites breaks, but both sites are perfectly healthy and both can still reach users. Each half looks around, sees no partner, and quite reasonably concludes the other one died. So both keep serving. Both keep writing. When the link comes back you have two divergent versions of the same data and no automatic way to say which is real.

The standard answer is a quorum: you need a majority to keep accepting writes, which is why these systems are built with three nodes rather than two — so a split always leaves one side in the minority, and the side that loses the vote stops serving.

The honest comparison

	Active-Passive	Active-Active
Second machine	Idle, invisible to production	Live, serving traffic
Routing function	No branch — returns the primary	Picks from the healthy pool
Failure response	Detect, fence, promote, repoint	Set a flag to false
Best-case downtime	~107 s automated (41 min if untested)	~seconds, capacity drops to 50%
Is the spare proven?	No — first run is your worst hour	Yes — it ran today
Write conflicts	None (one writer)	Yours to solve (versioning)
Session state	Can live in memory	Must be shared
Split brain	Not asked	Needs a quorum (3 nodes)
Normal utilisation	~100% of one machine	~50% of each

What this looks like when you serve an LLM

The same two shapes decide how you run inference, and the split is unusually clean — which makes it a good way to check you've understood the rule.

Your model-serving replicas are stateless: a GPU worker holding weights takes a request, returns tokens, and remembers nothing about you afterwards. That's the free case for active-active. Put every replica behind one pool, health-check them all continuously, and a dead GPU node is a flag flip and a capacity drop — not a promotion. You also get the thing an idle standby can never give you: a spare GPU that has been proven to load the weights and answer, because it answered a minute ago. A "warm standby GPU" that has never served a token is exactly the generator in the basement, and it costs the same per hour as one that's working.

Two caveats worth naming, because they're the same bill in new clothes:

The KV cache is session state. Multi-turn conversations and prefix caching make a replica feel stateful — routing a follow-up to a different node throws away the cached prefix and you pay to prefill again. That's the "shared session state" line on the bill, and the usual answer is the usual answer: session-affinity routing, or lift the cache somewhere both replicas can see. Note that this is a performance cost, not a correctness one — you get a slower answer, not a wrong one, which is exactly why the stateless tier is still the easy case.
The stateful tier behind it is not stateless. Your vector store, your metadata database, the table you write conversations and evals into — those have all the conflict and split-brain problems from the previous section. A vector index accepting writes on two sides can silently diverge in what it returns, and "the retriever quietly stopped seeing some documents" is the RAG version of the lost update: no error, plausible output, wrong answer.

Which is the general rule, arrived at from the other direction.

So which should you build? Almost certainly both

Not one for the whole company — one per layer.

Web servers, API processes, model replicas — anything that doesn't remember anything between one request and the next: run those active-active. They have no conflicts to have, so you get all of the upside and pay none of the bill.
Your main database, the single source of truth — active-passive is usually the honest answer. One writer, one version of the truth, a replica standing ready. That is not a compromise, that's the right tool.

Anyone who tells you their entire system is active-active is describing the front of it.

The sentence to keep

You are not choosing how many machines you own. In both designs, you own two.

You are choosing whether failing over is a routine or an event.

Active-passive makes it an event: a procedure that runs about once a year, at the worst possible moment, carried out by a tired person reading a document. Active-active makes it a routine: a flag that flips, on a path that already ran today.

And if you do keep an active-passive tier — which you probably should — then the only honest thing to do is to fail over to it on purpose, on a Tuesday afternoon, while everybody is awake.

Hash Tables Explained in Detail: Why a Dict Lookup Beats a List Scan by 62,000x

Vahid Aghajani — Mon, 10 Aug 2026 10:12:11 +0000

Originally published on software-engineer-blog.com.

Two hundred thousand users in a list. A login handler that looks for one email. It works, it passes review, and it costs 4.4 milliseconds — which sounds like nothing until two thousand people sign in per second and the box is suddenly doing 8.8 CPU-seconds of work every second. Nothing crashes. Nothing shows up in the logs. The service just stops answering.

The identical lookup out of a Python dict takes 71 nanoseconds. About 62,000 times faster, on the same data and the same machine — and not because the dictionary searches faster.

Mental model: A hash table never searches. It computes where the answer lives. The list looks at a hundred thousand rows; the dictionary looks at one.

This is unit 11 of "the missing CS degree", and it is the one data structure worth understanding properly, because you already use one in every dict, every set, and every "have I seen this before" check you have ever written.

The code you would write first, and it is not wrong

users = load_users()              # 200,000 rows, in a plain list

def find_user(email):
    for u in users:               # one string compare, per row, in order
        if u["email"] == email:
            return u              # average: 100,000 comparisons
    return None                   # worst case: all 200,000

# $ python bench.py
#   find_user(...)          4.419 ms     <- per login, average row
#   2,000 logins/second ->  8.8 CPU-seconds of work, per second

There is nothing bad in there. It is correct, it is readable, and for a list of twenty it is the right answer. The problem is the shape of the cost: double the users, double the work, every time, forever. That shape is called O(n), and changing it is the entire reason hash tables exist.

Build one, do not import one

The whole idea fits in nine lines, and only one of them matters:

# toytable.py  —  the whole idea, in nine lines

class HashTable:
    def __init__(self, size=8):
        self.buckets = [[] for _ in range(size)]   # 8 empty slots
        self.n = 0                                 # how many keys we hold

    def _index(self, key):                         # <- THIS is the hash table
        return hash(key) % len(self.buckets)       # any key  ->  0..7

    def get(self, key):
        for k, v in self.buckets[self._index(key)]:   # ONE slot. never the rest.
            if k == key:
                return v
        raise KeyError(key)

hash() is built into Python: hand it a string, it hands back a large integer. Modulo the number of buckets folds that integer down into the range 0 to 7. Key in, slot number out — that one line is the hash table. Everything else in the file is bookkeeping.

Note what get does not do: it never touches the other seven slots. That is where the 62,000x comes from.

Eight names, eight slots — and the collision is not a bug

You are folding billions of possible numbers into eight slots. Of course two of them land in the same place:

h("grace")    =        98,615,224   ->  % 8 = 0
h("alan")     =         2,996,632   ->  % 8 = 0
h("edsger")   =     2,987,424,544   ->  % 8 = 0

Three keys, one slot. Meanwhile slots 2, 3 and 5 got nothing at all. Neither of those is a defect — it is arithmetic. The question is never how to avoid a collision. It is what you do when one happens.

The answer here is chaining: a slot does not hold one entry, it holds a little list.

# toytable.py  —  set: the two lines that resolve a collision

    def set(self, key, value):
        bucket = self.buckets[self._index(key)]
        for pair in bucket:
            if pair[0] == key:        # this exact key is already in the slot
                pair[1] = value       # so it is an update, not an insert
                return
        bucket.append([key, value])   # a DIFFERENT key, same slot: it queues up
        self.n += 1

# >>> ht.buckets[0]
# [['grace', 2], ['alan', 4], ['edsger', 5]]    <- three keys, one slot
# >>> ht.get('alan')
# 4                                             <- 2 compares, not 200,000

All of collision handling is that one append. Getting a key back means: the right slot, then a short walk down its list. Short is the word doing the work — two or three, not two hundred thousand.

Load factor: the number that decides whether any of this stays fast

A hash table is fast while the chains are short. The load factor is what keeps them short:

load factor  =  keys stored  ÷  slots available

Eight keys in eight slots is a load factor of 1.0. It is the average number of keys sitting in a slot, which is the same thing as the number of comparisons a lookup costs. So it is not a tidiness metric — it is your lookup cost, written as one number.

Measured over 100,000 real keys:

load factor	average chain	longest chain
0.75	1.4	7
1.0	1.6	7
2.0	2.3	10
8.0	8.0	21

Eight times as crowded costs eight times the comparisons, and it degrades smoothly — which is exactly why nobody notices until it matters.

This is also why the textbook writes O(1) as "amortised, average case". The average case is a statement about the load factor, and the load factor is not a fact about your data. It is something you actively keep down.

The resize, and why every key has to move

# toytable.py  —  the four lines that keep the O(1) true

    def set(self, key, value):
        ...
        bucket.append([key, value])
        self.n += 1
        if self.n / len(self.buckets) > 0.75:    # over three quarters full
            self._resize()                       # so make the table bigger

    def _resize(self):
        old = self.buckets
        self.buckets = [[] for _ in range(len(old) * 2)]   # twice as many slots
        for bucket in old:
            for k, v in bucket:
                self.buckets[self._index(k)].append([k, v])   # EVERY key moves

It cannot copy the pairs across into the same positions. _index divides by the number of buckets, and that number just changed:

h("grace") % 8  = 0
h("grace") % 16 = 8      <- same key, same hash, different slot

So every key belongs somewhere new, and re-inserting all of them is what "rehashing" means. That occasional O(n) pass, spread over all the cheap inserts that preceded it, is the amortised in "amortised O(1)".

On our eight keys, the three-deep chain in slot 0 became a one and a two. That is all a resize ever does: it buys the chains room by making the divisor bigger.

Chaining vs open addressing — you use both every day

	Chaining	Open addressing
On a collision	The slot holds a list; the new pair is appended to it	Probe on to another slot by a fixed rule — there are no lists
Memory layout	Every entry costs a pointer, and the chain is scattered	Cache-friendly — probing stays inside one block of memory
Deleting	Simple: unlink it from the list	Genuinely awkward — you have to leave a tombstone behind
Who uses it	Our toy table, Java's `HashMap`; Go chains overflow buckets	Python's `dict`, and Ruby's `Hash`

Neither one avoids collisions. Nothing avoids collisions. They are two different answers to "this slot is taken".

When O(1) becomes O(n) — and the attack built on it

One bad hash function turns your hash table straight back into the list you replaced:

4,000 keys	longest chain	lookup
an ordinary hash	4	0.4 µs
every key forced into slot 0	4,000	74.5 µs

186 times slower. Same table, same keys, same machine — the structure is identical, only the hash changed.

And if an attacker can guess your hash function, they can send you exactly those keys on purpose. That is a real denial-of-service class, and it is why CPython randomises the hash of every string at start-up.

Three details about the dict you already use

CPython uses open addressing, not chaining, and grows the table at roughly two thirds full.
Since 3.7 a dict keeps insertion order — that comes from a second, compact array of indices, not from anything about hashing.
A key has to be hashable, which in practice means immutable: {[1, 2]: "x"} raises unhashable type: list.
hash(1), hash(1.0) and hash(True) are all 1 — so {1: "a", 1.0: "b", True: "c"} is a dict with one key, holding "c".

The same trade-off in LLM serving

This is not a legacy-backend concern that AI work has moved past. The inference stack is held together by hash tables, and by their one limitation:

Prefix / KV-cache reuse. Serving engines like vLLM cache blocks of KV state and look them up by a hash of the token prefix. Two requests that share an opening system prompt hit the same block and skip recomputing it. That lookup has to be O(1) on every single request, so it is exactly the structure above — and the "hash keys must be deterministic" rule is why block hashing has to be stable across replicas or the cache silently stops hitting.
Tokenizer vocabulary. Every token-to-id lookup is a hash map, run tens of thousands of times per second. It is the definition of "cheap hash function matters".
Exact-match response caching and dedup. "Have I answered this exact prompt before?", "have I already embedded this chunk?" — both are set membership, which is the one thing a hash table is unbeatable at.
And what it cannot do is why vector search exists. Hashing deliberately destroys the relationship between keys. A semantic cache needs the nearest prompt, not the identical one, and "nearest" is precisely the query a hash table cannot answer. That is why retrieval runs on an ANN index and not a dict — the same reason a database index is a B-tree.

The rule transfers exactly: hash for identity, something ordered for proximity.

When a hash table is the wrong choice

Unbeatable at	Cannot do it at all
Is this exact key present?	Every key between 10 and 50
The value for this exact key	The smallest key, or the next key after this one
Have I already seen this?	Anything starting with "sm", or sorted output
Dedup, caches, sets, indexes by id	The nearest match, or a predictable worst case

Hashing throws away order to buy you speed, and sometimes order was the thing you needed. It costs memory, too: a thousand integers in a list is about 8 kilobytes; a thousand pairs in a dict is about 37.

The whole thing in one breath

A hash table turns a key into a slot number by arithmetic instead of searching, which is why a dict lookup is 71 nanoseconds where a list scan is 4.4 milliseconds. Folding a huge number space into a small array must produce collisions, so a slot holds a short chain (or probes on to the next slot). Chains stay short only while the load factor stays low, which is why the table doubles at about three quarters full and rehashes every key — the divisor changed, so every key belongs somewhere new. That occasional O(n) pass is the "amortised" in amortised O(1), and a bad or attacker-chosen hash collapses the whole thing back to O(n).

Verdict

Reach for a hash table by default for anything keyed by identity: lookups by id, membership, dedup, caches. It is the best constant-factor win in everyday code, and the mental model is one sentence — it computes the address, it does not search for it.

Reach for something else the moment the question involves order or proximity: ranges, prefixes, min/max, sorted scans, nearest match. Name the replacement out loud — a B-tree, a sorted array, a trie, an ANN index — because in an interview, and in a design review, that is the half of the answer most people leave out.

Watch the full 13-minute walkthrough for the table built line by line, the collision and the resize on screen, and five real interview questions with an answer skeleton for each — or the 2-minute version for the short of it.

Write-Through vs Write-Back Caching — Explained in Detail

Vahid Aghajani — Sun, 09 Aug 2026 11:43:32 +0000

Originally published on software-engineer-blog.com.

A customer fixes their delivery address at 20:31. Green tick, "Saved." At 20:32 a routine deploy restarts the service. At 08:14 the next morning the parcel ships to the old address. Nothing crashed. The log tail is clean: zero errors, zero alerts, zero exceptions.

Every explanation of caching teaches the read path — you ask for something, the cache has it, you skip the database, everything is fast. Almost nobody tells you what happens on a write. That is where this class of bug lives.

Mental model: A cache is not a shortcut. It is a second copy. The only thing a write strategy decides is which copy becomes true first — and how long the other one is allowed to be wrong.

The code you already have

Most services never chose a write strategy. They chose a read strategy — cache-aside — and left the write path empty:

def get_order(order_id):
    hit = cache.get(order_id)          # the read path everyone writes
    if hit is not None:
        return hit
    row = db.fetch_one(order_id)
    cache.set(order_id, row)
    return row

def update_address(order_id, addr):
    db.execute("UPDATE orders SET addr = ? WHERE id = ?", (addr, order_id))
    db.commit()
    # ...and nothing at all happens to the cache

That is a write strategy — an accidental one. The database is right, the cache is stale, and the staleness lasts until the key expires. Which is exactly how a "saved" address gets served back as the old address.

Write-through: the database first, then the cache

Write-through makes both copies move together. Commit the durable write, then update the cache — in that order, always:

def update_address(order_id, addr):
    db.execute("UPDATE orders SET addr = ? WHERE id = ?", (addr, order_id))
    db.commit()                        # durable first
    cache.set(order_id, addr)          # then the second copy
    return "saved"

The two copies can never disagree for longer than the microsecond between those lines. If the process dies after commit(), the database is already correct and the cache just misses. Nothing is lost.

The bill comes on the hot path. Measured on a real database, not estimated:

operation	cost per write
durable write (commit + fsync)	5.948 ms
the same value into an in-memory cache	0.0012 ms
durable writes per second, one connection	171

Write-through means every single change — forever — pays that ~6 ms, and your write throughput per connection is capped somewhere near 171/s. For a settings page nobody notices. For a counter updated on every request, that ceiling is your outage.

Write-back: write the cache, say yes, flush later

Write-back (also called write-behind) inverts it. The cache takes the write, the user is told yes immediately, and a background flusher pushes the changes to the database in batches:

dirty = {}                             # key -> newest value not yet in the DB

def update_address(order_id, addr):
    cache.set(order_id, addr)
    dirty[order_id] = addr             # remember it needs flushing
    return "saved"                     # the user is told YES, right here

def flusher():                         # background, every 5 seconds
    while True:
        time.sleep(5)
        batch, dirty_now = list(dirty.items()), dict(dirty)
        dirty.clear()
        with db.transaction():
            for key, value in batch:
                db.execute("UPDATE orders SET addr = ? WHERE id = ?", (value, key))

Acknowledgements now cost about a microsecond instead of about six milliseconds, and the database absorbs a burst as a trickle.

The trap

Write-back does not make the write faster. It makes the write a promise.

For the window between the acknowledgement and the flush, the only copy of that fact is in RAM, on one machine. Here is the loss, reproduced:

>>> save_write_back("o-4471", "Bahnhofstrasse 12")
'saved'                                  # the user watched this succeed
>>> cache["o-4471"]
'Bahnhofstrasse 12'

# ...the process restarts before the flusher runs. RAM is gone.

>>> db.execute("SELECT addr FROM orders WHERE id='o-4471'").fetchone()
None                                     # not stale. NOT THERE AT ALL.

Not a stale row. No row. The same sequence under write-through returns the address, because the durable write happened before the user was ever told yes.

That is the whole failure at the top of this article: an ordinary deploy, inside the durability window, and a confirmed change that never existed.

Why anyone accepts that

Because of coalescing. A hot key updated 900 times does not need 900 durable writes — the dirty map keeps only the newest value, so the flush writes it once:

	one-at-a-time	batched behind a dirty map
database writes for 900 updates to one key	900	1
wall clock	5276.8 ms	13.3 ms (396× faster)

For a cart quantity, a session, a counter, a last_seen timestamp, that is an enormous win and the risk is genuinely acceptable — nobody is harmed if five seconds of "last seen" evaporate. For a confirmed delivery address or a payment record, the same trade is indefensible.

The durability window is a number you choose

Write-through's answer is "zero milliseconds." Write-back's answer is "however long until the next flush" — and if you cannot say that number out loud, you have not chosen it, you have inherited it.

Two guards make write-back safe to actually run:

MAX_DIRTY = 10_000

def update_address(order_id, addr):
    if len(dirty) >= MAX_DIRTY:        # 1. bound the queue
        flush_now()                    #    backpressure, never unbounded RAM
    cache.set(order_id, addr)
    dirty[order_id] = addr
    return "saved"

def flush_lag_seconds():               # 2. measure the real window, live
    return time.time() - last_successful_flush_at

The bound stops a flusher that has fallen behind from turning into an out-of-memory kill that takes every unflushed write with it. The metric is the one number to put on a dashboard and alert on: it is your data-loss exposure, in seconds, right now.

Write-around: the third strategy

Write-around writes the database and skips the cache entirely:

def import_row(order_id, addr):
    db.execute("UPDATE orders SET addr = ? WHERE id = ?", (addr, order_id))
    db.commit()
    cache.delete(order_id)             # invalidate, do not populate

It exists for writes that nobody is going to read soon — a nightly import, a bulk backfill. Populating the cache with a million rows that no user will request just evicts the keys people are reading, and your hit rate falls off a cliff right after the import finishes.

The same decision in LLM serving

This is not a legacy-backend concern. Every inference stack in production runs the same three strategies under different names:

A semantic cache in front of a model is cache-aside on the read. If you write a corrected answer or a moderation verdict into it and defer the durable write, that is write-back — and a pod restart loses the corrections you believe you shipped.
KV-cache and session state for a multi-turn conversation are the textbook write-back case: enormous update rates on one key, coalescing wins, and nobody is harmed by losing the last few seconds of an ephemeral attention state.
Usage and token accounting, on the other hand, is money. Batch it for throughput if you must, but the flush lag on that queue is the number of billable tokens you are willing to lose. That belongs on a dashboard, not in a comment.

The rule transfers exactly: batch what is cheap to lose, commit what is expensive to lose, and know your window.

Strategy	What happens on a write	Data-loss window	When to use
Write-through	Durable commit first (~6 ms), then the cache	Zero — the answer is true when you say it	Anything a user is told succeeded: addresses, orders, payments, permissions
Write-back	Cache + dirty map, acknowledge in ~1 µs, flush in batches	Up to the flush interval — seconds of confirmed data	High-rate, low-value state: counters, sessions, carts, last-seen
Write-around	Durable commit, cache invalidated and not populated	Zero (durability); costs a read miss	Bulk imports and backfills nobody will read back soon
Nothing (accidental)	Database updated, cache untouched	Zero loss, but unbounded staleness until TTL	Never on purpose — this is the bug, not a strategy

The whole thing in one breath

A cache is a second copy, so every write has to decide which copy becomes true first. Write-through pays a durable ~6 ms on every change and can never lose a confirmed write. Write-back acknowledges in microseconds and coalesces 900 updates into one database write, but for the length of its flush interval the only copy of a confirmed fact lives in RAM on one machine — and an ordinary restart in that window deletes a change your user watched succeed. Write-around keeps bulk writes from evicting your hot keys. Bound the dirty queue, export the flush lag, and you have turned an invisible risk into a number.

Verdict

Default to write-through. It is boring, it costs about six milliseconds, and "saved" means saved.

Reach for write-back deliberately, for a specific hot key whose last few seconds are genuinely disposable — and only with a bounded dirty map and a flush-lag metric on a dashboard.

The question is never "is my cache fast." It is: how many milliseconds of confirmed data am I willing to lose?

Watch the full 11-minute walkthrough for all of the code and the loss reproduced live, or the 2-minute version for the short of it.

Airflow vs Cron: When a Crontab Line Isn't Enough

Vahid Aghajani — Thu, 06 Aug 2026 07:51:28 +0000

Originally published on software-engineer-blog.com.

"Just use cron" and "we need Airflow" are both wrong about half the time.

They look like the same tool — both run your jobs on a schedule — so teams pick one out of habit and pay for it later. But scheduling isn't the real difference. The real difference is whether your work is one command on a clock or a graph of tasks that depend on each other.

The one-line mental model:

cron asks "is it time yet?"
Airflow asks "what's the state of my whole pipeline?"

cron: one line, one clock

cron is a time-based scheduler that has shipped on essentially every Unix box for about forty years. You give it a time expression and a command. That's the entire interface.

# min hour dom mon dow   command
0    2    *   *   *      /opt/etl/run.sh

At 02:00 every day, cron runs the script. There is no daemon to operate, no database to back up, no dashboard to log into. It is one of the most battle-tested pieces of software you will ever depend on, and for a single timed job nothing is simpler or more reliable.

The important thing to notice is what cron knows: the clock, and nothing else. It does not know what your command does, whether it succeeded, or whether anything else depends on it.

Where the crontab starts to hurt

The failure mode shows up the moment you have a second step. Say load must run after extract. cron has no way to express "after", so you do the thing everyone does — you guess a gap:

0    2    *   *   *      /opt/etl/extract.sh
10   2    *   *   *      /opt/etl/load.sh    # extract "should" be done by now

That works right up until the night extract runs long, or crashes. cron does not care. At 02:10 it fires load exactly on schedule — against stale data, partial data, or no data at all. And you will not find out from cron, because:

No dependency awareness. 02:10 is a wish, not a guarantee.
No retries. A transient network blip means the run is simply lost until tomorrow.
No backfill. The box was down for two days? Those two days never happen.
No observability. "Did it run?" is answered by SSH-ing in and grepping logs.

None of these are bugs. cron does exactly what it promised. You just asked it a question it was never designed to answer.

Airflow: the pipeline is the unit

Airflow attacks the other question. Instead of scheduling commands, you describe your work as a DAG — a directed acyclic graph of tasks, where the edges are dependencies:

from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="0 2 * * *", start_date=datetime(2026, 1, 1), catchup=True)
def daily_etl():

    @task(retries=3)
    def extract():
        return fetch_rows()

    @task()
    def transform(rows):
        return clean(rows)

    @task()
    def load(rows):
        write_warehouse(rows)

    load(transform(extract()))

daily_etl()

The last line is the whole point. load does not run at a time — it runs when transform has succeeded, which runs when extract has succeeded. The schedule only starts the graph; the graph decides the rest.

That single change buys you the entire list cron was missing:

Dependencies are declared, not approximated with a ten-minute gap.
Retries are a keyword argument (retries=3), applied per task.
Backfills are first-class: catchup=True and Airflow will run the windows it missed, because each run is tied to a data interval rather than to "now".
Observability is a web UI showing every run, every task, every log, colour-coded green and red.

And the bill for that

Airflow is not a binary you drop in /usr/local/bin. It is a system you now operate:

a scheduler process that must stay up,
a metadata database (Postgres) holding every task state,
a queue (Redis/Celery) or a Kubernetes executor,
a pool of workers,
and the upgrades, migrations and on-call that come with all of it.

To fire one nightly python etl.py, that is pure overhead — you have replaced a line of text with a distributed system, and now the scheduler itself can page you at 3 a.m. The joke that writes itself: you cannot run Airflow to make sure Airflow is running.

Side by side

Dimension	cron	Apache Airflow
Core question	"Is it time yet?"	"What's the state of my pipeline?"
Unit of work	One command	A DAG of tasks
Dependencies	None — approximated with time gaps	Declared in the graph
Retries	Whatever you hand-roll in the script	Built in, per task
Missed runs / backfill	Lost forever	Backfilled by data interval
Observability	grep the logs over SSH	Web UI, run history, per-task logs
Operational cost	Zero — already installed	Scheduler + Postgres + queue + workers
Scale-out	One box	Worker pool / Kubernetes
Best fit	A single timed job	A multi-step pipeline with state

The same split shows up in LLM pipelines

If you are building AI systems rather than classic ETL, don't assume this is a data-engineering-only question — the orchestration lane is where most RAG systems quietly rot.

A retrieval pipeline is a DAG whether you admit it or not: fetch sources → chunk → embed → upsert into the vector store → rebuild the index → smoke-test retrieval. Every one of those steps depends on the one before it, and two properties of LLM workloads make cron a particularly bad fit:

Model APIs are flaky and rate-limited. An embedding job over 200k chunks will hit a 429 or a timeout at some point. With Airflow that's retries=5 plus exponential backoff on one task; with cron the whole nightly script dies and you find out when answers go stale.
Re-embedding is a backfill problem. Switch embedding models, change your chunk size, or fix a parser bug, and you need to re-run the pipeline over a historical range of documents. That is exactly what a backfill is — and cron has no concept of a past window at all.

The failure is also nastier than a missed report. If embed fails and upsert runs anyway on a partial batch, you don't get an error — you get a vector store that is silently missing a slice of your corpus, and a chatbot that confidently answers "I don't have information about that." Partial state in a retrieval index looks exactly like a working system.

The heuristic carries over cleanly: a nightly "re-embed the docs folder" script is fine on cron. A multi-source ingestion pipeline feeding a production RAG endpoint — with evaluation gates and index swaps — wants a real orchestrator.

Verdict

Reach for cron when the work is genuinely one command on a clock: a nightly backup, a certificate renewal, a single script that is idempotent and cheap to re-run. Adding an orchestrator there buys you nothing and hands you a scheduler, a database and a queue to babysit.

Reach for Airflow (or Dagster, or Prefect — the model is the same) the moment step B needs step A to have succeeded, or you need retries, backfills, and a straight answer to "what ran last night?". The tell is simple: the first time you write a crontab line whose correctness depends on guessing how long the previous line takes, you have outgrown cron.

One command on a clock, or a graph of tasks with state. Match the tool to the shape of the work — not to what's trendy.

Watch the 2-minute version for the whole comparison in one pass.

25 Years of Object Detection in 4 Walls (Sliding Window YOLO DETR)

Vahid Aghajani — Wed, 05 Aug 2026 08:29:31 +0000

Originally published on software-engineer-blog.com.

A detector must emit a list of unknown length. A classifier cannot—one picture in, one label out, a fixed shape. This mismatch drove twenty-five years of object detection, and the field hit it four times in four different ways.

Mental model: Each breakthrough solved a hard constraint the previous era exposed, then revealed a new one. The real pivot wasn't architecture—it was the loss function, and whether you remove duplicates before or after training starts.

Wall 1: The Sliding Window (1990s–2011)

You cannot slide a classifier everywhere—it's too slow and it says WHAT, not WHERE. So the early field asked the easy question: "Is there an object in this 32×32 patch?" Then asked it everywhere. A 640×640 image yields roughly 145,000 overlapping windows. For each one, humans hand-drew features (edges, histograms, gradients). Each window ran through a trained classifier. Each detection got a confidence score.

The cost was computational. The ceiling was accuracy: hand-drawn features lose to learned ones.

Wall 2: Learning Features, Keeping Regions (2012–2014)

2012: AlexNet proved a neural network could learn better features than humans could design. ImageNet error dropped from ~26% to ~15%. But AlexNet is still a classifier—it sees one 224×224 image, emits one label.

R-CNN's insight (2014): Use a region proposal algorithm to narrow the search space from 145,000 windows to ~2,000 candidate regions, then run AlexNet on each one. The regions came from selective search (a hand-tuned algorithm). The mAP on PASCAL VOC 2012 jumped from 35.1 to 53.3.

The cost was speed: two stages, two forward passes per region. The ceiling was latency—video and real-time inference stayed out of reach.

Wall 3: One Stage, One Forward Pass—and a Hard Trade-off (2015–2019)

YOLO (2015) solved speed: one forward pass, one output tensor. No region proposals. No selective search.

But here is what nobody says about one-stage detectors: they do not emit a list. They emit a fixed grid of candidate boxes. A modern YOLO at 640×640 resolution emits roughly 8,400 candidate boxes—always. Before it sees the image. The shape is baked into the architecture.

Most of those boxes describe the same dog. To filter them down, every detector for twenty years ended the same way: non-maximum suppression (NMS). Greedy loop:

# Pseudocode: NMS is not learned, not differentiable
keep = []
while boxes:
    best = boxes.pop(highest_confidence)
    keep.append(best)
    boxes = [b for b in boxes if iou(best, b) < iou_threshold]  # Hand-picked threshold
return keep

NMS has three fatal flaws:

It is not learned. No gradient flows through it during training.
The IoU threshold is a human-picked number, retuned per dataset.
In a crowd, it deletes real people, because real people overlap.

YOLO v1–v5 and their descendants (SSD, RetinaNet, EfficientDet) all paid this bill. Faster inference came at the cost of a post-processing hack that was brittle and blind to the actual training signal.

Wall 4: Predict a Set, Not a Grid (2020–Present)

DETR (Detection Transformer, 2020) reframed the problem: you want a set out, so predict a set.

100 learned query slots. Each slot outputs one bounding box and one class label (or the special class "no object"). Always 100 queries. Always a hundred outputs.

The variable-length problem from wall one is gone—you have a fixed shape. But now you have a different problem: how do you train it? You have 100 predictions and maybe 3 real objects. Which prediction should match which object?

The breakthrough was the loss function, not the architecture. DETR uses the Hungarian algorithm:

# Pseudocode: Hungarian matching in the loss, not post-processing
cost_matrix = compute_costs(predictions_100, ground_truth_3)  # 100 × 3
best_matching = hungarian_algorithm(cost_matrix)  # one-to-one pairing

# Train the 3 matched predictions to be correct.
# Train the 97 unmatched predictions to output "no object".
# A second query that also finds the dog is punished during training.
# Duplicates are never created.

No hand-picked IoU threshold. No greedy loop. No post-processing that ignores the loss. Duplicates do not get removed—they are never trained to exist.

The cost was high: DETR needed 500 epochs to converge (versus ~100 for YOLO), and small objects were harder to detect. The next four years of research paid down both.

Where It Landed (Mid-2026)

By mid-2026, both families had converged on the same architectural shape:

One-stage detector (no separate region proposal network).
No hand-tuned anchors.
No NMS in the default pipeline.

YOLO26 ships NMS-free by default, using a Decoupled Head with soft spatial and class matching in the loss (inspired by DETR). RF-DETR (real-time DETR) is a transformer detector with architectural tricks (RoI queries, token pruning) to match YOLO's inference speed.

On the COCO benchmark, they tie:

Model	COCO mAP	Parameters	Approach
YOLO26x	57.5	55.7M	Grid-based, learned matching in loss
RF-DETR-L	56.5	129M	Set-based, Hungarian matching in loss

What still differs is the backbone: the feature extraction before the head. YOLO's backbone is trained on supervised detection data. RF-DETR's uses DINOv2, a self-supervised vision model trained on a billion unlabeled images. COCO is a tie because the head (grid vs. set) is now irrelevant—the bottleneck moved downstream.

This is why COCO scores alone are the wrong lens to pick a detector. You have to ask: which backbone suits your data and your label budget?

For LLM Inference: Why This Matters to Serving

If you serve detectors as part of a larger AI system (e.g., grounding for a vision-language model), the shift from NMS to learned matching changes latency profile:

YOLO with NMS: Inference time is deterministic. NMS is O(n log n) on the number of detections, but it runs after the network, so you cannot batch or fuse it with the next module.
DETR / NMS-free: All computation is in the neural network. Batch processing and quantization apply uniformly. No separate post-processing stage to stall on edge devices.
Hungarian matching (training only): Only happens during training. At inference, you just pick the 100 predictions; no matching step. But the 100 queries are learned to avoid duplicates, so you spend fewer cycles filtering.

For real-time serving, prefer NMS-free detectors (YOLO26, RF-DETR) because they fuse into quantized execution graphs and avoid the post-processing bottleneck.

Verdict

Reach for YOLO when you need simplicity and real-time speed on single-backbone setups; reach for DETR (or RF-DETR) when you want to plug in a better backbone (like DINOv2) and have latency headroom for transformer compute. The head stopped mattering in 2024. The backbone and your label budget are what differ now.

Watch the 17-minute deep dive for the full chain, the numbers, and the trade-offs.

Ollama vs vLLM: Local Dev Tool or Production Inference Engine?

Vahid Aghajani — Fri, 31 Jul 2026 16:35:46 +0000

Originally published on software-engineer-blog.com.

Ollama and vLLM both let you "run a large language model locally." Ask either one to serve 200 concurrent users on a single GPU, and the resemblance ends.

Mental model: Ollama is a single-user dev tool that trades throughput for simplicity; vLLM is a production inference server that trades setup complexity for multi-user concurrency.

What Ollama Does

Ollama wraps llama.cpp and quantized GGUF weights into a single command:

ollama run llama3

That's it. No Python environment to configure, no CUDA dependencies to compile, no batch-size tuning. The model downloads, loads, and starts answering you in seconds on your laptop's GPU. Ollama is optimized for developer friction—the fastest possible path from zero to "the model is running and I can ask it a question."

Under the hood, Ollama serves requests largely one at a time. When fifty concurrent API calls arrive, they don't run in parallel. They queue. Latency climbs. The GPU idles waiting for responses. The tool that "just works" for a solo developer becomes a bottleneck the moment real traffic hits.

What vLLM Does

vLLM is built around two architectural ideas that flip Ollama's trade-off:

PagedAttention manages GPU memory like an operating system pages RAM. Instead of allocating one contiguous block per request (wasting memory when sequences are short), vLLM divides the KV cache into fixed-size pages and reuses them across requests. One GPU holds many more concurrent requests.

Continuous batching slots new requests into the running batch mid-flight instead of waiting for a batch to finish. Request A doesn't have to complete before Request B starts generating tokens. They generate in parallel, packed efficiently into the same GPU compute.

The result: one GPU serving hundreds of concurrent requests, exposed through an OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token-unused")

response = client.completions.create(
    model="meta-llama/Llama-2-7b",
    prompt="Explain inference optimization",
    max_tokens=512
)

The cost of that throughput: you must provide a real GPU with real VRAM, install Python and CUDA dependencies, and configure the server deliberately—model path, tensor parallelism, GPU memory fraction, batch size. Even asking one solo question on your laptop means paying the full production setup tax.

They're Not Competitors—Different Axes

	Ollama	vLLM
Primary User	One developer, local machine	Production API serving N concurrent users
Request Handling	Largely sequential (queue)	Continuous batching (parallel)
GPU Memory Model	Fixed allocation per request	PagedAttention (page-based)
Setup Friction	One command, zero config	Python, CUDA, deliberate tuning
Concurrent Requests per GPU	~1–5	~50–500+
Best For	Local prototyping, tinkering	Production inference APIs
If You Pick Wrong	Production traffic queues, GPU underutilized	Local experiments pay ops overhead

The Inference Lens

In LLM serving, throughput depends on how you handle two latencies:

TTFT (time to first token): latency before generation starts.
TPOT (time per output token): latency per token generated.

Ollama's sequential queue inflates both. A second request waits for the first to finish generating all its tokens before the GPU even touches the prompt. vLLM's continuous batching decouples them: Request B's prompt can be processed while Request A is still generating output tokens, and both requests' output generation runs in parallel on the same GPU. One GPU's TPOT is divided among N concurrent requests.

The Verdict

Reach for Ollama when you're prototyping alone—you want model inference in one command with zero fuss. Reach for vLLM when you're serving real traffic—you need the GPU utilization, latency scaling, and concurrency that continuous batching and PagedAttention provide.

Don't use Ollama for production. Don't use vLLM for local tinkering.

Your Turn

You're serving one fine-tuned model to 200 concurrent API users on a single GPU. Ollama or vLLM—and what specifically breaks if you pick wrong?

Watch the 90-second reel for a visual walkthrough of this breakdown.

Autoscaling Explained: Why New Servers Always Arrive Two Minutes Too Late

Vahid Aghajani — Fri, 31 Jul 2026 06:37:02 +0000

Originally published on software-engineer-blog.com.

Your dashboard says autoscaling worked. Your users say the site was down. Both are right.

That contradiction is not a misconfiguration and it is not your fault. It falls straight out of how the mechanism works. Autoscaling is a reaction, not a prediction — and the reaction is structurally late by exactly one boot cycle.

Let's build the whole loop from first principles on one running example: LunchRun, an office lunch-ordering site. Everything below is cloud-agnostic — no orchestrator, no vendor product names. Every provider implements the same four moving parts.

First principles: nothing "senses" that you are busy

There is no magic in autoscaling. No component feels strain. A machine gets added for exactly one reason:

That's it. Everything else is detail about which number, where the line is, and how long it takes to act. And it is that last part — the latency — where all the pain lives.

Step 1 — something counts a number

A control loop samples a metric on a fixed interval, typically about every 60 seconds. The default metric almost everywhere is average CPU across the group. It is popular because it is free and universal, not because it is good.

CPU only guesses at pain. A machine can sit at 45% CPU while every request waits 4 seconds on a saturated database connection pool. CPU says "fine". Users say "broken".

Metric	What it actually measures	Fails when
Average CPU	How busy the processor is	The bottleneck is I/O, locks, or a connection pool — CPU stays low while latency explodes
Request rate	Incoming traffic volume	Requests get more expensive; the same rate now means twice the work
Queue depth	How many requests are waiting	Rarely — a growing queue is the pain, not a proxy for it
p95 latency	What users experience	It is a lagging signal — by the time it moves, you are already behind

If you can export it, scale on queue depth. The number of requests piling up in front of your workers is the closest thing to a direct measurement of "we do not have enough capacity right now."

Step 2 — you draw the line

The rule is a threshold plus a confirmation window:

IF  average CPU > 70%
FOR 2 consecutive samples   (≈ 2 minutes)
THEN launch 1 instance

The two-sample requirement is not padding. One sample over the line is a blip — a garbage collection pause, a backup job, a bot burst. Acting on a single sample gives you a scaler that thrashes on noise. So you deliberately buy stability with 60 more seconds of delay.

Then there are the bounds, and both of them cost you:

Minimum too low. LunchRun idles on 2 machines overnight to save money. At 11:55 the rush starts. The scaler has to climb from 2, one boot cycle at a time — and the spike is over before help finishes arriving.
Maximum too high. A retry storm or one runaway bot looks exactly like real demand. The group happily scales to 40 machines and hands you the bill. The ceiling is not a performance setting, it is a blast radius.

The heart of it: launched is not serving

This is the part that no marketing page states plainly. The moment the scaler decides to act, the clock does not stop. A newly launched machine is not capacity. It is a promise of capacity.

Stage	What happens	Typical cost
Boot	Hardware allocated, kernel and OS come up, network attaches	~30s
Image pull	Your container image is downloaded and unpacked	~25s
App start + warm	Runtime starts, config loads, connection pools and caches fill	~20s
Health checks	Two consecutive passes before it is trusted	~15s
Total	Only now does the load balancer route it one request	~90s

Every one of those numbers is a lever. A 1.2 GB image pulls slower than a 200 MB one. An app that eagerly warms a 50-connection pool starts slower than one that lazily opens connections. Two health checks 15 seconds apart are safer and slower than one. You are not stuck with 90 seconds — but you are never at zero.

The gap

Now add up the whole chain, from the first user who felt slowness to the first request served by new capacity:

 60s   notice   — the sample interval; the spike began just after a sample
 60s   confirm  — the second consecutive breach
 90s   boot     — launch → serving
─────
~210s  ≈ 3.5 minutes

Those three and a half minutes are carried entirely by the machines you already had. Not partially. Entirely. During the gap your existing group absorbs 100% of the load, and whatever it cannot absorb is queued, slowed, or dropped.

For LunchRun that is the whole lunch rush. Orders climb, latency climbs, timeouts start, and at 12:03 the new machines finally go healthy — right as the traffic curve flattens on its own. The autoscaling graph afterwards looks perfect: demand rose, capacity rose, both came down. It just did so several minutes after it mattered.

Scaling back down: the flapping trap

Scale-in has its own failure mode, and it is a loop.

Traffic drops. The scaler removes a machine. But the remaining machines now split the same work between fewer of them, so average CPU rises — possibly straight back over the scale-out threshold. The scaler adds a machine. CPU falls. It removes one. That is flapping: paying for boot cycles all afternoon while capacity oscillates.

The fix is a cooldown — after any scaling action, ignore the metric entirely for a few minutes and let the system settle before judging it again. Scale-in is normally made deliberately more conservative than scale-out: quick to add, slow to remove. Removing capacity you turn out to need costs a full boot cycle to undo.

The consequence: a reaction, not a prediction

Put the pieces together and the conclusion is structural, not situational:

Autoscaling is always late by exactly one boot cycle.

Which means it is genuinely good at one thing and useless at another:

✅ The ramp. Traffic that builds over 10–20 minutes — a workday warming up, a campaign spreading, a gradual regional shift. The loop tracks it comfortably and you barely notice it working.
❌ The spike. Ten times the traffic in 20 seconds. A push notification to 500,000 phones. A link hitting the front page. A thundering herd of clients all retrying at the same instant. That load is served entirely by the capacity you already had, because the reaction has not finished happening yet.

"Just autoscale it" was never an answer to a thundering herd.

What actually saves you sits in front of the scaler

If the scaler is structurally late, the absorption has to come from somewhere else. Three things, and none of them are scaling rules:

1. Headroom. Run the fleet at ~50% utilization, not 90%. That spare capacity is not waste — it is a boot cycle in the bank, prepaid. It is the only thing that can respond in zero seconds because it is already booted, already warm, already receiving traffic. Headroom is the single highest-leverage knob here, and it is an explicit money-for-safety trade you should make on purpose.

2. A queue. Put a buffer between accepting work and doing work. LunchRun takes the order immediately and confirms it; the kitchen works through the backlog at its own pace. The queue converts a failure into a delay, and slow beats failed by an enormous margin. It also gives you the best scaling metric you will ever have, for free: the queue's own depth.

3. Pre-warming on a schedule. The most under-used technique in the list. Lunch is at 12:00 every single day. Scale up at 11:45 because you know, not because a metric noticed. Any spike you can predict — business hours, a scheduled campaign, a Friday deploy, a known batch job — should never be handled reactively at all. This is the only mechanism in the whole toolkit that is genuinely a prediction.

The same math, with a much worse constant: autoscaling model serving

Everything above gets sharper the moment the thing you are scaling is an inference service, because the boot ladder gets dramatically longer.

A stateless web app pulls a 200 MB image. An LLM replica has to pull an image and load tens of gigabytes of weights into GPU memory, allocate a KV cache, and often run a compile or graph-capture warmup pass before the first token comes out at a sane speed. The "launched is not serving" stage stops being ~90 seconds and becomes 2–10 minutes. The gap does not shrink — it multiplies.

The metric choice matters more too. GPU utilization is an even worse proxy than CPU: a server can look 90% utilized while running one wasteful batch, or 40% utilized while requests wait. The signals that actually track pain in a serving stack are the queue-shaped ones:

Pending request / queue depth — the same principle as above, and the standard scaling signal for inference gateways.
Time to first token (TTFT) — climbs as soon as requests start waiting for a batch slot, which is exactly the early warning you want.
Time per output token (TPOT) — degrades when batches get too large, telling you the opposite story: you are past the good part of the throughput curve.

And the mitigations map one-for-one. Headroom becomes keeping spare replicas or spare KV-cache blocks. The queue becomes continuous batching plus an admission queue, which is why a well-run inference server degrades into "everyone waits a bit longer" instead of "everyone gets a 503". Pre-warming becomes keeping a warm pool of loaded replicas for known traffic patterns — the entire reason serverless GPU platforms sell "keep N warm" as a headline feature. It is the identical structural problem, just with a boot cycle long enough that nobody can pretend the reactive loop will save them.

The verdict

Autoscaling is a control loop, and every control loop has a response time. Yours is roughly: sample interval + confirmation window + boot time. Write that number down for your own system — it is usually between two and five minutes, and most teams have never measured it.

Then design around it honestly:

Scale on queue depth if you possibly can. CPU guesses at pain; a queue is the pain.
Shrink the boot ladder — smaller images, faster startup, lazier warmup, tuned health checks. Every second cut is a second the gap shrinks.
Set the maximum as a blast radius, not as an ambition.
Give scale-in a cooldown, or you will pay to watch capacity oscillate.
Buy headroom on purpose, and stop treating 50% utilization as waste.
Pre-warm everything you can predict — that is the only part of your system that isn't reacting.

Autoscaling is not there to save you from a spike. It never was. It is there so the ramp doesn't require a human, and so the buffer you built in front of it never has to stretch further than one boot cycle.

Watch the reel: Autoscaling — why it is always ~2 minutes too late · Follow along on Telegram.

Debouncing vs Throttling: Only the Last Event, or the Ones in Between?

Vahid Aghajani — Thu, 30 Jul 2026 15:34:47 +0000

Originally published on software-engineer-blog.com.

Ten keystrokes in a search box fire ten input events — and ten API calls, nine of them wasted before the tenth even lands.

That's the bug everyone has written once. The fix everyone reaches for is "just debounce it," and in a search box that fix is correct. Which is exactly why so many people think debounce is the answer to every noisy event stream. It isn't.

The real question is never "how do I fire fewer events?" The event itself is free — it's a function call in the browser's event loop. What isn't free is the work behind it: the network round trip, the layout recalculation, the write to disk. So you need a rule for which events become work. There are two rules, and they disagree.

Debounce waits for the stream to go quiet, then fires once.
Throttle lets one through, then ignores the rest until a fixed interval has elapsed.

Neither one is a timer that runs on its own. No events, no calls.

Debounce: wait for silence

Every incoming event cancels the pending callback and restarts the clock. The callback only survives to run if nothing interrupts it for the full delay.

function debounce(fn, delay = 300) {
  let timer;
  return (...args) => {
    clearTimeout(timer);              // cancel whatever was pending
    timer = setTimeout(() => fn(...args), delay);  // restart the clock
  };
}

const search = debounce((q) => fetch(`/api/search?q=${encodeURIComponent(q)}`), 300);

That's the whole mechanism — six lines, clearTimeout and setTimeout. Type ten characters quickly and you get one request, carrying the final query. Perfect, because in a search box only the last keystroke is real. The nine intermediate queries were never answers anyone wanted.

What debounce costs you

Move the same code to a drag handler and the trade-off flips into view. Drag a slider and you get exactly one call — after you let go. Which means that for the entire duration of the drag, nothing happens. No preview, no live value, no progress. The UI looks frozen, and users read frozen as broken.

Then there's the failure mode that most explanations skip entirely:

Debounce guarantees a call after silence. It guarantees nothing if silence never arrives. A user typing steadily for 60 seconds without a single 300 ms pause triggers zero saves. Wrap autosave in a plain debounce(save, 300) and you have shipped a feature that, under exactly the conditions it exists for — sustained writing — saves nothing at all.

Debounce can starve forever. Throttle cannot.

Throttle: watch the clock, not the silence

Throttle ignores whether the stream is quiet. It keeps a timestamp and asks one question per event: has the interval elapsed?

function throttle(fn, interval = 300) {
  let last = 0;
  return (...args) => {
    const now = Date.now();
    if (now - last >= interval) {     // window open?
      last = now;
      fn(...args);
    }
    // else: drop this event on the floor
  };
}

const onDrag = throttle((e) => updatePosition(e.clientX), 100);

Same 60 events a second, but now the handler fires steadily throughout the drag — it tracks your finger. Progress is guaranteed regardless of how long the input lasts, because the clock advances whether or not the user pauses.

What throttle costs you

The naive timestamp gate above is leading-edge only: it runs on the first event of each window and drops everything after. So the final event of the stream — the one that carries where the user actually stopped — arrives inside a window that's already closed, and gets dropped with the rest.

You end up storing a position that is one step stale, permanently. The slider snapped to 47 on screen, your backend thinks 43. That's why every real throttle implementation adds a trailing call: when the window closes, fire once more with the most recent arguments if any were dropped.

Side by side

	Debounce	Throttle
Trigger	Silence — N ms with no new event	The clock — one call per N ms
Mechanism	`clearTimeout` + `setTimeout` on every event	Timestamp gate (`Date.now() - last >= interval`)
Calls during a 5s burst	0 (then 1 at the end)	~5s / interval, spread evenly
Guarantees progress?	No — can starve under continuous input	Yes — fires regardless of pauses
Sees the final event?	Yes, always — that's the point	Only with a trailing call bolted on
Failure mode	Frozen UI mid-stream; zero calls if input never pauses	Permanently one step stale (leading-edge only)
Reach for it when	Autocomplete, validate-on-stop, resize settle	Drag, scroll, autosave, progress, telemetry

The rule worth memorising

Every framework, every language, same rule:

Ask that question about your handler and the choice answers itself. Autocomplete? Only the final query is a real query — debounce. Drag preview? Every intermediate position is a frame the user is watching — throttle. Autosave? The intermediate states genuinely matter and the stream may never pause — throttle, or debounce with a maxWait so it can't starve.

That maxWait option in lodash's debounce isn't decoration. It's the escape hatch for exactly the autosave bug above: fire after 300 ms of silence, but never go longer than 5 s without firing, whatever the user does.

The same two rules, one layer up: LLM and AI apps

This isn't a frontend-only pattern. The moment your app calls a model, the "event is free, the work behind it isn't" asymmetry gets much sharper — an input event costs nothing, an embedding call plus a vector search plus a generation costs money and hundreds of milliseconds.

Debounce the query, not the keystroke. RAG-backed search-as-you-type should never embed on every character. Debounce at 250–400 ms so one embedding + retrieval fires per thought, not per letter. Only the last query is a real query — textbook debounce.
Throttle the token stream into the UI. A streaming response arrives as many small chunks per second. Re-rendering markdown on every token burns main-thread time for frames nobody perceives. Throttle the render to ~30–60 ms — the intermediate tokens matter (that's the whole point of streaming), just not all of them. And you must add the trailing call, or the last chunk lands in a closed window and the answer renders permanently truncated by a few words. This is the leading-edge bug, shipped.
Throttle outbound calls to respect provider rate limits. A token-bucket throttle in front of the provider client is what keeps a burst of agent tool-calls under the requests-per-minute ceiling. Debounce would be actively wrong here — it would collapse a queue of distinct, all-necessary calls into one.
Debounce-with-maxWait for agent state checkpoints. Persisting conversation or agent scratchpad state on every step is wasteful; debouncing it alone risks a long-running agent that never pauses and therefore never checkpoints — the autosave starvation bug, now with an hour of lost work behind it.

Same two questions, bigger blast radius: does only the final one matter, and can this stream ever fail to pause?

Verdict

Debounce and throttle are not competing implementations of one idea — they encode two different beliefs about your event stream.

Use debounce when intermediate events are noise on the way to a real one: autocomplete, validation on stop, resize settle, search-as-you-type embeddings.

Use throttle when intermediate events are themselves the product, or when the stream might never go quiet: drag, scroll, progress, telemetry, streaming-token rendering, rate-limited API calls.

And whichever you pick, handle its cost explicitly — a maxWait on debounce so it can't starve, a trailing call on throttle so it can't go stale. Both are six lines of plain setTimeout and Date.now(). Reaching for the lodash import before you can write either one by hand is how the autosave bug ships.

Watch the reel: Debouncing vs Throttling in 100 seconds — the drag-handler test and both failure modes, animated.