DEV Community: Chen Yuan

Private AI Inference with Homomorphic Encryption: A Practical Guide to Computing on Encrypted Data

Chen Yuan — Sat, 15 Aug 2026 12:23:41 +0000

In 2009, Craig Gentry proved that it is possible to compute on encrypted data without ever decrypting it, and the result was widely treated as a theoretical curiosity. Sixteen years later, homomorphic encryption has crossed from conference papers into production pipelines: banks screen transactions against encrypted watchlists, hospitals run diagnostic models on data that never leaves their custody, and in August 2026 Google announced private AI features built on the same primitives. The gap between "possible in theory" and "usable in practice" is still wide, but it is no longer an argument against trying. This guide walks through what homomorphic encryption actually computes, how the CKKS scheme turns encrypted vectors into a workable substrate for machine learning, and the cost model that decides whether a private inference pipeline is worth building at all.

The Promise: Compute Without Reading

Ordinary encryption has a hard property: a ciphertext reveals nothing about the plaintext. AES-CTR, ChaCha20, RSA — all of them scramble data so thoroughly that an attacker holding the ciphertext and a supercomputer cannot recover the message without the key. That property is also the problem. If a server stores customer data encrypted at rest, every query requires shipping the data (or the key) somewhere a human or a process can read it. The moment the data is decrypted for computation, the confidentiality boundary moves from the storage layer to the memory of whatever process is doing the work.

Homomorphic encryption changes the terms. A homomorphic scheme is one where operations on ciphertexts correspond to operations on plaintexts: Enc(a) ⊕ Enc(b) = Enc(a + b). A server can add, multiply, and combine encrypted values and return the encrypted result, and the client — the only party holding the key — decrypts the final answer. The server learns nothing about the inputs, the intermediate values, or the output. For inference, this is the entire ballgame: the model owner never exposes weights, and the data owner never exposes the query.

Why Plain Encryption Breaks Computation

To see why this is hard, consider what AES does to a single byte. The S-box substitution and the ShiftRows/MixColumns rounds mix the input so completely that flipping one plaintext bit changes roughly half the output bits. That avalanche effect is what makes AES secure, and it is exactly what makes it unusable for computation. There is no way to run a + b on AES ciphertexts because the algebraic structure of the ciphertext has no relationship to the algebraic structure of the plaintext.

Fully homomorphic encryption takes the opposite approach. Instead of starting with a scrambling cipher and hoping arithmetic survives, it starts with an algebraic structure that naturally supports both operations. The classic construction works over polynomial rings: plaintexts are small polynomials, ciphertexts are pairs of larger polynomials, and addition and multiplication in the ring map to addition and multiplication of the encrypted messages. Security comes not from destroying structure but from noise — every operation grows a random error term that must stay small enough for decryption to still recover the message. Multiply too many times and the noise drowns the signal.

From Gentry's Bootstrapping to CKKS

Gentry's 2009 breakthrough had two parts. The first was the observation that a scheme with bounded noise can still be made unbounded: if the decryption circuit is shallow enough, the server can evaluate it homomorphically, producing a fresh ciphertext with reset noise. That process, bootstrapping, was the theoretical missing piece. The second part was a working scheme with a decryption circuit shallow enough to bootstrap. The catch was performance — early bootstrapping took minutes per operation.

The decade after Gentry produced a family of practical schemes. BGV and BFV handle encrypted integers with exact arithmetic. TFHE (CGGI) works over encrypted bits and is fast enough for small circuits like database lookups and comparisons. And CKKS, published by Cheon, Kim, Kim, and Song in 2017, introduced approximate arithmetic over encrypted real numbers, with a noise budget that behaves like floating-point error. That last property is what made machine learning viable: neural networks already tolerate small numerical errors, so a scheme that treats noise as precision loss fits the workload instead of fighting it.

CKKS also inherits a trick called SIMD packing from its predecessors. A single ciphertext in CKKS is not one number but a vector of hundreds of slots, and operations apply element-wise across the whole vector. A dot product — the inner loop of almost every inference step — becomes one ciphertext multiplication and a few rotations instead of hundreds of separate operations.

How CKKS Works Under the Hood

The construction is a polynomial ring R = Z[X] / (X^N + 1), typically with N = 4096 or 8192. Plaintexts are polynomials of degree less than N whose coefficients are the numbers you actually care about, scaled by a large factor and rounded. The scale factor is the scheme's version of fixed-point arithmetic: it sets how many bits of fractional precision survive a multiplication. After each multiply, the scale of the result grows, and the scheme provides a rescaling operation that brings it back down.

Noise is the real constraint. Each addition adds noise linearly; each multiplication roughly squares it. The scheme allocates a noise budget at encryption time, and every operation spends some of it. When the budget hits zero, decryption produces garbage. This is why parameter selection matters more in FHE than in any other part of a machine learning stack: the ring degree sets the maximum vector size and the top of the noise budget, the scale factor sets precision, and the multiplication depth you need sets how many levels the chain must provide before bootstrapping becomes necessary.

A minimal encrypted vector in Python, using a research-oriented binding, looks like this:

from openfhe import CKKSRNS, SecurityLevel, ScalingTechnique

# A fresh CKKS context with 40 levels of multiplicative depth
params = CKKSRNS()
params.SetMultiplicativeDepth(40)
params.SetScalingModSize(50)
params.SetSecurityLevel(SecurityLevel.HEStd_128_classic)

cc = CKKSRNS.GenCryptoContext(params)
cc.Enable(PKE)
cc.Enable(KEYSWITCH)
cc.Enable(LEVELEDSHE)
cc.Enable(ADVANCED)

keys = cc.KeyGen()
cc.EvalMultKeyGen(keys.secretKey)   # allow encrypted multiplication
cc.EvalRotateKeyGen(keys.secretKey, [1, 2, 4, 8])

plain = cc.MakePackedPlaintext([0.5, 1.5, 2.5, 3.5])
ct = cc.Encrypt(keys.publicKey, plain)

That is the entire setup. ct is now a ciphertext the server can store, transform, and return without ever seeing the values inside.

A Minimal Private Inference Pipeline

The classic private inference flow is split between a client that owns the data and a server that owns the model. The client encrypts its input, the server evaluates the model homomorphically, and the client decrypts the result. For a logistic regression — a single affine transform followed by a sigmoid — the encrypted computation is small enough to show in full:

def encrypted_predict(ct_x, ct_w, ct_b, cc, keys):
    # Encrypted dot product: one multiply, one add over packed slots
    ct_z = cc.EvalAdd(cc.EvalMult(ct_x, ct_w), ct_b)

    # Approximate the sigmoid with a low-degree polynomial
    # because division and exp are not directly supported
    sigmoid_approx = (
        0.5
        + 0.197 * ct_z
        - 0.004 * cc.EvalMult(ct_z, ct_z) * ct_z
    )
    return sigmoid_approx

Three details matter here. First, the sigmoid must be replaced by a polynomial approximation such as a Taylor or Chebyshev expansion, because CKKS supports only addition and multiplication. Second, the approximation degree is a direct trade against the noise budget: every extra multiply spends a level. Third, the whole vector of slots is processed in parallel, so one encrypted call classifies an entire batch of inputs, not a single sample.

For deeper networks, the same recipe repeats layer by layer: convolution becomes a sum of shifted and multiplied ciphertexts, ReLU becomes a polynomial like x^2-based approximation or a TFHE-style comparison, and pooling becomes rotations plus additions. The engineering problem is not expressing the model — it is keeping the depth inside the budget.

The Real Cost Model

Homomorphic inference is slow, and the honest framing is to say how slow and why. Ciphertexts are two or three orders of magnitude larger than the plaintexts they hold. Multiplication on a packed ciphertext is roughly a thousand to ten thousand times more expensive than the equivalent plaintext float operation, depending on ring degree and security level. Bootstrapping, when the noise budget runs out, costs on the order of tens of milliseconds to seconds per ciphertext — cheap enough to amortize over a packed batch, ruinous if applied per element.

The practical consequence is that private inference shifts the bottleneck from model quality to arithmetic budget. A model that runs in 2 milliseconds on plaintext floats can take seconds homomorphically, and the gap is dominated by the number of multiplications per slot, not the model's parameter count. Architectures that are friendly to FHE are the ones that keep multiplicative depth low: shallow MLPs, quantized networks, models with polynomial activations. Deep transformer stacks with attention and softmax are the worst case, because attention is a softmax followed by matrix products — softmax division is not a native operation, and its polynomial replacement is expensive.

When Private Inference Makes Sense

The honest answer is that homomorphic inference earns its cost in a narrow but real set of situations. The first is regulated data with a shared computation: hospitals collaborating on a model where patient records cannot leave each institution's boundary, or banks running joint fraud models over accounts they are legally barred from sharing. The second is API-based inference where the query itself is the secret — legal research, medical symptom triage, proprietary financial signals — and the client does not want the server to see what it is asking. The third is the emerging pattern behind the August 2026 announcements: consumer AI where the provider wants to process a user's data without being able to read it, as a product differentiator rather than a regulatory requirement.

What homomorphic encryption does not buy you is protection against a malicious server. A server that controls the evaluation can still drop requests, return garbage, or measure timing and access patterns. FHE guarantees confidentiality of the data against the server's curiosity, not integrity of the result against the server's malice. Teams that need both must add zero-knowledge proofs or commit-and-reveal protocols on top.

Choosing a Library

The ecosystem has consolidated around a few serious options. Microsoft SEAL is the reference implementation of BFV and CKKS in C++, with Python bindings via the pybind11-based extensions; it is battle-tested but expects the caller to manage parameters. OpenFHE is the community successor, actively maintained, and adds BGV, TFHE, and a unified API across schemes. TenSEAL, built on SEAL, provides a Pythonic API for CKKS over PyTorch tensors, though its maintenance has slowed. Zama's Concrete implements TFHE with a compiler that takes plain Python functions and emits bootstrapped circuits, which suits smaller integer workloads like lookups and decision trees. For a new project, the practical default is OpenFHE for custom CKKS work, and Concrete when the target workload is small and integer-shaped.

# OpenFHE exposes the same context pattern across schemes,
# so the pipeline above ports to BGV with two line changes:
params = BGVRNS()
params.SetMultiplicativeDepth(20)
cc = BGVRNS.GenCryptoContext(params)

The parameter selection itself is a black art that libraries are only beginning to automate. The rule of thumb is: choose the ring degree from the vector size and security level, choose the scaling modulus from the precision you need, then count the multiplicative depth of your exact model graph and add headroom for the approximations. Getting this wrong shows up not as a crash but as silent precision loss at the output — which makes an end-to-end test with known plaintexts the most important step in any FHE project.

A Decision Framework for Your First Private Inference Pilot

A useful heuristic before committing engineering time: if the plaintext model fits on a single machine and the deployment is a one-off computation, homomorphic encryption is probably the wrong tool — shipping the data under an agreement is simpler. If the computation is recurring, the inputs are sensitive, and the participants do not trust each other enough to share plaintexts, the calculus flips. Start with a single layer, measure the noise budget after every operation, and instrument the number of slots used per ciphertext. Most teams discover that the bottleneck is not the cryptographic primitives but the model's activation functions, and that replacing one ReLU with a polynomial buys more than a faster library ever could.

The field is also moving faster than its reputation suggests. Ciphertext compression, GPU kernels for CKKS multiplication, and programmable bootstrapping have each cut the effective cost by an order of magnitude within the last few years. The 2009 result was a proof that encrypted computation is possible; the current state of the art is a proof that it is affordable for the workloads where confidentiality is actually worth money. For anyone building AI products on other people's data, that is a gap worth watching — and a pilot worth running.

Originally published on Dispatch.

Why Your LLM Classifier Doesn't Need the Taxonomy: Hypothetical Classification with Embeddings

Chen Yuan — Fri, 14 Aug 2026 14:47:39 +0000

Classifying free-text queries into a fixed product taxonomy is one of the most common LLM workloads in production, and one of the most quietly expensive ones. A typical e-commerce catalog ships with 400 to 800 legal categories, each spelled out as a fully qualified path like Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables. Every time a search query needs a label, the whole vocabulary has to travel with the prompt: as a giant Pydantic Literal, as a JSON schema, or as a few hundred lines of enum text. Tokens are not free, and neither is latency. The prompt grows, the small model you wanted to use starts misbehaving, and the request that should have cost a fraction of a cent now needs the biggest model on the roster just to keep the output valid.

There is a cheaper pattern that most teams never consider. Instead of forcing the model to choose from the real taxonomy, you let a small, cheap model invent fake categories that sound plausible, and then resolve those hallucinations back into the real vocabulary with an embedding lookup. It sounds backwards. It works surprisingly well. This article walks through the idea, the implementation, the failure modes, and the measurements you should collect before trusting it.

The Standard Answer: Structured Outputs and the Taxonomy Tax

The conventional approach is structured outputs. You define the legal vocabulary as a type, hand it to the provider, and ask for a constrained decode:

from typing import Literal
from pydantic import BaseModel, Field

FullyQualifiedClassifications = Literal[
    "Furniture / Bedroom Furniture / Beds & Headboards / Beds",
    "Furniture / Living Room Furniture / Chairs & Seating / Accent Chairs",
    "Rugs / Area Rugs",
    # ... times 500
]

class QueryClassification(BaseModel):
    """Structured representation of a search query."""
    classifications: list[FullyQualifiedClassifications] = Field(
        description="Possible classifications for the product."
    )

Constrained decoding guarantees that the answer is drawn from the legal set, which is a strong property. But it carries a hidden cost that grows with the catalog. The schema has to be sent with every request, so every call pays the full vocabulary in input tokens. Classification is a high-volume, low-value-per-request workload, which is exactly the wrong place to spend tokens.

The second problem is that the guarantee is only as good as the model's willingness to stay inside the schema. Small models choke on a 500-element Literal; they either truncate it, or they start hallucinating categories that were never in the list, which defeats the entire purpose. The teams I have watched hit this wall usually respond by upgrading the model, and the unit economics of the whole pipeline follow them upward.

The Trick: Ask the Model to Make Things Up

The key realization is that the model does not need to know the taxonomy to know what kind of thing a query is talking about. A query like "brown coffee table" is obviously a piece of living room furniture. The hard part is not understanding the query; it is mapping that understanding onto a specific node in a tree with hundreds of leaves.

So stop sending the tree. Send an example of the shape of a classification, and ask the model to invent new ones:

hallucination_prompt = f"""
Your task is to create novel, never seen before furniture, home goods,
or hardware classifications that best fit a search query.

Product classifications look like:
Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables
Décor & Pillows / Decorative Pillows & Blankets / Throw Pillows
Furniture / Bedroom Furniture / Dressers & Chests
Kitchen & Tabletop / Kitchen Organization / Food Storage & Canisters
Baby & Kids / Toddler & Kids Bedroom Furniture / Kids Beds

Here is the query to generate classifications for:
brown coffee table
"""

The prompt now fits in a few hundred tokens instead of several thousand. The model has no legal vocabulary to violate, so it will happily produce a made-up path like Furniture / Living Room / Tables / Coffee. That output is useless as a label by itself. But it is a precise description of the query's meaning, written in the same style as the real taxonomy.

Resolving the Fake into the Real

Now the problem becomes: given a hallucinated path, find the real category it points at. Embeddings make this a nearest-neighbor search instead of a string-matching problem, which matters because the fake path and the real path share almost no literal characters.

The setup is small enough to live in memory. Embed every real category path once at startup with a compact sentence model, keep the vectors in a NumPy array or a tiny vector store, and embed the hallucinated path at request time. The label is the real category whose vector has the highest dot product with the fake one:

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
real_paths = [...]  # the full taxonomy, loaded once
real_vectors = model.encode(real_paths, normalize_embeddings=True)

def resolve(hallucinated_path: str) -> str:
    v = model.encode([hallucinated_path], normalize_embeddings=True)[0]
    idx = int(np.argmax(real_vectors @ v))
    return real_paths[idx]

For a catalog of a few hundred categories, this is a single matrix-vector product, well under a millisecond even on a laptop CPU. The expensive part of the pipeline is now the tiny LLM call, and that is the whole point: you have moved the cost from a big model with a giant schema to a small model with a short prompt plus a local lookup that costs nothing.

Why This Works: What Small Models Are Actually Good At

The pattern sounds like it should fail, so it is worth being precise about why it does not. Small models are bad at constrained generation over large vocabularies: their attention degrades as the constraint set grows, and their sampling drifts into invalid outputs. But they are good at open-ended paraphrase and at writing text in the style of an example. Inventing a plausible category path for "brown coffee table" is a style-matching task, not a constraint-satisfaction task. It is exactly the kind of fluent generation that even a 1B-parameter model does reliably.

The embedding model, meanwhile, is the component that actually holds the taxonomy. Sentence embeddings map both the fake path and the real paths into a space where meaning, not surface form, determines distance. Furniture / Living Room / Tables / Coffee and Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables share almost no tokens, but they land close together because the embedding space was trained on paraphrases. Each component does the part it is good at, and neither needs to be strong at the other's job.

When Cheap Models Are Enough, and When They Are Not

The pattern is not a universal replacement for structured outputs; it is a trade. You should measure three failure classes before committing.

The first is vocabulary drift. A small model inventing categories will sometimes produce a path that is semantically generic, like Furniture / Tables, which resolves to whichever real node is closest in the embedding space. If your taxonomy has many tables, the nearest neighbor may be wrong even though the query was specific. This is the dominant failure mode, and its rate is a property of your taxonomy, not of the model.

The second is resolution ambiguity. Some queries are genuinely under-specified: "black stand" could be a phone stand, a monitor stand, or a plant stand. The hallucinated path will be confidently wrong because the model had no way to know. This is not a bug in the pattern; structured outputs fail the same way, but they fail by returning a confidently wrong label with a schema-valid appearance.

The third is taxonomy churn. If your catalog changes often, the embedding index must be rebuilt, and a newly added category is invisible until then. Rebuilding a few hundred embeddings takes seconds, so this is an operational detail rather than a blocker, but it needs to be part of the deploy pipeline.

The pattern shines when the vocabulary is large, the queries are short, and you want the classification step cheap enough to run on every request without thinking about the bill. It is a poor fit when the taxonomy is tiny, when labels must be exact strings with no tolerance, or when you cannot tolerate any rate of misclassification and prefer the deterministic failure of constrained decoding.

Guardrails: Handling Unresolvable Hallucinations

In practice you want a confidence floor on the resolution step, not just an argmax. The dot product between the hallucinated vector and its nearest real neighbor tells you how convinced the system is. A path that resolves at 0.45 cosine similarity is a guess; one that resolves at 0.85 is a lock. The cheap fix is a threshold with two fallback tiers:

def classify(query: str, threshold: float = 0.6) -> tuple[str, str]:
    fake = invent_categories(query)          # small LLM, short prompt
    real, score = resolve(fake)              # embedding lookup
    if score >= threshold:
        return real, "auto"
    return run_structured_classification(query), "fallback"

Below the threshold you can escalate to the structured-output path on a bigger model, or route to a human review queue. Because the cheap path handles the bulk of traffic, the escalation path only fires on the ambiguous fraction, and your average cost stays low even though the worst case still uses the expensive machinery. A second guardrail worth adding is a short denylist of generic leaves that are known to swallow unrelated queries, so a generic hallucination can be rejected before it resolves to a useless bucket.

Measuring Quality Without a Labeled Set

The objection to this pattern is usually "how do I know it is accurate?" You can get surprisingly far without a hand-labeled test set. Sample a few hundred real search queries from your logs, run both the cheap path and the structured-output path on the same queries, and diff the labels. Disagreement does not mean the cheap path is wrong, but every disagreement is a case to eyeball. In practice the disagreements cluster into two piles: cases where the cheap path is semantically right and the schema path was forced into a wrong but valid leaf, and cases where the cheap path resolved to a generic neighbor. The first pile is evidence the pattern is working; the second pile tunes your threshold.

Track the resolution score distribution as a metric. If the mean score drifts down over weeks, the model vendor changed something or your taxonomy drifted; either way you want to know before the misclassification rate moves. A small dashboard with three numbers, mean score, escalation rate, and disagreement rate against the structured path, is enough to run this pattern in production with confidence.

Putting It Together

The complete pipeline is short enough to hold in your head: a tiny model invents a category path from a prompt of a few hundred tokens, an embedding lookup resolves that path to a real leaf in under a millisecond, and a similarity threshold decides whether the answer ships or escalates. The taxonomy never travels with the request, the cheap model stays cheap, and the expensive machinery only sees the genuinely ambiguous cases.

Hypothetical classification is one of those ideas that looks like a hack until you measure it. The vocabulary is the expensive part of classification, and the embedding model lets you carry it once, locally, instead of shipping it on every request. If your classification workload is currently paying the taxonomy tax on a big model, it is worth a weekend experiment: replace the schema with a hallucination prompt, add the lookup, and look at the disagreement rate. The numbers will tell you quickly whether the backwards idea is the right one.

Originally published on Dispatch.

A Write Vanished into Thin Air: The 16-Year-Old SQLite Bug That Corrupted Tailscale's Databases

Chen Yuan — Thu, 13 Aug 2026 06:45:04 +0000

Nineteen times in six months, a database that was supposed to be boring quietly corrupted itself. Each incident took the same shape: a backup monitor or data pipeline reported an error, PRAGMA integrity_check confirmed corruption, and an on-call engineer stopped a control-plane process, restored from a snapshot, and tried to explain what had happened. Nobody could. The trigger changed every time — different shard, different customer, different time of day. The only constant was that a committed write had somehow stopped existing.

The bug behind all of it had been sitting in SQLite's Write-Ahead Logging code since 2010. It took Tailscale and the SQLite core team months of forensics to find it, one fix release that had to be withdrawn, and two more months of waiting for positive proof that the real fix worked. This is the case study: how a rare data race hides inside one of the most tested codebases in the world, why Tailscale hit it when almost nobody else could, and what the hunt teaches about the difference between standard and non-standard ways of running "boring" technology.

A Database That Corrupted Itself

Tailscale's control plane looks like a single public endpoint, but internally it is a set of coordination servers, or shards. Each shard owns a slice of tailnets and has its own SQLite database, accessed exclusively by one Go process. That single-writer design is exactly how SQLite is meant to be used: one writer, serialisable transactions, no cross-process locking drama.

SQLite became Tailscale's primary database in 2022 precisely because it is boring. It is well-known, reliable, and widely deployed at far larger scale than any per-shard database at Tailscale. The backup pipeline was equally unremarkable: every few minutes, a complete snapshot of the database file was uploaded to S3. It ran without incident from early 2023.

Then in August 2025, a data pipeline that reads those S3 backups reported an error in one database. PRAGMA integrity_check confirmed corruption. SQLite corruption is possible but highly unusual, and it is not something you should encounter in normal operation. The team repaired the affected database and investigated. Nothing was found.

It happened again. And again. Nineteen separate corruption incidents over six months.

The Hunt: Ruling Out the Obvious

PRAGMA integrity_check;
-- expected: ok
-- actual (August 2025, backup pipeline): "database disk image is malformed"

Every obvious theory failed. No recent change touched the low-level code that interacts with SQLite — it had been written years earlier and had been silent since. There was no common factor between incidents: not a single shard, customer, tailnet feature, time of day, or load level. With no reliable trigger condition, the bug could not be reproduced synthetically. The team fell back to deploying passive forensic telemetry in production and waiting for the next corruption to happen live.

The wait was unpredictable. Incidents came hours apart or weeks apart. Between October and December there were six weeks of calm, and then the corruption returned as an unwelcome Christmas present. Because the diagnosis was not going to be quick, Tailscale signed a professional support contract with the SQLite developers, which gave them direct access to the core maintainers.

Together they mapped out theories: broken POSIX advisory locks cancelled by a separate thread calling close(), mismanaged memory owned by SQLite, or accidentally using SQLite from multiple threads while thread-safety was disabled. After every incident, more diagnostics were added, and one theory after another was systematically ruled out.

The Transactions That Didn't Bark

While the root cause was unknown, the platform still had to run. Recovery was automated aggressively: shards hard-stopped immediately upon detected corruption, a backup monitor continuously ran PRAGMA integrity_check over every snapshot, and runbooks improved. Response time dropped to under an hour.

Then the team built something that doubled as a forensic instrument: a transaction logging pipeline. Every SQL statement that modified the database was streamed to a separate log file. Because SQLite is a single-writer database with serialisable transactions, the transaction history is completely linear and deterministic — replaying it against the last good backup reconstructs the most recent state without touching the corrupted file.

# concept: deterministic replay of a single-writer transaction log
for tx in transaction_log:              # linear, in commit order
    if backup_db.transaction_id < tx.id:
        backup_db.execute(tx.sql)       # SQLite serialises writes anyway
# result: latest consistent state, corruption bypassed

The pipeline worked — and then it found the clue. In two incidents, the transaction logs failed to replay cleanly. A write that had been committed was invisible to later transactions. Data had vanished without an error. In a single-writer, serialisable database, that should be impossible.

How WAL and Checkpoints Work

To see why it was possible, you need the storage layer. A SQLite database is a file of fixed-size pages. In the default rollback-journal mode, updates are written into the main file, which means a reader blocks a writer. Write-Ahead Logging (WAL) changes the deal: new pages are appended to a separate WAL file, and readers keep reading the old main file until a checkpoint copies the new pages back.

Checkpointing is normally SQLite's own decision, invisible to the application. Tailscale's backup pipeline needed fast, consistent snapshots, so the control plane took manual control of the checkpoint process — a public, documented, supported configuration. It also checkpointed aggressively. This non-standard cadence became the prime suspect, and the metrics agreed: during corruption incidents, SQLite reported copying more pages from the WAL than the WAL contained. Ten pages in the WAL, twenty pages copied to the database. Something in the checkpoint path was hallucinating.

WAL file:        [p1][p2][p3] ... [p10]    10 pages
checkpoint log:  "copied 20 pages to db file"
→ impossible in a correct checkpoint

To see inside that path, the SQLite developers built a new debugging tool: a shim around the virtual filesystem layer — the OS interface that actually writes bytes to disk — which logs every change to the database. SQLite's layered design (parser and code generator, pager, virtual filesystem) makes this kind of instrumentation cheap: you wrap one layer instead of recompiling the world. The shim, tmstmpvfs, was deployed into Tailscale's live environment. They did not have to wait long.

The WAL-Reset Bug

The next corruption incident gave the SQLite developers the trace they needed, and the bug surfaced: a rare data race between a checkpoint and a write transaction.

The mechanics are tight. If a write lands at a specific moment during a checkpoint, the checkpointing process gets confused and believes some pages were already copied from the WAL into the main database when they were not. Those pages are never written to the database file, so the data in them is permanently lost — no error, no rollback, nothing. The database file ends up corrupt because other pages that reference the missing ones, such as index pages, are written anyway. A committed transaction simply stops existing.

SQLite named it the WAL-Reset bug and dated it back at least 16 years. It survived that long because it is rare — so rare that the SQLite developers could never reproduce it organically and had to add special testing logic that deliberately triggers the race just to verify that the fix works. The fix itself is small: an additional check in the checkpointing function that detects when the WAL has been reset by another thread.

Why did Tailscale hit it when the rest of the world did not? Because it checkpointed manually and aggressively. Even a bug triggered by a rare timing condition becomes inevitable at a high enough checkpoint rate — which is the quiet mathematical truth about rare bugs at scale.

A Fix, a False Alarm, and a Withdrawn Release

The fix shipped as SQLite 3.52.0. Tailscale rolled it out carefully: canary shards first, then, when it looked healthy, the rest of the control plane. The backup monitor promptly turned red across 13 databases.

The relief was short-lived: those databases were not corrupt. The release that fixed the race also contained an unrelated optimisation that subtly changed text-to-floating-point rounding, and Tailscale stored high-precision timestamps as text that was converted inside a virtual generated column. Indexes on computed values had gone stale, and PRAGMA integrity_check reported them as corruption. The canary shards simply had not contained timestamps that triggered the new rounding behaviour, so the phased rollout missed it.

The SQLite developers withdrew 3.52.0 and published 3.51.3, containing only the WAL-Reset fix. Tailscale reduced its timestamps to integer seconds — text-to-integer conversion is unambiguous — and the SQLite team later shipped an automated self-healing index feature in 3.53.0 that prevents the stale-expression-index problem entirely. Two bugs, one release, one withdrawal, and two lessons about what a "fix" release can smuggle in.

Proof, Two Months Later

// conceptual sketch of the tripwire Tailscale added to its SQLite driver;
// not the actual patch. It logs when a checkpoint overlaps a WAL reset.
func checkpoint(db *sqlite.Conn) error {
    if walWasResetByAnotherThread(db) {
        log.Warn("SQLitePartyMode: WAL reset overlapped checkpoint; corruption prevented")
    }
    return db.Checkpoint(sqlite.CheckpointPassive)
}

An absence of incidents proves nothing — the team already had one six-week false calm behind them. They wanted positive evidence that the race was occurring in production and that the fix was intercepting it. So they patched their SQLite driver to log a warning whenever a write overlapped a WAL reset. If the warning fired and the database stayed healthy, the fix had saved them.

They deployed the tripwire and waited. Weeks slipped by. Doubts crept in: was the warning broken? Was the theory wrong? Was the true bug still hiding somewhere in the darkness? Two months later, the alert finally fired: "SQLite attempted corruption … but the system prevented it." The exact conditions for the bug do occur in production, and the fix was what stood between Tailscale and corruption number twenty.

Since that alert, Tailscale ran another four months with zero database incidents — the only number that ends an incident investigation.

Reproducing the Unreproducible

The same bug offers a second, shorter story about tooling. Antithesis took SQLite 3.51.2 — still buggy — added standard database assertions ("no lost committed writes", "database is not corrupt"), and pointed a generic workload at it: writes and checkpoints running concurrently, exactly what production does all the time. Their deterministic testing platform caught the bug in fifteen minutes. The same workload against 3.51.3 came back clean.

Six months of forensics in production; fifteen minutes under deterministic instrumentation. The asymmetry is not a knock on the forensics — the production hunt produced the diagnosis, the shim, and the fix. It is a reminder that when a bug cannot be reproduced organically, tooling that makes time reproducible is worth more than another week of staring at logs.

What a 16-Year-Old Bug Teaches Us

The operational lesson is the one Tailscale drew itself: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well tested. Manual checkpoint control was documented, supported, and public — and it was still a corner of the design space that had never been exercised the way Tailscale exercised it. If your database is the load-bearing wall of your service, the wall you modified is the wall you should stress-test.

The engineering lesson is about what "rare" means. A bug with tight timing constraints that nobody can reproduce is not a bug that does not exist; it is a bug waiting for the right workload. The WAL-Reset bug existed for sixteen years before anyone hit it, and it will not be the last of its kind. The cheap insurance is the same in every layer: integrity checks that run continuously, transaction logs that can be replayed, and tripwires that fire before corruption becomes an incident.

And the human lesson is that the fix is never the end. The release that fixed the race broke something else. The proof of a fix for a rare bug is not green tests — it is the alert that fires in production two months later, saying the thing you feared almost happened, and did not.

Originally published on Dispatch.

AI Is Removing the Middle Class of Software Engineering

Chen Yuan — Wed, 12 Aug 2026 15:45:55 +0000

You can prompt an agent for three hours and ship a 25,000-line pull request. Nobody on your team can tell you why it works — or why it breaks at 2 AM.

The New Workflow

It's 2026. You're the senior engineer on a mid-size product team. Your job has always been the person who catches the architecture mistakes before they compound — the one who notices that a Kafka dependency was grafted onto a read-heavy query, or that someone denormalized the database because it was faster than fixing the ORM.

This morning, you open your inbox. There are seven pull requests.

The first one is 24,506 lines added, 3,938 removed, with a description that reads: "Implemented user analytics pipeline with event streaming." You pull the branch. It runs. The tests pass. When you ask the author where the data flows, they send you a link to a Claude conversation.

Somewhere in that 47-turn exchange, between confident architectural recommendations and polite apologies when the model changed its mind, is the design decision. You read all 47 turns. You still don't know why they chose Kafka.

This is not a hypothetical. This is what the post-AI-productivity era looks like for teams that adopted coding agents without updating their engineering discipline.

The speed limit has been removed. And the people who built their careers on being the speed limit are now obsolete.

What Changed

Before AI coding assistants, there was a natural throughput cap on software output. A senior engineer could review perhaps three meaningful pull requests per day. A team of ten could ship maybe fifteen high-quality merges per sprint. This cap wasn't arbitrary — it was enforced by the time required to actually understand what you were merging.

AI changed the cost structure, not the review requirement.

A developer armed with a capable agent can now produce 25,000 lines of code in a morning. The agent writes the code. The agent writes the tests. The agent writes the documentation. The agent even writes the PR description, which sounds coherent and professional. To the untrained eye — and many managers are untrained — the output looks indistinguishable from what a senior engineer would produce.

But the review requirement hasn't changed. Someone still needs to understand every line in that 25,000-line PR. Someone still needs to know whether the Kafka dependency is necessary, whether the database schema makes sense, whether the error handling covers the failure modes that will kill you in production at 3 AM.

The gap between production velocity and review capacity has become the defining structural problem of modern software teams.

The Two Types of Engineers

There are two kinds of engineers on every team, and AI has dramatically changed the value of each.

The first type knows what the code does. They can trace a request from the API endpoint through the service mesh to the database and back. They understand why the retry logic uses exponential backoff instead of fixed delay. They can explain, without consulting the agent, why the authentication flow requires a token refresh every 15 minutes rather than every hour. These engineers were valuable before AI. They are exponentially more valuable after it.

The second type can prompt an agent to produce working code. They don't know why the code works. They don't know what happens when the third-party API changes its response format. When asked "why did we choose this architecture?", they say "the model suggested it." These engineers were marginally productive before AI. They are dangerously unproductive after it.

The tragic insight is that the second type of engineer produces output that looks correct. The code runs. The tests pass. The feature works. This is precisely what makes them dangerous: nobody can easily distinguish their output from the first type's output without deep, time-consuming review.

And here's the crucial point — the first type of engineer now has less time to do that review. Because the second type produces 10x the volume, the first type must review 10x the code. The math doesn't work.

The Debt Compounds Faster Now

Technical debt has always been a problem. The difference in the post-AI era is that debt accumulates at a rate that exceeds the team's ability to pay it down.

Consider a simple example. An engineer uses an agent to add a new database table because it's "faster than modifying the existing schema." The agent writes the migration. The agent writes the model. The agent wires it into the API. Everything works. The PR is approved — or rather, it's too large to review thoroughly, so it gets merged with a few cosmetic comments.

# Before AI: This migration might take 2 days of careful planning
class AddUserAnalyticsTable(Migration):
    def up(self):
        # Carefully consider: will this break existing queries?
        # Will the index hurt write performance?
        # Do we need a gradual rollout?
        execute("CREATE TABLE user_analytics (...)"

# After AI: The agent writes this in 30 seconds
class AddUserAnalyticsTable(Migration):
    def up(self):
        # Agent generated: no review needed, right?
        execute("CREATE TABLE user_analytics (...)")
        # Agent also added 12 indexes "for performance"
        # Agent also refactored the ORM layer "for clarity"
        # Nobody noticed until prod broke

Six months later, you need to migrate that data. You need to update every service that touches it. You need to coordinate the rollout with zero downtime. The engineer who added the table has moved to a different team. The agent that wrote the migration is gone with it — no one on the team remembers the reasoning.

Fixing bad architectural decisions is always harder than making them. Before AI, the speed differential between "making the bad decision" and "fixing it" was manageable. A team could accumulate a few bad decisions per quarter and spend a sprint paying down the debt. Now, a team can accumulate thousands of bad decisions per quarter, and the debt payment phase never comes.

This is the credit-card metaphor that every senior engineer recognizes: you see the luxury car (the working feature), not the debt (the architectural complexity that will haunt you for years). AI makes it possible to buy a new car every week.

What Gets Worse With Scale

You might think this problem is contained to teams that adopt AI uncritically. It isn't. The dynamics scale in ways that make the problem worse at larger organizations.

At a startup of ten engineers, the senior person can still review everything personally. At a company of 500, the review bottleneck becomes structural. Middle management adds layers of approval that don't actually improve code quality — they just add process. The result is a organization where code ships faster but gets worse, and nobody can point to a specific failure because everyone was following the process.

The salary divergence is the economic signal. Companies that previously paid $150K–$200K for "solid mid-level engineers who can implement features" now find that those engineers are producing output that costs $20 to generate in API calls. The market corrects: those roles either disappear or drop to $60K–$80K for people who can actually evaluate and direct AI output.

Meanwhile, the engineers who can read code, understand systems, and make architectural judgments command $300K+ because they're the only ones who can prevent the org from collapsing under its own accumulated debt. The middle class isn't just shrinking — it's being replaced by a bimodal distribution with a thinning waist.

The Vibe Coder Career Path

Several years ago, I wrote about why the "vibe coding" career path — learning to prompt AI to build apps without understanding the underlying systems — is doomed. That analysis was speculative. The evidence now overwhelming supports it.

Here's what happens to a vibe coder on a real team:

Month 1:  Ship fast. Impressive output. Colleagues impressed.
Month 3:  Bugs appear they can't debug. PRs create dependencies.
Month 6:  Become a liability. Every touch requires senior review.
Month 12: Let go, or realize they can't compete.

The trajectory is predictable because the foundation is absent. AI gave these engineers the ability to produce output without the skills to evaluate it. That's not a career — it's a countdown.

What Senior Engineers Should Do

If you're the person who actually understands the system, the writing on the wall is clear: your value is increasing, but so is your workload. Here's how to protect yourself.

Refuse large PRs. A pull request larger than 400 lines should be a red flag, regardless of who wrote it or what tools they used. Insist on small, reviewable changes. This isn't anti-AI — it's pro-quality. Any engineer, human or augmented, should be able to explain a 200-line change in a single conversation.

# Good: Small, reviewable PR
def update_user_balance(user_id: int, amount: float) -> bool:
    """Update balance with proper locking."""
    with db.transaction():
        balance = get_balance_for_update(user_id)
        new_balance = balance + amount
        validate_balance(new_balance)
        set_balance(user_id, new_balance)
    return True
# ~15 lines. Anyone can review this in 2 minutes.

# Bad: 400-line PR from an agent
# The agent generated an entire microservice,
# 12 classes, 3 database tables, and a message queue
# in one shot. Nobody reviewed it thoroughly.
# It merged. It broke in prod 3 weeks later.

Demand explanations, not links. When an engineer can't explain a decision, don't accept a link to their AI conversation. That conversation contained the reasoning, yes — but if the engineer couldn't extract the relevant part from 47 turns of back-and-forth, they don't understand their own work. Ask them to explain it. If they can't, the PR doesn't merge.

Build judgment, not speed. Your differentiator is no longer how fast you can produce code. It's how well you can evaluate code that anyone (or anything) can produce. Invest in your ability to read systems, spot architectural flaws, and make tradeoff decisions that balance short-term shipping against long-term maintainability.

Mentor the evaluation skill. Help your junior engineers develop the judgment to evaluate AI output. The next generation of software engineers shouldn't be judged on how many lines they can generate — they should be judged on how well they can decide which lines are worth generating.

What Teams Should Change

Individual advice only goes so far. The structural incentives that reward volume over quality need organizational change.

Tie compensation to maintainability, not velocity. If you measure engineers by lines shipped or PRs merged, you will get exactly what you measure: lots of lines and lots of PRs, most of which nobody fully understands. Start measuring bug rates, mean time to recovery, and the age of code that hasn't been touched in six months.

# Wrong metric: Lines of code per sprint
metrics = {
    "prs_merged": 47,
    "lines_added": 12500,
    "lines_removed": 3200,
    # These numbers look great. They're also meaningless.
}

# Right metrics:
metrics = {
    "bug_rate_per_1000_lines": 2.3,
    "mttr_hours": 4.2,
    "code_age_no_review_months": 8.5,  # Alert: > 3
    "architectural_debt_score": 0.7,    # Scale 0-1
}

Require architectural documentation for non-trivial changes. A PR that introduces a new service, changes the data model, or modifies a shared library should come with a one-paragraph design rationale. Not a 47-turn AI conversation — a single paragraph that explains what the change is, why it was needed, and what the tradeoffs were. If the engineer can't write that paragraph, they don't understand the change.

Keep the human in the loop. This is obvious advice that teams repeatedly forget. AI should augment engineers, not replace them. The engineer who writes the prompt should also understand the output. If they don't, they shouldn't be the one shipping it.

Invest in the middle class. Paradoxically, the best way to save the middle class of software engineering is to stop treating "writing code" as the primary skill and start treating "evaluating code" as the primary skill. Mid-level engineers who learn to critically assess AI output — who can spot a subtle concurrency bug or a security vulnerability that the agent introduced — will be more valuable than ever. The question is whether organizations will recognize and reward that skill, or whether they'll continue optimizing for the wrong metric.

The Long View

This isn't the first time technology has disrupted the middle class of a profession. The printing press disrupted scribes. The spreadsheet disrupted accountants who calculated by hand. The compiler disrupted assembly programmers. Each time, the middle layer shrank, and the people at the top became more valuable while the people at the bottom either adapted or were displaced.

Software engineering is no different. AI is removing the middle class of coding — the engineers who can produce working code but don't deeply understand the systems they're building. What remains are two tiers: the engineers who understand systems and can direct AI, and the engineers who can't and are being replaced.

The question isn't whether this is happening. It's whether you're on the right side of the split.

The engineers who survive this transition won't be the ones who can prompt the most effectively. They'll be the ones who can look at a 25,000-line PR and say, "I don't understand three of these modules, and I'm not signing off on this until I do."

That's not anti-AI. That's just good engineering. And it's the skill that AI can't replicate — because it requires judgment, not pattern matching.

What's your team doing about AI-augmented code quality? Have you seen the middle-class squeeze firsthand? Share your experience in the comments.

Originally published on Dispatch.

A GPU Warp Is Just a Wide SIMD Vector

Chen Yuan — Tue, 11 Aug 2026 15:41:51 +0000

A GPU warp is not like a SIMD vector. It is one: thirty-two lanes, one instruction, thirty-two pieces of data. The only reason the connection stayed invisible for decades is that GPU toolchains never exposed the vector unit the way CPU intrinsics do — you wrote kernels in CUDA or WGSL, and the hardware's lane-level parallelism lived behind the compiler's back. Rust's portable SIMD has just changed that. VectorWare, a GPU-native software company, announced that core::simd types now compile to warp operations on NVIDIA hardware, which means the same Simd<T, N> source that lowers to AVX-512 on a laptop can run on a GPU with no rewrite. This article walks through what that mapping is, where it breaks, and why it matters for how we think about parallelism.

SIMD, in Three Sentences

Single Instruction, Multiple Data is the oldest form of parallelism that fits inside a thread. A scalar add takes two numbers and produces one sum. A SIMD add takes two vectors of eight f32 values and produces eight sums with one instruction — the arithmetic unit is wider, and the loop disappears.

Two properties make SIMD worth caring about. First, it is below the operating system: no threads, no scheduler, no context switches, just a wider execution unit. Second, it is data parallelism, which means the hardware can decide to be wide without the programmer managing any lifecycle. Write the vector operation, and the machine handles the rest.

The cost is that the width is a hardware fact. x86-64 has 128-bit SSE, 256-bit AVX, and 512-bit AVX-512. Arm has 128-bit NEON. A vector that does not fit the register width gets split into multiple instructions, and a vector that does not fill it wastes lanes. The programmer historically had to know which architecture they were on.

Rust's Portable SIMD: One Type, Many Targets

Writing SIMD in Rust used to mean reaching for architecture-specific vendor intrinsics in core::arch — _mm256_add_ps on x86-64, vaddq_f32 on Arm. Each intrinsic is tied to one instruction set, so a program that must run on more than one architecture carries a separate implementation per target, with cfg gates and duplicated tests.

Portable SIMD adds a layer of abstraction above those intrinsics. The core type is Simd<T, N>, a vector of N elements of type T, and the program writes its arithmetic, comparisons, reductions, and lane shuffles once. The compiler lowers those operations to whatever vector instructions the target CPU has:

#![feature(portable_simd)]

use core::simd::num::SimdFloat;
use core::simd::cmp::SimdPartialOrd;
use core::simd::{Select, Simd};

// Elementwise multiply: 32 products computed at once.
fn relu_dot(a: Simd<f32, 32>, b: Simd<f32, 32>) -> f32 {
    let products = a * b;

    // Per-lane comparison produces a mask, one boolean per lane.
    let positive = products.simd_gt(Simd::splat(0.0));

    // Keep the positive products, replace the rest with zero.
    let clamped = positive.select(products, Simd::splat(0.0));

    // Horizontal add across all lanes down to a single scalar.
    clamped.reduce_sum()
}

This is ordinary Rust: an owned value, checked by the borrow checker, composed with normal traits. Nothing in the signature says "GPU" or "x86" or "Arm" — the target is decided by the compiler, not by the source.

SIMT Is SIMD

NVIDIA calls the GPU execution model SIMT — Single Instruction, Multiple Thread. A warp issues one instruction, and each of its 32 lanes runs that instruction on its own data. Read that definition slowly, because it is the whole argument: one instruction operating on many data elements is exactly what SIMD means. The per-lane addressing that SIMT adds — each lane can index a different memory location — does not change the core mechanics.

A warp is a wide vector unit, and a portable SIMD vector maps onto it directly. A Simd<i16, 32> gives one i16 element to each of the warp's 32 lanes; adding two such vectors compiles to a single warp instruction in which every lane adds its element at once. What was a metaphor becomes a layout decision: store one lane's value per lane, and the arithmetic just works.

This completes a clean parallelism hierarchy. On the CPU, a thread contains SIMD lanes. On the GPU, a thread maps to a warp whose hardware lanes play the same role. In both cases, core::simd drives those lanes — the same type, the same operations, the same code.

What Each SIMD Operation Becomes on a Warp

The mapping is not just "elementwise arithmetic happens to line up." Each family of portable SIMD operations has a direct warp-level counterpart:

Portable SIMD operation	GPU counterpart
Elementwise ops (`+`, `*`, `simd_gt`)	Native warp arithmetic, one instruction
Horizontal reductions (`reduce_sum`, `reduce_max`)	Warp shuffle instructions that exchange values across lanes
Cross-lane shuffles (`simd_swizzle!`, rotates)	The same warp shuffle primitives CUDA uses for lane data exchange
Masks (`Mask<T, N>`, `select`, `any`, `all`)	Vote and ballot instructions for per-lane predicates

Reductions deserve a closer look, because they are the operation that does not exist on a typical CPU SIMD model at the source level. reduce_sum combines every lane into a scalar, and the GPU implements it with shuffle instructions: lane i exchanges its partial value with lane i + 16, adds, exchanges with i + 8, and so on down to lane 0. The result is produced in every lane, which is exactly the semantics reduce_sum needs when the vector is part of a larger computation.

// A reduction in portable SIMD...
let total: f32 = clamped.reduce_sum();

// ...is a butterfly of warp shuffles under the hood:
//   lane 0  += lane 16, lane 1  += lane 17, ...  (shuffle + add)
//   lane 0  += lane 8,  lane 1  += lane 9,  ...
//   lane 0  += lane 4,  lane 1  += lane 5,  ...
//   lane 0  += lane 2,  lane 1  += lane 3,  ...
//   lane 0  += lane 1
// Five shuffle-add pairs, O(log lanes), result in every lane.

Scalar values in the surrounding code — a loop counter, a constant — are computed identically by every lane and simply replicated across the warp, exactly like uniform values in CUDA. The distinction between "one value for the whole machine" and "one value per lane" falls out of Rust's own types: a plain f32 is uniform, a Simd<f32, 32> is varying. Data-parallel languages like ISPC make this split explicit with keywords; here it is just the type system doing its job.

The Lane-Count Problem

The one place the abstraction and the hardware do not line up is width. On a CPU, Simd<T, N> allows any N from 1 through 64, and the compiler splits or pads as needed. GPU hardware has a fixed width: 32 lanes on NVIDIA, 32 or 64 on AMD. The mapping is one-to-one only when N matches that width.

// NVIDIA warp: exactly 32 lanes.
let ideal: Simd<f32, 32> = /* one f32 per lane */;

// Too narrow: lanes 16..31 sit idle for every instruction.
let narrow: Simd<f32, 16> = /* half the warp does nothing */;

// Too wide: each lane must process multiple elements,
// and every operation becomes more than one instruction.
let wide: Simd<f32, 64> = /* strip-mined into two warp ops */;

When there is more work than the warp is wide, the program needs a way to say which lanes do what — a small "machine" with its own primitives for moving and combining data, plus invariants about which lanes are active and how much data each one holds. VectorWare's approach encodes that machine in Rust's type system: typed ballots, shuffles, reductions, scans, gathers, scatters, and atomics, with execution shape carried in const generics and trait bounds. Because the operations carry their shape in the types, many invalid programs cannot be constructed at all. That IR is architecture-agnostic — AMD wavefronts and Vulkan subgroups expose the same primitives — and it can run on the CPU through a reference interpreter, which gives differential testing for free.

What This Actually Buys You

The first benefit is portability in the strongest sense: the same source runs on the CPU and the GPU. Code and libraries that already use portable SIMD become candidates for GPU execution without a rewrite. A developer who wrote a Simd<f32, 32> hot loop for a laptop can compile the same function for a GPU kernel and get warp-level parallelism out of it.

The second benefit is that the semantics come along. Simd<T, N> is an ordinary owned value, which means the borrow checker, lifetimes, and type checking apply to it exactly as they do on the CPU. GPU programming has historically been a second-class citizen of the language: kernels had their own syntax, their own error modes, and their own way of doing memory. Mapping existing Rust types onto the GPU's native execution model removes an entire category of cross-language bugs — the kind where the CPU-side code believes one thing about layout while the kernel assumes another.

The third benefit is compositional. With threads mapped to warps, SIMD mapped to lanes, and async mapped to GPU concurrency, the natural next step is combining them: threads spreading work across warps, core::simd spreading data across lanes within each warp, and async structuring the concurrency between them. Each layer of parallelism uses the abstraction it was designed for, instead of a GPU-specific dialect.

The Honest Costs

The abstraction is zero-cost only when the width matches. Portable SIMD is still unstable — it requires nightly and the #![feature(portable_simd)] gate, and its surface may change before stabilization. Vectors narrower than the warp leave lanes idle; vectors wider than the warp turn each operation into more instructions. A Simd<f32, 16> on an NVIDIA warp is not a bug, but it is a wasted half of the machine, silently.

Not every cross-lane operation maps to an efficient warp instruction. Shuffles that match the hardware's supported patterns are cheap, but arbitrary permutations may need several instructions or a trip through shared memory. Horizontal operations like reductions and all/any act as synchronization points within the warp, which constrains how freely the scheduler can overlap work. And the mapping required compiler changes to stay sound — a reminder that "the GPU is just another vector target" is true at the type level but still young at the implementation level.

Conclusion: The Vector Was Always There

The takeaway is not that Rust now has a GPU framework. It is that the GPU's execution model was always a wide vector machine, and the abstraction gap was a toolchain artifact, not a hardware fact. Once Simd<T, N> maps onto a warp, the mental model collapses into something much simpler: there is data parallelism, and it can be written once. Whether the target is AVX-512, a 32-lane NVIDIA warp, or a 64-lane AMD wavefront is a lowering decision, not a design decision.

That is the shift worth paying attention to. GPU programming has spent two decades teaching developers a separate set of concepts — blocks, warps, shared memory, memory coalescing — on top of what is fundamentally the same lane-level arithmetic the CPU has always had. If portable SIMD becomes the common vocabulary, the next generation of libraries gets both targets for the price of one, and the hardware choice stops being a rewrite decision. The vector unit was always there. Now the type system can finally see it.

Originally published on Dispatch.

Give Your AI Agent a Sandbox: Disposable Docker Isolation in Practice

Chen Yuan — Mon, 10 Aug 2026 15:54:31 +0000

A container is a process with boundaries. It shares the host kernel, but it cannot see the host's files, processes, or network sockets unless you explicitly hand them over. That property — isolated by default, exposed by request — is exactly the contract an AI agent needs when it runs code on your machine. Agents are getting good enough to be useful, which means they are getting dangerous enough to be contained. This article walks through why a bare shell is not a boundary, where containers sit on the isolation spectrum, and how to wrap an agent's work in a disposable Docker sandbox without turning your deployment into an orchestration project.

Why a Bare Shell Is Not a Boundary

An agent that executes commands on your host gets the full surface of your machine. Every file the agent writes lands in your real filesystem. Every process it spawns can see your environment variables, and environment variables are where secrets live. Every port it opens is a port on your network. None of this requires malice. A model that writes a loop with a wrong exit condition can fill your disk. A tool that misparses a path can overwrite a file it was only supposed to read. A dependency pulled from a registry can ship with a post-install hook, and the agent will happily run it in your shell.

The failure mode is not exotic. The same properties that make an agent useful — it can read, write, install, and execute — are the properties that make it uncontained. The fix is not to trust the agent less; it is to change the environment the agent acts in, so that even a fully autonomous run has a hard ceiling on what it can touch.

The Isolation Spectrum

Isolation is a spectrum, and each rung buys a different boundary at a different price.

A virtual environment isolates Python packages. It does not isolate files, processes, or network access. A dependency that wants to write to $HOME still can. A subprocess that wants to bind a port still can. Venvs solve dependency conflicts, not containment.

A container isolates all three. It gets its own filesystem view, its own process tree, its own network namespace. It shares the kernel with the host, which makes it cheap to start and cheap to throw away. That last property is the one that matters for agents: a container is designed to be destroyed, and destroying it removes every trace of what ran inside.

A virtual machine isolates the kernel too. That is stronger, but it costs minutes of boot time, gigabytes of memory, and a layer of management tooling. For most agent workloads, the extra boundary is not worth the weight. The threat model for an agent run is accidental damage and untrusted dependencies, not a hostile kernel exploit.

The pragmatic default sits in the middle: run the agent in a container, mount in only the inputs it needs, mount out only the outputs it produced, and delete the container when the run ends.

A Minimal Sandbox in Five Lines

A sandbox image does not need to be complicated. The essentials are a base image, a working directory, and a non-root user. Running as root inside a container is a common mistake: root in the container is still root for anything the container is allowed to do, and the whole point of the exercise is limiting what a run can do.

FROM python:3.12-slim
RUN useradd --create-home --uid 1000 agent
WORKDIR /work
COPY --chown=agent:agent . /work
USER agent
ENTRYPOINT ["python", "main.py"]

That is the whole image. The agent's code runs as an unprivileged user in a filesystem that contains only the working directory. Everything else — the host's home directory, its sockets, its mounts — is simply not there.

Wiring the Sandbox into an Agent Loop

The interesting part is how the sandbox connects to the agent's loop. The agent needs to produce a command, and the harness needs to run that command in a fresh container, with resources capped and a deadline enforced. The container engine gives you the caps; Python gives you the deadline.

import subprocess
import shlex

def run_in_sandbox(command: str, workdir: str, timeout: int = 60) -> subprocess.CompletedProcess:
    docker = [
        "docker", "run", "--rm",
        "--network", "none",
        "--memory", "512m",
        "--cpus", "1.0",
        "--read-only",
        "-v", f"{workdir}:/work:rw",
        "agent-sandbox:latest",
        "sh", "-c", command,
    ]
    return subprocess.run(
        docker,
        capture_output=True,
        text=True,
        timeout=timeout,
    )

Three flags carry most of the safety. --network none means the container cannot reach the network, so a dependency or a misbehaving model cannot exfiltrate or download. --read-only makes the container's own filesystem immutable, so nothing inside the image can be modified at runtime. --memory and --cpus cap the damage a runaway loop can do. The only writable location is the mounted working directory, which is where the agent's inputs and outputs live anyway.

# the harness side: fail the run, keep the host clean
docker run --rm --network none --read-only \
  -v "$(pwd)/inbox:/inbox:ro" \
  -v "$(pwd)/outbox:/outbox:rw" \
  agent-sandbox:latest python main.py

Inputs go in read-only; outputs come out through a single writable mount. If the agent deletes everything in the container, the host notices nothing.

Making the Sandbox Disposable

Disposability is the property that turns a sandbox into a guarantee. A container you keep and reuse accumulates state: files left by a previous run, packages installed by a previous agent, environment drift that makes the next run behave differently. A container you create per run and destroy after the run has no history, and an agent with no history cannot be poisoned by one.

The --rm flag deletes the container when the process exits. That covers the container itself. The working directory is another matter: it is mounted from the host, so anything the agent writes there survives. That is usually what you want — the outputs are the point of the run — but it means the working directory is the state boundary, and it should be created fresh per run and cleaned up when the run is done.

Image tagging is part of the same discipline. latest moves; a pinned digest does not. If the sandbox image is rebuilt between runs, latest silently changes the environment the agent runs in, and a run that passed yesterday can fail today for reasons nobody can reproduce. Pin the image to a digest or a fixed tag so a sandbox run is reproducible.

When the Sandbox Bites Back

The sandbox is not free, and the costs show up in predictable places.

Image availability is the first. A fresh container engine has no images cached, and pulling a base image takes time and bandwidth. The fix is to pull before the agent runs, not during it. The same applies to dependencies: if the container installs packages at startup, every run pays the network cost — and with --network none, it pays with failure. Bake dependencies into the image at build time.

Mount performance is the second. Bind mounts are fine for code and small files, but a working directory with many files or heavy I/O can be dramatically slower inside a container than on the host, especially on macOS and Windows, where the mount crosses a virtualization boundary. For large artifacts, copy them into the container's own filesystem and copy results out, rather than streaming through the mount.

Docker-in-Docker is the third. An agent that builds images inside its sandbox needs the Docker socket, and mounting the Docker socket into a container is the same as giving the container root on the host — it undoes the isolation you just bought. If the agent must build images, run a dedicated daemon inside the sandbox, or route through a remote builder with its own limits. Never mount the host socket.

What a Sandbox Does Not Buy You

Containers are a boundary, not a safe. They share the kernel, so a kernel vulnerability can in principle cross the boundary, and containers do not stop resource exhaustion inside their own limits: an agent that is allowed 1 CPU can still peg that CPU. Secrets mounted into the container are secrets inside the container, and a read of a mounted secret file is a read the sandbox will permit. The sandbox shrinks the blast radius; it does not eliminate it.

The discipline that makes sandboxes work is the same discipline that makes any security control work: the control is only as good as the policy around it. Fresh working directories per run, pinned images, no socket mounts, network off unless the task needs it, and a deadline on every run. None of these are hard, and together they turn an agent from a process that can touch everything into a process that can touch one directory, briefly.

That is the trade worth making. The agent gets to be autonomous; the host gets to stay boring.

Originally published on Dispatch.

When the GPU Is Overkill: A Measurement-First Guide to CPU Inference

Chen Yuan — Sun, 09 Aug 2026 04:29:51 +0000

Which hardware should run this model? Most teams answer that question with habit instead of arithmetic. The model card says GPU. The cluster has a GPU quota. The last project used a GPU, so this one will too. Then the invoice arrives, and the model that should cost pennies is costing dollars, because nobody asked the only question that matters: how many tokens per second does this workload actually need?

That number exists before you buy anything. A background job that summarizes a document overnight needs a handful of tokens per second. A chatbot in front of a user needs dozens. A batch re-ranker that processes a queue while nobody watches needs whatever keeps the queue from growing faster than it drains. Those are three different hardware answers, and two of them do not involve a GPU at all. This article shows how to compute your ceiling before you spend, how to measure the machine you already have, and how to turn the result into a decision you can defend in a review meeting.

The Two Phases Have Different Bottlenecks

Every LLM generation run has two phases, and they are not the same kind of work. Prefill processes the whole prompt at once: every token attends to every other token, which is a wall of matrix multiplications. That phase is compute-bound. More FLOPs per second wins, and GPUs win that game by an order of magnitude.

Decode is the second phase. The model emits one token, feeds it back, emits the next. Each step moves the entire weight matrix from memory into the compute units, does a comparatively small amount of arithmetic on it, and writes one token out. The arithmetic per token is tiny. What dominates is the memory traffic: the weights have to cross the memory bus once per token, every token, until generation stops.

That is why decode speed on a given machine is set by memory bandwidth, not by FLOPs. The compute units sit idle waiting for weights to arrive. A GPU still wins decode because its memory subsystem is wider and faster, but the win is a bandwidth win, and bandwidth is a property a CPU can also have. The question is whether the CPU has enough of it for your workload.

The Decode Ceiling Is a Bandwidth Calculation

The ceiling is arithmetic, and it takes one line. A model reads its weights once per token, so the maximum tokens per second is the memory bandwidth divided by the size of the weights in bytes.

def decode_ceiling(bandwidth_gb_s: float, weights_gb: float) -> float:
    # pure arithmetic: bytes per second divided by bytes per token
    return bandwidth_gb_s / weights_gb

Plug in typical numbers to see the shape. A 7-billion-parameter model stored at 4 bits per parameter is roughly 3.5 GB on disk and in memory. A laptop with 40 GB/s of effective memory bandwidth computes a ceiling near 11 tokens per second. A machine with 100 GB/s of bandwidth doubles that. A GPU with 500 GB/s or more pushes the same model past 100. None of these are measurements of any particular product. They are the theoretical ceiling the machine cannot beat, and they come straight from the specification sheet.

Two consequences fall out immediately. First, the model size in the denominator matters as much as the hardware in the numerator: a 1-billion-parameter model at 4 bits is about 0.5 GB, which gives a CPU a ceiling of dozens of tokens per second. Small models on CPUs are not a compromise; they are the point where the arithmetic stops favoring the GPU. Second, a workload that needs 8 tokens per second has no business paying for a machine whose ceiling is 200. The GPU is overkill by a factor of twenty-five, and you pay for the whole factor.

Where the GPU Stops Being Worth It

Three workload shapes routinely fall under the CPU ceiling. The first is throughput-insensitive work: nightly summarization, document classification, log triage, anything that runs in the background and has no human waiting. Nobody notices whether a batch job finishes in four minutes or forty, as long as it finishes before the morning. The second is low-concurrency work. A single user or a single queue means one stream of generation at a time. GPUs earn their price when many streams share the hardware; a lone stream uses a fraction of it. The third is long-context re-reading. A job that feeds a large document to a model spends most of its time in prefill, and prefill on a CPU is slow enough to matter — but if the same document is processed every night on a schedule, slow is still fast enough.

Each of these maps onto the ceiling formula. Write down the required tokens per second. Multiply by the margin you want for spikes and retries. If the product stays under the measured ceiling of the CPU you already own, the purchase decision is already made and the answer is to buy nothing.

A Harness That Measures Your Machine

Spec sheets give ceilings. Your machine gives reality, and reality is what decides. The harness below measures decode throughput directly: load a GGUF model with llama-cpp-python, generate a fixed number of tokens, and divide the count by the wall time. No external server, no cloud account, no vendor dashboard.

import time
from llama_cpp import Llama

llm = Llama(model_path="model.gguf", n_ctx=8192, n_threads=8, verbose=False)

prompt = "Explain why a GPU is not always the right answer for inference."
start = time.perf_counter()
out = llm(prompt, max_tokens=512)
elapsed = time.perf_counter() - start

tokens = out["usage"]["completion_tokens"]
print(f"prefill+decode wall time : {elapsed:.2f}s")
print(f"decode throughput        : {tokens / elapsed:.1f} tokens/s")

Run the same script on the GPU box and on the CPU box, with the same model file and the same prompt length. The ratio between the two is the real answer to the purchase question, and it is usually smaller than marketing suggests, because decode is bandwidth-bound on both machines. Do not trust the number printed by a benchmark suite you did not write; trust the number you produced with your own workload shape.

Reading the Numbers

The measured throughput is a fact, not a verdict. Turning it into a decision requires the requirement side of the equation. A chatbot needs its first token fast and its subsequent tokens steady; the decode rate matters on every interaction. A batch worker needs the average rate to exceed the arrival rate of the queue; short bursts do not matter, sustained rate does. A report generator that runs once a day needs the total wall time to fit inside a maintenance window.

Compare the requirement against the measurement with an explicit margin. If the requirement is 8 tokens per second and the CPU measured 14, the decision is easy. If the CPU measured 10 and the requirement is 9, the margin is too thin for comfort and the GPU wins on headroom, not on peak speed. The margin is a policy choice, not a physics constant, which is exactly why it belongs in code.

The Decision Procedure

The whole process compresses into a small function that takes the requirement and the measurement and returns a verdict. Putting it in code forces the assumptions out of the conversation and into a file where they can be reviewed.

def choose_hardware(required_tokens_s: float, measured_cpu: float, margin: float = 1.5) -> str:
    needed = required_tokens_s * margin
    if measured_cpu >= needed:
        return "cpu"
    if measured_cpu >= required_tokens_s:
        return "cpu, thin margin — recheck after quantization"
    return "gpu"

The verdicts read as plain language on purpose. The first branch is the whole article in one line: measured throughput beats the requirement with margin, so the GPU is overkill. The second branch says the arithmetic is close enough to test one more variable before spending. The third branch is the only one that actually buys hardware.

What Quantization Changes

The denominator of the ceiling formula is bytes per parameter, and quantization is how you shrink it. A model at 8 bits per parameter takes twice the memory of the same model at 4 bits, which halves the CPU decode ceiling for the same bandwidth. Dropping from 8 bits to 4 bits doubles the ceiling and can move a machine from the third branch to the first.

The catch is that quantization trades bytes for accuracy, and the trade is task-dependent. A model that classifies boilerplate may survive 4-bit without a visible difference. The same model doing precise extraction from legal text may not. The correct test is cheap and mechanical: run your own evaluation set through both quantizations, compare the outputs, and let the comparison decide. Never assume the ceiling math is the only math in the room.

Guardrails and Regression Harness

The measurement is only trustworthy while the machine and the software stay the same. A dependency upgrade can silently change the runtime, a shared machine can lose bandwidth to a noisy neighbor, and a model file swap can change the effective size. The cheap protection is a regression harness that re-measures and asserts, run after every change that touches the pipeline.

import json
from pathlib import Path

def assert_decode_ceiling(path: str, floor: float) -> None:
    result = json.loads(Path(path).read_text())
    measured = result["decode_tokens_per_second"]
    assert measured >= floor, f"decode fell below {floor}: {measured:.1f}"

Wire the harness into CI next to the unit tests. It will not catch every regression, but it catches the expensive kind: the one where the model still answers correctly and the bill simply grows.

The conclusion is not a recommendation to abandon GPUs. It is a recommendation to stop guessing. The requirement is a number you can write down. The ceiling is a number you can compute from a spec sheet. The measured throughput is a number you can produce in five minutes on the machine you already own. When all three numbers exist, the hardware decision stops being a belief and becomes a calculation — and a surprising fraction of the time, the calculation ends in buying nothing.

Originally published on Dispatch.

Your Subprocesses Outlive Your Program. Here's How to Kill Them for Real.

Chen Yuan — Fri, 07 Aug 2026 13:42:10 +0000

At 02:00 the deployment script reported success. The log showed the worker finishing, the health check passing, and the script exiting with code zero. What the log did not show was the child process it had started ten minutes earlier, still alive, still holding the port, still appending to a log file that had already been rotated out from under it. The next deploy failed because the port was taken, and the postmortem said the obvious thing: the previous run did not clean up. Nobody had typed a wrong command. The script had simply killed the wrong thing.

Every language that starts a subprocess ships a kill button, and in every language that button is a lie half of the time. In Python the lie has three parts, and they fail in a fixed order: what terminate() actually sends, who is listening when the signal arrives, and what happens to the process after the signal lands. Most cleanup bugs live in the gap between the call and the corpse, and most of them survive code review because the happy path — a child that exits on its own — never exposes them. This article walks that gap from the first signal to the reaped process, and ends with a termination protocol that leaves nothing running, plus tests that prove it without a single sleep.

What terminate() Actually Sends

When you call process.terminate() on a subprocess.Popen object, Python sends the child a SIGTERM. SIGTERM is not a kill. It is a request to exit, delivered to a signal handler that the child may have replaced, may be ignoring, or may not be ready to handle yet. A well-behaved process exits. A process with a handler that flushes buffers, saves state, or finishes a network call can take seconds or minutes. A process with no handler at all exits immediately. And a process with a handler that decides, in code you do not control, not to exit simply keeps running.

process.kill() sends SIGKILL, which the kernel delivers without asking the process anything. Nothing can ignore it, nothing can defer it, and the process gets no chance to clean up. The two calls look like a toggle between gentle and brutal. In reality they are the two endpoints of a negotiation that the parent must manage, because the parent cannot know in advance which kind of child it is talking to.

import signal
import subprocess

proc = subprocess.Popen(["worker", "--queue", "high"])
proc.terminate()                # SIGTERM: a polite request
try:
    proc.wait(timeout=5)        # give the handler time to clean up
except subprocess.TimeoutExpired:
    proc.kill()                 # SIGKILL: the kernel takes over
    proc.wait()

Who Actually Receives the Signal

The second lie is delivery. With default settings, the signal goes to exactly one process: the direct child. If that child is a shell — and it is whenever you pass shell=True, and it often is when the executable you named is a wrapper script, a Makefile, or a language runner that spawns its own workers — the child forwards nothing. Your SIGTERM reached the shell sitting on top of a tree of three more processes, and the shell has its own ideas about what to do with signals meant for its children.

Even without a shell, real children spawn grandchildren. A build tool starts a compiler, the compiler starts a linker, a media encoder fans out to worker processes. The parent holds a handle only to the top of that tree, and killing the top leaves the rest running. The rest is the part holding the port, the lock file, or the half-written database row.

# shell=True wraps the command in /bin/sh -c
proc = subprocess.Popen(
    "ffmpeg -i in.mp4 out.mp4",   # sh spawns ffmpeg as its own child
    shell=True,
)
proc.terminate()                  # kills the shell; ffmpeg keeps transcoding

Killing the Whole Family

The fix is to stop addressing the process and start addressing the group. POSIX systems group processes precisely so that a signal can be aimed at a tree. Start the child in its own session with start_new_session=True, which makes it a process-group leader, and then signal every member of the group with os.killpg. The child you started, the grandchildren it spawned, and the shell in between all receive the signal in one call.

proc = subprocess.Popen(
    ["worker", "--queue", "high"],
    start_new_session=True,          # the child leads its own process group
)
os.killpg(proc.pid, signal.SIGTERM)  # the whole tree, politely
try:
    proc.wait(timeout=10)
except subprocess.TimeoutExpired:
    os.killpg(proc.pid, signal.SIGKILL)  # the whole tree, for real
    proc.wait()

Grouping changes the meaning of the kill from "tell the top to stop" to "tell everyone in this unit to stop". It is the difference between phoning the manager and evacuating the floor, and for a process tree it is the only way to be sure the floor is actually empty.

The Timeout That Lets the Child Keep Running

The third lie is about time. Popen.communicate(timeout=N) is the most common way to bound a subprocess, and its failure mode is quietly famous: the method raises TimeoutExpired, the caller catches it, and the child keeps running. The timeout bounded the parent's patience, not the child's life. The trap is extra nasty because it is not silent — the code clearly knows the call timed out — and yet the natural reaction, log and continue, is exactly the reaction that leaves the orphan behind.

try:
    out, err = proc.communicate(timeout=10)
except subprocess.TimeoutExpired:
    logger.error("worker exceeded 10 seconds")  # logged, and...
    # ...the worker is still running, still holding the queue

There is a second trap inside the same method. After a timeout, communicate() has left the pipes in an inconsistent state, and the standard library documentation is explicit about the recovery: kill the process first, then call communicate() again to drain whatever remains in the buffers. The second call is not optional bookkeeping. A child that was writing a large payload when it died can leave the parent blocked on a full pipe buffer unless the remaining bytes are read.

except subprocess.TimeoutExpired:
    proc.kill()
    out, err = proc.communicate()   # drain the pipes, then reap

A Termination Protocol That Ends

Put the pieces together and a reliable termination is a fixed sequence of four steps: signal the group politely, wait with a deadline, signal the group forcefully, and wait again until the corpse is collected. The deadline between the two signals is what gives a well-behaved process the chance to do its cleanup, and the second wait is what guarantees the slot in the process table is actually free before the parent moves on.

def terminate_tree(proc: subprocess.Popen, grace: float = 5.0) -> int:
    os.killpg(proc.pid, signal.SIGTERM)
    try:
        return proc.wait(timeout=grace)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGKILL)
        return proc.wait()

The grace period is a policy decision, not a tuning knob: it is the amount of cleanup time you are willing to pay before you accept the risk of a half-written file. The protocol works because each step is unconditional — no step asks whether the child looks cooperative, and no step re-checks the process state before deciding. The only conditional in the whole flow is the timeout, and a timeout is a deadline, not a guess about the child's mood.

Zombies, Orphans, and Who Reaps Them

A process that exits is not finished. It becomes a zombie: an entry in the process table carrying an exit status, waiting for its parent to read that status with wait(). A zombie costs almost nothing — no CPU, no memory worth naming — but it occupies a process-table slot, and a system with enough zombies eventually refuses to spawn new processes. The kernel keeps the corpse around because the parent might want to know how the child died. The parent's job is to collect the body.

Popen.wait() and communicate() do the reaping for you. The subtle part is what happens when the parent never calls them. A long-lived worker that spawns one-shot children and forgets to wait on each one accumulates zombies until the machine notices. A child whose parent died first is adopted by the nearest surviving ancestor, which usually means it stops being your problem at the same moment your code stops running — but only if your code actually stops. In a process that catches errors and keeps going, an un-reaped child stays un-reaped.

The asyncio world has the same shape with different plumbing. create_subprocess_exec returns a Process with the same wait() and communicate() methods, and the same escalation applies. The extra hazard is that a bare exception handler around an un-awaited subprocess leaves the child running until the event loop itself finishes, and the loop will happily finish while the child keeps working. Termination is not a detail the event loop manages for you.

Testing Lifecycles Without Sleep

Lifecycle bugs are invisible to the tests people usually write, because the usual test starts a process, asserts on its output, and lets it exit naturally. Every failure mode in this article involves a child that refuses to exit, so the test has to create one deliberately. The trick is to use Python itself as the unruly child: a short script that catches SIGTERM, prints that it received it, and keeps running until something stronger arrives.

STUB = """
import signal
import sys
import time

def hold(signum, frame):
    print("got SIGTERM, ignoring", flush=True)

signal.signal(signal.SIGTERM, hold)
print("ready", flush=True)
while True:
    time.sleep(0.05)
"""

def test_terminate_tree_reaches_everyone():
    proc = subprocess.Popen(
        [sys.executable, "-c", STUB],
        start_new_session=True,
        stdout=subprocess.PIPE,
    )
    assert proc.stdout.readline().strip() == "ready"
    os.killpg(proc.pid, signal.SIGTERM)
    assert proc.poll() is None           # the polite step was ignored
    os.killpg(proc.pid, signal.SIGKILL)
    assert proc.wait(timeout=5) == -signal.SIGKILL

Asserting the middle state — alive after SIGTERM, dead after SIGKILL — pins the test exactly where the bug lives. The same stub works for the shell case: start it with shell=True, terminate the top, and verify that the grandchild survives a plain terminate() but not a killpg. Every assertion is about process state, not wall-clock timing, so the test is deterministic and never sleeps.

Terminating a subprocess is not a call. It is a protocol with four obligations: know what your signal asks for, know who is listening, address the whole tree, and collect the corpse. Python provides every piece — SIGTERM, start_new_session, os.killpg, wait — and none of it works if the pieces run in the wrong order, and none of it is checked by the tests that let the child exit on its own. The next time a deploy fails because a port is taken, the question is not whether the previous run killed its child. The question is whether the child was killed for real.

Originally published on Dispatch.

Timeouts Are Contracts, Not Safety Nets

Chen Yuan — Thu, 06 Aug 2026 15:36:42 +0000

Setting a timeout does not give you a bounded operation. It gives you a number. Whether that number ever turns into an actual limit depends on three things the documentation rarely mentions: which phase of the call the timeout applies to, how much budget is left by the time the call runs, and whether the process really stops work when the limit fires. Most timeout bugs live in the gap between the number and the behavior, and they fail silently — the operation completes, just later than every promise you made upstream.

This article is a field guide to closing that gap. It covers the three timeout types that are actually different from each other, why per-call timeouts compose into unbounded totals, how to carry a deadline through a call graph instead, and what has to happen after a deadline fires for the whole mechanism to mean anything. None of it requires new infrastructure. All of it is a change in where the number is chosen and what the code does with it.

The Three Timeouts That Are Actually Different

When an HTTP client says "timeout", it usually means one of two things. A connect timeout bounds the phase before the request is sent: DNS resolution, TCP handshake, TLS negotiation. A read timeout bounds the silence between bytes once the request is in flight. Neither bounds the whole operation. A call can use its entire connect budget, then sit under its read budget, and the user experiences a duration that is the sum of both — or worse, the product, if the code retries.

The third kind is the one most clients hide: a total timeout that bounds everything, from the first byte of the request to the last byte of the response. Some libraries expose it explicitly. Some do not expose it at all, which means the only way to get one is to wrap the call yourself.

import httpx

# connect, read, write and pool are separate budgets
client = httpx.Client(
    timeout=httpx.Timeout(
        connect=3.0,   # DNS + TCP + TLS
        read=10.0,     # silence between bytes
        write=10.0,    # silence while uploading
        pool=3.0,      # waiting for a free connection
    )
)

Note what the configuration above does not contain: an overall limit. read=10.0 means ten seconds of silence, not a ten-second call. A server that trickles one byte every nine seconds for an hour satisfies that read timeout forever. If your mental model was "the request dies after ten seconds", the model was wrong, and nothing on the wire will tell you that. The total timeout is a different feature, and when the library does not ship one, it is your job to add it:

import asyncio

async def call_with_total(fn, total: float):
    # one cap around everything, including the time spent waiting to send
    return await asyncio.wait_for(asyncio.to_thread(fn), timeout=total)

A Per-Call Timeout Is Not a Deadline

Per-call timeouts look like they compose, and they do not. An HTTP handler that calls a database with a five-second timeout, inside a worker that has its own ten-second timeout, inside a queue consumer that waits fifteen seconds — that is a call graph whose worst case is the sum of every level, not the slowest one. Each layer protects itself, and none of them protects the user, because the user is the only participant with a real deadline.

The distinction that matters: a timeout is a limit on one operation; a deadline is a point in time by which a whole chain of operations must finish. If the chain has four hops and each hop gets its own generous timeout, the total is generous four times. The system is not slow because the timeouts are wrong. It is slow because the timeouts are permissive in the exact place where the sum is what the user experiences.

The fix is to stop treating timeouts as independent knobs and start treating them as a budget that is allocated once, at the entry point, and consumed as the work travels.

import asyncio

async def handle_order(request):
    # the only place a real number is chosen
    deadline = asyncio.get_running_loop().time() + 30.0
    await charge_card(deadline)
    await book_inventory(deadline)

async def charge_card(deadline: float):
    remaining = deadline - asyncio.get_running_loop().time()
    if remaining <= 0:
        raise DeadlineExceeded()
    await asyncio.wait_for(payment_provider.charge(), timeout=remaining)

Carry the Deadline, Convert at the Boundary

The pattern above is the whole trick: every function receives the deadline, computes the remaining budget when it actually needs a timeout, and converts that remaining budget into the timeout it passes to whatever it calls. The conversion happens at the boundary, so each layer still speaks the language of its own library — wait_for wants seconds, a socket wants a deadline, a database driver wants its own timeout object — while the total stays bounded by the number chosen at the entry point.

Two properties make this work. First, the deadline is never reset: no function along the path gets to start a fresh countdown, because a fresh countdown is exactly how the sum of the per-call timeouts sneaks back in. Second, the check happens before the call, not after: waiting until a timeout fires to discover the budget is gone is waiting for the failure you were trying to prevent.

The subtle part is the check itself. remaining <= 0 looks like a boundary check, but the real cost is inside the call: a call that starts with one millisecond of budget left will burn the entire underlying timeout machinery before failing, and it will fail with a confusing error instead of the clean decision the caller needs. Subtract a safety margin. If the remaining budget is smaller than what a meaningful operation needs, fail fast with the degraded path, not with the timeout handler.

The Fan-Out Trap and the Retry Trap

Parallelism hides the same problem in the other direction. Ten calls issued with asyncio.gather, each with its own five-second timeout, complete in five seconds in the worst case — the budget is not multiplied, because the calls overlap. But the moment a retry loop wraps those calls, the arithmetic changes. Three retries of a five-second call is fifteen seconds of worst case, and if the retry loop sits inside a caller that also retries, the totals compound exactly like the nested timeouts above.

Retries are where budgets die. The standard loop — try, sleep, try again, with a backoff factor — has no concept of a deadline, so it will happily spend fifteen seconds when the product promised three. The fix is to make the retry consume the same budget as the call.

async def call_with_budget(operation, deadline: float, attempts: int = 3):
    loop = asyncio.get_running_loop()
    for attempt in range(attempts):
        remaining = deadline - loop.time()
        if remaining <= 0:
            raise DeadlineExceeded()
        try:
            return await asyncio.wait_for(operation(), timeout=remaining)
        except (TimeoutError, TransientError):
            if attempt == attempts - 1:
                raise
            # backoff comes out of the same budget
            await asyncio.sleep(min(0.5 * (2 ** attempt), remaining / 2))

The sleep is the detail that usually gets skipped. It is real time, it is inside the user's deadline, and it must be charged to the budget like everything else. A backoff that waits two seconds after the budget expired is a bug that only shows up in production, because in tests the retry loop always succeeds on the first attempt and nobody ever sees the sleep.

Cancellation Is the Other Half

A timeout that fires but does not stop work is a timer, not a limit. In asyncio, wait_for cancels the task when the time is up, and cancellation is the mechanism that actually enforces the contract — the coroutine is asked to unwind, its finally blocks run, and control returns to the caller. The enforcement fails when the code being cancelled refuses to be cancelled. Catching CancelledError and continuing is possible, and it converts a timeout into a stall that the process can never escape.

# the anti-pattern: catching CancelledError defeats every timeout above you
try:
    result = await slow_operation()
except asyncio.CancelledError:
    cleanup()            # fine
    await finish_work()  # not fine — you are now the hung process
    raise

The same logic applies to CPU-bound work. A timeout cancels a coroutine at its next suspension point; a function that spins in a loop never suspends, so the cancellation waits at the gate. The answer is not a bigger timeout — it is to put CPU-bound work where the process can actually abandon it: a worker process, or explicit progress checks inside the loop that respect the deadline and raise on their own.

What Happens When the Deadline Fires

The last question is the one most designs forget: what does the caller receive? The answer is never "nothing". A user waiting on an operation that will not complete needs a decision — a degraded result, a cached value, an error the UI can render — and the deadline is the moment that decision gets made on your terms instead of the user's.

There is a second layer to this. A dependency that is slow is often a dependency that is down, and if every request waits out the full timeout before failing, a slow dependency becomes an outage: the timeout is spent, then the retry spends another, and the process fills with requests that are technically not stuck but are not doing anything either. A circuit breaker caps the damage by failing fast once the timeout has fired a few times in a row, instead of paying the full price on every request while the dependency recovers.

class CircuitBreaker:
    def __init__(self, threshold: int = 3, cooldown: float = 30.0):
        self.threshold = threshold
        self.cooldown = cooldown
        self.failures = 0
        self.opened_at = 0.0

    def _open(self) -> bool:
        return self.failures >= self.threshold

    async def call(self, operation):
        if self._open():
            raise ServiceUnavailable()
        try:
            result = await operation()
            self.failures = 0
            return result
        except TimeoutError:
            self.failures += 1
            raise

Measure the Contract, Not the Knob

A timeout value is a promise with a number on it, and promises need measurement. The two numbers that matter are the timeout itself and the percentile of call durations: if the p99 of a call sits close to its timeout, the system is returning errors that nobody is counting, because the requests that time out are precisely the ones that never make it into the duration histogram. The timeout is not a performance target; it is an upper bound, and durations that live near it are a finding, not a configuration.

Timeout outcomes deserve their own counters, separately from success and failure. How many requests hit the deadline? How many of those hit it on the first attempt versus the third? How many were saved by the degraded path? These are the numbers that tell you whether the contract is being kept, and they are invisible in a dashboard that only shows average latency — the average is pulled down by fast successes while the timeout stream stays hidden in the tail.

Tests have the same blind spot. A timeout test that waits for real seconds is slow, flaky, and never exercises the interesting cases — the deadline consumed by retries, the cancellation that cleans up mid-flight. Time-based code is testable by faking the clock: inject a clock that can be advanced, and every timeout path becomes a deterministic test instead of a sleep-and-hope.

def test_deadline_is_consumed_by_retries(fake_clock):
    calls = []

    async def flaky():
        calls.append(fake_clock.now())
        raise TransientError()

    with pytest.raises(DeadlineExceeded):
        fake_clock.run(
            call_with_budget(flaky, deadline=fake_clock.now() + 2.0, attempts=10)
        )
    # the backoff ate the whole budget: the loop exited before the final attempt
    assert len(calls) == 2

A timeout is a contract between the process and its caller. The number is the easy part; the contract is the phase it applies to, the budget it consumes, the work it actually cancels, and the answer the caller gets when it fires. Set the number once, carry the deadline through the call graph, cancel real work when it trips, and measure the distance between the promise and the behavior. That distance, not the knob, is the thing worth engineering.

Originally published on Dispatch.

Parsing an LLM's JSON Before the Last Token Arrives

Chen Yuan — Wed, 05 Aug 2026 07:23:27 +0000

A model that reliably returns valid JSON does not return valid JSON while it is typing. Those are two different guarantees, and most streaming clients quietly assume they are the same one. Constrained decoding and schema-enforced output modes promise that the finished response parses. They promise nothing about the seventeenth chunk, which might end in the middle of a key, inside an escape sequence, or right after a comma that has no element behind it yet.

So the usual compromise is to stream tokens for a progress spinner and buffer them all anyway, then parse once at the end. The user watches characters appear that the application itself refuses to read. Everything the model has already decided — the title, the first three list items, the classification label that arrived in the first fifty tokens — sits unusable in a string buffer until the closing brace lands.

The gap is closable. What follows builds a partial JSON reader that turns every chunk into the best complete value the stream can currently justify, then shows the two places where the obvious version of that idea silently corrupts data.

What a Streaming Model Actually Sends You

A streaming completion arrives as deltas of text with no relationship to JSON structure. Token boundaries follow the tokenizer's vocabulary, not the grammar. A single delta can carry {"ti, or tle": "Sh, or a lone backslash that only becomes meaningful when the next delta supplies the n.

The client loop is the easy part:

import json
from openai import OpenAI

client = OpenAI()

def stream_text(prompt: str):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        stream=True,
    )
    for chunk in response:
        delta = chunk.choices[0].delta.content
        if delta:
            yield delta

Every element of that generator is a fragment of a document that is not yet a document. The question is what a consumer can do with each one besides append it.

Why `json.loads` Is the Wrong Tool Mid-Stream

The standard library parser is all-or-nothing by design. Hand it a prefix and it raises, because a prefix of a JSON document is not a JSON document.

buffer = ""
for delta in stream_text(prompt):
    buffer += delta
    try:
        value = json.loads(buffer)   # raises on every chunk but the last
    except json.JSONDecodeError:
        continue

json.JSONDecoder.raw_decode looks like an escape hatch, and it is a useful one, but for a different problem: it decodes one complete value from the front of a string and reports where it stopped. That solves concatenated documents in a stream of separate objects. It does not help when the single object you want is truncated, because there is still no complete value at the front to decode.

The ijson package is closer. It is an event-driven parser that yields start_map, map_key, string, and end_map events as bytes arrive, which is exactly the streaming shape you want. Its constraint is that it is built for very large but eventually complete documents; a truncated feed ends in IncompleteJSONError, and you get events only for values that have already closed. For a rendering UI that wants to show a half-written string as it grows, event-per-completed-value is one step too coarse.

Writing a real incremental parser is the thorough answer, and it is more work than the problem deserves. A tokenizer, a state machine over the grammar, and a value builder that can be interrogated halfway through is a few hundred lines to write and considerably more to trust. The remaining approach is cheaper and reuses a parser that is already correct: repair the prefix into a valid document, parse the repair, and throw the repair away. The buffer itself is never modified, so a mistake in the repair logic costs one bad snapshot rather than a poisoned stream.

Closing What You Have So Far

The core idea is small. Track which containers are open, and when a snapshot is requested, append the closers that would finish them.

def naive_snapshot(buffer: str):
    stack = []
    for ch in buffer:
        if ch == "{":
            stack.append("}")
        elif ch == "[":
            stack.append("]")
        elif ch in "}]" and stack:
            stack.pop()
    try:
        return json.loads(buffer + "".join(reversed(stack)))
    except json.JSONDecodeError:
        return None

For a well-behaved prefix this works immediately. {"title": "Ship it", "tags": ["python" becomes {"title": "Ship it", "tags": ["python"]} and parses into a dictionary a template can render right now, several hundred tokens before the response ends.

It also fails constantly, and the failures are more interesting than the successes. {"tags": ["python", closes into a trailing comma. {"score": closes into a key with no value. {"score": 1. closes into a number literal that ends in a decimal point. Each of these is a decode error, so the snapshot comes back as None and the UI stalls until the stream happens to land on a lucky boundary.

Strings, Escapes, and the Repairs That Corrupt Data

The dangerous failure is different from the noisy ones. The noisy ones announce themselves: a trailing comma raises, the snapshot is None, and the next chunk usually fixes it. A brace inside a string value is not a brace, and nothing raises.

Consider the prefix {"note": "use {curly} braces. The scan above counts the { in the prose, pushes a second } onto the stack, never sees a matching close, and produces {"note": "use {curly} braces}}. That is not a decode error. Depending on where the stream stopped, a repair like this can produce a document that parses into the wrong shape — structure invented out of the model's prose.

This is not a rare input either. Any model writing about code, file paths, or template syntax emits braces and brackets inside string values constantly, and a JSON response about JSON is the worst case for a naive scanner. A snapshot that parses into the wrong shape is worse than one that fails to parse, because a consumer has no signal that anything went wrong.

Any scanner that does not know whether it is inside a string is guessing. The same applies to escapes: a buffer ending in a single \ is a partial escape sequence, and closing the string with " turns that trailing backslash into an escaped quote, which swallows the closer and pushes the corruption one level further out.

Correct handling needs three pieces of state carried between chunks: the container stack, an in-string flag, and an escape flag.

A Scanner That Remembers Where It Is

Each character is classified once, when it arrives, and never re-scanned. Every frame also records a safe truncation offset — the point at which the container held only complete elements — so a repair that fails can retreat to the last known-good boundary instead of returning nothing.

import json
from dataclasses import dataclass


@dataclass
class _Frame:
    close: str
    cut: int  # offset where this container held only finished elements


class PartialJSONStream:
    def __init__(self) -> None:
        self._buf: list[str] = []
        self._len = 0
        self._stack: list[_Frame] = []
        self._in_string = False
        self._escaped = False

    def feed(self, chunk: str) -> None:
        for ch in chunk:
            self._step(ch)
            self._buf.append(ch)
            self._len += 1

    def _step(self, ch: str) -> None:
        i = self._len
        if self._in_string:
            if self._escaped:
                self._escaped = False
            elif ch == "\\":
                self._escaped = True
            elif ch == '"':
                self._in_string = False
            return
        if ch == '"':
            self._in_string = True
        elif ch == "{":
            self._stack.append(_Frame("}", i + 1))
        elif ch == "[":
            self._stack.append(_Frame("]", i + 1))
        elif ch in "}]":
            if self._stack:
                self._stack.pop()
        elif ch == "," and self._stack:
            self._stack[-1].cut = i

The snapshot method generates repair candidates from most complete to most conservative and returns the first one that parses:

    def _closers(self, upto: int | None = None) -> str:
        frames = self._stack if upto is None else self._stack[:upto]
        return "".join(frame.close for frame in reversed(frames))

    def _candidates(self, text: str):
        head = text[:-1] if self._escaped else text
        if self._in_string:
            head += '"'
        head = head.rstrip()
        if head.endswith(","):
            head = head[:-1].rstrip()
        elif head.endswith(":"):
            head += " null"
        yield head + self._closers()
        for depth in range(len(self._stack) - 1, -1, -1):
            cut = self._stack[depth].cut
            yield text[:cut].rstrip().rstrip(",") + self._closers(depth + 1)

    def snapshot(self):
        text = "".join(self._buf)
        for candidate in self._candidates(text):
            try:
                return json.loads(candidate)
            except json.JSONDecodeError:
                continue
        return None

The fallback path is what makes partial numbers and half-typed literals harmless. A buffer ending in 1. or tru produces a first candidate that fails, then a truncated candidate that drops the unfinished element entirely and parses. The caller sees the object without that key rather than seeing nothing at all.

One property of this design has to be stated plainly to whoever consumes it: a string value in a snapshot may be a prefix of the real value. Rendering a partial description as it grows is the intended use. Comparing a partial value against an enum, treating it as a URL, or passing it to something that performs an action is a bug waiting for a slow token.

Turning Snapshots Into Field Events

Polling a whole snapshot per chunk is wasteful for consumers that only care when a field is done. JSON objects stream in order, which gives a simple and reliable completion rule: once a second key appears, the first key's value can no longer change. Every key except the last one in the snapshot is settled.

class FieldEmitter:
    def __init__(self) -> None:
        self.stream = PartialJSONStream()
        self._emitted: set[str] = set()

    def feed(self, chunk: str) -> list[tuple[str, object]]:
        self.stream.feed(chunk)
        snap = self.stream.snapshot()
        if not isinstance(snap, dict):
            return []
        events = []
        for key in list(snap)[:-1]:
            if key not in self._emitted:
                self._emitted.add(key)
                events.append((key, snap[key]))
        return events

    def finish(self) -> list[tuple[str, object]]:
        snap = self.stream.snapshot()
        if not isinstance(snap, dict):
            return []
        return [(k, v) for k, v in snap.items() if k not in self._emitted]

A downstream handler can now start work on classification while the model is still writing explanation, which is the whole point of streaming a structured response rather than a blob of prose. A routing decision can be dispatched, a database row can be reserved, a UI section can render as final rather than as a skeleton.

The ordering rule has one condition attached: it holds for objects the model writes in a fixed key order, which is what schema-constrained decoding produces. It does not hold for arrays of objects where a later element revises nothing but the last element is still growing, so treat the final element of a list as provisional in exactly the same way as the final key of an object.

Validation, Cancellation, and Output That Never Recovers

A snapshot is a draft, so validating it against the strict output model is wrong by construction — required fields are missing on purpose. Build a relaxed mirror of the model for snapshots and keep the strict one for the final value.

from pydantic import BaseModel, create_model


class Review(BaseModel):
    verdict: str
    reasons: list[str]
    score: int


def draft_of(model: type[BaseModel]) -> type[BaseModel]:
    fields = {
        name: (info.annotation | None, None)
        for name, info in model.model_fields.items()
    }
    return create_model(f"Draft{model.__name__}", **fields)


DraftReview = draft_of(Review)

Two failure modes need explicit handling around this. The first is a model that stops producing structure and starts producing an apology or a fenced code block; the snapshot goes None and stays None, so track consecutive unparseable chunks and abandon the stream rather than waiting for a close that is not coming. The second is early exit: when a snapshot already satisfies the strict model and the remaining fields are optional, cancelling the request stops paying for tokens nobody will read.

Cap the buffer as well. A partial parser will happily accumulate megabytes from a model stuck in a repetition loop, and a size ceiling on the buffer is the cheapest protection against that.

Testing One Character at a Time

The tests worth writing are the ones that feed input at the worst possible granularity, because a chunk size of one exercises every boundary a real tokenizer could ever produce.

import json
import pytest
from partial_json import PartialJSONStream

PAYLOADS = [
    {"verdict": "ship", "reasons": ["tests pass", "small diff"], "score": 8},
    {"note": 'use {curly} braces and a "quote"', "path": "C:\\tmp\\out.json"},
    {"nested": {"a": [1, 2, {"b": None}], "c": True}, "trailing": 1.5},
]


@pytest.mark.parametrize("payload", PAYLOADS)
def test_every_prefix_parses_or_declines(payload):
    text = json.dumps(payload)
    stream = PartialJSONStream()
    for ch in text:
        stream.feed(ch)
        snap = stream.snapshot()
        assert snap is None or isinstance(snap, (dict, list))
    assert stream.snapshot() == payload


@pytest.mark.parametrize("payload", PAYLOADS)
def test_settled_keys_never_change(payload):
    text = json.dumps(payload)
    stream = PartialJSONStream()
    seen: dict[str, object] = {}
    for ch in text:
        stream.feed(ch)
        snap = stream.snapshot()
        if isinstance(snap, dict):
            for key in list(snap)[:-1]:
                if key in seen:
                    assert seen[key] == snap[key]
                else:
                    seen[key] = snap[key]

The second test is the important one. It encodes the promise the emitter makes to its consumers — that a value handed out as settled is never revised — and it is the test that catches a scanner bug that only shows up when a brace appears inside a string, because the corrupted snapshot changes a key that had already been reported.

Property-based testing extends this cheaply. Generate arbitrary nested structures with Hypothesis, serialise them, feed every prefix, and assert the same two invariants. Any escape-handling mistake surfaces as a shrunk counterexample rather than as a support ticket about a mangled field.

What this buys is not a faster model. It is the removal of a wait that was never necessary — the interval between the moment a value is decided and the moment the closing brace lets the application admit it knows. The parser is roughly a hundred lines, the state it carries is three variables and a stack, and the correctness argument fits in two tests. That is a reasonable price for showing users an answer while it is still being written.

Originally published on Dispatch.

Lost Updates, Phantom Rows, and the Isolation Level That Prevents Them

Chen Yuan — Tue, 04 Aug 2026 10:02:55 +0000

Two Updates, One Row, No Agreement

Two application servers receive the same customer action at the same instant. Both read the customer's balance, both add a credit, both write the result. One credit disappears. Neither server did anything wrong in its own view — each read a consistent value, each wrote a new one, and the database accepted both writes without complaint.

This is a lost update, and it is not an application bug in the usual sense. The application code was correct for a world with one writer. The database, by default, does not promise that world. It promises that each individual statement sees a consistent snapshot, and that promise is strictly weaker than what the code assumed. The gap between the two is the entire subject of transaction isolation levels.

Isolation levels are the contract between the application and the database about what concurrent activity a transaction is allowed to observe. The SQL standard defines four of them, and every serious database implements at least three. The practical question is not which level is "best" — it is what each level actually guarantees, because the guarantees are narrower than most developers believe, and the default level is not the safest one.

The Four Anomalies That Isolation Is Supposed to Prevent

The standard defines isolation levels by the anomalies they allow or prevent. Four anomalies matter in practice.

A dirty read is reading a row written by a transaction that has not committed yet. The danger is not the read itself — it is that the writing transaction may roll back, and the reader has already built logic on top of a value that never existed.

A non-repeatable read is reading the same row twice in one transaction and getting different values, because another transaction committed a change in between. The first read was of a committed value; so was the second. They simply disagree with each other.

A phantom read is the row-count version of the same problem: a query with a filter returns a different set of rows on its second execution, because another transaction inserted or deleted rows that match the filter between the two runs.

A lost update happens when two transactions read the same value, both modify it based on what they read, and the second write silently overwrites the first. No error is raised. No constraint is violated. The database simply keeps the last write, and the earlier transaction's work evaporates.

Each isolation level is a position on a spectrum: which of these four anomalies the database is allowed to let through. The safest level prevents all four. The weakest prevents only the first. Everything between is a trade.

Read Uncommitted: The Level Postgres Refuses to Ship

The weakest standard level, read uncommitted, permits dirty reads. A transaction at this level may see rows that another transaction has modified but not committed — including rows that will be rolled back moments later.

PostgreSQL does not implement it. It is not an oversight; it is an architectural consequence. PostgreSQL's concurrency control is built on snapshots taken at the moment a statement begins, and a snapshot by construction only contains rows that were committed before the snapshot was taken. There is no code path that shows a transaction an uncommitted row, because there is no code path that reads outside its snapshot.

Setting the level to read uncommitted in PostgreSQL is legal and silently equivalent to read committed. This is worth knowing because it changes how you read documentation and advice from other databases: in MySQL, read uncommitted is real and reachable; in PostgreSQL it is a label with no effect. Code that depends on seeing uncommitted data to work around a locking problem will not work here, and the fix is not to find a lower level — there is none.

Read Committed: The Default You Already Run On

Read committed is PostgreSQL's default, and it is the level most applications use without thinking. Each statement gets its own snapshot, taken when the statement starts. The statement sees every row committed before that moment, and nothing committed after.

The guarantee is per-statement, not per-transaction. Two SELECTs in the same transaction can return different results if another transaction commits between them. This is the source of non-repeatable reads, and for most applications it is invisible, because most transactions are short and most applications do not re-read the same row with the same filter inside one transaction.

Read committed also does not prevent lost updates. Two transactions can both read a row under read committed, both compute a new value, and both write it; the writes serialize, but the reads do not, and the earlier computation is discarded. This is the default behavior of almost every PostgreSQL installation, which means the opening scenario — two credits, one balance — is the ordinary outcome, not a misconfiguration.

The mitigation for lost updates under read committed is not a different isolation level. It is making the update itself conditional; the statement re-reads the row inside its own snapshot and computes from the current value, which closes the read-modify-write gap for this specific pattern:

UPDATE accounts
   SET balance = balance + 50
 WHERE account_id = 42;

The same trick extends to the upsert form, which atomically decides between insert and update based on the row state at statement time:

INSERT INTO counters (key, value)
VALUES ('visits', 1)
ON CONFLICT (key)
DO UPDATE SET value = counters.value + 1;

Neither form needs a transaction boundary to be safe against lost updates, because the conflict check and the write happen in one atomic statement. The moment the application splits the operation into a SELECT followed by a separate UPDATE, it is back on the read-modify-write path, and the isolation level decides what happens next.

Repeatable Read: The Snapshot That Outlives the Statement

Repeatable read is where the semantics change qualitatively. PostgreSQL implements it by taking the snapshot once, at the first statement of the transaction, and reusing it for the whole transaction. Every statement in the transaction sees the same world, including the same set of committed rows and the same versions of those rows.

Non-repeatable reads disappear: reading the same row twice in one transaction returns the same value, because there is only one snapshot. Phantoms are also prevented for the same reason — the row set is fixed when the snapshot is taken, so a filtered query returns the same rows every time, regardless of what other transactions commit in between.

Lost updates, however, are not prevented; they are detected. When a transaction at repeatable read tries to update a row that a concurrent committed transaction also updated, PostgreSQL raises a serialization failure instead of silently overwriting. The application receives an error and must retry the transaction. This is a crucial distinction: the anomaly is not allowed to pass silently, but the transaction is also not made safe automatically. Code that never handles a serialization failure will crash at exactly the moment the data was about to be corrupted.

Applications that genuinely need repeatable-read semantics — reporting queries that must aggregate a consistent point-in-time view, or batch jobs that read a stable row set while users keep writing — should set the level deliberately and add retry logic around serialization failures, because under the default level those queries quietly mix data from different moments.

Serializable: Where the Database Starts Saying No

Serializable is the strongest standard level: transactions must behave as if they ran one after another, in some order, even though they actually ran concurrently. PostgreSQL implements it with serializable snapshot isolation, which runs transactions under repeatable-read snapshots and tracks read-write dependencies between them. When the dependency graph gains a cycle — meaning the concurrent schedule could not be linearized — the database aborts one of the transactions with a serialization failure.

The practical shape is identical to repeatable read plus stricter failure detection. The application still must retry aborted transactions; the difference is that the set of abortable schedules is larger, because serializable also catches interleavings where two transactions read and write overlapping ranges in ways that could not happen in any serial order.

The honest trade is throughput. Serializable transactions abort more often, and every abort forces the application to replay work. Databases that default to serializable are choosing correctness over concurrency; PostgreSQL defaults the other way, and the operator who wants serializable behavior must set it per session or per transaction and build the retry loop that makes it livable.

Choosing serializable is right when the correctness of a business invariant depends on the absence of any interleaving: seat allocation, inventory decrements, balance transfers across accounts. For those cases, the retry loop is cheap next to the audit that follows a silently lost update.

Choosing a Level Without Guessing

The four levels form a practical decision tree, and the answer is usually determined by two questions: does the transaction re-read data it has already read, and does it compute a new value from a value it read?

For short transactions that read once and write once, read committed is correct, and the update should use the atomic conditional forms that close the read-modify-write gap. For transactions that read the same rows or the same filtered set more than once and must see a consistent world, repeatable read is the level — with a retry handler around serialization failures. For transactions whose interleaving with other transactions would corrupt a business invariant, serializable is the level — with the same retry handler, applied more often.

Set the level in the connection string or session configuration, not by sprinkling per-statement hints, because the level is a property of the transaction, and a transaction is a property of the connection:

BEGIN;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
-- all statements in this transaction share one snapshot
COMMIT;

A mixed application where different transactions run at different levels is correct only if every code path sets its own level before the first statement of its transaction; the moment one path forgets, it silently inherits the default.

A Test That Shows Which Level You Are On

The fastest way to internalize the difference is to observe it. Two sessions, one table, one row, and a deliberate pause between read and write demonstrate each guarantee directly.

In session one, begin a transaction and read a row. In session two, update that row and commit. In session one, read again. Under read committed, the second read shows the new value; under repeatable read, it shows the old one — the snapshot wins. Run the same sequence with both sessions updating the same row from a read value, and the second commit under repeatable read raises a serialization failure, while under read committed it silently succeeds and discards the first update.

That contrast — an error versus silence — is the whole point of moving up the isolation spectrum. The database cannot make concurrent logic correct; it can only refuse to execute interleavings that would break the contract. Lower levels let the interleaving happen and keep the result. Higher levels abort the transaction and ask the application to try again. The isolation level you choose is not a performance knob. It is a statement about whether the database is allowed to be quietly wrong, and the application's retry loop is the price of answering no.

Originally published on Dispatch.

Your Async Logs Are Lying to You: Correlation IDs with Contextvars

Chen Yuan — Mon, 03 Aug 2026 14:53:46 +0000

The Three Logs That Never Connected

An order fails at 14:02. The support ticket quotes a customer who saw a payment error, then a retry, then another error. Three log lines carry the same timestamp window, and none of them agree on what happened.

The payment service logged payment.authorized at 14:02:11. The order service logged order.failed at 14:02:11. The worker that was supposed to reconcile them logged worker.idle at 14:02:12. Three services, three log streams, three different stories. Somewhere between those lines a state change was lost, and nobody can say where, because nothing in the logs connects them.

This is the classic symptom of a system where every log line is locally true and globally useless. The fix is a correlation ID: one opaque string that travels with every request, crosses every service boundary, and gets stamped onto every log line it touches. In synchronous code this is a solved problem — a header, a middleware, done. In async Python, the interesting part is that the mechanism you reach for first is quietly wrong, and the one that works is hiding in a corner of the standard library most people have never imported.

Why Thread-Local State Breaks in Async Code

The instinctive move is threading.local(). One thread, one request, one context — that is the mental model that made thread-local storage popular, and it is exactly the model that async code violates.

An asyncio event loop runs thousands of tasks on one thread. The thread does not change when the application switches from handling request A to handling request B; the task does. threading.local() keys its storage on the thread identity, so every task on the same thread reads and writes the same slot. Request A sets request_id = "a1", awaits I/O, and while it is suspended, request B sets request_id = "b2". When A resumes and logs its next line, the filter reads request_id and stamps the line with "b2".

import threading

local = threading.local()

async def handle(request_id: str) -> None:
    local.request_id = request_id
    await some_io()          # loop switches to another task here
    print(local.request_id)  # may already be a different request's ID

The result is not a missing ID. It is worse: a wrong ID. Log lines get stamped with the neighbor's request, and the timeline you are trying to reconstruct becomes actively misleading. This is the "locally true, globally useless" failure in its purest form, and it is why the standard advice for async code is to stop storing context in places the runtime can swap underneath you.

Contextvars: State That Travels With the Task

contextvars exists to fix precisely this. A context variable holds one value per execution context, and an asyncio task carries its own context. When the loop suspends one task and resumes another, each task sees its own copy of every context variable, with no shared slot and no cross-talk.

import contextvars

request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="")

async def handle(request_id_value: str) -> None:
    request_id.set(request_id_value)
    await some_io()            # loop can switch tasks freely
    print(request_id.get())    # still "a1" for this task

Two properties make this safe. First, assignment inside a task is visible only to that task and to whatever it awaits; sibling tasks never observe it. Second, when the task finishes, its context is discarded, so there is no leak into the next request. The value follows the logical unit of work, not the thread — which is precisely the semantics a correlation ID needs.

There is a cost worth naming: contextvars is not free. Setting and reading a context variable involves dictionary lookups in the current context, and every await can trigger context bookkeeping. In a logging hot path the cost is negligible next to the I/O you are already doing, but if you were planning to use a context variable inside a tight numerical loop, measure first.

A Correlation ID Middleware in Thirty Lines

The middleware has three jobs: accept the ID from upstream if one exists, generate a fresh one if it does not, and make sure the response carries it back so the caller can attach it to their own logs. With contextvars the implementation is short enough to read in one pass.

import contextvars
import uuid
from starlette.middleware.base import BaseHTTPMiddleware

request_id_var: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="")

class CorrelationMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        incoming = request.headers.get("X-Request-ID", "").strip()
        rid = incoming or uuid.uuid4().hex[:16]
        request_id_var.set(rid)
        response = await call_next(request)
        response.headers["X-Request-ID"] = rid
        return response

The same shape ports to aiohttp middlewares, FastAPI dependencies, and raw asyncio server handlers. The important part is not the framework hook — it is that the ID lives in request_id_var, so every downstream function that awaits inside this request can read it without threading a parameter through a dozen call sites.

This matters more than it looks. The alternative — passing the ID as an explicit argument to every function that logs — turns every signature into a carrier for plumbing. Adding a field to a dataclass, a parameter to a helper, or an argument to a third-party callback all become breaking changes. The context variable keeps the data flow implicit, and the call sites stay honest: a function logs what it needs because the context is already there.

The Context Copy Trap: Threads and Executors

Here is the corner that catches most implementations: asyncio.create_task() copies the current context, but work dispatched to a thread pool executor does not inherit it.

A task that calls loop.run_in_executor(None, blocking_call) hands blocking_call to a worker thread. That thread runs with the default context, in which request_id is still the empty default. The correlation ID silently vanishes exactly at the boundary where you most need it — the slow, blocking call that takes real seconds and logs real failures.

import asyncio
import contextvars

request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="")

def blocking_call() -> None:
    print(request_id.get())  # "" — the executor thread has the default context

async def handle(rid: str) -> None:
    request_id.set(rid)
    await asyncio.get_running_loop().run_in_executor(None, blocking_call)

asyncio.run(handle("a1"))

The fix is to propagate the context explicitly. Python 3.11 added asyncio.to_thread(), which copies the caller's context before dispatching — the one-liner replacement that makes the ID survive the boundary:

async def handle(rid: str) -> None:
    request_id.set(rid)
    await asyncio.to_thread(blocking_call)  # context is copied automatically

For code that must stay on run_in_executor, copy the context by hand and run the callable inside it:

ctx = contextvars.copy_context()
await loop.run_in_executor(None, lambda: ctx.run(blocking_call))

Pick one rule and apply it everywhere: prefer asyncio.to_thread, and audit every remaining run_in_executor for an explicit ctx.run. A correlation ID that dies in the first thread pool is worse than none, because the logs now look complete while silently missing the section of the timeline that actually matters.

A Logging Filter So Every Line Carries the ID

A middleware sets the variable, but the logs still need to read it. Python's logging.Filter runs against every record before it is emitted, which makes it the natural home for stamping:

import logging

class RequestIDFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_var.get() or "-"
        return True

handler = logging.StreamHandler()
handler.addFilter(RequestIDFilter())
logging.basicConfig(handlers=[handler], level=logging.INFO,
                    format="%(asctime)s %(request_id)s %(name)s %(message)s")

Every line — from application code, from library loggers, from warning paths — now carries the ID, because the filter does not care who created the record. One filter, attached to the root handler, covers code you wrote and code you did not.

This is the property that makes the three logs connect again. The payment line, the order line, and the worker line all carry the same X-Request-ID value, and the support ticket becomes a search: filter the aggregated stream by that one string and the entire timeline of the failed order is contiguous. The correlation ID does not make the logs truthful; it makes them sortable by truth.

Verifying the Timeline Comes Back

The implementation is not finished until the cross-talk regression is actually tested. Two tests cover the failure modes that matter: concurrent tasks must not share IDs, and executor work must inherit them.

import asyncio
import contextvars

def test_tasks_do_not_share_ids():
    seen: set[str] = set()

    async def worker(value: str) -> str:
        request_id.set(value)
        await asyncio.sleep(0)
        return request_id.get()

    async def main() -> None:
        results = await asyncio.gather(worker("a1"), worker("b2"))
        seen.update(results)

    asyncio.run(main())
    assert seen == {"a1", "b2"}  # no task saw its neighbor's ID

def test_executor_inherits_context():
    captured: list[str] = []

    def blocking() -> None:
        captured.append(request_id.get())

    async def main() -> None:
        request_id.set("a1")
        await asyncio.to_thread(blocking)

    asyncio.run(main())
    assert captured == ["a1"]

Run these under the CI job that runs your async tests. If either fails, the fix is one of the two patterns above — not a new configuration flag and not a global variable.

When Contextvars Are Not Enough

The context variable carries the ID through one process. The moment the request crosses a network boundary, the ID must ride in the wire protocol: an X-Request-ID header on HTTP, a traceparent header for OpenTelemetry, a message_id field on a queue. The middleware already propagates the header forward; the same header should be read from upstream and attached to outbound calls, so a chain of three services produces one searchable ID instead of three locally unique ones.

For systems that already run OpenTelemetry, the pragmatic move is to back the correlation ID with the trace context rather than reinventing it — read traceparent, store the trace ID in the context variable, and let the filter stamp it. The pattern does not change; the source of the string does.

And one honest limit: contextvars does not magically make logs from a crashed process appear. It closes the gap between log lines that were both written; it cannot resurrect the line that was never written because the process died mid-flush. For that you still need durable, buffered shipping. But for the ordinary failure — the one where every service logged something and nobody could connect the dots — the correlation ID is the difference between a timeline and a pile of timestamps.

Originally published on Dispatch.