DEV Community: Chen Yuan

Fix HTTP 429 in Python Requests Without Retrying POST by Accident

Chen Yuan — Tue, 13 Oct 2026 05:52:00 +0000

A Python data collector can run for an hour and then suddenly produce “429 Too Many Requests.” The usual emergency fix is a loop that sleeps for one second and repeats the HTTP call. That can make the situation worse. The code may ignore the server's stated waiting period, multiply load across several workers, or send a write operation twice after the server already processed it.

The better response is specific: inspect the actual failure, respect Retry-After when present, bound the number of attempts, and separate safe read operations from actions that change server state. This tutorial uses Requests and urllib3. Its examples show their intended behavior and include local unit tests instead of making speculative requests to a live API.

Find out what received the 429 before adding retries

A 429 means a server is applying a rate limit. It does not identify which rule was exceeded. The limit might be associated with an account, token, endpoint, organization, or network address. The HTTP standard allows a server to send a Retry-After field, but does not require one. Different services also expose different rate-limit headers, so there is no universal quota header your code can assume.

Begin with one diagnostic request. Record the method, host, path, status, and Retry-After if the response contains it. Do not print bearer tokens or full URLs containing private query parameters. If a batch worker has ten threads, determine whether they all share the same quota before adjusting one thread's settings.

import requests

url = "https://api.example.com/items"
try:
    response = requests.get(url, timeout=(3.05, 10))
    print("status:", response.status_code)
    print("retry-after:", response.headers.get("Retry-After"))
    response.raise_for_status()
except requests.ConnectTimeout:
    print("Could not establish a connection in time")
except requests.ReadTimeout:
    print("Connected server stopped sending data in time")
except requests.HTTPError as exc:
    print("Server returned:", exc.response.status_code)

Replace the placeholder URL with an endpoint that you are permitted to use. Requests timeouts should be explicit: otherwise an unresponsive call can wait indefinitely. The pair in this example sets separate connection and read inactivity timeouts. Neither value is a deadline covering the entire download. That distinction matters when retries and slow responses consume far more time than the developer expected.

Interpret Retry-After correctly

The HTTP Retry-After header supports two forms. One is a non-negative number of seconds, such as Retry-After: 120. The other is an HTTP date. A program that simply converts every header to an integer will break on the date form. A program that sleeps for one second regardless of the header may send the next request much earlier than the server asked.

RFC 9110 defines those forms, and RFC 6585 defines the 429 response. Neither establishes a universal waiting time for all APIs. The server may omit Retry-After, or a service may apply a quota that requires a longer pause outside the current request.

You do not normally need to implement both date parsing and retry scheduling yourself. urllib3's Retry component understands the header for certain response codes, including 429 and 503. When no applicable header is present, an exponential backoff setting provides a separate delay policy. These mechanisms are not interchangeable: an explicit server wait should not be silently replaced by an arbitrary short delay.

There is also a deadline problem. If the server asks for an hour and the job has two minutes left, sleeping for an hour is not a successful recovery strategy. Preserve the cursor or checkpoint and defer the remaining work. A healthy queue recognizes when it cannot finish within its allowed time.

Configure a bounded retry policy for GET

Requests uses urllib3 internally, and a Session can be configured with an HTTPAdapter. Make the policy explicit. In particular, list which methods may retry and which status codes are considered temporary. The following example limits automatic retries to GET and HEAD, then handles the exhausted-response case normally.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

policy = Retry(
    total=3,
    connect=3,
    read=0,
    status=3,
    other=0,
    allowed_methods=frozenset({"GET", "HEAD"}),
    status_forcelist=(429, 502, 503, 504),
    respect_retry_after_header=True,
    backoff_factor=0.5,
    backoff_jitter=0.2,
    raise_on_status=False,
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=policy)
session.mount("https://", adapter)
session.mount("http://", adapter)

response = session.get(
    "https://api.example.com/items?page=1",
    timeout=(3.05, 10),
)
response.raise_for_status()
items = response.json()
print("items:", len(items))
session.close()

The total parameter permits up to three retries after the original request. It does not mean three attempts in total. The selected status codes are deliberately narrow. A validation error such as 400 is not cured by trying the same malformed request repeatedly; a 401 usually needs an authentication fix rather than a backoff delay.

Setting read to zero is conservative. An error while reading can happen after the upstream server received the request. Although a properly implemented GET should be safe to repeat, it is still useful to decide consciously which transport failures your application will replay. Adjust this only after examining the semantics of the upstream endpoint.

Exponential backoff does not necessarily pause before the first retry. The documented urllib3 algorithm commonly begins with an immediate retry when there is no Retry-After instruction, then increases delays for later attempts. Backoff jitter adds small random variation to avoid a synchronized retry wave. Neither feature controls the combined rate of multiple workers in different processes.

Keep POST out of automatic retries until its contract is known

The dangerous case is not a clean 429 received from a GET endpoint. It is a lost response from a write operation. Imagine a POST that creates an invoice: the server stores the invoice, but the client times out while waiting for confirmation. Sending the same POST again could create a duplicate invoice. A timeout is not proof that nothing happened.

Do not widen allowed_methods to include every HTTP method merely because it makes transient errors disappear. The sample intentionally excludes POST. It also does not claim that every GET endpoint is implemented correctly or every PUT is automatically harmless. You have to understand the particular API's documented behavior.

Some providers offer idempotency keys for writes. A client assigns an identifier to one intended operation, retains it across permitted retries, and uses the provider's published contract to prevent duplicate execution. This protection works only when the server explicitly supports it. Adding an invented header to an arbitrary API does not create idempotency.

If the provider offers no dependable idempotency mechanism, persist the local action identifier before submission and query the official object or job status after an ambiguous timeout. If the server confirms completion, record success rather than repeating the request. If the result remains unknown, preserve an unresolved state for reconciliation. This is slower than pressing retry, but prevents silent duplicate side effects.

The same principle applies after retrieving data. Store external object IDs and update existing records instead of inserting a new database row for every repeated page fetch. A retry should not turn one record into two.

Prove the retry decisions without calling a live API

It is wasteful to consume a real service's quota just to find out whether a Python retry configuration behaves as expected. You can inspect the urllib3 policy using synthetic response metadata and Python's built-in unittest. The following tests make no network calls.

import unittest
from urllib3.response import HTTPResponse
from urllib3.util import Retry

class RetryPolicyTests(unittest.TestCase):
    def setUp(self):
        self.policy = Retry(
            total=3,
            allowed_methods=frozenset({"GET", "HEAD"}),
            status_forcelist=(429, 503),
            respect_retry_after_header=True,
        )

    def test_get_retries_429(self):
        self.assertTrue(self.policy.is_retry("GET", 429))

    def test_post_does_not_retry_429(self):
        self.assertFalse(self.policy.is_retry("POST", 429))

    def test_reads_retry_after_seconds(self):
        response = HTTPResponse(
            status=429,
            headers={"Retry-After": "5"},
            preload_content=False,
        )
        self.assertEqual(self.policy.get_retry_after(response), 5.0)

    def test_bad_request_is_not_retried(self):
        self.assertFalse(self.policy.is_retry("GET", 400))

if __name__ == "__main__":
    unittest.main()

The expected result is four passing tests. These checks establish that the chosen methods, status codes, and numeric Retry-After behavior match the policy. They do not prove that your specific API accepts a request or that a live account can safely retry a write.

For an integration test, use a local HTTP server programmed to return a short sequence of 429 and 200 responses, then count requests. Add cases for missing Retry-After, an HTTP-date, and exhaustion of the attempt budget. Do not simulate production by sending dozens of rapid calls to a third-party service. A local test can reveal a looping bug without affecting another operator's quota.

Separate rate limits from network and permission failures

A 429 is a real HTTP response. A connection timeout is not. Requests distinguishes ConnectTimeout and ReadTimeout, and its documentation explicitly states that a ConnectTimeout is safe to retry. A ReadTimeout is more ambiguous for a method that changes server state because the request may already have reached the server.

Treat a 401 as an authentication issue until evidence shows otherwise. Treat a 403 as an access or policy problem rather than assuming an ordinary quota reset will fix it. A 400 often reflects invalid input. These categories should produce different recovery actions, even when they all appear in a generic error dashboard.

Log only what is necessary. A host and path can identify the affected endpoint without copying secrets from a query string. If the path itself contains user identifiers, redact that too.

from urllib.parse import urlsplit

def safe_event(method, url, status, retry_after=None):
    parts = urlsplit(url)
    return {
        "method": method.upper(),
        "host": parts.hostname,
        "path": parts.path,
        "status": status,
        "retry_after": retry_after,
    }

print(safe_event(
    "GET",
    "https://api.example.com/items?access_token=private",
    429,
    "30",
))

This sample removes query parameters from its diagnostic object. It is not a complete privacy policy for every application. In practice you may also need endpoint-specific field redaction, a correlation ID, and a rule for how long diagnostic events are retained.

Give the entire job a quota and a recovery point

Per-request backoff is only one layer. If a crawler schedules one hundred URLs and gives every URL three retries, a broad outage can create up to four hundred attempts. Add five worker processes and the total load becomes much harder to reason about. The fix cannot be limited to the sleep call in one function.

Set a concurrency cap, request budget, and application-level deadline. For paginated data, persist the last fully processed cursor. On resumption, start from a known state and deduplicate by stable record identifiers. When an API supports batch retrieval or conditional caching, use its documented features instead of repeatedly downloading unchanged content.

Be careful about what success means. If a collector fetched nine of ten pages and exhausted the last retry, do not mark the dataset as a complete refresh. Keep the partial result and record which page remains. Explicit incomplete status is more useful than a fresh timestamp attached to missing data.

Review the workload before tuning the retry mechanism. Sometimes the real cause is a duplicate scheduler, accidental loop, unnecessary polling interval, or several deployments that all call the same API. Lowering the initial request rate can resolve the incident without adding a complicated retry strategy.

The rule to keep

For HTTP 429, read the server's response, honor the supported wait instruction, use a finite attempt budget, and retry only operations whose semantics justify it. For uncertain writes, reconcile before repeating. Then test the policy locally and measure total requests rather than simply counting errors that disappeared.

The primary references are urllib3 Retry, Requests timeout behavior, RFC 6585, and RFC 9110. The important distinction is between a temporary server response and proof that an intended operation was never performed.

Originally published on Dispatch.

Fix Broken Links in Python Crawlers with urljoin and a Same-Origin Check

Chen Yuan — Mon, 12 Oct 2026 04:19:07 +0000

A crawler reads a perfectly ordinary link: ../pricing. Your script appends it to the website address, requests the wrong page, and logs a failure. A second link begins with //, and the crawler quietly leaves the website it was supposed to inspect.

Both problems can hide behind a reassuring result: the HTTP request worked. What failed was the decision about which address to request.

This walkthrough builds a small resolver for a documentation crawler. It uses Python's standard library, needs no API account, and can be tested without visiting any website. The aim is specific: resolve ordinary links correctly, keep accepted links on one origin, and make rejected inputs visible.

Start with the page that actually supplied the link

Imagine a documentation page at https://docs.example.com/guides/install.html. It contains links to ../pricing, /faq, and ?lang=en. These are deliberately different inputs. Treating them all as fragments to concatenate hides those differences and makes debugging harder.

Use the current page address as the base. A website home page is often the wrong base for a link extracted from a nested document. Keep the raw extracted link in your diagnostic record so you can explain the eventual address.

Python provides urljoin for this operation. Its documentation also warns that an absolute second argument can replace the base address. Resolving an address and deciding whether to visit it are separate steps.

Run this small example before adding a network client:

from urllib.parse import urljoin

base = "https://docs.example.com/guides/install.html"
samples = [
    "../pricing",
    "/faq",
    "?lang=en",
    "//other.example.com/news",
]

for href in samples:
    print(href, "->", urljoin(base, href))

The first result ends in /pricing; the second ends in /faq. The query-only input keeps the document path. The fourth input points at the other host. These outputs are reproducible local examples, not a claim about any real site's behavior.

That last case is useful. The resolver has correctly interpreted the input. Your crawler still needs a rule telling it that the resulting host is outside its assignment.

Define the boundary before writing the helper

For this example, an origin means the scheme, lowercase hostname, and effective port. The expected origin is supplied separately from the current page. This separation prevents a redirected page from silently becoming a new trusted base.

The chosen policy accepts HTTP and HTTPS addresses only. It refuses embedded usernames or passwords, backslashes, control characters, and leading or trailing whitespace. It removes fragments because this crawler collects documents rather than positions within documents.

These are application choices for a constrained documentation crawler. A general browser, a sitemap generator, and a login redirect handler can need different behavior. Keeping the choices explicit makes the helper easier to review.

The helper also checks the base itself. If the caller accidentally passes an external page, even a relative link is rejected. Otherwise a redirect could move the crawler's base, and a comparison against that moved base would approve the wrong destination.

This gives you an answer to a practical review question: what does a successful return value promise? It promises a resolved HTTP address within the configured origin, under the deliberately narrow input rules below. It does not promise that a request has succeeded or that a destination is safe to fetch.

Use a resolver with a visible rejection result

Save the following as resolver.py. A rejected input returns None, which lets the calling loop skip it without mistaking it for a request failure. If your application needs rejection reasons, replace that result with a small result object.

from urllib.parse import urljoin, urlsplit, urlunsplit

def origin(parts):
    scheme = parts.scheme.lower()
    if scheme not in ("http", "https"):
        raise ValueError("unsupported scheme")
    if not parts.hostname:
        raise ValueError("missing hostname")
    if parts.username is not None or parts.password is not None:
        raise ValueError("credentials in URL")
    port = parts.port
    if port is None:
        port = 443 if scheme == "https" else 80
    return scheme, parts.hostname.lower(), port

def clean_input(value):
    if not isinstance(value, str) or not value:
        return False
    if value != value.strip() or "\\" in value:
        return False
    return not any(ord(ch) < 32 or ord(ch) == 127 for ch in value)

def resolve_link(base, href, allowed_origin):
    if not clean_input(base) or not clean_input(href):
        return None
    try:
        base_parts = urlsplit(base)
        if origin(base_parts) != allowed_origin:
            return None
        target = urlsplit(urljoin(base, href))
        if origin(target) != allowed_origin:
            return None
        return urlunsplit((
            target.scheme.lower(), target.netloc,
            target.path, target.query, ""
        ))
    except ValueError:
        return None

The effective-port comparison avoids treating an explicitly written default port as a different origin. A genuinely different port is rejected. Accessing a malformed port can raise an exception; the helper returns the same rejection result instead of letting that input terminate the whole crawl.

Notice what the code does not normalize. It leaves the query intact. A page such as /search?q=python can differ from /search?q=sqlite. Removing every query parameter to reduce duplicates would be a separate content policy, with different consequences.

It also preserves the output authority spelling. The comparison uses a normalized hostname and effective port, but the returned address is not a complete canonical identity. If your queue requires stricter deduplication, define that normalization independently and test it against your own site's URLs.

Test the outputs that matter to the crawler

A useful test set contains accepted links and rejected links. Testing only ordinary relative paths would miss the boundary that motivated the helper. You can run the following as test_resolver.py, beside the helper.

import unittest
from resolver import resolve_link

BASE = "https://docs.example.com/guides/install.html"
ALLOWED = ("https", "docs.example.com", 443)

class ResolverTests(unittest.TestCase):
    def test_expected_destinations(self):
        cases = [
            ("../pricing", "https://docs.example.com/pricing"),
            ("/faq", "https://docs.example.com/faq"),
            ("?lang=en",
             "https://docs.example.com/guides/install.html?lang=en"),
            ("#part",
             "https://docs.example.com/guides/install.html"),
            ("https://docs.example.com:443/faq",
             "https://docs.example.com:443/faq"),
        ]
        for href, expected in cases:
            with self.subTest(href=href):
                self.assertEqual(
                    resolve_link(BASE, href, ALLOWED), expected
                )

    def test_rejected_destinations(self):
        for href in [
            "//other.example.com/news",
            "https://docs.example.com.other.example/faq",
            "http://docs.example.com/faq",
            "https://docs.example.com:444/faq",
            "https://user@docs.example.com/faq",
            "https://docs.example.com:bad/faq",
            "mailto:hello@example.com",
            " /faq",
            "/fa\tq",
            "",
            None,
        ]:
            with self.subTest(href=href):
                self.assertIsNone(resolve_link(BASE, href, ALLOWED))

    def test_base_cannot_change_the_allowlist(self):
        self.assertIsNone(resolve_link(
            "https://other.example.com/page", "/faq", ALLOWED
        ))

if __name__ == "__main__":
    unittest.main()

Run python -m unittest -v test_resolver. The suite contains three test methods covering seventeen input cases. All decisions happen locally; there is no DNS lookup, HTTP traffic, or dependency installation.

The near-match hostname is especially useful for catching a tempting shortcut: searching the complete address for a trusted hostname string. A string can contain the expected text and still describe a different host. Here the test asks for the exact parsed hostname.

When a site-specific case fails, keep the failing input and expected destination. That becomes a durable example for the next person who changes the helper, instead of a comment saying that URL handling is complicated.

Keep resolution out of the request loop

Place resolution between extraction and queueing. The extractor supplies the page address and raw link. The resolver supplies an accepted destination or a rejection. The request worker operates only on accepted destinations.

Record those outcomes separately. An out-of-scope link is a policy skip, while a timeout is a fetch failure. Combining them in one failure counter makes the crawler look unreliable and sends debugging effort toward the network.

For a first version, a compact record containing the base, input, accepted address, and outcome is enough. Avoid logging sensitive query values from private pages. A documentation crawler can often operate on public pages and retain only the examples needed for troubleshooting.

You can also deduplicate the accepted address before placing it in the queue. Fragment removal already collapses section links for this example, but it does not decide whether different queries represent the same document. Make that decision from the site's behavior, rather than a universal rule.

Handle redirects at the network boundary

The helper checks a requested address before fetching. It does not control redirects followed by the HTTP client. Python's urllib.request includes redirect handling, so the request path needs its own explicit policy.

For a constrained crawler, either disable automatic redirects and inspect each next destination, or use a client configuration that checks every redirect before following it. Checking the final response address is useful for diagnostics, but it happens after the request.

The same-origin helper is also not a complete SSRF defense. A service fetching user-supplied URLs needs network-level protections and a threat model covering DNS, private destinations, and redirect behavior. Do not promote this small crawler utility into a security boundary for arbitrary internet inputs.

Those requirements belong in the fetch layer. Keeping them there lets the resolver remain small and gives each layer a testable responsibility.

Apply the fix to one broken page first

Pick a page where your crawler currently produces a bad destination. Capture its actual base address and one extracted link. Reproduce the result locally, add the intended result to the tests, and only then resume the crawl.

This is a small change with an immediate observable outcome: the queue contains the addresses you meant to visit, and rejected links no longer masquerade as network failures. It gives a useful first repair before adding browser automation or a larger scraping framework.

Which input has caused the most confusing result in your crawler: a nested path, a query-only link, or a redirect? An exact base-and-link pair is more useful than a general description of a broken scraper.

Sources: Python URL parsing, Python HTTP request handling, and Python unittest.

Originally published on Dispatch.

SQLite WAL Does Not Mean Unlimited Writers: A Practical Python Playbook

Chen Yuan — Sun, 11 Oct 2026 06:22:25 +0000

A small service can run happily on SQLite for months, then begin throwing database is locked as soon as two background workers start processing jobs. Switching to Write-Ahead Logging (WAL) often improves the situation, but it does not turn SQLite into a multi-writer database. That distinction is the first thing an engineer needs to understand before adding more retries or raising every timeout.

SQLite's official WAL documentation is precise: readers and a writer can operate at the same time, but there can still be only one writer at a time. This article builds a compact Python pattern around that reality. It covers transaction boundaries, connection lifetimes, bounded retries, crash-safe work identities, and checkpoint monitoring. The examples are illustrative recipes, not benchmarks; measure them under your own workload.

The lock is a symptom of transaction design

In rollback-journal mode, readers and writers interact through locking that can limit concurrency. WAL changes the commit protocol: writers append to a write-ahead log, while readers use a snapshot of committed data. That reduces interference between ordinary reads and writes. It does not eliminate contention among transactions that need the write lock.

Imagine a worker that starts a transaction, calls an external HTTP service, transforms a payload, and finally updates a row. Another worker arrives during that slow network call. Even if each SQL statement is tiny, the first worker is holding scarce database capacity for the entire external request. WAL cannot make that transaction cheap.

The first remedy is often architectural, not a PRAGMA: finish network calls, validation, and CPU-heavy work before acquiring a write transaction. Commit the smallest coherent database change, then release the connection promptly.

A write transaction should protect a database invariant, not the whole business workflow. If you need to coordinate an expensive workflow across failures, store durable task states and do the expensive work outside the transaction.

Initialize WAL once and configure connections deliberately

A minimal local database can use one-time setup followed by short-lived connections. Python's standard sqlite3 module is sufficient for this example, and SQLite's PRAGMA reference documents the relevant settings.

import sqlite3
from pathlib import Path

DB = Path("jobs.sqlite3")

def connect():
    con = sqlite3.connect(DB, timeout=5, isolation_level=None)
    con.execute("PRAGMA busy_timeout=5000")
    con.execute("PRAGMA foreign_keys=ON")
    return con

def initialize():
    with connect() as con:
        mode = con.execute("PRAGMA journal_mode=WAL").fetchone()[0]
        if mode.lower() != "wal":
            raise RuntimeError(f"WAL not active: {mode}")
        con.execute("""
            CREATE TABLE IF NOT EXISTS completed_jobs (
                job_key TEXT PRIMARY KEY,
                result TEXT NOT NULL,
                completed_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
            )
        """)

Run initialize() in a controlled setup step before starting the workers. A mode switch is not a request to make during every hot transaction. The example uses autocommit mode so transaction boundaries are explicit later; it does not imply that autocommit alone prevents lock contention.

Set busy_timeout on each connection. Its purpose is to wait for a conflicting lock to clear instead of failing instantly. Waiting is not free: a worker that waits for five seconds still uses time and capacity. A timeout is a stopgap, not permission to keep transactions open indefinitely.

Do not casually put a WAL database on a network file share. SQLite's documentation says WAL depends on same-host shared-memory mechanisms, so it is not a solution for multiple hosts sharing one database file.

Make the write atomic, short, and safe to retry

If a job completion message might be delivered twice, a unique key is part of correctness. A queue acknowledgement, HTTP response, or process exit can be lost after SQLite commits. Retrying blindly should not create duplicate completed-job rows.

This example uses a bounded attempt count, an explicit immediate transaction, and a uniqueness constraint. Keep any remote API request outside save_completed.

import random
import sqlite3
import time

def save_completed(job_key: str, result: str) -> bool:
    for attempt in range(4):
        con = connect()
        try:
            con.execute("BEGIN IMMEDIATE")
            cursor = con.execute(
                """INSERT INTO completed_jobs (job_key, result)
                   VALUES (?, ?)
                   ON CONFLICT(job_key) DO NOTHING""",
                (job_key, result),
            )
            con.execute("COMMIT")
            return cursor.rowcount == 1
        except sqlite3.OperationalError as exc:
            if con.in_transaction:
                con.execute("ROLLBACK")
            if "locked" not in str(exc).lower() or attempt == 3:
                raise
            time.sleep(0.05 * (2 ** attempt) + random.uniform(0, 0.03))
        except Exception:
            if con.in_transaction:
                con.execute("ROLLBACK")
            raise
        finally:
            con.close()
    raise AssertionError("unreachable")

The return value distinguishes a newly recorded job from one already recorded. The key must describe the logical job, not a randomly generated identifier for each attempt. Otherwise every retry defeats the unique constraint.

BEGIN IMMEDIATE asks for a write transaction at a predictable point. It can fail or wait when another writer owns the lock; that is expected. The retry policy belongs at this narrow boundary. Do not wrap a non-idempotent payment, email send, or publish operation in the same retry loop just because the database write is retryable.

The sleep is small and randomized to reduce synchronized retries. Four attempts are an example, not a recommended universal setting. A busy system may need fewer retries and a real backpressure mechanism; a batch importer can tolerate longer waits.

A retry is not a cure for slow readers and checkpoints

WAL introduces an extra responsibility: checkpointing. A checkpoint transfers committed frames from the WAL back into the main database. SQLite normally triggers automatic checkpoints after a default threshold of about 1,000 pages, but a long-lived read transaction can prevent a checkpoint from progressing past that reader's snapshot.

If your application streams a query result for minutes, keeps a cursor alive across an external request, or runs a long analytics transaction, the WAL may grow. That can affect read performance and disk usage. The documentation describes the reader end mark and the resulting checkpoint limitations in detail.

A simple operational inspection uses the built-in checkpoint PRAGMA:

def wal_status():
    with connect() as con:
        busy, log_pages, checkpointed_pages = con.execute(
            "PRAGMA wal_checkpoint(PASSIVE)"
        ).fetchone()
    return {
        "busy": busy,
        "log_pages": log_pages,
        "checkpointed_pages": checkpointed_pages,
        "behind": log_pages - checkpointed_pages,
    }

print(wal_status())

Treat these values as observations, not a magic health score. PASSIVE attempts checkpoint work without waiting aggressively for competing operations; the returned counters help you see whether frames remain. A separate monitoring process, measured during normal load, can reveal whether the backlog repeatedly grows.

Do not reflexively run TRUNCATE on every request or delete -wal and -shm files while the database is in use. Those files are part of SQLite's live coordination and recovery behavior. Consider more aggressive checkpoints only with a measured operational reason and a controlled maintenance window.

Use separate connections and measure the right latency

SQLite connections have thread-affinity behavior in Python by default. Sharing one connection among unrelated threads without a carefully designed ownership model adds another layer of confusion. A straightforward pattern is one connection per independent operation or one clearly owned connection per worker, closed at a known boundary.

Measure the duration of acquiring a transaction, the time between BEGIN and COMMIT, the count of SQLITE_BUSY outcomes, and the size of the WAL backlog. Total request latency alone cannot tell you whether the problem came from a database lock, network call, or slow business logic.

Use synthetic contention tests before deciding to increase limits in production. Launch several writers that attempt small independent inserts and a reader that scans the table. Then deliberately add a slow transaction and observe how lock waits change. Compare transaction length before and after moving the slow work outside the lock.

If writes are still serialized too often, try a single writer queue inside one process, batching independent inserts into short transactions, or moving to a server database when multi-host writes and higher concurrency are genuine requirements. The best choice depends on throughput, failure tolerance, and operational cost rather than database fashion.

A small diagnostic checklist

Before calling WAL broken, check whether your application has a long-running write transaction, whether every connection actually uses the expected journal mode and busy timeout, and whether any reader holds a snapshot while an external operation runs.

Then look at duplicate delivery behavior. If a worker crashes after commit but before acknowledging a message, will reprocessing preserve the business invariant? A primary key plus ON CONFLICT DO NOTHING handles the completed-job example, but a richer workflow may require explicit states and reconciliation.

Finally, examine the deployment topology. WAL belongs to a database file that its participating processes can coordinate on the same machine. If workloads must share mutable state across machines, adding more Python retry logic will not remove that architecture mismatch.

What to take into the next incident

The sentence "we already enabled WAL" is not a diagnosis. WAL makes a specific kind of read/write concurrency better; it does not supply multiple simultaneous writers, infinite transaction capacity, or automatic exactly-once processing.

The durable fix is usually a combination of short writes, stable job identities, bounded lock handling, intentional connection ownership, and measured checkpoint behavior. Those choices are less exciting than switching databases, but they are often enough to make a small service predictable again.

Sources: SQLite WAL mode, SQLite PRAGMA reference, and auto-checkpoint behavior.

Originally published on Dispatch.

A Retry Is Not a Recovery Plan: Designing Publish Jobs for Ambiguous Success

Chen Yuan — Fri, 09 Oct 2026 15:54:56 +0000

A publishing automation can fail after doing exactly what it was asked to do. The browser clicks Publish, the site accepts the article, and the connection drops before the response arrives. The worker sees a timeout. The publication sees a new article. A scheduler that treats every timeout as permission to retry creates a duplicate. A scheduler that treats the timeout as a permanent failure may leave a perfectly good article unrecorded.

This is not peculiar to writing platforms. Payment requests, webhook consumers, email deliveries, and infrastructure changes all encounter the same ambiguity: the caller knows that it sent an operation, but cannot prove whether the remote system committed it. For browser publishing, the ambiguity is especially visible because there may be several screens between the editor and the public URL.

The useful engineering goal is not to pretend that network delivery can be made infallible. It is to design a small recovery protocol that makes uncertainty explicit. The protocol should survive a crashed process, a restarted scheduler, and a different operator taking over the next morning. A single retry counter cannot represent these facts.

The dangerous interval is after submission, not before it

Consider a job that prepares a long Markdown article, opens the editor, applies tags and a cover image, and presses Publish. Before the button press, most failures are reversible. If a title did not fill correctly, the worker can inspect the editor, correct the field, and try the local interaction again. Once the press may have reached the server, the risk changes.

A common implementation wraps the entire job in a broad exception handler. If any stage raises an error, the outer queue retries the whole task. This is convenient for an image-download failure and unsafe for an uncertain publish response. The same retry policy should not govern both operations.

The exact point of uncertainty is not merely a successful HTTP status. For browser tasks it begins when a client dispatches an action with possible permanent effects. The page may navigate without returning a clean automation response. The platform might save a draft, schedule publication, or publish immediately. These outcomes need different follow-up checks.

Treat the action boundary as a state transition. Before calling the browser, write durable evidence that the attempt is about to happen. If the worker later disappears, another worker can distinguish an untouched task from one whose commit result is unknown. The transition should happen before the action, even though that means a crash may mark an attempt that never actually reached the platform. Conservatively verifying a nonexistent publication is much safer than blindly publishing twice.

Give each job a stable identity that survives a restart

A queue job ID is often an implementation detail. It may change when the message is moved, replayed, or re-created by a new scheduler. A publishing identity should describe the intended content and target: publication account, approved article revision, and the planned slot. A deterministic key can help you identify the same logical request even when the worker process changes.

The revision part matters. A correction to an article should not silently be treated as an identical request. Conversely, starting the same content in a new worker should not manufacture a fresh publishing identity. The system must choose that policy explicitly rather than relying on timestamps generated inside a retry loop.

Here is a minimal example. Its hash is for local matching; it is not a platform-provided idempotency guarantee.

import hashlib
import json

def publish_identity(site, account, slot, title, body):
    revision = hashlib.sha256(body.encode("utf-8")).hexdigest()
    payload = {
        "site": site,
        "account": account,
        "slot": slot,
        "title": title.strip(),
        "revision": revision,
    }
    encoded = json.dumps(payload, sort_keys=True).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()

Keep the identity and a content hash alongside the approved source. Later, if a published item has the same title but a different body, that is not proof that the intended job succeeded. It could be an old article, another revision, or a name collision. Identity should make verification stricter, not provide an excuse to accept a weak match.

Store the attempt before touching the irreversible button

A durable run record can be remarkably small. It needs a job identity, source revision, target account, current state, attempt timestamp, candidate draft URL, any known public URL, and a brief history of observed transitions. The record should not contain passwords, live session cookies, or a copy of browser authentication material.

One practical state machine separates local preparation from platform reconciliation:

PREPARED
  -> EDITOR_VERIFIED
  -> COMMIT_POSSIBLE
  -> PUBLISHED_VERIFIED

COMMIT_POSSIBLE
  -> RECONCILE_ONLY
  -> PUBLISHED_VERIFIED

PREPARED or EDITOR_VERIFIED
  -> REPAIR_REQUIRED

The important state is COMMIT_POSSIBLE. It means the system cannot promise the action was never delivered. The next invocation is not allowed to submit again merely because the worker lost its response. Instead, it must inspect the platform through read-only paths.

Write state changes atomically where practical. On a local filesystem, a process can write a complete JSON record to a temporary file and replace the prior record. On a database, use a transaction and a uniqueness constraint around the logical job identity. In a distributed queue, acquire a lease or use compare-and-swap so that two workers cannot both cross the action boundary. Each method has different crash and recovery behavior, but all are stronger than keeping the attempt only in process memory.

Do not label an attempt as successfully published when the button was clicked. A click confirms an action was issued, not that the public page now exists. That distinction is the core of the protocol.

Reconciliation is a read operation with a precise success contract

After an ambiguous response, visit the platform's own published list or the expected public article URL in the same authorized account context. First look for a durable identifier: the site's post ID, canonical URL, or publication record. Then verify enough content to ensure that this is the intended revision.

A useful success contract checks the author and publication, exact normalized title, a distinctive passage near the end, and the expected publication status. For technical posts it can also inspect the number of code blocks and the cover image. A slug alone is not sufficient when platforms permit edits, collisions, or redirects. The public page must actually be visible; a draft editor URL is not proof of publication.

The contract can be expressed as data rather than a vague boolean. An illustrative verifier might return:

{
  "status": "published_verified",
  "account": "dispatch-example",
  "public_url": "https://example.dev/reliable-publish-recovery",
  "checks": {
    "author": true,
    "title": true,
    "tail_passage": true,
    "code_blocks": true,
    "canonical": true
  },
  "verified_at": "2026-10-09T15:00:00Z"
}

These are example values, not evidence of a real publication. In production, every true value should come from a concrete read-back of the page or platform API. Record the raw URL, source hash, and observation time so a later maintainer can assess what was actually checked.

If the site is slow to hydrate, the verifier may need a bounded polling window. A single empty render immediately after navigation is weak failure evidence. However, bounded polling must not become an infinite wait. If the window expires without reliable proof, preserve RECONCILE_ONLY and report the uncertainty accurately.

An idempotency key is useful only when the receiver enforces it

An API can offer a documented idempotency key and store the result of the first accepted request. That is valuable when the server guarantees that a repeated key resolves to the same operation. A browser editor typically does not expose such a contract. Putting a key in the local job file does not make a second click harmless.

Even real API idempotency contracts have details: key retention periods, request-body consistency, account scope, and behavior during concurrent requests. Read the specific API documentation rather than assuming that a string header solves everything. For sites without a supported key, the recovery plan depends on remote observation and a strict at-most-one automatic commit policy.

At-most-one is deliberately different from exactly-once. There are unavoidable cases where a process records COMMIT_POSSIBLE and crashes before it sends the click. The cautious system can then fail to publish, pending verification or manual adjudication. This is a trade-off. For a public article, a missing item can be diagnosed and published later; duplicated posts may produce confusing subscriptions, bad search indexing, or separate comment histories.

Choose the cost of each failure consciously. If the platform provides an official create endpoint with verified idempotency semantics, prefer that contract. If it does not, never describe browser clicks as exactly-once delivery.

Separate editor repair from remote-outcome repair

A resilient runner should decide its next action from the saved state and the latest platform evidence, not solely from the exception type. For example, an editor element becoming hidden before the commit boundary can be fixed by refreshing a snapshot and using a known selector. An exception after the commit boundary calls for verification, not another click.

The decision rule can be written as a simple function. It intentionally avoids calling a network or browser tool itself:

def next_step(state, remote_evidence):
    if remote_evidence.get("verified_publication"):
        return "finalize_receipt"
    if state in {"COMMIT_POSSIBLE", "RECONCILE_ONLY"}:
        return "read_only_reconcile"
    if state == "EDITOR_VERIFIED":
        return "commit_once"
    if state == "PREPARED":
        return "finish_editor"
    return "inspect_and_record"

This is a planning function, not an entire publishing system. It assumes that some trusted component already gathered remote evidence and that the caller enforces the single-commit boundary. For a real implementation, make the transition itself durable and reject invalid transitions rather than trusting a caller-provided state string.

When an external challenge interrupts the public-page check, preserve the exact URL and current attempt state. A normal login or simple challenge may be completed within the existing authorized browser session. It should not trigger creation of a new browser profile, a second worker, or a fresh article. If evidence remains unavailable, stop at a truthful uncertainty state.

Make the receipt useful to the next operator

The final receipt should answer operational questions without requiring a replay of the whole session: which account was targeted, which source revision was approved, which action was attempted, whether publication was independently verified, what public URL was observed, and what work remains. Include a stable run ID and elapsed duration. Separate a technical script result from a business outcome.

A publish script that exits with status zero after filling the editor has not necessarily completed the business task. Likewise, a script that times out immediately after publication may have produced a valid public article. A reliable report can say the script had an error while the business outcome was independently verified. Collapsing both facts into one success flag loses information.

Here is a compact receipt shape:

{
  "run_id": "article-20261009-01",
  "content_revision": "sha256-of-approved-source",
  "platform_state": "PUBLISHED_VERIFIED",
  "attempt_count": 1,
  "public_url": "https://example.dev/reliable-publish-recovery",
  "verification": "public_page_and_content_checked",
  "next_action": "record_metrics_on_next_review"
}

For a blocked run, keep the same keys and use a specific failure stage. Do not invent zero views or zero interactions when those metrics have not been collected. The honest value is unmeasured. Archive the evidence needed to resume; leave unrelated browser tabs and user files alone.

The deployment checklist is shorter than the incident report

The first improvement is to identify every action that may create a permanent remote object. Surround each one with a persisted attempt marker and a platform-specific read-back. The second improvement is to write the verification contract before production: what exact URL, title, identity, and tail passage constitute proof? The third is to constrain retry logic so that uncertainty cannot create a second post.

Next, test three deliberately unpleasant cases in an authorized test environment: crash before the commit call, crash after dispatch but before receiving its result, and successful publication followed by a failed verifier. Each case should produce a distinct state and a deterministic next step. Re-run the worker from a new process to prove that local memory is not the source of truth.

Finally, measure outcomes across actual runs. Useful metrics include ambiguous commit events, publications verified after a tool error, duplicate incidents, and time spent in reconciliation. An automation system that reports only green executions can hide its most expensive defect.

A reliable publishing worker is not one that always finds a way to click Publish. It is one that knows when to click, when to look, and when the evidence is too weak to do either again. The retry queue still has a place, but recovery begins with a durable record and a fresh observation of what the platform actually did.

Originally published on Dispatch.

OpenSSH 10.6 Ships a Compression Side-Channel Fix as a Default: Why It Matters

Chen Yuan — Wed, 07 Oct 2026 07:12:12 +0000

OpenSSH 10.6 arrived on 2026-10-06 with eight security fixes, two potentially incompatible configuration changes, and a stated shift in release cadence driven by AI-assisted vulnerability reports. The single change with the widest blast radius is not a new feature. It is the decision to disable the LZ77 dictionary coder in the SSH protocol's compression layer by default, mitigating a chosen-plaintext side channel described in the preprint "Crossing the Streams: SSH Plaintext Recovery via a Common Compression Context in Multiplexed Channels" by Fabian Baumer and Marcus Brinkmann. This is the kind of change that looks small in a changelog and large in a production fleet, because it quietly alters the cost model of a configuration knob that operators have tuned for decades.

The Compression Side-Channel That Was Already Documented

The SSH protocol allows compression to be negotiated at the transport layer. When a client and server both enable Compression, the negotiated cipher protects a compressed byte stream rather than the raw packet payload. For years the OpenSSH documentation itself recommended against enabling compression on connections that mix trusted and untrusted traffic, because the compression ratio can leak information about the plaintext. That warning existed precisely because the class of attack was understood, even if a concrete exploit had not been shipped against every variant of the SSH multiplexing design.

The 10.6 release notes name the specific mechanism: attacker-controlled input on one channel can recognizably reflect into the total length of transmitted ciphertexts, because LZ77 replaces repeated strings with back-references into an encoder search buffer that is shared across all channels of a single SSH session. When a victim channel carries a secret and an attacker-controlled channel runs in the same session, the attacker can craft inputs whose compressed length changes depending on the secret's content. The observable quantity is the ciphertext length, which the attacker can measure by observing the network.

How LZ77 Leaks Secrets Through Ciphertext Length

LZ77 is a sliding-window dictionary coder. It replaces a repeated byte sequence with a back-reference, which is a pair encoding a distance and a length. The compressor maintains a search buffer that holds recently seen bytes. When the encoder encounters a substring that also appears in the search buffer, it emits a back-reference instead of the literal bytes. A back-reference is typically shorter than the literal run it replaces, which is how compression reduces the output size.

The leakage appears when the search buffer is shared across channels that have different trust levels. Consider a simplified model of the encoder state:

search buffer (shared):  [..........secret-prefix..........]
attacker input:          A B C D E F G
                         ^^^^^^^^^^^^^
if "A B C D" matches a run inside the secret prefix,
the encoder emits a back-reference and the output shrinks.
otherwise the encoder emits literals and the output grows.

The attacker does not read the secret directly. The attacker observes the length of the ciphertext produced for a given input, and infers whether the input matched something already present in the shared buffer. By submitting many crafted inputs and recording which ones shortened the output, the attacker narrows down the secret's content. This is the same family of attack that broke TLS under the CRIME and BREACH disclosures, where compression of attacker-controlled and secret-controlled data into a shared context produced a length oracle.

Why Disabling the Dictionary Coder Is the Default Now

The 10.6 fix targets the LZ77 dictionary coder specifically. OpenSSH's compression has historically used zlib, which is an LZ77-family compressor with a Huffman coding stage layered on top. Disabling the dictionary coder removes the back-reference substitution step while leaving the entropy coding stage functional. The practical effect is that compression still runs, but it no longer produces back-references that could make the compressed length depend on cross-channel buffer contents.

# ssh_config: the knob whose behavior changed under the hood
# this still enables compression, but the LZ77 dictionary coder
# is now disabled by the 10.6 transport change
Host production-ssh
    Compression yes
    # the documentation recommends application-level
    # compression for mixed-trust sessions instead

The release notes are explicit about the trade-off. Disabling the dictionary coder makes Compression less effective at reducing payload size. The same payload that previously compressed to a third of its original size may now compress to half or more of the original, depending on the data. The OpenSSH team recommends application-level compression layered on top of the SSH transport where compression is genuinely needed, because application-level compression operates on a single trust context and is immune to the cross-channel length oracle.

The GSSAPI Credential Persistence Problem

A second security fix in 10.6 closes a credential leak in the GSSAPI authentication path. Before this release, when a client attempted GSSAPIAuthentication and the attempt failed, the credentials established during that failed attempt could persist into the session state. If a later authentication method succeeded against the same connection, those lingering GSSAPI credentials could be made available to the authenticated session in a way that the operator did not intend.

The fix has two parts. First, sshd now stores GSSAPI credentials only when authentication has succeeded. Second, sshd resets the GSSAPI authentication state before each new authentication attempt begins, so that residual state from one attempt cannot be confused with the state of a later attempt. The first part stops the leak; the second part stops the confusion that made the leak possible.

# sshd_config: the GSSAPI knobs affected by the fix
# the behavior change is server-side; the client knobs still apply
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
# 10.6 ensures failed GSSAPI attempts do not leave
# credentials available to a later successful auth

This is a privilege-boundary fix rather than a cryptographic one. Its severity depends on whether an operator had GSSAPIAuthentication enabled alongside other methods, and whether the subsequent successful authentication was expected to inherit the GSSAPI context. In configurations where GSSAPI was the only enabled method, the practical exposure is narrower, because a failed GSSAPI attempt that does not fall back to another method simply rejects the connection.

Username Injection Through ProxyCommand and Match Exec

A third fix restricts the characters allowed in destination usernames entered on the command line. OpenSSH 10.6 refuses usernames that contain backslash or dollar-sign characters when those usernames are provided as command-line arguments. Usernames specified through the User directive in a configuration file are not subject to the new restriction.

The motivation is injection. OpenSSH exposes several configuration directives that interpolate values into shell contexts, including ProxyCommand, Match exec, and related directives that run external programs. When a username is supplied from an untrusted source and the configuration uses that username inside a shell-evaluated directive, a username containing shell metacharacters can produce a command the operator did not intend.

# the attack surface: a username from an untrusted source
# fed into a directive that runs an external program
# a username containing shell metacharacters such as
# backtick, dollar-sign, or semicolon, when interpolated
# into ProxyCommand or Match exec, runs an external program
# the operator did not authorize
ssh 'user<metacharacters>@host'
# 10.6: the command-line username is rejected before
# it reaches ProxyCommand or Match exec evaluation

The release notes are careful to say that this mitigation cannot be absolute, because the variety of shells and user configurations in use means a different metacharacter set could still produce injection in some environment. The recommendation remains to avoid exposing the ssh command line to untrusted input at all. The fix closes a known concrete vector without claiming to close the entire class.

authorized_keys Restrict and the Match Override Bug

A fourth fix corrects the handling of the restrict keyword in authorized_keys entries. The restrict keyword in an authorized_keys line applies a set of restrictions to that key, including disabling port forwarding, agent forwarding, X11 forwarding, and PTY allocation. Before 10.6, the restrict keyword was not being properly applied to tunnel forwarding, which is controlled by the PermitTunnel directive and is disabled by default.

The interaction that produced the bug is a Match block override. Some options, including AuthorizedPrincipalsFile, were documented as accepting the string none as a way to disable them. When an option set to none was overridden by a Match block, the override code path was interpreting the literal string none as a filename rather than as the documented sentinel value. The fix ensures that the none sentinel is honored under Match overrides.

# authorized_keys: the restrict keyword now fully applies
# including PermitTunnel, which was previously missed
restrict,port-forwarding="no",pty="no" ssh-ed25519 AAAA... user@host
# 10.6: PermitTunnel is now covered by restrict, closing
# the gap left by the 10.5 fix for the same keyword

This is a continuation of a fix shipped in 10.5. The earlier release addressed restrict handling for most forwarding types but missed tunnel forwarding. The 10.6 release closes that remaining gap, which means the restrict keyword now applies uniformly across the forwarding directives it is documented to control.

AI-Assisted Security Reports and the New Release Cadence

The release notes include an unusual statement about the source of recent security reports. The OpenSSH team writes that they have received a large number of security bug reports, many of which are findings produced by AI models or produced with AI assistance. The team distinguishes between reports that have security impact under a realistic threat model and reports that do not, and notes that AI-assisted reports are most valuable when combined with human triage, analysis, test cases, and proposed fixes.

The statement that drives the cadence change is the observation that some bugs identified by AI tools were subsequently discovered independently by a different researcher. The team infers from this that adversaries who do not report bugs are likely able to discover the same bugs. On that basis, OpenSSH will for now ship more frequent releases to put fixes into users' hands more quickly, rather than batching fixes until a planned release.

This is a release-process decision with a measurable operational consequence. Operators who pin to specific OpenSSH versions and update on a fixed cadence will need to account for a faster stream of security-focused releases. Operators who track the latest release already will see more frequent updates. The decision does not change the codebase's stability guarantees, but it does change the patch-management rhythm that operators have built around OpenSSH.

What These Changes Mean for Existing Configurations

The two potentially incompatible changes in 10.6 are both consequences of the security work. Compression is less effective because the LZ77 dictionary coder is disabled, which means bandwidth-sensitive workloads that relied on transport compression will see higher byte counts for the same payload. Destination usernames entered on the command line are now more strictly validated, which means any automation that constructs ssh command lines from untrusted input will need to handle the new rejection of backslash and dollar-sign characters.

Operators who already followed the documentation's recommendation to avoid compression on mixed-trust sessions will see no behavioral change, because the vulnerable configuration was already discouraged. Operators who expose the ssh command line to untrusted input remain exposed to the broader class of injection, and the 10.6 username validation is one closed vector within that class rather than a full defense. The GSSAPI credential fix and the authorized_keys restrict fix are server-side corrections that tighten privilege boundaries without requiring configuration changes from operators who were using those features as documented.

The release cadence shift is the change that outlasts the individual fixes. A faster stream of security releases means the interval between a vulnerability being reported and a fix being available shrinks, which is good for defenders. It also means the interval between releases shrinks, which raises the operational cost of pinning and validation. The OpenSSH team has stated the cadence change is for now, which leaves open the possibility that the cadence returns to the previous rhythm once the current surge of AI-assisted reports subsides. Until then, operators tracking OpenSSH should expect security releases to arrive on their own schedule rather than on the calendar of a planned major release. The fixes in 10.6 are the first batch shipped under that new discipline.

Originally published on Dispatch.

Linux on ARM Laptops Is Not a Kernel Problem Anymore

Chen Yuan — Tue, 06 Oct 2026 14:27:22 +0000

Qualcomm announced an early developer preview of Linux on Snapdragon X2 Series in September 2026, and the phrasing of that announcement tells you where the hard part lives. The post describes a validated reference environment of a Debian 13 user space on a custom kernel, and it lists what is already in the mainline tree: boot and system bring-up through UEFI plus systemd-boot, core I/O for USB, PCIe and the QUP peripherals, graphics work in Mesa through Freedreno, Turnip and Rusticl, and fastRPC driver upstreaming that opens up the Hexagon NPU.

Read that list again and notice what it is. Every one of those items is a hardware enablement problem, and every one of them has a technical answer that somebody can write and review. Drivers land. Device trees get reviewed. Kernel configuration options get set. The project has a public mailing list, a review process, and a definition of done that everyone involved already agrees on. If that were the whole story, then bringing a new ARM laptop into a distribution would be a matter of waiting for the next kernel release.

It is not the whole story, and the people who actually ship images on these machines have said so in print. The AArch64 Laptops community guide for distribution integrators opens with a blunt question about why this is so much harder than it looks, given the seemingly good kernel support, and then answers it. Some of the difficulty is technical, some of it is organizational, and a meaningful part of it is licensing. Recognizing which of those three you are looking at is what separates a bring-up that takes an afternoon from one that consumes a month.

The Kernel Side Is Now Boring, Which Is the Good News

Start with the part that works, because it sets the baseline for everything else. The AArch64 Laptops device status page has been tracking per-laptop feature checklists for years. A Snapdragon 850 laptop from 2018 has every box ticked: boot into the bootloader, boot the kernel from the rootfs partition, text console, SD card storage, USB, a framebuffer desktop, keyboard, touchpad, touchscreen, on-board UFS storage, WiFi, Bluetooth, LTE, accelerated graphics, and audio.

That list is unglamorous and it is exactly the point. The kernel team behind these machines has spent the intervening years pushing patches for each subsystem, and the pattern repeats across generations. Qualcomm posted the first Snapdragon X Elite patchset one day after announcing the platform in October 2023. By the Linux 6.8 and 6.9 cycle, the merged list already included pinctrl, interconnect, clocks, power domains, the SMMU, the QUP serial engine block, system cache, the PMIC sound machine driver, USB DWC3, reference board support, the ADSP and CDSP coprocessors, multimedia clocks, PCIe and eDP and USB PHYs, and NVMe over PCIe. The next cycle added USB host, on-board display, GPU, memory DVCS, CPU frequency, audio capture, external DisplayPort, suspend and resume, camera, and video decode.

For the X2 generation the staged plan continues in the same shape. The initial enablement stage covers boot through systemd-boot, low and high speed I/O including UART, I2C and SPI on the QUP block, the Mesa graphics work, fastRPC for local inference, and system-wide power and thermal management. The expanded enablement stage targets broader peripheral coverage and better reliability, described as the point where building and testing real workloads becomes practical rather than experimental.

Qualcomm's own framing of why they invest here is worth quoting in substance: out-of-tree patches may help early bring-up, but mainline support is what distribution maintainers and long-term developers can build on across kernel releases. That sentence is the thesis of this article expressed in one line. A vendor can enable their own reference device with a private tree and get a working demo. What they cannot do that way is make the platform something a distribution is willing to carry.

Device Trees Move the Problem Instead of Solving It

The first genuinely hard thing is the hardware description. On x86 the firmware describes the machine through ACPI, and the kernel walks the firmware's own description of power rails, controllers and buses. On most ARM platforms the kernel instead consumes a device tree, a much lower level document that does not contain arbitrary logic and therefore requires Linux to know far more detail about the board.

The consequence is spelled out in the distribution integrator guide. When an ACPI system powers up a USB controller, the firmware handles the sequencing and the kernel does not need to model every rail involved. On a device tree system the kernel models those rails and drives them itself. Components that x86 firmware programs invisibly, like the USB signal repeaters on a laptop mainboard, have to be described in the device tree and have a driver written for them, wired up correctly to receive events. If you enable the repeater driver in the kernel but leave the module out of the initramfs, you get a machine that boots from its internal disk and cannot boot from a USB drive, which is a memorable way to discover your mistake during an encrypted-disk install.

The guide's second structural point is that device trees and kernels do not separate cleanly the way ACPI and kernels do. Device trees are meant to be forward and backward compatible with kernel versions, and in practice they are not always, because drivers and the tree fragments that describe their hardware frequently land in the same patch series. If the tree describing your panel predates DisplayPort alt-mode support, a newer kernel will not conjure that feature, because the description itself is the input.

There is a third point in that guide which is the most uncomfortable one for anyone who has debugged peripheral failures. The ACPI that Qualcomm laptops ship with is not supported by Linux and probably will not be for the current generation, and the vendors do not ship a device tree in firmware as an alternative. So a distribution has to install device tree files into the EFI system partition and pick the right one at boot based on which laptop is actually running.

That selection problem has no clean answer. The guide describes two working approaches. In systemd version 257 and later, a unified kernel image can embed multiple device trees and select one by matching EFI hardware IDs, with the practical benefit that the device tree lives inside the signed PE binary and therefore survives secure boot validation. The alternative is a separate EFI driver that loads device trees from a well-known directory on the ESP using its own internal hardware ID database, the approach used by the generic ARM64 EFI target in postmarketOS. Both work. Neither is a standard, and a distribution has to pick one and maintain the device list.

Firmware That Cannot Be Redistributed

Here is the part that stops being an engineering problem. Many of the useful properties of these systems come from coprocessors, and those coprocessors run firmware. Some of that firmware is packaged in linux-firmware, but for most devices there is no clear license and packaging is not possible. The firmware therefore has to be pulled off the Windows partition, which means the only supported way for a user to get firmware updates is to keep dual booting, periodically boot back into Windows for the update, and copy the new firmware across.

The distribution guide states this plainly and admits that solving the licensing problem is outside its scope. The practical advice it gives is to build a manual or semi-manual process that users can follow to extract the firmware during or after installation, because that is the best option available today.

Tooling exists for the extraction step, and it is interesting precisely because of how many special cases it encodes. One widely used script mounts a Windows system partition, plain NTFS or BitLocker, locates the Qualcomm DSP firmware files under the Windows driver store directory, and builds a Debian package that installs them into the firmware updates directory before rebuilding the initramfs. It chooses the firmware set by reading the device tree model string and matching it against a hardcoded table.

That table is the artifact worth reading. It contains exact device strings for laptops from Acer, ASUS, Dell, HP, Lenovo, Medion, Microsoft and Samsung, and the script aborts with an error if your model is not listed, with a comment instructing you to extend the case statement to add support. The firmware files themselves are a grab bag of DSP images and JSON descriptors with names that encode SoC variants. The limitations section is equally blunt about scope: only NVMe devices are scanned automatically, and the script assumes an arm64 Debian or Ubuntu target with initramfs-tools present.

So the "just install Linux" path on these machines includes a step where you extract proprietary binaries from your Windows install, wrap them in a distribution package whose device list a volunteer maintained by hand, and then rebuild your initramfs so the display and audio coprocessors can come up on the next boot. None of that is a kernel bug. All of it is real work that every distribution has to either duplicate or depend on somebody else to maintain.

import subprocess, json
from pathlib import Path

DT_MODEL = Path("/proc/device-tree/model")
FW_DB = Path("/usr/share/qcom-firmware-extract/devices.json")

def read_model() -> str:
    # device-tree model strings carry a trailing NUL byte
    return DT_MODEL.read_bytes().rstrip(b"\x00").decode("utf-8")

def firmware_dir_for(model: str, db: dict) -> str | None:
    # the shipped table matches on exact or prefix strings such as
    # "x1e80100/LENOVO/21N1" and "glymur/ASUSTeK/UX3407NA"
    for entry in db["devices"]:
        if model == entry["model"] or model.startswith(entry["model"]):
            return entry["firmware_dir"]
    return None

def main() -> int:
    model = read_model()
    db = json.loads(FW_DB.read_text(encoding="utf-8"))
    fw_dir = firmware_dir_for(model, db)
    if fw_dir is None:
        # upstream behaviour: abort instead of guessing a firmware set
        print(f"unsupported device: {model}", file=sys.stderr)
        return 2
    return subprocess.call(["/usr/libexec/qcom-firmware-extract", "-d", fw_dir])

if __name__ == "__main__":
    raise SystemExit(main())

Configuration Work That Never Ends

The middle layer between kernel and firmware is kernel configuration, and the distribution guide treats it as its own discipline rather than a footnote. The upstream mandate is that the arm64 defconfig should carry everything needed to boot at least the well-supported laptops like the ThinkPad T14s and the X13s, and distributions are advised to derive their kernel configuration from that defconfig so new device support trickles down without anyone making a deliberate decision.

That advice sounds straightforward until you try to work out what your laptop needs on top of it. The guide offers a technique that is refreshingly honest about its own limits. The kernel ships a script that maps device tree nodes to the drivers and kernel config options that could bind to them, so you can feed it a compiled config and get back a list of options that are switched off but probably should be on. The example output pairs a clock controller node with its compatible string, the driver file that matches it, and the config symbol that enables it, which tells you the symbol's current value is off and needs to change.

The guide follows that technique with a warning that the tool produces incorrect and spurious output for some generic nodes, that its output should be validated against a second source of truth, and that options mentioning other vendors are probably false matches. It recommends falling back to searching the device tree for your panel, finding its compatible strings, grepping the driver directory for those strings, and then opening the Makefile to read off the config symbol. When all of that fails there is the deferred devices debug directory, which lists hardware that did not probe, and an IRC channel where people will help translate an error message into a symbol name.

Two details from the porting checklist explain why these machines are fragile in ways that look like bugs but are not. First, the guide advises always using the newest available kernel, because Snapdragon laptops pick up fixes and features with every release rather than stabilizing once. Second, it advises passing two command line options that tell the kernel to keep clocks and power domains running even when it believes nothing is using them, on the grounds that the schematics are usually unavailable and some of these resources are modeled incorrectly. That second item is a striking admission: on some devices the working configuration is one where you deliberately over-provision power rails because the description you have is wrong and you cannot see the hardware to correct it.

# map device tree nodes to the driver symbols that should be enabled
scripts/dtc/dt_to_config \
  --config .config \
  --exclude-flag n \
  --include-flag y \
  --short-name \
  arch/arm64/boot/dts/qcom/x1e80100-lenovo-yoga-slim7x.dts

# find the config symbol for a panel by its compatible string
grep -rnwI drivers -e "samsung,atna45dc02"
# -> drivers/gpu/drm/panel/panel-samsung-atna33xc20.c
grep -n "atna33xc20" drivers/gpu/drm/panel/Makefile
# -> obj-<CONFIG_DRM_PANEL_SAMSUNG_ATNA33XC20> += panel-samsung-atna33xc20.o

The Initramfs Is Where Bring-Up Gets Honest

If you want a single artifact that captures what is unusual about these platforms, it is the initramfs. On a typical x86 machine the set of modules needed to reach the root filesystem is generic and small, and initramfs generators have gotten good at inferring it. On an ARM laptop the list is longer and stranger, because basic things like display output and keyboard input depend on drivers that a generic machine would never need at that stage.

The guide's recommendation is to sidestep the inference entirely during bring-up: put every kernel module into the initramfs, accept the size cost, and remove them later once the machine works. The reason given is that partially populated initramfs causes race conditions and probe failures that are not retried, which turns a configuration mistake into an intermittent hardware failure that is much harder to diagnose. Only after everything works does the guide suggest using a script to compute the list of modules that are actually loaded and hardware specific.

There is a second-order consequence that the guide mentions in passing and that deserves more attention than it gets. The bare necessities for full disk encryption include being able to see a prompt and type into it, which on these machines means the display and input drivers must be present before the encrypted root can be decrypted at boot. A distribution that gets this wrong ships an installer that works on a reference device and bricks the experience on a laptop with a panel driver it did not include. The hardware is fine. The packaging was incomplete in a way that only shows up on real hardware.

# assemble a bootable image the way the bring-up guides do it:
# kernel image, device tree blob, and initramfs concatenated
cat arch/arm64/boot/Image.gz \
    arch/arm64/boot/dts/qcom/x1e80100-lenovo-yoga-slim7x.dtb \
    > Image.gz+dtb

mkbootimg --kernel Image.gz+dtb \
  --cmdline "ignore_loglevel earlycon clk_ignore_unused pd_ignore_unused" \
  --ramdisk final-initramfs.cpio.gz \
  --base 0x80000000 --pagesize 4096 \
  --output boot.img

What the X2 Announcement Actually Promises

Go back to the launch framing and the pieces line up. Qualcomm's stated target audience for the developer preview is distribution maintainers, toolchain developers, kernel contributors and hardware enablement engineers, and the post explicitly says that a turnkey install-and-go experience comes later, when distributions adopt what lands upstream. The reference environment is Debian 13 with a custom kernel, the build path starts from upstream sources, and the resources listed are kernel repositories, a Debian image recipe, the arm-msm mailing list, and a developer forum.

That is a vendor doing the part it is uniquely positioned to do, which is making the hardware describable and the drivers available upstream, and then declining to do the part that belongs to distributions. Meanwhile the consumer commitments are separate and have their own timeline, with partners planning Linux support in their devices in the first part of 2027.

The practical lesson for anyone who maintains packages, installers, or internal images for these machines is to stop treating ARM laptop support as a kernel project that will eventually complete. The kernel side is largely a solved and steadily improving problem. The remaining work is a configuration surface that changes with every kernel release, a device tree selection problem with two competing non-standard answers, a firmware distribution problem that licensing has pushed onto end users and their Windows partitions, and an initramfs completeness requirement where a single missing module turns into a laptop that will not boot to its login screen.

None of that gets closed by a release announcement. It gets closed by distributions writing down how they handle device tree selection, packaging the extraction tooling, documenting which modules must be in the initramfs, and sharing the device specific details that make the next laptop cheaper to support than the last one. The guide says as much near the end, in a sentence that reads like a description of the entire field: while there is plenty of work happening in the kernel, a lot of effort is still needed in distributions and middleware to make these devices properly nice to install and use. That sentence is the roadmap, and it does not contain the word upstream anywhere in it.

Originally published on Dispatch.

Page Tables Are Not Free: The 1/512 Rule That Breaks Under Shared Mappings

Chen Yuan — Mon, 05 Oct 2026 10:15:29 +0000

A 4 KiB data page costs 8 bytes to map. That is a ratio of 1 to 512. When one address space maps a page once, the page table overhead is roughly one 512th of the memory it describes, which is why most developers never think about it. The ratio is not a law. It is an assumption, and the assumption is that each page is mapped exactly once. The moment many address spaces map the same physical page, the ratio stops holding and page tables can consume as much memory as the data they describe.

Linux kernel developers have hit this wall repeatedly over two decades. The pattern is consistent, the workloads are not exotic, and the numbers are large enough to take down production servers. This article walks through that history in release order, because each case sharpens the same point: page table memory is a real, measurable cost, and it grows with the number of address spaces, not just the amount of data.

The Arithmetic Behind the Ratio

Start with the units. A standard 4 KiB page needs one page table entry to be mapped. On a 64-bit x86 kernel that PTE is 8 bytes. So a single mapping costs 8 bytes of page table memory for 4 KiB of data, which is 1/512 of the mapped size.

If N address spaces map the same page, each one needs its own PTE for it. The kernel does not share PTEs across processes. Every task has a pointer to its own mm_struct, and each mm_struct owns its own page table tree. So the cost scales with N even though the data page is shared once.

Run the arithmetic for the boundary case. If 512 processes map the same page, they need 512 PTEs. At 8 bytes each that is 4 KiB, which is exactly the size of a data page. The page tables now cost as much as the data they map.

That is the whole story in one number:

data page size:    4096 bytes
PTE size:          8 bytes
mappings to parity:  4096 / 8  = 512

Converted to code, the rule is a one-line function of how many address spaces map the same page:

pte_bytes = 8          # a PTE on 64-bit x86
page_bytes = 4096      # a 4 KiB data page
map_count = 512

# cost of mapping one page in N address spaces, in pages
pte_pages = (map_count * pte_bytes) / page_bytes   # 1.0 when N = 512

When you see a server with hundreds of processes attached to a large shared region, the page table overhead is no longer a rounding error.

The Assumption Sits Inside the Kernel

The 1/512 figure is not an arbitrary target. It comes from the geometry of the page table itself. A multi-level page table maps a virtual address through several levels, and each level holds pointers to the next, so the total overhead is a small fraction of the addressable range. Linus Torvalds made exactly this point in an early kernel mailing-list discussion about hashed page tables, arguing that a tree structure keeps neighboring entries adjacent so a single cache-line fill covers several TLB entries at once. Memory size, in that framing, was a secondary concern.

His own master's thesis acknowledged the memory side. Virtual memory mappings must be memory-efficient so the mapping information does not consume physical memory that could otherwise serve file-system caching or user programs. The 1/512 ratio is the kernel's way of buying structure with a modest fixed tax. The tax is low when the working set is mapped once. It stops being low when the same working set is mapped many times.

Andrea Arcangeli, 2002: A 64 GiB Machine Out of Memory

The first real production case is instructive because the machine had plenty of free RAM. Users with 64 GiB x86 machines kept running out of memory even though a large amount of memory was idle. Their workloads ran hundreds of processes, each mapping the same 1 GiB of shared memory.

The data pages were shared, but every process needed its own page tables to map them. On that architecture the page tables lived in a region with a hard ceiling, so the overhead collided with a physical limit. The fix was to move page tables out of that constrained region. It was an edge case created by the combination of a 32-bit constraint and a feature that widened physical addressing beyond what the platform normally supports.

The lesson survives. The workload was not malicious. It was a normal shared-memory deployment, and the page tables quietly doubled the memory footprint of the shared region.

Khalid Aziz, 2022: An Oracle Server and 878 GB of Page Tables

Twenty years later the same shape reappeared at a much larger scale. An Oracle database server with 512 GB of RAM was hitting the out-of-memory killer. When 1500 or more clients attached to a 300 GB shared global area, each client mapped the shared region with its own page tables.

With 8-byte PTEs, two thousand processes mapping the same 4 KiB page need 16 KiB of PTEs for that single page. In the worst case, where every process maps the whole shared area, the PTEs alone would consume roughly 878 GB. The proposed fix was to let processes share page tables, but the mechanism that would share PTEs across processes has not been merged into mainline.

Again the cause was not a leak. The data pages were shared correctly. The cost was purely the per-address-space page table replication.

Qi Zheng, 2021: MADV_DONTNEED Leaves Empty Page Tables Behind

The third case does not involve shared mappings at all. A single process reported 590 GiB of resident set size and 110 GiB of page tables, when mapping that much memory should only need about 1.2 GiB of PTEs. That is roughly a hundred times the expected overhead.

The workload used allocators that return memory to the kernel with madvise(MADV_DONTNEED) instead of munmap(). The distinction matters. Unmapping memory eventually frees the page tables. Advising the kernel that a range is not needed frees the data pages and clears the PTEs, but keeps the page table structures allocated. Empty page tables piled up.

A patch to reclaim those page tables took several rewrites and was eventually merged years later. The case is a reminder that a page table is a resource you have to release deliberately, and that the API you call determines whether it is released.

Huge Pages Are the Lever Database Operators Pull

Page table memory is proportional to the number of mappings, so the direct fix is to map more per entry. That is exactly what huge pages do. A 2 MiB huge page still needs a single PTE, so the overhead per byte drops by a factor of 512.

Database operators reached for this lever when page tables showed up in production incidents. One reproduction with PostgreSQL was forced to kill backends on a machine with 192 GiB of RAM and a large shared buffer. Page tables grew from tens of MiB to over 25 GiB as backends touched more of the cache, memory got pressured, and the machine started swapping. Switching to huge pages kept page tables around 61 MiB for the same workload.

A recent measurement with ClickHouse made the growth explicit. On a machine with 128 GiB and a 32 GiB shared buffer, each backend scanning a cached table added 31 MiB of page tables. At 200 connections, page tables took more than 6 GiB. With huge pages, the same 200 connections cost about 111 MiB.

Huge pages are not free. Reserving them needs unfragmented memory, and under load the same request can routinely fail. Transparent huge pages can stall on allocation, and kernel developers have debated whether defragmentation should be the default for years. There is no parameter-free win here, only a trade.

NUMA Machines Reintroduce the Same Argument

The memory cost of page tables has a latency twin on multi-node machines. When a thread runs on a node that does not hold the page tables, a TLB miss has to walk page tables in remote memory.

The research consensus is that you pay one way or the other. One approach replicates the entire page table tree on every node, which costs page-table memory times the number of nodes, and every change has to update every copy. Another approach copies a PTE to a node only when one of its threads faults on that node, and each page table page keeps a list of the nodes that hold a copy. Work on remote page tables showed they can slow an application as much as remote data, and the same effect was reproduced on an 8-socket machine with 8 TB of RAM.

So the design space is a balance between two costs: keep one copy and pay the latency of reading remote tables on every miss, or keep a copy per node and pay the cost of keeping them in sync, plus the memory multiplication.

What to Check in Your Own Workload

The cases above share a single, checkable symptom. Ask whether many processes, threads, or containers map the same large region. If the answer is yes, page table memory is worth measuring rather than assuming.

Linux exposes the metric directly. A single process reports its page table size, and the kernel tells you whether transparent huge pages are available:

# page table memory for one process (kB)
grep VmPTE /proc/<pid>/status

# is transparent huge page support enabled, and to what degree?
cat /sys/kernel/mm/transparent_hugepage/enabled

The VmPTE line is the per-process number to watch. On a shared-memory workload it grows with the number of address spaces, not with the amount of data, which is exactly the failure mode described above.

There is no single answer here, only a lever. This article presents a handful of real production incidents and the trade-offs involved, rather than a universal fix, because the right choice depends on the workload and the hardware. What is universal is the measurement.

The ratio of 1 to 512 is a convenient assumption. Under shared mappings it is a floor, not a cap.

Originally published on Dispatch.

Hard Budget Caps Are a Runtime Safety Primitive for AI Agents

Chen Yuan — Sun, 04 Oct 2026 05:31:35 +0000

An AI agent can fail without crashing, throwing an exception, or producing an obviously wrong answer. It can simply keep doing valid work for far too long.

That is a different failure mode from the ones most software teams are used to. A retry loop can keep calling a paid API. A coding agent can keep launching test environments. A background worker can keep generating images, embeddings, logs, or storage objects. Every individual action may be permitted, yet the overall run becomes economically unsafe.

This is why hard budget caps are starting to look less like billing features and more like runtime safety controls.

Simon Willison made the case directly on October 3: usage-priced services need hard limits that stop work instead of merely sending an alert after a threshold is crossed. Google Cloud has moved in the same direction with Spend Caps, which can pause eligible services after a configured budget is exceeded. The important shift is architectural: cost is becoming something software can enforce, not merely something finance reviews later.

For agent systems, that distinction matters even more because agents can create new work on their own.

A Budget Alert Is Not a Control Boundary

Traditional cloud budgets are usually observational. They answer questions such as:

How much has this project spent?
Is spending above the expected trend?

Those are useful questions, but none of them stop execution.

An agent needs a control boundary. If a run is allowed to consume at most 500 units of a metered resource, then unit 501 should fail closed. The exact unit may be API credits, GPU minutes, database writes, browser actions, tokens, or a normalized internal cost score.

The basic distinction is simple:

soft budget:
    observe -> alert -> keep running

hard budget:
    observe -> compare -> deny next action

The second model belongs on the execution path itself.

Google Cloud's current Spend Caps preview makes this idea concrete. Its documentation says eligible service usage can be paused when the configured cap is enforced, while resources remain intact. Google also warns that enforcement is not instantaneous because estimated costs are involved, which is an important reminder: a provider-side cap is useful, but an application should still maintain its own tighter limits.

Agents Turn Small Mistakes Into Repeated Actions

A normal application usually has a fairly stable request shape. Users click buttons, jobs enter queues, and traffic rises or falls within recognizable patterns.

Agents can create a feedback loop.

A planner decides that a task failed. The retry policy creates another attempt. The new attempt spawns a browser session. The browser session calls an image service. The image service output fails validation. The planner decides to try again with a revised prompt. Nothing in that sequence is necessarily broken.

The problem is multiplication.

One bad decision can become twenty valid paid actions. A second recovery rule can turn those twenty into a hundred. If the only cost control is an email, the system can continue while nobody is watching.

This is why an agent budget should be treated like a memory limit or a request deadline. It is not an accounting preference. It is a condition for continued execution.

Put the Budget Check Before the Tool Call

The safest place for a budget gate is immediately before a side-effecting or metered action.

A simple implementation can keep the policy outside individual tools:

from dataclasses import dataclass

@dataclass
class Budget:
    limit: int
    used: int = 0

    def reserve(self, units: int) -> None:
        if units <= 0:
            raise ValueError("units must be positive")

        if self.used + units > self.limit:
            raise RuntimeError("hard budget exceeded")

        self.used += units


budget = Budget(limit=500)

def call_metered_tool(tool, estimated_units, **kwargs):
    budget.reserve(estimated_units)
    return tool(**kwargs)

This example is intentionally small. The important property is ordering: reserve first, execute second.

If the budget check happens after the tool call, it is only telemetry. If a retry path can bypass the wrapper, it is not a hard cap. If a child agent receives a fresh counter instead of sharing the parent budget, spawning becomes a way to escape the limit.

The budget object therefore belongs at the same architectural level as authentication, authorization, and cancellation.

Use More Than One Budget Dimension

Money is not the only resource that needs a ceiling.

A useful agent runtime can enforce several independent dimensions:

run_budget:
  max_tool_calls: 80
  max_browser_writes: 12
  max_external_api_units: 500
  max_wall_clock_seconds: 1200
  max_child_agents: 3
  max_retries_per_step: 2

These limits solve different problems.

A wall-clock limit catches work that stalls without spending much. A tool-call limit catches rapid loops. A browser-write limit protects external accounts from repeated side effects. A retry limit prevents one failing step from consuming the entire run. A child-agent limit prevents recursive delegation from turning one task into an uncontrolled tree.

The dimensions should be independent because a run can be safe in one dimension and dangerous in another.

For example, a cheap API can still generate thousands of unwanted external objects. A browser automation task can cost almost nothing in infrastructure while repeatedly posting, liking, or modifying data. Financial cost alone would miss the real failure.

Reservations Are Better Than Post-Hoc Accounting

Many metered operations do not reveal their final cost until they finish. That creates a race: several concurrent workers can all see remaining budget and begin expensive work at the same time.

Reservations fix that.

Before starting an operation, reserve the maximum or a conservative estimate. After it finishes, reconcile the reservation with actual usage.

class Ledger:
    def __init__(self, limit):
        self.limit = limit
        self.committed = 0

    def reserve(self, estimate):
        if self.committed + estimate > self.limit:
            return None

        self.committed += estimate
        return {"reserved": estimate}

    def settle(self, ticket, actual):
        reserved = ticket["reserved"]
        self.committed += actual - reserved

A production implementation needs atomic storage, but the model is what matters. Concurrency should not create free budget.

This is similar to inventory systems. Two buyers should not both be able to purchase the last item because they read the same stock count before either write completes. Two agents should not both spend the same remaining allowance because they checked it at the same time.

Provider Caps and Application Caps Solve Different Problems

Provider-side limits are valuable because they are outside the application. If the application has a bug, the provider can still stop further billable usage.

Application-side limits are valuable because they understand intent.

A cloud provider sees requests. Your runtime sees tasks, users, sessions, tools, retries, and side effects.

That suggests a layered design:

A provider cap limits worst-case account exposure.
A project cap limits one application.
A user cap limits one tenant or operator.
A run cap limits one agent execution.
A step cap limits retries and side effects inside one workflow.

Google Cloud's Spend Caps are an example of the first two layers for eligible services. The feature can pause usage after the configured threshold is reached, and Google documents alert points below the cap as well. But an agent runtime can act earlier because it knows whether a request is part of a retry storm or a legitimate new task.

The two controls should reinforce each other rather than compete.

Budget Exhaustion Must Be a Normal Terminal State

Many agent systems treat every interrupted task as an error that deserves another attempt. That is dangerous when the interruption is the budget system doing its job.

Budget exhaustion should be a first-class terminal state:

SUCCESS
FAILURE
BLOCKED
CANCELLED
BUDGET_EXHAUSTED

A scheduler receiving BUDGET_EXHAUSTED should not automatically retry the same plan with a fresh allowance. That would turn the budget into a delay instead of a boundary.

The terminal record should include enough evidence to explain what happened:

configured limit;
consumed amount;
last successful step;
action that was denied;
retry counts;
child runs;
external side effects already completed.

That makes recovery explicit. A human or higher-level policy can choose to grant more budget, reduce scope, or stop permanently.

Retries Need Their Own Economics

Retries are one of the easiest ways to accidentally defeat a cap.

Suppose an image generation step fails quality validation. The system may reasonably try a second prompt. But if each retry receives the original full run allowance, the cost model is fictional.

Retries should spend from the same parent budget and usually become stricter as attempts accumulate.

One useful policy is:

retry_limits = {
    "network_timeout": 2,
    "rate_limit": 1,
    "validation_failure": 1,
    "authentication_failure": 0,
    "permission_denied": 0,
}

The important part is not the exact numbers. It is that retry permission is based on failure class, and every retry still consumes the original run budget.

This also improves behavior quality. An agent that knows it has one remaining attempt is more likely to gather evidence before acting than one that can keep trying indefinitely.

A Hard Cap Should Fail Closed Without Destroying State

Stopping spend should not mean destroying work.

Google Cloud explicitly describes its Spend Caps as non-destructive: eligible usage can be paused while data and resources remain. Agent runtimes should follow the same principle.

When a hard cap is reached:

stop new metered actions;
preserve logs and receipts;
keep already-created drafts or artifacts;
do not roll back verified external side effects blindly;
record the exact continuation point;
require an explicit new allowance before resuming.

This produces a resumable system instead of a disposable one.

It also separates two concerns that are often mixed together: stopping new risk and cleaning up old state. The first should happen immediately. The second should be deliberate and evidence-based.

Budget Policy Belongs in the Runtime, Not in Prompts

A prompt can tell an agent to be frugal. That is not enforcement.

Prompts influence decisions. Runtime gates constrain actions.

The safest architecture assumes that the model may misunderstand a limit, forget it after context compression, delegate to another process, or choose a plan whose cost estimate is wrong. A hard budget should still hold.

That means the policy must live outside the model loop:

model proposes action
        |
        v
policy checks permission
        |
        v
budget reserves capacity
        |
        v
tool executes
        |
        v
ledger settles actual usage

The model can participate by estimating cost and selecting cheaper plans, but it should not be the final authority over whether the action is allowed.

Cost Is Becoming Part of Correctness

The broader lesson is that correctness for autonomous software has changed.

A task is not correct merely because it eventually reaches the requested output. It also has to respect limits while getting there: time, external writes, retries, permissions, and spend.

That is why the current push toward real spend caps matters. Simon Willison's argument for default hard limits is not only a billing recommendation. It points toward a runtime design principle for agentic systems: every autonomous loop needs a maximum amount of damage it is allowed to cause before the infrastructure stops it.

Google Cloud's July 2026 Spend Caps preview shows that cloud platforms are beginning to expose this control at the billing layer. Agent frameworks should not wait for every provider to do the same.

A reliable agent should know what it is allowed to spend, reserve that capacity before acting, share the same ledger across retries and child workers, and stop cleanly when the allowance is gone.

That is not pessimism about autonomous systems. It is the mechanism that makes unattended execution practical.

Sources:

Originally published on Dispatch.

Zig 0.17 Split Its Build Into Two Processes: Why That Matters

Chen Yuan — Sat, 03 Oct 2026 11:12:32 +0000

The most important change in Zig 0.17.0 is not a language feature. It is the build system being split into two separate executables: one that evaluates your build.zig script (the configurer), and one that executes the build graph (the maker). This restructuring solves a problem that has been quietly bothering build systems for years: every time you edit your build script, the entire build system had to be rebuilt from source.

The Problem: Build Scripts That Reprogram the Build System

Most build systems treat the build configuration file as a program. Zig is no exception: build.zig is Zig source code that runs at configure time to declare what steps exist, what dependencies connect them, and what options are available. The trouble is that this program runs inside the build system own process. When you change build.zig, the build system itself must be rebuilt.

This creates a feedback loop. The build system cannot cache its own configuration because the configuration logic is the thing being changed. Every invocation after a build.zig edit starts from scratch: recompile the build runner, re-evaluate the script, then execute the graph. For small projects this is a rounding error. For large projects with complex build scripts, it compounds.

The previous design also forced the build system to observe certain arguments at configure time. If your build.zig checked whether scdoc was installed to decide whether to build man pages, that probe poisoned the configuration cache. The cache could not be reused across invocations because the configuration depended on the state of the host system at the moment it ran.

The Solution: Separate the Configurer From the Maker

Zig 0.17.0 breaks the build system into two processes. The configurer process evaluates build.zig and produces a compact binary serialization of the build graph. The maker process consumes that serialization and executes the steps.

+-------------------+     binary      +-------------------+
|  Configurer       | --------------->|  Maker            |
|  (evaluates       |   serialization |  (executes the    |
|   build.zig)      |                 |   build graph)    |
+-------------------+                 +-------------------+

The maker executable is built once and remains unmodified when build.zig changes. It is built with optimizations enabled, which matters more now that --watch and --fuzz modes exist. The configurer can be skipped entirely for certain CLI flags, meaning some zig build invocations bypass configuration altogether and go straight to the maker.

This separation also makes configuration serializable in a format that third-party tooling can consume. That format is the foundation of the new Build Server Protocol.

Configure Cache Poisoning: Naming the Side Effects

The split between configurer and maker introduced a precise vocabulary for a concept that was previously implicit: configure cache poisoning. A poisoned cache means the configure logic had side effects or did something the cache system could not track.

The distinction is subtle but important. A Run step that prints output has side effects at make time, which is expected and does not poison the cache. But checking for the existence of scdoc at configure time to choose a default option does poison the cache, because the result depends on host state the cache cannot see.

Zig 0.17.0 provides explicit alternatives to poisoning:

// Instead of probing the host at configure time:
const scdoc_path = b.findProgram(&.{ "scdoc" }, &.{});

// Declare the dependency explicitly:
std.Build.dependOnFileContents(b, "config.txt");
std.Build.dependOnDirectoryContents(b, "docs/");

Four declaration functions cover the common cases: file contents, file metadata (size, inode, mtime), directory contents, and directory metadata. When you declare dependencies instead of probing, the cache stays pure and the configurer can be skipped on identical inputs.

The findProgram function still exists for cases where you genuinely need the result at configure time. A new findProgramLazy function returns a LazyPath that defers the search to make time, keeping the cache clean.

Build Server Protocol: Exposing the Build Graph

With configuration serialized, the build system can serve it to external clients. Passing --listen=- starts the Build Server Protocol, which allows IDEs and other tools to inspect the build graph statically, receive notifications when steps start and complete, and request specific steps to build.

zig build --listen=-
# Build server now accepts protocol messages on stdin/stdout

The protocol is still early. It exposes post-configuration information: available steps, options, dependencies. Planned additions include module names and multiplexed compiler server protocol for type information, refactoring, and editing capabilities.

This is a breaking change for ZLS, the Zig language server. The maker and configurer split prevents ZLS from working with 0.17.0 in its current form. The Zig and ZLS teams are collaborating to restore and exceed the previous functionality through the build server protocol.

Incremental Compilation: The Payoff

The build system restructuring is not an end in itself. Its payoff is incremental compilation. Zig 0.17.0 fixes enough bugs in the ELF linker and the incremental compilation machinery that most projects targeting x86_64-linux can now use it.

zig build -fincremental --watch
# Listens for source changes and performs incremental rebuilds

The new ELF linker introduced in 0.16.0 gained full x86_64 support, static and shared library generation, GOT generation, copy relocations, GNU symbol versioning, DWARF debug information, and symbol hash table generation. It is not yet at feature parity with the legacy linker, but it is capable of building the vast majority of Zig projects on x86_64-linux. The legacy linker is disabled by default for that target.

The next release plans to eliminate the legacy ELF linker entirely and introduce a new Mach-O linker with incremental compilation support for macOS targets.

The Standard Library Gets a Safer Allocator

The build system is the headline, but the standard library changes matter for application code. std.heap.DebugAllocator is replaced by SafeAllocator, a thread-safe allocator with hard guarantees:

// SafeAllocator guarantees:
// - deinit reports all leaks and frees backing memory
// - allocation mismatches panic or segfault
// - allocations from other SafeAllocator instances with
//   different canaries cause a panic
// - double frees and resize/remap/free races panic or segfault

Every allocation is trailed by an AllocFooter containing metadata and stack traces, protected by a checksum to catch corruption from overwrites. Small allocations go into linearly-filled buckets; large ones go directly to the backing allocator. The allocator does not reuse memory, which means most writes after free cause a segmentation fault or are detected and reported.

Benchmarks published in the release notes show the new allocator reduces peak RSS by roughly half and cuts wall time by about a quarter when building the standard library tests, compared to the previous DebugAllocator under the same conditions.

Language Changes Worth Auditing

The @bitCast builtin was redefined to operate on the logical bit representation of a value rather than its in-memory layout. For integer and float types on little-endian targets, the behavior is unchanged. The new definition is endian-agnostic, which matters for cross-platform code. However, casts involving extern struct or extern union types are no longer permitted, and the change can break existing code without triggering a compile error. The release notes recommend auditing any @bitCast uses that involve array or vector types.

Two new builtins replace deprecated ones:

// Old (deprecated):
const n = @intFromEnum(my_enum);
const e = @enumFromInt(MyEnum, 42);

// New in 0.17.0:
const n = @backingInt(my_enum);
const e = @fromBackingInt(MyEnum, 42);

@backingInt works with all enums, tagged unions, and bitpacks with explicit backing integer types. @fromBackingInt infers its result type and performs safety checks for invalid tag values. Passing undefined as a backing integer for an enum yields safety-checked Illegal Behavior.

The @cImport mechanism, deprecated in 0.16.0, is now removed entirely. C translation moves to an external package with its own release cadence:

zig fetch --save git+https://codeberg.org/ziglang/translate-c

The formal grammar (grammar.peg) and the handwritten parser now agree, after a tool was written to generate a recursive descent parser from the grammar file and use it as a fuzz oracle. This unblocks future grammar changes and specification work.

What Breaks and What to Watch For

The maker and configurer split breaks response file use cases for std.Build.Step.Run. ZLS does not work with 0.17.0 yet. The @bitCast change is a silent break for code using array or vector types. Several notable regressions exist: compiler-rt fails to compile for soft float on x86, std.debug.simple_panic fails to compile, and weak Zig libc symbols cannot be reliably overridden.

LLVM loop vectorization remains disabled as a workaround for a miscompilation that affects the Zig compiler. The fix is merged into LLVM main but is not in LLVM 22, the version Zig 0.17.0 ships. Zig 0.18.0 will upgrade to LLVM 23 and re-enable the optimization.

The release spans 5 months, 206 contributors, and 925 commits. The roadmap to 1.0 continues: language proposals are being decided (around 25 accepted, 125 rejected this cycle), with 23 proposals still open on Codeberg and 61 on the legacy GitHub tracker. The build system split is infrastructure that the remaining language decisions will depend on.

Originally published on Dispatch.

Decision Models Belong on the Agent Hot Path: What Cloudflare Clef Changes

Chen Yuan — Fri, 02 Oct 2026 06:14:48 +0000

Cloudflare released Clef and Clef-flash on October 1, 2026, and the useful part is not that another model family exists. The useful part is that these models are designed for a narrow job that large language models often perform badly: choosing among a bounded set of actions quickly, with typed outputs and calibrated probabilities.

That distinction matters for agent systems. Many agent loops use a large language model for everything: interpret an event, decide whether it is risky, choose a route, pick a tool, select a priority, and only then generate text or execute an action. The result is flexible, but it is also expensive, variable, and difficult to test.

Clef points toward a different architecture. Put a small decision model on the hot path for bounded choices, then call a larger model only when the task actually needs open-ended reasoning or generation.

Cloudflare says Clef is based on a frozen Qwen3.8-27B backbone, while Clef-flash uses a frozen Qwen3.5-9B backbone. The company trains routing heads and low-rank adapters rather than turning the whole system into another general text generator. The output is a probability distribution over allowed schema choices.

That sounds modest. It changes a surprising amount.

A Decision Is Not the Same Thing as a Generation

Consider a support agent receiving a customer message. The system may need to answer three questions before anything else happens:

Is the request urgent?
Which team should receive it?
Should the system escalate to a human?

A language model can answer those questions, but the task itself is bounded. There is no need to generate paragraphs, chain through tool calls, or produce prose that another parser must interpret.

A decision model can represent the same job as typed choices:

schema = {
    "urgent": [True, False],
    "team": ["billing", "security", "support"],
    "escalate": [True, False],
}

The useful output is not a sentence saying that a ticket appears urgent. The useful output is a stable record with probabilities attached to each allowed choice.

That gives the caller something operational rather than conversational.

The distinction becomes more important as agents move into production systems. A free-form model may decide that a ticket is "probably urgent" or invent a new category such as "priority support." A schema-bound model cannot create a category that the downstream system does not understand.

This is less expressive by design. For routing, policy gates, moderation decisions, retries, model selection, and workflow transitions, that is often a feature.

Clef Treats Schema Choices as the Product

Cloudflare describes Clef as using a two-stage attention routing process. Each valid choice extracts context from the prompt, fields can cross-attend with other fields, and the system scores only the options that the caller declared.

The important engineering idea is not the exact attention layout. It is that the schema is part of inference rather than an instruction the model may or may not follow.

A typical agent router could ask for several fields at once:

{
  "input": "Production API latency doubled after a deploy",
  "questions": {
    "severity": ["low", "medium", "high", "critical"],
    "owner": ["application", "database", "network"],
    "rollback": ["yes", "no"]
  }
}

The caller already knows every legal answer. The model's job is to rank them.

That is a cleaner contract than asking a text model to emit JSON and then validating, repairing, or retrying the result. Structured generation has improved, but it still solves a broader problem than many control-plane decisions require.

This is why decision models fit naturally beside large language models rather than replacing them. The decision model narrows the path. The language model handles the parts that remain open-ended.

The Latency Numbers Change Where a Model Can Sit

Cloudflare's published benchmark table reports a median latency of about 209.3 milliseconds for Clef, 38.8 milliseconds for Clef-flash, and 524.1 milliseconds for Jev across its benchmark runs. Its p95 figures are about 238.6 milliseconds for Clef, 122.4 milliseconds for Clef-flash, and 536 milliseconds for Jev.

Those measurements are Cloudflare's own benchmark results, so they should not be treated as universal production numbers. Hardware, region, payload size, and schema complexity all matter.

The architectural implication is still useful.

A model that consistently returns a bounded routing decision in tens of milliseconds can sit in places where a general language model feels too heavy. Examples include request admission, abuse scoring, tool routing, fallback selection, alert triage, queue assignment, and policy checks before an expensive model call.

A simple router might look like this:

def handle_request(request):
    decision = clef_decide(request)

    if decision["risk"] == "high":
        return escalate(request)

    if decision["needs_reasoning"] == "yes":
        return call_large_model(request)

    return run_fast_path(request, decision)

The model is no longer the whole application. It is one bounded component in a larger system.

That makes latency easier to budget and failure easier to isolate.

Probability Calibration Is More Useful Than Confident Prose

One of the more interesting training details is Cloudflare's use of a Brier loss for probability calibration. The company also describes a reinforcement-learning objective called Reinforcement Learning for Calibrated Decisions, or RLCD.

Calibration matters because production systems rarely need only a label. They need to know how much confidence to place in that label.

Suppose a moderation gate returns the same top answer for two requests:

request A: allow 0.98, review 0.02
request B: allow 0.54, review 0.46

Treating those as identical "allow" decisions would throw away useful information.

A production controller can set explicit thresholds. High-confidence choices can continue automatically. Borderline choices can go to a larger model or a human. Very low-confidence cases can fail closed.

This is a better fit for policy systems than trying to infer confidence from the tone of generated text. Fluent prose is not a calibrated score.

The same pattern applies to incident routing, fraud detection, email classification, customer support queues, and tool selection. The more expensive the wrong action is, the more valuable calibrated uncertainty becomes.

Frozen Backbones Make the Fine-Tuning Story More Practical

Cloudflare says it freezes the Qwen backbones and trains routing components plus rank-256 low-rank adapters. That choice matters because it reduces the amount of model state that must change for a specialized task.

The company is pairing Clef with a new reinforcement-learning service. Its proposed pipeline uses existing Cloudflare infrastructure: AI Gateway to collect traffic, Workers AI for rollouts, Containers as scoring sandboxes, a Trainer component to update weights, and Workers AI with bring-your-own-model support for redeployment.

The loop can be expressed as a conventional control system:

production traffic
    -> capture examples
    -> score decisions
    -> train adapters
    -> evaluate calibration
    -> deploy candidate
    -> compare with current model

That is more operationally interesting than "fine-tune a model" as a standalone feature.

The hard part of applied reinforcement learning is often not the optimizer. It is constructing repeatable environments, capturing representative data, defining rewards, replaying failures, and deciding when a new model is safe enough to serve.

Cloudflare already operates traffic, compute, sandboxing, and model serving layers. Its RL product is an attempt to connect those pieces into one feedback loop.

Whether that service becomes broadly useful will depend on tooling quality and evaluation discipline, not just training speed.

Decision Models Can Cut Large Models Out of the Boring Branches

A typical agent architecture has many decisions that do not require language generation.

Which model should handle this request?

Should the system retry or stop?

Which tool is eligible?

Is the user asking for an action or an explanation?

Does this event belong in the security queue or the operations queue?

Should an expensive retrieval step run at all?

Large models can answer all of these. That does not mean they should.

If a system calls a large model for every branch, the cost is not only inference. It also inherits larger variance, longer tail latency, more prompt surface, more formatting checks, and more opportunities for prompt injection to influence control logic.

A bounded decision layer shrinks that attack and validation surface. The application declares legal outputs first. The model assigns probability to them second.

That does not remove security concerns. Inputs can still be adversarial, and a classifier can still be wrong. It does make the contract easier to reason about.

For many agent systems, the best design may be a cascade:

rules for obvious cases
    -> decision model for bounded ambiguity
    -> large model for open-ended reasoning
    -> human for high-impact uncertainty

Each layer handles the cases that justify its cost and flexibility.

Open Weights Matter Because Routing Logic Is Infrastructure

Cloudflare released Clef and Clef-flash under the Apache 2.0 license and says the hosted versions are compatible with the Jev API.

That is important because routing logic is not merely another content feature. Once a decision model sits in the request path, it becomes infrastructure.

Infrastructure benefits from portability. Teams may want to run the same decision model locally, inside a private network, near a data source, or on a different inference stack. Open weights make it possible to inspect behavior, benchmark on internal data, and move serving locations without redesigning the application contract.

Compatibility also reduces switching cost. If multiple decision models accept the same bounded schema interface, applications can compare them without rewriting every caller.

That creates a more interesting competitive dimension than raw benchmark rank. A team can ask which model is best calibrated on its own traffic, which one meets latency targets, and which one behaves most predictably under distribution shift.

Those are infrastructure questions, not chatbot questions.

The Main Risk Is Treating Every Choice as a Classification Problem

The appeal of fast structured decisions can produce the opposite mistake: forcing open-ended reasoning into a schema that is too small.

Some tasks genuinely need exploration. A novel production incident may not belong to any known category. A code-review problem may require tracing state across several files. A user may ask something that changes the available choices themselves.

A decision model is strongest when the action space is already known.

That suggests a useful design test. Before adding a decision model, ask four questions:

Are the legal outputs known before inference?
Can a wrong choice be detected or escalated?
Is latency or cost important enough to justify a specialized path?
Does the system benefit from explicit probabilities rather than prose?

If the answer to those questions is mostly yes, a decision model is a strong candidate.

If the model needs to invent the action space, explain tradeoffs, synthesize evidence, or create a plan that did not exist before the request arrived, a general reasoning model remains the better tool.

The Better Agent Architecture Is More Heterogeneous

Clef is interesting because it argues against the idea that every agent improvement requires a larger general model.

Production systems already use different components for storage, search, queues, policy, and execution. Model architecture is starting to follow the same pattern.

One model may extract. Another may rank. Another may make a bounded decision. Another may reason across a large context. A deterministic program may perform the final write.

That looks less magical than a single autonomous model controlling everything. It is also easier to measure.

The practical lesson is not that decision models will replace language models. It is that many agent systems are wasting language-model capacity on decisions that should be explicit, typed, and cheap.

Cloudflare's Clef release makes that separation concrete. The most useful question for an agent designer is now simpler: which parts of the workflow truly require generation, and which parts only require a reliable choice among known options?

Sources

Cloudflare Blog: https://blog.cloudflare.com/clef-decision-models/
Cloudflare Workers AI changelog: https://developers.cloudflare.com/changelog/post/2026-10-01-clef-workers-ai/
Hacker News discussion: https://news.ycombinator.com/news

Originally published on Dispatch.

MicroVMs at the Edge: What Netlify's Firecracker Migration Actually Changed

Chen Yuan — Thu, 01 Oct 2026 08:50:26 +0000

Netlify says its Edge Functions now handle about a billion invocations each day. That makes a small amount of per-request overhead expensive twice: users feel it as latency, and the platform pays it across a very large request volume.

The interesting part of Netlify's recent infrastructure migration is not simply that virtual machines became fast enough for edge compute. The deeper change is architectural. Netlify moved from a hosted V8-isolate execution service to Firecracker MicroVMs running inside its own edge network, while keeping the programming model for customers largely unchanged.

According to Netlify's engineering write-up, warm invocation overhead fell from roughly 25–40 ms on the previous infrastructure to about 5–6 ms at the median. Netlify also reports 47.4 percent faster p99 invocations, 99.998 percent availability, and log delivery that is about five times faster. Unikraft, which worked with Netlify on the execution layer, reports a 2 ms p99 for MicroVM startup in Netlify's fleet.

Those numbers matter, but the mechanisms behind them are more reusable than the benchmark headline.

The old boundary was outside Netlify's network

An Edge Function sits in the request path. If a request matches a configured route, the platform must select compute, start or reuse an execution environment, run customer code, and return the result before the request can continue.

On Netlify's previous architecture, a matching request left Netlify's own network for a hosted execution service based on V8 isolates. That provider handled execution, provisioning, and much of the runtime environment. The model offered operational simplicity, but it also created a network boundary Netlify did not control.

The new path keeps that work inside Netlify's network. The edge node that terminates TLS checks the deployment's Edge Function routes. If a route matches, it forwards the request to a regional compute node rather than sending it to an external execution provider.

The key lesson is that runtime startup time is only one term in request latency. Placement, network hops, image availability, service lookup, logging, and failure recovery all sit on the critical path too. Removing an external hop can therefore matter as much as making the runtime itself faster.

A request carries the machine specification with it

Before forwarding the request, the edge node writes a specification for the machine that should execute the function. Netlify describes three image layers in that specification: the runtime, a platform image, and the customer's Edge Function image. CPU, memory, and connection limits travel with the spec as well.

A simplified representation might look like this:

{
  "runtime": "deno-runtime-v42",
  "platform": "netlify-edge-v18",
  "function": "site-7f3a-image",
  "cpu": 1,
  "memory_mb": 512,
  "connection_limit": 64
}

The exact object above is illustrative, not Netlify's production schema. The important design idea is that enough information travels with the request to identify the execution service deterministically.

Netlify hashes the machine specification together with site-specific information to derive a service ID. Different deploys or environment-variable sets therefore map to different services and do not share a MicroVM. That makes deployment identity part of the isolation boundary instead of a convention layered on top of a shared process.

Rendezvous hashing keeps the fast path warm

Each region contains a group of compute nodes. Netlify uses rendezvous hashing so the same service normally lands on the same node. That improves locality: once a node has fetched an Edge Function image, later requests can reuse the copy already on disk, and an existing MicroVM or snapshot can stay useful for that service.

A minimal version of the placement idea can be expressed as:

def choose_node(service_id, nodes):
    scored = [(score(service_id, node.id), node) for node in nodes]
    return max(scored, key=lambda item: item[0])[1]

Real production placement has more inputs, but the property matters: stable input usually produces stable placement.

Pure stickiness creates another problem. A busy service can become a hot spot and saturate the node selected for it. Netlify therefore relaxes placement above a threshold and spreads a hot service across a slice of nodes. That is a useful pattern for state-light edge workloads: prefer locality when traffic is ordinary, then trade some cache warmth for capacity when one tenant becomes dominant.

Cold starts are mostly an image-distribution problem

A node that has never served a function needs the relevant images before it can run anything. Netlify reports that this cold path occurs on about 1.2 percent of invocations and takes about 9 ms on average.

The design avoids distributing every customer's code to every compute node in advance. A node fetches an image only when traffic for that service arrives in the region, stores it locally, and reuses it for later requests. Unikraft describes the same strategy as on-demand image resolution and notes that stale images can be pruned from the node cache automatically.

This changes the scaling equation. Pre-provisioning every deployment everywhere would consume storage and memory before demand exists. Fetch-on-first-use lets the fleet pay distribution cost where traffic actually appears.

The cache policy can be summarized as a small state machine:

request arrives
  -> image present: boot or restore MicroVM
  -> image missing: fetch image, cache image, then boot
  -> image idle long enough: prune local copy

That mechanism also decouples deploy frequency from fleet-wide synchronization. A new function image does not need to be pushed to every node before it can receive traffic.

The MicroVM is fast because it avoids doing unnecessary work

Netlify says each function runs in its own Firecracker MicroVM. The platform creates these environments in under a millisecond and reports roughly 2 ms p99 startup. The VM boots a stripped-down Linux environment rather than a conventional general-purpose guest with a large boot sequence.

The function files are mounted as an uncompressed EROFS image and memory-mapped. That allows the VM to read the parts of the bundle it actually touches instead of copying the entire function into memory before execution can begin.

There is another optimization after boot. When the JavaScript server starts listening, the platform snapshots the MicroVM. Idle instances can scale to zero. A later request can restore a new MicroVM from the snapshot, and the snapshot itself is memory-mapped so execution does not have to wait for every page to be loaded first.

The practical takeaway is that "VM" is too broad a performance category. A large cloud VM that boots an ordinary operating system and a purpose-built MicroVM restored from a memory-mapped snapshot have very different startup paths.

Isolation became a property of the compute boundary

Netlify also frames the migration as a security change. A service with different code or environment variables receives a distinct service ID, and different services do not share a MicroVM. If customer code escapes the JavaScript runtime, it still faces a VM boundary before it can reach another tenant or the compute host.

That is a stronger boundary than placing mutually untrusted programs in separate V8 isolates inside a shared process. It does not make the system invulnerable, and no virtualization boundary should be described that way. It does change where a compromise has to cross next.

For edge platforms, that matters because performance and tenant isolation are usually in tension. The notable result here is not that Netlify chose stronger isolation at any cost. It reports lower request overhead after making that change.

The migration therefore challenges a common assumption: process-level isolation is not automatically the only practical option for very low-latency serverless work. If the VM lifecycle is reduced far enough, hardware virtualization can fit inside a single-digit-millisecond budget.

Owning the runtime changes the feature ceiling

The previous hosted provider also controlled the runtime. Netlify says that limited features such as filesystem access, WASM loading, and native npm modules. In the new system, Deno remains the JavaScript runtime, but Netlify ships and patches its own build inside the MicroVM.

That means infrastructure ownership affects product capability, not just cost or latency. A runtime provider can be an excellent abstraction until the platform needs a feature below that abstraction.

Netlify points to native npm modules, filesystem behavior, and future runtime choices as areas that become more tractable with a full VM. Unikraft separately notes that Netlify can deploy its own runtime changes without waiting for Unikraft to coordinate the rollout.

The distinction is useful for platform design. Outsourcing execution minimizes the amount of infrastructure a team must operate. Bringing execution in-house increases the operational surface, but it also moves the constraint boundary. The right choice depends on whether the managed boundary still matches the product roadmap.

Reliability comes from boring control-plane decisions

Fast boot is only useful if the fleet remains healthy under real traffic. Netlify describes several less glamorous choices that make the system operable: local DNS resolvers on compute nodes, circuit breakers for rerouting, health tracking in a control plane, and separate compute-node fleets that can be rolled out beside the current fleet before taking traffic.

The control plane gives edge nodes a list of healthy compute nodes. During a deployment, a new fleet comes up alongside the existing one, scales to the required size, proves healthy, and only then takes traffic. That keeps rollout and rollback separate from the per-request execution path.

Unikraft adds another scale detail: Netlify can create hundreds of millions of MicroVM invocations per day, and customer traffic can spike far above its normal level. At that scale, tiny-probability faults stop being theoretical. The architecture must assume rare failures will eventually happen.

That is why observability is part of the performance design. Netlify records boot time, time to first port open, and time until user code starts. Those measurements let engineers distinguish a slow image fetch from a slow VM restore or a slow customer function instead of collapsing everything into one latency number.

The migration is a lesson in moving work off the critical path

Several optimizations in the new design share the same shape. They do not merely make an operation faster; they avoid doing it for most requests.

Images are fetched only on a cache miss. The full fleet is not updated for every function deployment. Idle MicroVMs scale to zero instead of consuming capacity. A restored snapshot is memory-mapped rather than copied in full before execution. Rendezvous hashing keeps a service near already-cached code until load makes that placement harmful. Compute nodes use local DNS rather than turning every lookup into another remote dependency.

This is a useful way to review any serverless request path. List every operation between accepting a request and producing the first response byte. For each operation, ask whether it must happen on every request, whether its result can be cached, whether it can be prepared lazily, and whether failure can be isolated without blocking unrelated tenants.

A rough review template is:

for each critical-path operation:
    measure its p50 and p99
    identify reusable state
    move reusable state closer to the request
    remove remote dependencies where practical
    define a bounded fallback when locality fails

The important word is bounded. A cache or sticky-placement strategy that has no escape route becomes a source of hot spots. A snapshot that cannot be invalidated becomes stale state. A local dependency without health checks becomes a silent single point of failure.

What to copy from this architecture

Most teams should not build an edge compute platform. The reusable ideas sit one level above the implementation details.

First, measure the entire request path, not just application execution time. A fast runtime can still sit behind slow routing, remote provisioning, or logging infrastructure.

Second, make locality intentional. Stable placement and local image caches reduce repeated work, but load-aware spreading is necessary when locality starts harming capacity.

Third, use identity to enforce isolation. Netlify derives a service ID from the machine specification and site-specific state, so a deployment change naturally creates a separate execution service. That is safer than hoping two logically different deployments remain separated by convention.

Fourth, treat cold starts as a distribution problem as well as a boot problem. If code or images must cross the network before execution, a very fast VM cannot hide an inefficient artifact-delivery path.

Fifth, keep rollout control outside the hot path. A control plane can decide which nodes are healthy and which fleet version should receive traffic without forcing every request to perform control-plane work.

Netlify's reported result is a warm Edge Function path of roughly 5–6 ms at the median, with cold image fetches affecting about 1.2 percent of invocations and averaging roughly 9 ms. The useful engineering story is how those numbers were obtained: fewer remote boundaries, on-demand artifacts, deterministic locality, fast snapshot restore, hardware-backed tenant separation, and explicit escape paths when the fast path becomes a hot spot.

That is a more durable lesson than "MicroVMs beat isolates." Architecture wins here because the whole request path changed.

Sources

Originally published on Dispatch.

How to Build a Post-Launch Eval Canary That Tells a Real LLM Regression From Sampling Noise

Chen Yuan — Wed, 30 Sep 2026 13:56:53 +0000

Is the model actually getting worse, or did I just get unlucky on a handful of prompts?

That question is why threads like "is it just me or is it dumber today" keep recurring, and it is the question a post-launch eval canary has to answer with a number instead of a feeling. The reference implementation here is livenerf, a long-running, deterministic-as-possible benchmark tracking whether Claude Opus 5.5 (released 2026-09-22) gets quietly worse after launch. This article teaches the reusable method behind it: how to freeze prompts, pin the harness, calibrate a panel of "sometimes right" questions, compute paired per-item statistics with clustered standard errors, run a control arm, watch output-token counts as an early signal, and pre-register the rule that decides when you are allowed to say "regression."

You do not need a frontier model or a cluster. You need a few hundred labeled questions, one machine, and the discipline to fix everything except the model.

Step 1: Accept that the harness is part of the treatment

A canary compares a model to itself across time. Every other moving part becomes a confound. The livenerf v0 runner executes a Claude Max subscription through headless Claude Code (claude -p), pins the CLI version, disables the auto-updater, and refuses to run when the version no longer matches — because a changed harness looks exactly like a changed model. The same logic applies to your system prompt, your sampling parameters, your tool list, and your working directory.

The cheapest enforcement is a guard that fails loudly. The code below is illustrative and was not executed for this article; treat the version string and command shape as placeholders for your own stack.

# illustrative, not executed
import re
import subprocess
import sys

PINNED_CLI = "claude-code 2.4.1"

def assert_pinned_harness() -> None:
    out = subprocess.run(
        ["claude", "--version"], capture_output=True, text=True, check=True
    ).stdout.strip()
    if out != PINNED_CLI:
        sys.exit(
            f"harness drift: expected {PINNED_CLI!r}, got {out!r}. "
            "Refusing to run; a harness change is indistinguishable from a model change."
        )

Version pinning only works if the run itself is hermetic. livenerf uses a short frozen system prompt, no tools, no MCP, no CLAUDE.md or memory, one turn, and a fixed empty working directory. Code answers are graded by hidden tests in a sandbox outside the model. That last decision matters more than it sounds: if an LLM judge scores your outputs, the judge can drift too, and you will be measuring two moving objects at once.

Step 2: Freeze a prompt file, not a prompt idea

A canary needs an artifact you can hash. Keep the system prompt, the item text, and the scoring function in version control, and make the runner read them from disk rather than from a database that someone can quietly edit.

# illustrative, not executed
import hashlib
import json
from pathlib import Path

PANEL = Path("panel/frozen_panel_v1.jsonl")
SYSTEM = Path("panel/system_v1.txt")

def load_frozen():
    system = SYSTEM.read_text(encoding="utf-8")
    items = [json.loads(line) for line in PANEL.read_text(encoding="utf-8").splitlines()]
    digest = hashlib.sha256(
        SYSTEM.read_bytes() + PANEL.read_bytes()
    ).hexdigest()[:12]
    return system, items, digest

Two files, one digest, no edits without a new panel version. The digest is what you log next to every run so a later comparison can prove it used the same questions.

Step 3: Calibrate toward "sometimes right" questions

Here is the part most homegrown evals skip, and it is the part that buys you statistical power. Under a logit-shift model, a question with pass rate p carries information p(1-p) per sample. Questions the model always answers correctly contribute nearly nothing, and questions it always fails contribute nearly nothing. Only questions near the middle of the difficulty curve move when capability moves.

livenerf screened 2,336 GPQA Diamond, MMLU-Pro, competition-math and AIME 2025–26 questions with 4 samples each. Opus 5.5 got roughly 93% right on the first try, and 97% of questions turned out to be always right or always wrong. The surviving 78 "sometimes right" questions became the frozen panel. A second effect showed up: questions selected for being sometimes-right regress toward the mean. On fresh samples their pass rate rose from 54.7% to 62.0%, so the power calculation has to use the fresh rates rather than the selection rates.

If you calibrate your own panel, do the same: sample each candidate question several times, keep the ones with intermediate pass rates, then re-estimate their rates on data you did not use for selection. Budget the panel around that middle band, because that is where the information is.

Step 4: Compare items, not just averages

A raw accuracy comparison between two time windows carries question difficulty inside it. If day 30 happens to sample harder items, the score drops for reasons that have nothing to do with the model. The fix is to compare each item to itself: for every question, take the score in the current window minus the score in the launch-week baseline, then average those per-item deltas.

Because each item is measured repeatedly, the per-item deltas are not independent, and naive standard errors will be too small. That is what clustered standard errors are for, and it is the approach livenerf takes, following Evan Miller's "Adding Error Bars to Evals". Cluster by question; with one observation per item per window, the per-item delta is the cluster.

# illustrative, not executed
import numpy as np

def paired_delta_with_clustered_se(baseline: np.ndarray, current: np.ndarray) -> dict:
    """baseline/current are item-level scores in [0,1], same item order."""
    d = current - baseline
    n = len(d)
    mean = d.mean()
    # cluster-robust variance with one cluster per item:
    # var(mean) = sum_i (d_i - mean)^2 / (n * (n - 1))
    var = ((d - mean) ** 2).sum() / (n * (n - 1))
    se = np.sqrt(var)
    return {
        "n_items": n,
        "delta": float(mean),
        "se": float(se),
        "ci99": (float(mean - 2.576 * se), float(mean + 2.576 * se)),
    }

The choice of 99% rather than 95% is deliberate: a canary fires repeatedly, so it needs a higher bar per look to keep false alarms rare. With one run per day of the whole panel, livenerf's own power calculation puts the detectable accuracy change at roughly 7.5 points per 10-day window. That is the instrument's resolution, and knowing it is what stops you from over-reading a 2-point wobble.

Step 5: Add a control arm and watch tokens first

The control arm is the cheapest way to avoid blaming the model for something else. livenerf runs claude-opus-5 on the same harness alongside the tracked model. If the control arm moves by the same amount, the change is probably infrastructure, a shared dependency, or the harness — not a silent downgrade of the tracked model.

Output-token counts are the early signal. In livenerf's validation, lowering effort to low cut output tokens by 62% and accuracy by 8.3 ± 4.5 points; effort medium cut tokens by 26% and accuracy by 4.2 ± 3.9 points. Tokens moved before accuracy did, which makes them a leading indicator worth logging on every run even when accuracy looks flat.

Step 6: Pre-register the decision rule

A canary that decides what counts as a regression after seeing the data is a vibes machine with extra steps. The livenerf decision rule was committed to public git before any series data existed, so the timestamp is meaningful. It calls a change only if three conditions hold together: the 99% interval excludes zero in two consecutive 10-day windows, the effect is at least 3 points, and the control arm does not show the same move. Null results and improvements get published just as loudly as regressions.

The two-window requirement is what protects against a single unlucky stretch. Here is a compact check you can adapt; it assumes you already computed intervals per window for the tracked model and the control.

# illustrative, not executed
MIN_EFFECT = 0.03

def call_regression(windows: list[dict], control: list[dict]) -> str:
    """windows: per-10-day dicts with 'ci99' and 'delta' for the tracked model."""
    if len(windows) < 2:
        return "insufficient data"
    last_two = windows[-2:]
    excludes_zero = all(w["ci99"][0] > 0 or w["ci99"][1] < 0 for w in last_two)
    big_enough = all(abs(w["delta"]) >= MIN_EFFECT for w in last_two)
    control_moved = any(
        c["ci99"][0] > 0 or c["ci99"][1] < 0 for c in control[-2:]
    )
    if excludes_zero and big_enough and not control_moved:
        return "regression called"
    return "no call"

Note what the rule does not do. It does not treat the launch-week baseline as ground truth. livenerf's README is explicit that launch week is a reference point, not a ceiling: launch week could be the worst week because of capacity strain, and past quality incidents turned out to be infrastructure bugs rather than deliberate downgrades. The baseline is the thing you compare against, not the thing you trust.

Step 7: Publish the limits alongside the numbers

An instrument that hides its blind spots will be misused. livenerf documents a concrete one: a same-family swap (Opus 5 in place of Opus 5.5) was not distinguishable in a validation's worth of samples, at -3.8 ± 6.3 points and -23% tokens. That is a known limit of the instrument, written down in advance so a null result is not later spun as a clean bill of health.

Every canary has a floor. The honest move is to state yours: how many points you can detect, over what window, with what false-alarm rate, and which kinds of changes you would miss entirely. Then keep the raw logs append-only, as livenerf does with Inspect .eval files, so anyone can re-analyze without re-running.

The method in one paragraph

Freeze the prompts and the harness, calibrate toward questions that are sometimes right, compare each item against its own baseline, cluster your standard errors by question, run a control arm, log tokens as a leading indicator, and pre-register the rule that authorizes the word "regression." None of that requires a lab. It requires deciding, before you look, what would count as evidence — and then publishing the null results with the same energy as the alarming ones. The alternative is the loop we already know: vibes versus vibes, one launch week at a time.

Originally published on Dispatch.