Nineteen times in six months, a database that was supposed to be boring quietly corrupted itself. Each incident took the same shape: a backup monitor or data pipeline reported an error, PRAGMA integrity_check confirmed corruption, and an on-call engineer stopped a control-plane process, restored from a snapshot, and tried to explain what had happened. Nobody could. The trigger changed every time — different shard, different customer, different time of day. The only constant was that a committed write had somehow stopped existing.
The bug behind all of it had been sitting in SQLite's Write-Ahead Logging code since 2010. It took Tailscale and the SQLite core team months of forensics to find it, one fix release that had to be withdrawn, and two more months of waiting for positive proof that the real fix worked. This is the case study: how a rare data race hides inside one of the most tested codebases in the world, why Tailscale hit it when almost nobody else could, and what the hunt teaches about the difference between standard and non-standard ways of running "boring" technology.
A Database That Corrupted Itself
Tailscale's control plane looks like a single public endpoint, but internally it is a set of coordination servers, or shards. Each shard owns a slice of tailnets and has its own SQLite database, accessed exclusively by one Go process. That single-writer design is exactly how SQLite is meant to be used: one writer, serialisable transactions, no cross-process locking drama.
SQLite became Tailscale's primary database in 2022 precisely because it is boring. It is well-known, reliable, and widely deployed at far larger scale than any per-shard database at Tailscale. The backup pipeline was equally unremarkable: every few minutes, a complete snapshot of the database file was uploaded to S3. It ran without incident from early 2023.
Then in August 2025, a data pipeline that reads those S3 backups reported an error in one database. PRAGMA integrity_check confirmed corruption. SQLite corruption is possible but highly unusual, and it is not something you should encounter in normal operation. The team repaired the affected database and investigated. Nothing was found.
It happened again. And again. Nineteen separate corruption incidents over six months.
The Hunt: Ruling Out the Obvious
PRAGMA integrity_check;
-- expected: ok
-- actual (August 2025, backup pipeline): "database disk image is malformed"
Every obvious theory failed. No recent change touched the low-level code that interacts with SQLite — it had been written years earlier and had been silent since. There was no common factor between incidents: not a single shard, customer, tailnet feature, time of day, or load level. With no reliable trigger condition, the bug could not be reproduced synthetically. The team fell back to deploying passive forensic telemetry in production and waiting for the next corruption to happen live.
The wait was unpredictable. Incidents came hours apart or weeks apart. Between October and December there were six weeks of calm, and then the corruption returned as an unwelcome Christmas present. Because the diagnosis was not going to be quick, Tailscale signed a professional support contract with the SQLite developers, which gave them direct access to the core maintainers.
Together they mapped out theories: broken POSIX advisory locks cancelled by a separate thread calling close(), mismanaged memory owned by SQLite, or accidentally using SQLite from multiple threads while thread-safety was disabled. After every incident, more diagnostics were added, and one theory after another was systematically ruled out.
The Transactions That Didn't Bark
While the root cause was unknown, the platform still had to run. Recovery was automated aggressively: shards hard-stopped immediately upon detected corruption, a backup monitor continuously ran PRAGMA integrity_check over every snapshot, and runbooks improved. Response time dropped to under an hour.
Then the team built something that doubled as a forensic instrument: a transaction logging pipeline. Every SQL statement that modified the database was streamed to a separate log file. Because SQLite is a single-writer database with serialisable transactions, the transaction history is completely linear and deterministic — replaying it against the last good backup reconstructs the most recent state without touching the corrupted file.
# concept: deterministic replay of a single-writer transaction log
for tx in transaction_log: # linear, in commit order
if backup_db.transaction_id < tx.id:
backup_db.execute(tx.sql) # SQLite serialises writes anyway
# result: latest consistent state, corruption bypassed
The pipeline worked — and then it found the clue. In two incidents, the transaction logs failed to replay cleanly. A write that had been committed was invisible to later transactions. Data had vanished without an error. In a single-writer, serialisable database, that should be impossible.
How WAL and Checkpoints Work
To see why it was possible, you need the storage layer. A SQLite database is a file of fixed-size pages. In the default rollback-journal mode, updates are written into the main file, which means a reader blocks a writer. Write-Ahead Logging (WAL) changes the deal: new pages are appended to a separate WAL file, and readers keep reading the old main file until a checkpoint copies the new pages back.
Checkpointing is normally SQLite's own decision, invisible to the application. Tailscale's backup pipeline needed fast, consistent snapshots, so the control plane took manual control of the checkpoint process — a public, documented, supported configuration. It also checkpointed aggressively. This non-standard cadence became the prime suspect, and the metrics agreed: during corruption incidents, SQLite reported copying more pages from the WAL than the WAL contained. Ten pages in the WAL, twenty pages copied to the database. Something in the checkpoint path was hallucinating.
WAL file: [p1][p2][p3] ... [p10] 10 pages
checkpoint log: "copied 20 pages to db file"
→ impossible in a correct checkpoint
To see inside that path, the SQLite developers built a new debugging tool: a shim around the virtual filesystem layer — the OS interface that actually writes bytes to disk — which logs every change to the database. SQLite's layered design (parser and code generator, pager, virtual filesystem) makes this kind of instrumentation cheap: you wrap one layer instead of recompiling the world. The shim, tmstmpvfs, was deployed into Tailscale's live environment. They did not have to wait long.
The WAL-Reset Bug
The next corruption incident gave the SQLite developers the trace they needed, and the bug surfaced: a rare data race between a checkpoint and a write transaction.
The mechanics are tight. If a write lands at a specific moment during a checkpoint, the checkpointing process gets confused and believes some pages were already copied from the WAL into the main database when they were not. Those pages are never written to the database file, so the data in them is permanently lost — no error, no rollback, nothing. The database file ends up corrupt because other pages that reference the missing ones, such as index pages, are written anyway. A committed transaction simply stops existing.
SQLite named it the WAL-Reset bug and dated it back at least 16 years. It survived that long because it is rare — so rare that the SQLite developers could never reproduce it organically and had to add special testing logic that deliberately triggers the race just to verify that the fix works. The fix itself is small: an additional check in the checkpointing function that detects when the WAL has been reset by another thread.
Why did Tailscale hit it when the rest of the world did not? Because it checkpointed manually and aggressively. Even a bug triggered by a rare timing condition becomes inevitable at a high enough checkpoint rate — which is the quiet mathematical truth about rare bugs at scale.
A Fix, a False Alarm, and a Withdrawn Release
The fix shipped as SQLite 3.52.0. Tailscale rolled it out carefully: canary shards first, then, when it looked healthy, the rest of the control plane. The backup monitor promptly turned red across 13 databases.
The relief was short-lived: those databases were not corrupt. The release that fixed the race also contained an unrelated optimisation that subtly changed text-to-floating-point rounding, and Tailscale stored high-precision timestamps as text that was converted inside a virtual generated column. Indexes on computed values had gone stale, and PRAGMA integrity_check reported them as corruption. The canary shards simply had not contained timestamps that triggered the new rounding behaviour, so the phased rollout missed it.
The SQLite developers withdrew 3.52.0 and published 3.51.3, containing only the WAL-Reset fix. Tailscale reduced its timestamps to integer seconds — text-to-integer conversion is unambiguous — and the SQLite team later shipped an automated self-healing index feature in 3.53.0 that prevents the stale-expression-index problem entirely. Two bugs, one release, one withdrawal, and two lessons about what a "fix" release can smuggle in.
Proof, Two Months Later
// conceptual sketch of the tripwire Tailscale added to its SQLite driver;
// not the actual patch. It logs when a checkpoint overlaps a WAL reset.
func checkpoint(db *sqlite.Conn) error {
if walWasResetByAnotherThread(db) {
log.Warn("SQLitePartyMode: WAL reset overlapped checkpoint; corruption prevented")
}
return db.Checkpoint(sqlite.CheckpointPassive)
}
An absence of incidents proves nothing — the team already had one six-week false calm behind them. They wanted positive evidence that the race was occurring in production and that the fix was intercepting it. So they patched their SQLite driver to log a warning whenever a write overlapped a WAL reset. If the warning fired and the database stayed healthy, the fix had saved them.
They deployed the tripwire and waited. Weeks slipped by. Doubts crept in: was the warning broken? Was the theory wrong? Was the true bug still hiding somewhere in the darkness? Two months later, the alert finally fired: "SQLite attempted corruption … but the system prevented it." The exact conditions for the bug do occur in production, and the fix was what stood between Tailscale and corruption number twenty.
Since that alert, Tailscale ran another four months with zero database incidents — the only number that ends an incident investigation.
Reproducing the Unreproducible
The same bug offers a second, shorter story about tooling. Antithesis took SQLite 3.51.2 — still buggy — added standard database assertions ("no lost committed writes", "database is not corrupt"), and pointed a generic workload at it: writes and checkpoints running concurrently, exactly what production does all the time. Their deterministic testing platform caught the bug in fifteen minutes. The same workload against 3.51.3 came back clean.
Six months of forensics in production; fifteen minutes under deterministic instrumentation. The asymmetry is not a knock on the forensics — the production hunt produced the diagnosis, the shim, and the fix. It is a reminder that when a bug cannot be reproduced organically, tooling that makes time reproducible is worth more than another week of staring at logs.
What a 16-Year-Old Bug Teaches Us
The operational lesson is the one Tailscale drew itself: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well tested. Manual checkpoint control was documented, supported, and public — and it was still a corner of the design space that had never been exercised the way Tailscale exercised it. If your database is the load-bearing wall of your service, the wall you modified is the wall you should stress-test.
The engineering lesson is about what "rare" means. A bug with tight timing constraints that nobody can reproduce is not a bug that does not exist; it is a bug waiting for the right workload. The WAL-Reset bug existed for sixteen years before anyone hit it, and it will not be the last of its kind. The cheap insurance is the same in every layer: integrity checks that run continuously, transaction logs that can be replayed, and tripwires that fire before corruption becomes an incident.
And the human lesson is that the fix is never the end. The release that fixed the race broke something else. The proof of a fix for a rare bug is not green tests — it is the alert that fires in production two months later, saying the thing you feared almost happened, and did not.
Originally published on Dispatch.
Top comments (0)