The test we called flaky was filing records under the wrong owner

TL;DR — A contention test in my agent tooling failed in 6.7% of standalone runs (24% in a noisy shared tree). I had labeled it a "known flake." The real failure: under load, an observability record was occasionally filed under the wrong owner. Total row count stayed correct, every writer process exited 0, and the only check that reliably notices this failure shape — a per-owner count assertion — was the one I kept rerunning until it went green. This post is about that failure shape: misattribution, which is invisible to every totals-based check, and which the flaky-test literature and the SQLite-locking literature each half-describe but don't connect.

Background: what's already well covered

Three adjacent topics are thoroughly written up, and I won't re-litigate them:

• "Your flaky test might be a real bug" is established wisdom: TestRail's guide on flakes masking genuine defects, and Test Flakiness: The Silent Killer of Engineering Trust on why "it passes sometimes" must not become an accepted state.
• SQLITE_BUSY despite a timeout has a canonical explainer in Bert Hubert's post (implicit transaction upgrades), and sqlite-net #778 documents lock errors when journal_mode=WAL runs before a timeout can be configured. Simon Willison's TIL covers the part many people miss: write-ahead logging (WAL) mode is persisted in the database file header, so you don't need to re-issue it per connection.
• Silent data failures in pipelines — rows that stay countable while becoming wrong — are a known blind spot in the data-quality world (e.g. Catching Silent Failures in Data Pipelines).

What I could not find written anywhere: what misattribution looks like as a test-design problem — which assertions can catch it at all, and how easily the one assertion that can gets killed as a flake.

The incident

My agent tooling runs each large language model (LLM) agent as an OS process. A small Rust hook fires on every agent event and appends one row to a shared SQLite database in WAL mode. This log is the single source of truth for "which agent is alive and what is it doing." A contention drill spawns 300 writer processes — 30 running concurrently, 10 events per owner — and asserts, per owner, that all rows landed.

It usually passed. Sometimes, under parallel load, one owner had 9 rows instead of 10. Reruns in isolation: green, green. Into the books it went as "known load flake — rerun solo, two consecutive greens = pre-existing."

When I banned that label and required a mechanism, the first failing run said everything:

seat-00     | 9     <- expected 10
env-seat-00 | 1     <- ...where did YOU come from

(Owner names lightly anonymized.) The record was not lost. It was filed under a different owner. Totals: still 300. Writer exit codes: all 0. Distinct owners: 30 → 31.

Which checks can even see this?

This is the part I wish someone had written down before:

Check	Record loss	Record misattribution
Total row count	catches	blind
Writer exit codes	catches (write errors only)	blind
"Did all N processes report?"	catches	blind
Distinct-owner count	blind (unless an owner loses every row)	catches only when the wrong owner is a new one (here 30 → 31); blind when a record moves between existing owners
Per-owner row count	catches	catches

Everything in the first three rows — the checks a reasonable engineer reaches for first — stays green while ownership moves. In my system the per-owner assertion existed only because the drill happened to be written that way, and it was the assertion I kept rerunning until it passed. The tripwire was almost deleted for doing its job.

Mechanism: a fallback that lies

The hook resolves which agent it belongs to like this:

session_id ──> registry DB lookup ──ok──> seat id        (authoritative)
                     │
                    Err  (swallowed: `Err(_) => {}`)
                     ▼
              environment variable                        (known-unreliable)

The environment variable is unreliable by design in my background-agent mode — a shared daemon hands every agent the first client's environment, which is exactly why the session-ID lookup exists as the primary source. So "registry momentarily unreadable" silently degraded into "file this event under whoever the environment claims." A different, wrong owner.

The trigger for "momentarily unreadable" was one line executed on every connection open, after busy_timeout(5000) was set:

conn.pragma_update(None, "journal_mode", "WAL")   // unconditional, every open

Under a 30-process open stampede this failed with, verbatim:

SqliteFailure(Error { code: DatabaseBusy, extended_code: 5 }, Some("database is locked"))

In my runs this statement was not protected by the busy handler: failing runs completed in 3–4 seconds total, nowhere near the 5-second timeout budget, so no retrying happened on that statement. (This is a different mechanism from the implicit-transaction upgrade in Bert Hubert's post, and different from sqlite-net #778 where the timeout couldn't be set before the pragma — here it was set and didn't cover it.) As Simon Willison's TIL notes, re-issuing the pragma is unnecessary anyway once the file is in WAL mode — which is what made the collision window pure waste.

What I changed, and what each change was worth

Configuration	Result
Baseline	6.7% fail (dedicated tree, standalone) / 24% (noisy shared tree)
Fix 1 only — issue the pragma only if the DB is not already WAL	1 failure in 10 standalone runs
Fixes 1+2 — also remove the env fallback	10 consecutive greens
Fixes 1+2+3 — also bounded retry on first WAL transition	20 for 20 under deliberate parallel load

• Fix 2 is the one that addresses misattribution itself: if a session ID exists but the registry read fails, don't fall back to a less trustworthy source. Degrade to a deterministic short-ID placeholder that the daemon later reconciles, and emit a marker event so the degradation is observable. During the 10-run verification, one green run recorded exactly one such marker — the degradation path fired for real and was absorbed correctly.
• Fix 3 exists because I almost stopped too early. After fixes 1+2 hit 10 consecutive greens, a follow-up 20-run campaign failed its first run with a different mode: 7 of 300 writer processes exited non-zero on a brand-new database file (30 processes stampeding the very first WAL transition, which fix 1's check cannot avoid). Bounded retry — 5 attempts, linear backoff, then fail loud — closed it. If I had stopped at 10 runs, "fixed" would once again be sitting on top of a live bug.

Average write cost was 11.6 ms per process against the 5000 ms budget — this was never a throughput problem.

Takeaways

1. Misattribution is a distinct failure shape from loss, and it defeats totals. If a record's owner can be derived from fallible context, put at least one per-owner (or per-attributor) assertion somewhere. Nothing else in the matrix above catches it.
2. A fallback from an authoritative source to a less-trustworthy one is not resilience — it is a data-corruption path. Degrade to an explicit placeholder and mark the degradation. (This was not the only swallowed-error fallback in my codebase — merely the first one caught in the act.)
3. Decide your consecutive-green bar before you start, and overshoot it. 10 was not enough here; the 20-run campaign caught a second failure mode on run one.

Limits / untested

• The journal-mode/busy-handler behavior above is what I observed on my setup (Rust + rusqlite, bundled SQLite, Windows, process-per-writer); I have not tested other bindings or platforms, and I make no claim about all SQLite statement types.
• I have not measured how much production misattribution the conditional-pragma fix alone would have prevented; production contention differs from the drill's.
• A continuous per-owner drift check (in the retention job, not just in tests) is designed but not built.

Repro notes

• Windows 11, Rust (rusqlite, bundled SQLite), one writer process per event (300 total, 30 concurrent) appending to one WAL-mode file; busy_timeout(5000) set on every connection before any other statement.
• In my runs the failure appeared only when real process startups collided on connection open. To watch misattribution live, group rows by owner and diff against the expected owner set — don't count totals.
• The systems are personal tooling and not distributed; owner names above are lightly anonymized.

Drafted with my local AI pipeline; all measurements are from real runs on my machine and reproducible from the notes above.