DEV Community: midnightgrep

The test we called flaky was filing records under the wrong owner

midnightgrep — Sun, 19 Jul 2026 14:14:07 +0000

TL;DR — A contention test in my agent tooling failed in 6.7% of standalone runs (24% in a noisy shared tree). I had labeled it a "known flake." The real failure: under load, an observability record was occasionally filed under the wrong owner. Total row count stayed correct, every writer process exited 0, and the only check that reliably notices this failure shape — a per-owner count assertion — was the one I kept rerunning until it went green. This post is about that failure shape: misattribution, which is invisible to every totals-based check, and which the flaky-test literature and the SQLite-locking literature each half-describe but don't connect.

Background: what's already well covered

Three adjacent topics are thoroughly written up, and I won't re-litigate them:

"Your flaky test might be a real bug" is established wisdom: TestRail's guide on flakes masking genuine defects, and Test Flakiness: The Silent Killer of Engineering Trust on why "it passes sometimes" must not become an accepted state.
SQLITE_BUSY despite a timeout has a canonical explainer in Bert Hubert's post (implicit transaction upgrades), and sqlite-net #778 documents lock errors when journal_mode=WAL runs before a timeout can be configured. Simon Willison's TIL covers the part many people miss: write-ahead logging (WAL) mode is persisted in the database file header, so you don't need to re-issue it per connection.
Silent data failures in pipelines — rows that stay countable while becoming wrong — are a known blind spot in the data-quality world (e.g. Catching Silent Failures in Data Pipelines).

What I could not find written anywhere: what misattribution looks like as a test-design problem — which assertions can catch it at all, and how easily the one assertion that can gets killed as a flake.

The incident

My agent tooling runs each large language model (LLM) agent as an OS process. A small Rust hook fires on every agent event and appends one row to a shared SQLite database in WAL mode. This log is the single source of truth for "which agent is alive and what is it doing." A contention drill spawns 300 writer processes — 30 running concurrently, 10 events per owner — and asserts, per owner, that all rows landed.

It usually passed. Sometimes, under parallel load, one owner had 9 rows instead of 10. Reruns in isolation: green, green. Into the books it went as "known load flake — rerun solo, two consecutive greens = pre-existing."

When I banned that label and required a mechanism, the first failing run said everything:

seat-00     | 9     <- expected 10
env-seat-00 | 1     <- ...where did YOU come from

(Owner names lightly anonymized.) The record was not lost. It was filed under a different owner. Totals: still 300. Writer exit codes: all 0. Distinct owners: 30 → 31.

Which checks can even see this?

This is the part I wish someone had written down before:

Check	Record loss	Record misattribution
Total row count	catches	blind
Writer exit codes	catches (write errors only)	blind
"Did all N processes report?"	catches	blind
Distinct-owner count	blind (unless an owner loses every row)	catches only when the wrong owner is a new one (here 30 → 31); blind when a record moves between existing owners
Per-owner row count	catches	catches

Everything in the first three rows — the checks a reasonable engineer reaches for first — stays green while ownership moves. In my system the per-owner assertion existed only because the drill happened to be written that way, and it was the assertion I kept rerunning until it passed. The tripwire was almost deleted for doing its job.

Mechanism: a fallback that lies

The hook resolves which agent it belongs to like this:

session_id ──> registry DB lookup ──ok──> seat id        (authoritative)
                     │
                    Err  (swallowed: `Err(_) => {}`)
                     ▼
              environment variable                        (known-unreliable)

The environment variable is unreliable by design in my background-agent mode — a shared daemon hands every agent the first client's environment, which is exactly why the session-ID lookup exists as the primary source. So "registry momentarily unreadable" silently degraded into "file this event under whoever the environment claims." A different, wrong owner.

The trigger for "momentarily unreadable" was one line executed on every connection open, after busy_timeout(5000) was set:

conn.pragma_update(None, "journal_mode", "WAL")   // unconditional, every open

Under a 30-process open stampede this failed with, verbatim:

SqliteFailure(Error { code: DatabaseBusy, extended_code: 5 }, Some("database is locked"))

In my runs this statement was not protected by the busy handler: failing runs completed in 3–4 seconds total, nowhere near the 5-second timeout budget, so no retrying happened on that statement. (This is a different mechanism from the implicit-transaction upgrade in Bert Hubert's post, and different from sqlite-net #778 where the timeout couldn't be set before the pragma — here it was set and didn't cover it.) As Simon Willison's TIL notes, re-issuing the pragma is unnecessary anyway once the file is in WAL mode — which is what made the collision window pure waste.

What I changed, and what each change was worth

Configuration	Result
Baseline	6.7% fail (dedicated tree, standalone) / 24% (noisy shared tree)
Fix 1 only — issue the pragma only if the DB is not already WAL	1 failure in 10 standalone runs
Fixes 1+2 — also remove the env fallback	10 consecutive greens
Fixes 1+2+3 — also bounded retry on first WAL transition	20 for 20 under deliberate parallel load

Fix 2 is the one that addresses misattribution itself: if a session ID exists but the registry read fails, don't fall back to a less trustworthy source. Degrade to a deterministic short-ID placeholder that the daemon later reconciles, and emit a marker event so the degradation is observable. During the 10-run verification, one green run recorded exactly one such marker — the degradation path fired for real and was absorbed correctly.
Fix 3 exists because I almost stopped too early. After fixes 1+2 hit 10 consecutive greens, a follow-up 20-run campaign failed its first run with a different mode: 7 of 300 writer processes exited non-zero on a brand-new database file (30 processes stampeding the very first WAL transition, which fix 1's check cannot avoid). Bounded retry — 5 attempts, linear backoff, then fail loud — closed it. If I had stopped at 10 runs, "fixed" would once again be sitting on top of a live bug.

Average write cost was 11.6 ms per process against the 5000 ms budget — this was never a throughput problem.

Takeaways

Misattribution is a distinct failure shape from loss, and it defeats totals. If a record's owner can be derived from fallible context, put at least one per-owner (or per-attributor) assertion somewhere. Nothing else in the matrix above catches it.
A fallback from an authoritative source to a less-trustworthy one is not resilience — it is a data-corruption path. Degrade to an explicit placeholder and mark the degradation. (This was not the only swallowed-error fallback in my codebase — merely the first one caught in the act.)
Decide your consecutive-green bar before you start, and overshoot it. 10 was not enough here; the 20-run campaign caught a second failure mode on run one.

Limits / untested

The journal-mode/busy-handler behavior above is what I observed on my setup (Rust + rusqlite, bundled SQLite, Windows, process-per-writer); I have not tested other bindings or platforms, and I make no claim about all SQLite statement types.
I have not measured how much production misattribution the conditional-pragma fix alone would have prevented; production contention differs from the drill's.
A continuous per-owner drift check (in the retention job, not just in tests) is designed but not built.

Repro notes

Windows 11, Rust (rusqlite, bundled SQLite), one writer process per event (300 total, 30 concurrent) appending to one WAL-mode file; busy_timeout(5000) set on every connection before any other statement.
In my runs the failure appeared only when real process startups collided on connection open. To watch misattribution live, group rows by owner and diff against the expected owner set — don't count totals.
The systems are personal tooling and not distributed; owner names above are lightly anonymized.

Drafted with my local AI pipeline; all measurements are from real runs on my machine and reproducible from the notes above.

A 110M-Parameter Model Instead of an LLM? Testing Japanese NLI Cross-Encoders on Real Conversation Logs

midnightgrep — Sun, 19 Jul 2026 04:27:41 +0000

TL;DR: I wanted a model small enough to bundle inside a memory tool (tens to hundreds of MB, CPU-only) that could judge whether two Japanese sentences make the same claim, contradict each other, or are unrelated — a Natural Language Inference (NLI) task. I benchmarked two NLI cross-encoders on 16 hand-labeled pairs written in the register of real work logs. The small Japanese model (111M params) failed paraphrase detection outright on this set (0/5). The multilingual one (279M) did better (10/16) and — with a confidence threshold — works today as a zero-false-positive contradiction detector. Neither solves "same claim" out of the box. Numbers, traps, and a possible way forward below.

Why I wanted this

I run a local, self-hosted memory system: every conversation turn with my AI assistant gets appended verbatim to SQLite. The tape never lies, but it accumulates. The obvious next layer is consolidation — merging entries that state the same fact, flagging entries that contradict each other.

The judgment at the core of consolidation is a three-way classification:

Given two sentences: same claim / contradiction / unrelated?

That's not generation. That's NLI, a task BERT-class models were solving before LLMs were cool. So instead of paying an LLM (API cost, GPU memory, or both) to babysit my logs, could a bundle-sized model (tens to hundreds of MB) do it? My constraints:

CPU only, imperceptibly light (the user should not be able to tell it's running)
Bundleable: same deployment shape as the OCR and embedding models my tool already ships (40–150MB each)
Fail-safe direction: false merges are worse than no merges. Low recall is acceptable; false positives are not.
Wall-clock time is unlimited (it's a sleep-time job)

The test set

16 hand-labeled Japanese pairs, written deliberately in the register of my actual logs — casual, technical, elliptical. Not textbook NLI sentences. Three axes:

5 paraphrase pairs (expected: entailment) — e.g. "検索ツールを構成から外した" vs "検索ツールは構成の一覧から削除された"
5 contradiction pairs — e.g. "バックアップはそのうちやろう" vs "バックアップは今すぐ実装する"
6 unrelated pairs, including 2 adversarial ones: same topic, different claims — e.g. "このツールは検索には悪くない" vs "このツールでは意味の蒸留までは厳しい". These are exactly what embedding cosine similarity gets wrong, and exactly what consolidation must not weld together.

(Tool names in the examples above are lightly anonymized; sentence structure and register are unchanged. Yes, n=16 is small — this is a smoke test to decide whether deeper investment is justified, not a paper.)

Hardware: consumer desktop CPU, single-pair inference, PyTorch, no batching.

Candidate 1: Japanese-only, 111M params

akiFQC/bert-base-japanese-v3_nli-jsnli-jnli-jsick — tohoku BERT-base v3, fine-tuned on JSNLI + JNLI + JSICK.

Trap #1 before any evaluation: the model's config.json has no id2label. You get LABEL_0/1/2 and nothing else. The mapping (0=entailment, 1=neutral, 2=contradiction) exists only in the README. If you guess the standard-but-wrong order, your accuracy numbers are garbage and you may not notice. I burned 20 minutes on this; a probe set of trivial pairs ("犬が走っている / 動物が走っている") is the fast way to pin the mapping empirically.

Results (16 pairs):

axis	correct
paraphrase (E)	0/5
contradiction (C)	2/5
unrelated (N)	5/6
total	7/16

Latency: 23.7 ms/pair (CPU, single pair, no batching).

(Counts are under the corrected label mapping — and yes, I first miscounted this table as 8/16 at 5 a.m.; recounting from raw logs before publishing is part of the job.)

On this set, the paraphrase failure is total: every single paraphrase pair — including near-trivial ones like "外した / 削除された" — collapsed to neutral. The model only caught contradictions that contained explicit negation ("使っていない" vs "毎日使っている"). A pragmatic contradiction ("そのうちやろう" vs "今すぐ実装する") sailed through as neutral.

Why: JSNLI is machine-translated SNLI, i.e. translated image captions. JNLI's pairs also derive from image captions (YJ Captions). The model has essentially never seen technical, colloquial, or elliptical Japanese. This is textbook distribution shift, and at 111M params there's no reserve of general knowledge to absorb it. The latency is spectacular — the semantics are not.

Candidate 2: multilingual, 279M params

MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 — mDeBERTa-v3-base fine-tuned on ~2.7M NLI pairs across many datasets. Japanese is mostly cross-lingual transfer.

Results (same 16 pairs):

axis	correct
paraphrase (E)	2/5
contradiction (C)	4/5
unrelated (N)	4/6
total	10/16

Latency: 389 ms/pair (CPU, single pair, no batching).

Better across the board except the two N misses (it saw contradictions that weren't there). Still weak on paraphrase, and it also missed the pragmatic "later vs now" contradiction. But here's the interesting part:

Threshold analysis. If I only act when P(contradiction) ≥ 0.95:

3 contradictions detected (at 0.976, 0.998, 0.984)
0 false positives in this set (the wrong contradiction calls scored 0.69 and 0.54 — comfortably below the bar)

That's exactly the fail-safe shape I need: a low-recall, high-precision contradiction alarm. Not a consolidator — an alarm. "This new log line disagrees with what your knowledge base says" is a useful, safe, shippable feature even when full consolidation isn't.

16× slower than the 111M model, but 389 ms on CPU is still nothing for a sleep-time job, and int8 ONNX quantization is commonly reported to buy back around 4× (I haven't measured that here).

What I concluded

These two bundle-sized NLI models did not solve same-claim detection on my logs. The core judgment of memory consolidation — "are these two sentences saying the same thing?" — is precisely the axis where both models are weakest. Hypothesis refuted, with numbers. (My embedding model had already failed this differently: cosine similarity says "same topic", which is emphatically not "same claim" — that conflation is how you weld contradictions together.)
Threshold-gated contradiction detection is usable today. High precision, low recall, zero LLM involvement, CPU-trivial. Deployed as an alarm, it fails silently — which is safe.
Read the README, not just the config. Missing id2label is a silent accuracy killer. Probe with trivial pairs before trusting any numbers.
The failure is specific, therefore fixable. The diagnosis is distribution shift, not model capacity. Which suggests:

The way forward (untested)

The one path I haven't run yet: domain distillation. Sample a few thousand sentence pairs from my actual logs, have a large LLM label them once, offline (the LLM appears at training time, never at runtime), and fine-tune the 111M Japanese model on pairs that look like my data. CPU fine-tuning of a model this size is slow but feasible, and sleep-time jobs have unlimited wall-clock budget.

Encouragingly, a paper at the Japanese NLP conference this year (ANLP 2025, Q4-2) studied domain adaptation for English-Japanese medical translation and found that supervised fine-tuning alone acquired domain knowledge about as well as continued pretraining + SFT — and that mixing in over 50% pretraining data actually degraded scores on low-frequency knowledge. Their domain isn't mine, so the caveat may or may not transfer, but it suggests I can try skipping the pretraining stage and go straight to labeled pairs. If the experiment happens, the follow-up post will have the numbers, whichever way they fall.

Repro notes

Python 3.11, transformers + torch CPU wheels + fugashi + unidic-lite (candidate 1 needs the MeCab tokenizer deps; candidate 2 needs sentencepiece)
On Windows, watch out for: pip long-path failures inside deep venv paths, and cp932 console encoding clobbering UTF-8 output (PYTHONIOENCODING=utf-8)
Both models load with AutoModelForSequenceClassification.from_pretrained(...); classify with softmax over logits of the (premise, hypothesis) pair

Drafted with my local AI pipeline; all measurements are from real runs on my machine and reproducible from the notes above. Opinions mine, benchmarks yours to check.