Seven times my test harness reported success while measuring nothing

Last week a stranger reproduced my benchmarks. Over four days he found six defects. Every single one of them had been reporting success.

That is the part worth writing down. Not that I had bugs — everyone has bugs. That my instrumentation, six separate times, returned a number that looked fine and meant nothing. A crash is loud. A harness that prints 86.1% when it measured the wrong thing is not, and I published every one of those numbers.

Here they are, in the order they were found.

1. except Exception: continue

The ablation loader walked a Spider 2.0 checkout and read each table's JSON:

for f in path.rglob("*.json"):
    try:
        d = json.loads(f.read_text())
    except Exception:
        continue

On Windows, open() refuses a path over 260 characters. 2,868 of the 7,892 paths in that checkout are over 260 characters. The except caught FileNotFoundError and counted every one of those tables as a table that does not exist.

So the benchmark ran against a database missing roughly 39% of its tables, and reported 86.1% recall at k=20. A smaller schema is an easier schema. The number was not wrong because the retrieval was bad; it was wrong because the corpus was a third smaller than the corpus I said it was.

The detail that makes this worse: the shortfall depends on where you cloned the repo. A 182-character root gives 2,868 unreadable files. A 183-character root gives 3,056. The benchmark's result was a function of the reviewer's directory layout, and nothing in the output said so.

2. A comment that had been lying for a day and a half

# Bounded, and only ever additive -- it cannot displace a ranked pick.

Forty lines below it:

if len(chosen) >= top_k:
    for i in range(len(chosen) - 1, -1, -1):
        if chosen[i].reason not in ("pinned", "covers"):
            taken.discard(chosen[i].doc.qname); del chosen[i]; break

It displaces a ranked pick. It has since a commit I made a day and a half earlier. I had quoted the comment in a public reply as though it were the behaviour, because I read the comment instead of the code directly above and below it.

Comments do not have tests. This one shipped in three releases.

3. An enum missing a value the code writes

Scored.reason documented five values: hybrid | vector | lexical | fk | pinned. The selector writes a sixth, covers. Anyone branching on that field — and the whole point of the field is that you branch on it — would silently fall through on one case in six.

4. A check with power zero by construction

He claimed 52 of 52 questions had a step where the returned-set size decreases as K grows. My check returned 0 of 52, and I was one keystroke from posting that as a contradiction.

My check compared returned(K) against returned(K−1). Top-K picks are nested and both expansion passes are unions, so returned cannot decrease in K. The test could never fire. It was not a disagreement with his result; it was a test with no power, returning the only answer it was capable of returning.

Comparing the overages instead of the totals: 52 of 52, exactly as he had it.

This is the one that still bothers me. A test that returns 0 looks like evidence of absence. It is indistinguishable, from the outside, from a test that measured something.

5. The aggregate that was never printed

The held-out evaluation built six fixtures. One of them declares three extra schemas and puts four objects in them — three tables all named account, plus a view joining across two. That name collision is most of what the fixture exists to test.

The harness opened a bare SQLite connection with no ATTACH, so executescript raised unknown database "billing" on the first of those statements and the run died there. After the five per-schema rows had printed. Before the TUNE / HELD OUT / OVERALL lines.

The file exists to produce an aggregate. It had never printed one. Five plausible rows of output scrolled past every time and I read them as the run having worked.

6. The fix that turned a crash into a silence

This is the one I would most like you to take away.

The obvious repair is to ATTACH DATABASE ':memory:' before running the DDL. It works. The script completes, the harness prints its aggregate, exit code 0.

In-memory databases are dropped when the connection closes. The harness closes that connection and bootstrap opens a new one. So:

256 objects indexed, all in `main`
objects named `account`: none

The three same-named tables and the cross-schema view — the hardest case in the fixture, the reason the fixture is there — are still absent. The difference is that now nothing crashes. The repair converted a loud failure into a quiet one, and both of us verified the repair by reading a number that the repair had not changed.

File-backed databases attached on a connect listener, with the schema list passed to bootstrap, gives 260 objects — main 256, billing 2, crm 1, sec 1. Recall did not move. The measurement did.

And one the reviewer didn't find

While writing this I checked the benchmark script that guards my README. Its last line is:

bench OK: README claims hold

The gate above that line checks recall thresholds and a 70% token-reduction floor. It does not check the token count the README prints (the README said 2,583; the script prints 2,812). It does not check the 260-object row at all — that schema is never built by that script. It asserted that the README holds, while not reading most of the README.

Seventh.

The pattern

Every one of these reported success or absence for a reason unrelated to the thing being measured.

• an exception handler that turned missing into does not exist
• a comment that described code that had changed
• an enum that described code that had grown
• a comparison that was monotone and therefore constant
• a crash sequenced after the output that made it look like a run
• a fix that replaced a crash with an absence
• a gate whose message claimed more than its assertions

None of these produce an error. All of them produce a number. That is the shape: the failure mode of a measurement is not a wrong answer, it is a plausible one.

The test I have taken from him is mechanical, and I recommend it. Sweep the apparatus. Change the K grid, the predicate, the clamp, the domain, the platform. If the number moves, you measured your apparatus. If it holds, you may publish it.

It caught all seven above. It also caught a number I had sent him two days ago: I reported 22 questions whose ceiling rises on a dense sweep; he measured 24. He was right — I had included a K value that is not in the published grid. Sweeping four grids gives 22, 23, 24, 29. The count is a property of the grid, not of the system, so the curve is what gets published and the count does not.

Why I am writing this instead of quietly fixing it

Because the numbers were public, and a correction that is not as visible as the claim is not a correction.

And because the thing that produced all seven fixes was not my discipline. It was a stranger on the internet with no stake in the project, who reproduced every figure before disagreeing with any of it, and who kept going for four days. I have never had review like that from a paid reviewer.

If you maintain something and somebody turns up to check your arithmetic: that is the most valuable thing that will happen to your project this year. Give them everything they ask for.

---

The library is schemagate — identity-scoped schema selection for text-to-SQL. Every figure above is reproducible from the repo; the benchmark file records what was withdrawn as well as what held.