Set membership is not pairing: a property test that was green for the exact bug it was written to catch

I have a small board that tracks 16 LLMs across 5 labs on a frozen 35-task suite. The charts are the easy part. The paragraph above them is the part I did not trust, because I had written that kind of paragraph by hand twice and both times it went false — true when I typed it, falsified by a later run.

So I stopped typing it. The paragraph is generated now. Each sentence is a claim: a predicate over the current numbers plus a renderer, returning a string or None. A claim whose predicate stops holding is dropped, not reworded. The generator can be silent. It can't be wrong.

That second sentence is the part I got wrong.

The sentence that lied

One claim compares a lab's cheaper models against that same lab's flagship. On the live board it produced this:

Google's cheaper tiers match or beat its own flagship — Gemini 3.5 Flash at 100% and Gemini 3.1 Flash-Lite at 95%, against Gemini 3.1 Pro at 95%

Correct. But the first version of that renderer collected the winners, took the highest score among them, and printed one number for the group:

scores = f"{_names(winners)} at {_pct(max(r.acc for r in winners))}"

Which reads: Gemini 3.5 Flash and Gemini 3.1 Flash-Lite at 100%. Flash-Lite scored 95%. The sentence put a real number next to a model that did not earn it.

Naming several models and then printing one number is not a formatting choice. It is a claim about all of them.

The test that should have caught it

I had written a property test for exactly this class of bug. It asserted that every percentage appearing in the prose was a score some model actually got:

real = {f"{p[-1]['acc'] * 100:.0f}%" for k, p in m["series"].items()}
printed = set(re.findall(r"\d+%", out["text"]))
assert printed <= real

It was green.

It is green because the assertion is true. 100% was a real score — Gemini 3.5 Flash got it. The test asked "does this number exist in the data?" The bug was "is this number attached to the right model?" Set membership where pairing was required.

The replacement checks the pairing:

_PAIR = re.compile(r"\b([AB] (?:Big|Small|Tiny)) at (\d+)%")

for label, printed in _PAIR.findall(out["text"]):
    assert f"{actual[label] * 100:.0f}" == printed

The fixture was wrong too

This is the part I would have missed if I had stopped at the test.

A fixture with one cheap model per lab cannot reach the failing state — with a single cheap model per group, a group is always unanimous, so a renderer printing rows[0]'s score is always right. That was the shape I started from, and I fixed it before the first commit, so take this part on my word rather than on the history. The property test was not weak because of its assertion alone. It was weak because the data it ran against could not produce a disagreement.

The fixture now gives LabA two cheaper tiers under one flagship, which is the shape Google actually has (Pro / Flash / Flash-Lite):

REGISTRY = [
    {"id": "lab-a:big",   "label": "A Big",   "group": "LabA", "tier": "flagship"},
    {"id": "lab-a:small", "label": "A Small", "group": "LabA", "tier": "mini"},
    {"id": "lab-a:tiny",  "label": "A Tiny",  "group": "LabA", "tier": "nano"},
    ...
]

and the baseline board puts those two cheap tiers at different scores, both at or above their flagship.

What it cost

Eight mutants run against the new guards. Three survived the first time — the tie guard, the single-flagship rule, and a float-truncation fix — and each one needed a test written before it was really covered. The repair was less covered than it felt.

The float one is my favourite, because it is a bug in the code whose entire job was to never overstate a number. Percentages truncate rather than round, so a model on 99.6% can never print as "100%" — a perfect score it did not get, in the one place a reader checks hardest. But:

# The epsilon is not decoration: 0.58 * 100 is 57.99999999999999 in binary
# floating point, so a bare floor prints a measured 58% as "57%". [...]
return f"{math.floor(value * 100 + 1e-9)}%"

A guard against overstating had started understating instead. That is still wrong prose about real data — just wrong in the flattering direction, which is the direction you don't check.

The suite is 125 tests now, stdlib only, 0.04s. It was 91 at the fix commit, 74 when the generator first landed, and 22 before the generator existed.

What I'd do differently

Ask what state would make this assertion fail, and name it out loud. I could not have named it for the set test — not because I hadn't thought about it, but because the fixture made it unreachable and I never noticed that the two facts were connected. An assertion whose failing state you cannot describe is decoration.

Mutate the code, not just the inputs. Property tests over generated data feel rigorous enough to skip this. Three of eight mutants survived my new, careful, adversarially-reviewed guards. That is the number that convinced me.

Assume the repair is uncovered. The fix arrives believing in itself. It has no tests yet.

And one more, which I found while writing this. The README badge for that repo says tests-91. It is 125. Four screens further down, the quickstart says # 22 tests — a badge and its own README disagreeing, both hand-typed, both true on the day they were written and false since. That is the identical failure mode the generator exists to eliminate, sitting in the repo that hosts the generator.

The verification was broken, not the code. It usually is.

Repo: github.com/egnaro9/model-drift — the broken test and the real one are both in tests/test_narrative.py.
Portfolio: egnaro9.github.io