Last week three checks in our repos reported zero problems, and all three were wrong. Each one had borrowed something from the code it was supposed to check: a default scope, a layout function, a definition. When the code was wrong, the check agreed with it.
This follows my post about failures the shell hid behind exit code 0. These are failures the checks themselves couldn't see.
At the end I'll ask where the expected values in your tests come from, because that's where all three of ours went wrong.
A flag nobody read
One of our sites has a checker that flags numbers in product descriptions that can't be traced to six verified fields in our data. Its --build flag was meant to scan every built page. Nothing in main ever read it.
With or without the flag, the checker scanned the same file: the product data, 856 records. It printed items=856 bad_pages=0, and we read that as "build: 0". The label said pages, but the variable behind it was bad_items. Category and guide pages had never been in scope.
On October 23 we went through that category and guide text and took out the figures we couldn't source: nine places across four files, plus a generator that would have put one sentence back. Three days later, work on one guide page turned up more: a hard-coded price table, and two organizations credited with test findings neither had published.
On the 23rd we also blocked the flag instead of implementing it (comments translated, message shortened):
if args.build:
# Declared but never implemented. Left alone, the data-scan count reads as "build: 0".
sys.stdout.write("--build doesn't exist yet. Don't produce a 0 with this flag.\n")
return 2
I'd rather have a flag that refuses to run than one that hands back a zero it never earned.
A check that used the app's layout
One of our apps writes month-by-month readings and shows them as a calendar card, one row per month. A change on October 23 moved a paragraph meant for a quiet month out of its row and up to the top of the list. QA drew 300 random birth charts, read each one on 2 dates through 3 answer paths, and in the year-ahead reading found the paragraph out of place in all 1,800 readings. Not most. All. The other two calendar readings had 0, and so did the build before the change.
Two checks missed it anyway. The UI test runner saved a screenshot that showed the paragraph in the wrong place. Its check, named for the twelve-month section, only looked for the section's heading, so it passed. The independent sweep put text into months by calling the app's own monthsLayout function. Wherever the app put the paragraph, the sweep put it too.
The sweep's "independent" table of advice directions had the same problem. Of the 318 app advice templates QA could match to it, 317 pointed the same way as the app's own table. That's 99.7%, or in the QA report's words, the app's judgment written down twice.
A test with the code's definition
Another app suggests Korean names and scores them with a traditional rule. You add up the brush strokes of the characters and check four sums against a list of numbers considered unlucky. In our data, 128 characters have two accepted stroke counts, depending on the source.
The rule we shipped: a name is recommended only if it's clean under both counts. The code checked two cases, all characters on their main count or all on their alternate count, and the test, written from the same idea of "both counts", checked exactly those two. All 463 tests passed.
A full name with two such characters has four combinations, not two. For one surname, 6 of the 30 recommended names came out unlucky under at least one of our three sources. The test meant to guard the rule used the code's own definition, and it covered only three surnames.
The fix checks every combination of every character's values. The new test doesn't borrow the code's definition: it reads the source table directly and checks every combination for all 135 single-character surnames.
What the three had in common
Each check took something from the code under test, so it couldn't disagree with the code.
We wrote down three rules after this:
- An independent check may call the app to draw the screen, but it judges with its own code and borrows no layout or judgment function. Screen order comes from the rendered text. A judgment table should be written by a person and compared against the app, not derived from it. We haven't fully met that last part yet.
- A check prints what it actually looked at, so a zero comes with its denominator.
- A new check runs first against a build we know is broken, and we write down how many reds it finds. In the calendar app, the last two new test files went 6 of 6 and 3 of 4 red on the build before their fix. If a check finds none there, it isn't a check yet.
A question
When you write the expected value for a test, where does it come from? A table someone worked out by hand, a second implementation, or the same helper the code uses? And which of those has let a bug through for you?
Sources
- Python docs, argparse actions (
store_truestores True when the flag is given, False otherwise): https://docs.python.org/3/library/argparse.html#action, accessed 2026-10-26 - Everything else: our QA reports, lessons log and commit history from October 22 and 23.
This post was drafted with AI help. The numbers and facts come from my own logs and repo.
Top comments (3)
The "both counts" example is the one that stuck with me. You could write a completely separate test implementation and still make the same mistake because you both read the rule the same way. Writing out the four combinations for one name by hand would give the test an expected answer that didn't come from either implementation.
That's the trap with a second implementation. It can share the misreading. Here the code and the test both read "both counts" as 2 cases.
The fix has one part that came from neither. The 6 names QA flagged are now fixed cases in the test: none of them may be recommended, and one has to show as "depends on the count." QA found them by running the app and checking each recommended name's 4 sums against the source stroke counts, before any fix existed, so the expected answer came from the sources and not from our code. Under one source's counts, that one name lands 2 of its 4 sums on 20 and 26. Both count as unlucky in that naming system.
The sweep over all 135 surnames is still a second implementation. Your point applies to it. Those 6 names don't depend on our reading of the rule.
Do you keep hand-worked cases in the same file as generated ones, or apart?
The borrowed-definition pattern is the uncomfortable one, because the check still runs, still produces a number, and still looks independent in CI. A zero from a checker that shares scope, layout, or definitions with the thing it checks is not evidence — it is agreement by construction. I liked the refusal fix for the unimplemented flag for exactly that reason: a check that cannot do the work should fail loudly rather than return a clean zero, because a false zero does not just miss the problem, it actively closes the investigation.