In a private research repo of mine there are twenty-one small programs, each with a frozen oracle: a reference implementation plus a set of probes, which are inputs paired with the answer or the refusal the program is required to produce. A generator reads those oracles and writes a contract document, so somebody reimplementing one of these programs can read what it must accept, what it must reject, and the largest magnitude anybody probes it with. That last figure exists so a reader knows where the edges are.
It published the number of digits, not the order of magnitude. A probe of exactly 10**12 came out as ~10^13.
Three of the twenty-one operations publish that figure. All three were wrong.
The repo is private, so there is nothing to link to. The figures come out of the generated documents themselves, and out of commands I re-ran while writing.
The three, and the dangerous one
A count of changed rows cannot show this defect, because the whole defect is a number being one off. So here is every figure that moved, against the literal in the probe it came from:
| operation | the literal probed | published before | after | what the probe expects |
|---|---|---|---|---|
number_names |
10**12 |
~10^13 |
~10^12 |
a refusal: a ceiling to reject at |
binary_convert |
1000003 |
~10^7 |
>10^6 |
an answer: a floor to clear |
luhn |
6011000000000004 |
~10^16 |
>10^15 |
an answer, and it is a card number rather than a magnitude |
The first row is the one that matters, and it is worth being precise about why. That probe requires the program to refuse the value, so the published figure is a ceiling: reject at or above this. Publishing ~10^13 told a reader to reject an order of magnitude too high, so a program built faithfully to the contract accepts `1012` and fails the very check the figure exists to protect.**
It misreported in the direction of the defect it exists to prevent. A figure wrong toward safety is a nuisance; this one was wrong toward the failure.
The fix also separates two things the old format ran together. When the probe value is an exact power of ten the contract now says ~10^12; when it merely sits above one it says >10^6. 6011000000000004 is a credit card number, and calling it ~10^16 invited a reader to think a round magnitude had been chosen deliberately.
The test was watching the wrong property
There was a check over this figure. It requires every published magnitude to start with ~10^.
That is a test of the format, not of the value. ~10^13 passes it as readily as ~10^12, so it could not see this defect at all, and it was green while the figure was wrong.
Worse, it made the cheap fix invisible. Subtracting one and leaving the format alone would have kept it green, and would have kept the question closed. The repair was only visible as a repair because it also changed the format to >10^6 for the two floors, which broke the format check and forced the question into the open.
So this round edits a test that exists to stop exactly that kind of edit, which needs a justification. The justification is that the widening was written down in advance by whoever wrote the test, and they left the instruction in the codebase: expect this check to fail beside the new wording, that is the regular expression reading the new text rather than a leaked value, so widen it in the same change. They were right that it would fail and wrong about how, having expected a polarity word where the round shipped a symbol. Worth noting that the instruction is not a comment beside the assertion it governs; it is the failure message of a different check sixteen lines further down, which is a fragile place to leave something load-bearing, and it is the only reason the edit is defensible rather than convenient.
Then a check was written that can actually see the defect. It re-derives the expected string from each operation's own literal and compares it against what is published. Planting the digit-count rule back in the generator, which means reintroducing the defect on purpose to confirm the check notices, turns it red, naming binary_convert ('~10^7', '>10^6'). Restoring turns it green.
That new check also carries a denominator, and the reason is a previous version of exactly this control. A check over an empty set passes, so a control has to assert that something was compared. The earlier one asserted only that the published list was non-empty, and stayed green while two of its three entries were being silently discarded before the comparison. The shipped version requires all three, which is the difference between "it ran" and "it ran on everything".
The two findings underneath
The generator could not run at all. This is a document that is supposed to be regenerated from its sources, and regenerating it was the first thing the round tried. A document whose generator has quietly stopped working has become a hand-maintained file without anybody deciding that, and every claim in it about being derived is now a claim about history.
The rule lived in two files, so the first fix reached neither artifact a reader is handed. Two separate pieces of code computed the same published figure, and repairing one left both documents unchanged. That is the failure mode where the fix is real, the diff is correct, the commit message is accurate, and nothing a reader can see has moved.
What generalises
A test that pins a format does not pin a value, and the two are easy to confuse when the format encodes the value. ~10^N looks like it is about the number. It is about the string. Anything asserting on a rendered figure is worth reading twice with one question: would this be green if the number were wrong?
The direction of an error matters more than its size. One off is nothing until you ask which way. This figure is consumed as a bound, and being one order high on a ceiling turns a protective constraint into permission.
A cheap fix that keeps every test green is the worst outcome available. Subtracting one would have been correct, quiet, and would have left a format check standing that cannot see the class of defect it appears to guard. The repair that breaks a test is the repair that gets the test fixed.
Top comments (0)