The check I added to my build last week has printed 0 arithmetic errors on every single run. Then I scrolled a live page and read one row of a conversion table:
C30 concrete = 30 MPa = 30 N/mm² = 300 kgf/cm²
30 MPa is 305.9 kgf/cm². That row is off by 1.93%, and the check that exists specifically to catch wrong numbers had waved it through.
Not a regex bug. Not a missing test case. One line of tolerance arithmetic — the part everybody writes in thirty seconds and never looks at again.
What the check actually does
The site publishes 242 pages in each of two languages, about 2.7 million characters of visible text. Buried in that prose are sentences like 1 in = 25.4 mm and 1 ft³ = 0.0283168 m³. Unit conversions are the one class of statement where correctness is decided by arithmetic alone — no editor, no source, no judgement. Multiply, compare, done.
So the check pulls every number unit = number unit out of the rendered HTML, converts both sides to SI, and compares:
FAC = {"mm": 1e-3, "m": 1.0, "in": 0.0254, "ft": 0.3048, "ft³": 0.028316846592, ...}
calc = num_a * FAC[unit_a] / FAC[unit_b]
if abs(num_b - calc) > tolerance(num_b):
flag(...)
Everything hinges on that last function, and the obvious choice is a percentage:
tolerance = lambda printed: abs(printed) * 0.01 # ±1%, "close enough"
It reads as reasonable. It is wrong in both directions at once, and the two failures are not symmetric — one is loud and one is silent.
A percentage is the wrong shape
Run the same 131 equations under three rules:
| Rule | Flagged | Correct values rejected | Real errors caught |
|---|---|---|---|
| ±1% of the value | 1 | 0 | 1 |
| ±0.1% of the value | 7 | 6 | 1 |
| half of the last printed digit | 0 | 0 | 0 |
The ±0.1% column is the interesting one. Six of the seven things it flagged are correct:
3/4 in = 19 mm (true 19.05)
1.5 in = 38 mm (true 38.10)
3 in = 76 mm (true 76.20)
80 ft³ = 2.96 yd³ (true 2.96296)
4 in = 0.333 ft (true 0.333333)
Those are not errors. They are a table rounded to whole millimetres, which is what a table of construction dimensions is supposed to look like. A relative tolerance treats precision as a property of the magnitude: bigger number, more slack. Printed precision does not work that way. It is a property of how the author typed it.
The same 1% window means wildly different things depending on where you apply it:
| Equation | 1% of it | Half the last printed digit | Ratio |
|---|---|---|---|
1 lb = 453.59237 g |
4.54 g | 0.000005 g | 907,094× |
1 ft³ = 0.0283168 m³ |
0.00028 m³ | 0.00000005 m³ | 5,663× |
1 in = 25.4 mm |
0.254 mm | 0.05 mm | 5× |
On the first row, a 1% tolerance will happily accept an error of four and a half grams on a value that was printed to eight significant figures. Across the corpus, the median 1% window is 5× the last printed digit; the 90th percentile is 13,788×; 58 of the 135 equations get a window at least ten times too wide. That is the silent failure — nothing is flagged, so nothing looks wrong.
And the "last digit" rule has a hole of its own
So derive the tolerance from the printed form: half of the last digit shown. 25.4 allows 0.05, 0.0283168 allows 5×10⁻⁸. Write more digits, you have to be more correct. That rule flagged nothing in the corpus, which felt like a win until it also failed to flag the 300 kgf/cm² row.
Because 300 is ambiguous. It can mean 300 ± 50, or 3.0×10² ± 5. Nothing in the string tells you which, and the rule I wrote assumes the worst case:
if "." in printed:
decimals = len(printed.split(".")[1])
return 0.5 * 10 ** -decimals
sig = len(printed.rstrip("0")) or 1
return 0.5 * 10 ** (len(printed) - sig)
For 1000, that returns ±500. Thirty of the 135 equations end in a zero, so their tolerance is enormous — 1 m = 1000 mm would pass as 1500, and 300 passes as 305.9. The strictest-looking rule in the file is the one with the widest hole.
There is no clever fix here. Printed precision is a claim, and a bare trailing zero declines to make it. The workable response is to stop guessing: either the copy prints significant figures explicitly (3.0×10², or 300.0), or the value gets queued for a human instead of counted as verified. My checker now reports three buckets — verified, failed, undecidable — and I look at the third. It is 30 rows. It takes ten minutes.
Four ways the parser lies, and how much it costs
The tolerance argument is the intellectual part. The parser is where you lose coverage without noticing, because every fix for a false positive is also a fix that quietly stops checking something.
I broke the checker on purpose, four ways, against the same 484 pages:
| Broken on purpose | Verified | Unverifiable | False errors |
|---|---|---|---|
| Correct | 131 | 10 | 0 |
| Unit table sorted alphabetically, not by length | 115 | 26 | 0 |
No fraction support (3/4 in) |
129 | 10 | 0 |
| No "deliberate counter-example" filter | 131 | 10 | 2 |
Unit ordering. re tries alternatives left to right, so with m before mm, the string 25.4 mm matches 25.4 m and the rest is noise. Sixteen equations stop being checked and the error list stays empty. Sort your unit table by descending length. This one is invisible in review and invisible in CI unless you assert the count.
Fractions. 3/4 in = 19 mm is a quarter-inch value written the way a carpenter writes it. Naively, 4 in = 19 mm is off by 101.6 mm and you get a spectacular false positive. The guard that prevents it — (?<![\w./]) — also means the equation never matches at all, so it silently stops being verified. Two equations in my corpus are in that state.
Deliberate counter-examples. One page teaches the classic mistake: "people assume 1 m² = 100 cm²". It is correct content and a wrong equation. A checker without a negative-context filter reports two errors (English and Chinese) on a page that is doing its job.
Cross-family equations. 1 m³ = 1 t is true for water and false for concrete. Ten equations in the corpus relate volume to mass; none of them can be verified without knowing the material, so they are listed for reading, not scoring.
The lesson I keep re-learning: assert the number of checks, not just the number of failures. A linter that checks nothing passes everything. Mine now fails the build if the verified count drops more than 20% between runs, which is how I noticed the alphabetical-sort experiment instead of shipping it.
The two errors it did catch
Both were found in the first week, both in the same family — precision drifting between two places that are supposed to agree.
A table cell where three-quarter inch had been rounded to 75 mm, while the cells directly above and below it were rounded to 19 and 38. Same table, three different rounding conventions, one cell wrong by the width of a millimetre.
And an intermediate value printed as 0.771 (true 0.7715) that the next formula in the chain multiplied as if it were 0.7715 — the printed derivation and the computed result disagreed in the third digit. Neither error is visible to a reader skimming for plausibility. Both are trivially visible to a machine that recomputes.
That is the whole case for this kind of check: not that it is clever, but that it is exhaustive on the one class of claim where exhaustiveness is cheap.
Running it on your own docs
You do not need my unit table — you need four things:
- The rendered output, not the templates. Half of my equations only exist after variable interpolation, and the HTML tags sit between the number and its unit in ways the source never shows.
- A unit table sorted by string length, with every unit's SI factor.
- A tolerance derived from the printed form, plus an explicit "undecidable" bucket for trailing zeros and for anything cross-family.
-
A count assertion in CI.
verified < previous * 0.8should fail the build. This is the part that catches the parser silently going blind.
The script is about 120 lines of Python, no dependencies, and runs in under a second over 2.7 million characters.
I maintain a set of construction-quantity calculators, which is why my prose is unusually full of conversion factors — the unit conversion table is the page that generated the most equations, and the one where a wrong digit costs somebody a truckload of concrete. But the pattern is not specific to construction. Any documentation with prices, benchmarks, tolerances, or version-dated numbers in it has this property: the numbers are testable, and almost nobody tests them.
The 0 arithmetic errors line in my build log was not evidence of correctness. It was evidence that the check ran. Those are different claims, and only one of them is worth putting in a build log.
Top comments (0)