I shipped a set of free investing calculators. Each one has a card on the index page showing a worked example: given these inputs, here is the answer.
Last week I found that one of those cards said $2,282 while the calculator underneath it produced $1,160.
Nobody had complained. Every test passed. TypeScript was happy. The number had been wrong for weeks.
Why this kind of bug is invisible
The calculation lives in a React component. The published example lives somewhere else entirely, in a plain manifest file that feeds the index cards.
Those two files have no relationship. Neither imports the other. So when I changed the model to let uninvested cash earn interest, the component's output moved and the manifest quietly did not.
This is not a code bug. The code was correct the whole time. It is a claim bug: the site asserted something that was no longer true, and no type system checks assertions about arithmetic.
I suspect this is extremely common. Most products publish numbers somewhere, in a pricing table, a docs example, a README, a marketing card, and almost nobody has a test that says "the thing we told people is still what the software does."
Why a unit test would not have caught it
The obvious fix is a unit test. But a unit test written against the same implementation only encodes whatever that implementation currently does. If I had asserted that the function returns 1,160, I would have been asserting that the code does what the code does. Useful for catching regressions, useless for catching a wrong published claim.
What I actually needed to check was different: does the number we publish match the number the tool produces? That needs a second, independent implementation, ideally written by someone who read the description rather than the source.
I do not have that someone. So I used a different language as a rough substitute for a different brain.
What I built
A small script in Python that reimplements every calculator from scratch, then compares its results against the figures published on the site.
Three decisions made it worth the afternoon.
It reads the inputs out of the component source. Rather than keeping its own copy of each calculator's default values, the script parses them out of the React files directly. If I change a default, the script picks it up automatically and the expected published figure fails. There is no second copy of the inputs to forget about, which is the failure mode that created the original bug.
It reimplements the maths from the description, not the code. I worked from what each tool is supposed to do rather than translating the TypeScript line by line. Translating would have carried any logical error straight across into the test.
Where a closed form exists, it checks both. The compound interest tool loops month by month. There is also a standard annuity formula that produces the same answer in one step. The script runs both and requires them to agree to nine decimal places. Two different routes arriving at the same number is much stronger evidence than one route agreeing with itself.
What it found
102 checks. Four defects.
The $2,282 that should have been $1,160.
The same card describing the result as "ahead after 11 years" on a setup that runs for 12 months. Two separate errors in one sentence, both invisible because nobody re-reads a card they wrote months ago.
An FAQ on the Coast FIRE calculator claiming that retiring at 55 instead of 65 raises the balance you need at 40 from $276,400 to $762,500. The real figure is $543,700. I had built a correct lookup table of these values and then, two paragraphs later, transcribed the wrong row into prose. $762,500 is the ten-year horizon. Retiring at 55 when you are 40 is fifteen years.
And two worked examples that quietly omitted inputs the tool actually uses. One said "$500 a month for 30 years" while the calculator also starts with $10,000 already invested, which makes the stated result impossible for a reader to reproduce.
The third one is the one that bothers me. It was not stale data or a refactor. I did the arithmetic correctly, wrote the correct numbers into a table, and then quoted the wrong cell in a sentence underneath it.
Extending it past the outputs
Once the script existed, the cheap win was pointing it at every number in the surrounding prose, not just the calculator results.
The dividend reinvestment calculator has an explainer noting that over twenty years the annual income grows 5.9 times, and that this decomposes into the dividend per share growing 3.2 times multiplied by the share count growing 1.85 times. It is a satisfying line. It is also three numbers and a multiplication, every part of which can be wrong independently.
So the script checks all three values, and then separately checks that multiplying the first two actually produces the third. That last check is about the sentence being internally consistent rather than about any single figure being right. If one number drifts, the claim stops holding together and the test fails. No human proofreader catches that on the fortieth read.
What is worth stealing
Test the claims you publish, not only your functions. Any number in marketing copy, a docs example or a card is an assertion your software should be able to verify.
Read the inputs from the source of truth. A test carrying its own copy of the defaults passes happily forever after the defaults change.
Use a genuinely separate implementation for anything with real arithmetic. A different language, a closed form, even a spreadsheet. Checking an implementation against itself only proves it is consistent with itself.
Compare with tolerances rather than exact equality. Floating point plus values displayed rounded means you want a small relative allowance, tight for derived maths and looser for anything shown to the nearest dollar.
The whole thing runs in under a second and is roughly two hundred lines. It has already caught four things that were live on pages real people read.
If you want to see what it is checking, the calculators are all here, free and with no account. But the transferable part is the idea rather than the tools: the numbers you publish are part of your product, and almost nobody tests them.
Top comments (0)