One day in August I stopped reading my code and started reading my terms of service. Not as a contract, but as a specification: every sentence in it that describes behaviour is a claim about the software. Fifteen of them did not hold.
Day 7
On day 7 the dashboard required proof of work. My own terms describe the check-in on that day as voluntary.
So anyone who had read the contract and believed it would have uploaded nothing — and walked straight into the consequence designed for people who do nothing at all. The product would have punished someone for trusting a commitment I made.
Nobody was affected. As of August 2026 no paying customer has ever started a sprint in this product, and there is no testimonial. That is not an excuse — it is the reason any of this can be written down. All fifteen findings hit nobody, because nobody is in there yet. There is no better moment coming.
Why I looked
This was not a code review. I suspected my external communication and my product had drifted apart, and I wanted to know by how much. So I took the only text that is legally binding and treated it as a requirements list.
Four steps, short enough to copy:
- Read the terms and the privacy policy as a specification, not as legalese. Mark every sentence that makes a statement about behaviour — "X is recommended but not mandatory", "a suspension occurs when …", "we typically reply within …".
- For each marked sentence, find the place in the product that executes it. Not the place that mentions it — the place that enforces it.
- Search for those same sentences across everything customer-facing: landing page, FAQ, checkout, emails, the machine-readable surfaces. A sentence rarely exists once.
- For every finding, record the direction: does the code deviate in the user's favour or against them?
Step 4 is the one I almost skipped, and it is the most important. It separates a legal problem from a fairness problem — and it explains why half of the findings survived as long as they did.
Three classes, not a bug list
Class 1 — the contract says A, the code does B
Day 7 belongs here. So does a second case: the product suspended a participant after two skipped days, and my contract did not provide for that — it enumerates the grounds for suspension exhaustively, and this one was not among them.
That is explicitly not a liability finding, and the entire difference is the direction: the code was stricter than the contract. The dangerous case is the mirror image — a contract promising something the software does not deliver. Same size of deviation, two completely different consequences.
Class 2 — the marketing says A, the contract says B
The leading example: in four places I advertised a response time that was shorter than the one I had contractually committed to. Nobody complains about a promise that is too generous. The error is silent by construction — there is no complainant for an oversized promise, right up until the one day you fail to keep it.
I cannot quote that sentence here. It now sits on a blocklist that runs across the repository on every build. I tried while writing, and the check went red. That is not a mishap, it is the intended behaviour, and it is the most honest evidence I can offer for the rest of this post.
The same class holds the sentence I liked least. In six places it said the participant receives a personal email. What went out was a template. The sentence was not a lie when it was written — it became one the moment I automated the sending. There were exactly two honest resolutions: write the email by hand, or stop promising it. Today the sentence appears in none of those six places.
Class 3 — statements nothing can structurally contradict
Two code paths could never fire. Not broken — unreachable. No test went red, because no test exists that asserts "this line is ever reached". No error was raised, because a path that never runs never fails.
This is the expensive class. It is indistinguishable from working software using every tool I have. I found neither by reading; I found them by running the whole chain end to end, once.
Fifteen
I found eleven instances by hand that working day. Three more turned up while proofreading the list — that is, inside the document I had created in order to close the matter. At fourteen I considered it complete.
That same evening I built a check that fails the build when a retired sentence reappears, and ran it for the first time. It immediately reported a fifteenth occurrence — same class, same wording, in a file neither I nor the proofreading pass had looked at.
Fifteen is not a decay rate. It is what a first measurement looks like. Fifteen instances of one class, in a product with no paying sprint behind it, all corrected the same day and verifiable. And the last of them was found not by me but by the first tool capable of looking for it. If you run this yourself and come out at zero, you probably did not measure — you confirmed.
The pattern
A comment has no assertion. A README has no CI. Your terms of service are a test suite nobody runs.
That is the actual discovery, and it is not that fifteen statements were wrong. It is that in a product like this two specifications exist and only one of them is tested — and the untestable one is the one that binds. The code has tests, linters, types, a pipeline and a reviewer. The text that governs in a dispute has none of it. It has an author and a date.
So the class does not come from carelessness. It comes from the same rule existing in four, six, eleven places, and none of them stating how many there are. Every change is an n-place operation with an unknown n. Someone changing a number from 24 to 48 is not asking a positioning question — so they do not ask anyone.
Two findings, then onward
Kleven et al. (2011) ran a randomised audit experiment with more than 40,000 Danish taxpayers. Evasion was close to zero on third-party-reported income and substantial on self-reported income. Same people, same incentives; the only difference is whether somebody else reports the figure. Scheel, Schijen and Lakens (2021) compared 152 standard psychology studies with 71 Registered Reports: 96 per cent positive results against 44. Once the hypothesis is fixed before the result — once it can fail — the success rate halves.
Both describe the same property, once in money and once in science: a statement that nothing can contradict stops behaving like a statement. Documentation is self-reported by default. That is not a question of character — it is a property of the reporting arrangement.
What I actually changed
The check is deliberately small — a gate that mostly reports noise gets ignored within two weeks. It checks wording, not correctness. It knows strings, not meaning, so it cannot tell a claim from its negation. And absence is invisible to it: a promise that quietly disappears produces no match. Of my fifteen cases it would have caught roughly a third — the cheap ones. One of them it actually did catch, on its very first run. It would have let the four most expensive ones through.
Which is the honest limit of the whole exercise. The expensive class needed a dry run of the entire chain, and a dry run is manual work you do once per chain, not once per day.
The twenty-minute version
Take the longest document that describes your product and is not code: terms, README, landing page, onboarding email. Mark every sentence that makes a statement about how your software behaves. Take three of them and find the place that executes them. Record the direction for each.
And the condition without which the rest is worthless: write down beforehand which claim you are checking and what result would refute your assumption. Otherwise you will end up confirming what you expected — that is the distance between 96 and 44 per cent, and it does not come from dishonesty.
Which fixes my next check, and it can go wrong: in a few weeks I will look at whether the fifteen corrected sentences still hold. If more than two of them have drifted again, then the build check was the wrong answer — the problem would be the number of places rather than the supervision of them, and the correct response would be to delete carriers instead of monitoring them.
Disclosure, so it isn't buried: I build a product in this space, so I'm not neutral on the conclusion. What I can evidence is a self-test — I ran my own 21-day sprint as a participant with the goal frozen before day one, and missed one of four criteria. What I can't evidence is that it works for anyone else: no paying customer has completed a sprint and there's no testimonial. The two studies above are neutral.
Longer version with the full source list on my own site: mvpbuilder.io/blog/terms-of-service-test-suite
Top comments (0)