I run a small measurement pipeline. One fixed set of 16 questions, asked once a day at 04:00 UTC against two answer engines, 32 data points per run, a single entity. It has been running for months and each day's result is published through an endpoint.
Six days ago I wrote a short reference file describing that setup. The reason was mundane. I had noticed the setup being described inconsistently wherever I quoted it: two engines in one place, three in another, five endpoints in a third. So I wrote the ground truth down once, in one file, and pointed everything at it.
Today I read that file against a live run. Every number in it was wrong.
Not the structure. The structure was fine. 16 questions, 2 engines, 32 points, 04:00 UTC. That is a property of the setup and it only changes when I change the setup.
What was wrong was everything I had helpfully written next to it.
documented 02 Sep headline 28.6% direct 88.9% engine A 27.1% engine B 30.2%
measured 07 Sep headline 23.4% direct 83.3% engine A 22.9% engine B 24.0%
Those are not typos and nothing broke. They were correct on the day I wrote them. They are readings, and a reading has an expiry date that a constant does not.
Why this is not just carelessness
The file was written to solve a drift problem and it produced the next one. That is the part worth generalising.
A constant and a reading look identical in prose. Both are a name, a colon and a number. Nothing in the format tells a later reader which of the two they are looking at, so the file gets quoted as a whole, and the reading gets quoted with the same confidence as the structure. The document is most dangerous exactly when it is most trusted.
A stale reading also fails quietly. A wrong constant contradicts something else and someone eventually trips over it. A stale reading is just a number that used to be true. Nothing anywhere asserts that it should still be true, so nothing can catch it.
What I changed
Three things, none of them clever.
Split by lifetime, not by topic. The file now holds only what changes when the setup changes. Anything that moves on its own schedule was taken out.
Make the reading unquotable from the document. There is now one command that prints the current values, and nothing else prints them. If you want a number, you fetch it.
Fail closed on age. The reader checks the snapshot timestamp. Past 36 hours it stops printing values and says so instead. An old number is not a smaller number, it is an unknown one, and those two have to look different at the point of use.
The rule
If a value in your documentation would be different tomorrow without anyone editing the file, it is not documentation. It is a cached read with no invalidation and no timestamp, sitting in the exact place people go when they want to be sure.
Keep the structure in the file. Keep the reading behind a call.
Top comments (0)