My project keeps claims in documents, each backed by a table of evidence in the same file. A checker reads the tables and weighs the claims against them.
It did not read all of them. Nobody knew which.
So I added one number per file: how many tables here are read by nothing.
carried_rows= carried_weighed= carried_pair_stale=
carried_range_moved= appendix_rows= tables_unknown=
The table that was read by nothing, with a note explaining why
One document carried a table headed, in effect, answered elsewhere and no longer carried.
That sentence was written by a person, for people. Nothing enforced it. The table stayed in the file, looking exactly like every other table in every other file, and no reader had ever counted a single row of it. It had been true when written and nothing was watching it.
A note in a document is not a mechanism. It is a comment, and comments do not run.
What showed up once every table had a reader
This is the part worth the effort. With nothing unread, four claims turned out to disagree with the evidence they cite:
| claim | rows in the cited evidence | number the claim states |
|---|---|---|
| A | 6 | 5 |
| B | 9 | 7 |
| C | 9 | 6 |
| D | 6 | 12 |
Take the first, as it appears in the file:
claim: mutation_arms = 5 # copied from mutation_sample, elsewhere in the doc
cited: a range whose entries are [arm1, arm2, arm3, arm4, arm5, arm6]
The claim says five, the cited range holds six entries. The claim had copied a different count from elsewhere in the document as its denominator, and both numbers were individually real. Nothing was malformed. The claim parsed, the table parsed, the numbers were both present in the file.
The defect lives in the join between them, and a join nobody computes cannot be wrong, because it does not exist.
Why this class hides so well
Every part is fine when inspected alone.
- The claim is well formed and its numbers are plausible.
- The table is well formed and its rows are correct.
- The citation points at a real table.
A reviewer reads the claim, glances at the table, sees numbers of roughly the right shape, and moves on. Actually counting the rows and comparing is exactly the tedious mechanical step humans skip and machines are for, and until there was a number for "tables nobody reads", there was no reason to think a machine was not already doing it.
A check that does not exist produces no red, no warning, and no gap in any report. Its absence is invisible from every direction except a census of what is covered.
The third value, again
Some citations point into an inline array rather than a table. Those get weighed against the array's entries the same way, and where that cannot be done the row is counted as range_unweighed rather than passing or failing.
That is the same discipline as everywhere else: a row the instrument could not weigh is named, kept in the denominator, and not quietly folded into the weighed ones.
weighed=118 unweighed=4 unread_tables=0
Three numbers that must be reported together. weighed alone is a statement about the subset the instrument happened to reach.
The rule
Count what is not covered, per file, as a number in the output. Coverage of a corpus of documents is not "we have a checker", it is a fraction, and the interesting part is always the tail nobody thought about.
And the smaller one: when a document says a section is obsolete, that sentence has a shelf life and no enforcement. If a table should stop being counted, delete it or mark it in a way the reader understands. Leaving prose to do the job of a declaration means the file and the tooling disagree, silently, from the moment the note is written.
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.