DEV Community

Mahiro Hirakawa
Mahiro Hirakawa

Posted on

One row in my ledger said 12 of 3. Six review rounds later, not a single verdict had changed.

There is a ledger here where each row compares our implementation against someone else's on one axis, and carries a letter: ahead, level, behind. Each letter is supported by numbers and by pointers into both codebases.

A reviewer reads rows and disagrees with them. Six rounds ran over one section of that ledger. Every round found real defects. Not one round changed a letter.

What the rounds actually found

Three kinds of thing, none of which is a wrong conclusion.

A closing sentence that was false about its own code. One row ended by saying the operation returns "not contained" for a failing name. The code returns three different outcomes: not contained, absent for a name outside the root, and unsettled for a race that ran past its bound. The sentence was not a simplification. It named one of three and implied it was the set.

Two counts glued together with a slash. One row wrote:

probes=12/3
Enter fullscreen mode Exit fullscreen mode

which reads as twelve of three. There was no ratio. There were two independent facts, twelve probes run and three conditions declared, and a slash between them because the surrounding rows all had a shape like n/m and this one wanted to match.

Cited ranges that did not hold what the row claimed. A reader that resolves every citation and compares its extent against the claim found several rows off:

range_wider     row 4:  cites 9 lines, needs 7
range_narrower  row 10: cites 6 lines, needs 12
Enter fullscreen mode Exit fullscreen mode

The second is the dangerous one. A citation narrower than its claim points at a fragment and asserts the whole.

kind changes the verdict changes whether you can check it
clause names 1 of 3 outcomes no yes
two counts written as a ratio no yes
cited range wider or narrower no yes

Six rounds without a letter moving is not obviously good news

From the outside this is indistinguishable from churn. Six passes, a pile of edits, the same conclusions as before. If someone asked what the review accomplished, "we corrected some numbers" is a weak answer and it is also the true one.

What changes the reading is what each round left behind. Over those six rounds the checking apparatus gained three readers it did not have: one that resolves a cited range and compares it to the claim, one that refuses a count written as a pair unless both halves come from declared sources, and one that parses the ledger's tables rather than trusting their shape.

The repaired rows show what the readers now require. A count stays a count, and a sample keeps its population beside it:

OK_CONTAIN   conditions=6
OK_ATTEST    probes=12  declared=3        was: probes=12/3
arms=9  sampled=7  ratio=7/7              a pair, with the thing it is a pair of
Enter fullscreen mode Exit fullscreen mode

So the rounds were not producing opinions about the rows. They were converting a prose ledger into a machine-checkable one, one defect class at a time, and the letters were never what was in question.

The test that tells churn apart from progress

After a review round, name what now runs that did not run before. If the answer is nothing, that round produced an opinion.

An opinion is worth having. It is just not worth six rounds, and it does not accumulate, so the seventh round rediscovers what the third one found.

The three readers above will catch their defect classes in every future row without anyone reading carefully. That is the part that compounds. The corrected rows are a one-time payment.

What I would do differently

I would say out loud, after round two, that the letters look stable and the evidence does not, because that is a different activity from reviewing the claims and it deserves to be named. Rounds three through six were an audit of the citation apparatus wearing the costume of a content review, and nobody said so, so each round started by re-reading rows that were not the problem.

Two things I keep

A count and a ratio look identical and mean different things. 12/3 passed several readings because it had the shape of the rows around it. Any number written as a pair needs both halves to come from somewhere declared, and a reader that enforces that is fifteen lines.

Review rounds that never move a verdict are measuring your evidence, not your claims. That is legitimate work and it is a different job. Name it when you notice it, because the two jobs want different reviewers, different stopping conditions, and, most of all, a different definition of done.

Top comments (0)