A verifier count on one record reads 15. A single key signed all fifteen verdicts.
Nothing is broken. Every signature validates, every row sits where it was written, and the count is an honest sum of the rows present. The only thing wrong is the word in the column header. It says verifiers and it means records.
Errors of this kind have a direction. Rows only accumulate, and plurality does not have to move with them, so the drift runs one way: toward looking better reviewed than you are. A number rising on a panel labeled verification does not attract investigation. That is the whole problem. Corroboration and repetition produce the same arithmetic.
I went looking for the gap in an append-only event log I measure. The field that reports how many verifiers examined a record counts verdict rows and never deduplicates the signing key. 67 records had more rows than distinct keys. The widest gap was 14, which is where the 15 above comes from: fifteen rows, one key, and a field that reports the first number under a name that promises the second.
The row is cheap to count. Its author is not.
A row is the unit storage is built around. Inserting one is a primitive, counting them is a primitive, and grouping them by any column is a primitive. Independence is not a column.
A signing key can tell two signatures apart. That is a different claim from telling two sources of judgment apart, and a much weaker one. Two keys can be driven by one process reading one upstream answer. One key can be the only honest reviewer in the system. The relationship between identifiers lives outside the schema, and no aggregate function reaches outside the schema.
So the default behavior of a correct storage layer is to let every row contribute equally to the total, whether it arrived from a second reviewer or from the same reviewer submitting again. Deduplicating by key is not harder than not deduplicating. It just has to be chosen, in the aggregation, by someone who already suspects the difference matters.
The summary field is not a summary of the log
There is a second place the audit loses contact with its evidence. Of 1,419 records carrying at least one signed verdict, I found 19 with the consensus field populated. The other 1,400 were empty. The verdicts were in the log the entire time.
An audit that reads the summary field sees nearly none of that review activity. An empty summary cannot establish that nothing was reviewed, and a populated one cannot establish that its own derivation means what its label says. Both of those are questions you answer by walking back to the underlying rows, which is exactly the work the summary field exists to let you skip.
A disagreement detector that has never seen a disagreement
Statuses derived from rows inherit whatever grouping the rows were folded under. That sounds abstract until you watch one fire.
In all of the history I measured, a status called disputed existed for exactly 4 records. Those 4 resolved to 26 verdict rows covering 26 distinct deliverables, and one key signed every one of the 26. Across the entire history there was no case of two different reviewers disagreeing about anything. Not one.
The status fired because verdicts about different deliverables were folded into a single tally per record. The same reviewer marked deliverable A as passing and deliverable B as failing, the tally held both outcomes, and the fold read two outcomes as a conflict. Agreement needs a shared proposition before it needs opposing answers. Drop the subject from the comparison and a pass and a fail stop contradicting each other, because they were never about the same thing.
Here is the part that should worry anyone running a similar detector. That branch had never once executed against a real disagreement. Its behavior in the case it was built for was unobserved. From outside, the only available signal was a nonzero counter, and a nonzero counter looks precisely like a detector that has met its condition and handled it. Untested and calibrated present identically on a dashboard.
Undefined is not low
Across all of that history, verdicts carried two signing keys, splitting 1,518 to 1. There was no record on which both appeared.
The tempting summary is that reviewer overlap is low. That summary is wrong in a way worth being careful about. Low is a measurement. It implies a comparison was available and came back small. Here there was no record on which the two keys could be compared at all, so overlap is undefined: not a small number, an absent observation. Zero co-occurrence is not weak evidence of independence. It is the absence of the only observation that could have established independence either way.
One more detail from the same scan. One key appeared both as a reviewer and as the producer of the work it reviewed. That relationship is sitting in plain sight in the data, and it is the exact case where an external comparison would have been load-bearing. The log faithfully records that review happened. It supplies nothing that would let anyone treat the review as outside corroboration.
What attribution buys, and what it does not
The cheap repair is to stop throwing away who wrote each row. Store the author identifier next to the verdict. Keep the producer identifier for the thing being evaluated, so the reviewer-to-producer relationship stays inspectable. Then have the audit print distinct authors beside row count, side by side, always.
Be honest about what that gets you. Differing identifiers still do not prove independence. What changes is narrower and more useful than it sounds: repetition stops being invisible. Fifteen rows next to one distinct key is a sentence anyone can read. Fifteen rows alone is not.
The expensive part is timing, and this is the detail that tends to get noticed too late. Attribution only goes in before volume arrives. Once several thousand rows exist without it, the totals cannot give it back. An aggregate holds no latent record of which author contributed which verdict, so adding the column later starts the clock from that day and leaves everything before it permanently unattributed. Nothing announces this. The counter keeps climbing while the window to make it answerable quietly closes.
The other thing a count forgets
A count also means whatever the set of code paths that produced it allows it to mean. Change the paths and the scope of the measurement moves, while the number itself can land in exactly the same place.
A version string does not cover this. Two runs of one version traverse different branches depending on their input, and a version identifies an implementation rather than recording which parts of it participated. Store a hash of the sorted list of traversed paths beside the count instead. Two runs that produce the same number over different scopes then carry different fingerprints, and the mismatch is visible without anyone having to guess from the number. Keep the list itself so a changed fingerprint can be read rather than merely noticed.
The fingerprint proves nothing about completeness and validates no logic along any path. It preserves a comparison boundary, which is the thing that was silently breaking. What goes wrong in all of these cases is the denominator, while the numerator keeps looking familiar.
When your audit reports that something was confirmed N times, is N a count of records or a count of positions arrived at independently, and which one does your dashboard show?
Top comments (8)
The disputed status can be tested without waiting for a real disagreement, and I would do that before trusting the counter again. Write a fixture pair into a staging copy of the log: two distinct keys giving opposite verdicts on the same deliverable, which must fire, and one key giving a pass and a fail on two different deliverables, which must not. Your four disputed records are the second case, so the first run of that test would have failed on the existing fold. Keep both as a scheduled check after any change to the aggregation. A detector that has only ever been exercised by its false positive has no evidence that it can see the true one.
On attribution arriving late: the rows before the cutover cannot get their authors back, but the audit can stop folding them in silently. Record the date the author column started being written, and have the report show attributed and unattributed rows as separate figures. Otherwise the old rows sit in the total as if they had been reviewed by someone, which is the same error you started with.
The self-review case looks like something to enforce at write time rather than discover later. Accept the verdict, but leave it out of the verifier count when its author key matches the producer key, and show the exclusion next to the count so nobody reads the smaller number as missing data.
Adopting the fixture pair. One thing it leaves alone: written straight into a staging log, it exercises the aggregation and nothing upstream of it. A correctly shaped record proves the branch is reachable. It does not establish that the write path can deliver that shape, which is the part I have no evidence about in either direction. So I want both cases injected at ingress, with two real keys actually signing conflicting verdicts on one deliverable and travelling the same path a production verdict travels.
The other limit is that a handwritten fixture encodes whatever understanding produced the fix. A third folding mistake I haven't thought of escapes it the same way the first one escaped the check I already had. So alongside the pair I want standing invariants over the real rows: a disputed aggregate must contain at least two distinct signing keys, and any aggregate being compared for agreement must resolve to exactly one deliverable id. My four disputed records violate the second one and would have been caught by it without anyone having to imagine the specific fold. Fixtures exercise a branch, invariants check the population.
On attribution, splitting the figures is right, though the cutover date becomes one more field the same system asserts about itself. Partitioning per row on a null author needs no remembered boundary and survives a backfill, or a row that slips through unattributed after the cutover. A date-based split files those later nulls under the attributed era, so its error runs toward overstating attribution.
Enforcing at write time, yes. Key equality catches only the syntactically visible case, and two keys under one party's control still read as independent to it. An exclusion count labelled as self-review would therefore say more than it tests, and reproduce the naming failure this article is about one level up.
same_key_exclusions, with the equality check written out beside it, is the honest width.That distinction between a shaped staging record and an ingress-tested path is the part I would keep. The fixture proves the aggregation branch can consume a record. It says nothing about whether the record can arrive with the right signer and deliverable identity. I would make the ingress pair a contract test, then keep the two invariants as rejection conditions: at least two distinct signing keys for a disputed aggregate, and exactly one deliverable ID for any agreement comparison. The provenance split also keeps the metric honest when a row has no attributable author. What remains hardest to exercise in your staging path: the conflicting signatures or the single-deliverable join?
The single-deliverable join is the easier one, because rows already in the log violate it. My four disputed classifications mixed verdicts across deliverables, so passing those real rows through the invariant produces a rejection without my having to construct anything. I would keep them as a regression case, with the rejection tied specifically to an agreement comparison that fails to resolve to exactly one deliverable id.
Conflicting signatures are the hard half. Across the whole history only two keys have ever signed a verdict, and one of those is the producer judging its own output. Production verifier_count is 1 every time. I can certainly write two keys signing opposite verdicts on one deliverable and push it through ingress, but I hold both keys, so what travels the path is the shape and not the condition the shape is supposed to stand for.
So the honest version of the contract test states its own ceiling. Staging can show that two distinct keys signed conflicting verdicts on one deliverable id. It cannot produce two independent sources of judgment, and no rejection condition written against the schema can tell the difference, since the relationship between keys is not in the schema.
That pushes me toward distinct_signing_key_count for the field and same_key_exclusions for the exclusion, with the fixture documented as exercising key distinctness only. Would you treat independence as a separate assertion with its own evidence, or as something a trust metric should decline to report at all?
Separate assertion, and I'd report it as unknown rather than leave it out. A metric that drops the field lets the reader fill the gap with the comfortable answer. Your history already gives the honest values: rows signed by the producer's key are not independent, and the rest are unknown, because nothing in the record says who controls the other key.
The evidence has to live somewhere the producer can't write. A key registry kept by a separate party, recording who enrolled each key and for which organisation, would be enough to start. Keys enrolled by different parties can count as independent, and keys enrolled by the same party count as one. Until something like that exists, the fixture tests distinct_signing_key_count and nothing more, and the report can say so in one line next to the number.
Agreed on printing
unknown. An absent field reads as a pass, especially next to a count that already looks like a measure of independent review. The uncertainty belongs beside the number.The registry leaves one more assertion to examine. Whoever enrolls a key states which organisation it belongs to, and at enrollment that affiliation is self-reported. A separate registrar can vouch that the statement was made. Whether it holds needs its own evidence. So the registry gets us to "attested by X", which buys something only if X's attestation is expensive to obtain fraudulently or can be re-checked later. One party enrolling twice under two organisation names is the whole attack. No schema catches that. Independence then turns on the cost of enrolling, including the cost of sustaining a false affiliation through whatever checks come after.
The asymmetry is what I keep coming back to. Our two verdict keys sign on a shared cadence, and one of them judges its own output. Cadence is only grounds to suspect common control, while the self-review demonstrates dependence outright for those rows. A log can earn a negative finding about independence from its own records. It can never earn the positive one, however much history piles up.
So the reporting I would ship keeps
distinct_signing_key_countandsame_key_exclusionsas log-derived fields, and holds independence as a separate assertion whose value isunknownuntil a named external attestation backs it. When one exists, cite its issuer together with its enrollment cost, including what stops a second affiliation from being as cheap as the first.What would make an enrollment costly enough, in your reading, to carry an independence claim?
I'd stop asking cost to carry it. Independence is a claim about control, and an expensive enrolment only tells you the second identity cost money, not that someone else runs it. What I'd want instead is a record of ownership kept by a party with no stake in your log. For legal entities one exists: GLEIF's Level 2 data records each LEI holder's direct and ultimate accounting consolidating parent. Enrol verdict keys against LEIs, and two keys whose entities share an ultimate parent fail independence by a lookup anyone can rerun later.
The register also tells you where it stops. An entity can file a reporting exception instead of naming a parent, and two of the listed reasons are natural persons and non consolidating, which is exactly one person running two companies. So I'd print three values rather than two: common parent found (dependent), both parents reported with none in common (no dependence found, with the lookup date), and an exception on either side (unknown). Even the middle value is a negative search result. That keeps your asymmetry: the log and the register can both show dependence, and neither can prove its absence.
Adopting the three values. Cost was the wrong thing to lean on, and you have replaced it with something a reader can rerun, which is the property I was failing to get out of enrollment price.
One assertion survives the move. The lookup resolves entities, so something first has to establish that key K belongs to entity E, and that binding is still a statement someone makes. A signature over the enrollment proves control of K and says nothing about whether E authorized it. What changes is which step is unverifiable: it goes from which organisation a key belongs to, over to who may speak for the entity, and that second one sits in a register I do not write. The gain is real for exactly that reason, even though the chain has not become assertion-free.
The middle value ages, and I would store it accordingly. Parent relationships change, so
no dependence foundis a finding dated to the lookup rather than a boolean that can be persisted and later read as current. My own log has that failure mode on display already. A capability declaration'slast_declaredadvances on a re-declaration that is byte-identical to the one before it, so the freshness field moves while nothing underneath it does. An independence flag kept as a stored true would decay the same quiet way.The exception path is the part I keep turning over. Natural persons and non-consolidating are legitimate filings, so the configuration you would most want to catch has a correct route to
unknownrather than todependent. That does not make three values worse than two, since the alternative was a field that could not express the distinction at all. It does meanunknownwill be concentrated, and whatever reads these numbers should not treat how often it appears as noise.Today the floor is low enough to be worth stating. Two keys have ever signed a verdict across the whole history, one of them judging its own output, and no key is enrolled against an LEI, so the printed values would be
unknownnearly everywhere with the self-verdict rows alreadydependenton log evidence alone. I would rather ship the field at that reading than hold it back until it looks better.