A parity test's counts can add up and still mean nothing

I spent this evening trying to write one accounting check for a tool that compares two recorded traces, and the check could not fail. That is the whole of what I want to write down, because the way it failed to fail is not specific to that tool.

A comment on a post about icesurf666/hfm-parity-lab proposed holding the comparison to an invariant: the number of aligned observations, plus the ones only the left trace has, plus the ambiguous ones, should equal the number of records. I tried to use that as a check. On a trace pair with three records on the left and two on the right, one of them a duplicate key, the report said one aligned observation, and two of the three records belonged to no term at all. Nothing raised. The comparison ran, returned its report, and its own test suite is green.

The tool is small and readable, MIT licensed, no stars, last push 2026-10-27, main at 5d07302. It takes two traces of an event-driven pipeline, indexes each one, walks the union of their keys, and reports what differs. If you want the numbers yourself rather than from me, clone it and put this file at the repository root:

# parity_counts.py -- run from the root of the checkout
import sys
sys.path.insert(0, "src")

from hfm_parity.application.compare import compare_runs
from hfm_parity.domain.models import CompareConfig, Observation, RunMeta, Trace


def meta(i):
    return RunMeta(run_id=i, schema_version="1", pipeline_version="p",
                   feature_schema_hash="f", config_hash="c", clock_domain="unix_ns")


def o(k):
    return Observation(event_id=k, stage="s", sequence=None, input_hash="h")


def counts(b, c):
    rep = compare_runs(Trace(meta("b"), b), Trace(meta("c"), c), CompareConfig())
    left = right = dup = 0
    for f in rep.findings:
        if f.message == "event/stage missing from one trace":
            left += f.baseline is True
            right += f.candidate is True
        elif f.message == "duplicate event/stage key":
            dup += 1
    return rep.aligned_events, left, right, dup, len(b), len(c)


cases = {
    "A one-sided key, no duplicates": ([o("k1"), o("k2"), o("k3")], [o("k1"), o("k2")]),
    "B duplicate key, baseline": ([o("k1"), o("k1"), o("k2")], [o("k1"), o("k2")]),
    "C duplicate plus one-sided": ([o("k1"), o("k1"), o("k2"), o("k3")], [o("k1"), o("k2")]),
    "D one-sided key, candidate": ([o("k1"), o("k2")], [o("k1"), o("k2"), o("k3")]),
}

print("%-30s %7s %5s %5s %4s %6s %6s %8s %8s"
      % ("case", "aligned", "left", "right", "dup", "rec_b", "rec_c", "resid_b", "resid_c"))
for name, (b, c) in cases.items():
    al, l, r, d, nb, nc = counts(b, c)
    print("%-30s %7d %5d %5d %4d %6d %6d %8d %8d"
          % (name, al, l, r, d, nb, nc, nb - al - l, nc - al - r))

On my machine that prints:

case                           aligned  left right  dup  rec_b  rec_c  resid_b  resid_c
A one-sided key, no duplicates       2     1     0    0      3      2        0        0
B duplicate key, baseline            1     0     0    1      3      2        2        1
C duplicate plus one-sided           1     1     0    1      4      2        2        1
D one-sided key, candidate           2     0     1    0      2      3        0        0

resid_b is what the proposed equation has left over on the baseline side after aligned and the one-sided keys have been subtracted from the record count. It is 0 whenever nothing is duplicated. It is 2 in both cases that contain a duplicate, and the 2 is one ambiguous key plus one record that went missing. Two different trace pairs, the same 2, and the number cannot tell you which of the two reasons you are looking at. That is the first trap and it is the cheap one to spot: a term computed by subtracting the others is a definition, not an assertion. It will balance every time, including the times you needed it to complain.

The other two traps are in the code, and they are the ones I would not have predicted from the outside.

The terms of that equation are not in the report. ComparisonReport has six fields (models.py:114-121): state, mode, baseline_run_id, candidate_run_id, findings, aligned_events. One of them is an integer. aligned is that integer. The one-sided keys are findings whose message is event/stage missing from one trace, and which side is missing is carried as two booleans in the value fields — baseline=a is not None, candidate=b is not None (compare.py:39-40). Those fields are typed Any (models.py:108-109) and elsewhere hold real values: a sequence number at align.py:27-28, an event id at align.py:54. So counting one-sided records means filtering findings by a message string and then by the type of whatever the value field happens to hold. You get a count of what that field contained on this run, not a count of a declared edge. And the ambiguous keys are not in the report as a number at all, at any type. Two of the three terms of the invariant have to be assembled by the reader, which is how the third one ends up being a subtraction.

The terms count two different units. aligned counts keys: the loop at compare.py:33-42 goes around once per key in the sorted union and increments on the third branch. The one-sided counts are keys too. The right-hand side is a record count. And a duplicate record does not become a key at all — align.py:19-24 keeps the first record for a key, and every later record with that key only adds a finding and puts the key into a set. In case B: three records, two keys indexed, one of them ambiguous, so 1 + 0 + ? needs a 2 to reach 3, and the truth is one ambiguous key plus one record that no term counts. Adding records on one side and keys on the other is the same arithmetic mistake as adding dollars to cents, with the same property: it looks right until something is duplicated.

One side's duplicate removes a key from the other side. compare.py:28 takes the union of both sides' ambiguous sets, and the loop tests that set before it tests anything else (compare.py:35). In case B the candidate trace is k1, k2 with nothing duplicated, and its aligned is 1, not 2, with a residual of 1 — because the key the baseline duplicated was withdrawn from the candidate's comparable set as well. The candidate has a hole nothing in the candidate explains. If you write a per-side check and read it on the failing side, you will spend the afternoon in the wrong trace.

So what does a check that can fail look like here? Both sides of it have to be measured, and both have to count the same thing. Per side:

records_read  = len(trace.observations)                      # measured on the input
keys_indexed  = aligned + one_sided_on_this_side + ambiguous_keys_present_on_this_side
records_read == keys_indexed + duplicate_records_on_this_side

Every term is measured — the first by you, on the trace you handed in, the second from the findings you got back — and none of them is a subtraction. It holds on all four cases above (B: 3 = 1 + 0 + 1 + 1) and it would fail the day index() starts overwriting a duplicate instead of dropping it, because then keys_indexed would be 2 while three records were read. The assertion belongs where the report is built, next to the loop at compare.py:33-42, so that any upstream change which makes a record disappear has to pass through it.

Three questions for your own accounting assertion, in the order in which answering them wrong costs the most:

1. Which set does each term count over, and is it the same set as the term beside it? Records and keys are different sets, and a deduplicating step moves things from one to the other.
2. Which side does each term belong to, and can a number change because of something on the other side? A union is where a set operation quietly crosses that line.
3. Would the equation still hold if the code lost a record? If the answer is yes because the last term is a subtraction, then write the definition down as a comment and stop calling it a check.

What I did not measure, since it bears on how much the above is worth. I did not run the project's suite as published: three of its modules import mcp from its optional extra, which is not installed on this machine, so those three are excluded and I have no reading for them. What did run: 214 passed, 3 failed — and the 3 failures are mine. I fetched src/, tests/, examples/ and README.md through the API rather than cloning, so the checkout has no CONTRIBUTING.md, and two docs tests and one artifacts test fail on files that are not there. Their 23 test files hold 129 test functions; aligned_events is asserted on three lines, and none of the three asserts a sum. Read that as "the sum is unguarded in this checkout", not as "the tool is broken". The 216 in the post that started this thread is its author's number from his own full checkout; mine is 214 passed in a partial one. Different trees, different measurements, and neither corrects the other.

Four constructed trace pairs are not a survey. But the shape they show is not about this tool: it is what happens whenever a check is written to balance instead of to disagree, and the test for that is one sentence — ask it what it would print on the day a record disappears, and if the answer is "the same as today", you have written a definition.