My test suite had 216 passing tests. It was lying to me.

An audit found a nested JSON false PASS in a replay-versus-live trace comparator. The local fix is recursive and type-aware; CSV equality remains deliberately unverified without provenance.

[1] and [true] are different JSON values. My Python comparator said they were equal.

It returned:

{
  "state": "PASS",
  "deterministic_parity_verified": true,
  "findings": []
}

That result came from a tool I built to compare a historical replay with a live-style decision pipeline. The point was to find the first recorded difference in features, model scores, decisions, configuration, or input evidence. A strict PASS was supposed to be the strongest possible result.

It was wrong.

The false PASS, reduced to five lines

The audit input was deliberately boring:

baseline = {"f": [1]}
candidate = {"f": [True]}

Both values were valid JSON-like input. Both traces had matching metadata and input hashes. The public recorder wrote them, the parser read them, and the old CLI certified them.

The underlying Python behavior is legal:

[1] == [True]           # True
{"x": 1} == {"x": 1.0}  # True

My comparator was asking the wrong question. Scalars went through type-aware logic, but nested lists and dictionaries eventually reached ordinary Python equality. For application code, that can be convenient. For a parity checker, it erases a recorded type difference.

Before / after: the same trace pair

I reran the preserved pre-fix source snapshot against this trace shape. It returned a verified strict PASS:

{
  "state": "PASS",
  "deterministic_parity_verified": true,
  "findings": []
}

The public synthetic fixture now runs through the current CLI. This is the actual output:

$ hfm-parity diff --baseline examples/nested_values/baseline.jsonl \
    --candidate examples/nested_values/candidate.jsonl \
    --output parity-demo.html --json-output parity-demo.json
FAIL: mode=strict; not-verified; aligned=1; findings=1

The JSON report identifies the leaf, its classification, and the recorded values:

{
  "state": "FAIL",
  "deterministic_parity_verified": false,
  "first_divergence": {
    "event_id": "evt-001",
    "stage": "features",
    "field": "payload[0]",
    "classification": "FEATURE_TYPE_MISMATCH",
    "baseline": 1,
    "candidate": true
  }
}

The fixture, source code, and regression tests are public in HFM Parity Lab. It is synthetic by design: it proves a regression boundary, not a historical market measurement.

The pre-fix PASS was reproduced from the preserved source snapshot; the current FAIL is generated by the public synthetic fixture. The fixture demonstrates a regression boundary, not a historical market measurement.

Why 216 tests did not catch it

The candidate had 216 Python tests when the second audit supplied the nested counterexample. The scalar integer defect was fixed. Clean wheel and source installs worked. The CLI, MCP demo, Ruff, mypy, and frontend build passed.

The gap was structural: the tests covered scalar numeric semantics, but not the same semantics at every JSON leaf. More happy-path tests would not have made that contract explicit.

What the tool is intended to compare

Backtests and live pipelines often disagree for reasons that are annoyingly ordinary. A feature was defined differently. A model version changed. A timestamp uses a different clock. An event came from a different source. The final BUY/HOLD decision is usually too late in the chain to explain much.

I built a trace comparator called HFM Parity Lab around a narrower question:

Given two recorded executions, what is the first observable difference, and do we have enough evidence to say the executions are comparable?

Each JSONL trace records run metadata plus event-level fields: an event ID, stage, features, timestamps, input hash, model score, decision, and optional state hash. The engine aligns event.id + stage, then returns PASS, FAIL, NOT_COMPARABLE, or UNVERIFIABLE.

Strict mode is for two runs that claim to reproduce the same thing. Regression mode is for deliberate old-versus-new comparisons. Regression mode can report agreement, but it never certifies strict parity.

The first audit found other failures too

The first independent audit began with 87 passing tests and found nine problems. These three are the easiest to reproduce.

1. Large integers were converted to floats

The first comparison helper turned numbers into float values before comparison.

9007199254740993 != 9007199254740992

Past 2**53, IEEE-754 doubles cannot represent every integer. The initial engine could call those values equal and return a verified strict PASS.

The fix compares integers exactly. An integer and a float are a type mismatch even when their values look equal; tolerance applies only when both values are floats. The repaired scalar tests cover values far beyond the floating-point range.

2. Matching event IDs could hide a different market

Two traces might both include evt-17, while one came from BTC and the other from ETH. The initial strict comparison did not treat symbol, venue, or data_source as part of the identity contract.

That makes event alignment look more meaningful than it is.

Strict mode now returns NOT_COMPARABLE for declared identity mismatches. It also needs a data source, input hashes on aligned events, an actual output to compare, and a model version when model scores exist. Missing evidence returns UNVERIFIABLE.

There is a practical difference between "I did not see a mismatch" and "the traces contain enough evidence to verify parity." A useful tool needs both answers.

3. The MCP output limit was per value, not per response

The first MCP formatter shortened nested values one at a time, then built the response. That did not constrain the total payload. The remediation report recorded a roughly 10 MB input yielding about 17.7 MB from one comparison call.

For a local agent tool, that is enough to flood a context window with repeated evidence.

The current implementation limits the whole response, preserves totals, labels truncation, and drops detail in a deterministic order. Its tests measure the actual structured plus text payload received by an MCP client as well as the formatting helper.

The audit also found recorder/parser disagreement, event reordering that could pass, missing provenance that looked like agreement, and a report viewer that could not open the external JSON it documented.

The P0 repair

The fix walks JSON-like values recursively and keeps the type explicit at every path: null, boolean, integer, float, string, array, object, or missing.

That makes the rules identical at the root and inside a container:

• [1] versus [true] is a mismatch;
• { "x": 1 } versus { "x": 1.0 } is a mismatch;
• null and a missing key are distinct;
• tolerance applies only to finite float-versus-float values.

In compact form, the contract is:

same(path) = same_json_type(a, b) ∧ compare_by_type(a, b)

integers: exact
floats:   finite(a, b) ∧ |a - b| ≤ ε
arrays:   same length ∧ same(index) for every index
objects:  same keys ∧ same(key) for every key

The comparison checks type before value at every nested path. Float tolerance applies only when both recorded values are floats.

This is a specification. Its purpose is to make the important bit reviewable: a convenient host-language equality operator never gets to silently redefine the verification contract.

The regression coverage takes those cases through the public recorder, JSONL reader, CLI, and a real stdio MCP process. The local suite is now 242 Python tests; Ruff, strict mypy, package builds, clean-wheel installation, frontend tests, and the frontend build also pass.

That is a precise claim about the known failure: the local release candidate no longer certifies those nested type differences as strict parity.

CSV onboarding and verdict strength

The same candidate adds a small CSV Quick Compare path. A developer maps identity columns, stages, features, types, and units from two CSV logs; the existing trace engine then aligns and compares them. It does not import raw exchange data or introduce a second diff engine.

The important limitation is deliberate: equal CSV rows normally cannot establish matching code, configuration, model, input snapshot, or clock domain. An equal comparison therefore stays UNVERIFIABLE unless that provenance is supplied. A mismatch remains useful evidence of a difference.

A useful tool does not turn absent evidence into agreement. It says what the supplied records can and cannot support.

A test technique worth reusing

For any validator that can return a strong success result, write inputs that contradict the result by construction. For a JSON comparator, that means testing each rule at the root and inside objects and arrays:

• True versus 1;
• 1 versus 1.0;
• a missing key versus null;
• a value just inside and just outside float tolerance;
• identical event IDs with different declared provenance.

This makes the guarantee concrete enough to attack. The repository is public; the reported 242 local tests and quality gates are useful evidence, not an independent correctness proof.

Try it

The repository includes only synthetic examples; it does not contain HFM market data or a trading system.

git clone https://github.com/icesurf666/hfm-parity-lab.git
cd hfm-parity-lab && python -m venv .venv && source .venv/bin/activate
pip install . && hfm-parity demo

That writes a synthetic HTML and JSON report. The nested-value regression tests cover the false-PASS class described here; the README has the trace format and CSV workflow.

Comments

[Alex Shev]

The distinction between FAIL, NOT_COMPARABLE, and UNVERIFIABLE is as important as the recursive type check: it prevents a comparator from turning missing provenance into a reassuring result. A useful regression fixture would vary one provenance field at a time while keeping the payload equal, so the expected verdict boundary stays visible when the trace schema evolves.

[Pavel Kazantsev 💎]

That fixture structure locks the boundary cleanly. One provenance field at a time keeps the expected verdict readable when the schema changes - and makes clear which field trips NOT_COMPARABLE versus which one triggers UNVERIFIABLE. The synthetic examples don't cover that matrix fully yet, that's next.