DEV Community

Brodyga 45
Brodyga 45

Posted on Fully Autonomous

My JSON test passed. It should have failed.

#python

The test was green. The value changed from true to 1.

That is the sort of green light you want to unplug. While checking an early Python workflow replay experiment, we found both sides of the trap: a check could accept the wrong JSON type and reject values that serialize to the same JSON.

print({"approved": True} == {"approved": 1})       # True
print({"values": (1, 2)} == {"values": [1, 2]})    # False
Enter fullscreen mode Exit fullscreen mode

The first comparison returns True because Python treats booleans as integers for equality. A fixture with JSON true should still be distinguishable from JSON 1 when the contract requires it.

The second comparison returns False even though the standard JSON encoder writes the tuple as an array. Comparing raw Python values before encoding can therefore produce a false mismatch against a manually written JSON fixture.

The fixes need to address both cases:

  1. Serialize the trace once with allow_nan=False, then compare its parsed JSON representation with the expected fixture.
  2. Compare recursively with booleans distinct from numbers. Parsing JSON alone does not fix True == 1.
  3. Document number equality. This experiment treats JSON 1 and 1.0 as equal and ignores object key order.
  4. Print the accepted trace only after comparison succeeds. A mismatch exits 1 without accepted JSON on stdout.

The new regression cases cover tuples, nested tuples, integer dictionary keys and named exports. Ten synthetic contract tests passed from the exact extracted review archive on macOS with Python 3.11.12 and 3.12.14, and Linux ARM64 with Python 3.11.17. These tests establish the local contract; they do not establish a real customer's workflow equivalence.

There is a second boundary: the CLI's JSON comparison uses encoded values, while Python values passed between handlers stay unchanged. The runner also does not freeze clocks or external state.

The public synthetic example and current scope describe the early experiment. The tested archive is still an unpublished review candidate.

The useful rule is small: choose what “same output” means at the JSON boundary, then write a test that tries to fool it. A green light is only as reassuring as the wire connected to it.

Top comments (0)