Match the Fixture Digest Before a Property Score Can Count

A property score is not a merge signal until the fixture digest matches a human pin. If the same patch produces different property results on two pins, record fixture drift. A flake note cannot explain either outcome.

Agent diffs fail in three ways that one CI summary collapses. The fixture bytes moved. A property did not hold. A known-flaky observation fired again. Those defects have different owners. Mixing them into one green or red line is how a moving tree gets certified, and how a real miss gets filed as noise.

This is a gate design, not a measured campaign. The exit codes below are policy choices. They are not benchmarks, and the script is a reference checker you still have to run yourself.

What one status line hides

A typical agent job log looks decisive and says almost nothing:

fictures: loaded
properties: 18 passed, 1 failed
flake_budget: within limit
result: review

"Loaded" does not name bytes. "Within limit" does not say whether the failed id was pre-listed. "Review" does not say whether the checker rewrote an oracle. Four nouns, one verdict, and no identity for the tree.

A clean remote workspace makes the gap sharper. Files that lived only on a laptop are absent. Caches differ. A property can pass because an input was missing and the checker skipped it, or fail because a file the author treated as stable was never pinned. Either result is about the tree, not about the patch logic.

The control is strict. Name the fixture bytes before you score behavior. Keep the flake observation in a third file that cannot be written unless the first two already agree.

Three ledgers, three questions

Ledger	Question it answers	Written only when
fixture_ledger.json	Which fixture bytes did this run use?	The manifest lists every fixture path, and the disk has no extras
property_ledger.json	Did pinned properties hold on that digest?	Live digest equals the human pin
flake_ledger.json	Did a pre-listed flaky test observe again?	Digest matches, oracles were not edited, and the test id is already on the flake pin

No ledger may copy recorded_at from another ledger. Shared clocks are how a later writer forges a causal story. Each file stores its own UTC timestamp and the digest it actually observed.

A human pin is a reviewed file in the repo. It is not a model suggestion, and it is not a CI cache entry. Example identifier only, not a recorded run:

{
  "pin_id": "fixpin-example-a",
  "algorithm": "sha256",
  "files": [
    "fixtures/orders_min.json",
    "fixtures/orders_boundary.json"
  ]
}

The digest is SHA-256 of path + NUL + file bytes for each listed path, in sorted path order. Sorting matters. A directory walk that follows filesystem order will disagree across machines even when the bytes match.

What belongs in each pin

Build the pins before you automate the refusal. A digest of an empty or decorative manifest is ceremony.

1. Fixture files are the smallest inputs that can falsify a claim. Include a boundary file, not only the happy path. Do not embed "current time" unless the property is about time. A moving timestamp makes dual-pin output noisy for a reason that is not the patch.
2. Each property has a stable id, a one-line claim, and a command that can fail. The claim names a domain drawn from the fixture pin. A predicate that cannot fail does not enter the pin. A check that treats the agent's new test edit as the oracle does not enter the pin either.
3. The flake pin is a separate reviewed commit. Each entry has a test id, the revision where a human accepted the label, and a review_by date the human fills in. This workflow does not compute flake rates, and it does not auto-extend that date. An id the agent appends is pin mutation, not evidence.

Workflow

Follow the steps in order. A later step must not heal an earlier failure. Commands assume Python 3.10 or newer and a checkout at ..

1. Build the digest from the pin, not from a free walk

python3 three_ledger.py digest \
  --pin pins/fixtures.json \
  --root . \
  --out ledgers/fixture_ledger.json

The command fails if a pinned path is missing. It also fails if a file under fixtures/ is not in the pin. Extra fixtures are unreviewed inputs, not free evidence. Exit 2 means the score is refused. Do not coerce that exit to success to keep a pipeline quiet.

2. Refuse to score properties on a mismatch

python3 three_ledger.py properties \
  --pin pins/fixtures.json \
  --expect-digest "$(jq -r .digest ledgers/fixture_ledger.json)" \
  --properties pins/properties.json \
  --command "python3 -m pytest -q properties/test_orders.py" \
  --out ledgers/property_ledger.json

If the live digest does not match expect-digest, the process exits 2 and does not create property_ledger.json. A missing property ledger is a rejected score, not an implicit pass. A green row from an unpinned tree is discarded, even when every assertion happened to succeed.

The property ledger contract, which your wrapper must write, looks like this:

{
  "kind": "property",
  "pin_id": "fixpin-example-a",
  "fixture_digest": "<digest from the fixture ledger>",
  "status_by_id": {
    "orders.total.non_negative": "pass",
    "orders.id.stable": "fail"
  },
  "oracle_moved": false,
  "recorded_at": "2026-09-25T12:00:00Z"
}

The timestamp above is a shape example. It is not a claim that this run occurred. oracle_moved is true when properties/ or tests/ differ from the base revision. In that case the property ledger is invalid, and you stop. You do not open the flake step to launder an edited oracle.

3. Compare two pins before anyone says flake

When the diff touches parsing, ordering, or defaults, run the same patch against two real fixture pins. Write two property ledgers. Then diff them.

python3 three_ledger.py dual-pin \
  --left ledgers/property_pin_a.json \
  --right ledgers/property_pin_b.json \
  --out ledgers/drift.json

If status_by_id differs, drift.json sets drift to true and the process exits 3. File fixture drift. Do not relabel the split as a flake. The bytes changed, or the pins disagree about which bytes are in scope.

If you only have one pin, skip this step and record dual_pin: not_run. Do not copy one ledger, rename it, and call it a second pin. Identical status maps with different recorded_at values are not drift. Time is not evidence.

4. Write a flake ledger only as a narrow observation

python3 three_ledger.py flake-observe \
  --fixture-ledger ledgers/fixture_ledger.json \
  --property-ledger ledgers/property_ledger.json \
  --flake-pin pins/flakes.json \
  --test-id orders.boundary.timeout \
  --out ledgers/flake_ledger.json

All four preconditions are required. The fixture digest equals the digest cited by the property ledger. oracle_moved is false. test-id is already on the human flake pin. The property ledger exists and its failing ids are not rewritten into this file. Exit 3 here means the flake write was refused. That is a gate decision. Retrying the same command will not create new facts.

5. Apply the merge table

Fixture digest	Property ledger	Dual-pin	Flake ledger	Merge posture
Missing or mismatch	Absent by rule	Any	Any	Reject. Do not score.
Match	Fail, oracles intact	Agree on fail	Any	Reject. The property miss stands.
Match	Invalid because oracles moved	Any	Any	Reject. Review the oracle edit on its own.
Match	Pass on pin A, fail on pin B	Disagree	Any	Reject. File fixture drift.
Match	Pass	Not run	Not requested	Human review. One pin is weaker evidence.
Match	Pass	Agree	Observation only, id pre-listed	Review. A flake note is not a pass.
Match	Any	Any	Writer tried to add an id	Reject. Pin mutation.

Review means a person still reads the diff. It does not auto-advance the change. An empty ledger set is a missing score, not a quiet yes.

6. Keep the checker small enough to test

The file below implements digest refusal and dual-pin classification. It does not implement properties or flake-observe. Those commands are specified above so the exits stay stable when you add them. Do not treat the paste as a timed result. It does not call a model.

#!/usr/bin/env python3
"""Proposed checker: property scores require a matching fixture digest."""

import argparse
import hashlib
import json
import sys
from datetime import datetime, timezone
from pathlib import Path

def digest_for(root: Path, files: list[str]) -> str:
    h = hashlib.sha256()
    for rel in sorted(files):
        path = root / rel
        if not path.is_file():
            raise SystemExit(f"missing fixture: {rel}")
        h.update(rel.encode())
        h.update(b"\0")
        h.update(path.read_bytes())
    return h.hexdigest()

def now() -> str:
    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")

def cmd_digest(args: argparse.Namespace) -> int:
    pin = json.loads(Path(args.pin).read_text())
    root = Path(args.root)
    fix_root = root / "fixtures"
    if not fix_root.is_dir():
        print("missing fixtures directory")
        return 2
    listed = set(pin["files"])
    on_disk = {
        p.relative_to(root).as_posix()
        for p in fix_root.rglob("*")
        if p.is_file()
    }
    extra = sorted(on_disk - listed)
    if extra:
        print("unpinned fixtures:")
        print("\n".join(extra))
        return 2
    digest = digest_for(root, list(pin["files"]))
    ledger = {
        "kind": "fixture",
        "pin_id": pin["pin_id"],
        "digest": digest,
        "algorithm": "sha256",
        "recorded_at": now(),
    }
    Path(args.out).write_text(json.dumps(ledger, indent=2) + "\n")
    print(digest)
    return 0

def cmd_dual(args: argparse.Namespace) -> int:
    left = json.loads(Path(args.left).read_text())
    right = json.loads(Path(args.right).read_text())
    same = left.get("status_by_id") == right.get("status_by_id")
    report = {
        "kind": "dual_pin",
        "drift": not same,
        "left_digest": left.get("fixture_digest"),
        "right_digest": right.get("fixture_digest"),
        "recorded_at": now(),
    }
    Path(args.out).write_text(json.dumps(report, indent=2) + "\n")
    return 0 if same else 3

def main() -> int:
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest="cmd", required=True)
    digest = sub.add_parser("digest")
    digest.add_argument("--pin", required=True)
    digest.add_argument("--root", required=True)
    digest.add_argument("--out", required=True)
    digest.set_defaults(fn=cmd_digest)
    dual = sub.add_parser("dual-pin")
    dual.add_argument("--left", required=True)
    dual.add_argument("--right", required=True)
    dual.add_argument("--out", required=True)
    dual.set_defaults(fn=cmd_dual)
    args = parser.parse_args()
    return args.fn(args)

if __name__ == "__main__":
    sys.exit(main())

Test the checker on synthetic files before it sits in front of a merge:

1. The pin lists fixtures/a.json, and that file is absent. Expect exit 2 and no output ledger.
2. The pin lists fixtures/a.json, and the disk also has fixtures/b.json. Expect exit 2.
3. Two property ledgers carry different status_by_id maps. Expect "drift": true and exit 3.
4. Two ledgers share a status map and differ only in recorded_at. Expect "drift": false.
5. Confirm the process never writes pins/flakes.json or pins/properties.json.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access fits this gate as a drafting aid, not as a pin authority. You can ask it for candidate property ideas against a schema you already own. Then read every predicate. A check that cannot fail, a weakened bound, or a comparison that imports the patch under test stays out of pins/properties.json. The pin changes only through review.

The free server option is a reasonable place to run the checker on a clean tree. That is the point of step 1. A fresh workspace will disagree with a laptop when fixtures were never committed. Let the digest fail. Do not paper over the failure by uploading a local flake pin, and do not assume the free option adds retention, a hardware profile, a quota, or a stable model name. Those details are outside this workflow. Pin the files you mean to score, and store the three ledgers as build artifacts next to the diff.

If you already have that workspace, run three_ledger.py digest there once on a synthetic pin before you attach the script to a merge button. A checker you have not watched fail is not a gate.

Limitations

The design does not prove the patch is correct. It shows only that the recorded property set held on the recorded bytes, or that it did not. Unpinned behavior stays unpinned.

Dual-pin comparison needs two real pins. Replaying one pin and renaming the ledger is not a second observation. Matching timestamps across ledgers should be treated as a writer bug.

The reference script does not execute pytest. The --command in step 2 is the integration you still have to wrap, including timeout and output capture. A hung property run is an infrastructure failure. It is not a flake observation, and it must not create flake_ledger.json.

SHA-256 of path plus bytes will not detect a property that reads an unlisted network resource. If a check can leave the machine, the ledger is incomplete. Ban outbound calls in the property pin, or record that ban as an explicit hole.

Who should skip this

Skip the split if the repo has no fixtures and no properties yet. Commit one fixture and one property that can fail. Then add the digest check. Starting with the script alone creates a green digest of nothing.

Skip it if you need a statistical flake model. This gate does not estimate rates or confidence intervals. It only refuses to let an unlisted failure consume a freeze write.

Skip it if the agent may rewrite pins in the same change as the product code. The design assumes pins move in a separate reviewed commit. Without that rule, the script can detect mutation and still lose to an approval click.

Closing

Score the digest first. Score properties only on that digest. Treat a split across pins as fixture drift. Leave flake notes in their own file, under those preconditions, and never as a substitute for either result.

That order is the strategy. The implementation can stay a short Python file, as long as the exits stay distinct and the pins stay human.