Lock the Pre-Fix Seed Before You Trust a Property Check

A green property check is not merge evidence for an agent patch. Merge evidence is a seed that failed on the pre-fix tree, a fixture locked to that seed, and a freeze on any flake that cannot replay under that same seed.

Agent diffs often arrive with new checks. Many of those checks restate the new code, snapshot the new output, or retry until timing noise disappears. Wrapping the same idea in a “property” function does not change the evidence. If the check has never failed on a named input, you have not shown that it can see the bug the patch claims to close.

This article proposes a seed ledger. It is a workflow, not a framework. The artifact is a JSON file, a small Python runner, and a merge rule that ignores unearned greens.

What the ledger is allowed to record

Three records matter. Everything else is commentary.

1. A property with an explicit generator and a seed budget.
2. A fixture promoted from a failing seed, hashed and human-locked.
3. A freeze on any failure that does not reproduce when the seed, clock, or payload is fixed.

Random passing runs belong in a log. They do not vote. Pass counts grow with retries. Seeds do not.

Why a pre-fix seed beats a pass count

Name the input that was illegal before the patch. If you cannot name it, you cannot tell an invariant from a souvenir of the new function body.

A typical agent patch on a parser adds assert parse(s) is not None for three strings it just emitted. CI is green. The old crash payload is gone from the tree. That is not a property check. It is a restatement.

A usable property is an invariant that was false on at least one pre-fix seed. After the patch, that same seed must pass. The fixture hash must not drift. Exploration seeds may find new bugs. They do not replace the pre-fix seed.

Ledger format

Keep seed_ledger.json next to the tests. Humans edit ownership fields. Agents may append candidates only under a draft key that the gate ignores.

{
  "properties": [
    {
      "id": "parse.no_throw_on_printable",
      "fn": "tests/properties/test_parse.py::prop_no_throw",
      "seed_budget": 64,
      "pre_fix_failing_seeds": [180229],
      "status": "owned"
    }
  ],
  "fixtures": [
    {
      "id": "fx-180229",
      "property_id": "parse.no_throw_on_printable",
      "seed": 180229,
      "sha256": "<digest of tests/fixtures/180229.bin>",
      "locked_by": "human",
      "path": "tests/fixtures/180229.bin"
    }
  ],
  "freezes": [
    {
      "id": "flake-time-skew",
      "test": "tests/test_clock.py::test_skew",
      "reason": "fails only with an unseeded clock",
      "cannot_gate": true,
      "required": false
    }
  ],
  "draft": []
}

Compute the digest from fixture bytes. Do not paste a hash from an article. Empty-file hashes are a common copy-paste failure; treat any digest you did not compute as untrusted.

Numbered workflow

Run the steps in order. Skipping the pre-fix replay is how tautologies get merged.

1. Write the claimed invariant in one sentence

State what the patch makes true. If the sentence only names the new function, stop. The agent is restating code. A property has to mention inputs, outputs, or a protocol, not the helper that was just added.

2. Recover at least one pre-fix failing seed

Check out the parent commit. Run the generator with a fixed RNG. Record every seed that raises or returns false. Zero failing seeds means the property does not see the defect, or there was no defect.

git switch --detach HEAD^
python find_failing_seeds.py --property parse.no_throw_on_printable --budget 64
git switch -

If the new property already passes on parent, the check is unearned. Calibrate it with a one-line mutant: revert the agent’s fix, confirm a seed fails, restore the fix, confirm the same seed passes. That mutant is a calibration tool. It is not a substitute for a fuzzer.

3. Lock the fixture as bytes, not as a test name

Serialize the input the seed produced. Hash it. Commit the bytes and the digest with a human in locked_by. Test names move. Byte hashes do not, unless someone edits the file.

4. Replay the locked seed on the patched tree

The property must pass on that seed. Then spend the remaining budget as exploration. Exploration failures become new fixtures or new freezes. They do not silently retry.

5. Freeze failures that have no seed

If a test failed and you cannot replay it with a seed, a clock, or a recorded payload, set cannot_gate to true. Frozen tests may still run. They must not be required for merge.

6. Gate on the ledger, not on pytest’s exit code alone

The merge job reads seed_ledger.json. It fails closed when a required property has an empty pre_fix_failing_seeds list, a fixture hash mismatches, or a freeze is marked required.

Runner example (starting point, not a drop-in gate)

Wire replay_seed to your real property function before this script can block a merge. The generator below is a stand-in.

#!/usr/bin/env python3
"""seed_ledger.py — replay pre-fix seeds, hash fixtures, refuse unearned greens."""
from __future__ import annotations

import hashlib
import json
import random
import sys
from pathlib import Path

LEDGER = Path("seed_ledger.json")
FIXTURE_DIR = Path("tests/fixtures")


def load() -> dict:
    return json.loads(LEDGER.read_text())


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def fixture_path(seed: int) -> Path:
    return FIXTURE_DIR / f"{seed}.bin"


def replay_seed(prop_id: str, seed: int) -> bool:
    """Replace this stub with an import of the real property function."""
    rng = random.Random(seed)
    sample = rng.randbytes(32)
    # Example invariant only: latin-1 decode must succeed.
    # Label: not executed against a production parser.
    try:
        sample.decode("latin-1")
        return True
    except Exception:
        return False


def cmd_replay(ledger: dict) -> int:
    errors = 0
    for prop in ledger["properties"]:
        if prop.get("status") != "owned":
            continue
        seeds = prop.get("pre_fix_failing_seeds") or []
        if not seeds:
            print(f"GATE: {prop['id']} has no pre-fix failing seed")
            errors += 1
            continue
        for seed in seeds:
            path = fixture_path(seed)
            if not path.exists():
                print(f"GATE: missing fixture for seed {seed}")
                errors += 1
                continue
            digest = sha256_bytes(path.read_bytes())
            locked = next((f for f in ledger["fixtures"] if f["seed"] == seed), None)
            if locked is None or locked.get("sha256") != digest:
                print(f"GATE: hash mismatch for seed {seed}")
                errors += 1
                continue
            if not replay_seed(prop["id"], seed):
                print(f"GATE: property still fails on locked seed {seed}")
                errors += 1
    for freeze in ledger.get("freezes", []):
        if freeze.get("cannot_gate") and freeze.get("required"):
            print(f"GATE: frozen test listed as required: {freeze['id']}")
            errors += 1
    return errors


def cmd_promote(seed: int, property_id: str) -> None:
    FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
    rng = random.Random(seed)
    data = rng.randbytes(32)
    path = fixture_path(seed)
    path.write_bytes(data)
    print(f"wrote {path} sha256={sha256_bytes(data)}")
    print(f"set pre_fix_failing_seeds and locked_by=human for {property_id}")


if __name__ == "__main__":
    if not LEDGER.exists():
        sys.exit("missing seed_ledger.json")
    ledger = load()
    if len(sys.argv) < 2 or sys.argv[1] == "replay":
        sys.exit(1 if cmd_replay(ledger) else 0)
    if sys.argv[1] == "promote":
        cmd_promote(int(sys.argv[2]), sys.argv[3])
        sys.exit(0)
    sys.exit("usage: seed_ledger.py [replay|promote SEED PROPERTY_ID]")

Commands:

python seed_ledger.py promote 180229 parse.no_throw_on_printable
# human copies the printed digest into seed_ledger.json
python seed_ledger.py replay
echo $?   # 0 only when locked seeds pass and hashes match

A minimal CI step looks like this. Adapt the checkout action to whatever you already pin.

# Example job fragment. Not a complete workflow.
steps:
  - name: Replay locked seeds
    run: python seed_ledger.py replay

Do not let a flaky suite retry its way past replay. Retries are how unseeded failures fake a pass.

Decision table

Observation	Ledger action	Merge vote
Property fails on parent with seed S	Promote S to fixture; locked_by=human	Continue to post-patch replay
Property never fails on parent; budget exhausted	Status unearned; keep out of required	Block if this was the only new check
Mutant of the agent fix fails on S; restore passes	Treat S as a calibrated pre-fix seed	Continue
Post-patch still fails on S	Keep fixture; reject the patch	Block
Post-patch passes S; hash unchanged	Log exploration seeds only	This property may vote
Failure only when time or network is unseeded	Freeze; cannot_gate=true	No required vote
Agent rewrites fixture bytes	Hash mismatch	Block

The table is the policy. The script only encodes it.

Drafting properties without polluting the ledger

Candidate invariants can come from a checklist, a human, or a model. The source does not matter. Ownership does. Keep drafts under draft until a human copies an id into properties with a pre-fix seed.

Scratch drafting does not have to live on the merge agent. MonkeyCode's free model access and free server option can host that scratch loop so draft text stays off the locked fixture paths. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate still reads only the human-owned ledger.

One rule is enough. A model may propose a property function. It may not write locked_by, sha256, or cannot_gate.

Limitations

This workflow assumes you can check out a parent commit and re-run a generator. It does not replace a fuzzer for memory unsafety. It does not prove an invariant. It only records that the invariant was once false on a named input and is now true on that same input.

Hash-locked fixtures go stale when the input schema changes. Plan a human rotation. Do not auto-refresh hashes from agent output.

An integer seed is not a substitute for recorded bytes. If the generator is not deterministic across platforms or library versions, store the payload. Replay the payload. Keep the seed as an index, not as the artifact.

Frozen flakes can hide real races. Sample them in a separate, non-gating job. That job must not be in required.

Who should not use this

Do not use a seed ledger as the only control on safety-critical or compliance-bound code. A recorded seed is a regression handle, not a proof.

Do not apply it to screenshot or layout tests whose “seed” is a GPU driver. Those need a different freeze policy.

Do not apply it if the patch has no stated invariant. A ledger full of unearned properties is noise. Classify the patch first.

Skip a remote drafting loop if fixtures contain secrets or production payloads. Keep those on a machine you already trust.

What to count

Track three integers per week. Do not track pass rate as the primary number.

1. Properties that have at least one pre-fix failing seed.
2. Hash mismatches caught at replay.
3. Freezes created versus freezes retired.

If (1) is zero, the suite is not testing agent patches. If (2) stays zero while agents still touch fixtures, the hash is not in the gate. If (3) only grows, flakes are being hidden instead of seeded.

A property that cannot fail on a named seed cannot protect a merge. Lock the seed first.