Cite a Flake Freeze Only After the Fixture Digest Closes

An agent patch may cite a flake freeze only after a binding record closes. The record ties three artifacts computed from the pull request tree: a property scope id, a fixture digest, and the hash of a retained counterexample. A repeated failure line in a log is not one of those artifacts. If any field is missing, or if it was copied from another tree, the citation is rejected and the patch stays open.

This is a proposed review gate, not a measured rollout. No pass rate, quota, or runner size is claimed below. The commands and the checker are unexecuted examples. Point them at your own paths before you treat an exit code as evidence.

What must agree

A property scope names the predicate and the input domain the patch claims to hold. A fixture digest hashes the generator source, the scope file, and the seed lock. A retained counterexample is the failing input saved as bytes, not reconstructed from stdout.

Those three have to name the same scope. A digest from last week's generator cannot authorize a freeze on this week's predicate. A counterexample that does not hash to the citation cannot authorize it either.

Step 1: Read scopes from the diff

Start with the patch. List property and fixture paths before you open the CI log. A log can only describe a run that already happened. The gate needs the files the run was supposed to use.

git fetch origin main
git diff --name-only origin/main...HEAD -- tests/properties tests/fixtures tests/bindings

Each changed property file needs a scope document in the same diff. The document is data. Do not ask the gate to recover a domain from a docstring.

scope_id: order_total.non_negative
predicate: tests/properties/order_total.py:test_total_non_negative
domain:
  currency: [USD, EUR]
  qty: {min: 1, max: 20}
generator: tests/fixtures/gen_orders.py
schema_version: 3

Reject the patch when the predicate or the domain changed and the scope file did not. That split is how a freeze citation keeps an old id while the check underneath it moves.

The predicate itself stays small. This copy is an unexecuted illustration, not a test that has been run in CI.

def test_total_non_negative(case):
    total = case["qty"] * case["unit_cents"]
    assert total >= 0

A failure here can mean a bad fixture, a missing production guard, or a domain that allows negative unit prices. The binding record does not pick among those causes. It only stops a freeze from citing a scope the patch did not lock.

Step 2: Hash generator bytes, not generated output

Hashing the generated JSON is weaker than hashing the generator. An agent can leave yesterday's output file untouched and still rewrite the script that will run on the next seed. The digest has to cover the script.

python - <<'PY'
import hashlib, pathlib
parts = [
    "tests/fixtures/gen_orders.py",
    "tests/properties/order_total.scope.yaml",
    "tests/fixtures/seeds.lock",
]
h = hashlib.sha256()
for p in parts:
    blob = pathlib.Path(p).read_bytes()
    h.update(p.encode())
    h.update(bytes([0]))
    h.update(blob)
print(h.hexdigest())
PY

Store the hex only after Step 3 retains a counterexample. A digest file with a null counterexample hash is an open record. Do not commit it, and do not let a freeze file point at it.

Include the schema version in the hashed scope file so a format change alters the digest. Two runners can disagree after a silent parser change while an old hex stays green. If you bump the version, recompute the digest in the same commit.

Step 3: Retain one counterexample from a bounded run

Run the property check with the seed and the domain from the scope file. Cap the example count. An unbounded search makes the retained input unreproducible even when the seed is fixed, because the stop condition moves.

PROPERTY_SCOPE=tests/properties/order_total.scope.yaml COUNTEREXAMPLE_OUT=tests/bindings/order_total.non_negative.cex.json python -m tools.property_run --seed 17 --max-examples 200

On failure, the proposed runner writes a single JSON object. Passing runs write nothing. A freeze candidate cannot be born from a pass.

{
  "scope_id": "order_total.non_negative",
  "seed": 17,
  "example_index": 44,
  "input": {"currency": "EUR", "qty": 3, "unit_cents": -1},
  "predicate_result": "fail",
  "shrink_steps": 6
}

Then hash that file. The citation will carry this hex, not the example index. Indexes shift when the generator changes. Bytes do not.

python -c "import hashlib, pathlib; p=pathlib.Path('tests/bindings/order_total.non_negative.cex.json'); print(hashlib.sha256(p.read_bytes()).hexdigest())"

If the runner cannot shrink to one input, stop the workflow. Multi-process races and live network fixtures do not belong in this record. Quarantine them under a different policy.

Step 4: Classify that file on a clean worktree

Copy the counterexample and the scope into a second worktree at the same commit. Evaluate the predicate on that input only. Do not rerun the generator. A fresh sample can miss the retained case and make an unstable check look closed.

git worktree add --detach /tmp/bind-check HEAD
cp tests/bindings/order_total.non_negative.cex.json /tmp/bind-check/tests/bindings/
python -m tools.property_eval --scope tests/properties/order_total.scope.yaml --case tests/bindings/order_total.non_negative.cex.json

Label the worktree result class_local. A second environment you control may supply class_isolated. Both labels must describe the copied file, not a newly generated sample.

A disagreement between the labels leaves the record open. Fix the fixture, the predicate, or the runner image before anyone writes a citation. Do not paper over the split with a broader freeze reason string.

Step 5: Write the citation from the closed triple

The citation is data. It names the triple and the two class labels. It does not name a test the patch just created, and it does not paste a log excerpt in place of a hash.

{
  "scope_id": "order_total.non_negative",
  "fixture_digest": "<sha256 from step 2>",
  "counterexample_sha256": "<sha256 of the cex file>",
  "class_local": "fail",
  "class_isolated": "fail",
  "freeze_reason": "retained_input_failed_in_both_classifiers"
}

Leave expiry to whatever policy the repository already enforces. This gate does not invent a lifetime. If you add an expiry field, compute it from that policy in the same commit as the citation. A date typed by the agent with no rule beside it is not a control.

Decision table

Read this table left to right. Only one cell authorizes a citation. Do not collapse the other rows into a mostly-failed judgment.

Local class	Isolated class	Digest equals patch bytes	Gate result
pass	pass	yes	No citation. The retained input holds.
fail	fail	yes	Citation allowed for this scope id only.
fail	pass	yes	Reject. Environment split.
fail	fail	no	Reject. Recompute the digest in CI.
fail	absent	yes	Reject. Classifier missing.
absent	any	any	Reject. No counterexample to bind.

The gate is boolean on the triple. A reviewer note cannot flip a reject row to allow. Update the files, recompute the hashes, and run the checker again.

Proposed checker

This script is unexecuted proposal code. It compares three local files. It does not call a model, and it does not decide whether the predicate is meaningful.

#!/usr/bin/env python3
"""Reject a freeze citation whose binding record is open."""
import hashlib, json, sys
from pathlib import Path

def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def main() -> int:
    freeze = json.loads(Path(sys.argv[1]).read_text())
    binding = json.loads(Path(sys.argv[2]).read_text())
    cex = Path(sys.argv[3])
    keys = ("scope_id", "fixture_digest", "counterexample_sha256")
    bad = [k for k in keys if freeze.get(k) != binding.get(k)]
    if bad:
        print("binding mismatch:", ",".join(bad))
        return 2
    if freeze.get("class_local") != "fail" or freeze.get("class_isolated") != "fail":
        print("citation requires fail on both classifiers")
        return 3
    if sha256_file(cex) != freeze["counterexample_sha256"]:
        print("counterexample bytes differ from citation")
        return 4
    print("binding record closed")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

python tools/check_binding.py tests/freezes/order_total.non_negative.freeze.json tests/bindings/order_total.non_negative.json tests/bindings/order_total.non_negative.cex.json

Call that from the pull request job. A review comment that says the failure looks intermittent does not close the record. The exit code does, and only for the files it hashed.

Pin tools/check_binding.py to a trusted revision when the same patch is agent-authored. Otherwise the diff can weaken the comparison and then satisfy it. Review that hunk outside the generated change, or load the checker from a ref the agent cannot edit.

Drafts from a free model, replay on a free server

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator notes supplied for this draft say that MonkeyCode provides free model access and a free server option. No model name, quota, hardware shape, duration, or benchmark is stated here. Those details were not provided, and they change. Confirm the current terms before you wire either option into a required check.

Use free model access, when you have it, to draft a candidate scope file or a first predicate. Keep the draft untrusted. The digest hashes the text you merge, including a confident domain that omits a currency production code accepts. A fluent YAML file is not a class label, and it is not a closed record.

Use a free server, when you have it, as one place to compute class_isolated in Step 4. Replay the retained counterexample you uploaded. Do not ask that server to regenerate the fixture suite or to write the freeze file.

If the server cannot return the same case file, treat class_isolated as absent and follow the table. Missing evidence is a reject, not a skip. A remote label never replaces the local digest computed from the pull request tree.

A workable split is narrow. Let the model propose the quantity bound in the scope file. Have a reviewer confirm that bound against the production invariant before the digest is computed. Generation stays cheap. The binding stays in the repository you control.

Who should not use this gate

Skip it when the failure cannot be stored as one input file. Timing-sensitive UI checks, multi-process races, and tests that require a live third party will not yield a stable counterexample. Forcing them into the schema produces fake precision. Use a different quarantine, or drop those tests from the agent patch.

Skip it when CI cannot recompute the digest from the pull request tree. A hex pasted from a laptop is the case the checker is meant to catch. If your runners mutate fixture files before hashing, fix that mutation first. Otherwise the digest column is theater.

Skip it when nobody will read the predicate. The checker can prove the citation matches the files. It cannot prove the property is non-tautological.

A predicate that asserts true after catching every exception will bind cleanly and still tell you nothing. Read the predicate on every agent patch that adds one. Do not replace that read with another freeze file.

What can merge

Merge when the scope id, the fixture digest, and the counterexample hash agree, and when any freeze citation is limited to that triple. Keep the patch open when the citation names a digest CI cannot recompute from the same commit. The production change can still be correct. It is not reviewable as a freeze until the record closes.

If you already use MonkeyCode's free model access to sketch scope YAML, leave the checker in the repository and run it on the pull request tree. The sketch can shorten the first draft. It should not be the step that marks the binding closed.

Comments

[DEV SUPPORTS]

You need to verify your account.

tr.ee/dev-to