Class the Shrunk Witness Before a Property Failure Can Be Frozen

A property failure is not a flake freeze. It becomes eligible for a freeze only after the input is shrunk, retained, replayed on a pinned fixture, and classed as unstable rather than consistently wrong. A witness that fails on every replay is a regression. Silencing it paints the suite green without making the patch true.

Agent patches miss this distinction in a mechanical way. Production code changes. The same diff then relaxes the property that went red. CI still prints the old test name, and the name now passes. Line coverage often stays flat because the lines still ran. They ran a weaker statement.

The workflow below is a gate, not a prompt for more tests. Classify the shrunk input first. Freeze only the flake class. Send oracle edits to another review. If one patch touches both the implementation and the checker, refuse the freeze outright.

What a freeze is allowed to hide

A freeze is a temporary silence for one assertion and one retained payload. It expires. It is not a file-level skip, and it is not permission to rebuild fixtures until the red case disappears from the log.

Bind that silence to the witness hash, not to the test name. Agents are good at keeping names. They are also good at swapping generators. The log line looks familiar while the input that used to fail is gone. If the shrunk payload still fails under the pin, there is nothing honest to freeze.

Four labels before any ignore file exists

Leave the run unclassified until a witness file is on disk. The sample policy then emits one label.

Label	Evidence the harness requires	Next step
regression	The same witness fails on every replay, and implementation paths changed	Block the patch. Do not write a freeze file.
flake	The same witness fails on some replays and passes on others, with one fixture digest	Write a freeze bound to the assertion, the digest, and the witness hash.
oracle_defect	Implementation paths are unchanged, and the witness contradicts the spec	Open a separate oracle review. No freeze on the implementation ticket.
mixed_edit	Implementation paths and oracle paths both change	Block. Split the diff before classification is trusted.

These rows are policy. They are not telemetry. This article does not report a flake rate, a model score, or a suite percentage. The replay count later is a constant you set in review, then check into the harness.

1. Pin the fixture before anyone classifies

Hash the fixture bytes and record the generator version. A run without that lock stays unclassified. Do not ask a model to invent the digest from a summary.

python3 -m witness_gate pin \
  --fixture tests/fixtures/orders.json \
  --generator-version 3 \
  --out .artifacts/fixture.lock

The lock is an input to later steps. Store it outside the patch diff so one commit cannot refresh the pin and the witness together. Reviewers compare the lock digest to the fixture blob in the job, not to a digest pasted into the pull-request description.

2. Shrink the failure and retain the payload

On the first property failure, shrink to the smallest payload the checker still rejects. Store the payload and a canonical hash. Editing the assertion until the failure vanishes is not minimization. It is an oracle edit.

# Proposed harness for review. Not a log from a finished rollout.
import hashlib
import json
from pathlib import Path

def witness_hash(payload: dict) -> str:
    raw = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(raw).hexdigest()

def retain(assertion_id: str, payload: dict, fixture_digest: str, out_dir: Path) -> dict:
    digest = witness_hash(payload)
    record = {
        "assertion_id": assertion_id,
        "fixture_digest": fixture_digest,
        "witness_hash": digest,
        "payload": payload,
    }
    path = out_dir / f"{assertion_id}.{digest[:12]}.json"
    path.write_text(json.dumps(record, indent=2, sort_keys=True) + "\n")
    return record

Take the shrink seed from a schedule stored beside the lock. Do not take it from the wall clock. A witness that cannot be reloaded byte for byte on a second machine is not evidence a reviewer can re-run.

The checker itself should be able to fail closed. This example rejects a negative line total and a claimed total that does not match the lines.

def prop_order_total_non_negative(order: dict) -> bool:
    total = sum(line["qty"] * line["price"] for line in order["lines"])
    if total < 0:
        return False
    return total == order["claimed_total"]

A generator that only emits non-negative qty and price never hits the first branch. That is a weak property, not proof the patch is safe. Keep at least one explicit negative payload in the witness set even when the random generator prefers valid orders.

3. Replay that payload, not a fresh draw

The original property run asks whether some input fails. Replay asks whether this input fails reliably. Those are different questions. Mixing them is how a new random seed gets mislabeled as a fix.

def classify(failures: int, runs: int, impl_changed: bool, oracle_changed: bool) -> str:
    if runs < 1 or failures < 0 or failures > runs:
        return "unclassified"
    if impl_changed and oracle_changed:
        return "mixed_edit"
    if oracle_changed and not impl_changed:
        return "oracle_defect"
    if failures == runs:
        return "regression"
    if failures == 0:
        return "not_reproduced"
    return "flake"

The sample job uses five replays. Five is a review constant in the example. It is not an empirical optimum, and it is not a host quota. If any replay records a different fixture digest, discard the set and pin again. Do not average results across pins.

not_reproduced is not a pass. It means this witness no longer demonstrates the original failure. Shrink distortion is a common cause. Keep the unshrunk input and replay that before you declare the red run gone. Do not convert "could not reproduce" into a freeze file.

A table-driven check keeps the labels from drifting when someone later edits the function.

def test_classify_table() -> None:
    cases = [
        (5, 5, True, False, "regression"),
        (2, 5, True, False, "flake"),
        (0, 5, True, False, "not_reproduced"),
        (5, 5, False, True, "oracle_defect"),
        (2, 5, True, True, "mixed_edit"),
        (3, 0, True, False, "unclassified"),
    ]
    for failures, runs, impl, oracle, expected in cases:
        assert classify(failures, runs, impl, oracle) == expected

4. Write a freeze only for the flake label

def freeze_allowed(kind: str, witness_present: bool, digest_matches: bool) -> bool:
    return bool(witness_present and digest_matches and kind == "flake")

The freeze record needs an assertion id, a fixture digest, a witness hash, an expiry, a failure count, and a run count. Drop any field and the writer exits non-zero. A paragraph in the pull request is not a record. It cannot be replayed, and it cannot be expired by a job.

python3 -m witness_gate freeze \
  --assertion prop_order_total_non_negative \
  --lock .artifacts/fixture.lock \
  --witness .artifacts/witnesses/prop_order_total_non_negative.ab12cd34ef56.json \
  --failures 2 \
  --runs 5 \
  --expires-in-hours 24 \
  --out .artifacts/freeze.json

The 24-hour flag gives the command a concrete shape. It is not a studied best duration, and it is not a limit of any hosting plan. Pick an expiry your reviewers will actually reopen. A freeze nobody reopens is a skip with extra paperwork.

5. Split implementation paths from oracle paths

IMPL_PREFIX = "src/"
ORACLE_PREFIX = "tests/properties/"

def split_paths(paths: list[str]) -> tuple[bool, bool]:
    impl = any(p.startswith(IMPL_PREFIX) for p in paths)
    oracle = any(p.startswith(ORACLE_PREFIX) for p in paths)
    return impl, oracle

Feed split_paths from the changed-file list. Do not feed it a model summary of the diff. If both flags are true, stop with mixed_edit. The agent has a direct motive to weaken the checker in the same commit that needs the checker. Separate commits restore an order a human can read: spec review first, or code review against an unchanged oracle.

Prefixes are local convention. Set them before you enable the block. A tree that stores properties under src/ will mislabel every change until the prefixes match the repository. Generated stubs that the agent rewrites on each run should be treated as oracle paths, or excluded from the patch entirely.

6. Re-run the same commands in review

python3 -m witness_gate pin \
  --fixture tests/fixtures/orders.json \
  --generator-version 3 \
  --out .artifacts/fixture.lock
python3 -m witness_gate check \
  --lock .artifacts/fixture.lock \
  --properties tests/properties \
  --out .artifacts/witnesses
python3 -m witness_gate classify \
  --lock .artifacts/fixture.lock \
  --witnesses .artifacts/witnesses \
  --diff-paths changed.txt
python3 -m witness_gate freeze \
  --assertion prop_order_total_non_negative \
  --lock .artifacts/fixture.lock \
  --witness .artifacts/witnesses/prop_order_total_non_negative.ab12cd34ef56.json \
  --failures 2 --runs 5 --expires-in-hours 24

changed.txt holds one repository path per line, taken from the review diff. Classify reads that list. If the label is regression or mixed_edit, stop. The sample interface has no force switch. Adding one would recreate the hole this gate is meant to close.

7. Reviewer checks before approval

1. Match .artifacts/fixture.lock to the fixture blob in the job. Ignore a digest that appears only in the description.
2. Load the witness payload and evaluate prop_order_total_non_negative on it again. A screenshot of a green job is not the payload.
3. Compare the classify label to the table. A flake result with failures == runs is a caller bug, not a borderline flake.
4. Reject the patch when changed.txt contains both prefixes, even if the latest property run is green.
5. If a freeze file is present, require expiry, assertion id, and witness hash. A freeze without a witness hash is invalid under this policy.

Where a drafter may sit, and where it may not

The verdict function stays in the repository. A language model can draft the property stub and a short note that quotes the witness JSON. It should not be the process that selects the label.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access fits that drafting step. The free server option fits the replay job, so the fixed replay loop is not tied to one laptop. Those are availability claims only. This article does not name a model, a quota, a machine size, a runtime cap, or a promise that the free option is permanent.

If the harness already runs in CI, point the free server at witness_gate check and keep classify on the protected runner. Read the drafted note when it quotes the witness file. Discard it when it disagrees with that file. The gate still decides.

Limitations to keep on the policy page

Shrinking distorts timing bugs. A race on a large fixture can minimize into a tiny payload that no longer races, then land in not_reproduced. Store the unshrunk failure beside the witness when the property touches clocks, locks, or process scheduling.

A fixed replay count misses rare flakes. This ticket is for failures you can provoke again under the pin. Leave rare cases on the suite's ordinary retry policy. Do not open a freeze to burn down a count of red builds.

Hashing a live HTTP body is not a pin. Tests that need the public network, a shared staging tenant, or a clock outside the process are out of scope until those inputs exist as bytes you can hash.

The path split fails open when prefixes are wrong, and it fails closed when generated files are mislabeled as oracles. Review the prefix list when code generation lands. Otherwise the block becomes noise, and reviewers start bypassing it.

Who should not use this

Skip the workflow if fixtures are rebuilt on every job and have no stable bytes to hash. Skip it if witnesses cannot be stored as review artifacts. Skip it if policy requires the spec and the implementation to change under a single approval. A freeze file in that setting is a second, weaker approval path.

Also skip it when the suite has no properties. There is nothing to shrink and nothing honest to freeze. Add one checker that can fail closed before you add ignore machinery.

CODEOWNERS on the oracle directory complements this gate. It does not replace it, and the gate should not replace owners. If owners already block checker edits, keep that block. Do not add a freeze command those owners cannot see.

The outcome worth keeping is narrow. Consistent witness failures stay red. Unstable witnesses get a short, hashed silence with an expiry a human must reopen. Spec changes wait for their own review. An agent that can edit both sides of the check does not get to call that edit a flake.