Shrink, Replay on Base, Then Promote the Witness

A property failure on an agent patch belongs in a fixture file. It does not belong in a flake freeze.

Shrink the counterexample until the input is small. Replay that seed on the base revision. Write both verdicts into a ledger.

A freeze may cover only residual noise that never produced a stored counterexample. That order is the strategy. The rest is bookkeeping.

Teams lose the witness when both failure classes share one exception file. A property check searches an input space and returns a concrete value when an invariant breaks. An integration test returns an unstable observation from one command.

The witness needs a fixture. The unstable observation may need a time box. Those containers are not interchangeable.

Three records, three jobs

Keep three artifacts, and do not let one answer for another.

1. The property spec names the invariant, the generator bounds, and the shrink budget.
2. The witness ledger stores the minimized input, the base verdict, the patch verdict, and the pin hashes.
3. The freeze file lists non-property tests, an owner, and an expiry. It has no seed column.

A ledger row is evidence. A freeze row is a temporary exception. Evidence outranks the exception. A name that appears in both files makes the freeze entry invalid.

No flake rate, model score, or team outcome is claimed here. The commands and snippets are a proposed harness. They were not executed for this draft.

Pin, run, shrink, promote

Run the steps in order. Skip one, and a real miss can be renamed as noise.

1. Pin the corpus before the agent edits

Hash the fixture directory and the property module on the base revision. Store both hashes in the ledger header. If the agent diff changes either hash, stop. You are no longer replaying the same check.

python -m seedgate.ledger pin --base origin/main --out .seedgate/ledger.json

A pin is a content hash, not a review comment. Generator edits fail closed. Otherwise the next red run can be explained away as a spec move.

2. Execute properties on a fixed seed schedule

Use a recorded seed list for the merge gate. A fresh random stream is fine for local exploration. It is a weak merge signal, because a later reviewer cannot name the miss.

python -m seedgate.props run \
  --seeds .seedgate/seeds.txt \
  --patch HEAD \
  --out .seedgate/props.json

seeds.txt holds one integer per line. props.json stores the seed, pass or fail, and the raw counterexample on failure. Keep the raw value. A summary string is not replayable, and shrinking needs the original witness.

3. Shrink, then replay the base

Shrink before you classify. A 4,000-element list is a poor fixture. A three-element list that still breaks the invariant is a regression case.

python -m seedgate.ledger shrink --props .seedgate/props.json --max-steps 80
python -m seedgate.ledger replay-base --base origin/main --ledger .seedgate/ledger.json

Base replay uses the shrunk input only. It does not rerun the generator. The question is narrow: did this input already fail before the patch?

A shrink step is legal only when the candidate still fails. If it passes, discard it and try another axis. If no axis preserves the failure, stop and keep the last failing value.

4. Promote or quarantine

Promote when the base passes and the patch fails. Quarantine the seed when the base fails too. Quarantine means the generator or the base build is already unstable. It does not clear the agent patch.

python -m seedgate.ledger promote \
  --ledger .seedgate/ledger.json \
  --fixtures tests/fixtures/promoted/
python -m seedgate.freeze check \
  --ledger .seedgate/ledger.json \
  --freeze .seedgate/freeze.json

freeze check exits non-zero when a freeze entry names a property, or when a failing ledger row shares an invariant id with the freeze file. Promotion and freeze closure belong in the same commit. A freeze that survives promotion is a bookkeeping bug.

5. Freeze only the leftover

After promotion, inspect integration tests the diff did not touch. One failure plus two immediate passes can enter the freeze, if an owner and an expiry are present. The freeze must not cite a ledger seed. If it does, the check fails.

Touched tests are outside this exception. An edit plus a failure is a review item, not noise.

Decision table

The first matching row wins. any means the column does not change the action.

Kind	Base	Patch	Diff touched the test	Named in freeze	Action
property	pass	fail	any	any	promote the fixture and delete the freeze entry
property	fail	fail	any	any	quarantine the seed; do not clear the patch
property	unknown	fail	any	any	stop; missing base replay is not a pass
property	any	pass	any	yes	reject the freeze entry
integration	n/a	flake	no	no	freeze candidate only with owner, expiry, and rerun evidence
integration	n/a	fail	yes	any	reject the freeze; review the edit

A base failure does not acquit the patch. The diff may still have changed behavior around a bad seed, so quarantine blocks a green merge until a human separates base debt from the patch. A passing property named in the freeze file is still a policy error. Properties do not belong in that file, even when they are green.

Illustrative harness

This module is a proposal. It was not run in CI for this article. It encodes the table. It does not discover invariants, and it does not execute tests.

from enum import Enum

class Action(Enum):
    PROMOTE = "promote"
    QUARANTINE = "quarantine"
    FREEZE_CANDIDATE = "freeze_candidate"
    REJECT_FREEZE = "reject_freeze"

def decide(kind, base_ok, patch_ok, diff_touched_test, named_in_freeze):
    if kind == "property" and patch_ok is False and base_ok is True:
        return Action.PROMOTE
    if kind == "property" and patch_ok is False and base_ok is False:
        return Action.QUARANTINE
    if kind == "property" and patch_ok is False and base_ok is None:
        return Action.REJECT_FREEZE
    if kind == "property" and named_in_freeze:
        return Action.REJECT_FREEZE
    if kind == "integration" and patch_ok is False and diff_touched_test:
        return Action.REJECT_FREEZE
    if kind == "integration" and patch_ok is False and not diff_touched_test:
        return Action.FREEZE_CANDIDATE
    return None

def shrink_list(values, fails, max_steps=80):
    current = list(values)
    steps = 0
    while len(current) > 1 and steps < max_steps:
        steps += 1
        candidate = current[1:]
        if fails(candidate):
            current = candidate
            continue
        candidate = current[:-1]
        if fails(candidate):
            current = candidate
            continue
        break
    return current

decide returns None when nothing failed. Callers must treat None as "no exception recorded," not as approval of the rest of the diff. FREEZE_CANDIDATE is not a grant. The caller still needs one failure, two immediate passes, an owner, and an expiry before writing the freeze file.

shrink_list only drops list ends. Integers, maps, and nested objects need their own reducers. The shared rule is the preserve check. If fails(candidate) is false, that candidate is illegal.

A promoted fixture is small JSON. Commit it beside the code under test.

{
  "seed": 184729,
  "shrunk": [1, 0, 1],
  "invariant": "fail_count_never_counts_a_pass",
  "base": "pass",
  "patch": "fail",
  "base_sha": "pinned-at-review"
}

The next run loads tests/fixtures/promoted/ before it draws new seeds. A ledger that is not loaded is a note, not a gate. Load order is fixed: promoted fixtures, then the pinned seed list, then optional exploratory seeds. Exploratory seeds cannot block or clear a merge by themselves.

Where a free model and a free server fit

The verdict stays in the ledger commands. A model should not be the system of record for pass or fail. It can still help in two bounded places, and only if you already have access.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft a first reducer for a new invariant. The free server option can host replay-base so a laptop is not the only runner. Those are availability claims only.

This article does not name a model, a quota, a machine size, or a retention period, because none of those details were supplied. Treat the drafted reducer as an untrusted patch. Run it against the raw counterexample. If the shrunk value passes, discard the draft.

A reducer that loses the failure is an unreviewed test edit, not a helper. If a free server seat is already available, point the replay job at that runner and commit only the ledger and the fixtures. The seat is optional. The promotion rule is not.

Limitations

Properties must replay from a serialized input. Timing tests, UI tests, and checks that call a live third party do not shrink cleanly. Leave them out of the ledger. Forcing them in produces fixtures nobody can rerun.

Shrinking can hide a second bug. Keep the raw counterexample for one review cycle after promotion. If the shrunk case passes and the raw case fails, the reducer is wrong, and the fixture is not ready.

Quarantine grows when the base is already red. Cap the list, and block new agent merges until a human clears the base. A long quarantine file means stop generating patches. It does not mean the freeze should grow.

This ledger does not judge deleted assertions, weakened oracles, or rewritten expected files. Those edits need their own checks. A green property run is not a substitute for that review.

Who should skip it

Skip the ledger if inputs cannot be serialized and nobody owns the fixture directory. A freeze file alone will mix witnesses with noise again.

Skip a free-server replay when the diff touches secrets, production data, or credentials. Convenience is not a reason to upload that patch.

Skip model-drafted reducers when the preserve check cannot run on a machine you control. A reducer you cannot falsify should not land.

Promote the seed that failed. Leave the freeze for noise that never earned a fixture row.