Key the Flaky Waiver to a Shrunk Example Hash

A flaky-test freeze is a waiver, not a passing result. For an agent patch, record that waiver only when the diff leaves fixture factories and property strategies untouched, a failed property has been shrunk to a hashed witness, and that witness has been replayed outside the agent workspace. A freeze stored under a bare test name does not meet the bar.

Agent patches often turn a red property green by editing the generator that drew the input, or by narrowing the property until the failing case cannot appear again. The suite looks clean. The production failure is still reachable. A later freeze then points at the old test name and hides the redefined input.

This note is a proposed gate, not a measured study. Commands and snippets below are unexecuted examples. Pin dependency versions in your own lockfile. Do not treat the path lists as a complete map of every repository.

What the gate is allowed to decide

The gate answers one question: may this patch receive a short flake freeze?

It does not decide correctness. It does not score a model. It does not replace review.

Diff class	Shrink witness	Clean replay	Freeze decision
Production paths only	Present, hash matches the stored example	Outcomes flip under a predeclared count, generator hashes match base	Eligible for a short freeze
Production paths only	Absent	Any	Deny
Fixture factory, conftest.py, strategy file, or seed file edited	Any	Any	Deny
Test or property file edited	Any	Any	Deny until a separate line-level oracle review exists
Freeze config edited in the same patch	Any	Any	Deny
Production edits mixed with generator edits	Present	Stable pass	Deny, and split the patch

Read the rows from top to bottom. A later column never overrides an earlier deny. "Short" means whatever expiry your store already supports. This draft does not invent a duration.

Step 1. Classify paths before any test run

Start from the name-only diff against the merge base. Map each path with a policy file you review. Do not ask the drafting model to label its own diff. A drifted label is another unaudited edit.

# Proposed, unexecuted. Replace BASE with your merge-base SHA.
git diff --name-only BASE...HEAD > /tmp/agent-paths.txt

# Proposed classifier. Not executed in this draft.
from pathlib import Path

GENERATOR_PARTS = ("conftest.py", "fixtures/", "strategies/", "hypothesis/")
FREEZE_PARTS = ("freeze/", "flaky_allowlist")
SEED_NAME = "tests/seeds/agent_gate.json"

def classify(path: str) -> str:
    name = Path(path).as_posix()
    if any(part in name for part in FREEZE_PARTS):
        return "freeze_config"
    if any(part in name for part in GENERATOR_PARTS) or name.endswith(SEED_NAME):
        return "generator"
    if name.startswith("tests/"):
        return "test_body"
    return "production"

The function is path policy only. It does not parse assertions, and it will mis-file a generator that lives inside a test function. That miss is why every test_body result denies the freeze until a separate line-level oracle diff is reviewed. If you cannot run that second diff, keep the deny.

Substring matches can false-positive on vendored trees. Prefer anchored prefixes from your own layout if you adopt the sketch. Same path list must yield the same classes. Check the classifier in as code, and fail the job if the policy file is missing. A missing map is not a pass.

Step 2. Draw properties from a checked-in seed list

A fresh random draw on every job cannot support a waiver. You would be freezing a name, not an input. Check in a seed list, and run a prefix of it.

# Proposed seed plan. Unexecuted.
import json
from pathlib import Path

def seed_plan(path: str = "tests/seeds/agent_gate.json", limit: int = 32) -> list[int]:
    seeds = json.loads(Path(path).read_text(encoding="utf-8"))
    if not isinstance(seeds, list) or limit < 1 or limit > len(seeds):
        raise ValueError("seed budget out of range")
    return [int(item) for item in seeds[:limit]]

Thirty-two is a budget knob, not a measured optimum. Raise it when the property is cheap. Lower it when setup is heavy. Write the chosen prefix length into the witness so a later reader can see which seeds were eligible.

Hypothesis, or an equivalent library, should receive those seeds explicitly. If the runner cannot accept a seed, do not open a freeze path for that property. Example tests can still be locked another way. They are outside this gate.

Step 3. Shrink the failure, then hash the witness

On failure, shrink before storing. The witness holds the property id, the seed budget, the smallest example that still fails, and the hashes of the generator files at the merge base.

# Proposed witness writer. Unexecuted.
import hashlib
import json

def witness(property_id: str, seed_budget: int, example: dict, generator_hashes: dict) -> dict:
    payload = json.dumps(example, sort_keys=True, separators=(",", ":")).encode("utf-8")
    if len(payload) > 8192:
        raise ValueError("example exceeds illustrated 8 KiB cap")
    return {
        "property_id": property_id,
        "seed_budget": seed_budget,
        "example_sha256": hashlib.sha256(payload).hexdigest(),
        "example": example,
        "generator_hashes": generator_hashes,
    }

The 8 KiB figure is an illustration of a cap, not a tuned limit. Enforce some cap. An uncapped witness becomes a dump of the workspace. Canonical JSON matters because key order would otherwise change the hash without changing the example.

The freeze key is (property_id, example_sha256, generator_hashes). It is not the test function name. Renaming the test must not keep the waiver. Editing a generator must not keep it either, because those hashes change.

If shrinking hits a step cap, store nothing and deny. An incomplete shrink is not a partial pass.

Step 4. Replay the witness outside the agent workspace

Replay on a clean tree. The agent workspace may contain edited ignore files, a warm cache, or a half-applied format pass. None of that belongs in the waiver decision.

# Proposed replay sketch. Unexecuted. Do not point this at a dirty agent worktree.
git worktree add --detach /tmp/gate-replay HEAD
# Place only the witness file into that worktree, then run the one property
# that knows how to load example_sha256. The -k expression is a placeholder.

Interpret the clean run narrowly. A stable pass means the earlier failure was contamination, or a generator edit the classifier should already have denied. A stable fail on the same hash is a real counterexample. Fix the product code, or reject the patch. Do not freeze it.

A freeze is eligible only when that same witness flips between pass and fail across a count written in policy, and the generator hashes still match the base. This draft states no flip rate. Choose a count the queue can afford, record the outcomes, and fail closed when the count is incomplete.

A review note can use this shape. Values below are placeholders, not a captured run.

{
  "diff_class": "production",
  "property_id": "order_total_non_negative",
  "example_sha256": "<64 hex characters>",
  "generator_hashes_match_base": true,
  "replay_outcomes": ["fail", "pass", "fail"],
  "freeze": "eligible_short"
}

Three outcomes are an example policy, not a recommendation derived from data. If your store cannot attach this note to the waiver, do not record the waiver.

Where drafting and replay hosting fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate does not need a model. Drafting is a separate lane. Operator-supplied availability for this workflow is limited to two facts: MonkeyCode offers free model access, and it offers a free server option.

This article does not assert model names, quotas, hardware, durations, or permanence. If either option is missing on a given day, freezes that lack a witness stay denied.

Use free model access only to draft the production diff. Pass the shrunk witness in as the failing example, and keep generator paths out of the draft instructions. After the draft, run the classifier again. A draft that touches conftest.py or the seed file is discarded, not frozen.

Use the free server as the clean replay host when the replay should not share a machine with the patch build. The server is a workspace boundary, not an oracle. If it cannot accept the job, leave the freeze unset. A busy queue is not a reason to promote the patch.

Readers who already separate drafting from admission can try that split with those two options, then keep classification and hashing local. The property checks stand if the product is never called.

Limitations

Path rules miss generators written inside test functions. That is why test-file edits deny the freeze by default. Shrink may not finish, so the runner needs a step cap and must treat an incomplete shrink as deny. A clean host can still share DNS, clocks, or a container image with the agent job, so an environmental flake can survive the move.

Hash equality does not prove the property is meaningful. A tautology shrinks to a trivial example and still hashes. No false-accept rate is reported here. The table is a policy. Teams that need a rate have to run their own mutation sample and publish the method. Do not import a rate from a different gate.

Who should skip this

Skip the gate when the suite has no properties and only snapshot tests. A snapshot has no shrink step this key can use. Skip it when the failure depends on a physical device or a third-party UI with no serializable input. Skip it for patches that touch authentication, cryptography, or migrations.

Those diffs need review, not a waiver. Skip the gate if the freeze store can key only on a test name. A name-only store recreates the hiding problem this policy closes. Also skip it when the seed file is not reviewed. An agent that rewrites tests/seeds/agent_gate.json has changed the draw even if no factory path appears in the diff. Put that file in the generator class before you trust the table.

Closing check

Before a waiver is written, the review note needs four facts: the diff class is production-only, the witness hash is present, generator hashes match the base, and the clean replay flipped under the predeclared count. A missing fact keeps the freeze closed. The patch can still be repaired. It cannot be excused by a test name.