Hash the Sampler Output Before You Freeze an Agent-Patch Failure

An agent-patch failure is not a flake until two conditions hold. The sampler output must share one digest, and the red signal must be harness noise rather than an oracle miss. Property misses and fixture drift stay unfrozen. The artifact below is a decision table plus a local classifier you can run with the standard library.

Same red status, different clocks

An agent-patch job usually runs on three clocks. The generator samples bytes. A fixture file stores those bytes. A property oracle then checks the patched code against the file, not against a fresh sample.

CI often prints one word for all three failures: FAILED. A timeout, a changed completion, and a broken invariant then look identical in the job list. Collapsing them trains the suite to mute real defects.

The digest step sits before any freeze decision. Without it, a second run is not a rerun. It is a new input that happens to share a test name.

Four classes, one door

Label the pair of runs before you write a freeze record. Only one label may open the door.

Class	Fixture digests	Oracle on that digest	Transport	Freeze
property_miss	equal	fail and fail	ignored	no
fixture_drift	unequal	not comparable	ignored	no
harness_flake	equal	no recorded fail	at least one fault	yes, request only
unclassified	missing, or equal with a split oracle	unclear	unclear	no

Read the rows left to right. Equal digests make the oracle comparable. Unequal digests mean the generator moved, so a freeze would rename a new input as infrastructure noise.

A double fail on one digest is a patch problem or a weak property. It stays a miss even if the server was slow on the same run. A split oracle with no transport fault is not a flake either. That pair stays unclassified until a separate replay policy exists. This note does not define shrinking, replay budgets, or witness size.

harness_flake is narrow on purpose. The digest matches. No oracle result is False. At least one run records a transport fault, such as a timeout, a connection reset, or a completed call that fails a length check you declared in advance. The classifier may then allow a freeze request. It still does not set the expiry, and it does not merge the patch.

Numbered workflow

1. Persist sampler bytes before any assertion

Write the raw completion to a fixture path. Hash the bytes with SHA-256. Store the digest beside the run id.

Do this when the body is empty, too. A hash of zero bytes is a real pin. A missing hash is a different state, and the classifier treats missing as unclassified.

Confirm the pin with the standard library before the job continues:

python -c "import hashlib,pathlib; p=pathlib.Path('fixtures/sample.json'); print(hashlib.sha256(p.read_bytes()).hexdigest())"

A 64-character hex line means the file is eligible for comparison. A missing path means the run is not eligible for a freeze, regardless of how red the log looks.

2. Point the property oracle at the file

The oracle takes a path. It does not call the model, and it does not open a socket to the generator host.

If a property needs a second sample, give that sample its own fixture id. A hidden live call inside the oracle makes the digest meaningless, because the checked input is no longer the pinned file.

3. Record transport on its own fields

Keep the transport record small and explicit. Suggested fields are transport_ok, a status code, and a timeout flag. Fill them in the client, not by parsing the oracle message.

An oracle line that says a key is missing can be a product bug in the patch. Inferring a flake from that string will launder the bug. If the client retried silently, surface the retry count as its own field. A hidden retry makes transport_ok look clean and pushes a real harness fault into unclassified, which is the safer misclassification.

4. Classify the pair, then open at most one door

Load two run records. A missing digest stops at unclassified. Different digests stop at fixture_drift and should open a fixture-refresh ticket, not a freeze.

Matching digests and two oracle failures stop at property_miss. Keep the patch blocked. A single oracle failure beside a pass is also blocked from the freeze door, even if transport was dirty. The miss might be real. Only the narrow harness_flake row may set the freeze flag to true.

5. Leave expiry and merge rights outside the module

The function returns a class and a boolean. It does not choose a duration, a quota, or a machine size. Put expiry, owner, and unfreeze conditions in the workflow config reviewers already read.

A true boolean means this red result may be muted until that external expiry. It does not mean the patch is correct, and it does not clear a property miss from another job.

Example classifier

The module is a self-contained example for local reproduction. It does not call a hosted model, and it is not a report of a production run. Treat the control flow as the spec. If you change a return, update the driver in the next section in the same commit.

import hashlib
from dataclasses import dataclass
from enum import Enum

class FailureClass(str, Enum):
    PROPERTY_MISS = "property_miss"
    FIXTURE_DRIFT = "fixture_drift"
    HARNESS_FLAKE = "harness_flake"
    UNCLASSIFIED = "unclassified"

@dataclass(frozen=True)
class RunRecord:
    fixture_sha256: str | None
    oracle_passed: bool | None
    transport_ok: bool | None

def pin_fixture(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()

def classify(left: RunRecord, right: RunRecord) -> FailureClass:
    if not left.fixture_sha256 or not right.fixture_sha256:
        return FailureClass.UNCLASSIFIED
    if left.fixture_sha256 != right.fixture_sha256:
        return FailureClass.FIXTURE_DRIFT
    if left.oracle_passed is False and right.oracle_passed is False:
        return FailureClass.PROPERTY_MISS
    any_miss = left.oracle_passed is False or right.oracle_passed is False
    transport_bad = left.transport_ok is False or right.transport_ok is False
    if any_miss:
        return FailureClass.UNCLASSIFIED
    if transport_bad:
        return FailureClass.HARNESS_FLAKE
    return FailureClass.UNCLASSIFIED

def freeze_allowed(kind: FailureClass) -> bool:
    return kind is FailureClass.HARNESS_FLAKE

The final return is a closed door. A green pair should not be passed into classify at all. If it is, refusing the freeze is still the right default. There is nothing to mute.

Commands that lock the four rows

Save the module as freeze_gate.py. The driver uses only the standard library. Python 3.11 or newer accepts the str | None hints. On 3.9 or 3.10, switch those hints to Optional from typing and leave the control flow alone.

The driver is an unexecuted example in this draft. Run it locally before you trust a fork of the module.

python - <<'PY'
from freeze_gate import FailureClass, RunRecord, classify, freeze_allowed, pin_fixture

digest = pin_fixture(b'{"patch":"return sorted(xs)"}')
other = pin_fixture(b'{"patch":"return xs"}')

cases = {
    "property_miss": (
        RunRecord(digest, False, True),
        RunRecord(digest, False, True),
        FailureClass.PROPERTY_MISS,
    ),
    "fixture_drift": (
        RunRecord(digest, False, True),
        RunRecord(other, False, True),
        FailureClass.FIXTURE_DRIFT,
    ),
    "harness_flake": (
        RunRecord(digest, None, False),
        RunRecord(digest, True, True),
        FailureClass.HARNESS_FLAKE,
    ),
    "split_oracle": (
        RunRecord(digest, True, True),
        RunRecord(digest, False, True),
        FailureClass.UNCLASSIFIED,
    ),
}

for name, (left, right, expected) in cases.items():
    got = classify(left, right)
    assert got is expected, (name, got)
    assert freeze_allowed(got) is (expected is FailureClass.HARNESS_FLAKE)
print("four local cases passed")
PY

Expected stdout is four local cases passed. An assertion error prints the case name and the class you actually returned. Do not edit the expected class to match a weaker function. Edit the function, or admit the row is a different policy.

How to read a disagreement

Take harness_flake first. The fixture hash matches, so both runs saw the same sampler bytes. The first run never reached the oracle because transport failed. The second run reached the oracle and passed. The red job is eligible for a freeze request. It is not eligible for a silent permanent skip.

Take property_miss next. Same hash, two oracle failures, transport marked healthy. Slow hardware would not change the row. The invariant failed on the pinned input. Block the patch.

Take fixture_drift third. The completions differ by one expression in the example, and the digests differ. Refresh the fixture or allocate a new fixture id. Writing a freeze here hides generator movement inside an infrastructure label.

Take split_oracle last. Same bytes, clean transport, pass beside fail. That pattern can be a clock inside the property, a race in the code under test, or an oracle that still calls out. Freezing it would protect the wrong layer. Leave the row unclassified and fix the test until the oracle is a pure function of the fixture file.

Where a free model lane fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free model lane on MonkeyCode is a usable sampler for step 1. A free server lane there is a usable place to execute step 2. Both are availability options supplied for this draft. Neither option makes two completions the same input, and neither option turns a property miss into a flake.

Keep the sampler call inside step 1. Persist the bytes, compute the digest, and stop using the network. Step 2 should still pass with egress disabled. If it does not, the oracle is coupled to the generator, and the table does not apply until that coupling is gone.

No model name, token quota, hardware shape, or retention window is stated here. Those figures were not verified for this draft. If a console shows a limit, treat the limit as a constraint on how you capture step 1, not as proof that a later completion matches the fixture.

If generator calls are already split from assertions, that free model lane and that free server lane are enough to try the classifier on a scratch branch before it is wired into a merge gate.

Limitations

The digest does not score property quality. A check that cannot fail will pass on a pinned file and still test nothing. Review properties separately. Reject any check whose expected result is implied by the assertion text alone.

Fixture stores grow. Content addressing removes duplicate bytes, but it does not delete abandoned pins. Put retention in the store policy. Do not hide deletion inside classify, or the class of an old run will change under you.

Transport flags are only as honest as the client. Silent retries, rewritten status codes, and bodies truncated after a 200 will poison the row. Prefer unclassified over a guessed flake when those fields are absent.

This example does not shrink counterexamples, does not choose an expiry, and does not grant merge rights. A reviewer still owns the patch. The job summary should show the class even when the boolean is false, so a miss is visible beside a muted harness fault from another shard.

Who should skip it

Skip the gate when tests have no separable sampler input. A pure unit suite can decide flakes without fixture digests.

Skip it when the property must call a live model by design. Pinning will fight that design on every run, and the drift row will dominate for reasons that are not actionable.

Skip it when a freeze is treated as approval. The freeze flag is a mute request with an external expiry. It is not a safety claim about the patch, the model, or the server.

Fields to print in the job summary

Print four lines. A summary that contains only FAILED cannot support the table.

fixture_sha256=<64 hex chars>
failure_class=harness_flake
freeze_allowed=true
fixture_path=fixtures/<digest>.json

Add those lines before you add another retry wrapper. Retries without a digest compare the wrong thing, then ask the freeze door to forgive the comparison.