Only a Flake-Class Exit May Open the Lease

An agent-patch failure should leave the pipeline as one class, not a blended skip. The classes are a property violation, a fixture miss, and a flake candidate. Only the flake candidate may become a time-boxed freeze, and only after the property verdict on that same pin is pass.

Anything broader hides a regression inside a waiver. The workflow below is a gate: a decision table, a six-step run order, and a classifier that refuses a lease when the inputs do not match. The code is a proposal. It was not executed against a production suite for this article.

What each class is allowed to do

A property violation means an invariant the patch was supposed to preserve did not hold. A typical case is a public function that must still reject a negative quantity, or a record that must still round-trip through its serializer. That class blocks merge. Frequency does not matter. A check that fails once and a check that fails on every run are both ineligible for a freeze.

A fixture miss means the run is not comparable to the pin. The digest, the schema version, or the oracle file diverged. The patch may be innocent. The run is still invalid. Re-pin under review, or reject the job. Do not freeze a miss, and do not update the oracle inside the same job that is asking for a waiver.

A flake candidate is the narrow remainder. The property verdict passed. The pin matched. A bounded replay disagreed with the original red result. That pattern can justify a lease with an owner and an expiry. The lease does not delete the test. It does not transfer to the next failure.

Decision table

Read the table as a gate. If two rows seem to match, take the stricter exit. Missing fields are incomplete, not flake.

Property verdict	Fixture pin	Replay witness	Exit	Freeze allowed
fail	match	any	2 property	no
pass or fail	mismatch	any	3 fixture	no
pass	match	disagreement inside a full budget	4 flake	yes, with expiry and owner
pass	match	red on every replay	5 stable red	no
missing	any	any	1 incomplete	no

Exit 5 is the row that gets skipped in review. A failure that returns on every replay is not a flake. It is a stable red. A passing property does not waive a different test that is stably red. Classify that test on its own row. Borrowing the property pass is how a lease swallows an unrelated regression.

Workflow

1. Pin the fixture before the agent patch is applied. Record the digest, the schema version, and the command that produced the pin. The patch job refuses to start when that record is absent. This step creates a comparison point. It does not by itself sign a verdict.
2. Run property checks against that pin. Keep the predicate source in the repository, not in a model transcript. A failing property exits 2 and stops the pipeline. Do not start a replay in order to talk a property failure into a flake.
3. If a non-property test fails, re-check the pin. Mismatch exits 3. Stop. A fixture miss is an input error, even when the diff looks small.
4. Replay the failing test on a host that is not the session that proposed the patch. This proposal uses a budget of 5 attempts. The number is a policy choice, not a measured optimum. Store each attempt id, exit code, and duration in seconds. A short run is incomplete. Fill the budget before calling the classifier.
5. Emit a freeze record only when the classifier returns flake, the property verdict is pass, the pin status is match, the owner is non-empty, and the expiry is in the future. The replay host records attempts. It does not write the lease. A reviewer, or a separate policy job, accepts the record before merge.
6. Leave the test in a canary lane for the life of the lease. On expiry, the record is invalid. The next red run starts again at step 3. There is no automatic renewal.

Proposed classifier

The snippets assume Python 3.10 or newer for the type-union syntax. They are unexecuted examples. The assertions specify the gate. They are not a reported measurement from a live suite.

from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum

class Exit(Enum):
    INCOMPLETE = 1
    PROPERTY = 2
    FIXTURE = 3
    FLAKE = 4
    STABLE_RED = 5

@dataclass(frozen=True)
class Attempt:
    attempt_id: str
    passed: bool

@dataclass(frozen=True)
class Report:
    property_verdict: str | None
    pin_status: str | None
    origin_passed: bool | None
    replays: tuple[Attempt, ...]
    now: datetime
    expires_at: datetime
    owner: str

def classify(report: Report, budget: int = 5) -> Exit:
    if report.property_verdict not in {'pass', 'fail'}:
        return Exit.INCOMPLETE
    if report.pin_status not in {'match', 'mismatch'}:
        return Exit.INCOMPLETE
    if report.pin_status != 'match':
        return Exit.FIXTURE
    if report.property_verdict != 'pass':
        return Exit.PROPERTY
    if report.origin_passed is None or len(report.replays) != budget:
        return Exit.INCOMPLETE
    greens = sum(1 for item in report.replays if item.passed)
    if report.origin_passed is False and greens == 0:
        return Exit.STABLE_RED
    if report.origin_passed is False and greens > 0:
        return Exit.FLAKE
    return Exit.INCOMPLETE

def freeze_allowed(report: Report, exit_code: Exit) -> bool:
    if exit_code is not Exit.FLAKE:
        return False
    if not report.owner.strip():
        return False
    if report.expires_at <= report.now:
        return False
    return True

One specification check covers the row reviewers most often want to skip. Property failure is decided before replay length, so a short replay cannot launder it.

def test_property_fail_cannot_freeze():
    now = datetime(2026, 10, 13, tzinfo=timezone.utc)
    report = Report(
        property_verdict='fail',
        pin_status='match',
        origin_passed=False,
        replays=(Attempt('r1', True), Attempt('r2', False)),
        now=now,
        expires_at=datetime(2026, 10, 20, tzinfo=timezone.utc),
        owner='ci-bot',
    )
    exit_code = classify(report)
    assert exit_code is Exit.PROPERTY
    assert freeze_allowed(report, exit_code) is False

Change the property verdict to pass, fill five replays, and keep at least one green attempt. The class becomes flake. freeze_allowed still depends on owner and expiry. Clear the owner and the lease closes again. Class and permission are not the same decision. That is why the proposal uses two functions.

An illustration of a property, also unexecuted, shows the kind of predicate the verdict refers to. It is not a suite. It is not evidence that any agent patch passed it.

def qty_must_reject_negative(place_order):
    result = place_order(qty=-1)
    if result.rejected is not True:
        raise AssertionError('negative quantity was accepted')

If this function fails, the exit is 2. Replays of an unrelated integration test do not change that.

Commands that keep the classes apart

Run the property file and the suspect test as separate processes. A single log line should not be the only place both verdicts exist.

python -m pytest tests/properties -q --tb=line
echo property_exit=$?

python -m pytest tests/integration/test_order_replay.py -q --tb=line
echo suspect_exit=$?

python -m agent_gate.classify \
  --property-verdict pass \
  --pin-status match \
  --origin-passed false \
  --replay green,green,red,green,green \
  --owner ci-bot \
  --expires-at 2026-10-20T00:00:00Z

The third command is a proposed CLI shape for the functions above. It should print a line such as exit=4 class=flake freeze=eligible, or exit non-zero when the lease is refused. If that module does not exist yet, call classify() from a one-file script. Do not encode the decision in a shell conditional that only inspects the suspect exit code.

A practical debug pass on a red agent patch is then mechanical. Read the property exit first. If it is non-zero, stop and diff the predicate against the production change. If the property exit is zero, compare the fixture digest with the pin. Only a match proceeds to the five-attempt replay. Paste attempt ids into the report. Run the classifier. If it returns stable red, file a product or test fix. If it returns flake, the lease is eligible for review, not merged by the replay host.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator supplied two availability claims for this workflow: free model access, and a free server option. No model name, quota, hardware shape, duration, or benchmark is stated here, because none was verified for this draft. Confirm the current terms before you depend on either. Neither claim is a promise that the option remains available.

Use free model access in one place only: drafting candidate property predicates from the diff. The draft is untrusted input. A reviewer, or a checker that does not share the patch-authoring transcript, accepts the predicate into the repository before CI can treat it as a verdict. Do not ask the model to label a red test as flaky. That label comes from classify() and the replay witness.

Use the free server option as the replay host in step 4 only when the job can run apart from the session that proposed the patch. Separation is the requirement. The vendor is not. If that option cannot run the suspect test alone, use another host you already control. The classifier needs a witness outside the authoring session. It does not need a particular machine class.

The replay host still does not open the freeze. It appends attempt rows. Eligibility stays in freeze_allowed, and a person or a policy job still accepts the record.

Product options do not replace the gate. If the three exit classes are already separate fields in your repo, try the free model access and the free server option only as the predicate-draft step and the replay host. Read the current access notes first, and keep classify() in the repository either way.

Limitations

The budget of 5 will miss rare flakes. It will also call some environment faults flakes when the fault happens to disappear inside the window. Raising the budget costs latency. Lowering it makes exit 4 too easy. Write the budget into the lease so a later reader can see the policy that was applied.

A green replay does not prove nondeterminism. It proves the failure did not reproduce under the recorded attempts. Clocks, network calls, and order dependencies can still hide. The canary lane is what keeps that residual visible after the lease starts.

Property checks can be hollow. A predicate that asserts a constant, or that rewrites the oracle to match the patch, will pass, and this classifier will trust that pass. Review the predicate diff with the same suspicion as the production diff. This workflow does not detect a captured oracle.

Expiry is not enforcement unless CI loads the record and rejects expired leases. A pull-request comment is not a lease. A green canary is not a renewal. Requiring the replay length to equal the budget also means a loop that stops on the first green cannot open a lease. That is intentional. One lucky pass is not a witness.

Timestamps in the examples use 2026-10-13 only as a fixed clock for the specification. They are not a claim about a measured run.

Who should not use this

Skip the workflow when you have no pinned fixture, no property file, and no second place to replay. The classifier will return incomplete, which is honest, but it will not strengthen a suite that has nothing to compare.

Skip it when the failing check is a release gate for safety, billing, or migration correctness and policy forbids waivers. A flake lease is a waiver. Some suites should stay red until a human fixes the test or the product.

Skip it when the same agent writes the patch, the property, and the freeze justification. The separation is the control. Collapsing those roles removes the reason to trust exit 4.

What the lease must contain

A valid record stays small. Class, property verdict, pin status, attempt ids, budget, owner, expiry, and the canary job name are enough. Do not store a model transcript as the witness. Store the commands and the exit codes.

class=flake
property=pass
pin=match sha256:<recorded-in-ci>
origin=fail
replays=5 budget=5 greens=4
owner=ci-bot
expires_at=2026-10-20T00:00:00Z
canary=tests/integration/test_order_replay.py

The digest line is a placeholder shape, not a real hash. If any required line is missing, the record is incomplete. Incomplete does not merge as a freeze. The job merges when the suspect test is green without a lease, or it does not merge.

Count the class before you count the retries. Exit 4 is a narrow door. Exits 2, 3, and 5 exist so that door cannot be used for everything else.