A Run Budget for Agent Patch Verification: Stop Rechecking Until Green

An agent patch is a cheap hypothesis and an expensive promise. The generation loop ends when the diff is written. The verification loop ends when a check happens to pass. On a constrained model quota or a bounded CI server, that second loop is where the cost multiplies.

Unlimited retries treat flakiness as a scheduling problem. It is not a scheduling problem. It is an evidence problem.

The budget-free retry loop

Most of us have seen this pattern:

run the suite
one check fails
re-run the check
passes
mark the patch as verified

If the check fails because the patch broke something, a re-run does not fix the defect; it just refunds the failure. A refunded failure has no place in an audit trail.

A normal test runner will show you the first passing run and forget the failures. The attempt number is the lost signal.

The budget gate model

A run budget converts verification from an open loop into a spend plan. The rules are small:

1. Every patch gets a fixed budget B of verification attempts.
2. All checks run in a fixed order, decided before the patch arrives.
3. Every invocation of every check consumes exactly one attempt.
4. A check must pass within B; otherwise, the gate rejects and the report shows where B went.

The model makes two assumptions. First, no check has an infinite entitlement to retries. Second, a check that needs many attempts to pass is itself a finding.

Reference implementation

The gate needs state: how many attempts are left, and which check is consuming them. That is a class with a counter and no dependencies:

from dataclasses import dataclass
from typing import Callable, Optional

@dataclass
class BudgetReport:
    name: str
    attempt: int
    passed: bool
    budget_exhausted: bool = False

Verifier = Callable[[], bool]

class BudgetGate:
    def __init__(self, budget: int):
        self.budget = budget
        self.spent = 0

    def run(self, name: str, verifier: Verifier) -> BudgetReport:
        while self.spent < self.budget:
            self.spent += 1
            if verifier():
                return BudgetReport(name, self.spent, True)
        return BudgetReport(name, self.spent, False, budget_exhausted=True)

Orchestration:

def verify_patch(patch) -> Optional[BudgetReport]:
    gate = BudgetGate(budget=8)

    checks = [
        ('compile', patch.compile),
        ('property', patch.properties),
        ('fixture_contract', patch.fixture_contract),
        ('regression', patch.regression),
    ]

    for name, verifier in checks:
        report = gate.run(name, verifier)
        if not report.passed:
            return report

    return None

When the budget runs out, run returns passed=False with budget_exhausted=True. The caller stops at the first non-passing check and takes the report.

This implementation is intentionally simple. It does not schedule tests in parallel, it does not triage failures, and it does not decode test framework output. It shows the cost flow.

The cost flow is the audit.

Reading the budget report

Run the gate on a symptomatic patch. The transcript can look like this:

compile passed: attempt 1
property failed: attempt 2
property failed: attempt 3
property passed: attempt 4
fixture_contract passed: attempt 5
regression failed: attempt 6
regression failed: attempt 7
regression rejected: budget exhausted at attempt 8

The rejection itself is not the main signal. The main signal is the gap between attempts 2 and 4: the property check behaved like a coin toss. A normal test runner hides that gap because the first passing run becomes the only remembered result. The budget pipeline exposes it as evidence.

The budget size sets the sensitivity. Three illustrative settings:

Budget	Behavior	Blind spot
1	Any flaky check rejects	Cannot distinguish flaky from broken
3	Absorbs common jitter but records it	A fail-every-two-runs pattern is absorbed
8	Gives every check several chances	Can mask a test that fails first and passes later

Ordering checks by denial value

A budget is wasted if the order is wrong. Putting the full regression suite first means spending half the budget before you even know whether the patch compiles.

A practical order moves from cheap preconditions toward semantic specificity:

Priority	Check	Why first	Cost per attempt
1	Compile / schema	Blocks everything else	Small
2	Scoped property	Rejects broken invariants with a few samples	Small-medium
3	Fixture contract	Catches data shape drift before it reaches tests	Medium
4	Selected regression	Closest to production semantics	High

Lock the order before you start. Reordering while the quota is burning is another form of drift.

Who should not adopt this

• A deterministic suite does not gain anything from a budget. It only gains noise.
• An emergency hotfix should not wait for a budget report. Unblock first, audit after.
• A check that already runs exactly once does not need a wrapper. Counting one attempt adds no information.

Where the budget meets a disposable runner

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If the patch generator in front of this loop sits on a free model endpoint, the budget protects exactly the quota you do not want to burn on flaky retries. If the execution environment is the free server option, a disposable CI runner keeps the spent counter simple and isolated from local state.

Neither pairing is required. Bounded retries are useful on their own, and a disposable runner is useful on its own. The pairing just makes the limits explicit: you can see every attempt and where it died.

The final check

A green suite is a screenshot. A budget report is an artifact.

When a re-run turns a failure into a pass, you are not hearing a miracle. You are hearing a flakiness discovery. The budget does not remove flakiness. It gives you a record, a number that can be audited like any other finding.

Stop rechecking until the suite is green. Recheck until the budget shows where the evidence went.