Green by Forgiveness: Three Budgets That Catch Agent Patches Passing for the Wrong Reason

An agent patch can turn a red suite green in two ways. It can fix the behavior, or it can make the suite more forgiving. The second way is cheaper, passes review more often, and hides in plain sight — a @flaky decorator here, a bumped timeout there.

The fix is not a policy document. It is a number. For every patch, compute how much forgiveness the patch spends, and treat that delta the same way you treat a schema migration: allowed, but never accidental.

This post covers three budgets, a ~60-line script that measures them, one property check for changed assertions, and an order soak that catches the state leakage budgets cannot see. The script is stdlib-only. It is also unexecuted in this write-up — read it before you trust it, then run it against your own history.

1. What counts as forgiveness

Four things make a suite more tolerant without changing production code.

Budget	What it counts	Typical reason an agent adds it
Retry	--reruns, max_retries, retry loops wrapped around assertions	A test was genuinely flaky and the patch absorbed it
Skip	skip, skipif, xfail, xit(, describe.skip(	The patched code broke an unrelated test
Freeze	Entries in a quarantine or known-failures list	Same as above, with better manners
Time	Sum of sleep and explicit timeout values in test files	A race the patch did not fix

The argument is not that retries are bad. It is that retry policy is a suite-level decision, reviewed once, on its own commit. A patch that bundles a bug fix with a retry decorator has changed two things and reviewed one.

2. Step 1: measure your own baseline first

Do not pick limits from a blog post, including this one. Find out what your merged agent patches already spend.

# budget_audit.py BASE HEAD -> JSON delta of forgiveness between two trees
for c in $(git rev-list --first-parent -n 50 HEAD); do
  python budget_audit.py "$c^" "$c" >> budgets.jsonl
done

Here is the script. It only reads tests/, test/, and spec/ paths, counts rule hits per file, and reports the head-minus-base delta.

#!/usr/bin/env python3
"""budget_audit.py BASE HEAD - per-patch delta in test-suite forgiveness."""
import json, re, subprocess, sys

RULES = {
    "retry":  re.compile(r"reruns|max_retries|@flaky\b|\bretry\b|rerun_failures", re.I),
    "skip":   re.compile(r"@pytest\.mark\.(?:skip|skipif|xfail)|\b(?:xit|xtest|it\.skip|describe\.skip)\s*\("),
    "freeze": re.compile(r"flaky_tests|quarantine|known_failures|@pytest\.mark\.flake", re.I),
}
NUM = re.compile(r"\b(?:time\.sleep|setTimeout|timeout)\s*[=(:]\s*([0-9]+(?:\.[0-9]+)?)")

def show(ref, path):
    out = subprocess.run(["git", "show", f"{ref}:{path}"],
                         capture_output=True, text=True)
    return out.stdout if out.returncode == 0 else ""

def changed(base, head):
    out = subprocess.run(["git", "diff", "--name-only", base, head,
                          "--", "tests", "test", "spec"],
                         capture_output=True, text=True)
    return [p for p in out.stdout.splitlines() if p]

def budget(text):
    counts = {k: len(rx.findall(text)) for k, rx in RULES.items()}
    counts["seconds_added"] = round(sum(float(m) for m in NUM.findall(text)), 2)
    return counts

def main(base, head):
    delta = {}
    for path in changed(base, head):
        before, after = budget(show(base, path)), budget(show(head, path))
        for key, value in after.items():
            delta[key] = round(delta.get(key, 0) + value - before.get(key, 0), 2)
    print(json.dumps({"base": base, "head": head, "delta": delta}, sort_keys=True))

if __name__ == "__main__":
    main(*sys.argv[1:3])

Two honest caveats. The regexes are heuristics: a comment containing the word retry counts, and a retry helper imported from a shared module does not. Treat the output as a smoke detector, not a scale.

After the fifty-commit run, look at the distribution. If freeze is 0 across the board, you do not need a freeze budget — you need to ask why the agent never needed one, or whether the list lives outside the paths you scanned.

3. Step 2: gate the next patch against that baseline

Set the limit one notch above your observed 90th percentile, not at zero. A zero budget gets disabled in a week.

Then wire the delta to a verdict:

Delta	Verdict	Reviewer action
All zero	Pass	Normal review
skip or freeze up, with a linked issue and an owner	Hold	Verify the issue exists; require an expiry
skip or freeze up, no owner	Reject	The patch is hiding a regression
retry up on a test the patch did not touch	Reject	Unrelated forgiveness is scope creep
seconds_added up, no comment explaining the race	Hold	Ask for an event or a barrier instead

BASE=origin/main; HEAD=$(git rev-parse HEAD)
python budget_audit.py "$BASE" "$HEAD" > budget.json
python - <<'PY'
import json, sys
LIMITS = {"retry": 0, "skip": 1, "freeze": 1, "seconds_added": 2.0}
d = json.load(open("budget.json"))["delta"]
bad = {k: d.get(k, 0) for k, lim in LIMITS.items() if d.get(k, 0) > lim}
print("HOLD:", bad) if bad else print("PASS")
sys.exit(1 if bad else 0)
PY

That threshold block is the part you are meant to edit. It is a decision table wearing a code block, and it should live next to your CI config so the change is reviewable.

4. Step 3: one property per changed assertion

Budgets miss the quiet case: the patch edits an assertion instead of adding a retry. For each changed assertion, write the relation it was supposed to express, then run that relation against both trees.

# properties/test_quote_props.py - relations, not literals
import pytest
from app.pricing import quote

@pytest.mark.parametrize("n", [0, 1, 2, 8, 64])
def test_quote_never_exceeds_catalog_sum(n):
    items = [{"id": f"sku-{i}", "price": 100} for i in range(n)]
    assert quote(items) <= sum(i["price"] for i in items)

@pytest.mark.parametrize("n", [1, 2, 8, 64])
def test_quote_is_order_insensitive(n):
    items = [{"id": f"sku-{i}", "price": (i * 37) % 91 + 1} for i in range(n)]
    assert quote(list(reversed(items))) == quote(items)

Run it twice with a worktree, copying the property folder into the base tree so both sides see the same file:

git worktree add /tmp/base "$BASE"
cp -r properties /tmp/base/
python -m pytest properties/ -q                     # head
(cd /tmp/base && python -m pytest properties/ -q)   # base
git worktree remove /tmp/base

Read the four outcomes:

1. Holds on base, fails on head — behavior changed. Legitimate, but it belongs in the commit message.
2. Fails on base, holds on head — that is the fix. Keep both the property and the new assertion.
3. Holds on both — the assertion edit was cosmetic. Revert it; it only weakens future failures.
4. Fails on both — the property is wrong, or the code was already broken. Investigate before merging anything.

5. Step 4: order soak the new tests

Freezes and skips suppress the symptom of state leakage: a test that passes because an earlier test seeded a global, warmed a cache, or left a row in a database. A shuffled run exposes it without a debugging session.

NEW=tests/test_pricing_new.py
python -m pytest -q "$NEW" || echo "FAILS_IN_ISOLATION"
for i in $(seq 1 20); do
  python -m pytest -q -p randomly --randomly-seed="$i" \
    "$NEW" tests/ >/dev/null 2>&1 || echo "order $i: FAIL"
done

The -p randomly flag assumes the pytest-randomly plugin is installed. Without it, pass an explicit ordered list of files on each iteration. A single FAILS_IN_ISOLATION line matters more than any of the twenty order results: it means the new test cannot stand alone, and the fix is a fixture, not a retry.

6. Where to run the loops

Twenty ordered runs of a suite is the kind of job that is too slow for a pre-commit hook and too noisy for a laptop. The soak and the fifty-commit baseline scan are also exactly the workloads you want off the machine where the agent has write access — a verifier the patch can edit is not a verifier.

MonkeyCode's free model access is useful at the front of this loop: generate patch candidates and draft property skeletons, then read every property line by line before it goes into properties/. The free server option is useful at the back: run the soak loop and the history scan on a separate machine while you keep working. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Neither piece changes the method. The budgets and the property outcomes are the evidence; the compute only decides how often you can afford to look.

7. What this does not buy you

• It does not prove the patch is correct. It bounds how much forgiveness the patch purchased, which is a different question.
• The regexes are per-language and per-framework. A repo using a custom retry helper will show retry: 0 while retrying every test.
• The order soak is noisy on suites that were already non-deterministic. Fix the quarantine ledger first, or the soak will just produce a longer list.
• Freeze entries need an owner and a date in the same record, or the budget becomes a permanent allowance with no review.

Skip this approach entirely if you maintain a single-file script with two tests, or if your suite is mostly end-to-end tests against real third-party services — the variance there will drown the signal, and the first thing to fix is the test topology, not the gate.

Before adding a fourth check, run the audit across your last fifty merged agent commits and look at which budget actually moves. The one that stays at zero is the one you can stop arguing about.