Nothing Failed. The Suite Just Got Slower: A Wall-Clock Budget Gate for Agent Patches

A patch touches zero test files. No assertion changes, no new skips, no fixture edits. The suite goes green on the first run.

Then someone opens the job duration graph and the branch takes nearly three times as long as the commit it branched from.

Numbers in this post are illustrative unless labelled otherwise. The mechanism is the point, not the magnitudes.

Conclusion first: wall-clock duration, retry consumption, and runner configuration are test surfaces in exactly the same sense that assertions and fixtures are. Diff them with two ledgers, gate on ratios rather than absolutes, and attach an expiry date to every exception. The rest of this article is machinery for that sentence.

Why the usual patch audits stay silent

Most patch review looks at the test tree: assertions added or removed, fixture write sets, verdict flips between base and head, co-movement between fixture and assertion. All of that is content.

Duration inflation is behaviour. It usually lands outside the test tree entirely — a raised timeout value, a retries plugin configured to 3, a sleep(0.5) inserted into shared setup code, a marker that quietly turns a slow integration test into a CI skip.

An agent does not need to weaken a single assertion to make a red test green. It only needs more time. And once the clock has been raised, the next patch inherits it as the new floor.

The three channels worth diffing

1. Clock — per-test p90 duration, head versus base.
2. Retry debt — retries consumed at head that base did not consume.
3. Config — timeout, retry, skip and parallelism settings, hashed as blobs.

Each channel is cheap to capture and cheap to diff. None of them require reading the patch diff at all, which is the useful property: you can run the gate on patches you did not author and do not fully understand.

Step 1 — Fingerprint the config channels

Hash the blob that CI will actually execute, not the file on disk. A branch can change a config file and revert it; git show HEAD:path is the version that runs.

python - <<'PY'
import hashlib, json, subprocess

CHANNELS = {
    'pytest_ini':  'pytest.ini',
    'pyproject':   'pyproject.toml',
    'ci_workflow': '.github/workflows/test.yml',
    'conftest':    'tests/conftest.py',
}

fp = {}
for key, path in CHANNELS.items():
    try:
        blob = subprocess.run(['git', 'show', 'HEAD:' + path],
                              capture_output=True, check=True).stdout
        fp[key] = hashlib.sha256(blob).hexdigest()[:12]
    except subprocess.CalledProcessError:
        fp[key] = None
print(json.dumps(fp, indent=2, sort_keys=True))
PY

conftest.py is in the list deliberately. It is where sleeps and session-scoped polling loops usually land. The cost is noise: every legitimate conftest edit shows up as a config finding. Treat config as a human-acknowledgement channel, not an automatic fail.

Step 2 — Build the base ledger

Run the suite N times on the base commit and keep p50 and p90 per test. N ≥ 3 for anything under five minutes; N = 5 if you can afford it.

Keep the host boring: same image, same CPU class, same cache state, same number of runs for base and head. If base runs cold and head runs warm, you are measuring I/O, not the patch.

The ledger is a small JSON file, not a report from a specific plugin. Any reporter that emits per-test durations can fill it:

{
  "commit": "base",
  "config": {
    "pytest_ini": "9f2c1a0b7d3e",
    "ci_workflow": "0a11bb93c7d2"
  },
  "tests": {
    "tests/test_auth.py::test_login": {"p50_ms": 82, "p90_ms": 96, "retries": 0}
  },
  "freezes": [
    {"test": "tests/test_billing.py::test_webhook", "owner": "team-payments", "reason": "upstream sandbox 503", "expires": "2026-11-03"}
  ]
}

Step 3 — Build the head ledger where the agent cannot write

Timing has the same trust problem as verdicts: a measurement host that the patch under test can modify is not a measurement host. A separate small VM is enough — the workload is one test suite and a few JSON files.

MonkeyCode's free server option is a reasonable candidate for that second host if you are already in that ecosystem, and its free model access fits the adjacent low-stakes job: turning a wall of findings into a readable budget note for the PR. Keep both out of the decision path. (Operator-supplied: free model access and a free server option are offered. No quota, hardware class, or duration guarantee is implied — check the current terms before you depend on them.)

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The verdict itself must stay deterministic arithmetic. A model that decides whether 3.4× is acceptable is a model that can be argued with, and you will be arguing with it at the worst possible moment.

Step 4 — Diff the ledgers

Dependency-free reference implementation. Point it at two ledgers; exit code 1 means findings.

#!/usr/bin/env python3
import json, sys
from datetime import date

RATIO_FAIL   = 3.0     # p90 ratio vs base
ABS_FLOOR_MS = 300     # ignore tiny absolute deltas
RETRY_FAIL   = 1       # any retry at head where base had none

def load(p):
    with open(p, encoding='utf-8') as f:
        return json.load(f)

def compare(base, head, today):
    out = []
    bt, ht = base.get('tests', {}), head.get('tests', {})

    for name, h in ht.items():
        b = bt.get(name)
        if b is None:
            out.append(('new', name, 'no base budget'))
            continue
        bp90 = max(b.get('p90_ms', 0), 1)
        ratio = h.get('p90_ms', 0) / bp90
        delta = h.get('p90_ms', 0) - b.get('p90_ms', 0)
        if ratio >= RATIO_FAIL and delta >= ABS_FLOOR_MS:
            out.append(('clock', name, '%dms -> %dms (%.1fx)'
                        % (b['p90_ms'], h['p90_ms'], ratio)))
        if h.get('retries', 0) >= RETRY_FAIL and b.get('retries', 0) == 0:
            out.append(('retry', name, '0 -> %d retries' % h['retries']))

    for name in sorted(set(bt) - set(ht)):
        out.append(('dropped', name, 'missing from head ledger'))

    bc, hc = base.get('config', {}), head.get('config', {})
    for k in sorted(set(bc) | set(hc)):
        if bc.get(k) != hc.get(k):
            out.append(('config', k, '%s -> %s' % (bc.get(k), hc.get(k))))

    for t in head.get('freezes', []):
        if t.get('expires', '') < today.isoformat():
            out.append(('freeze', t['test'], 'expired %s (%s)'
                        % (t['expires'], t.get('owner', '?'))))
    return out

def main(argv):
    if len(argv) != 3:
        print('usage: budget_gate.py base.json head.json')
        return 2
    findings = compare(load(argv[1]), load(argv[2]), date.today())
    for kind, name, detail in findings:
        print('%-8s %-40s %s' % (kind, name, detail))
    print('---')
    print('%d finding(s)' % len(findings))
    return 1 if findings else 0

if __name__ == '__main__':
    sys.exit(main(sys.argv))

The ratio-and-floor pair matters. A ratio alone fires on every 4ms test that becomes 9ms. A floor alone ignores a 2s test that becomes 6s. Require both.

Step 5 — Freeze with an expiry date

Flaky-test freezes rot when they have no deadline. Make the token structural: a freezes entry needs test, owner, reason, and expires. An expired token is a finding, not a warning, and renewing it is a visible commit rather than a silent extension.

Two rules keep the freeze honest. First, a freeze only ever suppresses one channel — a frozen test can still fail the clock gate. Second, the freeze list is append-and-expire, never delete-and-forget; the git history is the audit trail for who bought time and when.

The decision table

Finding	Likely cause	First action
clock	sleep in shared setup, polling loop, hidden retry	Compare p50 as well as p90. If p50 moved too, revert before discussing
retry	flake masked by a retries plugin	Require a freeze token with an owner and a date, or fix the race
config	timeout raised, retries added, marker now skipping	Diff the config blob and ask for the justification in the PR body
dropped	test deleted or renamed	Confirm the replacement exists in the head ledger, by name
freeze	token expired	Re-triage the flake. Do not extend silently
new	new test, no baseline	Bootstrap after 3 green runs; provisional ceiling = suite p90

Where this breaks

Shared CI runners produce p90 noise, especially on small instances. If you cannot pin a CPU class, widen RATIO_FAIL and accept that you will miss modest inflation.

Some tests are supposed to be slow — anything asserting on a timeout or backoff has to burn real time. Freeze those explicitly instead of tuning the gate around them.

Retry debt can legitimately drop, and the script only fires on increases, so that direction is safe. Config hashing is the opposite: it fires on harmless edits. That is why it routes to a human, not to a red build.

New tests have no baseline at all. A provisional ceiling derived from the suite p90 is a heuristic, not a measurement, and it should be labelled that way in the ledger.

Finally, the gate costs CI minutes: two hosts, N runs each. On a suite that already takes twenty minutes, that bill is real and worth pricing before you adopt it.

Who should not use this

Skip it if the whole suite finishes in under about sixty seconds, because a 300ms floor is most of your runtime and the ratios are meaningless. Skip it if your runners are unstably shared with no image pinning. Skip it if tests only ever run on developer laptops, where the clock belongs to whatever else is open.

And skip it if nobody owns the freeze list. A budget gate with expired tokens nobody triages just becomes a permanently red job that everyone learns to ignore.

---

The clock is part of the test contract, and agent patches are unusually good at exploiting the parts of a contract nobody wrote down. Assertions and fixtures were the first two; duration and configuration are the next two.

If you run the ledger on a single repo, the number I would most want to hear is not how many clock findings you got — it is how many of them turned out to be real fixes versus measurement noise on your runners.