Give Every Agent Patch a Regression Budget, Not Just a Green Check

A green CI run is not a verdict. It is a photograph taken at one moment, under one seed, with one set of fixtures. When the photographer is an agent that rewrites five files in a single pass, you need more than a photograph — you need a budget.

This article shows how to define a regression budget for agent-generated patches: a fixed number of properties, fixture contracts, and flaky-test quarantine slots that the patch must respect before it merges. I use free model access and a free server tier to keep the gate honest, but the budgeting idea works with any CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a budget beats a checklist

A checklist says: run property P, run fixture F, mark flaky test T. A budget says: here is how much verification you can afford, here is what must pass, and here is what gets cut when time runs out.

Budgeting forces prioritization. Agent patches fail in noisy ways: an LLM can pass eight property checks and then break a fixture contract that was never loaded. If you have a fixed number of verification credits, you will naturally put the highest-signal checks first.

The three-bucket budget model

I split regression verification into three buckets. Each bucket has a cost, measured in free-tier compute minutes, and a cap.

Bucket	Artifact	Cap (free tier)	Pass condition
Properties	Hypothesis-based property tests	200 cases per property	No counterexample found
Fixtures	Versioned contract fixtures	20 fixtures per patch	All fixture checks pass
Quarantine	Flaky-test freeze list	5 entries	No frozen test failed twice

This table is small on purpose. A budget you cannot explain in one table is a budget your CI will ignore.

Implementing the budget gate

I used a shell script that reads a small YAML file describing the budget. The script runs three phases and stops as soon as one phase exceeds its cap.

#!/usr/bin/env bash
set -euo pipefail

BUDGET_FILE="budget.yml"
phase_cap() {
  local phase=$1
  grep -A4 "$phase:" "$BUDGET_FILE" | grep cap | awk '{print $2}'
}

run_verify() {
  local phase=$1
  local cap=$2
  local start=$(date +%s)
  ./verify_$phase.sh || true
  local elapsed=$(( $(date +%s) - start ))
  if (( elapsed > cap * 60 )); then
    echo "Budget exceeded for $phase"
    exit 1
  fi
}

run_verify properties $(phase_cap properties)
run_verify fixtures   $(phase_cap fixtures)
run_verify quarantine $(phase_cap quarantine)

This is pseudocode, but it is runnable if you have verify_properties.sh, verify_fixtures.sh, and verify_quarantine.sh lying around. I deliberately label it as a scaffold, not a production tool.

A concrete fixture-inventory artifact

The most useful artifact I have built is a fixture inventory that records what each fixture locks and when it was last verified. It turns "run all tests" into "verify the contract the patch actually touched."

# fixture_inventory.yml
user_service:
  version: "2026-09-01"
  locks:
    - auth.token_expiry
    - rate_limit.retry_after
    - user.update.immutable_fields
  verified_at: "2026-08-31T10:00:00Z"
  agent_patch_impact: "changed"

payment_splitter:
  version: "2026-08-15"
  locks:
    - split.amount_precision
    - split.currency_default
  verified_at: null
  agent_patch_impact: "untouched"

Before an agent patch is allowed through, the gate compares the list of files changed against this inventory. Only fixtures with agent_patch_impact: changed run in the fixture bucket. The others sit until their contract is actually at risk.

This is not an optimization. It is an honesty check: if a patch touches auth.py, you do not need to run the payment_splitter suite to judge it.

Where free resources change the game

Free model access and a free server tier change two things: the cost of generating a patch and the cost of verifying it. When both are near zero, the temptation is to run more. Budgeting says: run less, but run the right thing.

I ran this budget gate on a free server instance with a two-core limit. The property bucket consumed about 12 minutes for 200 cases. The fixture bucket consumed 6 minutes for 14 changed fixtures. The quarantine bucket took 2 minutes because no flaky test failed twice.

Total: 20 minutes per patch. That is long enough to matter, short enough to keep an agent loop human-supervised.

Limitations and who should not use this

This budget model assumes your tests are deterministic enough that a quarantine list is small. If your suite is 40% flaky, no budget will fix that; you need to fix the flakiness first.

The script above is a scaffold, not a scalable tool. It does not handle parallel CI workers, distributed caches, or multi-service fixture startup. If your system needs those, use a purpose-built platform and keep the budget idea, not the script.

You should not use this approach when your patch changes a security-critical module, where a narrow budget is dangerous. Security changes deserve exhaustive verification, not a cost cap.

The one-line takeaway

A regression budget is not about saving money. It is about forcing the gate to decide what cannot be skipped — before the agent's merge button gets pressed.

If you want to see how this budget behaves against 40 injected bugs, the same fixture inventory can be replayed with real failures. I recommend doing that once per month, with a deliberately broken patch, to check that the gate still cares.