The Agent Patch Test Budget: Spend CI Minutes Where They Catch Faults

Agent-generated patches have a strange property: the cheaper they are to produce, the more expensive they are to validate. Once generation costs approach zero, your CI pipeline becomes the loop's bottleneck. The fix is not more gates — it is a budget that decides which gates run on which patch, and how many minutes each one may spend.

The gates themselves are no longer the interesting problem. Property checks, fixtures, and a flaky freeze are table stakes for anyone merging agent-written code. The open problem is allocation: a one-line doc change does not deserve the same gauntlet as a rewrite of a lock-free queue.

The loop is test-bound

A patch arrives from the agent. It compiles. Its own tests pass. Then the real cost begins.

Most teams run the same gauntlet for every patch: fixture regression, property checks, mutation spot-checks, full suite. That is correct when patches are rare and expensive to produce. When an agent can emit a dozen candidates per hour, the gauntlet becomes the throughput limit.

The constraint that matters is test time, not token cost. A patch that costs nothing to generate but twenty minutes to validate is priced by the validator.

A budget, not a checklist

Classify each patch by risk, then assign a time budget and a gate set. The table below is the one I run.

Risk	Trigger	Gates	Budget
S	≤ 50 added lines, one module	Fixtures + flaky freeze	120 s
M	51–200 added lines, one module	Fixtures + targeted properties + flaky freeze	600 s
L	> 200 lines or cross-module	Fixtures + full properties + freeze review	1800 s

Three rules keep the budget honest:

1. Classify by diff size, then escalate on module crossings. A 30-line change that touches three top-level directories is L-risk, not S-risk.
2. The flaky freeze runs outside the budget. It is cheap, and it protects every other gate from noise.
3. A gate that times out fails the patch. Ask the agent to split the diff instead of raising the budget.

Size is a heuristic, not a verdict. A one-line change to a lock-free queue is L-risk; the classifier starts from line count and escalates when the diff crosses module boundaries.

The artifact: gate_budget.sh

Here is the classifier I use. It diffs against the merge base, counts added lines and touched modules, and prints the gate plan. The CI step reads that plan and executes only the selected gates.

#!/usr/bin/env bash
# gate_budget.sh — classify an agent patch and print its gate plan
set -euo pipefail

BASE="${1:-origin/main}"
BUDGET_S="${BUDGET_S:-120}"
BUDGET_M="${BUDGET_M:-600}"
BUDGET_L="${BUDGET_L:-1800}"

added=$(git diff --numstat "$BASE"...HEAD | awk '{s+=$1} END {print s+0}')
files=$(git diff --name-only "$BASE"...HEAD)
modules=$(echo "$files" | awk -F/ '{print $1}' | sort -u | wc -l)

risk="S"
[ "$added" -gt 50 ] && risk="M"
[ "$added" -gt 200 ] && risk="L"
[ "$modules" -gt 1 ] && risk="L"

case "$risk" in
  S) budget=$BUDGET_S; gates="fixtures freeze" ;;
  M) budget=$BUDGET_M; gates="fixtures properties freeze" ;;
  L) budget=$BUDGET_L; gates="fixtures properties_full freeze_review" ;;
esac

echo "risk=$risk added=$added modules=$modules budget=${budget}s"
echo "gates=$gates"

The freeze list is a plain file. A test that fails twice in a row is quarantined; a frozen test must pass five consecutive runs before it returns:

#!/usr/bin/env bash
# freeze.sh — quarantine flaky tests, require 5 clean runs to unfreeze
set -euo pipefail
test_name="$1"
marks=".marks/$(echo "$test_name" | tr '/' '_')"

if grep -qxF "$test_name" freeze_list.txt; then
  if ./run_one "$test_name"; then
    echo "clean" >> "$marks"
    if [ "$(grep -c clean "$marks")" -ge 5 ]; then
      sed -i "/^$test_name$/d" freeze_list.txt
      echo "unfroze: $test_name"
    fi
  else
    rm -f "$marks"
  fi
else
  if ./run_one "$test_name"; then
    rm -f "$marks"
  else
    echo "fail" >> "$marks"
    if [ "$(grep -c fail "$marks")" -ge 2 ]; then
      echo "$test_name" >> freeze_list.txt
      rm -f "$marks"
      echo "froze: $test_name"
    fi
  fi
fi

Why each gate earns its minutes

Fixtures are the cheapest fault-per-minute for known behavior. They encode the module's regression history. Their weakness is coverage: they only know what you already know.

Property checks cover what you do not know. They run random operations against an invariant and catch the patch that is correct on every fixture but breaks a boundary condition the fixtures never touch.

// property: size never exceeds capacity under random push/pop
void property_capacity_holds() {
  BoundedQueue<int> q(8);
  std::mt19937 rng(42);
  std::uniform_int_distribution<int> op(0, 1);

  for (int i = 0; i < 10'000; ++i) {
    if (op(rng) == 0) q.push(i);
    else if (!q.empty()) q.pop();
    assert(q.size() <= q.capacity());
  }
}

The generator is the test. A property check with a degenerate generator is a fixture in disguise.

The flaky freeze catches no faults at all. It protects the gates that do. A flaky test produces a red build that means nothing; the agent sees it, "fixes" the wrong thing, and the loop burns another cycle. Quarantining that noise is the cheapest insurance in the pipeline.

Where the free tier fits

The budget only matters because the generation side stopped being the cost. I run candidate patches through MonkeyCode's free model access, so asking for a dozen variants costs no tokens. The free server option hosts the gate runner itself, which means the validation side has no standing infrastructure bill either.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That changes the constraint. The scarce resource is no longer model tokens or server time — it is CI minutes and review attention. The budget table is how I spend those.

Who should not use this

• Safety-critical systems. If a fault can injure someone or move money at scale, run the full gate set on every patch. A budget is an optimization, not a license to skip.
• Teams with no CI pressure. If the full suite finishes in two minutes, the budget adds complexity without benefit. Use the checklist.
• Unstable generators. Property checks inherit the quality of their input distribution. Three distinct inputs is not a property test.

The takeaway

The agent loop is only as fast as its slowest gate. Classify the patch, budget the minutes, and keep the freeze list honest. The gates stay the same — the order and the spending are what change.

If you run gate runners for agent patches, I would be curious what your budget table looks like. Mine started at 120/600/1800 seconds with freeze thresholds at 2/5, both tuned by watching where the loop actually stalled.