Budget the Suite Delta Before You Merge an Agent Patch

An agent patch is mergeable only when the suite delta stays inside three budgets. Green CI is a binary signal. It cannot tell you that a property vanished, that a fixture was rewritten to match a bug, or that a flake freeze grew by one slot to hide a race.

Those three failures show up when generation is cheap. A model can propose another patch in minutes. The loop hunts a green check. The suite pays.

This article is a proposed gate, not a report of a production rollout. Files, commands, and thresholds below are a method you can run against your own tree. Label every number you have not measured on your runner as a policy choice, not a benchmark.

The three budgets

Define the budgets at HEAD, before any agent workspace exists. The agent may change implementation files. It may not change the contract files that encode the budgets.

1. Property budget. Named properties must still execute on the symbols the patch can affect. A patch may add properties. It may not delete, skip, or rewrite an existing property without a human-owned ticket.
2. Fixture budget. Characterization fixtures are hashed at HEAD. The agent may read them. It may not rewrite bytes to make assertions pass.
3. Flake budget. Known flakes occupy slots in a capacity-limited freeze file. Capacity is an integer. The freeze file is not a skip list the model is allowed to grow.

A checkbox asks whether tests passed. A scorecard asks what the suite became. Volume of new tests is not safety. A test that asserts a stub returns itself is volume. A property that still forbids silent truncation on callers of a changed encoder is safety.

Isolate generation from grading

Disclosure: This article was prepared as part of MonkeyCode's product outreach. One practical split is to generate the implementation patch with MonkeyCode's free model access on the free server option, then score the suite delta in CI that the model cannot edit. Cheap generation is useful. Cheap grading is not. The product is not the gate. The contract files are.

Treat the generator as untrusted input. The workspace that produces the diff must not be the workspace that records HEAD hashes, freeze capacity, or property names.

repo/
  contracts/
    properties.toml      # human-owned names + owning modules
    fixtures.lock        # sha256 of characterization fixtures
    flake_budget.json    # capacity + dated freeze slots
  src/
  tests/
  tools/score_suite_delta.py

Merge policy for contracts/ is simple. Humans may edit it. Agent diffs that touch it fail closed.

# Proposed pre-receive / CI check. Not a claim about any host.
git diff --name-only origin/main...HEAD | grep -E '^contracts/' && exit 1

Contract file shapes

properties.toml is a name list, not a test runner. Each row is a property the suite must still collect after the patch.

# contracts/properties.toml
[budget]
min_collected = 8
forbid_delete = true

[[property]]
id = "encoder.no_silent_truncation"
module = "tests/properties/test_encoder_bounds.py"
nodeid = "tests/properties/test_encoder_bounds.py::test_no_silent_truncation"

[[property]]
id = "cache.ttl_monotonic"
module = "tests/properties/test_cache_ttl.py"
nodeid = "tests/properties/test_cache_ttl.py::test_ttl_monotonic"

fixtures.lock is a manifest. Hash the bytes the tests read, not the test file that wraps them.

# Rebuild the lock only from a human checkout of HEAD.
find tests/fixtures/characterization -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum > contracts/fixtures.lock

flake_budget.json is capacity plus inventory. Dates in the sample assume a review window measured in days, not a promise that any flake will heal on that date.

{
  "capacity": 5,
  "entries": [
    {
      "nodeid": "tests/test_cache.py::test_ttl_race",
      "reason": "timing",
      "expires": "2026-09-25",
      "ticket": "QA-4412"
    }
  ]
}

Capacity is the budget. Occupied slots are the spend. A patch that adds a freeze entry without removing an expired one is over budget even if CI is green.

Property checks that the agent does not own

Keep properties in a directory the write allowlist excludes. The example below is labeled proposed. It does not claim a measured catch rate.

# tests/properties/test_encoder_bounds.py
from hypothesis import given, strategies as st
from src.encoder import encode_frame

@given(st.binary(min_size=0, max_size=4096), st.integers(min_value=1, max_value=512))
def test_no_silent_truncation(payload: bytes, limit: int) -> None:
    frame = encode_frame(payload, limit=limit)
    if len(payload) <= limit:
        assert frame.payload == payload
        return
    assert frame.truncated is True
    assert len(frame.payload) == limit
    assert frame.payload == payload[:limit]

The oracle is the invariant, not an example the model wrote next to the bug. If the agent also authors the property, the scorecard must still require the HEAD property ids to remain collected. New properties can raise the score. They cannot replace missing HEAD ids.

Score the suite delta

The scorer reads HEAD contracts, the patch diff, and a pytest collection report. It prints JSON. Humans set thresholds. The script does not invent pass/fail theater beyond those thresholds.

# tools/score_suite_delta.py
from __future__ import annotations

import hashlib, json, subprocess, sys, tomllib
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
CONTRACTS = ROOT / "contracts"


def git_names(rev_range: str) -> set[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", rev_range], text=True
    )
    return {line.strip() for line in out.splitlines() if line.strip()}


def load_properties() -> dict:
    return tomllib.loads((CONTRACTS / "properties.toml").read_text())


def fixture_hash_ok() -> bool:
    locked = (CONTRACTS / "fixtures.lock").read_text()
    current = subprocess.check_output(
        "find tests/fixtures/characterization -type f -print0 | sort -z | xargs -0 sha256sum",
        shell=True, text=True,
    )
    return locked == current


def flake_spend() -> dict:
    budget = json.loads((CONTRACTS / "flake_budget.json").read_text())
    occupied = len(budget["entries"])
    return {
        "capacity": budget["capacity"],
        "occupied": occupied,
        "remaining": budget["capacity"] - occupied,
    }


def collect_nodeids() -> set[str]:
    raw = subprocess.check_output(
        ["pytest", "--collect-only", "-q", "tests/properties"], text=True
    )
    return {line.strip() for line in raw.splitlines() if "::" in line}


def main() -> int:
    changed = git_names("origin/main...HEAD")
    if any(path.startswith("contracts/") for path in changed):
        print(json.dumps({"error": "agent_touched_contracts", "changed": sorted(changed)}))
        return 2

    props = load_properties()
    collected = collect_nodeids()
    missing = [
        row["nodeid"] for row in props["property"] if row["nodeid"] not in collected
    ]
    flakes = flake_spend()
    score = {
        "properties_required": len(props["property"]),
        "properties_missing": missing,
        "fixture_hash_ok": fixture_hash_ok(),
        "flake_remaining": flakes["remaining"],
        "contract_files_untouched": True,
    }
    print(json.dumps(score, indent=2))
    if missing or not score["fixture_hash_ok"] or flakes["remaining"] < 0:
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it as a second job, after the ordinary suite. Ordinary pytest still answers "did this revision fail a test?" The scorer answers "did this revision spend suite capacity it does not own?"

python tools/score_suite_delta.py
# exit 0: budgets held
# exit 1: missing property, fixture drift, or flake over-capacity
# exit 2: contract path edited in the agent range

Numbered workflow

Use this sequence on every agent patch, including patches produced on a free server. Do not invert steps 1 and 3. Budgets that are written after the patch are rationalizations.

1. Freeze HEAD contracts. Rebuild fixtures.lock from the default branch. Confirm flake_budget.json capacity and expiry dates. Confirm every property id still collects on HEAD.
2. Open an isolated workspace. Copy implementation paths only. Deny writes to contracts/, CI YAML, pytest.ini, and skip/xfail markers if those live outside the freeze file.
3. Generate the implementation patch. A free model and a free server are enough for this step because the grader does not live in that workspace.
4. Run the ordinary suite. Record pass/fail. Do not merge on this signal alone.
5. Run score_suite_delta.py. Require exit 0. Keep the JSON next to the patch as a review artifact.
6. Apply the decision table. If the scorecard fails, reject. Do not regenerate until the failing budget is named. Regeneration without a named budget is how skip lists grow.

Decision table

Suite delta	Ordinary CI	Scorecard	Action
Properties still collected; fixture hashes match; freeze slots unchanged	pass	pass	Review implementation only
New property added; HEAD properties intact	pass	pass	Accept the extra property as optional credit
HEAD property nodeid missing from collection	pass or fail	fail	Reject. Restoration of the property is required
Characterization fixture bytes changed	pass	fail	Reject. Fixtures are not an API the agent may edit
Freeze entry added; remaining ≥ 0; ticket present	pass	pass only if policy allows human freeze edits	Do not accept from the agent lane
Freeze entry added; remaining < 0	pass	fail	Reject. Capacity is the budget
Contract path appears in git diff	pass	exit 2	Reject closed
Ordinary tests fail; budgets held	fail	pass	Reject for the product bug, not for suite spend

The last row matters. A held budget does not excuse a red suite. The two signals answer different questions. Collapse them and you are back to a checkbox.

What the freeze file is not

A freeze slot is a dated lease with a ticket. It is not pytest.mark.skip. It is not xfail without a bug id. It is not a timeout increase hiding in pytest.ini.

Expired rows must burn down. A proposed nightly check:

python - <<'PY'
import json, datetime
from pathlib import Path
budget = json.loads(Path("contracts/flake_budget.json").read_text())
today = datetime.date.fromisoformat("2026-09-11")
expired = [e for e in budget["entries"]
           if datetime.date.fromisoformat(e["expires"]) <= today]
if expired:
    raise SystemExit("expired flake leases: " + ", ".join(e["nodeid"] for e in expired))
PY

If a flake is still real on the expiry date, a human renews the lease and spends another slot. The agent does not renew leases. Renewal is how a temporary quarantine becomes a permanent skip.

Limitations

This gate does not detect a tautological property that was already on HEAD. It only requires that named nodeids still collect. Classification of new tests is a separate review step.

Fixture hashing does not help if your "characterization" directory is empty and all expected values live inline in test functions. Inline goldens are editable without touching fixtures.lock. Move bytes you do not want rewritten into the hashed tree, or extend the lock to those test files and accept the extra merge friction.

Property collection does not prove the properties ran with a useful example budget. --collect-only is cheap and incomplete. Pair the scorer with a required pytest nodeid run for tests/properties if your runner can afford it. Do not claim a catch rate you have not measured.

Flake capacity is a social control. It fails if humans inflate capacity to unblock a release. Publish capacity next to the freeze file history. A silent capacity bump is the same class of spend as an agent-authored skip.

The method also assumes a single default branch and a git range origin/main...HEAD. Forks that rewrite history will hash the wrong HEAD. Pin the merge-base explicitly if your repo is not linear.

Who should not use this

Do not use three-budget scoring as a substitute for a test suite that already cannot run deterministically on CI hardware. If the ordinary runner is noise, the scorecard will freeze noise.

Do not use it on snapshot-heavy UI trees until snapshots are split into human-owned characterization and agent-editable sandboxes. Hashing a 4,000-file snapshot directory will block every visual change, including the ones you wanted the agent to make.

Do not use it if the agent is also the merge bot. A model that can edit GitHub Actions YAML can delete the scorer. The write deny list is part of the method, not an optional style rule.

Teams with fewer than a handful of property tests will see a vacuously green property budget. Start by naming the invariants you already believe, then turn them into collected nodeids. An empty properties.toml is not a gate.

Close

Merge on suite delta, not on a green check. Property names, fixture hashes, and flake capacity are the units. Generation can stay cheap. Grading cannot.

If you already produce patches on a free server, keep contracts/ and tools/score_suite_delta.py outside that workspace and review the JSON next to the diff. That is the whole split.