The Model Writes the Patch. Three Files Decide If It Merges.

An agent patch is a candidate, not a proof. The merge decision belongs in three files the model does not write: an invariant module, a hashed fixture lock, and a flake freeze with an explicit expiry. If any of those files is missing, a green CI run is only a prompt match.

This is a proposed workflow, not a production postmortem. Treat the code as a template. Run it locally, then on a disposable server, before an agent touches the default branch.

The failure mode, stated plainly

Agent patches optimize for the tests they can see. They also optimize for the tests they can delete. A second model call does not fix that. It repeats it.

The cheap pattern is circular grading. The same class of model that produced the diff is asked whether the diff is safe. That loop fails when the bug is an invariant the prompt never named.

Properties catch unnamed invariants. Fixture hashes catch silent output drift. A dated freeze file stops flakes from becoming permanent holes. None of those artifacts should be authored in the same change as the patch.

What stays out of the model's context

Do not paste the freeze list into the patch prompt. Do not paste the fixture lock either. The agent should see the failing production symptom and the public test names. It should not see the oracle.

If the agent can read flakes.freeze.json, it can "fix" a flake by deleting the test. If it can read the fixture hashes, it can rewrite the fixture. Keep the judge offline. Path-filter the evaluation tree so a patch that touches it is rejected before pytest runs.

Repository layout

Use a dedicated evaluation tree. Keep it small.

eval/
  invariants_test.py
  check_fixtures.py
  freeze_gate.py
  summarize_flakes.py
  fixtures/
    order_total.json
    retry_budget.json
  fixtures.lock
  flakes.freeze.json
  run_eval.sh
src/
  billing.py

The agent may edit src/. It may add tests under a visible suite. It may not edit eval/ in the same change. Enforce that with a name-only diff against the merge base.

Step 1: Encode properties as tests, not as prompts

Prompts rot. Properties compile.

The example below pins a billing helper. Discounts cannot make a total negative. Quantities are non-negative. Rounding is banker's rounding to cents. Those rules are the product. They do not belong in a chat transcript.

# eval/invariants_test.py
from decimal import Decimal, ROUND_HALF_EVEN
import pytest
from src.billing import apply_discount

CENTS = Decimal("0.01")

@pytest.mark.property
@pytest.mark.parametrize(
    "amount,rate",
    [
        (Decimal("19.99"), Decimal("0.00")),
        (Decimal("19.99"), Decimal("0.15")),
        (Decimal("0.01"), Decimal("1.00")),
        (Decimal("1000.00"), Decimal("0.33")),
    ],
)
def test_discount_never_goes_negative(amount, rate):
    total = apply_discount(amount, rate)
    assert total >= Decimal("0.00")
    assert total == total.quantize(CENTS, rounding=ROUND_HALF_EVEN)
    assert total <= amount

That is not open-ended fuzzing. It is a pinned property set you can grow. If you later add random search, keep the seed in eval/, not in the agent prompt. A property is a statement that must hold for a class of inputs. A unit assertion holds for one input. Agent patches often satisfy the reproducer and violate the class. Put the class in eval/.

Step 2: Lock fixtures by digest, not by filename

A fixture that can be rewritten is not a lock. Hash the bytes. Filename stability is not integrity.

# eval/check_fixtures.py
import hashlib, json, pathlib, sys

ROOT = pathlib.Path(__file__).parent
LOCK = ROOT / "fixtures.lock"

def digest(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def main() -> int:
    expected = json.loads(LOCK.read_text())
    files = sorted((ROOT / "fixtures").glob("*.json"))
    actual = {p.name: digest(p) for p in files}
    if actual != expected:
        print("fixture lock mismatch")
        print("expected", json.dumps(expected, indent=2, sort_keys=True))
        print("actual", json.dumps(actual, indent=2, sort_keys=True))
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Regenerate the lock only in a human-reviewed change:

python - <<'PY'
import hashlib, json, pathlib
root = pathlib.Path("eval/fixtures")
lock = {
    p.name: hashlib.sha256(p.read_bytes()).hexdigest()
    for p in sorted(root.glob("*.json"))
}
pathlib.Path("eval/fixtures.lock").write_text(
    json.dumps(lock, indent=2, sort_keys=True) + "\n"
)
PY

Fixture content is the expected I/O for a golden path. The lock is the integrity check. Split those roles. If an agent patch "fixes" a test by editing order_total.json, the digest check fails before pytest runs.

Step 3: Freeze flakes with an expiry, never with a comment

A skipped test without a date is a deleted test. Use a machine-readable freeze file. Require an owner, an expiry, and a fail count.

{
  "version": 1,
  "rules": [
    {
      "nodeid": "tests/test_retry.py::test_backoff_jitter",
      "reason": "timing depends on host clock",
      "owner": "payments",
      "expires": "2026-09-21",
      "max_runs_observed": 12,
      "fail_count": 3
    }
  ]
}

The runner must refuse two things: expired rows, and rows that freeze a property test under eval/.

# eval/freeze_gate.py
from datetime import date
import json, pathlib, sys

FREEZE = pathlib.Path("eval/flakes.freeze.json")
PROTECTED_PREFIX = "eval/"

def main() -> int:
    today = date.today().isoformat()
    data = json.loads(FREEZE.read_text())
    errors = []
    for row in data.get("rules", []):
        nodeid = row["nodeid"]
        if nodeid.startswith(PROTECTED_PREFIX) or "/invariants_" in nodeid:
            errors.append(f"cannot freeze property test: {nodeid}")
        if row["expires"] < today:
            errors.append(f"expired freeze: {nodeid} expired {row['expires']}")
        if int(row.get("fail_count", 0)) < 2:
            errors.append(f"not enough evidence to freeze: {nodeid}")
        if not row.get("owner"):
            errors.append(f"freeze missing owner: {nodeid}")
    if errors:
        print("\n".join(errors))
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Evidence threshold matters. One red run is not a flake. Three fails in twelve runs on the same nodeid is a candidate. The freeze is temporary. When it expires, the test returns to the gate or it is deleted in a reviewed change. There is no third state.

Collect that evidence from junit, not from chat:

pytest tests/ --count=12 --junitxml=eval/out.xml || true
python eval/summarize_flakes.py eval/out.xml

# eval/summarize_flakes.py
import collections, sys, xml.etree.ElementTree as ET

def main(path: str) -> None:
    root = ET.parse(path).getroot()
    seen = collections.Counter()
    failed = collections.Counter()
    for case in root.iter("testcase"):
        nodeid = f"{case.attrib.get('classname')}::{case.attrib.get('name')}"
        seen[nodeid] += 1
        if case.find("failure") is not None or case.find("error") is not None:
            failed[nodeid] += 1
    for nodeid, n in seen.items():
        f = failed[nodeid]
        if 2 <= f < n:
            print(f"{nodeid} fail={f} runs={n}")

if __name__ == "__main__":
    main(sys.argv[1])

Print a candidate. Do not auto-write the freeze file. A script that mints skips will hide races faster than any agent will.

Step 4: Generate and judge on different machines

Keep the roles split. Generation proposes bytes. Judgment runs commands the model cannot edit.

1. Generate the patch with a coding model. Do not stream eval/ files into that session.
2. Apply the patch on a clean worktree.
3. Run eval/run_eval.sh on a server that is not your laptop's dirty tree.
4. Merge only if the path filter, fixture lock, freeze gate, and properties all pass.

A minimal runner:

#!/usr/bin/env bash
set -euo pipefail
if git diff --name-only origin/main...HEAD | grep -E '^eval/'; then
  echo "eval/ is read-only in agent patches"
  exit 1
fi
python eval/check_fixtures.py
python eval/freeze_gate.py
pytest eval/invariants_test.py -q

CI can repeat the same filter without new logic:

# .github/workflows/agent-eval.yml
name: agent-eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: bash eval/run_eval.sh

You can execute that script on any Linux userland with Python and pytest. A disposable server is enough. Production hardware is the wrong place to discover that a patch inverted a discount.

MonkeyCode is relevant here only as the generation side and as a place to run the script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator-supplied capability, not a benchmark: free model access can propose a candidate patch, and a free server option can run eval/run_eval.sh. That is not a quota, a hardware spec, a model list, or a permanence claim. The judging files stay in git either way. Keep the eval tree out of the prompt. The server is an execution target. It is not an oracle.

Decision table

Observation	Action	Merge?
Property fails on pinned inputs	Reject patch. Do not freeze.	No
Fixture digest changes	Reject until a human regenerates the lock	No
Visible unit test deleted, properties still pass	Reject. Deletion is not a fix.	No
Non-eval test flaps ≥3/12, expiry set ≤14 days, owner set	Freeze that nodeid only	Conditional
Freeze expired	Re-run. Fix or delete with review.	No until resolved
eval/ appears in the agent diff	Reject the change	No
Properties pass, lock intact, no eval edits	Accept for review	Yes, with review

Conditional merge still needs a human. The freeze is a scheduler, not an approval. Bulk expiry extensions are a process failure. Treat them as an incident, not as maintenance.

What this does not prove

Properties do not prove UX. They do not prove latency. They do not prove that a prompt injection in some other layer is impossible.

A fixture lock can overfit. If your golden JSON encodes one customer's tax table, you will merge patches that break every other table. Keep fixtures representative, then add a property for the rule underneath.

A freeze file can hide a real race. The expiry is the safety valve. If your team ignores expiry, the file is worse than nothing. Delete the mechanism rather than extend dates in bulk.

Host-dependent tests will also lie. If jitter, time, or core count is not pinned, summarize_flakes.py will recommend freezes forever. Pin the runner image before you trust fail counts.

Who should not use this

Do not use this as the only gate for safety-critical, medical, or unsupervised financial production. Do not use it if you have no human owner for freeze rows. Do not use it if the agent is allowed to edit eval/. Do not use an LLM-as-judge to "confirm" a property failure. The failure is already the result.

Teams without a reproducible server image will struggle. Flakes that depend on CPU count will freeze forever. Pin the runner, then freeze tests, not the other way around.

Close the loop

The model is a patch source. The three files are the judge. Keep them on different sides of the change, and expire every freeze you are not willing to own.