Score Agent Patches on a Frozen Surface. Ledger the Flakes.

A green CI run is the wrong merge signal for an agent patch. If the agent can skip a flake, rewrite a fixture, or weaken a property, green only means the scoreboard moved. Split the suite into a scoring surface and a diagnostic surface. Score the patch only on the first. Put every freeze on a ledger the agent cannot edit.

This is a testing strategy, not a model review. The artifact is a four-class map, a freeze ledger, and a CI check that fails when scoring files change shape.

The unit you are actually scoring

Agent patches optimize whatever you measure. Test names are a weak measure. Skip markers are weaker. A property that still runs, on frozen inputs, with an unchanged fixture digest, is a stronger one.

Flakes do not belong on that surface. They belong on a ledger. The ledger is metadata. It is not a test the agent is allowed to “fix” by deleting an assertion.

Diagnostic tests can stay noisy. They help humans. They must not vote on merge.

Four ownership classes

Map every test path before the agent is allowed to touch production code. One class per file. No mixed files.

1. Scoring properties. Universal checks over a pinned input corpus. They must not import wall-clock time, network, or process-wide RNG. Failure is a merge blocker.
2. Scoring fixtures. Read-only bytes the properties consume. Hash them. A patch that retouches fixture bytes is a different change than a logic patch. Split it.
3. Diagnostic tests. Example-based tests, UI checks, live client calls, anything order-sensitive. They may flake. They never gate the agent.
4. Freeze ledger. Records why a test left the scoring surface. The agent gets no write access to this file.

If a file cannot be classified, it is diagnostic until a human says otherwise. Unclassified paths must not sit in the scoring glob. Silence is how flakes re-enter the scoreboard.

Artifact: the freeze ledger

Keep the ledger next to the suite, not in a chat log. JSON is enough. YAML is fine. The fields matter more than the format.

{
  "version": 1,
  "entries": [
    {
      "test_id": "tests/diagnostic/test_invoice_poll.py::test_poll_settles",
      "flake_class": "timing",
      "moved_from": "tests/scoring/test_invoice_poll.py",
      "evidence_hash": "sha256:8f3c1a0b9d2e4c77a1b0c3d4e5f60718293a4b5c6d7e8f90123456789abcdef0",
      "moved_at": "2026-09-21",
      "owner": "payments",
      "note": "Passes alone; fails under xdist when poll window is 50ms."
    }
  ]
}

flake_class is a closed set: timing, order, network, rng, env. If you cannot pick one, the test is not frozen. It is unclassified, and it stays off the scoring glob until someone names the class.

evidence_hash is a hash of the last failing artifact: log excerpt, reproduced command, or captured schedule. Unfreeze requires a new evidence hash from a stability window, not a deleted skip.

Do not encode “expires on” as the primary control. Expiry without a replay just returns the flake to the scoreboard. The control is relocation plus evidence.

Numbered workflow

1. Inventory the suite

List test files. Tag each path. Commit the map as scoring_map.yaml.

scoring_properties:
  - tests/scoring/properties/**/*.py
scoring_fixtures:
  - tests/scoring/fixtures/**/*
diagnostic:
  - tests/diagnostic/**/*.py
ledger: tests/scoring/flake_ledger.json
forbidden_to_agent:
  - tests/scoring/**
  - tests/scoring/flake_ledger.json
  - scoring_map.yaml

The glob is the contract. Reviewers should reject PRs that add a scoring file without a map update.

2. Move flakes off the scoring surface

When a scoring test flakes, do not xfail it in place. Cut it over to tests/diagnostic/. Add a ledger row. Leave a one-line pointer in the scoring tree if you need grep-ability, not a skip.

# tests/scoring/MOVED.md
# Invoice poll timing: see tests/diagnostic/test_invoice_poll.py
# ledger id: tests/diagnostic/test_invoice_poll.py::test_poll_settles

The scoring glob must not collect pytest.mark.skip, xfail, or flaky marks. Those marks are how an agent converts a blocker into a pass.

3. Freeze the scoring digest in CI

Hash the scoring properties, the fixtures, the map, and the ledger. Store the digest as a CI artifact from main. A candidate patch is scored against that digest. If the patch changes any of those files, fail with a distinct exit code. Do not fold that failure into “tests failed.”

# check_scoring_surface.py — reference implementation, not a published benchmark.
from __future__ import annotations

import hashlib, json, sys
from pathlib import Path

ROOT = Path(".")
MAP = json.loads(Path("scoring_map.json").read_text(encoding="utf-8"))
LEDGER = Path(MAP["ledger"])

CLOSED_FLAKE = {"timing", "order", "network", "rng", "env"}
FORBIDDEN_MARKS = ("pytest.mark.skip", "pytest.mark.xfail", "pytest.mark.flaky")

def sha_paths(patterns: list[str]) -> str:
    h = hashlib.sha256()
    files: list[Path] = []
    for pat in patterns:
        files.extend(sorted(ROOT.glob(pat)))
    for path in files:
        if path.is_file():
            h.update(path.as_posix().encode())
            h.update(path.read_bytes())
    return h.hexdigest()

def fail(msg: str, code: int) -> None:
    sys.stderr.write(msg + "\n")
    raise SystemExit(code)

def main() -> None:
    if not LEDGER.exists():
        fail("missing freeze ledger", 2)
    ledger = json.loads(LEDGER.read_text(encoding="utf-8"))
    for row in ledger.get("entries", []):
        if row.get("flake_class") not in CLOSED_FLAKE:
            fail(f"unclassified freeze: {row.get('test_id')}", 3)
        if not str(row.get("evidence_hash", "")).startswith("sha256:"):
            fail(f"freeze without evidence: {row.get('test_id')}", 3)

    for path in ROOT.glob("tests/scoring/**/*.py"):
        text = path.read_text(encoding="utf-8", errors="replace")
        for mark in FORBIDDEN_MARKS:
            if mark in text:
                fail(f"skip/xfail on scoring surface: {path}", 4)

    digest = {
        "properties": sha_paths(MAP["scoring_properties"]),
        "fixtures": sha_paths(MAP["scoring_fixtures"]),
        "ledger": hashlib.sha256(LEDGER.read_bytes()).hexdigest(),
    }
    Path("scoring_digest.json").write_text(json.dumps(digest, indent=2) + "\n")
    print(json.dumps(digest))

if __name__ == "__main__":
    main()

Exit 2/3/4 are policy failures. They are not product bugs. Keep them separate from pytest’s exit 1 so dashboards do not lump “agent cheated the suite” with “agent broke an invariant.”

4. Replay properties from a corpus, not from live RNG

Scoring properties read inputs from tests/scoring/fixtures/corpus/. Each file is one case. The runner hashes (property_id, input_sha, passed) into an outcome ledger. Compare that ledger to main. New failures are regressions. New passes on old inputs are the only allowed “improvement” signal.

# property_replay.py — proposal you can run as a CI step.
from __future__ import annotations

import hashlib, json, traceback
from pathlib import Path

CORPUS = Path("tests/scoring/fixtures/corpus")

def cases() -> list[Path]:
    return sorted(p for p in CORPUS.rglob("*") if p.is_file())

def run_property(name: str, data: bytes) -> bool:
    # Replace with a real import of your property. Keep I/O out of the property.
    from scoring_properties import PROPERTIES
    return bool(PROPERTIES[name](data))

def main() -> None:
    rows = []
    for path in cases():
        prop = path.parent.name
        payload = path.read_bytes()
        input_sha = hashlib.sha256(payload).hexdigest()
        try:
            passed = run_property(prop, payload)
            err = ""
        except Exception as exc:
            passed = False
            err = f"{type(exc).__name__}: {exc}"
        rows.append(
            {
                "property": prop,
                "case": path.name,
                "input_sha": input_sha,
                "passed": passed,
                "error": err,
            }
        )
    Path("outcome_ledger.json").write_text(json.dumps(rows, indent=2) + "\n")
    failed = [r for r in rows if not r["passed"]]
    if failed:
        raise SystemExit(f"{len(failed)} scoring properties failed")

if __name__ == "__main__":
    main()

Live generation of inputs can still happen on a schedule, off the merge path. New corpus cases land through a human PR. That keeps the agent from farming easy cases or dropping hard ones.

5. Generate the patch without the scoring tree in context

The generator sees production code and, if needed, diagnostic tests. It does not see tests/scoring/, scoring_map.yaml, or the freeze ledger. If those files appear in the prompt, the agent can aim at the scoreboard.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access is enough to draft candidate patches under that constraint. MonkeyCode’s free server option is a place to run check_scoring_surface.py and property_replay.py on a worktree that mounts the scoring surface read-only. Do not treat either as a quota, hardware, or uptime claim. The method does not depend on a particular model name.

A practical split:

# generation worktree: production + diagnostic only
git checkout -B agent/job-142
# scoring job: read-only mount of tests/scoring from origin/main
python check_scoring_surface.py
python property_replay.py

If the patch touches forbidden_to_agent, stop. Do not pytest first. Policy before verdict.

6. Unfreeze only with a stability window

To return a test to the scoring surface, record N consecutive diagnostic runs on the same command, same shard count, same fixture digest. Hash that log. Replace the ledger row with a removed entry that keeps the old evidence. Do not delete history.

Growing the ledger is allowed. Silent shrinkage is not. A patch that removes a freeze row without a removed record fails the map check.

Decision table

Event	Scoring surface	Diagnostic surface	Ledger
Property fails on frozen input	Block merge	Ignore	Unchanged
Fixture bytes change	Block as policy (exit 4 class)	n/a	Unchanged
New skip/xfail in tests/scoring/	Block as policy	n/a	Unchanged
Timing flake on an example test	Do not add it here	Keep, mark noisy	Add timing row if it ever sat in scoring
Agent rewrites a diagnostic test	Do not score it	Allowed	Unchanged
Agent edits ledger or map	Block as policy	n/a	Reject shrinkage
Human unfreeze	Move file back only after stability window	Drop after move	Append removed + new evidence

The table is the review checklist. If a PR cannot be placed in one row, it is two changes. Split it.

What this strategy does not claim

It does not prove the agent understood the domain. It proves the patch did not quietly retune the exam.

It does not remove flakes from the repository. It removes their vote.

It does not replace mutation testing, contract tests at HTTP boundaries, or a sealed oracle host. Those are complementary controls. This article is only the suite split and the ledger.

Corpus quality still dominates. A frozen set of trivial inputs will stay green while production breaks. Review new corpus cases the same way you review schema migrations.

Who should not use this

Skip the split if the repo has fewer than a handful of honest properties. A scoring surface of two tests is a ritual, not a gate.

Skip it if the team already lets application code seed pytest marks from feature flags. The ledger cannot outrun a runtime skip.

Skip it if merge is gated on a hosted SaaS suite the agent can reconfigure through the same credentials it uses to open the PR. Policy files in git are the point.

Do not point a generator at the ledger “to clean it up.” Cleanup is a human change with evidence hashes. That is slower. That is the cost of a stable scoreboard.

Limits you should write down

Shard count, locale, and filesystem case-folding still leak into properties if you let them. Pin the pytest invocation in the scoring job. Same flags every time.

PYTEST_ADDOPTS=
python -m pytest tests/scoring/properties -q -p no:xdist --randomly-seed=0
python check_scoring_surface.py
python property_replay.py

If you need xdist for diagnostic tests, keep it there. Do not share addopts across surfaces.

Outcome ledgers grow. Rotate corpus files with a dated directory, not by overwriting cases. Overwrite is how a hard input disappears.

Close

Score the patch on a surface the agent cannot edit. Relocate flakes instead of skipping them. Hash fixtures. Replay properties from files. Treat ledger edits as policy, not as product work.

If you already keep generation off the scoring tree, a free model plus a free server is a sufficient pair to try the split on one service before you touch the rest of the monorepo.

Comments

[Raju Dandigam]

Separating scoring tests from diagnostic tests is a strong way to stop “green” from becoming a movable target. The trust root also has to include the scoring glob, digest logic, CI command, and ledger—not only the fixtures—because changing any of those can silently shrink the surface. I like the idea of comparing them against a signed baseline or protected manifest outside the agent’s writable worktree. How do you handle legitimate scoring-map changes without making that policy review so heavy that teams route around it?