Classify Every Test Before You Score an Agent Patch

An agent patch does not need a larger suite. It needs a sealed label on every nodeid: invariant, characterization, or nondeterministic. Those labels decide which failures may retry, which files may change, and which relations must hold. Without the labels, a green job is one number. With them, you can take three independent votes and merge only on agreement.

A passing pytest run is not a merge decision. It is one noisy vote. Agent-generated diffs fail in ways named tests are slow to notice: they drop a relation without deleting a test, they add fixture bytes nobody hashed, and they turn jitter into a skip. Classify first. Then score channels the patch cannot rewrite.

Why the suite collapses into one bit

Cheap patches raise the volume of diffs that look locally correct. Coverage moves. CI stays green. The defect is usually not a crashing line. It is a weakened relation, an unregistered input, or a retry that hides jitter.

Those three defects share a cause. The agent can see the same files the tests use. If classification, fixtures, and flake policy live in the worktree, the patch can negotiate with them. A gate that reads only the process exit code has already lost.

The rest of this article is a proposed scorecard. Treat the code as a runnable template, not as a production incident report. No pass-rate claims are attached.

The missing artifact is the classification file

Write one sealed table before any candidate runs. Keep it off the writable worktree. The apply step must not mount it for write.

# /opt/sealed/classification.txt
# nodeid<TAB>kind
tests/test_parse.py::test_ascii\tinvariant
tests/test_parse.py::test_emoji\tcharacterization
tests/test_net.py::test_timeout\tnondeterministic

Kind is a closed enum. invariant rows are relations that must hold for every merge. characterization rows document current behavior; a change is information, not a veto, unless a human reclassifies the row. nondeterministic rows may consume retry quota. No other kind exists in this design.

Parse it once. Fail closed on unknown kinds and duplicate nodeids.

# sealed/classify.py — proposed parser, unexecuted example
from pathlib import Path

ALLOWED = {"invariant", "characterization", "nondeterministic"}

def load_classification(path: Path) -> dict[str, str]:
    table = {}
    for raw in path.read_text().splitlines():
        line = raw.split("#", 1)[0].strip()
        if not line:
            continue
        nodeid, kind = line.split("\t", 1)
        kind = kind.strip()
        if kind not in ALLOWED:
            raise ValueError(f"unknown kind for {nodeid}: {kind}")
        if nodeid in table:
            raise ValueError(f"duplicate nodeid: {nodeid}")
        table[nodeid] = kind
    return table

Numbered rule for classification:

1. The file lives outside the candidate worktree.
2. Every test the job may run has a row.
3. An unclassified nodeid is a fail vote, not an implicit characterization.
4. Reclassification is a human change with no product code in the same diff.

Three votes that consume the labels

Keep properties, fixture digests, and a retry-quota ledger beside the classification file. Each channel casts pass, fail, or abstain. The merge rule is conjunctive. Any fail blocks. abstain is legal only on the quota channel, and only when no nondeterministic row ran.

Vote 1 — Out-of-tree properties

A property is a relation over inputs the patch is not allowed to edit. Put the modules in a sibling tree. Do not keep them next to in-repo tests/ if the agent can open that directory.

# sealed/properties/test_order_invariants.py
from decimal import Decimal

def property_totals_match_lines(order: dict) -> None:
    lines = order["lines"]
    expected = sum(
        Decimal(str(x["qty"])) * Decimal(str(x["unit_price"])) for x in lines
    )
    actual = Decimal(str(order["total"]))
    if actual != expected:
        raise AssertionError(f"total {actual} != lines {expected}")

def property_ids_are_unique(order: dict) -> None:
    ids = [line["sku"] for line in order["lines"]]
    if len(ids) != len(set(ids)):
        raise AssertionError("duplicate sku in one order")

Run those functions against fixtures the digest registry already approved. The agent may change src/. It may not change sealed/properties/.

Discovery can stay small:

# sealed/runner.py
import importlib.util
from pathlib import Path

def load_properties(root: Path):
    mods = []
    for path in sorted(root.glob("test_*.py")):
        spec = importlib.util.spec_from_file_location(path.stem, path)
        mod = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(mod)
        mods.append(mod)
    return mods

Numbered rule for this vote:

1. Properties live outside the candidate worktree.
2. Each property is a pure function: input dict in, assertion out.
3. A single assertion failure is fail, not a skip.
4. Import errors are fail. Silence is not a pass.

Invariant rows in the in-repo suite are a second belt, not a substitute. If an invariant nodeid flickers, that is vote 1 adjacent failure. It does not touch quota.

Vote 2 — Fixture digest registry

Fixtures are data. Data drifts. An agent that “fixes” a test by appending a friendly row has not fixed the system. Record a SHA-256 for every fixture file the properties may read. Store the registry beside the classification file.

# sealed/digests.py
import hashlib
import json
from pathlib import Path

def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def vote_fixtures(fixture_dir: Path, registry_path: Path) -> dict:
    registry = json.loads(registry_path.read_text())
    observed = {}
    for path in sorted(fixture_dir.rglob("*")):
        if path.is_file():
            rel = str(path.relative_to(fixture_dir))
            observed[rel] = sha256_file(path)

    extra = sorted(set(observed) - set(registry))
    missing = sorted(set(registry) - set(observed))
    changed = sorted(
        rel for rel in observed.keys() & registry.keys()
        if observed[rel] != registry[rel]
    )
    if extra or missing or changed:
        return {
            "vote": "fail",
            "extra": extra,
            "missing": missing,
            "changed": changed,
        }
    return {"vote": "pass", "count": len(observed)}

Human-signed expansions are a separate change. They do not ride along with a behavior patch. Characterization tests may read new behavior. They may not smuggle new bytes into sealed fixtures.

Numbered rule for this vote:

1. Every readable fixture has one digest.
2. New files are extra, not helpful.
3. Content change without a registry edit is fail.
4. The registry file is not in the agent's write set.

Vote 3 — Retry-quota ledger

Flakes are not a character trait of a test. They are a budget. A suite that retries until green is spending a resource nobody counted. Keep a small JSON ledger outside the worktree. Decrement only when a nondeterministic row passed after retry. When remaining quota is zero, further jitter is fail.

This is not a named-test freeze. Frozen names rot. A quota is an integer the patch cannot increment.

# sealed/quota.py
import json
from pathlib import Path

def vote_quota(ledger_path: Path, retries_used: int, nondet_ran: bool) -> dict:
    ledger = json.loads(ledger_path.read_text())
    remaining = int(ledger["remaining"])
    if retries_used < 0:
        return {"vote": "fail", "reason": "negative retry count"}
    if retries_used == 0 and not nondet_ran:
        return {"vote": "abstain", "remaining": remaining}
    if remaining - retries_used < 0:
        return {
            "vote": "fail",
            "reason": "quota exhausted",
            "remaining": remaining,
            "retries_used": retries_used,
        }
    return {
        "vote": "pass",
        "remaining_after": remaining - retries_used,
    }

Only nondeterministic rows may consume quota. Invariants that flicker are fail. Characterization rows that flicker are recorded, then ignored by the merge rule unless you later promote them.

Numbered rule for this vote:

1. Classification is sealed before the run.
2. Invariant flicker cannot buy a retry.
3. Quota only decreases during a candidate job.
4. A human raises the integer in a change that contains no product code.

Wiring the labels to a merge exit code

The job is a sequence. Keep it boring.

1. Create a clean worktree from the candidate SHA.
2. Apply the agent diff to src/ only. Refuse patches that touch sealed/, .github/, or classification.txt.
3. Load classification. Fail if the in-repo suite lists a nodeid with no row.
4. Compute fixture digests. Cast vote 2.
5. Load properties. Run them on registered fixtures only. Cast vote 1.
6. Run in-repo tests with retries disabled for invariant and characterization. Record retries only for nondeterministic. Cast vote 3.
7. Write scorecard.json. Exit 0 only on unanimous pass (quota may abstain).

Proposed combiner:

# sealed/scorecard.py
import json
import sys

def combine(v1: dict, v2: dict, v3: dict) -> int:
    votes = [v1["vote"], v2["vote"], v3["vote"]]
    if "fail" in votes:
        print(json.dumps({"result": "reject", "votes": votes}, indent=2))
        return 1
    if v1["vote"] != "pass" or v2["vote"] != "pass":
        print(json.dumps({"result": "reject", "votes": votes}, indent=2))
        return 1
    print(json.dumps({"result": "merge", "votes": votes}, indent=2))
    return 0

if __name__ == "__main__":
    sys.exit(combine(
        json.loads(sys.argv[1]),
        json.loads(sys.argv[2]),
        json.loads(sys.argv[3]),
    ))

Commands you can paste into a job. Adjust paths. Do not treat the snippet as a benchmark.

git fetch origin "$CANDIDATE_SHA"
git worktree add /tmp/candidate "$CANDIDATE_SHA"

if git -C /tmp/candidate diff --name-only "$BASE_SHA"...HEAD \
    | grep -E '^(sealed/|\.github/|classification\.txt$)'; then
  echo "patch touches sealed paths"
  exit 1
fi

python3 /opt/sealed/classify.py /opt/sealed/classification.txt
python3 /opt/sealed/digests.py
python3 /opt/sealed/runner.py --fixtures /opt/sealed/fixtures
python3 /opt/sealed/quota.py
python3 /opt/sealed/scorecard.py "$V1" "$V2" "$V3"

Path refusal is the first real control. If step 2 is missing, the later votes are theater.

Decision table

Properties	Digests	Quota	Action
pass	pass	pass	merge
pass	pass	abstain	merge
pass	pass	fail	reject
fail	pass	pass	reject
pass	fail	pass	reject
fail	fail	fail	reject
abstain	any	any	reject

Properties and digests may not abstain. If a property file fails to import, that is fail. If classification cannot parse, that is fail before any vote runs.

Where candidate generation fits

The scorecard does not care which process wrote src/. It cares that the process cannot write the labels or the votes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you generate candidates with MonkeyCode's free model access and apply them on its free server option, keep /opt/sealed/ on a path the job mounts read-only. The free model and free server are a source of diffs. They are not a source of classification, properties, digests, or quota. Do not copy the scorecard into the same tree the model can edit. No model names, hardware, duration, or rate limits are claimed here.

Limitations

This design does not measure performance, memory, or latency unless you encode those as properties with explicit bounds. It does not replace a security review. It will not notice a logic bug that still satisfies the relations you wrote.

The digest vote is brittle against intentional, reviewed fixture growth. That brittleness is useful. It is also noisy if fixtures are generated at runtime. Generate them in sealed storage, then hash the output, or skip vote 2.

The quota vote can be gamed if the agent can edit the ledger or the classification file. If CI checks out one tree and gives the agent write access to it, the design is already void. Unseeded randomized tests look like flakes. Seed them, or classify them nondeterministic and accept the quota cost. Do not mix those two choices in one job.

Characterization rows can hide regressions if you never promote them. Schedule a human pass over rows that changed outcome twice in a week of candidates. Promotion is a classification edit, not a patch to src/.

Who should not use this

Skip the scorecard if the agent is allowed to edit workflows, owners files, or any path that defines the votes. Skip it if the product has no stable relations you can state in code. Skip it for throwaway prototypes where a human will never merge the diff. Skip it if your only tests are UI screenshots; those belong in a different gate.

Teams with a single shared folder and no sealed path should fix that first. The votes are not portable to a directory the model can rm.

What to implement first

Do not start with quota. Start with path refusal: reject any patch that touches the classification file or the property tree. Then hash two fixtures. Then add one property that would have caught the last bad merge. Then label every nodeid. The scorecard gets stricter as the relation set grows. It does not get stricter because a suite got greener.

If you already emit agent diffs from a remote free-model job, point that job at sealed storage before anyone opens the pull request. That is the whole loop.