Green Tests Can Be a Weakened Oracle: A Three-Bucket Plan for Agent Patches

Agent-generated patches often go green by shrinking the oracle, not by fixing the code. They drop an assertion. They skip a race. They assume a precondition the production path does not guarantee. Merge on that signal and you bought a quieter suite, not a safer one.

This article proposes a three-bucket test plan for those patches. Bucket A is an immutable contract layer the patch cannot rewrite. Bucket B is metamorphic pairs: relations between inputs that stay true even when you lack a perfect expected value. Bucket C is a seeded flake quarantine that records a reproduction command instead of pytest.mark.skip. The plan is a workflow you can implement. It is not a production case study, and the thresholds below are knobs, not fleet measurements.

Why a green unit suite is a weak merge signal

A conventional unit test encodes one example. An agent that can edit both src/ and tests/ can satisfy the example by changing either side. That is the cheapest path to a passing job. It is also the path that deletes the only check that would have caught the next patch.

Flakes make the hole larger. A human under time pressure skips them. An agent does the same, at higher volume, with a commit message that looks responsible. A skip is a silent hole. A seeded quarantine is a hole with a map.

Metamorphic testing covers a third gap. You may not know the exact parse tree for every input. You can still require that normalize(normalize(x)) == normalize(x), or that shuffling a commutative batch does not change the aggregate. Relations survive when goldens rot. Goldens do not survive an agent that rewrites them.

The three buckets

Name the buckets in CI so a patch cannot “fix” a failure by moving the file.

1. Contracts. Frozen fixtures, schema hashes, public signatures, and error-code tables. The agent may not modify this tree in the same commit as production code.
2. Relations. Metamorphic pairs and small properties with an explicit seed. Failures here block the merge.
3. Quarantine. Known flakes, each with a seed, a last reproduction command, and a budget of allowed silent runs. Exceed the budget and the job fails. No skip marks.

If a change needs to update a contract, it ships in a separate, human-reviewed commit. That split is the policy. Everything else is enforcement.

1. Inventory the suite by who may touch it

Start with a layout the diff can see.

tests/
  contracts/          # agent write-denied in the same commit as src/
  relations/          # metamorphic pairs; agent may add, not delete
  quarantine/         # seeded flakes only; no pytest.mark.skip
  unit/               # ordinary examples; treated as hints, not oracles

List the files. Fail closed if a path is unclassified.

find tests/contracts tests/relations tests/quarantine tests/unit \
  -name 'test_*.py' -print | sort

If your runner cannot enforce path ACLs, check git diff --name-only in CI. A contract file in the same commit as src/ is a rejected patch, not a debate.

2. Reject patches that weaken the oracle

Run this before the suite. The script below is a proposed gate. It looks for deleted assert, new skips, bare xfail, and except Exception that did not exist on the base revision. Tune the patterns. Do not treat the output as a proof of malice. Treat it as a merge blocker that a human can override with a reason file.

# oracle_shrink.py — proposed gate, not a measured detector
from __future__ import annotations

import argparse
import re
import subprocess
import sys
from pathlib import Path

WEAKENERS = [
    (re.compile(r"^-\s*assert\b", re.M), "deleted assert"),
    (re.compile(r"^\+\s*(pytest\.mark\.skip|@pytest\.mark\.skip)", re.M), "new skip"),
    (re.compile(r"^\+\s*@pytest\.mark\.xfail", re.M), "new xfail"),
    (re.compile(r"^\+\s*except Exception\s*:", re.M), "broad except"),
    (re.compile(r"^\+\s*assert True\b", re.M), "tautology"),
]

CONTRACT_PREFIXES = ("tests/contracts/",)


def git_diff(base: str) -> str:
    out = subprocess.check_output(
        ["git", "diff", "--unified=0", base, "--", "tests", "src"],
        text=True,
    )
    return out


def changed_paths(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", base],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", default="origin/main")
    args = parser.parse_args()
    paths = changed_paths(args.base)
    src_changed = any(p.startswith("src/") for p in paths)
    contract_changed = any(p.startswith(CONTRACT_PREFIXES) for p in paths)
    if src_changed and contract_changed:
        print("reject: src/ and tests/contracts/ changed in the same commit")
        return 2

    diff = git_diff(args.base)
    hits = []
    for rx, label in WEAKENERS:
        if rx.search(diff):
            hits.append(label)
    if hits:
        print("reject: oracle shrinkage:", ", ".join(hits))
        return 2
    print("oracle_shrink: no hardcoded weakeners in tests/ diff")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it as a required check.

python oracle_shrink.py --base origin/main

A tautology is not subtle. assert True, assert result is not None after the function already returned a default, and except Exception: pass are the usual cheap greens. Catch the cheap ones in the diff. Leave the subtle ones to relations.

3. Put meaning in relations, not in one example

Metamorphic testing asks: if I transform the input in a way that should preserve, invert, or commute the output, does the program still agree with itself? You do not need the true answer for every row. You need a relation the agent cannot satisfy by editing a single golden.

The example below is labeled as an unexecuted illustration. Swap the domain for yours. Keep the structure: a generator, a relation, a seed.

# tests/relations/test_csv_aggregate_relations.py
# Proposed example. Not executed against a live service.
from __future__ import annotations

import csv
import io
import random
from decimal import Decimal

from mypkg.aggregate import totals_by_sku  # your function


def _rows(n: int, rng: random.Random) -> list[dict[str, str]]:
    skus = ["A", "B", "C"]
    out = []
    for _ in range(n):
        out.append(
            {
                "sku": rng.choice(skus),
                "qty": str(rng.randint(1, 9)),
                "price": f"{rng.randint(1, 20)}.00",
            }
        )
    return out


def _csv(rows: list[dict[str, str]]) -> str:
    buf = io.StringIO()
    w = csv.DictWriter(buf, fieldnames=["sku", "qty", "price"])
    w.writeheader()
    w.writerows(rows)
    return buf.getvalue()


def test_shuffle_does_not_change_totals():
    rng = random.Random(20260906)
    rows = _rows(40, rng)
    left = totals_by_sku(_csv(rows))
    rng.shuffle(rows)
    right = totals_by_sku(_csv(rows))
    assert left == right


def test_split_concat_is_additive():
    rng = random.Random(20260906)
    rows = _rows(40, rng)
    mid = len(rows) // 2
    whole = totals_by_sku(_csv(rows))
    a = totals_by_sku(_csv(rows[:mid]))
    b = totals_by_sku(_csv(rows[mid:]))
    skus = set(whole) | set(a) | set(b)
    for sku in skus:
        wa = a.get(sku, Decimal("0")) + b.get(sku, Decimal("0"))
        assert whole.get(sku, Decimal("0")) == wa

Two properties, one seed. Shuffle is a permutation relation. Split-concat is an additive relation. An agent that hard-codes { "A": 1 } will fail the second as soon as n grows. An agent that deletes the test will fail bucket policy if deletions in tests/relations/ are blocked.

Add a deletion check next to the shrink gate:

git diff --name-status origin/main -- tests/relations \
  | awk '$1 ~ /^D/ { print; bad=1 } END { exit bad }'

Additions are welcome. Deletions need a human.

4. Quarantine flakes with a seed, never with skip

A skip has no reproduction. A quarantine record does. Store one JSON object per flake. The job fails if the test passes for the whole budget (the flake died and should return to the main suite) or if it fails without matching the stored seed (a new failure, not the known one).

{
  "id": "test_parser_stream[crlf-chunk]",
  "file": "tests/quarantine/test_parser_stream.py",
  "seed": 20260906,
  "repro": "pytest tests/quarantine/test_parser_stream.py -k crlf-chunk --seed 20260906",
  "silent_runs_allowed": 8,
  "silent_runs_seen": 0,
  "last_status": "fail",
  "note": "intermittent CRLF split across chunks; not a skip"
}

Proposed runner fragment:

# quarantine_run.py — proposed classifier
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

LEDGER = Path("tests/quarantine/ledger.json")


def run_one(item: dict) -> str:
    cmd = item["repro"].split()
    proc = subprocess.run(cmd, capture_output=True, text=True)
    return "pass" if proc.returncode == 0 else "fail"


def main() -> int:
    ledger = json.loads(LEDGER.read_text())
    rc = 0
    for item in ledger:
        status = run_one(item)
        if status == "pass":
            item["silent_runs_seen"] += 1
            if item["silent_runs_seen"] >= item["silent_runs_allowed"]:
                print(f"quarantine graduated: {item['id']} passed too often; move it back")
                rc = 2
        else:
            item["silent_runs_seen"] = 0
            item["last_status"] = "fail"
        print(f"{item['id']}: {status} silent={item['silent_runs_seen']}")
    LEDGER.write_text(json.dumps(ledger, indent=2) + "\n")
    return rc


if __name__ == "__main__":
    sys.exit(main())

The important rule is negative. pytest.mark.skip in tests/ is a gate failure, the same class as a deleted assert. Quarantine is not a junk drawer. If nobody can reproduce with the stored seed, the record is invalid and the test returns to relations or contracts as a hard failure.

5. Classify the job before you read the patch notes

Use a table, not a feeling.

Signal	Bucket	Merge?	Action
src/ + tests/contracts/ in one commit	A	No	Split the commit
Deleted assert / new skip / tautology	A/B	No	Restore the oracle
Relation failure	B	No	Fix code or add a human-reviewed relation
New relation file only	B	Yes, if contracts stay green	Keep
Quarantine fail with stored seed	C	Yes, if budget not exceeded	Update last_status
Quarantine fail with a different traceback	C	No	Treat as a new regression
Quarantine pass past budget	C	No	Graduate the test back
Unit examples fail, relations pass	unit	Maybe	Examples are hints; inspect

“Maybe” is the only soft cell. Unit examples are cheap for agents to game. Do not let them veto a relation-clean patch without reading the diff. Do not let them approve a relation-dirty patch at all.

A minimal CI shape:

set -e
python oracle_shrink.py --base origin/main
pytest tests/contracts tests/relations -q --randomly-seed=20260906
python quarantine_run.py

Pin the seed. Unpinned order is how flakes escape the ledger.

Where a free model and a free server fit

Candidate patches are cheap to produce. Oracle integrity is not. If you already generate patches from a free model and execute the suite on a free server, spend that budget on reruns of buckets B and C, not on a longer unit file the agent can edit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode’s free model access and free server option are relevant here only as a place to run the gate. They do not replace contracts, relations, or the ledger. A free endpoint that returns a patch is another untrusted author. The three buckets do not care who wrote the diff.

Limitations

This workflow will not save a suite that never had an oracle. If the program has no invariant you can state, metamorphic pairs become assert True with extra steps. Write the relation first. If you cannot, do not generate patches against that module.

The shrink gate is syntactic. It will miss an assertion rewritten as a weaker helper. It will also flag a legitimate refactor that moves assert into a utility. That is an acceptable false positive for a merge gate. Override with a signed reason file, not by deleting the check.

Seeded quarantine assumes you can pass a seed into the test. Hidden time sources, live network, and unordered hash iteration will ignore the seed. Freeze those at the process edge or keep the test out of CI.

The layout assumes one repo and one CI job. Monorepos with generated code need a tighter path filter. Safety-critical systems need more than relations; this article does not claim coverage of certification evidence.

Who should not use this

Do not use this plan if the agent is required to update goldens in the same commit as production code and you have no second reviewer. The policy depends on that split.

Do not use it for one-off scripts you will not run twice. The ledger is overhead.

Do not use it as a substitute for type checks, linters, or a real fuzzer on parsers. Relations are a merge signal. They are not a proof.

Do not skip bucket A because relations feel more advanced. Relations without frozen contracts are another suite the agent can edit.

Keep the classifier, change the author

The reusable artifact is the classifier: shrink gate, relation seed, quarantine ledger. Swap the patch source whenever you want. The merge rule stays the same. Green is not enough when the agent holds the eraser.

If you already run candidate patches through MonkeyCode’s free model and free server path, keep this runner even when the model changes. The oracle is the part that does not get cheaper.