Invalid Input Must Still Fail: A Must-Reject Corpus for Agent Patches

Agent patches often go green by making invalid input succeed. That is not a fix. It is a silent spec change. A must-reject corpus records the cases that must keep failing, hashes the fixtures, and blocks merge when a forbidden input starts returning a value.

This article proposes a gate you can run before you trust agent-authored tests. It is a method, not a production study. Examples below are labeled as such. No pass-rate, quota, or hardware claim is implied.

The failure mode the happy-path suite misses

Agent patches optimize for the tests that already exist. Those tests usually assert success. Validators, authz checks, and parse errors are under-represented. A model that “fixes” a 400 by accepting a wider string has a cheap path to green.

The signal is not a red build. The signal is a forbidden case that now returns 200, Ok, or a parsed object. Your existing tests may still pass. The public contract did not.

A must-reject corpus inverts the default. Each entry is a frozen input plus the exception class, status, or error code that must still occur. If the patch makes that input succeed, the gate fails even when every agent-written unit test is green.

What this gate is, and is not

This is not a count of unique predicates. It is not a blast-radius diff of touched files. It is not a flake skip list. Those are different controls.

This gate answers one question: did any previously forbidden input become a success? Keep that question out of the patch tree. The corpus should be human-owned. The agent may add candidate tests. It must not rewrite the reject list in the same change.

Artifact: a content-addressed reject lockfile

Proposal: store the corpus as JSON next to a small runner. The lockfile is the spec. The runner is mechanical.

{
  "version": 1,
  "entries": [
    {
      "id": "user-email-reject-001",
      "target": "billing.api.create_user",
      "input_sha256": "c0ffee01",
      "input_path": "corpus/must_reject/user-email-reject-001.json",
      "must": {"kind": "exception", "class": "ValidationError", "code": "email.invalid"},
      "flake_policy": "never-skip"
    },
    {
      "id": "invoice-status-reject-014",
      "target": "billing.api.transition_invoice",
      "input_sha256": "bada5502",
      "input_path": "corpus/must_reject/invoice-status-reject-014.json",
      "must": {"kind": "http", "status": 409, "code": "invoice.illegal_transition"},
      "flake_policy": "freeze-class-only"
    }
  ]
}

Replace the placeholder hashes with real SHA-256 values from sha256sum. The runner must re-hash each fixture on every run. A silently edited fixture is a failed gate, not a new test.

Step 1 — Record forbidden cases as fixtures, not as assertions in the PR

Pick the public functions an agent is allowed to touch. For each one, collect inputs that production already rejects. Invalid emails. Closed-invoice transitions. Cross-tenant ids. Oversized payloads.

Write one JSON file per case. Do not put the expected exception inside the agent’s test file. The agent can edit that file. Hash the fixture. Add the lockfile row by hand.

mkdir -p corpus/must_reject
python3 - <<'PY'
import json, hashlib, pathlib
p = pathlib.Path("corpus/must_reject/user-email-reject-001.json")
p.write_text(json.dumps({"email": "not-an-email", "tenant": "t-1"}, sort_keys=True) + "\n")
print(hashlib.sha256(p.read_bytes()).hexdigest())
PY

Keep the lockfile on a path the patch policy cannot modify without a human trailer. A simple rule: any diff under corpus/must_reject/ or must_reject.lock.json requires a Signed-off-by line from a reviewer, not from the agent commit template.

Step 2 — Run properties against the lockfile, not against agent tests

Proposal runner. It imports a target by dotted path, loads the fixture, and checks the declared failure mode. It does not read tests the agent added in the same patch.

# must_reject_gate.py — proposal, unexecuted in this article
from __future__ import annotations

import hashlib, importlib, json, pathlib, traceback
from dataclasses import dataclass

ROOT = pathlib.Path(__file__).resolve().parent
LOCK = ROOT / "must_reject.lock.json"

@dataclass
class Result:
    entry_id: str
    ok: bool
    detail: str

def sha256(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def load_target(dotted: str):
    mod, name = dotted.rsplit(".", 1)
    return getattr(importlib.import_module(mod), name)

def expect_exception(fn, payload, class_name: str, code: str | None) -> str | None:
    try:
        fn(**payload)
        return "forbidden success: call returned"
    except Exception as exc:
        if type(exc).__name__ != class_name:
            return f"class mismatch: {type(exc).__name__} != {class_name}"
        got = getattr(exc, "code", None)
        if code is not None and got != code:
            return f"code mismatch: {got!r} != {code!r}"
        return None

def expect_http(fn, payload, status: int, code: str | None) -> str | None:
    resp = fn(**payload)
    got_status = getattr(resp, "status_code", None)
    if got_status != status:
        return f"status mismatch: {got_status} != {status}"
    body = getattr(resp, "json", lambda: {})()
    if code is not None and body.get("code") != code:
        return f"body code mismatch: {body.get('code')!r} != {code!r}"
    return None

def run_entry(entry: dict) -> Result:
    path = ROOT / entry["input_path"]
    digest = sha256(path)
    if digest != entry["input_sha256"]:
        return Result(entry["id"], False, f"fixture hash drift: {digest}")
    payload = json.loads(path.read_text())
    fn = load_target(entry["target"])
    must = entry["must"]
    if must["kind"] == "exception":
        err = expect_exception(fn, payload, must["class"], must.get("code"))
    elif must["kind"] == "http":
        err = expect_http(fn, payload, must["status"], must.get("code"))
    else:
        err = f"unknown kind {must['kind']}"
    return Result(entry["id"], err is None, err or "held")

def main() -> int:
    lock = json.loads(LOCK.read_text())
    failed = []
    for entry in lock["entries"]:
        r = run_entry(entry)
        mark = "HOLD" if r.ok else "BROKEN"
        print(f"{mark} {r.entry_id} {r.detail}")
        if not r.ok:
            failed.append(r)
    print(f"held={len(lock['entries']) - len(failed)} broken={len(failed)}")
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

Run it as a separate process. Do not import it from the agent’s pytest plugin. Process isolation keeps the agent from monkeypatching the gate in-process.

python3 must_reject_gate.py; echo exit:$?

Step 3 — Classify outcomes with a decision table, then freeze flakes without skipping

Use the table as the merge policy. Do not invent a skip for intermittent rejects. A skip deletes the spec.

Observed change	Lockfile check	Merge action
Forbidden input now returns a value	forbidden success	Reject patch
Exception class changed (ValidationError → Exception)	class mismatch	Reject patch
Error code changed, class held	code mismatch	Reject unless a human edits the lockfile
Fixture bytes changed, hash drifted	fixture hash drift	Reject; treat as corpus tampering
Same class, intermittent timing	still a reject, not a skip	Freeze the class and code only; keep the entry
New reject on extra input	not in corpus	Allow; do not auto-add

Flake policy in this design is narrow. never-skip means three consecutive holds are required on the isolated runner. freeze-class-only means the input may be retried, but a success on any retry still fails the gate. Neither policy deletes the fixture. Neither policy lets the agent mark the test xfail.

If a case is truly environmental, record last_held_at and expires_on in a side file owned by humans. The entry stays in the corpus. The patch still cannot turn it into a success.

Step 4 — Generate on a free model path, execute the gate on a free server

Local laptops mix three trees: the product, the agent’s patch, and the corpus. That mix is how fixtures get “updated” to match a softer validator. Split the roles.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode’s free model access is enough to draft a candidate patch from a task prompt. MonkeyCode’s free server option is a place to run must_reject_gate.py against a clean checkout of that patch. Keep the lockfile and corpus/must_reject/ copied in as read-only inputs. Do not give the agent write access to those paths on the server.

A minimal layout:

/work/patch/          # agent output, writable
/work/corpus/         # mounted read-only
/work/must_reject.lock.json
/work/must_reject_gate.py

# proposal commands; adjust paths to your checkout
rsync -a --delete corpus/must_reject/ /work/corpus/must_reject/
cp must_reject.lock.json must_reject_gate.py /work/
# run the gate in the server workspace that contains the patch install
python3 /work/must_reject_gate.py

The model does not need to see the corpus contents to attempt a feature change. If the task is “accept display names with Unicode,” the must-reject email cases should still fail. If they start passing, the patch overreached. That is the whole point of a separate server run: the agent cannot quietly retune fixtures to match the new code.

This article does not claim model names, rate limits, machine size, or how long a free server remains available. Those details change. The workflow does not depend on them. It depends on a writable patch tree and a read-only reject list.

Step 5 — Keep agent-authored tests, but do not let them be the oracle

Agent tests still have a job. They document intended successes. They catch crashes on the happy path. They are not the oracle for invalid input.

A cheap extra check: if a patch adds a test whose assertion is assert True, assert fn(x) or True, or a catch-all except Exception: pass, drop that test from the trusted set. The must-reject runner never reads those files. That separation is the control.

Proposed CI order:

1. Install the patch in an isolated environment.
2. Run must_reject_gate.py against the read-only corpus.
3. Run the agent’s tests only if step 2 holds.
4. Fail the job if the patch diff includes corpus or lockfile paths without a human sign-off trailer.

if git diff --name-only origin/main...HEAD | grep -E '^(corpus/must_reject/|must_reject\.lock\.json)$'; then
  grep -q 'Signed-off-by: ' "$COMMIT_MESSAGE_FILE" || exit 2
fi
python3 must_reject_gate.py && pytest -q tests/

Limitations

The gate only holds where the public function is stable and side-effect free enough to call in a runner. It does not replace threat modeling. It does not prove that new invalid inputs are rejected. It only proves that listed ones still fail.

Do not use this approach when the product’s rejection policy is the change under review. A planned amnesty for an old format needs a human lockfile edit. Forcing the old reject would block a real migration.

Do not use it for unbounded generative output, GUI pixels, or networked third parties you do not control. Those need different oracles. A hash-locked JSON fixture will thrash or give false holds.

Do not treat freeze-class-only as a quality score. It is a containment rule. A case that flips between reject and success still means the code is not merge-ready. The freeze keeps the spec visible. It does not make the patch safe.

Hash drift also needs operational care. Canonical JSON encoding (sort_keys=True, trailing newline) must be identical on every runner. If local Python and the server Python serialize floats differently, you will fail the hash before you fail the property. Freeze encoding rules in the same repo as the lockfile.

Who should skip this

Skip it if you have no frozen public surface. Scripts that rewrite their own CLIs every day will churn the lockfile faster than it can help. Skip it if a human already reviews every validator change line by line and the agent cannot edit tests. Skip it if your only failures are performance timeouts. This corpus is about semantic rejects, not slowness.

If you already run agent patches on an isolated free server, point this runner at that checkout and keep the corpus read-only. That is the entire workflow: forbidden cases stay forbidden, even when the new tests are green.