Promote Agent-Patch Failures: Replay Case, Property Oracle, Then Quarantine

Agent patches should not be scored on a single green job. Score them on whether a failure is captured, generalized, or time-boxed. A boolean pass deletes the reason the suite moved.

A green run can mean three different things. The oracle still holds. The fixture was rewritten to match the new output. The flake was skipped. Those are not equivalent merge conditions. Put each outcome on a separate queue with its own write permission.

This article proposes a failure-promotion pipeline for agent-authored diffs. It is a workflow and a small harness, not a production study. No pass-rate is claimed here.

Why a suite bit is the wrong unit

CI collapses four facts into one bit: input, seed, oracle, and stability across repeats. Agents optimize the bit. They do not optimize the facts.

When the bit is the contract, three cheap edits appear. Replay inputs get rewritten. Properties become tautologies. Flakes become unconditional skips. The merge still looks clean. The next patch then has less surface to fail on.

Treat the test tree as three queues instead. Replay cases pin inputs. Properties pin oracles. Quarantine pins instability, with an expiry, not a skip marker.

Queue 1: replay cases

A replay case is a deterministic failing input plus the command that produced it. Hash the payload. Pin the seed. Store the expected observation only when the oracle is already known.

If the observation is unknown, store observed and leave expected empty. Empty expected is not a pass. It is an unpromoted capture.

replays/
  2026-09-23T14-02Z_parse_header.json
  2026-09-23T14-11Z_retry_budget.json
seeds.lock

Proposed replay record:

{
  "id": "parse_header_a3f1",
  "command": "pytest tests/test_parse.py::test_header -q",
  "seed": 1729,
  "input_sha256": "e3b0c44298fc1c149afbf4c8996fb924...",
  "payload": {"raw": "X-Trace: 00-abc"},
  "observed": {"status": 500, "code": "TRACE_PARSE"},
  "expected": null,
  "promoted_to": null
}

The agent may append files under replays/. It may not rewrite input_sha256 on an existing id. A changed hash is a new case, not an edit.

Queue 2: property oracles

A property is a replay case that gained an independent oracle. The oracle must be writable without calling the function under test. assert f(x) == f(x) is not an oracle. assert parse(header).trace_id == header.split("-")[1] is.

Promotion rule, labeled as a proposal:

1. The same replay id passes on three consecutive seeded runs.
2. A human-authored expected value is present.
3. The property file imports the oracle from oracles/, not from the patched module’s test helpers if those helpers changed in the same diff.

# proposed: properties/test_trace_header.py
from oracles.trace import trace_id_from_raw

def test_trace_id_roundtrip(header: str) -> None:
    parsed = parse_header(header)
    assert parsed.trace_id == trace_id_from_raw(header)

Keep oracles/ out of the agent write set when the patch also touches production parsers. Otherwise the agent can move the goalpost and the property together.

Queue 3: quarantine with expiry

Quarantine is for mixed results under a pinned seed. It is not a skip list.

Run the same node seven times. If the tuple (outcome, observed_hash) is not constant, the case is unstable. Do not mark pytest.mark.skip. Write a quarantine proposal. CI must ignore proposals until a human moves the file into quarantine/active/.

# quarantine/active/retry_budget.yaml
id: retry_budget_9c2
seed: 1729
command: pytest tests/test_retry.py::test_budget -q
runs: 7
pass_count: 4
fail_count: 3
expires_at: 2026-10-07T00:00:00Z
repro: |
  python harness/replay.py --id retry_budget_9c2 --repeat 7
owner: humans-only

After expires_at, the gate fails closed. The failure returns to the replay queue. Agents may not extend expires_at. They may not create files under quarantine/active/.

Write-permission matrix

Path	Agent	Human	CI if missing
replays/*.json (new ids)	append	edit	warn
replays/*.json (existing hash)	deny	edit	fail
properties/	propose in PR	merge	fail if oracle import sits in patched package
oracles/	deny when prod parser changes	merge	fail
quarantine/proposed/	append	review	ignore
quarantine/active/	deny	merge	fail open tests; fail closed after expiry
seeds.lock	deny	edit	fail

The matrix is the contract. The suite size is not.

Harness artifact

The following is a proposed classifier, not a measured benchmark. It reads intent files, runs seeded repeats, and prints a queue decision. Label every unexecuted branch as such.

#!/usr/bin/env python3
"""Proposed promotion gate. Not executed against a public corpus here."""
from __future__ import annotations

import hashlib, json, subprocess, sys
from dataclasses import dataclass
from pathlib import Path

REPEAT = 7
PROMOTE_AFTER = 3

@dataclass
class Verdict:
    queue: str
    reason: str
    ok_for_agent_merge: bool

def sha(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()

def run_seeded(command: str, seed: int) -> tuple[int, str]:
    env_cmd = f"PYTHONHASHSEED={seed} {command}"
    proc = subprocess.run(env_cmd, shell=True, capture_output=True, text=True)
    body = (proc.stdout + proc.stderr).encode()
    return proc.returncode, sha(body)

def classify(record: dict) -> Verdict:
    seed = int(record["seed"])
    results = [run_seeded(record["command"], seed) for _ in range(REPEAT)]
    codes = {code for code, _ in results}
    bodies = {body for _, body in results}

    if record.get("expected") is None and codes == {0}:
        return Verdict("replay", "pass without oracle; do not promote", False)

    if len(codes) == 1 and len(bodies) == 1:
        if record.get("expected") and codes == {0}:
            return Verdict("property", "stable seeded pass with oracle", True)
        return Verdict("replay", "stable failure; keep as captured case", True)

    return Verdict("quarantine_proposal", "mixed outcomes under pinned seed", False)

def main(path: str) -> int:
    record = json.loads(Path(path).read_text())
    locked = json.loads(Path("seeds.lock").read_text())
    if record["id"] in locked and locked[record["id"]] != record["seed"]:
        print("seed mutation on existing id")
        return 2
    verdict = classify(record)
    print(f"{verdict.queue}: {verdict.reason}")
    return 0 if verdict.ok_for_agent_merge else 1

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Companion lock file:

{
  "parse_header_a3f1": 1729,
  "retry_budget_9c2": 1729
}

CI sketch:

# proposed ci fragment
steps:
  - run: python harness/check_write_set.py --deny oracles,quarantine/active,seeds.lock
  - run: python harness/promote.py replays/*.json
  - run: python harness/expire_quarantine.py --now 2026-09-23T00:00:00Z

check_write_set.py should diff the patch against the deny list. A one-line change to seeds.lock is a failed agent merge even if every test is green.

Numbered workflow

1. Freeze seeds.lock before the agent runs. Record the hash of that file in the job log.
2. Apply the agent patch. Reject the job if the deny list moved.
3. Re-run known replay ids with their locked seeds. New hashes on old ids fail the job.
4. Capture any new deterministic failure as a replay JSON. Empty expected is allowed. Empty command is not.
5. For ids that passed three seeded repeats and have an oracle import outside the patched package, open a property promotion. Do not auto-merge that promotion in the same job as the production patch.
6. For mixed results across seven repeats, write quarantine/proposed/ only. A human copies the file to quarantine/active/ with expires_at set no more than fourteen days out.
7. On expiry, delete the active freeze and return the id to replays/. Do not let the agent renew the clock.

Step 5 is the part most pipelines skip. Promotion is a separate review. Mixing production edits and oracle edits in one diff is how the goalpost moves.

Decision table for the merge job

Observation	Queue	Merge agent patch?
Seeded repeats identical, oracle present, oracles/ untouched	property	yes
Seeded repeats identical, expected null	replay	yes, with warning
Seeded repeats mixed	quarantine proposal	no
Existing replay hash changed	replay mutation	no
quarantine/active/ or seeds.lock in the diff	contract edit	no
Active freeze past expires_at	expired	no, until recaptured
New tests added, no new replay ids, production parser changed	missing capture	no

The last row matters. An agent that only adds passing tests after changing a parser has not captured the old failure mode. The replay queue should grow when behavior changes. It should not shrink.

Drafting properties without handing over the freeze file

Property candidates can come from a model that sees the unified diff and the replay JSON, then emits oracle sketches into a review comment. Humans copy accepted sketches into oracles/ in a follow-up commit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is enough to draft those oracle sketches from a diff, and the free server option can run the seeded replay loop in harness/promote.py. Keep quarantine/active/ and seeds.lock off that write path. The value is the queue split, not the vendor.

If you already have a runner, use it. The harness above does not require a particular host.

Limitations

Seeded repeats do not prove concurrency bugs. Seven runs will not exhaust a race. If the failure needs real time or a live network, pin a recorded transport fixture first or leave the case in replay with expected: null.

Hashing stdout is brittle when timestamps leak into logs. Strip clocks before hashing, or the quarantine queue will fill with noise.

Fourteen-day expiry is a policy default in this proposal, not a measured optimum. Shorten it if the module ships daily. Lengthen it only with a human note in the YAML, not with an agent commit.

The write-permission matrix assumes one agent per patch. Two agents in series can still launder a skip if the second agent is allowed to edit tests the first agent added. Close that by treating any test file touched in the last two agent commits as untrusted until a human commit lands on oracles/.

Who should not use this

Do not use this pipeline on a repo with no deterministic unit surface. Snapshot-heavy UI suites will hash-churn and starve the property queue.

Do not use it as a replacement for code review of authorization changes. An oracle that checks HTML shape will not catch a missing permission bit.

Do not point an unsupervised agent at quarantine/active/ because “the suite is noisy.” Noise is the signal that the freeze exists. Deleting the freeze is how the suite goes quiet and stays wrong.

The merge question is not “did tests pass?” It is “which queue absorbed the failure, and who was allowed to write that queue?” If you cannot answer that from the job log, the green bit is not evidence.