Quarantine Test Drafts Until the Oracle Hash Is Unchanged

Agent patches fail merge for a boring reason. The suite went green because the same writer owned the assertions. A merge vote that includes tests from the patch write-set is not a test of the product. It is a test of the agent's willingness to agree with itself.

The fix is not "write more tests." The fix is a quarantine. Property hypotheses and fixture edits stay out of the merge predicate until a content-addressed oracle hash on the base branch is unchanged. Flaky names go into a freeze registry instead of a skip. Skips do not vote. Frozen names do not vote. Drafts do not vote.

This article is a reference workflow, not a production report. The code is a labeled proposal you can run locally. It does not claim catch rates, latency, or fleet size.

What votes, and what does not

Four objects, four privileges.

1. Hypothesis drafts — proposed properties, often agent-authored. They may run as diagnostics. They never vote.
2. Locked properties — invariant checks stored outside the agent's write-set and hashed on the base branch. They vote.
3. Golden fixtures — content-addressed inputs the product must accept. Format changes require a human hash update, not a silent rewrite.
4. Freeze registry — named tests that recently flickered. Frozen names are excluded from the vote and expire by date, not by hope.

If a patch touches (2) or (3), the gate fails closed. The agent can still propose a follow-up that updates the oracle. That follow-up is a different change with a different reviewer.

Why equality on agent fixtures is the wrong default

Equality tests are precise and brittle. They are also the cheapest thing an agent can satisfy: emit the bytes the assertion expects, then point the assertion at those bytes.

Property checks invert that. They state a relation that remains true across many inputs: round-trip, idempotence, order preservation, bound checks. The agent can still cheat by deleting the property. That is why the property file's digest is part of the merge predicate, not only the pytest exit code.

Green is a status. Ownership of the assertion is the control. Separate them, or the status is noise.

Proposed layout

repo/
  oracle/
    properties/
      test_roundtrip.py
      test_bounds.py
    fixtures/
      manifest.json
      cases/
        invoice_v1.json
    freeze.json
    HASHES
  drafts/
    hypotheses/          # never on the merge vote
  src/
  tests/                 # ordinary unit tests; vote only if not in write-set

oracle/ is read-only for agent patches. drafts/ is writable. The gate computes a digest of oracle/ from the merge base, then refuses the patch if the digest moved.

Step 1 — Hash the oracle tree

Proposed helper. Unexecuted until you wire it to CI.

# oraclekit/hashlock.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ORACLE_ROOT = Path("oracle")
HASH_ALGO = "sha256"


def file_digest(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def tree_manifest(root: Path = ORACLE_ROOT) -> dict[str, str]:
    entries: dict[str, str] = {}
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        if rel == "HASHES":
            continue
        entries[rel] = file_digest(path)
    return entries


def write_hashes(root: Path = ORACLE_ROOT) -> str:
    manifest = tree_manifest(root)
    blob = json.dumps(manifest, indent=2, sort_keys=True).encode("utf-8")
    digest = hashlib.sha256(blob).hexdigest()
    payload = {"algo": HASH_ALGO, "tree": digest, "files": manifest}
    (root / "HASHES").write_text(
        json.dumps(payload, indent=2) + "\n", encoding="utf-8"
    )
    return digest


def read_locked_tree(root: Path = ORACLE_ROOT) -> str:
    payload = json.loads((root / "HASHES").read_text(encoding="utf-8"))
    return payload["tree"]


def assert_oracle_unchanged(root: Path = ORACLE_ROOT) -> None:
    current = hashlib.sha256(
        json.dumps(tree_manifest(root), indent=2, sort_keys=True).encode("utf-8")
    ).hexdigest()
    locked = read_locked_tree(root)
    if current != locked:
        raise SystemExit(
            f"oracle tree moved: locked={locked[:12]} current={current[:12]}"
        )

Run python -m oraclekit.hashlock --write only on a human commit that intends to change the oracle. Agent patches run a --check against the merge-base HASHES file, not the working-tree copy they might have rewritten.

The digest is a detector. It is not access control. Protect the path with CODEOWNERS or the equivalent. Anyone who can rewrite HASHES on the base branch owns the contract.

Step 2 — Load fixtures by digest, not by a retargetable path

oracle/fixtures/manifest.json maps case names to sha256 and a schema version. Property tests load bytes by digest. They do not load a sibling file the agent can replace and then re-point.

{
  "invoice_v1": {
    "file": "invoice_v1.json",
    "sha256": "replace-with-real-digest",
    "schema": 1
  }
}

# oracle/properties/test_roundtrip.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

import pytest

from src.codec import decode, encode

CASES = Path("oracle/fixtures/cases")
MANIFEST = json.loads(Path("oracle/fixtures/manifest.json").read_text())


def load_case(name: str) -> bytes:
    meta = MANIFEST[name]
    data = (CASES / meta["file"]).read_bytes()
    digest = hashlib.sha256(data).hexdigest()
    if digest != meta["sha256"]:
        pytest.fail(f"fixture {name} digest mismatch")
    return data


@pytest.mark.oracle
@pytest.mark.parametrize("name", sorted(MANIFEST))
def test_encode_decode_roundtrip(name: str) -> None:
    original = load_case(name)
    round_tripped = encode(decode(original))
    assert decode(round_tripped) == decode(original)

The assertion is a relation, not a golden output sitting next to the agent's patch. If the product changes a codec default, a human updates the fixture digest in the same commit that changes the codec contract. That commit is not an agent merge.

Add a second locked property for bounds. Same marker. Same hash tree.

# oracle/properties/test_bounds.py
import pytest
from src.codec import decode

@pytest.mark.oracle
def test_amount_non_negative(invoice_bytes: bytes) -> None:
    doc = decode(invoice_bytes)
    assert doc.amount >= 0
    assert doc.line_count >= 1

Keep domain facts here. Do not keep "equals this exact JSON the model just wrote."

Step 3 — Freeze flakes by identity, not by skip

A skip is an unrecorded abstention. A freeze is recorded, dated, and counted.

Proposed oracle/freeze.json:

{
  "version": 1,
  "frozen": [
    {
      "nodeid": "tests/test_api.py::test_list_order",
      "reason": "order flicker on empty page boundary",
      "first_seen": "2026-09-12",
      "expires": "2026-09-26",
      "owner": "platform-tests"
    }
  ]
}

Proposed pytest hook:

# oracle/conftest.py
from __future__ import annotations

import json
from datetime import date
from pathlib import Path

import pytest

FREEZE_PATH = Path("oracle/freeze.json")


def _frozen_map() -> dict[str, dict]:
    payload = json.loads(FREEZE_PATH.read_text(encoding="utf-8"))
    return {row["nodeid"]: row for row in payload.get("frozen", [])}


def pytest_collection_modifyitems(config, items) -> None:
    frozen = _frozen_map()
    for item in items:
        if item.nodeid in frozen:
            item.add_marker(pytest.mark.freeze)


def pytest_runtest_setup(item) -> None:
    row = _frozen_map().get(item.nodeid)
    if row is None:
        return
    today = date.today().isoformat()
    if row["expires"] < today:
        pytest.fail(f"freeze expired for {item.nodeid}: {row['reason']}")
    pytest.skip(f"frozen until {row['expires']}: {row['reason']}")

Treat an expired freeze as a failure, not as a quiet re-enable. Quiet re-enable is how flakes return as merge green. Fourteen days is a policy example, not a measured optimum. Pick a TTL a human will actually honor.

Do not let an agent append to freeze.json. An agent that can freeze its failures has rebuilt skip.

Step 4 — Fail closed on policy, product, or empty vote

Proposed gate. Unexecuted example. Three failure classes, one binary merge bit.

# oraclekit/merge_gate.py
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

NON_VOTING_MARKERS = {"freeze", "draft", "hypothesis"}


def git_write_set(base: str, head: str) -> set[str]:
    raw = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...{head}"],
        text=True,
    )
    return {line.strip() for line in raw.splitlines() if line.strip()}


def hashes_at(rev: str) -> str:
    blob = subprocess.check_output(["git", "show", f"{rev}:oracle/HASHES"])
    return json.loads(blob)["tree"]


def gate(base: str, head: str, report: Path) -> int:
    write_set = git_write_set(base, head)
    oracle_hits = [p for p in write_set if p.startswith("oracle/")]
    if oracle_hits:
        print("oracle write-set is non-empty:")
        print("\n".join(oracle_hits))
        return 2

    if hashes_at(head) != hashes_at(base):
        print("oracle tree digest drifted from merge base")
        return 2

    results = json.loads(report.read_text(encoding="utf-8"))
    voting_failed = 0
    voting_passed = 0
    frozen = 0
    drafts = 0
    for row in results["tests"]:
        markers = set(row["markers"])
        if markers & NON_VOTING_MARKERS:
            frozen += int("freeze" in markers)
            drafts += int(bool(markers & {"draft", "hypothesis"}))
            continue
        if "oracle" not in markers:
            continue
        if row["outcome"] in {"failed", "error", "timeout"}:
            voting_failed += 1
        elif row["outcome"] == "passed":
            voting_passed += 1

    print(json.dumps({
        "voting_passed": voting_passed,
        "voting_failed": voting_failed,
        "frozen": frozen,
        "drafts_ignored": drafts,
    }))
    if voting_failed:
        return 1
    if voting_passed == 0:
        print("no voting oracle tests ran")
        return 3
    return 0


if __name__ == "__main__":
    sys.exit(gate(sys.argv[1], sys.argv[2], Path(sys.argv[3])))

Exit 2 is an oracle-policy failure. Exit 1 is a product failure. Exit 3 is an empty vote, which is also a failure. An empty vote is how a deleted property suite sneaks through.

Timeouts belong with failures, not with skips. A hung locked property is a red vote. Do not recode it as freeze without an owner and an expiry.

Step 5 — Draft hypotheses without giving them the gavel

Agents are useful at proposing invariants. They are not useful at certifying those invariants.

A draft file lives under drafts/hypotheses/ and carries pytest.mark.hypothesis. The merge gate drops those outcomes. A human promotes a draft by moving the file into oracle/properties/, regenerating HASHES, and opening a non-agent change.

# drafts/hypotheses/test_idempotent_encode.py
import pytest
from src.codec import decode, encode

@pytest.mark.hypothesis
@pytest.mark.draft
def test_encode_idempotent_on_decoded_doc(invoice_bytes: bytes) -> None:
    doc = decode(invoice_bytes)
    once = encode(doc)
    twice = encode(decode(once))
    assert once == twice

Passing is not a lock. A draft that passes is still a draft. Promotion is a path change plus a digest update, not a green check.

This is the only place a free remote coding environment belongs in the method. MonkeyCode's free model access can emit candidate properties from a failure log. MonkeyCode's free server option can run the locked oracle/ suite so the agent's laptop is not the runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those are availability facts. They are not claims about model names, quotas, hardware, duration, or quality. Draft output still enters quarantine. Server results still hash oracle/ from the merge base.

If you promote a draft, promote the invariant, not the example the model copied from a failing fixture.

Decision table

Observation	Merge vote	Next action
Locked property fails	Fail	Fix product, or human oracle change
Draft hypothesis fails	Ignore	Inspect; maybe promote later
Draft hypothesis passes	Ignore	Still ignore; passing is not a lock
Fixture digest mismatch	Fail closed	Restore bytes or human manifest update
Test in freeze.json, not expired	Abstain	Do not count as pass
Freeze expired	Fail	Re-qualify or extend with owner
Patch writes oracle/	Fail closed	Split into an oracle PR
Zero locked properties ran	Fail closed	Empty vote
Ordinary unit test in write-set	Demote from vote	Keep as diagnostic

The table is the policy. Pytest is just the collector.

What this does not catch

Property checks only encode invariants you can state. They miss visual layout, copy changes, and one-off business exceptions that have no algebraic shape. If you cannot write the relation in a sentence, do not pretend a hash will invent it.

A freeze registry can hide a real regression for its TTL. Short expiries reduce that window. They also create toil. An unowned freeze is a skip with extra JSON.

Content-addressed fixtures assume the input corpus is the contract. If your contract is "whatever production sent this morning," the manifest will churn and humans will rubber-stamp digests. Do not use this layout for that corpus. Use production replay in a non-voting replay lane instead.

A remote free runner is not a trust boundary. The digest detects drift. It does not stop a force-push to oracle/HASHES. Path locks and review rules still matter.

The gate also cannot see tests the collector never loaded. A broken conftest.py that silently drops oracle items produces exit 3 if you keep the empty-vote rule. Drop that rule and the same breakage looks like success.

Who should not use this

Do not install a freeze registry on a ten-test hobby repo. You will spend more time on JSON than on the product.

Do not quarantine drafts if the agent is not allowed to touch tests at all. A write-set ban is simpler. Use quarantine when agents are allowed to propose tests and you need a promotion path.

Do not hash-lock oracles for exploratory spikes. Spikes need cheap mutation. Promote the layout when a merge lane exists and green results are used as evidence.

Do not treat the gate as a substitute for review of product behavior. The oracle says the relations you encoded still hold. It does not say those relations are the ones users needed.

A minimal local run

Proposed sequence. Adjust paths to the repo.

python -m oraclekit.hashlock --write
pytest oracle/properties -q --json-report --json-report-file=/tmp/oracle.json
python -m oraclekit.merge_gate origin/main HEAD /tmp/oracle.json

If you add drafts:

pytest drafts/hypotheses oracle/properties -q \
  --json-report --json-report-file=/tmp/all.json
python -m oraclekit.merge_gate origin/main HEAD /tmp/all.json

The second command may print draft failures. The gate should still exit 0 when locked properties pass and oracle/ is untouched. That split is the entire method.

Keep freeze rows in the same human commit as the decision to stop counting a name. Point the three commands at a clean checkout if the agent's working tree is not a runner you trust. A free remote server is sufficient for that checkout. It is not required. The hash and the freeze file are the policy. The runner is just a place they execute.