Seal the Oracle Host Before an Agent Patch Touches Your Tree

The agent that writes a patch should never mount the bytes that decide whether that patch is correct. If generation and verification share one working tree, expected values leak into context, and the suite starts agreeing with itself. Split the work. Keep property checks, fixture bytes, and flake freezes on a replay host the generator cannot read.

This is a testing plan, not a model review. The merge question is narrow: did the diff survive oracles it could not see?

Why a shared tree fails

Most agent workflows clone the full repo, including tests/, golden files, and skip lists. That is convenient. It is also an oracle leak.

A property that lives next to the code under edit is prompt-adjacent. A fixture the model can open() is no longer an independent check. A freeze file the model can rewrite is not a quarantine; it is a mute button.

The failure mode is quiet. The job stays green. The diff looks small. The suite no longer measures the parent behavior you thought you still had.

Three artifacts that must stay off the generator

Treat verification as a second filesystem, not a second pytest flag.

1. Property modules — boolean claims about behavior, named and versioned, with no literals copied from production code.
2. Fixture objects — input bytes plus a digest. The replay host loads them. The generator never lists the directory.
3. Freeze ledger — flaky test ids, expiry, owner, and the last parent SHA that justified the freeze. The generator cannot write this file.

If any of the three is visible to the session that authors the patch, score the run as unverified. Do not argue about intent. Visibility is the defect.

Workflow: generate on one host, replay on another

The steps below are a proposed procedure. They are not a report of a production rollout.

1. Inventory the sealed surface. List property ids and fixture paths on the replay host only. Record sha256 for every fixture. Commit that inventory as sealed/MANIFEST.json.
2. Check out a generator workspace that omits sealed/. Sparse checkout, a second clone with sparse-checkout, or a tarball that excludes the directory. Confirm with a command, not a comment in the PR.
3. Author the patch in that workspace. The model may read src/ and any tests you deliberately keep public. It must not receive fixture bodies, expected hashes, or the freeze ledger.
4. Ship only the diff. Export git format-patch or a merge-request SHA. Do not rsync the generator tree onto the replay host.
5. Apply the diff on the replay host. Mount sealed/ read-only. Run properties, fixture hits, and the freeze ledger as three result files, not one job color.
6. Emit a merge packet. One JSON document: parent SHA, patch SHA, manifest digest, property outcomes, fixture hits, freeze breaches. Merge policy reads that file. Humans can still override. The packet is what you store.

A cheap way to keep the split real is to generate the patch in a free model session and replay the sealed suite on a free server that session cannot SSH into. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit that split if you already wanted two hosts; the method does not require the product, and this article does not claim named models, quotas, or hardware.

Directory layout

Keep public tests and sealed oracles in different trees. The generator clone should fail to even list the second tree.

repo/
  src/
  tests/
    public/                 # visible to the generator
  tools/
    replay_harness.py       # runs only on the replay host
  # The following path exists only on the replay host:
  sealed/
    MANIFEST.json
    properties/
      p_idempotent_put.py
      p_reject_empty_key.py
    fixtures/
      put_ok.bin
      put_empty_key.bin
    freeze.json

Verify the generator cannot see sealed paths before you prompt it.

# On the generator host. Expected: no such file.
test ! -e sealed && echo "sealed absent" || echo "ORACLE LEAK"

git sparse-checkout list
git ls-files sealed | wc -l
# Expected: 0

Manifest and freeze schema

sealed/MANIFEST.json is the inventory. Properties are code. Fixtures are bytes. The manifest binds them with digests so a silent fixture edit is a failed replay, not a green skip.

{
  "schema": "sealed-oracle-v1",
  "parent_sha_at_lock": "REPLACE_WITH_PARENT_SHA",
  "properties": [
    {"id": "p_idempotent_put", "path": "properties/p_idempotent_put.py"},
    {"id": "p_reject_empty_key", "path": "properties/p_reject_empty_key.py"}
  ],
  "fixtures": [
    {"id": "put_ok", "path": "fixtures/put_ok.bin", "sha256": ""},
    {"id": "put_empty_key", "path": "fixtures/put_empty_key.bin", "sha256": ""}
  ]
}

Fill the hashes from the replay host. Never paste them into the generator prompt.

python - <<'PY'
from pathlib import Path
import hashlib, json
root = Path("sealed")
man = json.loads((root / "MANIFEST.json").read_text())
for fx in man["fixtures"]:
    data = (root / fx["path"]).read_bytes()
    fx["sha256"] = hashlib.sha256(data).hexdigest()
(root / "MANIFEST.json").write_text(json.dumps(man, indent=2) + "\n")
print("manifest digests written")
PY

Freeze records belong on the same host. A freeze without expiry is a deleted test. A freeze the patch can edit is not a freeze.

{
  "schema": "freeze-ledger-v1",
  "entries": [
    {
      "test_id": "public.test_network_retry",
      "reason": "parent flake, not patch-induced",
      "owner": "platform-ci",
      "expires_utc": "2026-09-28T00:00:00Z",
      "parent_sha": "REPLACE_WITH_PARENT_SHA",
      "max_runs_seen_flaky": 3
    }
  ]
}

Replay harness (labeled example)

The following script is a compact example you can run locally. It is not a claim that any specific patch was scored with it. It checks three things: fixture bytes still match the manifest, each property function returns true against those bytes, and no freeze is expired or missing an owner.

# tools/replay_harness.py
from __future__ import annotations

import hashlib, importlib.util, json, sys
from datetime import datetime, timezone
from pathlib import Path

SEALED = Path("sealed")


def load_json(path: Path):
    return json.loads(path.read_text())


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_property(path: Path):
    spec = importlib.util.spec_from_file_location(path.stem, path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    if not hasattr(mod, "holds"):
        raise SystemExit(f"{path} has no holds(fixture_bytes) -> bool")
    return mod.holds


def check_fixtures(manifest) -> list[dict]:
    rows = []
    for fx in manifest["fixtures"]:
        path = SEALED / fx["path"]
        digest = sha256(path)
        rows.append({
            "id": fx["id"],
            "hit": path.is_file(),
            "digest_match": digest == fx["sha256"],
            "sha256": digest,
        })
    return rows


def check_properties(manifest, fixtures: dict[str, bytes]) -> list[dict]:
    rows = []
    for prop in manifest["properties"]:
        holds = load_property(SEALED / prop["path"])
        failures = [fid for fid, body in fixtures.items() if not holds(body)]
        rows.append({
            "id": prop["id"],
            "ran": True,
            "failed_fixtures": failures,
            "ok": not failures,
        })
    return rows


def check_freezes(ledger, now: datetime) -> list[dict]:
    rows = []
    for entry in ledger.get("entries", []):
        expires = datetime.fromisoformat(entry["expires_utc"].replace("Z", "+00:00"))
        rows.append({
            "test_id": entry["test_id"],
            "has_owner": bool(entry.get("owner")),
            "expired": expires <= now,
            "parent_sha_present": bool(entry.get("parent_sha")),
        })
    return rows


def main() -> int:
    manifest = load_json(SEALED / "MANIFEST.json")
    ledger = load_json(SEALED / "freeze.json")
    fixtures = {
        fx["id"]: (SEALED / fx["path"]).read_bytes()
        for fx in manifest["fixtures"]
    }
    packet = {
        "schema": "merge-packet-v1",
        "manifest_sha256": sha256(SEALED / "MANIFEST.json"),
        "fixtures": check_fixtures(manifest),
        "properties": check_properties(manifest, fixtures),
        "freezes": check_freezes(ledger, datetime.now(timezone.utc)),
    }
    fx_ok = all(r["hit"] and r["digest_match"] for r in packet["fixtures"])
    prop_ok = all(r["ok"] for r in packet["properties"])
    freeze_ok = all(
        r["has_owner"] and r["parent_sha_present"] and not r["expired"]
        for r in packet["freezes"]
    ) if packet["freezes"] else True
    packet["merge_ok"] = fx_ok and prop_ok and freeze_ok
    Path("merge_packet.json").write_text(json.dumps(packet, indent=2) + "\n")
    print(json.dumps({"merge_ok": packet["merge_ok"]}, indent=2))
    return 0 if packet["merge_ok"] else 2


if __name__ == "__main__":
    sys.exit(main())

A property module is one function. Keep it small. Pass fixture bytes in; return a boolean. Do not import production helpers that the patch just changed unless that import is the claim under test.

# sealed/properties/p_reject_empty_key.py
def holds(fixture_bytes: bytes) -> bool:
    # Example claim: empty keys are rejected. Replace with your parser.
    if fixture_bytes.startswith(b"KEY=\n"):
        return b"ERR_EMPTY_KEY" in fixture_bytes or True  # replace the or True
    return True

Label the or True as a placeholder. A property that cannot fail is not sealed evidence. Replace it before you enforce merge_ok.

Merge decision table

Score the packet, not the CI emoji. Use the table as policy text in the repo, next to the harness.

Condition	Packet fields	Merge
Fixture missing or digest mismatch	fixtures[].hit or digest_match is false	Block. Inventory drifted.
Property returns false on any sealed fixture	properties[].failed_fixtures non-empty	Block. Oracle rejected the patch.
Property never loaded	properties[].ran is false	Block. Silent skip is not a pass.
Freeze expired or owner blank	freezes[].expired or has_owner is false	Block. Quarantine is unpaid.
Freeze ledger changed in the agent diff	git diff includes sealed/freeze.json	Block. Generator wrote the mute list.
Public tests fail, sealed packet is green	public lane red, merge_ok true	Do not auto-merge. Inspect. Public tests may be tautologies or real regressions.
Sealed packet green, freeze ledger untouched	merge_ok true	Eligible for human review.

The last row is eligibility, not a ship decision. Oracle isolation answers whether the check was independent. It does not answer whether the product change is wanted.

What to log when a property fails

A boolean is not enough. Store the fixture id, the property id, and a short counterexample note. If you shrink inputs, write the shrunk bytes to an artifact directory on the replay host. Do not paste those bytes back into the next generator prompt unless you intend to leak the oracle again.

python tools/replay_harness.py
python - <<'PY'
import json
from pathlib import Path
p = json.loads(Path("merge_packet.json").read_text())
for row in p["properties"]:
    if not row["ok"]:
        print(row["id"], "failed on", row["failed_fixtures"])
PY

If the packet cannot name a fixture id, you do not have a replay. You have a log line.

Limitations

This plan does not measure coverage, mutation score, or user-facing quality. It measures whether verification used bytes the generator could not read.

It will not help if sealed/ is in the same Docker layer the agent mounts, if fixture bodies are duplicated under tests/public/, or if someone pastes MANIFEST.json into the prompt to "help the model." Sparse checkout is only as strong as the check in step 2.

Do not use this as a merge gate for security-sensitive diffs, cryptography, or anything where the sealed fixtures themselves are secrets you cannot store on the replay host. Do not use it as an excuse to skip reading the patch. A green packet can still rename a public API, drop a metric, or delete an error path that no fixture named.

Teams with no independent expected values should not adopt the harness yet. You would be sealing an empty directory. Solo scripts that never grow fixtures gain process without signal. If your freeze ledger is editable from the generator host, delete the ledger until that write path is gone.

The example property above is unfinished on purpose. Wire holds() to a real parser before you block merges on merge_ok.

Closing

Independent tests are a hosting problem as much as a source problem. Put properties, fixture digests, and freeze expiry on a machine the patch author never mounts, then score the packet those three lanes emit. If you try the split on a throwaway repo, the useful output is merge_packet.json, not a screenshot of a green job.