Pin Property Seeds Outside the Agent Write Tree

Property checks do not gate a merge when the agent can rewrite the seeds, the strategy, or the oracle in the same commit as the production diff. Example assertions fail the same way, only faster. They travel with the code they bless.

Split the run. Pin property seeds and oracle modules on a path the patch cannot write. Hash fixtures at review. Send flakes to a quarantine executor that still runs and never votes. One merge bit remains. It comes from the sealed path only.

A single pytest process is the wrong unit of trust

Most agent workflows still issue one command: pytest. That process mixes three trust levels. Patch-local examples, human-owned properties, and historically flaky nodes all collapse into the same exit code.

A green run then means “nothing in this mixed bag failed.” It does not mean the oracle was untouched. It does not mean fixtures still match the reviewed bytes. It does not mean a flake was absent rather than rewritten into a skip.

Treat those as separate executors. Keep a single merge bit. Derive it from the sealed property executor, plus any human-owned invariants that also live off the write tree.

Executor 1: the patch tree is telemetry

The agent may change production code. It may add example tests under tests/examples/. Those tests are a log of what the model believed it did.

Run them. Store JUnit XML. Do not let a new assertion, a deleted failure, or a rewritten golden file turn the merge bit on. Executor 1 answers a narrow question. It does not answer whether the change holds.

Executor 2: seeds, strategy, and oracle leave the write tree

A property check needs three pinned artifacts. A seed vector. A strategy that builds inputs from those seeds. An oracle that accepts or rejects post-patch behavior. All three belong off the write tree.

A layout that makes the boundary visible:

repo/
  src/                     # writable by the patch
  tests/examples/          # writable by the patch
  .sealed/
    oracle/
      seeds.json
      properties.py
      MANIFEST.sha256
    fixtures/
      MANIFEST.sha256
    quarantine.json

CI must mount .sealed/ read-only. If the patch touches that directory, reject before any test process starts. A comment in CONTRIBUTING.md is not a mount.

Step 1 — hash the sealed tree at review and in CI

Proposal (unexecuted example). The manifest is a sorted map of relative path to hex digest. Recompute it in the check job. Compare bytes, not intent.

# sealed_hash.py — proposal
from __future__ import annotations

import hashlib
import json
from pathlib import Path


def file_sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def manifest(root: Path) -> dict[str, str]:
    out: dict[str, str] = {}
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        if rel == "MANIFEST.sha256":
            continue
        out[rel] = file_sha256(path)
    return out


def write_manifest(root: Path) -> None:
    body = json.dumps(manifest(root), indent=2, sort_keys=True) + "\n"
    (root / "MANIFEST.sha256").write_text(body, encoding="utf-8")


def assert_unchanged(root: Path) -> None:
    expected = (root / "MANIFEST.sha256").read_text(encoding="utf-8")
    actual = json.dumps(manifest(root), indent=2, sort_keys=True) + "\n"
    if actual != expected:
        raise SystemExit(f"sealed path drifted: {root}")

Step 2 — load seeds from JSON, never from patch-local parametrize

The seed file is the reproducibility contract. Values only. No lambdas, no imports, no embedded Python.

{
  "cases": [
    {"id": "empty-buf", "n": 0, "flag": false},
    {"id": "one-block", "n": 4096, "flag": true},
    {"id": "odd-align", "n": 7, "flag": false}
  ]
}

Proposal (unexecuted example). Executor 2 imports production code as a built artifact. It does not collect tests from the patch tree.

# sealed_runner.py — proposal
from __future__ import annotations

import importlib.util
import json
import os
from pathlib import Path

SEALED = Path(os.environ["SEALED_ORACLE_PATH"])


def load_oracle():
    spec = importlib.util.spec_from_file_location(
        "sealed_properties", SEALED / "oracle" / "properties.py"
    )
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod


def main() -> None:
    seeds = json.loads((SEALED / "oracle" / "seeds.json").read_text())
    oracle = load_oracle()
    failures = []
    for case in seeds["cases"]:
        try:
            oracle.check(case)
        except Exception as exc:
            failures.append((case["id"], type(exc).__name__, str(exc)))
    if failures:
        for row in failures:
            print("FAIL", row)
        raise SystemExit(len(failures))
    print("sealed properties passed", len(seeds["cases"]))


if __name__ == "__main__":
    main()

If the agent adds a seed that always passes, the manifest changes and the job fails closed. If it removes a failing seed, the same gate fires. That is the point of pinning bytes rather than trusting a collected suite.

Step 3 — hash fixtures the patch may read, not the ones it may write

Example goldens under tests/examples/ can move with the patch. They are executor-1 evidence. Input fixtures that the oracle depends on are different. Those files sit under a hashed directory. Drift is a failed check, not a warning.

python3 sealed_hash.py --assert .sealed/oracle
python3 sealed_hash.py --assert .sealed/fixtures
git diff --name-only origin/main...HEAD | grep -E '^\.sealed/' && exit 1

Executor 3: quarantine, not deletion, not a timer

Flakes do not disappear because they annoy a merge. Move them.

quarantine.json is a reviewed file inside the sealed path. Each entry names a node id, the last trusted hash of that test file, and a reason string. Executor 3 runs those nodes on a side channel. Failures write reports. They do not flip the merge bit. They also do not expire on a clock.

{
  "entries": [
    {
      "nodeid": "tests/legacy/test_retry.py::test_backoff_window",
      "file_sha256": "6b1d...",
      "reason": "timing depends on host load; do not vote"
    }
  ]
}

Thaw is a human edit of that catalog. Someone removes the entry, updates MANIFEST.sha256, and accepts that the node returns to a human-owned invariant list or stays deleted. Automatic re-enable is how flakes re-enter the vote. A TTL is a re-enable with extra steps.

Numbered check job

1. Freeze the sealed path. Paste the current MANIFEST.sha256 into the merge request body so review sees the pin.
2. Generate the agent patch against src/ and tests/examples/ only. Diff the write set. Fail closed if .sealed/ appears.
3. Recompute fixture hashes. Compare to .sealed/fixtures/MANIFEST.sha256.
4. Run executor 2 with SEALED_ORACLE_PATH on a read-only mount. Do not pass the patch tree as a test root in this process.
5. Run executor 1 for telemetry. Publish XML next to the merge bit. Do not OR it into the gate.
6. Run executor 3 from quarantine.json. Archive stdout. Leave the merge bit unchanged.
7. Merge only if executor 2 passed and both manifests matched.

export SEALED_ORACLE_PATH=/mnt/sealed
python3 sealed_hash.py --assert "$SEALED_ORACLE_PATH/oracle"
python3 sealed_hash.py --assert "$SEALED_ORACLE_PATH/fixtures"
python3 sealed_runner.py
pytest tests/examples --junitxml=telemetry.xml || true
python3 quarantine_runner.py --catalog "$SEALED_ORACLE_PATH/quarantine.json" \
  --junitxml=quarantine.xml || true

The || true on executors 1 and 3 is deliberate. Those processes must not be the shell status that CI treats as green. Executor 2 is the status.

Decision table

Signal	Source	Merge bit
.sealed/ path in the diff	write-set grep	fail closed
MANIFEST.sha256 mismatch	sealed hash	fail closed
property failure on a pinned seed	executor 2	fail closed
example test fail or new pass	executor 1	record only
fixture bytes the oracle reads changed	fixture manifest	fail closed
example golden rewritten in the patch	executor 1	record only
quarantined node fail or pass	executor 3	record only
catalog thaw without manifest update	sealed hash	fail closed

Isolation between generate and check

Generation and checking should not share a writable workspace. A model that can see .sealed/properties.py will try to edit it. A check job that installs the patch tree over the oracle path undoes the seal.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can hold that split: the model proposes a diff against src/ and tests/examples/; the server job mounts the sealed oracle path read-only and refuses to copy the patch tree over it. That is an isolation layout. It is not a quality claim about any model.

Pin PYTEST_ADDOPTS, --confcutdir, and the working directory on the check side. Prefer a second checkout that contains only .sealed/ plus the built artifact. Parent-directory conftest.py files are a common way a “sealed” process stops being sealed.

Limitations

Hashing a directory does not prove the properties are the right ones. A sealed oracle that only checks result is not None is still a tautology. It is just a tautology the agent cannot edit in-band. Review still has to read properties.py.

Quarantine hides duration. A test that fails twice a week can sit in the catalog for months if no one thaws it. That is safer than a flip-flopping gate. It is not a substitute for deleting dead tests.

Seeds must be data. A seed file that contains code is an oracle with extra syntax. Keep JSON numbers, strings, booleans, and lists. Rebuild richer inputs inside the sealed properties.py, which is itself hashed.

This plan also assumes CI can mount a path read-only. If it cannot, stop. A naming convention is not a seal.

Who should not use this

Do not install three executors on a repo with no agent in the loop. Human pull requests already have review. The extra mounts cost more than they return.

Do not use pinned seeds as a stand-in for an oracle that lives outside the process: a device, a ledger, a clock, a cluster. Those faults will not appear in seeds.json.

Do not thaw quarantine.json from the same session that produced the patch. The catalog is a human-owned file. If the check host cannot keep it read-only, the rest of the layout is theater.

The merge bit should answer one question: did the sealed properties still hold for the pinned seeds after the patch, with fixtures that still match the reviewed bytes? Everything else is evidence. Keep it. Do not let it vote.

Start with one property module and one quarantined node. Expand the seed list only after the mount and the manifest check fail closed in a dry run.