Pin Property Seeds in CI Before You Score an Agent Patch

A fresh property run that reports zero failures is not a merge score. It is a new roll of the dice. Agent patches should be judged against pinned RNG seeds and a checked-in corpus of shrunk examples, not against whatever Hypothesis or pytest-randomly happens to explore on this CI worker.

That distinction matters more as models start writing both the production diff and a nearby test file. The test file can be green for reasons that have nothing to do with the bug. A locked seed file is cheaper evidence. It is also boring enough to automate.

What a new seed actually measures

Property tests sample a space. The sample is a function of the seed, the deadline, and the example database. Change any of those and you are not repeating the last experiment. You are running a different one.

Agent patches exploit that gap without intending to. A model regenerates a test module. CI picks a new seed. The unlucky input that failed last Tuesday is not drawn. The job is green. The bug is still in main.

A proposed gate therefore treats exploration and scoring as two jobs. Exploration may use a rotating seed on a schedule. Scoring must replay a frozen seed and a frozen corpus. If those two disagree, the corpus wins.

Three files the patch cannot own

Keep independent oracles in a path the agent is not allowed to write. A CODEOWNERS rule or a CI path check is enough. Do not rely on the model to leave the files alone.

1. oracle/seeds.toml — the exact pytest-randomly seed and the Hypothesis deadline used for scoring.
2. oracle/corpus.jsonl — one shrunk failing input per line, with a stable id, the target test, and a SHA-256 of the payload.
3. oracle/quarantine.json — tests that failed under the pinned seed but are not yet diagnosed. Quarantine removes them from the pass set. It does not rotate the seed.

The production patch may change application code. It may add tests under tests/. It may not rewrite the oracle. If the diff touches oracle/, fail closed.

Proposed scoring workflow

Label the following as a proposed CI sequence, not as a production incident report. It is designed so a reviewer can run it locally with the same bytes CI will use.

1. Confirm the agent diff does not modify oracle/.
2. Hash oracle/corpus.jsonl and compare it to oracle/corpus.sha256.
3. Run the scoring suite with the pinned seed only.
4. Replay every corpus line against the patched tree.
5. Classify each line: still fails, now passes, or errored.
6. Apply the decision table below. Do not look at the agent-written test file for the merge bit.

Exploration can still run in a separate, non-blocking job with a free seed. New failures from that job are candidates for the corpus. They are not a pass condition for the current patch.

Artifact: lockfile, corpus line, and judge

oracle/seeds.toml is deliberately small. Deadlines belong here so a model cannot “fix” a slow property by raising the timeout in the test body.

# oracle/seeds.toml
[random]
pytest_randomly_seed = 20260921

[hypothesis]
deadline_ms = 400
max_examples_score = 40
# Scoring is replay. Do not let this job discover new examples.
suppress_database_write = true

Each corpus line is one independent oracle. The expect field is the only verdict the judge is allowed to use. fixed_by is optional and must match a reviewed ticket, not a model comment.

{"id":"inv-014","test":"tests/test_invoice_total.py::test_totals_are_non_negative","payload_sha256":"c8f3a1b0e44d6c91a7d2f0ab19e6c3d4e5f60718293a4b5c6d7e8f9012345678","payload":{"items":[{"qty":0,"price":"-0.01"}],"tax_bps":0},"expect":"fail","fixed_by":null}

The judge is a replay loop. It does not import the agent’s new assertions. Proposed Python, unexecuted here, using pytest as a subprocess so the oracle process cannot be patched by the same diff it is scoring.

# tools/score_agent_patch.py
# Proposed judge. Replay pinned seeds + corpus. Do not execute as-is against prod.
from __future__ import annotations

import hashlib, json, subprocess, sys, tomllib
from pathlib import Path

ORACLE = Path("oracle")
SEEDS = tomllib.loads((ORACLE / "seeds.toml").read_text())
SEED = str(SEEDS["random"]["pytest_randomly_seed"])


def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def deny_oracle_writes(diff_names: list[str]) -> None:
    touched = [n for n in diff_names if n.startswith("oracle/")]
    if touched:
        raise SystemExit(f"oracle path modified: {touched}")


def load_corpus() -> list[dict]:
    raw = (ORACLE / "corpus.jsonl").read_bytes()
    expected = (ORACLE / "corpus.sha256").read_text().strip()
    if hashlib.sha256(raw).hexdigest() != expected:
        raise SystemExit("corpus hash mismatch")
    return [json.loads(line) for line in raw.splitlines() if line.strip()]


def replay(line: dict) -> str:
    env = {
        "PYTEST_ADDOPTS": f"--randomly-seed={SEED} -q",
        "CORPUS_ID": line["id"],
        "CORPUS_PAYLOAD": json.dumps(line["payload"]),
    }
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", line["test"], "-p", "no:cacheprovider"],
        env={**dict(**{k: v for k, v in __import__("os").environ.items()}), **env},
        capture_output=True,
        text=True,
    )
    if proc.returncode == 0:
        return "pass"
    if proc.returncode == 1:
        return "fail"
    return "error"


def main(diff_names: list[str]) -> int:
    deny_oracle_writes(diff_names)
    quarantine = {
        row["test"]
        for row in json.loads((ORACLE / "quarantine.json").read_text())
    }
    failures = []
    for line in load_corpus():
        if line["test"] in quarantine:
            continue
        got = replay(line)
        expect = line["expect"]
        if expect == "fail" and got != "fail":
            failures.append((line["id"], "lost failing example", got))
        if expect == "pass" and got != "pass":
            failures.append((line["id"], "regressed known-good example", got))
        if got == "error":
            failures.append((line["id"], "replay errored", got))
    for item in failures:
        print(item)
    return 1 if failures else 0


if __name__ == "__main__":
    names = sys.argv[1:]
    raise SystemExit(main(names))

A thin CI wrapper keeps the seed visible in logs. Logs without the seed are not reproducible, and therefore not a score.

# Proposed CI fragment. Scoring job only.
set -euo pipefail
git diff --name-only origin/main...HEAD > /tmp/diff-names.txt
pytest -p no:cacheprovider \
  --randomly-seed="$(python -c 'import tomllib; print(tomllib.load(open("oracle/seeds.toml","rb"))["random"]["pytest_randomly_seed"])')" \
  tests/oracle_replay
python tools/score_agent_patch.py $(cat /tmp/diff-names.txt)

Decision table for the merge bit

Use one table. Do not average it with the color of the agent-authored tests.

Observation under pinned seed	Corpus effect	Merge bit
Agent tests green, corpus unchanged, all expect=fail still fail	No new evidence	Reject if the patch claimed to fix a corpus id
Agent tests green, one expect=fail now passes, fixed_by is set and reviewed	Claimed fix matches	Allow that id to flip to expect=pass in a human follow-up commit
Agent tests green, one expect=fail now passes, fixed_by is null	Unexplained disappearance of a failing example	Reject
Agent tests green, corpus line errors	Oracle cannot run	Reject
Agent tests red, corpus stable	Agent tests are not the score	Ignore the agent file; keep scoring the corpus
Pinned seed fails intermittently	Flake, not a seed problem	Add test to quarantine.json; do not change seeds.toml
Exploration job finds a new shrink	Candidate oracle	Append to corpus on main after review, not inside the agent PR

The important row is the unexplained disappearance. Models delete or weaken examples when they are allowed to edit the suite. A hashed corpus makes that deletion visible as a hash mismatch or as a lost fail.

Seed rotation is a scheduled job

Seeds go stale. A seed that never hits a branch is a weak oracle. Rotation still belongs on a calendar, not on every agent PR.

A proposed weekly job, separate from merge:

1. Run exploration with a new seed and a write-enabled Hypothesis database.
2. Collect unique shrinks that the current corpus does not already hash.
3. Open a human PR that only appends corpus lines and updates corpus.sha256.
4. After that PR merges, bump pytest_randomly_seed in a second, also-human PR.

Splitting those two commits keeps blame readable. If a later agent patch “fixes” a failure by reverting the seed, the diff is obvious.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Generation and judging should not share a process. A free model endpoint is a reasonable place to request the candidate patch. It is not a reasonable place to decide whether the patch is true. The judge needs the pinned seed, the corpus bytes, and a wall clock, not another sample from the same model.

A free server option is useful as the machine that runs tools/score_agent_patch.py. The laptop that prompted the model is a biased worker: it may have extra fixtures, a warmer Hypothesis database, or a different pytest plugin set. Replaying the corpus on a clean server image removes that bias. It does not require naming models, quoting quotas, or claiming a particular hardware profile. If the server is busy, queue the judge. Do not fall back to “the agent tests were green locally.”

If you already generate patches that way, point the scoring job at the same corpus files CI uses. That is the only product-shaped suggestion in this article.

Limitations

Pinned seeds do not create oracles. They only make yesterday’s oracles repeatable. A corpus of ten invoice payloads will not catch a tax-rounding bug in a path those payloads never enter.

Hypothesis example databases can rot when function signatures change. A replay error must be a reject, not a skip. Skipping trains the pipeline to drop evidence.

Quarantine is a leak. Every test in quarantine.json is a hole in the score. Cap the list and expire entries by date in review, not by a model suggestion.

This workflow also assumes you already have at least one property or parameterized test worth pinning. If the suite is all tautological unit tests of the agent’s own helpers, seed pinning will faithfully reproduce a useless pass.

Who should not use this

Skip the corpus gate if the change cannot execute the tests: generated docs, comment-only diffs, or vendored lockfiles. Skip it if the team cannot keep oracle/ human-owned. A corpus the agent is allowed to rewrite is worse than no corpus, because it produces a false merge bit.

Also skip it when the properties have no shrinking story — for example, tests that only assert “the function returns a dict.” There is nothing to pin. Write a stricter invariant first.

Teams that already dual-checkout parent and child for failure evidence can add seed pinning without replacing that check. The two oracles answer different questions. Dual-checkout asks whether the parent already failed. Seed pinning asks whether today’s pass is the same experiment as last week’s pass. Use both if you have the minutes. If you have only one job, pin the seed. A new dice roll is not a control.