Pin Time and Commit Seeds Before You Score an Agent Patch

A merge score that reads the wall clock, draws an unpinned random stream, or still collects a test the suite already treats as flaky is not a score. It is a coin flip that happened to land. Replay is the constraint. If the same patch, the same fixtures, and the same seeds cannot produce the same pass/fail bits on a second worker, the first worker's result is not evidence.

This article specifies a scoring harness for agent-generated patches built from three artifacts: hermetic fixtures, a committed seed corpus for property checks, and a freeze file that excludes flaky tests from collection. The harness is an example. It is not a production incident report, and it does not claim a measured catch rate.

Property checks without those artifacts do not constrain an agent. They constrain a lucky seed.

Moving oracles cannot grade a diff

A property that calls random.randint on every CI worker is a lottery. A fixture that calls datetime.utcnow() is a lottery with a calendar. A flaky test left in the job and marked xfail is worse than either. It teaches the patch generator that silencing a check is a legal edit.

The failure mode is mechanical. The agent changes a parser. The property draws inputs that never hit the new branch. The job passes. A later worker draws a different seed and fails on the default branch. The merge record contains no explanation for the flip.

Cached expected outputs fail in a different way. They overfit one example. Agents can keep that example intact while changing every other input. Properties and metamorphic relations avoid the expected-value problem. They only do so when the input stream itself is an artifact, not a fresh RNG.

Three artifacts, one replayable score

Treat the following as source, not as cache.

1. Hermetic fixtures. Replace wall clock, ambient environment, and live HTTP with byte-locked doubles. The patch under test may read them. It may not rewrite them.
2. Seed corpus. Every property check consumes a committed list of seeds. After a human triages a counterexample, a new seed may be added. The agent may not delete or reorder seeds.
3. Flake freeze. A freeze file lists property ids that are out of scoring. Frozen ids must be absent from collection, not present and skipped. An in-place skip is a signal the agent can copy.

Those three files are the oracle. Source under src/ is the candidate. Mixing them in one writable tree is how a generator learns to edit the exam.

Workflow

1. Extract non-deterministic calls from the scoring tree

Search tests and properties for time, UUID, unseeded random, and live I/O. The commands below are a starting grep, not a complete inventory for every language.

rg -n "datetime\.(utcnow|now)|time\.time|uuid\.uuid4|random\.(random|randint)" tests properties
rg -n "requests\.(get|post)|httpx\.|urlopen" tests properties
rg -n "pytest.mark.(xfail|skip)|unittest.skip" tests properties

Record each hit as either "must become a fixture" or "must not run during scoring." Do not leave a third category. Ambient entropy that remains in the scoring tree will dominate any later property you add.

2. Pin time and I/O in one fake

The example below is stdlib-only. It is a proposal, not a library.

# harness/fakes.py
from __future__ import annotations

from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Callable


@dataclass(frozen=True)
class ByteExchange:
    url: str
    request_body: bytes
    response_status: int
    response_body: bytes


@dataclass
class FakeClock:
    instant: datetime

    def now(self) -> datetime:
        return self.instant

    def advance_ms(self, ms: int) -> None:
        from datetime import timedelta
        object.__setattr__  # clock is mutable on purpose during a single test
        self.instant = self.instant + timedelta(milliseconds=ms)


@dataclass
class FakeTransport:
    exchanges: dict[str, ByteExchange]
    seen: list[str] = field(default_factory=list)

    def fetch(self, url: str, body: bytes = b"") -> tuple[int, bytes]:
        self.seen.append(url)
        if url not in self.exchanges:
            raise AssertionError(f"unpinned URL in scoring path: {url}")
        ex = self.exchanges[url]
        if body != ex.request_body:
            raise AssertionError(f"request body drift for {url}")
        return ex.response_status, ex.response_body

Load exchanges from fixtures/*.json. If production code still calls urlopen after injection, fail the harness. A mock that silently performs real I/O is not hermetic. It is a trap.

3. Commit seeds next to the property

Local development may generate extra random inputs. Scoring CI must not. Scoring reads only seeds/<property_id>.json. Missing seed files are oracle_incomplete, which must exit non-zero. Do not treat an incomplete oracle as a pass.

# harness/seeds.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

SEED_ROOT = Path("seeds")


def load_seeds(property_id: str) -> list[dict[str, Any]]:
    path = SEED_ROOT / f"{property_id}.json"
    if not path.is_file():
        raise SystemExit(f"oracle_incomplete: missing {path}")
    payload = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(payload, list) or not payload:
        raise SystemExit(f"oracle_incomplete: empty corpus {path}")
    return payload


def assert_seed_diff_is_additive(before: list, after: list) -> None:
    if after[: len(before)] != before:
        raise SystemExit("seed corpus was rewritten or reordered")

A seed is a structured input, not a hash of the last passing run. Hashes hide the case that later fails. Structured seeds can be shrunk and promoted.

Example corpus for a header parser:

[
  {"raw": "Name: value", "expect_name": "Name"},
  {"raw": "Name:\tvalue", "expect_name": "Name"},
  {"raw": "Name: value\r\nX: y", "expect_name": "Name"},
  {"raw": ": empty-name", "expect_reject": true}
]

4. Freeze flakes by exclusion, not by skip

freeze.json is a collector filter. Frozen ids must not appear in the collected set. If they do, the suite still contains an in-place skip, and the harness refuses to start.

# harness/freeze.py
from __future__ import annotations

import json
from pathlib import Path

FREEZE_PATH = Path("freeze.json")


def load_freeze() -> set[str]:
    if not FREEZE_PATH.is_file():
        return set()
    data = json.loads(FREEZE_PATH.read_text(encoding="utf-8"))
    ids = data.get("excluded_property_ids", [])
    if not isinstance(ids, list) or not all(isinstance(x, str) for x in ids):
        raise SystemExit("oracle_incomplete: malformed freeze.json")
    return set(ids)


def assert_collected_obeys_freeze(collected: set[str], frozen: set[str]) -> None:
    leaked = collected & frozen
    if leaked:
        raise SystemExit(f"frozen ids still collected: {sorted(leaked)}")

Example freeze file:

{
  "excluded_property_ids": [
    "prop.http.retry_after_parse"
  ],
  "reason": {
    "prop.http.retry_after_parse": "intermittent under parallel workers; excluded from scoring until retimed"
  }
}

The agent diff must not touch freeze.json, seeds/, or fixtures/. Those paths are human-only because they change the oracle. A freeze is not a skip annotation the generator is allowed to imitate.

5. Generate on a different host than you score

Keep generation and scoring on separate checkouts. The generator may propose edits under src/. It must not receive a writable mount of seeds/, fixtures/, or freeze.json.

A practical split is to generate the patch on one host and score it on another. MonkeyCode's free model access and free server option can occupy the generate side of that split so the scorer never shares a writable checkout with the model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The only product claims used here are free model access and a free server option. This article does not name models, quotas, hardware, duration, or pass rates.

The scorer host clones the candidate branch, verifies the oracle paths are unchanged, then runs properties against the seed corpus. If generation and scoring share a working tree, fixture edits become an available strategy. Host split is process control. It is not, by itself, a security boundary.

6. Promote shrunk counterexamples; never retry-away a fail

When a property fails on a seed, shrink it and write a fixture. The next score run must include that pair. Do not discard a counterexample because a retry passed.

# harness/promote.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any


def shrink_header_case(case: dict[str, Any]) -> dict[str, Any]:
    raw = case["raw"]
    while len(raw) > 1:
        candidate = dict(case, raw=raw[:-1])
        if still_fails(candidate):
            raw = raw[:-1]
            case = candidate
        else:
            break
    return case


def append_seed(property_id: str, case: dict[str, Any]) -> None:
    path = Path("seeds") / f"{property_id}.json"
    corpus = json.loads(path.read_text(encoding="utf-8"))
    if case not in corpus:
        corpus.append(case)
        path.write_text(json.dumps(corpus, indent=2) + "\n", encoding="utf-8")


def still_fails(case: dict[str, Any]) -> bool:
    raise NotImplementedError("wire to the property under test")

still_fails is a hook. Wire it to the same function the scorer uses. Promotion is a human action after the hook returns true. Automated promotion from the generator's own retry loop is how tautological fixtures get merged.

Decision table

Symptom in the scoring job	Replayable?	Required action
datetime.now() or time.time() in a property	No	Inject FakeClock
Unseeded random in a property	No	Replace with committed seeds
xfail / skip still collected	No	Exclude via freeze.json
Live HTTP or DNS	No	Byte-locked FakeTransport
Patch deletes or reorders seeds	No	Reject the patch
Patch edits freeze.json or fixtures/	No	Reject the patch
Missing seed file	No	Exit oracle_incomplete
All collected properties hold on the corpus	Yes, as far as the corpus goes	Human review still required

The last row is not a merge license. It is a statement that the oracle replayed. Coverage outside the corpus is untested by this harness.

Example scorer

# harness/score.py
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

from freeze import assert_collected_obeys_freeze, load_freeze
from seeds import load_seeds

ORACLE_PATHS = ("seeds/", "fixtures/", "freeze.json", "harness/")


def oracle_paths_changed(base: str, head: str) -> list[str]:
    diff = subprocess.check_output(
        ["git", "diff", "--name-only", base, head, "--", *ORACLE_PATHS],
        text=True,
    )
    return [line for line in diff.splitlines() if line]


def collect_property_ids() -> set[str]:
    return {p.stem for p in Path("seeds").glob("*.json")}


def run_property(property_id: str) -> None:
    cases = load_seeds(property_id)
    mod = __import__(f"properties.{property_id}", fromlist=["hold"])
    for i, case in enumerate(cases):
        if not mod.hold(case):
            raise SystemExit(f"property_failed:{property_id}:seed={i}")


def main() -> None:
    base, head = sys.argv[1], sys.argv[2]
    changed = oracle_paths_changed(base, head)
    if changed:
        raise SystemExit("oracle_edit_forbidden: " + ",".join(changed))
    frozen = load_freeze()
    collected = collect_property_ids() - frozen
    assert_collected_obeys_freeze(collect_property_ids(), frozen)
    if not collected:
        raise SystemExit("oracle_incomplete: nothing left to score")
    for property_id in sorted(collected):
        run_property(property_id)
    print(json.dumps({"replayable": True, "properties": sorted(collected)}))


if __name__ == "__main__":
    main()

Run it as a separate job after generation, not as a pytest plugin inside the agent's working tree.

python harness/score.py origin/main HEAD

A property module is a pure function over a case dict:

# properties/header_name.py
from __future__ import annotations

from typing import Any

from src.header import parse_name, Rejected


def hold(case: dict[str, Any]) -> bool:
    raw = case["raw"]
    if case.get("expect_reject"):
        try:
            parse_name(raw)
        except Rejected:
            return True
        return False
    return parse_name(raw) == case["expect_name"]

Keep hold free of I/O. If the unit under test needs time or HTTP, pass the fakes in through production seams, not through globals the agent can overwrite in the test package.

Limitations

A fake clock does not detect true races. It only removes calendar drift from the score. Concurrent code still needs a different oracle: thread sanitizers, deterministic schedulers, or a human-written concurrency test that is not in the agent's write set.

A seed corpus can overfit. Ten seeds that all share one shape will green-light a parser that fails on the eleventh. Promotion of shrunk failures slows that drift. It does not stop it. Rotate humans onto corpus review the same way you rotate reviewers onto production code.

A freeze file that is never triaged becomes a junk drawer. Exclusion is safer than in-place skip because the agent cannot see a skip mark to copy. It is not safer than fixing the flake. Frozen ids need an owner and a review date. This article does not specify expiry policy; expiry without triage just reintroduces noise.

Host split does not encrypt the oracle. If the generator can read seeds/ over the network, it can still overfit. Writable mounts are the minimum split. Confidential oracles need an additional control that this harness does not provide.

The example uses JSON files and a Python importer. Other languages need the same three artifacts, not this file layout. Translating the layout without pinning time and seeds does not preserve the property.

Who should not use this approach

Do not use this harness as a merge license for safety-critical, cryptographic, or access-control patches. Replayable properties on a seed corpus are still a sample. Those changes need a human reviewer who can reason about cases the corpus never lists.

Do not use it if the agent is allowed to edit freeze.json, seeds/, or fixtures/. The workflow collapses as soon as the candidate can rewrite the oracle.

Do not use it on suites that have no seams for time and I/O. Injecting fakes by monkeypatching private tests the agent also owns will not hold. Production code needs constructors or parameters that accept a clock and a transport.

Do not use it as a substitute for compiler errors, linters, or typechecks. Those gates answer different questions. A replayable property score that ignores a type error is still a broken merge process.

Teams whose tests are already hermetic, whose properties already consume committed seeds, and whose flaky ids are already excluded from collection do not need this harness. They need a review process for new seeds. Adding a second generator host on top of a working oracle is optional.

The score this article defines is narrow on purpose: same fixtures, same seeds, same collected ids, same bits. Anything wider than that is a different experiment and should be labeled as such.