Don't Xfail the Flake. Journal Its Seed.

Xfail is how an agent patch hides. A generated change that makes a test flicker is evidence. Skip that test and you delete the only cheap signal the suite produced.

This workflow uses three on-disk artifacts as the merge gate: a seed journal, a fixture digest, and a property budget. A green pytest run is not the gate. Replay is.

Why green is cheap

Agent patches often arrive with extras. New tests written by the same model. skip or xfail on anything that still moves. The suite stays green. The behavior does not.

Property checks fail on a seed. If you do not store that seed, you cannot separate a regression from scheduler noise on a small free server. Fixtures fail in a different way. The agent rewrites a golden file to match the new code. The assertion still passes. The contract moved.

The policy is narrow. Replay known flakes until they are deterministic. Hash fixtures the agent could rewrite. Spend a fixed example budget on properties, then stop.

Where a free model and a free server fit

You can generate the patch with a free model and run the suite on a free server. That pairing is enough for this method. It is not enough to trust the model as the author of both the code and the tests that certify it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are one place to generate a patch and execute the commands below. The journal format does not depend on that pairing. Paste a diff in by hand and keep the same three files.

The three artifacts

Keep them next to the suite. Prefer a directory the agent is not asked to edit.

1. seed_journal.jsonl — one record per non-deterministic failure.
2. fixture_digest.json — sha256 of every golden file the patch is allowed to see.
3. property_budget.toml — max examples and deadline for property search.

Proposed journal row (JSON Lines):

{"id":"tests/test_normalize.py::test_normalize_is_idempotent","seed":384421,"env":"a1b2c3d4e5f6g7h8","status":"open"}

status stays open until replay passes twice in a row on the same env digest. It is not xfail. Deleting an open row is a failed review, not cleanup.

Step 1 — Freeze the process environment

Hash randomization and plugin drift create flakes that look like product bugs. Pin them before the agent runs.

export PYTHONHASHSEED=0
export PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
python -m pytest -p hypothesis tests -q

Record an env digest at the start of every run. Proposed helper:

# env_digest.py
from __future__ import annotations

import hashlib
import os
import sys

import pytest


def env_digest() -> str:
    parts = [
        sys.version.split()[0],
        pytest.__version__,
        f"hashseed={os.environ.get('PYTHONHASHSEED', 'unset')}",
        f"autoload={os.environ.get('PYTEST_DISABLE_PLUGIN_AUTOLOAD', '0')}",
    ]
    raw = "|".join(parts).encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:16]


if __name__ == "__main__":
    print(env_digest())

If the digest changes, old journal rows are not comparable. Do not close them. Stamp the new env and replay.

Step 2 — Classify tests before the patch lands

Do this as a checklist. Not as a feeling.

1. Property — pure function, no clock, no network. Eligible for a budgeted search.
2. Fixture — bytes on disk that define a contract: golden JSON, recorded payloads, sample binaries.
3. Seedy — anything that has failed with a different result under the same env digest.

Everything else is out of scope for this policy. UI timing tests do not belong in the journal. They belong in a slower suite you do not run on every agent patch.

Example property test. Treat it as a template, not as a run against a named model.

# tests/test_normalize.py
from hypothesis import given, settings, strategies as st

from app.normalize import normalize_ids


@settings(max_examples=80, deadline=200)
@given(st.lists(st.integers(min_value=0, max_value=10_000), max_size=40))
def test_normalize_is_idempotent(ids):
    once = normalize_ids(ids)
    twice = normalize_ids(once)
    assert once == twice
    assert sorted(once) == once

Eighty examples is a budget. It is not a quality score. Raise it only after replay is clean.

Step 3 — Hash fixtures the agent can see

Golden files are the easiest lie. The model updates code and the expected bytes in the same diff.

# fixture_digest.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("tests/fixtures")
OUT = Path("fixture_digest.json")


def file_sha(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def build_digest() -> dict[str, str]:
    files = sorted(p for p in ROOT.rglob("*") if p.is_file())
    return {p.as_posix(): file_sha(p) for p in files}


if __name__ == "__main__":
    OUT.write_text(json.dumps(build_digest(), indent=2) + "\n", encoding="utf-8")

CI check:

python fixture_digest.py
git diff --exit-code fixture_digest.json

If the digest changes, the patch must explain the contract change in a human-written note. A model comment in the same diff does not count. Split the fixture change into a separate review, or reject the patch.

Step 4 — Journal flakes instead of marking xfail

Forbid xfail on this suite. Capture a seed from the failure text. Hypothesis already prints a replay hint; scrape that rather than private exception attributes.

# conftest.py
from __future__ import annotations

import json
import os
import re
from pathlib import Path

import pytest
from hypothesis import Phase, settings

from env_digest import env_digest

JOURNAL = Path("seed_journal.jsonl")
SEED_RE = re.compile(r"seed\s*=\s*(?P<seed>\d+)")

settings.register_profile(
    "ci",
    max_examples=80,
    deadline=200,
    phases=(Phase.explicit, Phase.reuse, Phase.generate, Phase.shrink),
)
settings.load_profile(os.environ.get("HYPOTHESIS_PROFILE", "ci"))


def pytest_runtest_setup(item):
    if item.get_closest_marker("xfail"):
        pytest.fail("xfail is forbidden on this suite; journal the seed instead")


def pytest_runtest_logreport(report):
    if report.when != "call" or not report.failed:
        return
    text = str(report.longrepr) if report.longrepr else ""
    match = SEED_RE.search(text)
    row = {
        "id": report.nodeid,
        "seed": int(match.group("seed")) if match else None,
        "env": env_digest(),
        "status": "open",
    }
    with JOURNAL.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")

Replay an open row:

python -m pytest tests/test_normalize.py::test_normalize_is_idempotent \
  --hypothesis-seed=384421 -q

Proposed closer. Two consecutive passes on the same env digest. One pass can be luck on a contended free server.

# replay_open.py
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

from env_digest import env_digest

JOURNAL = Path("seed_journal.jsonl")


def load_rows() -> list[dict]:
    if not JOURNAL.exists():
        return []
    return [json.loads(line) for line in JOURNAL.read_text(encoding="utf-8").splitlines() if line.strip()]


def main() -> int:
    env = env_digest()
    open_rows = [r for r in load_rows() if r.get("status") == "open" and r.get("env") == env]
    if not open_rows:
        print("no open rows for this env")
        return 0
    failed = 0
    for row in open_rows:
        cmd = [sys.executable, "-m", "pytest", row["id"], "-q"]
        if row.get("seed") is not None:
            cmd.append(f"--hypothesis-seed={row['seed']}")
        first = subprocess.run(cmd).returncode
        second = subprocess.run(cmd).returncode
        if first != 0 or second != 0:
            print(f"REPLAY FAIL {row['id']} seed={row.get('seed')}")
            failed += 1
        else:
            print(f"REPLAY PASS {row['id']} seed={row.get('seed')} (still open until you edit status)")
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

Closing a row is a human edit. The script only reports. That split is deliberate. An agent that can flip open to closed will do it.

Step 5 — Run order on a small machine

Free servers are easy to starve. Order the work so the cheapest diagnostic runs first.

1. Replay every open journal row for this env digest. If replay fails, reject the patch. Do not search for new properties yet.
2. Rebuild fixture_digest.json and demand a clean diff unless a human note is in the review.
3. Run property tests with the budget file.
4. Run the remaining deterministic unit tests.
5. Stop. Do not start the UI suite on this box.

# property_budget.toml
max_examples = 80
deadline_ms = 200
max_workers = 1

export PYTHONHASHSEED=0
export PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
export HYPOTHESIS_PROFILE=ci
python env_digest.py
python replay_open.py || exit 1
python fixture_digest.py
git diff --exit-code fixture_digest.json || exit 1
python -m pytest tests -q

Keep shrink enabled. A shrunk example is the only form worth journaling. Parallel workers are not worth it here. They scramble timing and make replay harder to trust.

Decision table

Symptom	Action	Not an action
Property fails with a seed	Journal the seed, reject the patch	Mark xfail
Same test fails with a new seed, same env	Keep both rows open	Delete the old row
Golden file bytes change	Require a human note, split the review	Trust the agent's comment
Failure only when PYTHONHASHSEED is unset	Pin the seed, do not journal as product	Rerun until green
Timeout under deadline_ms	Shrink the strategy, or mark out of scope	Raise the deadline without bound
Replay passes once on a busy box	Keep status=open	Close the row
Agent deletes an open row	Fail review	Treat it as cleanup

Limitations

This policy does not detect a wrong spec. If the property encodes the same misconception as the code, both will agree. Write properties for invariants you can state without the implementation: idempotence, monotonicity, round-trip, bounds.

Seed journals do not help when the flake is time, network, or another process. Those tests are out of scope. Moving them into the journal produces rows you cannot replay.

The env digest is coarse. It does not capture CPU count, load, or disk latency. That is why closing a row needs two passes. That is why property tests run with one worker on a free server.

The agent can edit the journal if the file sits in the same tree as the patch. If you cannot keep it outside the write path, check seed_journal.jsonl into review and refuse deletions of open rows.

Budgets are not coverage. Eighty examples will miss rare branches. Increase examples only after replay is quiet, and only on functions whose input strategy is honest.

Who should not use this

Do not use this as a merge gate if your suite is mostly browser or mobile UI. There is no useful seed.

Do not use it if you cannot pin PYTHONHASHSEED and plugin load. The journal will lie.

Do not use it as an excuse to let the agent author the properties. The model can generate candidates. A human has to accept the invariant.

If you need a skip list to ship today, this policy will block you. That is the point. Park non-deterministic tests in a separate suite and keep that suite off the agent path.

Close

A green bar after an agent patch is a hypothesis about the suite, not about the product. Journal the seed. Hash the fixture. Budget the search. Close rows only when replay is boring.

If you copy one file, copy the journal. Everything else exists to keep that file small.