One Run in Two Hundred: Debugging a Silent Fixture Fallback

The bug was not a race condition. It was a swallowed parse error.

One test read a fixture while another test rewrote the same temp file. The reader got half a JSON document. Then it fell back to empty defaults and passed.

The symptom: a green suite with a red job

Our ingest test failed about once every two hundred runs. Always on CI, never on my laptop. But why only on CI? More workers, same fixed temp path.

The failure moved around too. Different test names, different assertions. That pattern usually points at shared state or timing, so I looked at threads first. Wrong turn.

The log line that mattered was logged at debug level. Nobody reads debug logs on a green build.

DEBUG ingest.fixtures fixture unreadable, using defaults path=/tmp/fixture-shared.json

That line is not a warning. That line is the bug introducing itself.

Step 1: Make the silent path loud

Before hunting anything, count your silent fallbacks. This one command changed the whole investigation for me.

rg -n "except Exception" --type py src/ scripts/ | wc -l

If a parse failure returns a default, you do not have a bug. You have a guess. Failures must raise.

# before: a bug that hides itself
def load_fixture(path: str) -> dict:
    try:
        with open(path) as fh:
            return json.load(fh)
    except Exception:
        log.debug("fixture unreadable, using defaults")
        return {}

# after: loud, typed, and greppable
class FixtureError(RuntimeError):
    """Raised when a required fixture cannot be read."""

def load_fixture(path: str, *, required: bool = True) -> dict:
    try:
        with open(path) as fh:
            return json.load(fh)
    except (OSError, json.JSONDecodeError) as exc:
        if required:
            raise FixtureError(f"{path}: {exc!r}") from exc
        return {}

The failure rate went from 1 in 200 to 200 in 200. That is progress.

Step 2: Capture the artifact at failure time

Guessing from console text is slow. I wanted the exact bytes that broke it. So the harness keeps one directory per attempt.

# flake_hunt.py
import os, subprocess, sys

def run_once(i: int, outdir: str) -> str:
    env = dict(os.environ, PYTHONHASHSEED=str(i))
    attempt = os.path.join(outdir, f"attempt-{i:04d}")
    os.makedirs(attempt, exist_ok=True)
    log = os.path.join(attempt, "run.log")
    with open(log, "wb") as fh:
        proc = subprocess.run(
            [sys.executable, "-m", "pytest", "tests/test_ingest.py",
             "-x", "-q", f"--seed={i}"],
            stdout=fh, stderr=subprocess.STDOUT, env=env, timeout=120,
        )
    return attempt if proc.returncode != 0 else ""

if __name__ == "__main__":
    outdir = sys.argv[1]
    failures = [d for i in range(int(sys.argv[2])) if (d := run_once(i, outdir))]
    print(f"{len(failures)} failing runs kept under {outdir}")

Then let it grind in the background.

mkdir -p .artifacts/flake
python flake_hunt.py .artifacts/flake 200

Note the --seed flag. If your runner ignores it, read PYTHONHASHSEED inside the test instead. Same idea, fewer knobs.

Step 3: Reduce to the smallest input

Two hundred logs are too many to read. So sort them by size and take the cheap ones first. What is the cheapest artifact to read? The smallest one.

find .artifacts/flake -name run.log -printf "%s %p\n" | sort -n | head -5

The smallest failing log named the file directly: /tmp/fixture-shared.json. Fixed name, every worker, every run. The cause was plain after that.

Worker A opened that path for writing. Worker B opened it for reading. B parsed a partial document, and the silent handler turned that into empty defaults.

# before: one path shared by parallel workers
tmp_path = "/tmp/fixture-shared.json"

# after: one directory per worker, deleted on exit
from tempfile import TemporaryDirectory
with TemporaryDirectory(prefix="fixture-", dir=artifact_dir) as td:
    tmp_path = os.path.join(td, "fixture.json")

Parallel-safe names are not a style preference. They are a correctness requirement.

Step 4: Prove the fix with the same loop

A fix without a reproduction is just a hope. So I reran the identical 200 seeds.

python flake_hunt.py .artifacts/flake-after 200

Zero failing runs. The suite also got about one second slower per run. That is the honest price of isolation, and I paid it gladly.

Step 5: Where a free model tier actually helped

Reading 200 logs by hand is dull work. So I used AI assistance for triage only, never for the fix. I ran the long repetition loop on a free server option and pointed free model access at the failure logs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow that worked, in order:

1. Concatenate the five smallest failing logs into one file.
2. Ask for ranked hypotheses, each with the evidence it needs.
3. Test the cheapest hypothesis first, by hand.
4. Throw away any hypothesis you cannot test in five minutes.

The model's first hypothesis was a data race in the worker pool. The logs said no. Its third suggestion, "check for shared temp paths", was correct and specific.

So the free server mattered for one reason. A 200-run loop is a background job, not a laptop job. Token allowances and server availability are operator-supplied claims in this article. Quotas change, so check the project page instead of trusting a number in a blog post. That includes any number I could quote today.

A short decision table

Symptom	Cheapest next action	Stop when
Fails 1 in N runs, varies by test	Count silent except handlers	The count is zero
Fails only under parallelism	Log the resolved temp path	Paths differ per worker
Fails after a refactor	Diff the fallback defaults	Defaults are identical
Fails only on CI	Compare seeds and env vars	Seeds are pinned

Limitations and who should skip this

This approach costs wall-clock time and disk space. A 200-run loop is fine for a fast suite and painful for a slow one. Pick your run count from suite duration, not from enthusiasm.

Do not bother if your suite is already deterministic. Do not paste security-sensitive logs into any hosted tool, free or not. And do not ship an AI-suggested patch you cannot explain, because the next failure will be yours alone.

If your suite is flaky and you want a cheap place to run the loop, the free tier is a reasonable starting point. The project is open source, so its limits are readable before you commit.

The lesson is smaller and sharper than "flakiness is hard". A silent fallback converts a loud crash into a quiet lie. Fix the fallback first, and the bug will find you on the first run.