48-Hour Field Notes: Catching a Flake That Only Fails on Someone Else's Machine

Every team has one test that passes on your laptop and fails somewhere else, and after the third "works for me" it stops being a test and becomes folklore. I gave myself 48 hours to make one of those flakes boring, and the only thing I refused to accept was a fix I could not reproduce on demand. What follows is what I actually tried, what wasted my first afternoon, and the harness I now keep in a tools/ folder next to every project that has ever shrugged at me.

If you have ever re-run a CI job three times hoping for green, you already know why this matters more than another round of guessing.

The rules I set before touching any code

Debugging sessions rot when you start editing source before the failure is repeatable, so I wrote constraints first and treated them as non-negotiable.

1. No source edits until the same failure appears twice under a recorded command.
2. Every run writes a machine-readable artifact: exit code, elapsed time, tail of stderr, environment overlay.
3. Every hypothesis gets written down before I test it, because untested theories that live only in my head survive far too long.
4. Anything I cannot explain after 48 hours gets a written summary and a follow-up ticket, not a heroic third night.

That fourth rule is the one people skip, and it is the one that keeps a 48-hour investigation from quietly becoming two weeks of nothing.

Hours 0–8: freeze the run and make the failure boring

The first phase is pure capture. Copy the failing command verbatim from the CI log, run it locally, and snapshot the environment next to the result rather than reconstructing it later from memory.

# snapshot the environment next to the failure, while it is still true
mkdir -p artifacts
{ uname -a; python -V; pip freeze; locale; ulimit -a; } > "artifacts/env-$(date +%s).txt"

By hour six I had one important negative result: the failure did not follow the code, it followed the conditions. A single-threaded local run with my default locale never failed, which told me almost nothing. What it did tell me was that I needed a matrix instead of a debugger.

So stop asking "why does it fail?" for a moment and ask instead: fail relative to what? That reframing is half the work.

Hours 8–20: write hypotheses down, then try to kill them

Once I had a condition matrix sketched, I used MonkeyCode's free model access to generate candidate causes from the captured artifacts instead of from my memory of the code. I fed it the trimmed stderr, the locale output, the dependency list, and the exact command, then asked for ranked hypotheses plus the one observation that would falsify each.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project is open source, and the free tier — free model access plus a free server option, with a free token allowance the project describes in the millions — is why this step cost me nothing but wall-clock time. Check the project's own page for current numbers, because I did not independently verify quotas, limits, or how long any of it stays free.

Two habits made that step useful instead of noisy:

• I pasted artifacts, never credentials, tokens, or customer data.
• I treated every suggestion as an untrusted guess and required a kill test before touching source.

Of roughly a dozen hypotheses, most died cheaply, which is exactly what I wanted. Cheap deaths are the point.

Hours 20–36: move the repro to a clean room

My laptop is a terrible witness. It has stale virtualenvs, a warm kernel page cache, and a .env file I forgot about in 2024. So I moved the matrix to a fresh machine using MonkeyCode's free server option, cloned the repo, installed from the lockfile, and ran the harness there with nothing else running.

That single change eliminated an entire class of false leads. On the clean box, the failure appeared under a narrower set of conditions than it ever had locally, and the difference pointed straight at environment drift rather than at the logic I had been staring at for two days.

Here is the harness shape. It is a template, not a benchmark: swap the matrix for the dimensions your own failure seems to track.

# tools/repro_matrix.py — run one test under a grid of conditions, log every outcome.
import itertools, json, os, pathlib, subprocess, sys, time

ART = pathlib.Path("artifacts"); ART.mkdir(exist_ok=True)
LOG = (ART / "matrix.jsonl").open("a", encoding="utf-8")

MATRIX = {
    "LC_ALL": ["C", "en_US.UTF-8"],
    "TZ": ["UTC", "America/New_York"],
    "workers": ["1", "4"],
    "nofile": ["1024", "65535"],
}

def run_case(cmd, overlay, case_id):
    env = {**os.environ, **overlay}
    start = time.monotonic()
    proc = subprocess.run(cmd, env=env, capture_output=True, text=True, timeout=900)
    rec = {
        "case": case_id,
        "overlay": overlay,
        "returncode": proc.returncode,
        "seconds": round(time.monotonic() - start, 2),
        "stderr_tail": proc.stderr[-2000:],
    }
    LOG.write(json.dumps(rec) + "\n"); LOG.flush()
    return rec

def main(cmd):
    keys = list(MATRIX)
    for n, combo in enumerate(itertools.product(*(MATRIX[k] for k in keys)), 1):
        overlay = dict(zip(keys, combo))
        rec = run_case(cmd, overlay, f"case-{n}")
        print(f"[{n}] {overlay} -> {rec['returncode']}", flush=True)

if __name__ == "__main__":
    main(sys.argv[1:] or ["pytest", "-q", "tests/test_the_flaky_one.py"])

One hard rule kept me honest: a case counts as a reproduction only if it fails twice with the same overlay recorded in matrix.jsonl. A single red row is a rumor, not a finding.

Hours 36–48: what broke, and what I would repeat

What broke, in order of how much time it cost me:

• I let the matrix grow to six dimensions before running anything, which turned a 10-minute experiment into a two-hour wait.
• I copied stderr tails that had been truncated by the CI provider, so two hypotheses were built on evidence that did not exist.
• I forgot that the clean-room box had a different file-descriptor limit, which produced a second, unrelated failure I chased for an hour.

What I would repeat without hesitation:

• Writing hypotheses and their kill tests before running anything.
• Keeping raw artifacts as JSONL rather than reading screenshots of logs.
• Separating "where I generate ideas" from "where I run the test," because the two have different failure modes.

When this workflow earns its 48 hours (and when it does not)

Situation	Local loop	Clean-room run	Free model access
Flake never reproduces locally	Weak	Required	Useful for ranking causes
Failure tracks environment or locale	Weak	Strong	Useful for drafting the matrix
Failure tracks timing and load	Weak	Strong	Marginal
Plain logic bug with a stack trace	Strong	Unnecessary	Unnecessary
Data corruption or security incident	Not suitable	Not suitable	Never send the artifacts

Limitations and who should not use this

This approach costs wall-clock time and pays you back only in clarity, so it is a poor fit if you need a hotfix in the next 30 minutes. Do not paste production logs, credentials, or personal data into a hosted model, free or otherwise. Do not build a permanent pipeline on a free tier without checking its current terms, because quotas and limits change and I have not benchmarked any of them.

If your flake is really a one-line typo, none of this is worth it — read the traceback first.

If you want to try the split

The pattern is simple: generate hypotheses somewhere separate from where you execute the test, then let the evidence decide. If you want to try it without provisioning anything, MonkeyCode is open source and its free model access plus free server option is how I ran the second half of these notes; read the current terms before you depend on them.