48-Hour Field Notes: My Race Condition Vanished on a Free Server — It Was Just Slower

The green run that almost fooled me

On Wednesday I moved a flaky test off my laptop and onto a remote box, and it passed 200 times in a row. For about an hour I believed I had accidentally fixed something, which is a dangerous thing to believe at 2 a.m. The test was a concurrent append to a small ledger file, and on my laptop it failed roughly one run in fifteen. So what did the move actually change — the code, the data, or just the shape of the machine underneath it?

Nothing in the code changed. That is the whole point of this post: the failure rate of a timing-sensitive bug is a property of the environment, not of your test name. If you only ever run the repro on one machine, you are measuring that machine's scheduler, not your program's correctness.

What I actually did in the first twelve hours

I stopped trying to "reproduce the bug faster" and started trying to reproduce the sensitivity instead. Three things happened, in this order:

1. I wrote down the exact conditions of the original failure (laptop, on battery, browser open, test file unmodified for six days).
2. I ran the same test 200 times on the remote box and got zero failures, which I initially misread as "not reproducible."
3. I built a sweep that runs the test under deliberately worse scheduling conditions, because the remote box was simply too forgiving.

That third step is the one I would repeat. The rest was mostly me arguing with my own assumptions.

The artifact: a scheduler-sensitivity sweep

Here is the harness, stripped down. It does not try to find the bug; it tries to find the profile under which the bug is willing to appear.

#!/usr/bin/env bash
# sweep.sh — run one test under several scheduling profiles
set -u
RUNS="${RUNS:-200}"
TEST_CMD="${TEST_CMD:-python -m pytest tests/test_ledger.py::test_concurrent_append -q}"

run_profile () {
  local name="$1"; shift
  local fails=0
  for _ in $(seq 1 "$RUNS"); do
    if ! "$@" >/dev/null 2>&1; then fails=$((fails + 1)); fi
  done
  printf '%-14s %s/%s failed\n' "$name" "$fails" "$RUNS"
}

run_profile "default"     bash -c "$TEST_CMD"
run_profile "one-core"    taskset -c 0 bash -c "$TEST_CMD"
run_profile "tight-gil"   env PY_SWITCH_INTERVAL=0.000001 bash -c "$TEST_CMD"
run_profile "cpu-starved" cpulimit -l 15 -- bash -c "$TEST_CMD"

The tight-gil profile needs one line of glue in conftest.py, because there is no environment variable for sys.setswitchinterval out of the box:

# conftest.py
import os
import sys

if (raw := os.getenv("PY_SWITCH_INTERVAL")):
    sys.setswitchinterval(float(raw))

Then, before trusting any of it, I check that the profile actually took effect on this machine. A profile that silently does nothing is worse than no profile at all.

# probe.py — is this box coarse or fine grained right now?
import statistics
import sys
import threading
import time

def median_gap_ms(n: int = 2000) -> float:
    hits: list[int] = []

    def worker() -> None:
        for _ in range(n):
            hits.append(time.perf_counter_ns())

    t = threading.Thread(target=worker)
    t.start()
    worker()
    t.join()
    gaps = [b - a for a, b in zip(hits, hits[1:])]
    return statistics.median(gaps) / 1e6

print(f"switchinterval={sys.getswitchinterval() * 1e3:.3f}ms "
      f"median_gap={median_gap_ms():.3f}ms")

My result table looks like this shape, and yours will have different numbers in it:

profile        failures
-----------------------
default        <n1>/200
one-core       <n2>/200
tight-gil      <n3>/200
cpu-starved    <n4>/200

The shape is the finding, not the digits. If default is clean and one constrained profile is not, your bug is scheduling-sensitive and your fast machine has been lying to you by omission.

Where free model access and a free server actually helped

The remote box here was MonkeyCode's free server option, and the model I used for a second opinion was their free model access; both are operator-supplied availability claims, and I am describing my own use rather than a spec. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two uses were genuinely useful, and a third was a trap:

• Useful: drafting additional budget profiles for the sweep, which I then reviewed line by line before running anything.
• Useful: running the 800-run sweep off my laptop, so a sleeping lid or a browser tab could not contaminate the timing I was trying to measure.
• A trap: asking a model to "fix" the race directly. It produced a plausible lock that serialized the whole ledger and made the test pass by deleting the concurrency it was supposed to verify.

That last point matters more than the first two. Free access to a model is cheap; free access to a critic that does not get to commit is the actual workflow.

What broke, and what I would not repeat

• Wall-clock timestamps in the test assertions. They passed on the slow box and failed on the fast one, for reasons unrelated to the bug.
• Treating a clean 200-run sweep as proof. Zero failures under one profile is one data point about one scheduler.
• Running the sweep in CI first. CI is shared and noisy, so I could not tell whether a failure came from the bug or a neighbor.

I also had to stop myself from quoting the sweep's timings as performance data. The free server may be shared, so latency numbers from it describe contention on that host and nothing about your code — a lesson I apparently need to relearn periodically.

A short decision table

Question you are asking	Laptop	Free remote box	Constrained local profile
Does it fail at all?	Yes, sometimes	Sometimes not	Best chance
Is it scheduling-sensitive?	Hard to say	No	Yes
How fast is my code?	Noisy	No — shared host	No
Can I run 800 iterations cheaply?	Battery dies	Usually fine	Fine

Who should not do this

Do not use this sweep if your defect is deterministic, because you will burn a day measuring a scheduler that was never involved. Do not use it if you need real throughput numbers, since a shared free host cannot give them to you. And do not run it against anything holding production credentials, because a remote box you did not provision is the wrong place for secrets.

What I would repeat on Monday

Build the sensitivity sweep before the fix, run it somewhere your laptop cannot interfere, and keep the model in the reviewer's chair instead of the committer's chair. If a green run on a new machine ever reassures you, ask the obvious question first: did the bug go away, or did the timing? If you run the sweep yourself, I would like to know which profile finally reproduced your bug — that is the setting I want to steal.