Testing Race Conditions: Making Nondeterminism Reproducible

A passing concurrency test usually proves that one schedule was safe, not that the program was safe. If the failure depends on two operations landing in a narrow order, adding more loop iterations often repeats the same harmless order. The useful change is to control the schedule, record it, or explore it systematically.

Why a Passing Stress Test Proves Almost Nothing

The usual stress test starts several workers, runs the operation many times, and checks the final result. That catches bugs when the operating system happens to choose the needed interleaving. It does not ask the scheduler to choose that interleaving, and it does not preserve the choice when a run fails.

A green stress test therefore has a narrow meaning: the observed executions did not violate the assertion. The Go race detector documentation states the same limit for its dynamic detector: it can find races that happen at runtime, but cannot find a race in code that the test never executes. Code coverage and schedule coverage are different measurements.

A failing stress test is useful when it preserves its seed and inputs. A green stress test that consumes minutes has not shown that the schedule space was examined.

The Failure Mode: Interleavings, Not Code Paths

The first diagnostic step is to name the failure precisely. A data race is an unsynchronized concurrent access to the same memory location where at least one access writes. A race condition is broader: the program's result is wrong because events occur in an order the design did not permit. A program can avoid a data race with a mutex and still have an ordering bug inside the locked operations.

Four categories show up often in test reports:

• A data race has conflicting memory accesses without a valid synchronization edge.
• An atomicity violation splits a check and an update that should behave as one operation.
• An ordering assumption expects event A before event B, but no barrier, channel, or condition establishes it.
• A lost wakeup lets a notification occur before a waiter records that it is waiting.

The detector must match the category. A sanitizer is suited to unsynchronized memory accesses. A barrier can force both sides of a check-then-act window. Events and conditions can make a required order explicit. A small model checker can explore alternative transitions. No single green result covers all four categories.

This distinction also prevents a common bad fix: adding a sleep. A sleep changes timing without specifying the order. It may make a failure disappear on one machine and return on another. A synchronization primitive states what the test requires and gives the test a point at which to assert it.

Instrument the Scheduler Instead of the Clock

Replace guessed delays with a protocol. In the example below, request A writes one tenant, request B waits until that write happened and overwrites the shared client, and A then reads the tenant. The events force the bad order without depending on how long either thread sleeps.

import threading
from concurrent.futures import ThreadPoolExecutor


class Client:
    def __init__(self):
        self._tenant = None

    def set_tenant(self, tenant):
        self._tenant = tenant

    def get_tenant(self):
        return self._tenant


def test_request_state_is_not_shared():
    client = Client()
    first_write = threading.Event()
    second_write = threading.Event()

    def request_a():
        client.set_tenant("tenant-a")
        first_write.set()
        second_write.wait(timeout=1)
        return client.get_tenant()

    def request_b():
        first_write.wait(timeout=1)
        client.set_tenant("tenant-b")
        second_write.set()

    with ThreadPoolExecutor(max_workers=2) as pool:
        observed_a = pool.submit(request_a)
        pool.submit(request_b).result()
        assert observed_a.result() == "tenant-a"

The assertion fails on every run because the test has specified the interleaving. The production repair would normally make tenant state local to a request or protect a complete operation, not add another delay. The test is useful because it names the invariant: request A must not observe state written for request B.

Use a barrier for phases, an event for milestones, and a condition for state predicates. Each primitive makes the scheduler part of the test fixture.

For a notification path, start the waiter first and acknowledge that it has reached the waiting point before sending the signal:

import threading


def test_waiter_observes_a_notification():
    started = threading.Event()
    notified = threading.Event()
    finished = threading.Event()
    values = []

    def waiter():
        started.set()
        if not notified.wait(timeout=1):
            raise AssertionError("notification was lost")
        values.append(42)
        finished.set()

    thread = threading.Thread(target=waiter)
    thread.start()
    assert started.wait(timeout=1)
    notified.set()
    assert finished.wait(timeout=1)
    thread.join(timeout=1)
    assert values == [42]

The test does not claim that every notification implementation is correct. It checks one contract with an explicit order and a bounded failure. A real condition-variable test should apply the same idea around the predicate and the wait, rather than relying on a sleep that merely makes the waiter likely to run first.

Deterministic Replay: Turn a Rare Failure into a Test Fixture

A replayable failure needs more than a random seed. Record the seed, the generated input, the schedule decisions, the build identity, and any injected faults that can change the path. A seed is sufficient only when every decision comes from the same deterministic scheduler and input generator.

Wall-clock reads, operating-system randomness, network responses, and uncontrolled background tasks can break replay. If they matter to the bug, replace them with recorded inputs or a virtual interface. Otherwise, a test that says it is replaying a failure is only rerunning a similar experiment.

The manifest should be written before the test process exits, even when the assertion fails. A small helper can make the contract visible:

import json
import random
from pathlib import Path


class Schedule:
    def __init__(self, seed):
        self.seed = seed
        self.rng = random.Random(seed)
        self.decisions = []

    def choose(self, runnable):
        index = self.rng.randrange(len(runnable))
        choice = runnable[index]
        self.decisions.append(choice)
        return choice


def save_replay(path, schedule, input_data, build_id):
    manifest = {
        "schema_version": 1,
        "seed": schedule.seed,
        "decisions": schedule.decisions,
        "input": input_data,
        "build_id": build_id,
    }
    Path(path).write_text(json.dumps(manifest, indent=2), encoding="utf-8")

The important property is that replay consumes the record rather than silently generating new choices. Keep the manifest as a regression fixture when the defect is fixed.

Data Race Detection With Sanitizers in CI

ThreadSanitizer, or TSan, instruments the program at compile time and uses a runtime library to detect data races in executions that actually occur. The Clang documentation describes typical TSan slowdown as about 5x-15x and typical memory overhead as about 5x-10x. Those figures are tool guidance, not a promise for a particular service.

TSan does not enumerate schedules. It can miss a race in an unexecuted path, and Clang warns that code generally needs to be compiled with the sanitizer flag. Precompiled or uninstrumented libraries can cause missed reports or false positives because the detector cannot see all synchronization. A clean run is evidence about that instrumented execution, not a proof about every possible execution.

Go has a related detector behind -race. Its documentation requires cgo and, on non-Darwin systems, a C compiler, and it lists the supported platform combinations. The same page documents the GORACE option halt_on_error, which can make the first report terminate the process instead of allowing a CI job to continue.

env GORACE=halt_on_error=1 go test -race ./...

Run this stage on code paths that matter, not only on a tiny smoke test. Keep sanitizer jobs separate when needed, but make their failure visible and actionable. Do not link the sanitizer runtime into a production executable as a substitute for testing; Clang explicitly warns that its runtime is not intended for production security constraints.

Model Checking and Bounded Exhaustive Search

Dynamic tools observe schedules. Model-checking tools generate schedules. For a small state machine, systematic exploration can be cheaper than trying to make a large integration test fail by chance.

Rust Loom runs concurrent Rust tests while permuting executions under its supported portion of the C11 memory model, and it uses state reduction to limit combinatorial growth. Its own documentation also lists unsupported behaviour and warns that a passing result is not a sound proof for every C11 execution. Tests must use Loom's instrumented synchronization types so the checker can see the choices.

AWS Shuttle takes a different trade-off. It controls and randomizes scheduling, and its documentation describes deterministic reproduction of failing tests, but it is not an exhaustive checker. That makes it useful for larger test cases where full exploration is too expensive, while leaving a clear limit on what a green run means.

The boundary is the model. A semaphore accounting algorithm, connection-pool admission rule, or circuit-breaker transition may fit in a bounded model. A database server, network, external queue, or multi-service deployment does not become exhaustively checked merely because the client has a model-checking test. Keep the model small enough that its state and assumptions can be reviewed.

A Practical Workflow for a Concurrency Bug Report

Production reports usually describe symptoms, not interleavings: a response is truncated, a request receives another tenant's data, or a counter is occasionally wrong. Capture the request identifiers, input, build version, relevant logs, synchronization events, and the first invalid observation. Then reduce the report to a local invariant.

Cloudflare's account of a hyper HTTP bug is a useful example. Its Images service saw intermittent truncation for larger images while responses still returned HTTP 200 and no application error. Cloudflare reports spending six weeks tracing the issue; in one case, roughly 200 KB arrived when the response was expected to be 3.3 MB. The investigation eventually isolated an incomplete flush before connection shutdown and added a deterministic test; the article says the fix took four lines of code.

A practical sequence is:

• Identify the shared state and the operation that should protect it.
• Classify the defect as a data race, atomicity violation, ordering bug, or lost wakeup.
• Replace timing guesses with a barrier, event, condition, or scheduler hook.
• Record the input and schedule before rerunning the test.
• Verify that the unfixed version fails and the repaired version passes.

A lost-update report can become a deterministic regression test with two barriers, one after both workers read and one after both workers write:

import threading


def test_counter_update_is_atomic():
    counter = [0]
    read_barrier = threading.Barrier(2)
    write_barrier = threading.Barrier(2)
    iterations = 100

    def increment():
        for _ in range(iterations):
            value = counter[0]
            read_barrier.wait()
            counter[0] = value + 1
            write_barrier.wait()

    threads = [threading.Thread(target=increment) for _ in range(2)]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()

    assert counter[0] == 2 * iterations

The unsynchronized version fails because both workers read the same value in each round. A fixed implementation can protect the read-modify-write operation with a lock, after which this test should be adjusted so the synchronization in the fixture does not create an artificial deadlock. The key is that the regression test describes the lost update directly.

What to Keep, What to Delete, and What to Watch

An ordinary application repository should keep deterministic tests for critical shared state, replay manifests for injected schedules, and a sanitizer job when the language supports one. Delete long stress loops that only repeat a schedule; retain short stress runs when they complement, rather than replace, a forced-interleaving test.

A library repository should add model-checking tests for small concurrent algorithms. A bug fixed in a reusable queue, pool, or state machine can affect every caller, so the extra constraints and instrumented primitives are worth maintaining. Keep a deterministic fixture for every concurrency defect that reached the issue tracker.

For authorization, payment, replication, and other components where a race can corrupt trust or data, use all three layers where practical: forced schedules for known invariants, sanitizer coverage for observed memory races, and bounded exploration for small core algorithms. The watch item is always the same: a fix is merged after a green stress run, but the failing schedule was never captured. A race becomes a regression test only when its order is part of the test's data.

The goal is not to remove nondeterminism from the whole system. It is to put nondeterminism behind an interface that a test can control, record, and challenge. Once the schedule is visible, concurrency debugging becomes an engineering task instead of a request to get lucky.

---

Originally published on Dispatch.