A Race You Can't Reproduce Is Still a Race: Testing Model-Generated C++ Concurrency Fixes

In my last two posts I built a small grading loop for coding models on C++ tasks: compile the model's answer, run it, and score it. That worked fine for undefined behavior, because sanitizers make UB deterministic enough to grade. Then I tried the same loop on a data race, and the grader happily passed a fix that was still broken.

Concurrency bugs don't fail on demand. A test that passes 50 times can still be wrong. This post is about the workflow I ended up with: what a coding model is actually useful for here, what it is not reliable for, and a harness you can copy that refuses to let a lucky run count as a pass.

The setup: a deliberately racy counter

Here's the task I gave the model. It's a stripped-down version of a pattern I've seen in real codebases: a shared cache of computed results with a hit counter.

// racy.cpp — the broken version
#include <thread>
#include <vector>
#include <unordered_map>
#include <iostream>

struct Cache {
    std::unordered_map<int, int> table;
    long hits = 0;   // shared, unguarded

    int lookup_or_compute(int key) {
        auto it = table.find(key);
        if (it != table.end()) {
            ++hits;            // <-- data race #1
            return it->second;
        }
        int value = key * key; // pretend this is expensive
        table.emplace(key, value); // <-- data race #2 (map is not thread-safe)
        return value;
    }
};

int main() {
    Cache cache;
    std::vector<std::thread> threads;
    for (int t = 0; t < 8; ++t) {
        threads.emplace_back([&cache, t] {
            for (int i = 0; i < 10000; ++i)
                cache.lookup_or_compute((i + t) % 64);
        });
    }
    for (auto& th : threads) th.join();
    std::cout << "hits=" << cache.hits << "\n";
}

Here's the trap: on a quiet machine this program often prints a plausible-looking number and exits 0. If your grader is "compile, run, check exit code," you will green-light broken code all day. That was my first mistake when I reused my earlier UB-grading loop for this task.

Why the compile-and-run grader fails here

For the UB tasks I graded before, AddressSanitizer and UndefinedBehaviorSanitizer convert latent bugs into immediate, deterministic failures. A race is different: it needs a specific interleaving to manifest as visible corruption, and even then the crash may show up one run in a thousand — or never, until the code ships to a machine with more cores and a different scheduler.

So the grading loop needs two changes:

1. ThreadSanitizer (TSan) instead of "does it crash." TSan detects the race on the access pattern, not on observable corruption. One instrumented run is worth a thousand uninstrumented ones.
2. Repetition with perturbation. Even with TSan, I re-run under varied thread counts and CPU counts, because some interleavings only appear under contention.

The workflow, step by step

This is the loop I now run when a model proposes a concurrency fix:

#!/usr/bin/env bash
# race_grade.sh — grade one candidate fix. Exit 0 only if truly clean.
set -euo pipefail
SRC="$1"

# 1. Baseline: does it even compile with strict warnings?
g++ -std=c++17 -Wall -Wextra -Werror -c "$SRC" -o /dev/null

# 2. TSan build. -O1 keeps optimizations on so TSan sees realistic codegen.
g++ -std=c++17 -fsanitize=thread -g -O1 "$SRC" -o /tmp/candidate_tsan

# 3. Run under TSan several times, varying scheduler pressure.
for run in 1 2 3 4 5; do
    taskset -c 0-1 /tmp/candidate_tsan > /dev/null   # squeezed onto 2 cores
    taskset -c 0-7 /tmp/candidate_tsan > /dev/null   # full width
    TSAN_OPTIONS="halt_on_error=1" /tmp/candidate_tsan > /dev/null
done

# 4. Correctness of the counter: every iteration eventually hits the cache,
#    so after warm-up the hit count must be exact. Print it and diff against
#    the expected value computed single-threaded.
g++ -std=c++17 -O2 "$SRC" -o /tmp/candidate_plain
ACTUAL=$(/tmp/candidate_plain | grep -o 'hits=[0-9]*' | cut -d= -f2)
EXPECTED=75968   # computed from a single-threaded reference run
echo "candidate hits=$ACTUAL expected=$EXPECTED"
[ "$ACTUAL" = "$EXPECTED" ]

echo "PASS"

Note step 4. A fix that silences TSan by wrapping everything in one global mutex can still be wrong in a subtler way (or just ruin the point of the exercise), so I also check the final counter against a value I computed from a single-threaded reference run. If you adapt this harness, compute your own expected value the same way — don't copy mine.

taskset is Linux-specific; on macOS you can drop it and rely on the repetition, or run the binary under load from a background process. The idea matters more than the tool: vary the scheduler's choices.

What the model actually did with this

I fed racy.cpp to a few free models through MonkeyCode, which gives free access to a selection of coding models and a free server option, so this kind of repeated "generate candidate → grade → regenerate" loop doesn't cost anything per attempt. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The interesting part is the spread of answers across attempts:

• Attempt 1 (common): make hits a std::atomic<long> and call it fixed. TSan immediately flags the unordered_map accesses. Partial fix — the model fixed the race it could see (the ++ on a plain long) and missed the structural one.
• Attempt 2: guard the whole lookup_or_compute body with a std::mutex. This passes TSan and the counter check. Correct, but it serializes everything — for a read-heavy cache that's a real regression, and a naive grader can't tell.
• Attempt 3 (best, one attempt out of several): std::shared_mutex with a read lock on the hit path, an upgrade to a write lock on miss, plus the atomic counter. Passed everything.

Two observations that generalize:

1. The model is good at naming the right primitive (shared_mutex, atomic) once you show it the TSan report. It is noticeably worse at finding all the races on its own. Pasting the sanitizer output back into the prompt was the single biggest improvement to fix quality — much bigger than rewording the problem statement.
2. The most common failure was fixing the loudest race and stopping. My grader caught this only because TSan checks the whole program, not the line the model edited. If I had graded by "does the diff look reasonable," attempt 1 would have scored full marks.

What the harness can't tell you

Be honest about what this proves:

• TSan has blind spots. It doesn't understand custom synchronization built on raw atomics with memory_order_relaxed, and it can't reason about lock-free algorithms beyond detecting the raw races. If the model proposes a lock-free fix, TSan passing means much less.
• Passing N runs is not a proof. It's evidence. The expected-value check in step 4 only works because this particular task has an exact, computable answer; most real code doesn't.
• Performance is graded by you, not the script. The mutex-everything fix passes. Whether it's acceptable is a design decision. I now keep a crude throughput timing in a separate step, but I treat it as advisory, not pass/fail.
• A free tier is fine for this loop, but size your expectations. Candidate generation, sanitizer runs, and regenerations for one task took a handful of model calls — well within what a free option covers. If you want to sweep dozens of tasks across many models nightly, plan your own infrastructure or check current limits first; I haven't stress-tested where the boundaries are.

Who should skip this approach

If your concurrency code is lock-free or built on custom memory-ordering reasoning, this harness will give you false confidence — you want model checkers (e.g., CDSChecker-style approaches) or formal review, not a model-plus-TSan loop. And if the fix touches code you don't fully understand, no grading loop substitutes for reading it; the harness tells you the patch isn't obviously broken, not that it's right.

Takeaway

For UB, sanitizers turned coding models into something I could grade. For concurrency, the same trick works only after you fix the grader first: TSan for detection, repetition under varied scheduling for coverage, and an exact correctness oracle so that "no crash" can't impersonate "correct." The model earns its keep generating and revising candidates against that harness — especially when you feed the sanitizer report back in — but the harness, not the model, is what decides.

If you want to try the loop yourself, race_grade.sh above plus any free model access (I used MonkeyCode's) is enough to reproduce everything in this post on one machine. I'd be curious what pass rates you see on your own racy examples — my sample here is one task and a handful of attempts, nothing more.