Part 1 — `std::thread` Fundamentals: A Race Condition, Explained Down to the Assembly

Notes & References

Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist, CMU 15-213 (Computer Systems: A Programmer's Perspective), and various CppCon talks.

Everything here is my own code, compiled and run — not just transcribed. These are my personal notes and references for future programming. LLMs were utilized for restructuring and sanity-checking the explanations.

---

The setup

One expensive, CPU-bound function. Every version in this article calls it —
only how many threads and how they share state changes.

void Thread::complexFunction(int& cnt) {
    const double PI_HALF = 1.5707963267948966;
    for (int i = 0; i < 100000000; ++i) {
        double angle = (double(i) / 100000000) * PI_HALF;
        cnt += (int)(std::sin(angle) * 2);
    }
}

Run it 4 times, back to back, on one thread:

void Thread::singleThread() {
    t.start();
    int cnt = 0;
    for (int i = 0; i < 4; i++) {
        complexFunction(cnt);
    }
    t.stop("Single Thread", true);
    std::cout << "cnt: " << cnt << "\n";
}

[Single Thread]: 3.04755 seconds
cnt: 266666664

That number, 266666664, is our source of truth for the rest of this
article. Every "correct" version below has to reproduce it.

---

The problem: the obvious multithreaded version is wrong

The natural next move: instead of calling complexFunction 4 times on one
thread, launch 4 threads and let them all run it at the same time, all
writing into the same cnt.

void Thread::simpleMultiThreadRaceCondition() {
    t.start();
    int cnt = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnt)));
    }
    for (auto& th : threads) {
        if (th.joinable()) {
            th.join();
        }
    }
    t.stop("Multi Thread Race Condition", true);
    std::cout << "cnt: " << cnt << "\n";
}

(Quick note on the mechanics here, since this is the first threaded code in
the series: std::ref(cnt) is required because std::thread copies its
arguments by default, and a plain int& parameter wouldn't survive that —
std::ref forces the real reference through. .join() in the loop after is
what blocks main until every worker is actually finished; skipping it and
letting threads go out of scope early is an instant crash, not a subtle bug.)

Run it:

[Multi Thread Race Condition]: 1.18949 seconds
cnt: 66666666

Faster, sure — that's expected, 4 cores instead of 1. But 66666666 is not
266666664. It's not close. It's suspiciously close to exactly
one quarter of the right answer — as if 3 out of every 4 threads'
work just vanished. The code compiled fine, ran fine, crashed nothing. It
just handed back the wrong number, silently.

---

Why: cnt += x is not one instruction

Here's the thing to unlearn: cnt += (int)(std::sin(angle) * 2); looks
like a single step in your source code. It is not one step for the CPU. At
-O0, it compiles to three separate instructions:

mov  eax, [cnt]      ; READ:   load cnt's current value into a register
add  eax, edx        ; MODIFY: add the new value to it
mov  [cnt], eax       ; WRITE:  store the result back into cnt

Read the value. Modify it. Write it back. Three distinct steps, three
distinct moments in time — and on a 4-core machine, all four threads are
doing this to the same address, at the same time, with no rule saying
one thread's three steps have to finish before another thread's three steps
begin.

Think of it like four people sharing one notebook with a running total
written in it, and no rule about taking turns:

1. Person A reads the notebook: "it says 100."
2. Before A writes anything back, Person B also reads it: "it also says 100."
3. A computes 100 + 3 = 103, and writes 103 into the notebook.
4. B, still working off the 100 it read a moment ago, computes 100 + 5 = 105, and writes 105 into the notebook — overwriting A's update completely.

Nobody gets an error. Nothing crashes. The notebook just silently ends up
wrong, and there's no record that A's update ever happened at all. This is
called a lost update, and it's what's known as a data race: two
threads touching the same memory, at least one of them writing, with no
coordination between them at all.

Do that millions of times a second, across four threads, for 100 million
iterations each, and you get a final number that isn't "slightly off" — it's
wrong in a way that happens to look patterned (close to 1/4 of the true
answer) purely because of how evenly the four threads' work happened to
collide on this run. Run it again and you'd likely get a different wrong
number. That's the part that should feel uncomfortable: the wrongness
itself isn't even consistent.

---

Solution attempt 1: lock the notebook — a mutex

If the problem is four threads reading and writing the same memory with no
coordination, the direct fix is to add coordination: force only one thread
at a time to be allowed inside the read-modify-write, no matter how many
threads are running.

void Thread::complexFunctionWithMutex(int& cnt) {
    const double PI_HALF = 1.5707963267948966;
    for (int i = 0; i < 100000000; ++i) {
            std::lock_guard<std::mutex> lock(mtx);
            double angle = (double(i) / 100000000) * PI_HALF;
            cnt += (int)(std::sin(angle) * 2);
    }
}

void Thread::simpleMultiThreadMutex() {
    t.start();
    int cnt = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunctionWithMutex, this, std::ref(cnt)));
    }
    for (auto& th : threads) {
        if (th.joinable()) {
            th.join();
        }
    }
    t.stop("Multi Thread Mutex", true);
    std::cout << "cnt: " << cnt << "\n";
}

std::lock_guard<std::mutex> lock(mtx); is the notebook's lock: whichever
thread grabs it first gets to finish its entire read-modify-write
uninterrupted, and every other thread has to wait its turn. No two threads
can be mid-update at the same time anymore — the exact gap that caused the
lost update is closed.

[Multi Thread Mutex]: 13.2063 seconds
cnt: 266666664

Correct — matches the single-threaded baseline exactly, every run. But look
at the time: 13.2 seconds, against a single-threaded baseline of 3.05
seconds. Using 4 cores made this over 4x slower than using one core and
no threading at all.

The reason is that we locked the mutex inside the loop, on every single
iteration — 400 million lock/unlock operations total, most of them spent
with three threads waiting idle for the fourth to finish its turn. We didn't
just add coordination — we added coordination so fine-grained that the four
threads are barely running in parallel at all; they're mostly taking turns,
plus paying the overhead of acquiring a lock 400 million times on top of that.

Correctness, fixed. Performance, worse than not threading in the first
place. That's a real trade-off, and it's hit constantly:
a mutex fixes the wrong answer problem, but a careless mutex can make your
"parallel" code slower than serial code.

---

Solution attempt 2: don't share the notebook at all

Here's the question worth sitting with: did these four threads ever actually
need to share cnt while they were working? Each thread's math is
completely independent of the other three — thread 2 never needs to know
what thread 1 computed mid-loop. The only reason we had a race at all is
that we chose to point all four threads at the same variable.

So — don't:

void Thread::simpleMultiThreadStoragePerThread() {
    t.start();
    int sum = 0;
    std::vector<int> cnts(4, 0);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i])));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt;
    }
    t.stop("Multi Thread Storage Per Thread", true);
    std::cout << "cnt: " << sum << "\n";
}

Give each thread its own slot (cnts[i]) instead of a shared one, let all
four run completely uninterrupted with no locking at all, and only add the
four totals together after every thread has joined — at which point no
thread is writing anymore, so there's nothing left to race.

[Multi Thread Storage Per Thread]: 1.12478 seconds
cnt: 266666664

Correct, and the fastest version by a clear margin — faster than the mutex
version by more than 10x, and even slightly faster than the naive (wrong)
race-condition version, because there's no locking overhead at all.

---

The full picture

Version	Time	cnt	Correct?
Single Thread	3.05s	266666664	✅ baseline
Race Condition (shared cnt, no lock)	1.19s	66666666	❌ lost updates
Mutex (shared cnt, locked every iteration)	13.2s	266666664	✅ but 4x slower than serial
Storage Per Thread (no sharing at all)	1.12s	266666664	✅ fastest overall

The lesson isn't "always use a mutex" or "always avoid shared state" — it's
that the fastest fix for a race is often to make the race impossible
rather than to police it. A mutex is the right tool when threads
genuinely need to coordinate over shared, mutable state. Here, they never
did — the sharing was the bug, not something that needed defending.

---

Takeaways

• x += y is read-modify-write at the hardware level, not one atomic step — that gap between "one line of C++" and "three real instructions" is where every data race in this series starts.
• A data race doesn't announce itself. It can produce a plausible-looking wrong number, and a different wrong number on the next run.
• A mutex restores correctness by serializing access — but locking too finely (every iteration, in this case) can make multithreaded code slower than not threading at all.
• If threads don't actually need to share mutable state, removing the sharing beats synchronizing it — no lock, no contention, no overhead.