False Sharing, Measured: Two Threads, Zero Shared Variables, Still Slow

Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist,
CMU 15-213 (Computer Systems: A Programmer's Perspective), various CppCon talks,
and LLMs for restructuring and sanity-checking. Everything here is my own code,
compiled and run — not just transcribed.

---

Recap

Part 6 ended on a genuinely strange claim: two threads can share zero
variables in their own program logic — no mutex, no atomic touching the
same address, nothing that looks like synchronization in the code at all —
and still slow each other down, purely because the coherency protocol
invalidates an entire 64-byte cache line whenever any part of it is
written. This article makes that claim concrete, with real code from this
series.

---

The setup: separate variables, same benchmark

Back in Part 1, simpleMultiThreadStoragePerThread() gave each thread its
own int in a std::vector<int> cnts(4, 0) to avoid the shared-cnt race
entirely:

void Thread::simpleMultiThreadStoragePerThread() {
    t.start();
    int sum = 0;
    std::vector<int> cnts(4, 0);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i])));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt;
    }
    t.stop("Multi Thread Storage Per Thread", true);
    std::cout << "cnt: " << sum << "\n";
}

No two threads ever write the same int. By every rule covered in Parts 1
and 2, this is correct and race-free. What Part 1 didn't check is where
those four ints actually live in memory. A std::vector<int> packs its
elements contiguously — cnts[0], cnts[1], cnts[2], cnts[3] sit right
next to each other, 4 bytes apart. A 64-byte cache line holds 16 ints.
All four of cnts' elements — every thread's private counter — almost
certainly land in the same single cache line.

By Part 6's rule, that line is now a hot contention point: every write by
Thread 0 to cnts[0] invalidates the whole line for every core holding it
— including the cores running Threads 1, 2, and 3, which never touch
cnts[0] at all.

---

The simple fix already sitting in this series' code: alignas(64)

struct AlignedCount {
    alignas(64) int cnt = 0;
};

void Thread::simpleMultiThreadStoragePerThreadWithAligned() {
    t.start();
    int sum = 0;
    std::vector<AlignedCount> cnts(4);
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.push_back(std::thread(&Thread::complexFunction, this, std::ref(cnts[i].cnt)));
    }

    for (auto& t : threads) {
        if (t.joinable()) {
            t.join();
        }
    }

    for (auto& cnt : cnts) {
        sum += cnt.cnt;
    }
    t.stop("Multi Thread Storage Per Thread With Aligned", true);
    std::cout << "cnt: " << sum << "\n";
}

alignas(64) forces the compiler to place each AlignedCount on its own
64-byte boundary — meaning cnts[0].cnt, cnts[1].cnt, cnts[2].cnt, and
cnts[3].cnt each end up in a different cache line, with no other
thread's data anywhere nearby. Nothing about the algorithm changed. Nothing
about correctness changed — this was already race-free. The only thing
that changed is memory layout, and by Part 6's cost ladder, that's the
entire difference between "every write is the expensive dirty-write-while-
another-core-holds-it case" and "every write after the first is free,
purely local."

---

The controlled experiment: prac::FalseSharing

The per-thread-storage benchmark above mixes false sharing in with 100
million iterations of sin(cos(x)) math, which makes the effect hard to
isolate. prac::FalseSharing strips everything else away and measures
only the cache-line effect:

struct SharedLine {
    alignas(8) std::atomic<uint64_t> a{ 0 };
    alignas(8) std::atomic<uint64_t> b{ 0 };
};

struct PaddedLine {
    alignas(64) std::atomic<uint64_t> a{ 0 };
    char padding[64 - sizeof(std::atomic<uint64_t>)];
    alignas(64) std::atomic<uint64_t> b{ 0 };
    char padding2[64 - sizeof(std::atomic<uint64_t>)];
};

class FalseSharing {
    template <typename T>
    double runTest(const char* label, Timer t) {
        T data;
        auto worker1 = [&data]() {
            for (uint64_t i = 0; i < kIters; ++i) {
                data.a.fetch_add(1, std::memory_order_relaxed);
            }
        };
        std::function<void()> worker2 = [&data]() {
            for (uint64_t i = 0; i < kIters; ++i) {
                data.b.fetch_add(1, std::memory_order_relaxed);
            }
        };

        t.start();
        std::thread t1(worker1);
        std::thread t2(worker2);
        t1.join();
        t2.join();
        double count = t.stop("", false);
        double m_incs = (2.0 * kIters) / (count * 1e6);

        std::cout << label << " - Time: " << count << " seconds, Throughput: " << m_incs << " M increments/sec\n";

        if (data.a.load(std::memory_order_relaxed) != kIters || data.b.load(std::memory_order_relaxed) != kIters) {
            std::cerr << "Error: Incorrect final values!\n";
        }
        return count;
    }

public:
    void run(Timer t) {
        double sec1 = runTest<SharedLine>("SharedLine", t);
        double sec2 = runTest<PaddedLine>("Padded", t);
        std::cout << "Speedup: " << sec1 / sec2 << "x\n";
    }
};

Two threads. Each does kIters relaxed fetch_adds — on its own private
atomic. Thread 1 only ever touches data.a. Thread 2 only ever touches
data.b. By construction, there is no data race here at all, in either
struct — the only thing that changes between SharedLine and PaddedLine
is memory layout:

• SharedLine — a and b are 8 bytes apart, alignas(8). Both almost certainly land in the same 64-byte cache line. Every fetch_add on a is a lock-prefixed instruction (Part 6) that has to win exclusive ownership of that line — and since Thread 2 is constantly writing b in the same line, ownership is bouncing back and forth on essentially every operation. This is the expensive "write while the other core holds it dirty" case from Part 6's cost ladder, over and over.
• PaddedLine — a and b are each forced onto their own 64-byte line via alignas(64) plus explicit padding to fill out the rest of the line. After the very first access, each thread holds its own line Modified permanently. Every subsequent fetch_add is a local hit — the cheapest case on Part 6's ladder, zero interconnect traffic.

---

Results — run back to back, same machine

[Multi Thread Storage Per Thread]: 1.17731 seconds
cnt: 266666664

[Multi Thread Storage Per Thread With Aligned]: 1.24702 seconds
cnt: 266666664

SharedLine - Time: 4.58649 seconds, Throughput: 87.2127 M increments/sec
Padded - Time: 1.80964 seconds, Throughput: 221.038 M increments/sec

Version	Time	Correct?
simpleMultiThreadStoragePerThread()	1.17731s	✅
simpleMultiThreadStoragePerThreadWithAligned()	1.24702s	✅

Version	Time	Throughput	Notes
SharedLine	4.58649s	87.2 M incs/sec	a/b share one cache line
PaddedLine	1.80964s	221.0 M incs/sec	a/b on separate cache lines

prac::FalseSharing shows the effect exactly as predicted — padding gives
a ~2.5x speedup, almost identical to the throughput ratio (221 / 87.2 ≈
2.5x). This is the clean, isolated proof that Part 6's mechanism is real
and it's expensive.

But look at the other two numbers. Padding simpleMultiThreadStoragePerThread
made it slightly slower — 1.247s versus 1.177s. Same claimed fix, same
underlying cache-line issue, and it did nothing. Worse than nothing. That's
not a contradiction — it's the most useful result in this entire article,
and it's worth understanding exactly why before reaching for alignas(64)
on everything.

---

Why the exact same fix helped in one case and did nothing in the other

The difference between the two benchmarks isn't the cache-line layout —
both genuinely have the false-sharing setup Part 6 describes. The
difference is how much work sits between one write and the next.

prac::FalseSharing's loop body is almost nothing: one fetch_add, loop
back, do it again. The cost of the cache-line ping-pong — tens of cycles
for a BusRdX round trip (Part 6) — is comparable to or larger than the
cost of the actual work being protected. There's nothing to hide the
contention behind, so it dominates the total time. That's exactly why the
throughput ratio (2.5x) lines up so cleanly with the time ratio: almost
every cycle in SharedLine's run is either doing the increment or waiting
on ownership of the line.

complexFunction's loop body is sin(cos(angle)) — real floating-point
transcendental math, on the order of dozens to hundreds of cycles per
iteration, every single time through the loop. Against that cost, the
occasional cache-line invalidation is comparatively small — it's still
happening, but it's a much smaller fraction of a much bigger number. The
expensive math is already keeping each core busy long enough that the
ownership ping-pong has less opportunity to actually stall anything; by the
time a core wants to write its counter again, there's a reasonable chance
the line has settled back to it, or the wait overlaps with work that would
have happened anyway.

flowchart LR
    A["Cheap loop body<br/>(fetch_add only)"] --> A2["Cache-line contention ≈<br/>or > cost of the work itself"]
    A2 --> A3["Contention dominates —<br/>2.5x slowdown, clearly visible"]

    B["Expensive loop body<br/>(sin/cos math)"] --> B2["Cache-line contention <<<br/>cost of the work itself"]
    B2 --> B3["Contention is noise —<br/>no measurable effect"]

    style A3 fill:#FAECE7,stroke:#993C1D,color:#4A1B0C
    style B3 fill:#E1F5EE,stroke:#0F6E56,color:#04342C

And the ~6% regression from padding AlignedCount isn't random noise to
wave away either — it has a real explanation. alignas(64) isn't free:
each AlignedCount now occupies a full 64-byte line instead of packing 16
to a line, which means more total memory touched, more lines pulled into
cache, and worse memory density overall. When the false-sharing cost you're
paying without padding is already smaller than the math dominating the
loop, padding adds real overhead to fix a problem that wasn't actually
costing you anything measurable — a pure loss.

The rule this leaves you with: false sharing's cost isn't fixed — it's
relative to how much work happens between writes. Cheap, frequent writes to
a contended line: the effect can dominate everything, like the 2.5x here.
Expensive work per write: the effect can vanish into the noise, or even
reverse once padding's own memory cost is added. There's no substitute for
benchmarking your actual access pattern before deciding alignas(64) is
worth paying for.

---

Why this is the strangest bug in the series so far

Every bug this series has covered until now had a correctness signal you
could point at: Part 1's race gave a wrong number. Part 2's compiler hazard
gave a hang or a skipped wait. This one gives neither. SharedLine and
PaddedLine both compute the exact right answer, every time, verified by
the if check in runTest itself. Nothing is wrong with the program's
output. The only symptom is time — the same correct answer, taking longer,
for a reason that appears nowhere in the C++ source.

This is exactly why Part 6 spent as long as it did on MESI and bus
transactions before this article ever showed a line of benchmark code:
without that vocabulary, "two threads with separate variables run slower
together than apart" looks like nonsense. With it, it's a direct, mechanical
consequence of one fact — cache lines are the real unit of ownership, not
your variables — playing out exactly as predicted.

---

Takeaways

• Two threads can be completely race-free by every rule in Parts 1–2 and still contend, if their separate variables happen to share a cache line.
• std::vector<int>'s contiguous layout is exactly the shape that causes this — Part 1's "fix" (per-thread storage) solved the correctness problem and unknowingly reintroduced a performance problem from a different layer.
• alignas(64), with enough padding to actually fill the line, is the fix: force each hot variable onto its own cache line so writes never have to fight another core for ownership.
• This bug produces no wrong answers, no crashes, no hangs — the only symptom is unexplained slowness, which makes it far easier to miss than anything else covered so far in this series.
• The effect requires genuine multi-core contention to observe — it won't show up on a single-core machine or if the OS happens to schedule both threads on the same core.