Your Multithreaded Code Is Correct and Still 8x Slower: False Sharing Explained

The problem

Eight cores. Eight threads. Each thread owns a private counter and increments it in a tight loop, millions of times a second, with zero locks and zero shared state at the logical level. No mutexes, no atomics contention, no obvious bottleneck in the code.

Throughput should scale close to linearly. Instead you measure it and get roughly the same total throughput as one thread — sometimes worse. Nobody touched the same variable. There's no race condition. And it's still catastrophically slow.

I hit this exact case profiling a stats-collection layer: a Counter counters[NUM_THREADS] array, one slot per thread, each thread only ever writing its own index. On paper, embarrassingly parallel. In practice, adding cores made things worse.

Why it happens

The CPU doesn't move data in individual bytes or even individual variables — it moves it in cache lines, almost always 64 bytes on modern x86 and ARM. When a core writes to any byte in a cache line, the cache-coherency protocol (MESI, or a variant of it) invalidates every other core's cached copy of that entire line, not just the byte that changed.

An int counters[8] array is 32 bytes. On a 64-byte cache line, all eight counters — one per thread — can live in a single cache line, or split across two. It doesn't matter that thread 3 only ever writes counters[3] and never looks at counters[5]. As far as the hardware is concerned, every write to that line by any core forces the other cores' copies to be invalidated and re-fetched.

The result is the cache line physically ping-ponging between L1 caches across cores, even though there is zero logical data dependency between the threads. Each of those cross-core transfers costs on the order of tens to over a hundred nanoseconds — dwarfing the single-digit-nanosecond cost of the increment itself. You end up serializing on cache coherency traffic that never shows up as a lock, a mutex, or anything else you'd normally look for in a profiler's call graph.

This is false sharing: a performance bug with no corresponding correctness bug. Your code is right. Your memory layout is wrong.

What to do about it

1. Detect it before you "fix" it. Don't guess — false sharing has a specific fingerprint: HITM events (cache-line hit in "Modified" state, transferred cache-to-cache). On Linux:

perf c2c record -- ./your_binary
perf c2c report

This will point you at the exact cache line and the specific cores/threads fighting over it. Intel VTune's memory-access analysis surfaces the same signal on non-Linux setups.

2. Pad or align the hot data to cache-line boundaries.

struct alignas(64) PaddedCounter {
    std::atomic<uint64_t> value;
    char pad[64 - sizeof(std::atomic<uint64_t>)];
};

PaddedCounter counters[NUM_THREADS];

Now each counter owns its own cache line. No cross-core invalidation traffic, because no two threads' hot variables share a line anymore.

3. Prefer thread-local accumulation over shared arrays when you can. Instead of N threads writing into a shared array of N slots, give each thread a genuinely private (thread-local or stack-local) counter and aggregate once at the end. This sidesteps the layout problem entirely instead of papering over it with padding.

4. Don't pad everything reflexively. Padding trades memory and cache footprint for coherency traffic. On data that's read-mostly, or accessed by a single thread anyway, padding just wastes cache capacity and can hurt performance elsewhere. Measure with perf c2c first, pad the specific structures it flags, and stop there.

Key takeaways

• False sharing produces zero logic errors — your tests pass, your assertions hold, and your scaling is still terrible.
• The unit of cache coherency is the cache line (typically 64 bytes), not the variable — layout decisions you never think about become hardware-level contention.
• perf c2c turns "this is inexplicably slow" into "these two cores are fighting over this exact cache line," which is the difference between guessing and fixing.
• Padding and thread-local accumulation are targeted fixes for a measured problem, not a default you apply to every struct.