False Sharing: Why Adding Threads Made Your Code Slower

The problem

A team I worked with had a per-thread request counter: one long per worker thread, stored in a flat array, no locks, no shared state by design. Four threads, four counters, each thread only ever touches its own slot.

They scaled from 4 threads to 16 to handle more load. Throughput didn't scale — it got worse. Per-increment latency went from about 1.2ns on 4 threads to over 18ns on 16. No new locks had been added. No logic had changed. On paper, there was zero sharing between threads. And yet the more cores they threw at it, the slower each one got.

Why it happens

The bug isn't in your code's logic — it's in the CPU's cache coherence protocol, and it's invisible if you only reason about your program at the level of variables.

x86 and ARM caches move data in fixed 64-byte chunks called cache lines. That's the actual unit of coherence traffic between cores — not your long, not your int, not your struct field. When counters[16] is a plain array of 8-byte longs, eight of them fit on a single 64-byte line. So even though thread 3 and thread 7 never touch each other's counters in your source code, their counters can physically live on the same cache line.

Every time thread 3 writes its counter, the MESI coherence protocol invalidates that entire line in every other core's cache — including thread 7's copy, even though thread 7's data on that line hasn't changed at all. Thread 7 then has to fetch the line again on its next access, at full cross-core latency instead of an L1 hit. Both threads end up fighting over line ownership (a Request-For-Ownership round trip) purely because of memory layout, not because their data is actually related. This is false sharing: logically independent data, physically sharing a coherence unit.

It's nasty specifically because a normal CPU profiler will not show it to you. A flame graph will point at the increment instruction and say "this line is hot" — which is true, but misleading. The instruction itself is one cycle. The 15+ extra nanoseconds are coherence traffic that doesn't show up as "your code," it shows up as ordinary-looking time spent on an add.

What to do about it

The fix is to make sure independent hot data can't share a line: pad or align each thread's slot to 64 bytes.

struct padded_counter {
    long value;
    char pad[64 - sizeof(long)];
} __attribute__((aligned(64)));

struct padded_counter counters[16];

Now each counter owns its own cache line. No thread's write can ever invalidate another thread's copy, because they're not on the same line anymore. In that same 16-thread test, padding took per-increment latency from ~18ns back down to ~1.3ns — roughly back to single-thread speed, just distributed across cores.

The cost is memory: you're deliberately wasting up to 56 bytes per counter to buy back coherence. For a handful of hot counters or ring-buffer indices, that trade is trivial. For a large array of per-item state, you'd only pad the specific hot fields, not the whole structure.

To actually find this in the wild, don't start from a flame graph. On Linux, perf c2c (cache-to-cache) is built for exactly this — it shows you which cache lines are bouncing between cores and which offsets within them are contended. If you see unexpectedly high cross-core cache misses on data your logic says is private, check the layout before you check the algorithm.

Key takeaways

• Cache lines (64 bytes on most x86/ARM chips), not variables, are the real unit of contention — "independent" data can still collide if it's packed together in memory.
• False sharing gets worse as you add threads, which makes it look like a scaling problem when it's actually a layout problem.
• Standard CPU profilers/flame graphs won't show you this — the hot line looks innocent because the instruction really is cheap; the cost is coherence traffic.
• perf c2c (or your platform's equivalent cache-line profiler) is the right tool, not a sampling profiler.
• The fix — padding/aligning to 64 bytes — is cheap in memory and can recover an order of magnitude in throughput on hot per-thread state: counters, ring-buffer indices, spinlock flags, sharded stats.