False Sharing: Why Your Lock-Free Counter Is Slower Than a Mutex

A team I was pairing with had "fixed" a hot path by replacing a mutex-protected counter with four separate atomic counters — one per worker thread — summed up on read. No lock contention, right? Throughput dropped by roughly 6x under load, worse than the mutex version it replaced.

Nothing in the code was wrong. The four atomic counters were correctly implemented, lock-free, and uncontended at the logical level. The CPU didn't see it that way.

The problem

Nothing in the code was wrong, and yet the "lock-free" rewrite was dramatically slower than the lock it replaced. That contradiction is the whole point of this post.

Why it happens

Modern CPUs don't move memory in single bytes or even single words — they move it in fixed-size cache lines, almost always 64 bytes on current x86 and ARM hardware. When four 8-byte counters sit next to each other in an array or a struct, all four fit inside one or two cache lines.

Here's the part that bites people: cache coherence protocols (MESI and its variants) track ownership of a cache line, not of the individual variable inside it. When core 0 writes to counter[0], it has to take exclusive ownership of that cache line — even though cores 1-3 only care about counter[1], counter[2], and counter[3]. The moment core 1 writes to counter[1], it has to invalidate core 0's copy and pull the line into its own cache. Then core 2 does the same. Then core 0 writes again and has to re-fetch.

The four threads never touch each other's data. They are still fighting over the same 64 bytes of silicon, every single write, because the coherence protocol can't see "variable" — it only sees "cache line." This is false sharing: contention introduced by physical memory layout, invisible at the source-code level, and it gets worse the more cores you throw at the problem, which is exactly backwards from what you want out of a "fix."

The slowdown is not subtle. Depending on core topology and how many sockets are involved, published benchmarks (and the ones I've personally run) put false-sharing penalties anywhere from 4x to 20x+ versus the padded version — because every write now costs a cross-core cache-line transfer (tens to low hundreds of nanoseconds) instead of a local L1 hit (~1 ns).

What to do about it

The fix is almost insultingly simple once you know what you're looking for: pad or align the hot fields so each one owns its own cache line.

// Before: all four counters likely share 1-2 cache lines
struct Counters {
    long count[4];
};

// After: each counter gets its own 64-byte line
struct PaddedCounter {
    long count;
    char pad[64 - sizeof(long)];
};
struct PaddedCounter counters[4];

Most languages give you a primitive for this instead of hand-rolled padding:

• Java: @jdk.internal.vm.annotation.Contended (or the older sun.misc.Contended)
• C++: alignas(64) on the struct or field
• Go: the runtime pads several internal sync primitives for this exact reason; for your own hot structs, add explicit _ [N]byte padding fields
• Rust: #[repr(align(64))]

The second, less obvious trap: this isn't limited to arrays of counters. It happens any time a frequently-written field sits in the same struct as a frequently-read-by-other-threads field. A lastUpdated timestamp written on every request, sitting right next to a config pointer that every thread reads — same bug, different shape. Before reaching for sharding or lock-free structures as a performance fix, check whether you've actually solved contention or just relocated it one layer down, into the cache-coherence fabric.

Key takeaways

• Cache coherence tracks ownership at the cache-line level (typically 64 bytes), not the variable level — "no shared variables" doesn't mean "no contention."
• False sharing turns logically-independent, lock-free writes into real cross-core traffic, and it scales worse with more cores, not better.
• The fix is padding/alignment to give hot fields their own cache line — a one-line change once you know to look for it.
• Profile with hardware counters (perf c2c on Linux is purpose-built for this) before assuming a concurrency bug is a logic bug. Sometimes the bug is in the memory layout, not the algorithm.

Comments

[Odinaka Robert Nnamani]

I have hit this same trap in reverse: I assumed lock-free meant faster, and I blamed the scheduler before I looked at layout. Your lastUpdated-next-to-config example is the one I will carry into my own reviews, because that pattern hides in ordinary request code, not just in counters. When you profiled with perf c2c, did the hot line show up immediately, or did you have to isolate the counters first?