top says both threads are at 100% CPU. Your profiler says the hot function is a simple counter increment. Everything looks busy and nothing looks wrong — and the program is still half as fast as it should be. The reason isn't in your code; it's in where your data lives relative to the cache. Three effects do most of this damage: cache misses, TLB misses, and false sharing. None of them show up as a function name in a flame graph.
The headline number for the worst of them, measured on one machine with CoreTracer: two threads doing the same increment loop finish in 841 ms or 8,118 ms depending only on how two integers are laid out in a struct.
Cache miss — the stall you can't see
The CPU is fast; memory is not. An L1 hit is a few cycles; a miss that goes to L2, L3, and finally DRAM costs on the order of hundreds of cycles. During that time the core stalls — it retires no instructions, yet top still counts it as 100% busy, because "busy" to the OS means "not idle," not "doing useful work."
This is why a profiler can mislead. It tells you which function ran hot; it rarely tells you the function was hot because every iteration missed cache and sat waiting on DRAM. A hash lookup where each probe pulls a cold bucket, a linked-list walk with no spatial locality, a lookup table bigger than L2 — all of these read as "CPU-bound" while actually being memory-latency-bound. The fix is never "optimize the function"; it's "change the access pattern so the data is there when you reach for it."
TLB miss — the tax on every address
Before the CPU can even fetch your data, it has to translate the virtual address to a physical one, and it caches those translations in the TLB. Miss the TLB and the hardware walks the page tables — several dependent memory accesses of its own, often costing more than the cache miss that might follow.
TLB misses scale with how scattered your memory is and how many mappings are live. The place this bites hardest in production is dense multi-tenant hosts: many small processes or containers, each with its own page tables, thrashing a shared TLB. An inference server packing many tenants onto one box pays this quietly on every access, and no application-level metric attributes it. Huge pages exist precisely to shrink this tax — one TLB entry covering 2 MB instead of 4 KB — which is why they matter for large working sets.
False sharing — the one that punishes concurrency
This is the sharpest of the three, because it turns adding threads into a slowdown. Coherence hardware tracks ownership at cache-line granularity — 64 bytes on x86 — not per variable. So two threads writing two different variables that happen to live in the same line will fight over that line as if they shared it.
Here is the setup from CoreTracer, reduced to the part that matters:
#define CACHE_LINE_SIZE 64
// a and b land in the same 64-byte cache line
typedef struct { volatile int a; volatile int b; } shared_false_t;
// padding pushes b onto the next line
typedef struct {
volatile int a;
char padding[CACHE_LINE_SIZE - sizeof(int)];
volatile int b;
} padded_t;
Two threads, pinned to separate physical cores, each hammering its own field:
false_sharing_thread1: bind_thread_to_core(0); for (i…) { s->a++; __sync_synchronize(); }
false_sharing_thread2: bind_thread_to_core(1); for (i…) { s->b++; __sync_synchronize(); }
Thread 1 only ever touches a, thread 2 only ever touches b. Logically independent. But with shared_false_t, a and b are in the same line, so every write by core 0 invalidates core 1's copy of the whole line and vice versa. The line ping-pongs across the interconnect on every iteration. Swap in padded_t and the two fields sit on different lines — the contention disappears and nothing else changes.
The numbers from that run, same work throughout:
| Layout | Wall time | vs. padded |
|---|---|---|
| Padded (a and b on separate lines) | 841 ms | 1× |
| False sharing (a and b adjacent) | 4,284 ms | 5.1× |
| True ping-pong (both threads hammering one shared int) | 8,118 ms | 9.7× |
Same instructions, same iteration count, both cores at 100% in every run. The only variable is layout: a struct-field order in the first two rows, and in the third, two threads deliberately fighting over a single variable — the pure-contention ceiling. Between "padded" and "ping-pong" there is nearly a 10× difference that no CPU-utilization graph will ever explain.
How to actually see it
top and application profilers can't distinguish useful cycles from stall cycles. perf can:
perf stat -e cycles,instructions,cache-misses,L1-dcache-load-misses,dTLB-load-misses ./bench
perf c2c record ./bench # then: perf c2c report — points at the exact false-shared line
Watch IPC (instructions per cycle) collapse and cache-misses climb between the packed and padded runs, and perf c2c will name the cache line two cores are fighting over. That's the difference between "CPU is high" and "high doing what."
Why this matters if you write services, not benchmarks
These three are the mechanism behind "I added cores and it got slower" and "the profile looks flat but latency is bad":
- Cache misses on shared lookup structures — routing tables, feature stores, inference KV caches — where each access pulls a cold line.
- TLB misses on multi-tenant hosts with high page-table pressure; the denser the packing, the worse.
- False sharing in exactly the place teams add it by accident: per-thread counters, metrics, and sharded state packed tightly into one struct "to be cache-friendly," which does the opposite.
The unifying lesson is that memory layout is performance, not an optimization pass you do later. A field reorder can be a 2× win; 64 bytes of padding can be the difference between concurrency that scales and concurrency that ships as a serial bottleneck. For AI infrastructure the stakes compound: inference servers run multi-tenant on cores you don't choose, and all three effects get worse exactly when the box is busiest. The benchmarks are in CoreTracer — clone it, run perf, and watch the numbers move.
Related
- Video: Cache Miss, TLB Miss & False Sharing
- GitHub: CoreTracer — the false-sharing / cache / TLB benchmarks above, plus lock-free and memory-ordering experiments.
- Companion: Store→Load Reordering — x86 vs ARM64, the other place cache coherence and ordering surface.
Top comments (2)
VTune's microarchitecture exploration analysis does surface exactly this — L1/L2 miss rates, TLB miss cycles, and it'll flag false sharing directly (blames it on the offending cacheline/address). Same story with perf record -e cache-misses,dTLB-load-misses plus c2c for false sharing on Linux.
Fair point, and c2c and VTune's microarch exploration are the right tools for it, worth having in the thread. The gap the piece is about is the default reflex: a flat time profile shows the hot line and calls it a memory stall, without separating an L1 miss from a TLB miss from a coherence miss. The counters you listed do separate them, but usually only once you already suspect it and know to reach for them, which tends to be after the flat profile already sent you the wrong way. So it's less that the tools can't name it, more that the profile most people run by default won't, and naming it needs exactly the counters you mentioned.