Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist,
CMU 15-213 (Computer Systems: A Programmer's Perspective), various CppCon talks,
and LLMs for restructuring and sanity-checking. Everything here is my own code,
compiled and run — not just transcribed.
Recap
Part 3 left one number sitting there deliberately: a register access takes
roughly 1 cycle; a trip to main RAM takes roughly 200 cycles. This
article picks that gap up and asks the obvious next question — if RAM is
really 200x slower than a register, how does any real program run at a
reasonable speed at all? The answer is caches, and understanding why they
work is what makes the rest of this hardware track (and eventually, false
sharing) make sense instead of feeling like memorized trivia.
The problem caches solve
A CPU core executes an instruction in roughly 1 cycle. A single access to
main RAM takes roughly 200 cycles. If every memory access actually went all
the way to RAM, the CPU would spend the overwhelming majority of its time
just waiting — no matter how fast its execution units are.
Caches aren't an optimization bolted on as an afterthought. They're
load-bearing for a modern CPU to be useful at all.
The hierarchy, level by level
| Level | Typical latency | Typical size | Scope |
|---|---|---|---|
| Registers | ~1 cycle | Tens of bytes | Per core |
| L1 cache | ~4 cycles | 32-64KB | Per core (split: instructions / data) |
| L2 cache | ~12 cycles | 256KB-1MB | Per core |
| L3 cache | ~40 cycles | A few MB to tens of MB | Shared across all cores on the chip |
| Main RAM | ~200 cycles | GBs | Shared, off-chip |
The pattern going down every step: size goes up roughly an order of
magnitude, latency goes up roughly 3-5x, and — this is the detail worth
underlining — what's visible to that level shrinks from "this core only"
(registers, L1, L2) to "every core on the chip" (L3, RAM). That shift, from
private per-core storage to shared storage, is the seed of the
entire cache-coherency problem two articles from now, and it's the reason a
mutex or an atomic operation touching L1-local data behaves completely
differently from one touching something every core has to agree on.
(Numbers vary by CPU generation and vendor — treat the table as
order-of-magnitude intuition, not a spec sheet for any specific chip.)
Why a pyramid, not one big fast memory
Why not just make all of RAM as fast as L1? Two physical reasons, not
engineering laziness:
- Speed and size trade off directly in circuit design. The circuitry that makes L1 fast — more transistors per bit, shorter wire distances to the core — is also what makes it expensive and physically large per byte stored. Multi-gigabyte capacity at L1 speed isn't a software problem to solve; it's not economically or physically feasible with current technology.
- Physical distance costs real time. Even at near-light-speed signal propagation, a memory bank farther from the core takes measurably longer to respond. L1 is built into the core. RAM sits on separate chips, connected over a bus. That distance alone adds latency, independent of the memory technology used.
So instead of one tier, you get a pyramid: a tiny amount of very fast storage
right next to the execution units, progressively larger and slower tiers
moving outward, with RAM as the big slow tier at the bottom.
How the hierarchy is actually used
When the CPU needs an address, it doesn't go straight to RAM — it checks
levels in order, fastest first:
A successful lookup at any level is a hit. Failing to find it and having
to check the next level down is a miss. Every miss that eventually
resolves — even all the way from RAM — copies the data upward into the
faster levels it passed through, on the expectation you'll want it again
soon.
Why caching works at all: locality
Caching only pays off because real programs exhibit locality —
predictable patterns in what memory they touch.
- Temporal locality — if you accessed an address recently, you're likely to access it again soon. (A loop counter, read and written every iteration.)
- Spatial locality — if you accessed an address, you're likely to soon access addresses near it. (Iterating an array — element 5 is immediately followed by element 6.)
Temporal locality is why keeping recently-used data around instead of
evicting it immediately is worth doing. Spatial locality is the direct
motivation for the next article: instead of fetching exactly the one
byte/int requested, hardware fetches a whole contiguous block around it —
because spatial locality says you'll probably want the neighbors too. That
block is called a cache line, and it's the single fact the rest of this
hardware track — and eventually false sharing — is built on.
Glossary added this part
| Term | Meaning |
|---|---|
| Cache hit | Requested data found at this level — fast path |
| Cache miss | Requested data not found here — check the next, slower level |
| Latency | Time between requesting data and having it available |
| Temporal locality | Tendency to re-access the same address soon |
| Spatial locality | Tendency to access nearby addresses soon after one |
Takeaways
- Without caches, a CPU capable of 1-cycle instructions would spend nearly all its time waiting on 200-cycle RAM accesses. Caches aren't a bonus, they're required for a modern CPU to function at a usable speed.
- The hierarchy is a pyramid, not a single fast tier, for two physical reasons: speed/size trade off in circuit design, and physical distance costs real time regardless of technology.
- Registers, L1, and L2 are private per core. L3 and RAM are shared across every core on the chip — that shift from private to shared is exactly where multi-core problems start, two articles from now.
- Caching only works because of locality — temporal (reuse soon) and spatial (neighbors soon) — and spatial locality specifically is why hardware always fetches more than you asked for.
Top comments (0)