Store Load Reordering: x86 vs ARM64, and the Bug Intel Was Hiding

Here is a bug report that keeps recurring, in different words, every time a team moves to ARM: "the same code works on our Intel CI and on the developers' older MacBooks, but it corrupts data / deadlocks / returns impossible values on Graviton." No source change, no compiler change, no new code. The bug was always there. x86 was hiding it.

The mechanism underneath is store→load reordering, and it's one of the few places where two mainstream CPU architectures genuinely disagree about what your program means. This post takes it apart with a small litmus test you can run yourself, real numbers from running it on ARM64, and the reason a memory fence turns an "impossible" outcome back into an impossible one.

The test that shouldn't be able to fail

Two threads, two shared variables, both starting at zero:

// Thread 1          // Thread 2
X = 1;               Y = 1;
r1 = Y;              r2 = X;

Reason about it in program order and r1 == 0 && r2 == 0 looks impossible. Walk the interleavings:

1. Thread 1 finishes first → X=1, Y=0 → r1=0, r2=1
2. Thread 2 finishes first → Y=1, X=0 → r1=1, r2=0
3. They interleave after at least one store lands → r1=1, r2=1

There is no ordering of these four operations in which both loads read zero. For both to read zero, each thread's load would have to run before the other thread's store — but also before its own store, which comes first in the source. Under a sequentially consistent machine, that outcome cannot happen.

Real hardware produces it anyway.

Why the CPU makes the impossible happen

The CPU is allowed to reorder each thread into this:

// Thread 1 (as executed)   // Thread 2 (as executed)
r1 = Y;   // load first      r2 = X;   // load first
X = 1;                       Y = 1;

Now the timeline that was forbidden is trivial: Thread 1 loads Y (still 0), Thread 2 loads X (still 0), then both stores land. r1 == 0 && r2 == 0.

Why would a CPU do this? A store isn't finished when the instruction retires — it goes into the store buffer and drains to cache later. A load to a different address has no visible dependency on that pending store, so the core is free to let the load pass the buffered store and keep the pipeline busy. From a single thread's point of view nothing changed; X and Y are independent, so reordering X=1 and r1=Y is invisible to that thread. The other core is where the reordering becomes observable.

That's the key idea: the reordering is legal precisely because it's invisible to the thread doing it. It only leaks through a second observer.

The litmus test, in real code

This is the harness from CoreTracer, trimmed to the essentials:

volatile int X = 0, Y = 0, r1 = -1, r2 = -1;

// Thread 1: X = 1; r1 = Y;
void *thread1_func(void *p) {
    X = 1;
    if (use_fence) memory_barrier();   // real CPU fence
    else           compiler_barrier(); // stops the compiler only
    r1 = Y;
    ...
}
// Thread 2 is the mirror image: Y = 1; ...; r2 = X;

// After both threads finish an iteration:
if (r1 == 0 && r2 == 0) reorder_detected++;

Two details matter. First, the barrier:

void memory_barrier() {
#if defined(__x86_64__)
    __asm__ volatile("mfence" ::: "memory");   // x86
#elif defined(__aarch64__)
    __asm__ volatile("dsb sy" ::: "memory");   // ARM64
#endif
}
void compiler_barrier() {
    __asm__ volatile("" ::: "memory");         // compiler fence, no CPU effect
}

Second, the volatile and the compiler barrier are deliberate. This is a litmus test, not production code — volatile isn't the right tool for real concurrency (that's what C11 atomics are for). Here it exists to stop the compiler from optimizing the accesses away or reordering them itself, so that the only reordering left to observe is the hardware's. That separation is the whole point: it lets you tell a compiler reorder apart from a CPU reorder.

What the numbers actually say

Run it on ARM64 across a million iterations and the three cases separate cleanly:

Barrier	r1==0 && r2==0 rate (ARM64)
None	~2.3%
Compiler barrier only	~1.8%
Full memory barrier (dsb sy)	0.0%

Read those three rows carefully, because they tell the whole story:

• 2.3% with nothing — the reordering is not exotic. It's happening on roughly one in forty iterations.
• 1.8% with a compiler barrier — stopping the compiler from reordering barely moves the number. That's the proof that the CPU, not the compiler, is doing this. A compiler_barrier() (or Go's //go:nosplit-style tricks, or marking things volatile) does nothing about it.
• 0.0% with a real fence — one dsb sy and the outcome is gone entirely.

On x86 the same test behaves very differently. Store→load is the only reordering TSO permits, and the store buffer drains aggressively, so on a short run you often see zero hits and have to push iterations way up to catch any at all. The test harness even says as much when it comes back empty: "try more iterations or a different CPU." The video runs both sides live and prints the counts if you want to watch the gap open up rather than take the table's word for it.

x86 TSO vs ARM64: what each one lets slide

The reason the same binary behaves differently is that the two architectures publish different rules for which reorderings are allowed:

Reordering	x86 (TSO)	ARM64 (weak)
Load → Load	❌ no	✅ yes
Load → Store	❌ no	✅ yes
Store → Store	❌ no	✅ yes
Store → Load	✅ yes	✅ yes

x86's Total Store Order is almost sequentially consistent: it forbids three of the four reorderings and allows only store→load, the one our litmus test targets. That single allowed case is why the bug can appear on Intel at all — just rarely. ARM64's model is weak: unless you insert a barrier, essentially anything can move. Code that leaned, without knowing it, on x86 forbidding load→load or store→store has no such guarantee on ARM, and those cases are far more common than the narrow store→load window.

This is why "it only broke on ARM" is the usual shape of the incident. x86 was silently upholding guarantees the source language never actually promised.

Why the fence makes it truly impossible

Put a full barrier between the store and the load on each thread and r1==0 && r2==0 goes from rare to provably impossible — the 0.0% row above.

A fence forces a serialization point: every store issued before it must be globally visible before any load after it can execute. Trace the argument. Suppose r1 == 0, i.e. Thread 1's load of Y saw zero. With the fence, Thread 1's X = 1 was already globally visible when that load ran. So the load happened before Thread 2's Y = 1 (that's the only way Y could still be 0), which means Thread 2's Y = 1 — and therefore everything before it on Thread 2, including its load r2 = X — happened after X = 1 was visible. So r2 must read 1. r1 == 0 forces r2 == 1; both-zero cannot occur. The fence removed the reordering that made it possible, and with it the contradiction.

That's what mfence / dsb sy buy you, and why they cost cycles: they're draining the store buffer and establishing a global order where the hardware would otherwise let things float.

Why this matters if you never write assembly

You are not going to hand-write mfence in a Go service. You don't need to. But the same physics reaches up into the languages you do use:

• Atomics compile differently per architecture. A Go atomic.Store / atomic.Load or a Rust Ordering::SeqCst maps to specific hardware behavior. On x86 much of it is close to free because the CPU already provides most of the ordering; on ARM64 the compiler must emit real barrier instructions, which cost cycles. Same source, different generated code, different performance and different bugs when the ordering is under-specified.
• Lock-free data structures that pass on x86 CI can break on ARM prod. Graviton, Apple Silicon, GCP's Tau T2A, NVIDIA Grace — all weakly ordered. A queue or a flag protocol that "tested fine" on Intel runners can carry a latent ordering bug that only ARM exposes, and only under the right scheduling, which is what makes it a 3am incident instead of a CI failure.
• AI infrastructure is moving onto ARM fleets. Inference and serving increasingly run on Graviton and Grace. Concurrency code written with an unstated x86 assumption ships with bugs that were invisible until the hardware stopped hiding them.

The takeaway isn't "add fences everywhere." It's that porting concurrent code from x86 to ARM is not "recompile and run." The compiler honors the source language's memory model — Go's, Rust's, C++11's — and the bugs that surface on ARM are ones those models always permitted. x86's stronger model was doing you a favor you didn't know you were relying on, and ARM is where the bill comes due. If your code is lock-free, verify it on the architecture you actually ship on.

Related

• Video: Store→Load Reordering — x86 vs ARM64 Real-World Test
• GitHub: CoreTracer — the litmus harness above, plus assembly-level out-of-order and cache experiments.

Comments

[Raknaos]

The store-buffer explanation is the clearest version of this I've seen — most write-ups jump straight to "add a fence" without showing why the hardware is allowed to produce r1==0 && r2==0 in the first place. The part I'd underline even harder: the compiler reorders too, so a source-level fence is not enough once the toolchain has already swapped things before the CPU sees them.

Curious about the numbers when you ran the litmus loop on Graviton — how rarely does the forbidden outcome actually show up in a tight loop? My assumption is that it hides in CI because the rate is so low, and the first real sighting is months later in production.