From Mutex to Lock-Free: How a Few Cache-Line Decisions Made a Go Pipeline 4x Faster

A hands-on tour of six performance-engineering techniques: cache-line padding, atomic semaphores, sharding, SPSC ring buffers, work stealing, and mmap zero-copy I/O. Explained for engineers who want to understand how lock-free systems actually work in modern Go.

---

The Setup

Every team I have worked with eventually builds a pipeline like this one: events flow in, workers process them, and results get persisted. It usually looks like:

• One shared queue guarded by a mutex.
• A pool of workers popping from that same queue.
• A mechanism to bound in-flight work so a traffic spike does not exhaust memory.
• File-based persistence.

That design is correct, simple, and usually the right first version. Once it shows up in a CPU profile, someone has to make it fast. This post serves as the playbook for that day.

We will build the same pipeline twice in Go:

1. The naive version: mutex queue, channel-based semaphore, and buffered file I/O. This is what most teams ship first.
2. The optimal version: N shards, each with a lock-free single-producer/single-consumer ring buffer, a Chase-Lev work-stealing deque, a cache-line-padded atomic semaphore, and an mmap zero-copy store.

Here are the results upfront, measured on an 8-core machine processing 4,000,000 events:

Pipeline	Duration	Throughput	Speedup
Naive: mutex queue + channel sem + buffered I/O	4.87 s	~821K events/sec	1×
Optimal: sharded lock-free + mmap	1.21 s	~3.3M events/sec	4.04×

A standalone micro-demo demonstrates a 3.46× performance difference from a single struct layout change:

unpadded (both counters on one cache line): 543.5ms
padded   (each counter on its own line):    157.1ms

All concepts here transfer directly to C++, Rust, Java, or any other language running performance-critical workloads.

---

Part 0: A 90-Second Go Primer

You do not need extensive Go experience to read this code. Five core constructs cover the mechanics:

go doSomething()      // Spawn a lightweight thread ("goroutine")
sync.Mutex            // Mutual-exclusion lock
var x atomic.Int64    // Standard atomic integer type 
chan struct{}         // Typed channel used as a token bucket semaphore
defer f()     // Schedule f() to execute when the enclosing function returns

Go's runtime multiplexes goroutines onto OS threads, making goroutine creation cheap. Everything else here is standard systems engineering.

The domain model uses fixed-size structures:

type Event struct {
    ID        uint64
    Timestamp int64
    ShardKey  uint64     // Route key for shard placement
    Payload   [32]byte   // Fixed-size work payload
}

type ResultRecord struct {
    ID        uint64
    Timestamp int64
    Checksum  uint64     // Output from simulated work
    Worker    int32
    _         int32      // Explicit padding to keep struct aligned to 8 bytes
}

---

Part 1: The Naive Pipeline and Its Bottlenecks

The naive engine relies on standard library primitives:

type NaiveQueue struct {       // Workers share a single queue
    mu    sync.Mutex           // Guarded by a mutual exclusion lock
    items []*Event
}

A single producer goroutine claims a permit from a channel semaphore, increments a pending counter, and enqueues the event:

type ChanSemaphore struct { ch chan struct{} }

func (s *ChanSemaphore) TryAcquire() bool {
    select {
    case s.ch <- struct{}{}:
        return true
    default:
        return false
    }
}

func (s *ChanSemaphore) Release() { <-s.ch }

If the token bucket is full, TryAcquire fails and the producer yields CPU time via runtime.Gosched().

Worker goroutines pop events, compute checksums, and write records to disk using encoding/binary through a bufio.Writer guarded by a sync.Mutex.

While correct and maintainable, this architecture encounters three distinct bottlenecks at high concurrency:

1. Lock contention: every enqueue, dequeue, and output write contends for locks.
2. Cache-line invalidation: shared queue pointers and mutex state continually invalidate L1 cache across cores.
3. Syscall overhead: writing records through reflection and mutex-guarded buffered I/O triggers frequent OS context shifts.

---

Part 2: False Sharing and Cache Line Geometry

Modern CPUs do not load memory single bytes at a time. They fetch memory in 64-byte cache lines. When a CPU core updates a byte, its cache coherency protocol (such as MESI) invalidates that entire 64-byte line across all other core caches.

Coherency operates on cache lines, not individual fields.

type unpaddedCounters struct {
    a atomic.Int64   // Bytes 0 to 7
    b atomic.Int64   // Bytes 8 to 15 (same cache line)
}

If goroutine 1 continuously increments a while goroutine 2 increments b, both cores fight for ownership of the exact same 64-byte line. Every write on core 1 invalidates the line on core 2. This is false sharing: invalidation traffic caused by independent variables occupying shared cache lines.

We eliminate false sharing by padding struct fields to 64-byte boundaries:

const cacheLineSize = 64

type paddedCounters struct {
    a atomic.Int64
    _ [cacheLineSize - 8]byte   // 56 bytes of padding
    b atomic.Int64
    _ [cacheLineSize - 8]byte   // 56 bytes of padding
}

Running two goroutines across 20,000,000 iterations per counter yields:

=== False sharing: two goroutines each hammering their own counter ===
unpadded (both counters on one cache line): 543.538652ms
padded   (each counter on its own line):    157.098564ms
padding speedup: 3.46x

Adding 112 bytes of padding yields a 3.46× speedup without altering algorithmic logic.

---

Part 3: Sharding

Contention scales with the number of concurrent writers to shared state. Eliminating shared state removes contention structurally.

Rather than operating a single queue, we split the engine into N independent shards:

Each shard owns:

• A Single-Producer Single-Consumer (SPSC) ring buffer for event ingress.
• A Chase-Lev work-stealing deque for dynamic load balancing.
• A cache-line-padded atomic semaphore for flow control.

The demux goroutine hashes incoming events using MurmurHash3 to assign shards:

shard := int(hashKey(ev.ShardKey) % uint64(numShards))

Hashing by ShardKey ensures all events for a given key route to the same shard, preserving per-key processing order.

---

Part 4: The SPSC Ring Buffer

A Single-Producer Single-Consumer (SPSC) queue has exact ownership boundaries: one goroutine writes to tail and one goroutine reads from head.

Because ownership is non-overlapping, SPSC queues require zero compare-and-swap (CAS) loops. The producer only updates tail, while the consumer only updates head. In Go, we implement this cleanly using generics ([T any]) and standard atomic types (atomic.Uint64, atomic.Pointer[T]):

type SPSCRingBuffer[T any] struct {
    buf  []atomic.Pointer[T]
    mask uint64

    _    [cacheLineSize]byte
    head atomic.Uint64 // Consumer-owned cursor
    _    [cacheLineSize - 8]byte
    tail atomic.Uint64 // Producer-owned cursor
    _    [cacheLineSize - 8]byte
}

func NewSPSCRingBuffer[T any](capacityPow2 uint64) *SPSCRingBuffer[T] {
    if capacityPow2 == 0 || capacityPow2&(capacityPow2-1) != 0 {
        panic("capacity must be a power of two")
    }
    return &SPSCRingBuffer[T]{
        buf:  make([]atomic.Pointer[T], capacityPow2),
        mask: capacityPow2 - 1,
    }
}

func (r *SPSCRingBuffer[T]) Enqueue(item *T) bool {
    tail := r.tail.Load()
    head := r.head.Load()
    if tail-head >= uint64(len(r.buf)) {
        return false // Buffer full
    }
    idx := tail & r.mask
    r.buf[idx].Store(item)
    r.tail.Store(tail + 1) // Publish item
    return true
}

func (r *SPSCRingBuffer[T]) Dequeue() (*T, bool) {
    head := r.head.Load()
    tail := r.tail.Load()
    if head >= tail {
        return nil, false // Buffer empty
    }
    idx := head & r.mask
    item := r.buf[idx].Load()
    r.head.Store(head + 1) // Consume item
    return item, true
}

Design considerations:

1. Power-of-two bitmask indexing (tail & r.mask): replaces expensive integer division instructions with a single bitwise AND.
2. Explicit cache-line padding: prevents consumer updates to head from invalidating producer access to tail.
3. Generic atomic.Pointer[T] storage: provides compile-time type safety and avoids interface allocations or runtime type assertions.

---

Part 5: Work Stealing via Chase-Lev Deque

Sharding prevents lock contention, but static hashing can cause load imbalances if certain shard keys require more processing. Work stealing addresses this by allowing idle workers to steal work from busy workers.

We use a Chase-Lev double-ended queue (deque):

• The owner pushes and pops work from the bottom (LIFO order), maximizing L1 cache locality for recent tasks.
• Thieves steal work from the top (FIFO order), taking the oldest tasks.

We write the Chase-Lev deque generically using atomic.Int64 and atomic.Pointer[T]:

type WSDeque[T any] struct {
    _      [cacheLineSize]byte
    top    atomic.Int64
    _      [cacheLineSize - 8]byte
    bottom atomic.Int64
    _      [cacheLineSize - 8]byte
    mask   int64
    buf    []atomic.Pointer[T]
}

func (d *WSDeque[T]) PushBottom(item *T) bool {
    b := d.bottom.Load()
    t := d.top.Load()
    if b-t >= int64(len(d.buf)) {
        return false // Queue full
    }
    d.buf[b&d.mask].Store(item)
    d.bottom.Store(b + 1)
    return true
}

func (d *WSDeque[T]) PopBottom() (*T, bool) {
    b := d.bottom.Load() - 1
    d.bottom.Store(b)
    t := d.top.Load()

    if t > b {
        d.bottom.Store(b + 1)
        return nil, false // Queue empty
    }

    item := d.buf[b&d.mask].Load()

    if t == b {
        // Single item remaining: race against prospective thieves
        ok := d.top.CompareAndSwap(t, t+1)
        d.bottom.Store(b + 1)
        if !ok {
            return nil, false // Thief won the race
        }
        return item, true
    }

    return item, true // Uncontended pop
}

func (d *WSDeque[T]) Steal() (*T, bool) {
    t := d.top.Load()
    b := d.bottom.Load()
    if t >= b {
        return nil, false // Queue empty
    }
    item := d.buf[t&d.mask].Load()
    if !d.top.CompareAndSwap(t, t+1) {
        return nil, false // Lost race to owner or another thief
    }
    return item, true
}

The worker priority loop follows a set hierarchy:

1. Drain up to drainBatch (64) events from its local SPSC ring into its Chase-Lev deque.
2. Pop locally from its Chase-Lev deque (LIFO).
3. Attempt to steal work from a randomly selected victim shard's deque (FIFO).
4. Yield or sleep briefly if no work is available globally.

Randomized victim selection distributes thief CAS operations across shards, avoiding centralized steal contention. In our benchmark, 242,218 steals occurred across 4M events (~6% rebalanced work).

---

Part 6: Lock-Free Atomic Semaphores

Channel operations involve Go runtime scheduler locks and sleeping goroutine management (sudog). For microsecond-scale flow control, we replace channel semaphores with a non-blocking padded atomic counter:

type PaddedSemaphore struct {
    count atomic.Int64
    _     [cacheLineSize - 8]byte
    max   int64
    _     [cacheLineSize - 8]byte
}

func NewPaddedSemaphore(max int64) *PaddedSemaphore {
    return &PaddedSemaphore{max: max}
}

func (s *PaddedSemaphore) TryAcquire() bool {
    for {
        c := s.count.Load()
        if c >= s.max {
            return false // Permit limit reached
        }
        if s.count.CompareAndSwap(c, c+1) {
            return true
        }
    }
}

func (s *PaddedSemaphore) Release() {
    s.count.Add(-1)
}

Advantages:

1. Zero runtime scheduler involvement: executed strictly in user space via hardware atomic instructions.
2. Non-blocking semantics: TryAcquire allows the producer to yield CPU (runtime.Gosched()) rather than parking the thread.
3. Cache-line isolation: padding isolates each semaphore counter onto its own cache line.

---

Part 7: Persistence using mmap and unsafe.Slice

The naive implementation calls binary serialization and buffered I/O under a mutex. The optimal pipeline maps file storage directly into process virtual memory using syscall.Mmap and wraps it in a typed slice with unsafe.Slice:

// Open file, size it, and map into address space
data, _ := syscall.Mmap(int(file.Fd()), 0, size, syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)

// Zero-copy reinterpret of memory bytes as a ResultRecord slice
records := unsafe.Slice((*ResultRecord)(unsafe.Pointer(&data[0])), capacity)

Writing a record requires only an atomic slot reservation and a memory store:

func (m *MMapStore) Write(rec ResultRecord) bool {
    idx := m.cursor.Add(1) - 1
    if idx >= int64(len(m.records)) {
        return false
    }
    m.records[idx] = rec // Direct memory write; kernel handles disk sync asynchronously
    return true
}

Because ResultRecord has fixed size and explicit byte alignment, writing directly to mmap memory avoids serialization and syscalls on the execution hot path.

---

Part 8: End-to-End Performance Benchmark

Running both pipelines on an 8-core x86-64 Linux system over 4,000,000 events:

=== False sharing: two goroutines each hammering their own counter ===
unpadded (both counters on one cache line): 543.538652ms
padded   (each counter on its own line):    157.098564ms
padding speedup: 3.46x

generating 4000000 synthetic events across up to 997 shard keys, 8 shards, GOMAXPROCS=8

=== Naive pipeline: mutex queue + channel semaphore + buffered file I/O ===
processed=4000000/4000000 duration=4.870354709s throughput=821295 events/sec

=== Optimal pipeline: sharded SPSC rings + work-stealing deques + padded atomic semaphores + mmap ===
processed=4000000/4000000 duration=1.205254565s throughput=3318801 events/sec steals=242218

end-to-end speedup: 4.04x

The 4.04× speedup results from combining these techniques into a single system design. Eliminating queue mutexes, false sharing, scheduler context switches, and I/O serialization allows all available CPU cores to execute continuously without stall states.

---

Engineering Takeaways

1. Cache lines are the unit of hardware sharing: isolate hot atomic counters to separate 64-byte boundaries to eliminate false sharing.
2. Topologically eliminate contention: SPSC rings require no CAS loops because pointer ownership is single-threaded per direction.
3. Combine sharding with work stealing: sharding isolates hot execution paths, while work stealing handles dynamic load imbalance across shards.
4. Avoid runtime scheduler overhead in microsecond loops: atomic CAS counters provide non-blocking flow control without parking goroutines.
5. Leverage zero-copy memory mapping: memory-mapped storage combined with structured byte alignments eliminates serialization overhead on output paths.

---

Running the Code

The complete benchmark source code and reproduction steps are hosted on GitHub.

For representative numbers, run the benchmark on a dedicated multi-core system. Single-core environments force thread serialization and mask cache-line invalidation costs.

deepkpat / vortex

Vortex — A Sharded, Lock-Free Event Pipeline in Go, Built Twice

A self-contained, single-file (main.go) demonstration of how a "naive" event-processing pipeline — mutex queue + channel semaphore + buffered file I/O — becomes a 4× faster "optimal" pipeline using classic performance-engineering techniques. No external dependencies; the Go standard library only.

Run it

go run vertex/main.go

Run on a machine with ≥ 4 cores for representative numbers. On a single-core sandbox the goroutines time-share one core and every effect is understated.

What it demonstrates

#	Technique	Naive	Optimal
1	Cache-line padding	counters sharing one line (~3.5× slower)	each hot field on its own 64-byte line
2	Semaphore	buffered channel (ChanSemaphore)	padded atomic CAS loop (PaddedSemaphore)
3	Sharding	one shared queue	NumCPU independent shards, hash-routed
4	Queue	mutex-guarded slice (NaiveQueue)	lock-free SPSC ring buffer (SPSCRingBuffer)
5	Load balancing	none	Chase-Lev work-stealing

…

View on GitHub