Why a Lock‑Free Ring Buffer Beats Mutex Queues in a 200 kRPS Rust Service

In a telemetry ingest service we were hitting 200 k requests per second on a single 8‑core box. The naive design used a Mutex<VecDeque<T>> shared between the async dispatcher and a pool of worker threads. Under load the mutex became a hotspot, inflating tail latency and causing occasional lock‑starvation spikes.

The Bottleneck

A Mutex serializes all access, even when threads are only reading the queue length or pushing a new element. On a busy core the lock holder repeatedly evicts other threads from the L1 cache, and the kernel’s futex wake‑up adds a few microseconds of latency per contention event. Profiling with perf showed the mutex lock function consuming ~12 % of total CPU cycles, dwarfing the actual message‑processing work.

Switching to a Lock‑Free Ring Buffer

We replaced the mutex queue with a bounded, lock‑free ring buffer based on atomic indices. The producer claims a slot by atomically incrementing a head counter; the consumer reads from a tail counter. Because the buffer is fixed‑size we can safely use Relaxed ordering for the data payload and Acquire/Release only on the indices, which eliminates cache‑line ping‑pong.

use std::sync::atomic::{AtomicUsize, Ordering};
use std::cell::UnsafeCell;

pub struct RingBuffer<T> {
    buf: Vec<UnsafeCell<Option<T>>>,
    mask: usize,
    head: AtomicUsize,
    tail: AtomicUsize,
}

impl<T> RingBuffer<T> {
    pub fn with_capacity(cap: usize) -> Self {
        let cap = cap.next_power_of_two();
        Self {
            buf: (0..cap).map(|_| UnsafeCell::new(None)).collect(),
            mask: cap - 1,
            head: AtomicUsize::new(0),
            tail: AtomicUsize::new(0),
        }
    }

    pub fn push(&self, val: T) -> Result<(), T> {
        let h = self.head.load(Ordering::Relaxed);
        let t = self.tail.load(Ordering::Acquire);
        if h.wrapping_sub(t) == self.buf.len() {
            return Err(val); // full
        }
        unsafe { *self.buf[h & self.mask].get() = Some(val) };
        self.head.store(h.wrapping_add(1), Ordering::Release);
        Ok(())
    }

    pub fn pop(&self) -> Option<T> {
        let t = self.tail.load(Ordering::Relaxed);
        let h = self.head.load(Ordering::Acquire);
        if t == h {
            return None; // empty
        }
        let val = unsafe { (*self.buf[t & self.mask].get()).take() };
        self.tail.store(t.wrapping_add(1), Ordering::Release);
        val
    }
}

The implementation stays within a single cache line for each index, so producers and consumers only contend on distinct lines. Benchmarks on the same hardware dropped the per‑request latency from 1.8 ms (95th percentile) to 0.6 ms and eliminated the periodic latency spikes.

Tradeoffs and Edge Cases

• Bounded Capacity – The ring buffer must be sized to handle burst traffic; otherwise back‑pressure logic is required to drop or block.
• Memory Safety – Using UnsafeCell sidesteps Rust’s aliasing rules; we must guarantee that a slot is never accessed concurrently, which the atomic index protocol enforces.
• False Sharing – If the buffer size is not a power of two, the mask trick fails and indices can wrap on the same cache line, re‑introducing contention.
• CPU‑Bound Workloads – When processing is compute‑heavy, the lock‑free gain diminishes because the critical path shifts to the worker thread rather than the queue.

Conclusion

Lock‑free structures shine when contention is high and the critical path is the synchronization primitive itself. In our case, eliminating the mutex reduced cache thrashing and gave a predictable latency envelope.

How have you handled high‑throughput queuing in Rust or other systems? Do you prefer lock‑free designs, bounded channels, or a hybrid approach?

Comments

[Francisco Grandón]

Great breakdown, Elias. The 12% CPU burn on futex contention under high telemetry load is all too familiar—nothing humbles a modern multicore CPU quite like threads violently knocking each other out of L1 cache over a shared Mutex.

Moving to atomic head/tail counters with Acquire/Release semantics is textbook cache-friendly systems programming. The detail about using Relaxed ordering for the bounded payload slot once the index claim is secured is the real MVP here—people often slap SeqCst everywhere out of fear and end up paying for memory barriers they never actually needed.

That said, keeping it SPSC/bounded requires discipline when consumer hiccups happen. It reminds me of the classic realization: computers became a million times faster in forty years, yet nothing cures cache-line ping-pong quite like a 1970s circular ring buffer. Turns out Dijkstra and LMAX were right all along—the fastest lock is still the one you never acquire.

Solid write-up and clean Rust snippets!