DEV Community: Parikalp Bhardwaj

What Does Arc Actually Do? Atomic Reference Counting Under the Hood

Parikalp Bhardwaj — Wed, 26 Aug 2026 19:26:05 +0000

Following Arc::clone from a line of Rust down to a single machine instruction. Article 2 of **Below the Abstraction* — a series that takes everyday Rust abstractions and follows them down to the kernel. Every number and every trace below was produced by running the programs shown, on the machine described in the appendix.*

Why `Arc` Exists

In the last article we created OS threads and watched Linux build them with clone3. One detail from that investigation matters here more than anything else. Four threads in one process had byte-identical memory maps:

$ diff <(cat /proc/$PID/task/$PID/maps) <(cat /proc/$PID/task/$TID/maps) \
    && echo "identical"
identical

Every thread sees the same address space. A pointer in one thread is a valid pointer in all of them.

Which raises the question this article is about: if several threads can reach the same heap allocation, who is allowed to free it?

Start with a value that has exactly one owner:

let message = String::from("hello from Rust");

Move it into a thread:

use std::thread;

fn main() {
    let message = String::from("hello");

    thread::spawn(move || {
        println!("{message}");
    })
    .join()
    .unwrap();
}

The worker now owns message. The main thread cannot use it any more. That is not the threading system being awkward — that is Rust's ownership model doing exactly its job. One owner, one drop, no ambiguity.

But real systems need something ownership alone does not give you:

A configuration object read by twenty worker threads.
Routing state shared across a server's request handlers.
A cache used by many handlers at once.
A database connection pool referenced from every task.

Conceptually:

                     Shared Config
                          │
           ┌──────────────┼──────────────┐
           │              │              │
           ▼              ▼              ▼
       Worker 1       Worker 2       Worker 3
           │              │              │
           ▼              ▼              ▼
       Requests        Requests       Requests

We do not want to copy the whole configuration for each worker. We want several parts of the program to share ownership of one allocation, and for that allocation to be freed exactly once, when the last of them is finished.

That is the problem Arc<T> solves.

What Is `Arc<T>`?

Arc stands for:

Atomically Reference Counted

It is a pointer that provides shared ownership of a heap-allocated value. The idea in one picture:

multiple Arc handles
        ↓
same heap allocation

For example:

use std::sync::Arc;

fn main() {
    let value = Arc::new(String::from("hello"));

    let a = Arc::clone(&value);
    let b = Arc::clone(&value);

    println!("{value}");
    println!("{a}");
    println!("{b}");
}

This does not create three Strings. It creates one, and three handles to it:

Stack

value ───────┐
a ───────────┼───────┐
b ───────────┘       │
                     ▼
                 Heap allocation
              ┌──────────────────┐
              │ strong count = 3 │
              │ weak count       │
              │                  │
              │ String           │
              │ "hello"          │
              └──────────────────┘

That diagram is the mental model. Later in this article we will read those two counts out of memory directly and check that the picture is accurate — it turns out to be right in shape and slightly wrong in one detail.

A note on vocabulary

Since this series goes low-level, let me define terms as they come up rather than assume them.

Heap allocation — a block of memory obtained at runtime (ultimately via malloc, which as we saw in article 1 sometimes becomes an mmap syscall). It lives until something explicitly frees it, unlike stack memory which disappears when the function returns. This is why sharing across threads needs the heap: a spawned thread can outlive the function that created it, so it cannot borrow that function's stack.

Handle — a small value you hold that refers to something bigger elsewhere. An Arc<T> is a handle; the T is elsewhere.

Why Not Just Use `&T`?

Before reaching for Arc, it is fair to ask whether a plain reference would do.

let value = String::from("hello");
let reference = &value;

This compiles because the compiler can prove reference cannot outlive value.

thread::spawn breaks that proof. A spawned thread is an independent OS task — article 1 showed Linux scheduling it separately, with its own TID and its own 2 MiB stack. It can still be running long after the function that spawned it has returned and its stack frame is gone. So Rust will not let you hand it a borrow of that stack.

Arc<T> sidesteps the problem by giving each participant something it genuinely owns, while the actual value sits on the heap where nobody's stack frame can take it away.

Why Not `Rc<T>`?

Rust has a second reference-counting pointer:

Rc<T>   =  Reference Counted
Arc<T>  =  Atomically Reference Counted

They do the same job. The difference is one word: atomically.

Rc updates its count with an ordinary CPU increment. That is fine inside one thread and unsafe across several. Arc updates its count with an atomic instruction, which is safe across threads and costs more.

The compiler enforces the distinction. Here is the same program written with Rc and handed to thread::spawn:

error[E0277]: `Rc<Config>` cannot be sent between threads safely
  --> threads_rc.rs:18:36
   |
18 |           handles.push(thread::spawn(move || {
   |                        ------------- ^------
   |                        |             |
   |  ______________________|_____________within this `{closure@threads_rc.rs:18:36: 18:43}`
   | |                      |
   | |                      required by a bound introduced by this call
...
   = help: within `{closure@threads_rc.rs:18:36: 18:43}`, the trait `Send` is not implemented for `Rc<Config>`
note: required because it's used within this closure
note: required by a bound in `spawn`

error: aborting due to 1 previous error

(Trimmed in the middle where it echoes the closure body; the rest is verbatim.)

What that error is really saying

The error mentions Send. It is worth stopping here, because Send and its partner Sync are how Rust decides what is allowed near a thread — and most explanations of them are one sentence long and help nobody.

Here is the plain version.

Rust needs to answer two questions about every type:

Question one: is it safe to hand this value to another thread?

That is Send. If a type is Send, you can move it from thread A to thread B and thread B may use it. Most types are fine here. A String is Send — hand it over, the receiving thread now owns it, the sending thread has given it up, nobody is confused.

Question two: is it safe for two threads to look at this value at the same time?

That is Sync. If a type is Sync, two threads may hold &T references to the same value simultaneously and nothing bad happens. A u32 sitting behind a shared reference is Sync — both threads can read it all day, because reading cannot break anything.

You never write impl Send for MyType. The compiler works it out for you: if everything inside your type is Send, your type is Send. It only becomes something you think about when it goes wrong, which is exactly what happened above.

So why is Rc not Send?

Because of the counter. Rc bumps its count with a plain, non-atomic increment. If Rc were Send, you could move one to another thread, and then two threads would be incrementing and decrementing the same ordinary integer at the same time. That is the lost-update problem from the previous section, and it ends with the allocation being freed while somebody still holds a pointer to it.

So the library says: not Send. Not "discouraged", not "be careful" — the type simply cannot cross a thread boundary, and the compiler stops you.

Compare the two:

	`Send`?	`Sync`?	meaning
`Rc<T>`	no	no	one thread only, both for moving and for sharing
`Arc<T>`	yes*	yes*	can be moved between threads and shared by them

* when T itself is Send + Sync — Arc can only be as thread-safe as the thing inside it. This is the same point as the "Arc does not make T thread-safe" section coming up.

And this is the part I find genuinely impressive. The distinction between "safe to share across threads" and "not safe" is not a convention in a style guide, or a comment, or a lint you can ignore. It is in the type system. The unsafe version does not compile. You cannot ship this bug.

Keep that error in mind, because it is going to close a loop. By the end of this article you will have seen the exact machine instruction that is the whole difference between the type that compiles and the type that does not. It is one word long.

What "Reference Counting" Actually Means

Reference counting is a very old idea and a simple one. The allocation carries a number: how many owners currently exist. Owners appearing bump it up; owners leaving bump it down; whoever takes it to zero cleans up.

let data  = Arc::new(42);        // strong = 1
let data2 = Arc::clone(&data);   // strong = 2
let data3 = Arc::clone(&data);   // strong = 3

drop(data2);                     // strong = 2
drop(data3);                     // strong = 1
drop(data);                      // strong = 0  ->  destroy the value

The model:

Arc::new(T)
    ↓
strong = 1

Arc::clone()
    ↓
strong += 1

drop(Arc)
    ↓
strong -= 1

strong reaches zero
    ↓
drop T

Nothing surprising so far. The surprising part is that += 1 and -= 1 cannot be ordinary arithmetic once more than one thread is involved.

What "Atomic" Actually Means

Say the count is 2, and two threads clone at the same moment. An ordinary increment is not one operation — the CPU has to read the value, add one, and write it back. Three steps. Two threads can interleave them:

Thread A reads 2
Thread B reads 2
Thread A writes 3
Thread B writes 3

Two new owners were created. The count says 3. It should say 4.

This is called a lost update, and for a reference count it is fatal. The count is now permanently one too low, so the allocation will be freed while a real owner still holds a pointer to it. That owner then reads freed memory — a use-after-free, the exact class of bug Rust exists to prevent.

An atomic read-modify-write (RMW) fixes it. The hardware performs read, add, and write as one indivisible step: no other core can observe or interleave with the middle of it.

CPU 0                 shared count                 CPU 1

fetch_add(1)   ───────►     2      ◄─────── fetch_add(1)

                            ↓

                            4

Arc uses atomic RMW for both directions. Rc does not. That is the whole difference — and it is a real one, not a formality.

How that atomicity is achieved differs by CPU architecture, and we will look at the actual instruction later in this article rather than speculate about it.

`Arc` Does Not Make `T` Thread-Safe

This is the most common misconception about Arc, and it is worth being blunt about.

Arc<T> gives you shared ownership. It does not give you synchronized mutation.

let counter = Arc::new(0);

You cannot write *counter += 1 from several threads. Arc hands out &T — a shared reference — and shared references do not permit arbitrary mutation. That is not Arc being restrictive; it is the only sound thing it can do, because Arc has no idea how to make your type safe to mutate concurrently.

For shared mutable state you combine Arc with something that does know:

Arc<Mutex<T>>
Arc<RwLock<T>>
Arc<AtomicU64>

The separation of responsibility is clean, and worth memorising:

Arc
│
└── Who owns this allocation, and when is it freed?

Mutex / RwLock / Atomic
│
└── How is concurrent access to the value synchronized?

Do not merge those two jobs in your head. Nearly every confused question about Arc comes from merging them.

The compiler checks this too

Remember the asterisk on the Send/Sync table earlier — Arc<T> is only thread-safe if T is. Here is what that looks like when you get it wrong.

RefCell<T> gives you mutation through a shared reference, which sounds like exactly what we want. It enforces the borrow rules at runtime with a counter instead of at compile time. But that counter is an ordinary integer, not an atomic one — the same problem as Rc. So RefCell is not Sync: two threads must not look at one at the same time.

Wrap it in an Arc and try to send it to a thread anyway:

let shared = Arc::new(RefCell::new(0));
let a = Arc::clone(&shared);
thread::spawn(move || {
    *a.borrow_mut() += 1;
});

error[E0277]: `RefCell<i32>` cannot be shared between threads safely
   |
   = help: the trait `Sync` is not implemented for `RefCell<i32>`
   = note: if you want to do aliasing and mutation between multiple threads,
           use `std::sync::RwLock` instead
   = note: required for `Arc<RefCell<i32>>` to implement `Send`

Read that last line slowly, because it is the whole idea in one sentence. Arc<RefCell<i32>> is not Send — you cannot move it to a thread — because RefCell<i32> is not Sync. The Arc did not fix anything. It faithfully passed the question through to the type inside it, and the answer came back no.

Arc is a wrapper around ownership, not a thread-safety spray. And the compiler even tells you the fix: use RwLock.

Why `Arc<Mutex<T>>` is everywhere

Arc<Mutex<HashMap<String, String>>>

Two mechanisms, stacked:

Thread 1 ── Arc ──┐
Thread 2 ── Arc ──┼──► Mutex ──► HashMap
Thread 3 ── Arc ──┘

Arc lets several independently spawned threads own the same Mutex. The Mutex controls exclusive access to the HashMap. Remove the Arc and you cannot share it; remove the Mutex and you cannot mutate it safely.

It is a common pattern and a reasonable default. It is not automatically the right architecture — article 3 benchmarks when it is and when RwLock or sharding beats it.

The Question

For this article:

What actually changes in memory when we create, clone, move, and drop an Arc<T>? And what changes when several OS threads clone and drop the same Arc at once?

Concretely, the things to find out:

Arc allocation layout
strong reference count
weak reference count
pointer identity
clone behaviour
drop behaviour
atomic increments and decrements
memory ordering
cache-line effects
cross-thread ownership

The Hypothesis

Arc::clone does not clone the inner T. It creates another owning pointer to the same allocation and atomically increments a shared strong count. The last owner to drop destroys the value and frees the memory. Being atomic, those updates cost more than ordinary ones — and cost more still when several CPU cores touch the same count.

Now let's test each clause.

Experiment 1 — Prove Shared Allocation and Reference Counting

Start with the smallest program that can settle it. No dependencies:

cargo new rust-arc-internals
cd rust-arc-internals

src/main.rs:

use std::sync::Arc;

#[derive(Debug)]
struct Data {
    value: u64,
}

impl Drop for Data {
    fn drop(&mut self) {
        println!("Dropping Data {{ value: {} }}", self.value);
    }
}

fn main() {
    let original = Arc::new(Data { value: 42 });

    println!("after Arc::new");
    print_arc_state("original", &original);

    let clone_a = Arc::clone(&original);

    println!("\nafter clone_a");
    print_arc_state("original", &original);
    print_arc_state("clone_a", &clone_a);

    let clone_b = Arc::clone(&original);

    println!("\nafter clone_b");
    print_arc_state("original", &original);
    print_arc_state("clone_a", &clone_a);
    print_arc_state("clone_b", &clone_b);

    drop(clone_a);

    println!("\nafter dropping clone_a");
    print_arc_state("original", &original);
    print_arc_state("clone_b", &clone_b);

    drop(clone_b);

    println!("\nafter dropping clone_b");
    print_arc_state("original", &original);

    println!("\nleaving main");
}

fn print_arc_state(name: &str, arc: &Arc<Data>) {
    println!(
        "{name}: ptr={:p}, strong={}, weak={}, value={}",
        Arc::as_ptr(arc),
        Arc::strong_count(arc),
        Arc::weak_count(arc),
        arc.value,
    );
}

Three details in that program are deliberate.

Arc::clone(&original) rather than original.clone(). Both work. The first makes it obvious at the call site that you are cloning the handle, not the data. In concurrent code that distinction is the difference between a pointer copy and a deep copy, and it is worth spelling out.

Arc::as_ptr returns a raw pointer to the inner value. We only print it — never dereference it. If three handles print the same address, they refer to one allocation, and the "three Data values" theory is dead.

A custom Drop. This is the strongest evidence available. Reference counts are just numbers we are being told; a destructor running is an event we can observe. If Dropping Data prints exactly once, at exactly the right moment, the model is confirmed by behaviour rather than by report.

Predict before you run

Worth answering for yourself first:

Will all three Arcs print the same pointer?
What is strong_count after each clone?
Does cloning invoke Data's own clone? (It cannot — Data does not implement Clone.)
When does Data::drop run?
How many times does it run?
What does weak_count show when we never created a Weak?
Does drop(clone_a) free the allocation?

The output

after Arc::new
original: ptr=0x563fea01ed70, strong=1, weak=0, value=42

after clone_a
original: ptr=0x563fea01ed70, strong=2, weak=0, value=42
clone_a: ptr=0x563fea01ed70, strong=2, weak=0, value=42

after clone_b
original: ptr=0x563fea01ed70, strong=3, weak=0, value=42
clone_a: ptr=0x563fea01ed70, strong=3, weak=0, value=42
clone_b: ptr=0x563fea01ed70, strong=3, weak=0, value=42

after dropping clone_a
original: ptr=0x563fea01ed70, strong=2, weak=0, value=42
clone_b: ptr=0x563fea01ed70, strong=2, weak=0, value=42

after dropping clone_b
original: ptr=0x563fea01ed70, strong=1, weak=0, value=42

leaving main
Dropping Data { value: 42 }

Every clause of the hypothesis, confirmed:

One address, 0x563fea01ed70, printed by all three handles at every stage. One allocation, three pointers.
The count tracks handles exactly: 1, 2, 3, then 2, then 1.
Dropping Data prints once, after leaving main — when original, the final handle, goes out of scope at the end of main. Dropping the two clones destroyed nothing.
weak_count is 0 throughout, since we never made a Weak.

Note the ordering of the last two lines. leaving main is the last statement in the function; the destructor runs after it, as original goes out of scope. Rust drops locals at the end of the scope, in reverse declaration order. Small thing, but if you have ever wondered why a "shutting down" log line appears before the cleanup it is describing, that is why.

Experiment 2 — The Same `Arc` Across OS Threads

Single-threaded proof is nice; the whole point of Arc is threads. So:

use std::sync::Arc;
use std::thread;

struct Config { name: String }
impl Drop for Config {
    fn drop(&mut self) { println!("  Config::drop ran for {}", self.name); }
}

fn main() {
    let cfg = Arc::new(Config { name: "prod-eu-west-1".into() });
    println!("main thread: handle at {:p} on the stack, points to {:p} on the heap",
             &cfg as *const _, Arc::as_ptr(&cfg));
    println!("strong before spawning = {}", Arc::strong_count(&cfg));

    let mut handles = Vec::new();
    for i in 0..4 {
        let cfg = Arc::clone(&cfg);          // one increment, once per thread
        handles.push(thread::spawn(move || {
            println!("  worker {i}: tid={:?}  handle at {:p}  heap {:p}  strong(snapshot)={}",
                     thread::current().id(),
                     &cfg as *const _,
                     Arc::as_ptr(&cfg),
                     Arc::strong_count(&cfg));
        }));
    }
    for h in handles { h.join().unwrap(); }

    println!("strong after all workers joined = {}", Arc::strong_count(&cfg));
    println!("dropping the last handle now:");
}

Output:

main thread: handle at 0x7ffedfc480b8 on the stack, points to 0x55a027533af0 on the heap
strong before spawning = 1
  worker 1: tid=ThreadId(3)  handle at 0x7fe28effdcf0  heap 0x55a027533af0  strong(snapshot)=4
  worker 0: tid=ThreadId(2)  handle at 0x7fe28f1fecf0  heap 0x55a027533af0  strong(snapshot)=3
  worker 2: tid=ThreadId(4)  handle at 0x7fe28edfccf0  heap 0x55a027533af0  strong(snapshot)=2
  worker 3: tid=ThreadId(5)  handle at 0x7fe28ebfbcf0  heap 0x55a027533af0  strong(snapshot)=2
strong after all workers joined = 1
dropping the last handle now:
  Config::drop ran for prod-eu-west-1

There is a lot in there.

Each thread's handle lives on its own stack; the value lives in one shared place. The stack addresses are 0x7fe28effdcf0, 0x7fe28f1fecf0, 0x7fe28edfccf0, 0x7fe28ebfbcf0 — spread about 2 MiB apart, which is exactly the per-thread stack spacing article 1 measured. The heap address is 0x55a027533af0 for all four. Four private handles, one shared allocation.

The workers did not run in order. Worker 1 printed first. Nothing schedules threads in spawn order — the kernel picks, as article 1's PSR column showed.

The counts are 4, 3, 2, 2 — and this is the important one. They are not "wrong". Each is a truthful reading of the count at the instant that thread looked, while other threads were concurrently dropping their handles. Two threads happened to read 2.

This is the first genuinely concurrent lesson of the article: Arc::strong_count is a snapshot, not a fact. By the time the value reaches your variable it may already be stale. It is useful in tests and while debugging; any logic of the form if Arc::strong_count(&x) == 1 { ...assume exclusive... } is a race. (Rust gives you Arc::get_mut and Arc::try_unwrap for that, which do the check atomically.)

The last drop is still exactly one drop. Four threads incremented, four decremented as they exited, main dropped the last handle, and Config::drop ran once. No coordination, no locking, no leader election — just a counter that every participant agrees on.

That is a distributed-systems idea running inside a single process: no node knows who is last, but the shared counter means exactly one of them finds out.

Experiment 3 — What Is Actually in That Allocation?

We have proved there is one allocation and that a number inside it tracks owners. Now let's look at it directly.

The counters are private fields, but the standard library declares the header with #[repr(C)], which fixes the field order. That means we can compute where they are from the data pointer:

#[repr(C)]
struct FakeArcInner<T> { strong: AtomicUsize, weak: AtomicUsize, data: T }

fn counts<T>(a: &Arc<T>) -> (usize, usize) {
    unsafe {
        let data: *const T = Arc::as_ptr(a);
        let inner = (data as *const u8).sub(16) as *const FakeArcInner<()>;
        (
            (*inner).strong.load(Ordering::Relaxed),
            (*inner).weak.load(Ordering::Relaxed),
        )
    }
}

To be clear: this is layout-dependent pointer arithmetic that depends on an implementation detail. It is fine for an experiment and has no place in real code. We are doing it because reading the bytes is more convincing than reading the docs.

sizes / alignment
  size_of::<Arc<u32>>()          = 8
  size_of::<Arc<[u8; 4096]>>()   = 8
  size_of::<Rc<u32>>()           = 8
  size_of::<&u32>()              = 8
  size_of::<Arc<[u32]>>()        = 16  (unsized: fat pointer)
  size_of::<Option<Arc<u32>>>()  = 8  (niche optimisation)
  size_of::<AtomicUsize>()       = 8
  align_of::<AtomicUsize>()      = 8

one Arc<u32> holding 42
  Arc::as_ptr(&a)   = 0x560b747d1d70   (points at the DATA)
  &a as *const _    = 0x7ffdc796b830   (the Arc handle itself, on the stack)
  header starts at  = 0x560b747d1d60
  strong/weak read from the header = (1, 1)
  Arc::strong_count / weak_count   = (1, 0)

Four things to take from this.

An Arc<T> is eight bytes — the size of a pointer. Arc<[u8; 4096]> is also eight bytes. The handle does not grow with the payload, because the payload is not in the handle. Moving an Arc moves one machine word.

The one exception in that list is Arc<[u32]> at sixteen bytes. A slice has no fixed length known at compile time, so the pointer has to carry the length alongside the address. Two words instead of one. Rust calls that a fat pointer, and you get one whenever the thing you are pointing at has a size only known at runtime — slices and trait objects, mostly.

Option<Arc<T>> is also eight bytes. The compiler knows an Arc's pointer is never null, so it uses null to represent None. This is called a niche optimisation: the type already had an impossible value lying around, so the enum tag can hide inside it. Option<Arc<T>> costs nothing over Arc<T>, which is why idiomatic Rust uses it freely.

The pointer aims at the data, not at the start of the allocation. as_ptr gives 0x…d70; the header begins 16 bytes earlier at 0x…d60. Dereferencing an Arc therefore needs no arithmetic at all — the pointer is already at the value. The bookkeeping sits behind you. That is a deliberate design choice favouring the common operation.

The raw weak count is 1 while Arc::weak_count() reports 0. This is the one place my mental model was wrong, and it is not a bug. All the strong handles collectively hold a single implicit weak reference. It exists because the value and the memory are freed at two different moments.

When the strong count hits zero, the value is destroyed — its destructor runs. But the memory block cannot go back to the allocator yet, because a Weak might still be pointing at it, and that Weak needs somewhere valid to look when someone calls upgrade(). So the block is only released when the weak count hits zero too. The implicit weak reference is what holds the block open for as long as any strong handle exists. Arc::weak_count subtracts that implicit one so the number matches the count of Weaks you created.

Watching both counters move confirms it:

after two clones
  Arc::as_ptr(&a) = 0x560b747d1d70
  Arc::as_ptr(&b) = 0x560b747d1d70
  Arc::as_ptr(&c) = 0x560b747d1d70   <- all three point at the same allocation
  header counts   = (3, 1)

after one downgrade (Weak)
  header counts = (3, 2)   (strong, weak)
  strong_count=3  weak_count=1
after dropping the two clones
  header counts = (1, 2)
after dropping the Weak
  header counts = (1, 1)

The standard library's own definition matches the bytes we just read:

#[repr(C, align(2))]
struct ArcInner<T: ?Sized> {
    strong: Atomic<usize>,
    weak: Atomic<usize>,
    data: T,
}

So the diagram from the top of the article was right, with one correction — the weak count starts at 1, not 0:

                 Heap allocation
              ┌──────────────────┐
   offset 0   │ strong = 3       │
   offset 8   │ weak   = 1       │   <- the implicit one
   offset 16  │ String "hello"   │   <- Arc::as_ptr points HERE
              └──────────────────┘

Experiment 4 — What Does `Arc::new` Allocate?

Rather than reason about the size, let's log it.

Rust lets you swap out the global allocator — the thing that actually hands out heap memory. If we replace it with one that prints every allocation and free, we can watch Arc::new do its work:

unsafe impl GlobalAlloc for Logger {
    unsafe fn alloc(&self, l: Layout) -> *mut u8 {
        let p = unsafe { System.alloc(l) };
        if LOG.load(Ordering::Relaxed) {
            emit(format_args!("    alloc  size={:<5} align={:<3} -> {:p}\n",
                              l.size(), l.align(), p));
        }
        p
    }
    // dealloc likewise
}

One trap, which I fell into: the logger must not allocate. My first version used format!, which allocates, which re-enters the logger, which allocates again:

Arc<u8>  (payload 1 bytes, align 1)

thread 'main' (2577) has overflowed its stack
fatal runtime error: stack overflow, aborting

That is the guard page from article 1 catching runaway recursion — the same PROT_NONE page, doing the same job. The fix is to format into a fixed stack buffer and call write directly.

With that, here is the real cost of Arc::new. Each block creates an Arc, clones it, drops the clone, then drops the original:

Arc<()>: payload 0 bytes, align 1
    alloc  size=16    align=8   -> 0x560a0e179d60
    free   size=16    align=8   <- 0x560a0e179d60

Arc<u32>: payload 4 bytes, align 4
    alloc  size=24    align=8   -> 0x560a0e179d60
    free   size=24    align=8   <- 0x560a0e179d60

Arc<[u8; 100]>: payload 100 bytes, align 1
    alloc  size=120   align=8   -> 0x560a0e179480
    free   size=120   align=8   <- 0x560a0e179480

Arc<Aligned64>: payload 64 bytes, align 64
    alloc  size=128   align=64  -> 0x560a0e179d80
    free   size=128   align=64  <- 0x560a0e179d80

Arc<str> from "hello, arc" (10 bytes, unsized)
    alloc  size=32    align=8   -> 0x560a0e179ae0
    free   size=32    align=8   <- 0x560a0e179ae0

Collected:

`Arc<T>`	payload	allocated	overhead
`()`	0	16	16
`u8`	1	24	23
`u32`	4	24	20
`u64`	8	24	16
`[u8; 100]`	100	120	20
`Aligned64` (`align(64)`)	64	128	64
`str` (`"hello, arc"`)	10	32	22

Sixteen bytes of header, then the payload, rounded up for alignment. An Arc<u32> allocates 24 bytes to hold 4 — 83% of it is bookkeeping. That is fine for a config object and expensive for a million small nodes. If you find yourself writing Arc<u32> at scale, consider one Arc over a slab of values instead of one Arc per value.

The align(64) row is the one to remember. If you mark a struct #[repr(align(64))] — a common trick to give it its own cache line, which article 8 is about — the data field has to start at offset 64, so the header padding grows from 16 bytes to 64. You quadrupled the per-object overhead. That may well be the right trade. Just know you made it.

Cloning allocates nothing. Every block shows exactly one alloc and one free despite a clone in between. The hypothesis said clone copies a pointer and bumps a number; the allocator log agrees.

Experiment 5 — What Does `clone` Compile To?

Now the part I actually wanted to see.

A quick note on method, since this is where the article goes properly low-level. Disassembly means taking the compiled binary and printing the machine instructions it contains, which is what objdump does. It is the ground truth: whatever the source says, this is what the CPU will run.

Arc::clone is generic and normally gets inlined into its caller, which leaves no separate function to look at. So we force one:

#[unsafe(no_mangle)]
#[inline(never)]
pub extern "C" fn arc_clone(a: &Arc<u64>) -> Arc<u64> { Arc::clone(a) }

#[unsafe(no_mangle)]
#[inline(never)]
pub extern "C" fn rc_clone(a: &Rc<u64>) -> Rc<u64> { Rc::clone(a) }

Compile with optimisations and disassemble:

$ rustc -O -C panic=abort disasm.rs -o disasm
$ objdump -d --no-show-raw-insn -M intel disasm

0000000000013e80 <arc_clone>:
   13e80:   mov    rax,QWORD PTR [rdi]
   13e83:   lock inc QWORD PTR [rax]
   13e87:   jle    13e8a <arc_clone+0xa>
   13e89:   ret
   13e8a:   ud2

0000000000013eb0 <rc_clone>:
   13eb0:   mov    rax,QWORD PTR [rdi]
   13eb3:   inc    QWORD PTR [rax]
   13eb6:   je     13eb9 <rc_clone+0x9>
   13eb8:   ret
   13eb9:   ud2

Line them up:

Rc::clone     inc      QWORD PTR [rax]
Arc::clone    lock inc QWORD PTR [rax]

The entire difference between Rc and Arc is the four-letter lock prefix.

Same instruction. Same operand. Same three-instruction function. Rc is not a simpler algorithm — it is the same algorithm with the atomicity removed. Everything written about Arc being "the thread-safe one", and that whole Send compile error from earlier, reduces on x86-64 to one prefix byte.

Reading the instructions one at a time:

mov rax, QWORD PTR [rdi] — load the pointer out of the Arc handle. rdi holds the function's first argument.
lock inc QWORD PTR [rax] — atomically add one to the value at that address. That address is the strong count, at offset 0 of the header, exactly where experiment 3 found it. The lock prefix tells the CPU to make this read-modify-write indivisible with respect to other cores.
jle / ud2 — the overflow guard. ud2 is an undefined instruction: executing it crashes the process immediately. If the count ever exceeded isize::MAX, wrapping around toward zero would free memory that other threads still hold, so the library chooses to abort instead.

From the source:

const MAX_REFCOUNT: usize = (isize::MAX) as usize;

Note it is isize::MAX, not usize::MAX. Half the range is deliberately given away so the check can be a single cheap signed comparison — the jle above. You cannot hit this by accident; you could hit it with mem::forget in a loop.

And drop:

0000000000013e90 <arc_drop>:
   13e90:   mov    rax,QWORD PTR [rdi]
   13e93:   lock dec QWORD PTR [rax]
   13e97:   jne    13ea1 <arc_drop+0x11>
   13e99:   mov    rdi,QWORD PTR [rdi]
   13e9c:   jmp    13d30 <...Arc$LT$T$C$A$GT$9drop_slow...>
   13ea1:   ret

Four instructions in the common case. Decrement; if the result is not zero, return. Only when it reaches zero does it jump to drop_slow, which runs the value's destructor and releases the memory.

That structure — tiny hot path, expensive path moved out of line — is what you would hand-write if you were implementing this yourself in C. The compiler produced it for you.

So the full picture of a clone:

Arc::clone(&a)
   ↓
load the pointer                      (1 instruction)
   ↓
lock inc the strong count             (1 instruction, atomic)
   ↓
check for overflow, return            (2 instructions)

Three or four instructions, no branches taken, no function calls, no allocation. On paper, nearly free. The rest of this article is about why "on paper" is doing a lot of work in that sentence.

Experiment 6 — The Memory Orderings

Here is something that always looked arbitrary to me. Increment and decrement are mirror images, but Arc treats them differently:

clone increments with Relaxed ordering.
drop decrements with Release ordering, then performs an Acquire fence.

What is a memory ordering? Modern CPUs and compilers reorder memory operations for speed. An ordering is a constraint you attach to an atomic operation telling them what they may not reorder around it. Three of them matter here:

Relaxed — no constraint at all. Just make the arithmetic atomic and nothing else.
Release — "everything I did before this point must be visible to whoever observes this operation."
Acquire — the matching half. "Everything the other side did before releasing is now visible to me."

Release and Acquire come in pairs. One thread publishes, another thread picks up. Neither is useful alone.

Crucially, ordering is not about the count being correct — atomicity handles that. It is about the data the count protects.

The standard library explains the increment:

"Using a relaxed ordering is alright here, as knowledge of the original reference prevents other threads from erroneously deleting the object... Increasing the reference counter can always be done with memory_order_relaxed: New references to an object can only be formed from an existing reference, and passing an existing reference from one thread to another must already provide any required synchronization."

let old_size = self.inner().strong.fetch_add(1, Relaxed);

The argument is worth restating slowly, because it is genuinely subtle. To clone an Arc, you must already hold one. However you got it — a channel, a Mutex, a thread::spawn closure — that handover already established the necessary ordering. The increment therefore publishes nothing new that another thread must observe in a particular order. So it can be as weak as an atomic gets.

Dropping is not like that. When you drop, you are announcing "I am finished touching this value" to whichever thread happens to drop last — because that thread is going to run the destructor. Everything you did to the value must be visible to it before it starts tearing the value down. That is precisely a release. And the thread that sees the count hit zero needs the matching acquire before it may safely destroy anything:

#[cfg(not(sanitize = "thread"))]
macro_rules! acquire {
    ($x:expr) => {
        atomic::fence(Acquire)
    };
}

#[cfg(sanitize = "thread")]
macro_rules! acquire {
    ($x:expr) => {
        $x.load(Acquire)
    };
}

(A nice detail: the thread-sanitizer build uses a load instead of a fence, because TSan models fences poorly and would report false positives. The ordering is chosen partly for the benefit of a debugging tool.)

So what do these orderings cost? I isolated the two operations into a tiny #![no_std] crate so I could see the generated code without the rest of Arc around it:

#[unsafe(no_mangle)]
pub fn clone_op(strong: &AtomicUsize) -> usize {
    strong.fetch_add(1, Ordering::Relaxed)
}

#[unsafe(no_mangle)]
pub fn drop_op(strong: &AtomicUsize) -> bool {
    if strong.fetch_sub(1, Ordering::Release) != 1 { return false; }
    fence(Ordering::Acquire);
    true
}

On x86-64:

clone_op:
    movl    $1, %eax
    lock        xaddq   %rax, (%rdi)
    retq

drop_op:
    lock        decq    (%rdi)
    sete    %al
    jne .LBB1_2
    #MEMBARRIER
.LBB1_2:
    retq

Look closely at #MEMBARRIER. That is an assembler comment. The Acquire fence generates zero instructions.

That is not the compiler ignoring you. x86-64 has a strongly ordered memory model (often called TSO, for Total Store Order): the hardware already refuses to reorder loads with loads or stores with stores, and a lock-prefixed read-modify-write is already a full barrier. The fence is genuinely redundant on this architecture, so it costs nothing.

Compare against the strictest possible version to confirm:

drop_op_seqcst:
    lock        decq    (%rdi)
    sete    %al
    retq

SeqCst and Release-plus-fence emit the same real work here.

An honest caveat. This tells you something about x86-64, not about orderings in general. On a weakly ordered architecture — AArch64, RISC-V, POWER — Relaxed, Release and SeqCst compile to genuinely different instructions with genuinely different costs, and the Acquire fence becomes a real barrier. I wanted to show that contrast with actual output, but this machine has only the x86-64 standard library installed and I could not reach the toolchain server to add an AArch64 target. Rather than paste assembly I did not generate, I am marking it unmeasured. If you are on an ARM Mac, rustc -O --emit asm --target aarch64-apple-darwin on that snippet will show you what I could not.

What this does establish: on x86-64, Arc's careful ordering choices are free. The cost of Arc is not the orderings. It is the lock prefix — and specifically, what that prefix does to the other cores.

Experiment 7 — What Does It Cost?

Single thread, nothing contended, 20 million iterations of each operation:

single thread, uncontended, 20000000 iterations each
  *arc                (deref)             0.44 ns/op
  Arc::strong_count   (load)              0.56 ns/op
  Rc::clone + drop                        4.60 ns/op
  Arc::clone + drop                      12.89 ns/op
  Arc::downgrade + drop Weak             17.96 ns/op
  Weak::upgrade + drop Arc               18.01 ns/op

Reading through an Arc costs 0.44 ns — one pointer dereference, no atomic, effectively free. Cloning and dropping costs 12.89 ns, about 29x more than reading. And Arc is 2.8x the cost of Rc for the identical operation. That factor of 2.8 is the price of one lock prefix.

The ratios are the useful part. Thirteen nanoseconds is not much in absolute terms — article 1 measured thread creation at roughly 37 *micro*seconds, three thousand times more. But it is a lot compared with the thing people assume it is comparable to. Arc::clone is not "basically free, like a borrow". It is thirty times a borrow.

And that is the uncontended number.

Stressing the Design — More Than One Core

Everything so far was one thread. Now the question that decides whether your server scales: what happens when several threads clone the same Arc?

Three cases. clone private: each thread has its own separate Arc, so each touches its own counter. clone shared: all threads clone one Arc, hammering one counter. deref shared: all threads read through one shared Arc without cloning.

median of 5 runs, 6000000 clone+drop pairs per thread per run

 threads    clone private     clone shared      ratio     deref shared
       1         12.91 ns         12.93 ns       1.0x          0.64 ns
       2         13.39 ns         60.76 ns       4.5x          0.62 ns
       4         27.90 ns        119.73 ns       4.3x          1.54 ns
       8         53.03 ns        252.25 ns       4.8x          1.35 ns

Three findings, and the middle one is the point of the article.

Sharing the counter costs about 4.5x

At two threads, a clone goes from 13.4 ns to 60.8 ns. The instruction did not change. What changed is the cache line.

A cache line is the unit in which CPUs move memory — 64 bytes on this machine (getconf LEVEL1_DCACHE_LINESIZE says so). Cores do not fetch individual bytes; they fetch lines into their own private caches. To write to a line, a core must hold it exclusively — no other core may have a copy.

So when two cores both want to lock inc the same counter, the line containing it must bounce between them. Core 0 takes exclusive ownership, increments, then core 1 must take it away, and so on. Neither core can proceed while the other holds it.

     Core 0 cache            Core 1 cache
   ┌──────────────┐        ┌──────────────┐
   │ strong count │ ◄────► │ strong count │
   └──────────────┘        └──────────────┘
          ▲                       ▲
          └── the line ping-pongs ─┘

The atomic operation is fast. Acquiring the line is not. This is cache coherence showing up in your latency numbers, and it is why "just wrap it in an Arc" can quietly put a ceiling on how well a hot path scales.

Read-only sharing is free

The deref shared column stays under 2 ns at every thread count. Many cores may hold the same line simultaneously as long as they are only reading — coherence only forces exclusivity for writes.

So sharing immutable data across threads costs nothing. Sharing a counter costs a great deal. The awkward part is that Arc makes you do the second in order to get the first.

The `clone private` column rises too — and that is article 1's fault

This machine has 2 cores. At 4 and 8 threads the private numbers roughly double and quadruple: 12.9, 13.4, 27.9, 53.0. That is not atomics. That is exactly the oversubscription curve from article 1 — throughput holds, per-unit latency degrades in proportion, because there are more runnable threads than cores.

Which is why the ratio column exists. Dividing shared by private cancels the scheduling effect and isolates the contention.

The Mistake That Shows Up in Production

Here is a shape I have written myself more than once. Some shared thing every request needs — config, a routing table, a connection pool, a compiled regex set — behind an Arc. Then, inside the request loop:

let per_msg = Arc::clone(&cfg);     // the mistake
acc = acc.wrapping_add(handle(&per_msg, i));

instead of simply borrowing what you already own:

acc = acc.wrapping_add(handle(&cfg, i));

Both compile. Both are correct. One of them makes every worker thread write to the same cache line on every single message.

3000000 messages per thread, shared Arc<Config>
 threads    clone per message       borrow the Arc   overhead
       1             12.60 ns              0.81 ns      15.6x
       2             80.79 ns              0.64 ns     126.1x
       4            165.67 ns              1.16 ns     142.8x

At one thread the unnecessary clone costs 16x. At two threads, 126x. At four, 143x.

Look at how the two columns behave differently. The borrow column is flat — it does not care how many threads are running, because reading shared immutable data scales perfectly. The clone column gets dramatically worse with concurrency.

The penalty for this mistake grows with the number of cores you add. You will not find it on a laptop running a single-threaded test. You will find it when you deploy to a 32-core box and discover that scaling stops at four.

The fix is not to avoid Arc. It is to clone at the right granularity: once per task, not once per message.

for _ in 0..workers {
    let cfg = Arc::clone(&cfg);      // one atomic increment, once
    thread::spawn(move || {
        for msg in inbox {
            handle(&cfg, msg);        // borrow inside the loop
        }
    });
}

One increment per thread instead of one per message. That is the difference between the two columns above.

A habit that falls out of this: write functions that take &T, not Arc<T>. If a function does not need to keep the value after it returns, it does not need a reference count — it needs a borrow. Putting Arc<T> in a signature is an instruction to every caller to perform an atomic read-modify-write, and most of them did not need to.

`Weak`, Cycles, and the Leak Rust Does Not Prevent

Reference counting has one classic failure mode, and Rust does not save you from it.

If two objects hold Arcs to each other, they keep each other alive forever:

Parent ──Arc──► Child
   ▲              │
   └─────Arc──────┘

Neither count can reach zero, because each is held up by the other.

struct Node { name: &'static str, peer: Mutex<Option<Arc<Node>>> }
impl Drop for Node {
    fn drop(&mut self) { println!("    Drop::drop ran for {}", self.name); }
}

two nodes pointing at each other with Arc:
    strong_count(a) = 2, strong_count(b) = 2
    ...leaving the scope now
    (nothing printed above? then neither destructor ran)

Nothing printed. Both handles went out of scope, both counts fell from 2 to 1, neither reached zero, neither destructor ran, and that memory is gone for the life of the process. No unsafe, no warning, no panic.

Rust's safety guarantees do not include "no leaks". Leaking is safe. It is just not what you wanted.

Weak<T> is the fix. A Weak points at the allocation without owning the value — it increments the weak count, not the strong one, so it cannot keep the value alive. Make the back-edge weak and the cycle breaks:

same shape, but the back-edge is a Weak:
    strong_count(a) = 1, weak_count(a) = 1
    ...leaving the scope now
    Drop::drop ran for B
    Drop::drop ran for A

And a Weak tells you honestly when the thing is gone:

what a Weak sees after the last strong reference goes:
    while alive: w.upgrade() = Some(99)
    after drop:  w.upgrade() = None
    strong_count via Weak = 0

upgrade() is the entire point of Weak: a fallible promotion back to an Arc that returns None rather than handing you a dangling pointer.

That check is not free. From the cost table, Weak::upgrade plus dropping the resulting Arc costs 18.01 ns against 12.89 ns for a plain clone — because upgrade cannot be a blind increment. It has to atomically check that the strong count is non-zero and increment it in one indivisible step, or it would lose the race against a concurrent final drop.

Where this matters: parent/child trees, observer registries, caches holding handles to objects they must not keep alive, and graphs generally. If your object graph can contain a cycle, one direction must be Weak.

What I Could Not Measure Here

Two gaps, stated rather than papered over.

AArch64 code generation. As above — no cross-target standard library, no reachable toolchain server. "The orderings are free" is a claim about x86-64 only.

Strong/weak false sharing. The two counters sit at offsets 0 and 8, which I confirmed are inside the same 64-byte line:

  strong at offset 0  -> 0x55c8d6339d60
  weak   at offset 8  -> 0x55c8d6339d68
  64-byte line of strong = 0x55c8d6339d40
  64-byte line of weak   = 0x55c8d6339d40
  same cache line?       true

In principle, then, a thread doing downgrade/upgrade should interfere with a thread doing clone, even though they touch different words — that is what false sharing means. I built that benchmark and could not interpret the result.

The problem: downgrade is simply a more expensive operation than clone — 18 ns against 13 ns even with no contention at all. My benchmark reported the slower of the two threads, so it was mostly measuring that difference rather than any interference between them. I could publish the numbers, but they would not mean what the heading claimed.

Article 8 is about false sharing specifically and will do this properly, with matched operations. I would rather this article be shorter and correct.

Production Connection

API servers and gRPC services. Arc<AppState> is the standard pattern and it is a good one — until the state is cloned per request rather than per connection or per worker. The 126x figure above is what that costs at two threads. Clone at task boundaries; borrow inside them.

Databases, caches, vector stores. Arc<[u8]> or Arc<str> for shared immutable pages, embeddings, or interned keys is excellent: reads scale perfectly and the payload is never copied. But mind the allocation table — 16 bytes of header means Arc per small object is heavy. Prefer one Arc over a slab of values rather than an Arc around each value.

Message brokers and Kafka-style consumers. Passing Arc<Message> down a pipeline is right — no copying regardless of payload size. Watch the fan-out: if N consumers each clone the same Arc per message, that is N atomic RMWs on one cache line per message, and article 1's backpressure chain gains a new first link.

Low-latency and trading systems. 13 ns uncontended, 61 ns contended, and the contended figure is variable because it depends on which core last owned the line. If your tail-latency budget is in the hundreds of nanoseconds, a contended Arc clone on the hot path is both a cost and a variance source. Clone once at setup; pass &T thereafter.

Blockchain nodes and AI infrastructure. Both are the same shape: one large immutable thing — a state snapshot, a loaded model, a mempool view — read by many workers. That is Arc at its best, provided the clone count is proportional to workers, not to operations.

Everywhere. If your object graph can contain a cycle, one direction must be Weak.

Conclusion

The hypothesis mostly held. Arc is a pointer to a heap header of two counters plus the value; clone bumps a count without touching the data; the last drop frees. What the investigation added:

An Arc<T> is 8 bytes, and so is Option<Arc<T>>. The pointer aims at the data; the counters live 16 bytes behind it.
The header is 16 bytes — 24 allocated for a u32, so 83% overhead. An align(64) payload pushes header overhead to 64 bytes.
Arc::clone and Rc::clone differ by exactly one thing: lock inc versus inc. Three instructions each, plus an overflow guard that aborts at isize::MAX.
Arc::drop is four instructions in the common case. Destroying the value and freeing the memory happen in a separate function it jumps to only when the count hits zero.
On x86-64 the memory orderings are free. The Acquire fence emits #MEMBARRIER — a comment, not an instruction. The orderings still matter; they just do not cost anything on this architecture.
Uncontended: 12.89 ns to clone and drop — 2.8x an Rc, and 29x a plain deref at 0.44 ns.
Contended: about 4.5x worse, and the penalty persists as threads increase. Read-only sharing, by contrast, costs nothing at any thread count.
Arc::strong_count is a snapshot, not a fact. Four threads read 4, 3, 2, 2 from the same counter.
Cloning per message instead of per task cost 126x at two threads, and got worse with more cores — the worst possible property for a bug to have.
Cycles of Arc leak silently. Safe, warning-free, permanent.

The thing I keep returning to is that lock prefix. One prefix on one instruction is the entire difference between the single-threaded and the multi-threaded reference count. And the reason it is expensive has nothing to do with the instruction itself — it is that a cache line can only be owned by one core at a time.

Arc does not really cost you an atomic. It costs you exclusive ownership of a cache line, on every clone, contested with every other thread doing the same.

Article 3: Mutex vs RwLock in Rust — Benchmarking Real Contention. We can now share ownership across threads, but everything shared so far has been immutable. The moment we want to mutate shared state we need mutual exclusion — and the two obvious choices behave very differently under load.

Note where Arc<Mutex<T>> puts things: the lock sits in the same allocation as the reference count we just measured, quite possibly in the same cache line. The contention effects from this article do not go away when you add a lock. They compound.

Then article 4 follows a blocked Mutex into the kernel, and lands back on the same futex syscall article 1 found inside join().

Appendix: Environment and Reproducibility

kernel:  6.18.44-fc-v21 (x86_64)
distro:  Ubuntu 24.04.4 LTS
rustc:   1.95.0 (59807616e 2026-04-14)
glibc:   2.39-0ubuntu8.7
objdump: GNU binutils 2.42
cpu:     Intel(R) Xeon(R) Processor @ 2.80GHz, 2 cores
cache line: 64 bytes (getconf LEVEL1_DCACHE_LINESIZE)

On comparing with article 1: this is not the same host. Article 1 ran on a machine reporting 2.10 GHz; this one reports 2.80 GHz, and the container migrated mid-session. Every timing in this article was re-run on the machine above so the numbers are internally consistent. Do not compare absolute nanoseconds across the two articles — ratios are comparable, raw times are not.

File	Purpose
`exp1/`	the cargo project from experiment 1
`threads_arc.rs`	the same `Arc` across four OS threads
`threads_rc.rs`	the `Rc` version, for the `Send` compile error
`layout.rs`	reads the strong/weak header directly; sizes and pointer identity
`alloc_log.rs`	logging `#[global_allocator]`; real `Arc::new` allocation sizes
`disasm.rs`	`#[no_mangle]` wrappers for arc_clone / arc_drop / rc_clone / rc_drop
`orderings.rs`	`#![no_std]` isolation of the two atomic ops, for `--emit asm`
`ops.rs`	per-operation cost table
`bench3.rs`	contention: private vs shared vs read-only, median of 5
`realistic.rs`	clone-per-message vs borrow, across thread counts
`cycle.rs`	`Arc` cycle leak, `Weak` back-edge, `upgrade()` after drop
`addrs.rs`	cache-line placement of the two counters

Commands:

$ cargo run                                  # experiment 1
$ rustc -O threads_arc.rs -o threads_arc && ./threads_arc
$ rustc -O -C panic=abort disasm.rs -o disasm
$ objdump -d --no-show-raw-insn -M intel disasm | awk '/<arc_clone>:/,/^$/'
$ rustc -O --emit asm --target x86_64-unknown-linux-gnu orderings.rs -o -
$ MALLOC_ARENA_MAX=1 ./ops
$ MALLOC_ARENA_MAX=1 ./bench3
$ MALLOC_ARENA_MAX=1 ./realistic

MALLOC_ARENA_MAX=1 is inherited from article 1: glibc otherwise gives each thread its own malloc arena, which distorts memory accounting. It does not affect the timings here, but it keeps the runs comparable.

Caveats:

Two cores. The contention numbers would likely be worse on a bigger machine. More cores means more competition for the line, and on a multi-socket server the two cores fighting over it may be on physically separate chips, which makes each transfer much more expensive.
Virtualised. Timing under a hypervisor is noisier than bare metal, which is why the contention table is a median of five runs.
x86-64 only. Everything about lock prefixes and free fences is specific to this memory model.
The single-thread table is one run. Re-running ops.rs moves figures by roughly ±1% (Arc::clone came out at 12.89 and 12.99 ns on consecutive runs). The contention table is a median because it is far noisier.
Layout poking is not an API. The sub(16) arithmetic works because ArcInner is repr(C) and I checked the source. It is fine for an experiment and wrong for real code.
Addresses vary per run. The pointer values in different sections come from different runs of different programs, so they will not match each other — only the relationships within a single output block are meaningful.

Thanks for Reading

If you got this far — thank you.

Article 1 ended with me admitting I had the cost model backwards: I assumed thread creation was the expensive part, and the measurements said blocking was. This one had a smaller version of the same thing. I expected Arc's memory orderings to be where the cost lived, because that is the part everyone writes about. On x86-64 they are free, and the cost turned out to be somewhere much less glamorous: one cache line, and which core owns it.

I also had the weak count wrong, and only found out by reading the bytes.

If you spot something wrong here, I would much rather hear it now than leave it standing while eight more articles get built on top of it. And if you run the contention table on a machine with real core counts, I would like to see it — 4.5x on two cores is almost certainly the friendly version.

Next up — Article 3: Mutex vs RwLock in Rust: Benchmarking Real Contention.

GitHub: GITHUB_URL
LinkedIn: LINKEDIN_URL

What Actually Happens When Rust Spawns an OS Thread?

Parikalp Bhardwaj — Sat, 15 Aug 2026 12:38:06 +0000

Following std::thread::spawn from Rust to Linux. Article 1 of **Below the Abstraction* — a series that takes everyday Rust abstractions and follows them down to the kernel. Every number below was measured on a real machine, not recalled.*

Why I'm Starting With Threads

I have written a lot of Rust that creates threads. Backend services, gRPC servers, blockchain node components, async applications, background workers, infrastructure glue. thread::spawn is one of those calls that becomes invisible after the first hundred times you type it.

At some point that stopped being acceptable to me.

I had used thread::spawn many times before, but I had never actually verified what Linux saw when the call happened. I knew the shape of the answer — "it makes an OS thread" — but I could not have told you which syscall was issued, how much memory the call reserved, what the kernel scheduler now had to track, or why the same program behaves very differently at 4 threads and at 4,000.

That gap matters more than it looks. Almost every hard problem I have hit in production — tail latency that appears only under load, memory that grows without a leak, a service that goes from fine to unusable somewhere between 500 and 2,000 concurrent connections — turned out to be a question about the layer below the one I was writing in.

There is a second reason this exists. I like low-level work — it is the part of the stack where you can actually check the answer instead of arguing about it. And the fastest way I know to find out whether I really understand something is to try to explain it to someone else. Writing forces the gaps into the open: you can hand-wave your way through a conversation about threads, but you cannot hand-wave a syscall trace. So this series is a little bit me teaching, and mostly me learning in public.

It has one rule, and it is the rule I want to apply to everything:

Don't just use the abstraction. Follow it down until you understand the system underneath it.

Threads come first because everything else in the series stands on them. Arc, Mutex, RwLock, futexes, thread pools, Tokio's scheduler, epoll, async tasks — all of them are, in the end, statements about what an OS thread is, what it costs, and when the kernel takes it away from you. You cannot meaningfully say "a Tokio task is cheaper than a thread" until you have measured what a thread costs.

This article is the measurement.

Why Low-Level Knowledge Matters

There is a version of this argument that is just nostalgia — "real engineers know assembly." That is not the argument. The argument is that modern backend and distributed systems fail in ways that are only legible one or two layers down.

Here is the stack this series keeps returning to:

Distributed System
        ↓
Network Communication
        ↓
Sockets
        ↓
Async Runtime / Threads
        ↓
Synchronization
        ↓
Memory
        ↓
System Calls
        ↓
Kernel
        ↓
CPU + Hardware

You write at the top. You get paged at the bottom.

Performance

Every high-level abstraction eventually decomposes into a small set of concrete, countable things:

allocations
syscalls
locks
atomics
cache misses
context switches
network packets

An API surface tells you nothing about how many of each you just bought. Vec::push is amortized O(1) and also, sometimes, an mmap. thread::spawn is one line and also, as we are about to see, a mmap, an mprotect, a clone3, and a new schedulable entity that the kernel must now consider on every scheduling decision for the life of that thread.

You cannot reason about performance only from the API surface.

Concurrency

A line like:

Arc<Mutex<T>>

looks like a type. It is closer to a system:

atomic reference counts
memory ordering
lock state
parking
waking
kernel interaction
cache coherence

At low contention none of that is visible. At high contention it is the only thing that is visible — the difference between a lock that spins briefly in userspace and one that puts a thread to sleep in the kernel shows up as a CPU utilization graph that makes no sense, or a p99 that is 40x the p50. Articles 2, 3 and 4 take that apart.

Async Systems

Most Rust developers write:

tokio::spawn(...)

long before they understand:

Future
Poll
Waker
runtime scheduler
reactor
epoll
kernel readiness notifications

I think async Rust is genuinely confusing if you meet it first. It makes far more sense in the other order: OS threads, blocking, scheduling, and I/O readiness first — then async as a specific answer to a specific cost. That is why this series does not start with Tokio.

Distributed Systems

This is the part I care most about, and it is the part that is usually skipped.

Distributed systems problems become local systems problems, almost always. A latency graph in Grafana is a distributed artifact; the thing producing it is a single machine doing something specific.

Request latency
    ↓
network wait
    ↓
socket readiness
    ↓
runtime scheduling
    ↓
task wakeup
    ↓
lock acquisition
    ↓
cache/database access

Or, the shape I have personally chased more than once:

Kafka consumer slowdown
        ↓
backpressure
        ↓
queue growth
        ↓
memory growth
        ↓
scheduler pressure
        ↓
latency spikes

Every arrow in those chains is a local-machine phenomenon. "The cluster is slow" is a summary; the mechanism is a scheduler, a lock, a page fault, or a socket buffer.

A distributed system is not something separate from operating systems. It is multiple operating systems communicating over an unreliable network. If you do not understand what one machine does under load, a hundred of them will not be easier.

The Series Roadmap

Each article is a single investigation, and each one depends on the one before it.

What Actually Happens When Rust Spawns an OS Thread? — establish what a thread is, and what it costs.
What Does Arc Actually Do? Atomic Reference Counting Under the Hood — now that threads share memory, how is ownership shared safely?
Mutex vs RwLock in Rust: Benchmarking Real Contention — shared memory needs mutual exclusion; which primitive, and when?
What Happens When a Rust Mutex Blocks? Parking, Waking, and Futexes — the moment a lock stops being userspace-only and calls the kernel.
Building a Thread Pool in Rust From Scratch — the direct consequence of article 1's cost measurements.
Why Tokio Tasks Aren't Threads: 10,000 Tasks vs OS Threads — the same workload, re-measured against the numbers from article 1.
From epoll to Tokio: What Happens When Rust Waits for Network I/O? — where the runtime actually blocks.
False Sharing in Rust: CPU Cache Lines and Multithreaded Performance — going below the kernel, into the hardware.
Building a Concurrent LRU Cache in Rust — everything above, applied to one realistic data structure.
Tracing Rust async/await All the Way Down to epoll — the full path, end to end.

The progression is deliberate: threads → shared state → synchronization → blocking → pooling → async → I/O readiness → hardware → applied → full trace.

Series Philosophy

Every article follows the same investigation model:

Question
   ↓
Hypothesis
   ↓
Small Rust Program
   ↓
Run It
   ↓
Inspect Linux
   ↓
Measure
   ↓
Read Source Code
   ↓
Explain Internals
   ↓
Stress the Design
   ↓
Production Connection
   ↓
Conclusion

And one hard constraint: nothing is invented. No benchmark numbers, memory figures, syscall traces, or scheduler behaviour is written from memory or plausibility. Every number and every trace line in this article was produced by running the programs shown, on the machine described in the appendix, and pasted in. Where a result surprised me or contradicted what I expected, I say so rather than smoothing it over — a couple of the measurements below did exactly that.

Your numbers will differ. The shapes should not.

Why Rust for This Series

Rust is unusually good for this kind of work, and it is worth being specific about why rather than cheerleading.

What helps:

Ownership and borrowing make the memory model explicit at the type level, so "who owns this buffer across threads" is a compile-time question rather than a debugging session.
Send and Sync encode thread-safety as traits. When you study synchronization primitives, having the safety rules written in the type system is a genuine teaching aid.
No hidden runtime. std::thread::spawn is a thin wrapper over the platform's native threads. There is no green-thread layer or VM between you and clone3, which is exactly what you want when the goal is to see the syscall.
Explicit atomics with explicit orderings (Relaxed, Acquire, Release, SeqCst) force you to state memory ordering rather than inherit it.
Zero-cost abstractions mean the assembly usually corresponds to the source in a way you can follow.
First-class FFI lets you drop to libc and call the raw syscall when you want to compare.

What genuinely hurts:

Compiler complexity. The borrow checker is a real cost when you are prototyping a data structure whose whole point is aliasing.
Async type complexity. Deeply nested impl Future types, Pin, and lifetime errors in async code are hard, and error messages in that area are still rough.
Native dependencies and build friction. Anything touching C libraries brings the usual pain, plus bindgen/cc on top.
unsafe and FFI boundaries. The moment you go low-level you lose the guarantees that made Rust attractive, and you are back to C-level discipline without C's decades of tooling defaults.
Ecosystem maturity for systems tooling. C and C++ have a longer tail of profilers, sanitizers, and debugger integrations that "just work". Rust's are good and improving, but not equal.
Debugging low-level async. A stack trace through a poll chain is much less informative than a stack trace through blocking calls. This is a real regression in observability, and article 10 will have to deal with it head-on.

I am using Rust here because it is the best available lens on these mechanisms, not because it removes them.

The Question

What actually happens after this line?

std::thread::spawn(...)

Specifically:

Which syscall is issued, and with which arguments?
What memory is reserved before the syscall, and by whom?
What does Linux create, and how does it name and track it?
What does it cost — in time, in virtual memory, in resident memory?
What changes when there are more runnable threads than CPU cores?

The Hypothesis

My starting hypothesis, before running anything:

std::thread::spawn creates a native OS thread. Rust's standard library delegates to pthread_create, which allocates a stack, then asks the kernel to create a new task sharing the caller's address space. Linux tracks that task independently and schedules it independently. The thread is not free: it costs a stack-sized virtual memory reservation and a kernel-visible schedulable entity.

That is a hypothesis, not an answer. Let's check every clause of it.

The Experiment

The smallest program that spawns exactly one thread:

// hello_thread.rs
use std::thread;

fn main() {
    let handle = thread::spawn(|| {
        println!("hello from a spawned thread");
    });
    handle.join().unwrap();
    println!("main thread done");
}

Built with rustc -O -g hello_thread.rs -o hello_thread, then traced:

$ strace -f -c ./hello_thread

The full syscall summary for the whole process life is 89 syscalls. The interesting ones:

  0.00    0.000000           0        16           mmap
  0.00    0.000000           0         8           mprotect
  0.00    0.000000           0         5           munmap
  0.00    0.000000           0         2           gettid
  0.00    0.000000           0         1           futex
  0.00    0.000000           0         2           set_robust_list
  0.00    0.000000           0         2           rseq
  0.00    0.000000           0         1           clone3

One clone3. That is the whole thread creation, from the kernel's point of view. Everything else around it is preparation and cleanup.

Note what is not there: there is no thread_create syscall on Linux. There is no separate "thread" object. Threads and processes are the same kernel primitive — a task_struct — created by the same syscall family, differing only in which resources they share. That is the first thing the trace teaches, and it reframes everything that follows.

What the Trace Actually Shows

Here is the relevant window from the full trace (strace -f), with the process ID 3616 as the main thread and 3617 as the spawned thread. This is verbatim output, trimmed only to the region around the spawn:

3616  mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7ff8a9bff000
3616  mprotect(0x7ff8a9c00000, 2097152, PROT_READ|PROT_WRITE) = 0
3616  rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
3616  clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM
              |CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
              child_tid=0x7ff8a9dff990, parent_tid=0x7ff8a9dff990, exit_signal=0,
              stack=0x7ff8a9bff000, stack_size=0x1fff40, tls=0x7ff8a9dff6c0}
             => {parent_tid=[3617]}, 88) = 3617
3616  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
3616  futex(0x7ff8a9dff990, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 3617, NULL,
            FUTEX_BITSET_MATCH_ANY <unfinished ...>
3617  rseq(0x7ff8a9dfffe0, 0x20, 0, 0x53053053) = 0
3617  set_robust_list(0x7ff8a9dff9a0, 24) = 0
3617  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
3617  mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff8a1a00000
3617  munmap(0x7ff8a1a00000, 39845888)  = 0
3617  munmap(0x7ff8a8000000, 27262976)  = 0
3617  mprotect(0x7ff8a4000000, 135168, PROT_READ|PROT_WRITE) = 0
3617  gettid()                          = 3617
3617  write(1, "hello from a spawned thread\n", 28) = 28
3617  madvise(0x7ff8a9bff000, 2076672, MADV_DONTNEED) = 0
3617  exit(0)                           = ?
3616  <... futex resumed>)              = 0
3616  write(1, "main thread done\n", 17) = 17
3616  exit_group(0)                     = ?

There is a lot in there. Let's go line by line.

1. The stack is allocated in userspace, before the kernel is involved

mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0)
mprotect(0x7ff8a9c00000, 2097152, PROT_READ|PROT_WRITE)

2101248 bytes is 2 MiB + 4096. The whole region is mapped PROT_NONE — no access at all — and then all of it except the first page is flipped to read/write.

That leftover 4 KiB at the bottom is the guard page. It is a page with no permissions sitting immediately below the stack, so that a stack that grows too far hits an unmapped page and faults instead of silently scribbling over whatever mapping happens to be next in the address space.

This is worth pausing on: the kernel did not allocate the thread stack. glibc did, in userspace, with an ordinary anonymous mmap, before clone3 was ever called. The kernel is handed a pointer to memory that already exists. A "thread stack" is not a kernel concept — it is just anonymous memory that a thread happens to use as a stack.

MAP_NORESERVE is not set here, but the mapping is PROT_NONE until mprotect and untouched afterwards, so no physical pages are committed yet. We will measure exactly that in a moment.

2. Why 2 MiB?

Not because Linux says so. The main thread's stack limit on this machine is 8 MiB:

prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0

The 2 MiB is Rust's choice. In the standard library's Unix thread implementation:

#[cfg(not(any(
    target_os = "l4re",
    target_os = "vxworks",
    target_os = "espidf",
    target_os = "nuttx"
)))]
pub const DEFAULT_MIN_STACK_SIZE: usize = 2 * 1024 * 1024;

We can confirm the plumbing empirically rather than trusting the constant. RUST_MIN_STACK should change the size of that mmap, and it does:

$ RUST_MIN_STACK=1048576 strace -f -e trace=mmap,clone3 ./hello_thread
mmap(NULL, 1052672, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, ...) 
clone3({... stack_size=0xfff40 ...})

$ RUST_MIN_STACK=8388608 strace -f -e trace=mmap,clone3 ./hello_thread
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, ...)
clone3({... stack_size=0x7fff40 ...})

1052672 = 1 MiB + 4096. 8392704 = 8 MiB + 4096. Same guard page, different stack. The request travels from a Rust env var, through Builder, into pthread_attr_setstacksize, and out as the size of an mmap. That is the whole chain, visible in one command.

3. `clone3` is the actual thread creation

clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM
        |CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, ...}) = 3617

Each flag is a decision about what the new task shares with the old one:

Flag	Meaning	Consequence
`CLONE_VM`	share the address space (`mm_struct`)	this is what makes it a thread and not a process
`CLONE_FS`	share filesystem info (cwd, umask, root)	`chdir` in one thread affects all
`CLONE_FILES`	share the file descriptor table	fd 7 means the same socket in every thread
`CLONE_SIGHAND`	share signal handlers	one handler table per process
`CLONE_THREAD`	join the same thread group	same PID/TGID; `getpid()` matches
`CLONE_SYSVSEM`	share System V semaphore undo state
`CLONE_SETTLS`	install the given TLS pointer	`thread_local!` needs this
`CLONE_PARENT_SETTID`	write the new TID into the parent's memory	the parent learns the child's TID
`CLONE_CHILD_CLEARTID`	clear that word on exit and futex-wake it	this is how `join()` works

exit_signal=0 means the parent process is not sent SIGCHLD when this task dies, because it is a thread, not a child process.

The first four or five flags are the entire difference between fork and "spawn a thread". A process is the same call with fewer sharing flags. Once you have seen this, "threads share memory, processes don't" stops being a rule you memorised and becomes an argument you passed.

4. `join()` is a futex, and the kernel does the wake

Look at the pairing:

3616  futex(0x7ff8a9dff990, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 3617, ...)
...
3617  exit(0)
3616  <... futex resumed>) = 0

The address 0x7ff8a9dff990 is exactly the child_tid pointer passed to clone3. CLONE_CHILD_CLEARTID told the kernel: when this task exits, zero that word and perform a futex wake on it. The parent then simply waits on that futex with the expected value 3617 (the child's TID).

So handle.join() compiles down to: sleep on a futex until the kernel clears the child's TID. There is no polling. There is no "thread finished" callback. It is one memory word and one kernel wait queue.

That single futex line is the seed of article 4. Mutex, Condvar, join, channel blocking, and Tokio's own parking all end up at the same syscall.

5. The child does more work than you'd expect before running your closure

3617  rseq(0x7ff8a9dfffe0, 0x20, 0, 0x53053053) = 0
3617  set_robust_list(0x7ff8a9dff9a0, 24) = 0
3617  mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff8a1a00000
3617  munmap(0x7ff8a1a00000, 39845888)  = 0
3617  munmap(0x7ff8a8000000, 27262976)  = 0
3617  mprotect(0x7ff8a4000000, 135168, PROT_READ|PROT_WRITE) = 0

rseq registers restartable sequences (used by glibc for fast per-CPU operations). set_robust_list registers the list the kernel walks if this thread dies while holding a robust mutex.

Then something bigger: a 128 MiB PROT_NONE reservation, two munmaps to trim it down to a 64 MiB aligned region, and an mprotect of 132 KiB.

That is glibc's malloc creating a new per-thread arena. The first time a new thread allocates, glibc may give it its own arena to avoid contention on the main arena's lock. The 128 MiB mmap is a reservation of address space, not memory — it is PROT_NONE, so not a single physical page is committed.

I did not expect this to show up in a program whose thread only calls println!. It matters, because it shows up in virtual memory accounting in a way that looks alarming and isn't — as the next section demonstrates.

Also note:

3617  exit(0)
3616  exit_group(0)

The thread exits with exit, which terminates one task. The main thread ends with exit_group, which terminates the whole thread group. Same distinction, visible in the trace.

What Linux Sees

A trace shows the transition. /proc shows the steady state. This program spawns four named threads and sleeps:

// four_threads.rs
use std::thread;
use std::time::Duration;

fn main() {
    let mut handles = Vec::new();
    for i in 0..4 {
        let h = thread::Builder::new()
            .name(format!("worker-{i}"))
            .spawn(move || {
                thread::sleep(Duration::from_secs(20));
                i
            })
            .unwrap();
        handles.push(h);
    }
    println!("pid = {}", std::process::id());
    for h in handles {
        h.join().unwrap();
    }
}

While it runs:

$ ls /proc/$PID/task
5481  5483  5484  5485  5486

$ for t in /proc/$PID/task/*; do
>   printf "%s comm=%-12s State=%s\n" "$(basename $t)" "$(cat $t/comm)" \
>     "$(awk '/^State:/{print $2,$3}' $t/status)"
> done
5481  comm=four_threads State=S (sleeping)
5483  comm=worker-0     State=S (sleeping)
5484  comm=worker-1     State=S (sleeping)
5485  comm=worker-2     State=S (sleeping)
5486  comm=worker-3     State=S (sleeping)

Five kernel tasks. Each has its own TID, its own state, its own scheduler accounting. thread::Builder::name() is not just a Rust-side label — it reaches /proc/<tid>/comm, which means it shows up in ps, top, perf, and every kernel-level tool. That is a free observability win that a lot of Rust code leaves on the table.

ps agrees:

$ ps -o pid,tid,psr,pcpu,stat,comm -L -p $PID
  PID   TID PSR %CPU STAT COMMAND
27548 27548   0  0.0 Sl   four_threads
27548 27549   1  0.0 Sl   worker-0
27548 27550   0  0.0 Sl   worker-1
27548 27551   1  0.0 Sl   worker-2
27548 27552   0  0.0 Sl   worker-3

One PID, five TIDs, and PSR shows the kernel has already spread them across both CPUs.

From a worker's own status file:

$ grep -E '^(Name|State|Tgid|Pid|Threads|Cpus_allowed_list)' /proc/$PID/task/27552/status
Name:   worker-3
State:  S (sleeping)
Tgid:   27548
Pid:    27552
Threads:    5
Cpus_allowed_list:  0-1

Pid: 27552, Tgid: 27548. Inside the kernel, "PID" means the task ID and "TGID" means what userspace calls the process ID. getpid() returns the TGID; gettid() returns the PID. This is why the trace showed gettid() — the Rust runtime wants the task identity, not the process identity.

Confirming `CLONE_VM`

The claim "threads share an address space" is testable:

$ diff <(cat /proc/$PID/task/$PID/maps) <(cat /proc/$PID/task/$TID/maps) \
    && echo "identical"
identical

Byte-identical memory maps for two different tasks. One mm_struct, five tasks pointing at it. That is CLONE_VM, observed rather than asserted.

The stacks, and their guard pages

Dumping the anonymous mappings with their neighbours:

7f3b0abfc000-7f3b0abfd000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0abfd000-7f3b0adfd000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

7f3b0adfd000-7f3b0adfe000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0adfe000-7f3b0affe000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

7f3b0affe000-7f3b0afff000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0afff000-7f3b0b1ff000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

7f3b0b1ff000-7f3b0b200000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0b200000-7f3b0b400000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

Four threads, four 2 MiB stacks, four 4 KiB ---p guard pages, laid out back to back. Note the accounting consequence: each thread costs two VMA entries, not one. That will matter when we look at limits.

The guard page, doing its job

A thread with a deliberately small stack and unbounded recursion:

// overflow.rs
use std::thread;

fn recurse(n: u64) -> u64 {
    let pad = [n; 1024];             // 8 KiB of stack per frame
    std::hint::black_box(&pad);
    if n == 0 { 0 } else { recurse(n - 1) + pad[0] }
}

fn main() {
    let h = thread::Builder::new()
        .name("deep".into())
        .stack_size(64 * 1024)       // deliberately small: 64 KiB
        .spawn(|| recurse(1_000_000))
        .unwrap();
    println!("join result: {:?}", h.join().is_err());
}

Running it:

$ ./overflow

thread 'deep' (27182) has overflowed its stack
fatal runtime error: stack overflow, aborting
Aborted
$ echo $?
134

And the signals, from strace:

[pid 27928] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7ff67a257cc0} ---
[pid 27928] --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=27927, si_uid=0} ---
[pid 27928] +++ killed by SIGABRT +++

SEGV_ACCERR — an access permission error, not SEGV_MAPERR. The address is mapped; the thread simply had no permission to touch it. That is the PROT_NONE guard page.

This also explains the sigaltstack calls that appear near the top of the trace for every thread:

mmap(NULL, 16048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0)
mprotect(0x7ff8aa0b9000, 4096, PROT_NONE)
sigaltstack({ss_sp=0x7ff8aa0ba000, ss_flags=0, ss_size=11952}, NULL)
rt_sigaction(SIGSEGV, {sa_handler=0x560dd7ce53a0, ..., sa_flags=...|SA_ONSTACK|SA_SIGINFO}, ...)

Rust installs a SIGSEGV handler with SA_ONSTACK and gives every thread a small alternate signal stack — with its own guard page. It has to: when you overflow a stack you cannot run a signal handler on that stack. Roughly 16 KiB per thread of extra mapping buys you a readable "has overflowed its stack" message instead of an unexplained segfault.

An engineering consequence that costs nothing to apply: the message names the thread. thread 'deep' (27182) has overflowed its stack is dramatically more useful in a production log than thread '<unnamed>'. Name your threads.

What a Thread Actually Costs

Now the part that changes how you design systems.

Virtual memory

This program spawns batches of parked threads and reads its own /proc/self/status at each step:

// cost.rs (excerpt)
for target in [0usize, 100, 500, 1000, 2000, 4000] {
    // ... spawn up to `target` live threads, wait until all are parked ...
    println!("{:>8} {:>12} {:>10} {:>8}",
             target, stat("VmSize:"), stat("VmRSS:"), stat("Threads:"));
}

Default run:

live-thread footprint (kB from /proc/self/status)
 threads       VmSize      VmRSS  Threads
       0        70856       2408        1
     100      1193108       3488      101
     500      2020572       7312      501
    1000      3054836      12108     1001
    2000      5123528      21688     2001
    4000      9260740      40868     4001

The jump from 0 to 100 threads is over a gigabyte of virtual memory, which is far more than 100 x 2 MiB. That is the malloc arena behaviour from the trace: glibc creates up to 8 * ncores arenas, each reserving 64 MiB of address space. Capping it isolates the stack cost:

$ MALLOC_ARENA_MAX=1 ./cost 200

 threads       VmSize      VmRSS  Threads
       0         5320       2404        1
     100       210200       3420      101
     500      1038060       7252      501
    1000      2072852      12044     1001
    2000      4142436      21624     2001
    4000      8281508      40784     4001

Now it is clean and linear: roughly 2,069 kB of virtual memory per thread (2 MiB stack + guard page + TLS + thread descriptor). 4,000 threads costs about 7.9 GiB of virtual address space.

The arena arithmetic checks out too. In the default run, 100 threads added 1,122,252 kB of VmSize; 100 x 2,069 kB of stacks accounts for 206,900 kB, leaving 915,352 kB, which is 14 x 65,536 kB. Fourteen 64 MiB arenas, against a ceiling of 8 * 2 cores = 16.

Two lessons, and the second is the one that actually costs people time:

Threads are expensive in virtual memory, cheap in resident memory. 4,000 threads cost 7.9 GiB of VmSize but only about 40 MiB of VmRSS — roughly 10 kB resident per thread. The stacks are mapped, not touched. Physical pages arrive on first write.
VmSize is not memory usage. If you alert on virtual size, a multi-threaded Rust service on glibc will page you for nothing. Alert on RSS, or on Committed_AS if you care about overcommit headroom.

Does stack size actually matter?

If RSS is what matters and stacks are lazily backed, does stack_size matter at all? Measured, 1,000 threads each, arenas capped:

stack_size=    16384  n=1000  VmSize delta=    49584 kB (   49 kB/thread)  VmRSS delta= 13712 kB ( 13 kB/thread)
stack_size=    65536  n=1000  VmSize delta=    85584 kB (   85 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)
stack_size=   262144  n=1000  VmSize delta=   277584 kB (  277 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)
stack_size=  2097152  n=1000  VmSize delta=  2069584 kB ( 2069 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)
stack_size=  8388608  n=1000  VmSize delta=  8213584 kB ( 8213 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)

VmSize tracks the requested stack almost exactly, with a constant overhead of about 21 kB per thread on top (guard page, thread control block, static TLS) — visible in every row from 64 KiB upward: 85 - 64 = 21, 277 - 256 = 21, 2069 - 2048 = 21, 8213 - 8192 = 21. The 16 KiB row breaks the pattern because the request is below the platform minimum and gets raised; that is the cmp::max(stack, min_stack_size(...)) we will see in the source. VmRSS is flat at roughly 9.7 kB per thread regardless of whether you asked for 64 KiB or 8 MiB of stack.

So: shrinking stacks buys you address space, not RAM. On 64-bit that is usually not the constraint you are fighting — which means stack_size tuning is mostly worth doing when you are hitting VMA or overcommit limits, not as a general memory optimisation. That is a different conclusion from the folklore, and it only shows up if you measure both numbers.

Time

Cost of a thread that does nothing at all — spawn immediately followed by join, 1,000 iterations:

spawn+join, 1000 iterations (nanoseconds)
  min    = 11206
  p50    = 47496
  p90    = 119084
  p99    = 230279
  max    = 505619
  mean   = 70174

A round trip of about 47 us at p50, with a p99 of 230 us and a worst case over half a millisecond.

Separating creation from teardown — 10,000 threads created with 64 KiB stacks, all created before any join:

creating 10000 threads (64 KiB stacks)
  spawn() call    p50 =    36569 ns
  spawn() call    p99 =   300113 ns
  spawn() call    max =  2687274 ns
  wall to create all  = 546.666318ms  (18293 threads/sec)
  wall to join all    = 43.088635ms

Roughly 18,000 threads/sec of creation throughput on two cores, with a p50 of 37 us for the spawn() call itself and a tail reaching 2.7 ms.

Put that next to a latency budget. If your API server's p99 target is 10 ms and you spawn a thread per request, creation alone consumes 0.4% of the budget at p50, 3% at p99, and 27% in the worst case observed — before your handler has read a single byte. At 20,000 requests/sec, thread creation alone saturates both cores.

That is the number that justifies article 5.

But the obvious fix is less obvious than it looks

I expected the "reuse a thread instead" comparison to be dramatic. It wasn't:

N = 20000
  spawn+join per item :     46675 ns   (total 933.507137ms)
  channel round-trip  :     37140 ns   (total 742.79602ms)
  ratio               :       1.3x

Handing work to an already-running thread over an mpsc channel and waiting for the reply is only 1.3x faster than creating a whole new OS thread for it.

That result is real, and it is more instructive than the one I was expecting. Both paths are dominated by the same thing: blocking, and being woken by the scheduler. A strict ping-pong handoff costs two context switches per item, and on a 2-core machine those wakeups cost roughly what a clone3 costs. Thread creation is not the expensive part in this shape of workload — blocking is.

Which reframes the lesson. The value of a thread pool is not "spawning is slow". It is that a pool lets many items be in flight across a fixed set of threads without a sleep/wake cycle per item. And it points straight at the async argument: the reason Tokio tasks are cheap is not primarily that they skip clone3 — it is that switching between tasks does not go through the kernel scheduler at all. Article 6 gets to test that claim against these exact numbers.

I am keeping this result in because it is the kind of thing that gets quietly dropped when a benchmark doesn't say what the author wanted it to.

Stressing the Design: More Threads Than Cores

This machine has 2 cores. What happens when 64 threads all want CPU?

Each thread does an identical fixed amount of CPU-bound work, and reports its own wall time and its own involuntary context switches, read from /proc/thread-self/status:

// oversub2.rs (excerpt)
let handles: Vec<_> = (0..t).map(|_| thread::spawn(move || {
    let t0 = Instant::now();
    let n0 = self_stat("nonvoluntary_ctxt_switches:");
    std::hint::black_box(burn(WORK));
    (t0.elapsed().as_millis() as u64,
     self_stat("nonvoluntary_ctxt_switches:") - n0)
})).collect();

 threads     min_ms     p50_ms     max_ms     total_ms    nonvol_ctxt
       1        372        372        372          372              3
       2        373        380        380          380             46
       4        654        755        759          760            369
       8       1494       1506       1511         1517            819
      16       2955       2981       3014         3015           1739
      32       5711       5928       6035         6047           3557
      64      11763      12023      12130        12139           6942

Read this carefully, because the interesting result is what didn't happen.

Throughput is preserved. 64 threads each doing 372 ms of work is 23.8 seconds of CPU work, completed in 12.1 s of wall time on 2 cores — a speedup of about 1.96x, which is essentially both cores fully busy. The scheduler did not fall over. There is no throughput collapse from oversubscription here.

Latency is destroyed. The same unit of work takes 372 ms when it has a core to itself and 12,023 ms when it is one of 64 threads sharing two cores. That is 32x. If that unit of work is a request, your p50 just moved by a factor of 32 while your dashboards show healthy CPU utilization and healthy throughput.

Involuntary context switches scale linearly. 6,942 involuntary switches across 64 threads is about 108 preemptions per thread. Each thread consumed roughly 372 ms of actual CPU time, so it was preempted after about 3.4 ms of CPU each time. That is the scheduler doing exactly what it was designed to do — and 6,942 switches is 6,942 register saves, TLB and cache disturbances, and runqueue operations that the 1-thread run did not pay for.

The engineering consequence: oversubscription is a latency problem, not a throughput problem. Adding threads past core count does not get more work done; it spreads the same work over more concurrent, slower units. For a batch job that is harmless. For a request/response service it converts a fast p50 into a uniformly slow one — and it does so without tripping any of the signals people usually watch. Neither CPU utilization nor throughput will tell you. Only per-request latency will.

This is also why "just add more threads" fails for a service that is slow because it is CPU-bound, and works for a service that is slow because it is blocking on I/O. Those two look identical on a throughput graph and are opposite problems.

Where the ceiling actually is

The limits on this box:

threads-max:    64113        # /proc/sys/kernel/threads-max
pid_max:        32768        # /proc/sys/kernel/pid_max
max_map_count:  65530        # /proc/sys/vm/max_map_count
RLIMIT_STACK:   8192 kB
max user processes (-u): 32056
MemTotal:       8216168 kB
CommitLimit:    4108084 kB

The one people forget is max_map_count. We measured that each thread costs two VMAs — stack plus guard page. At 65,530 mappings that caps you near ~32,000 threads from mapping pressure alone, which happens to land in the same range as pid_max. You will typically hit a limit like this, or CommitLimit, well before you run out of RAM.

So "how many threads can I create" has at least four independent answers — threads-max, pid_max, RLIMIT_NPROC, max_map_count — plus overcommit policy. None of them is the number you would guess from RSS.

Reading the Source: How the Call Gets There

The trace tells us where we ended up. The source tells us how.

std::thread::spawn is a thin wrapper over Builder, which eventually calls the platform implementation. On Unix that is Thread::new, in the standard library's sys/thread/unix.rs:

pub unsafe fn new(stack: usize, init: Box<ThreadInit>) -> io::Result<Thread> {
    let data = init;
    let mut attr: mem::MaybeUninit<libc::pthread_attr_t> = mem::MaybeUninit::uninit();

    assert_eq!(libc::pthread_attr_init(attr.as_mut_ptr()), 0);
    let mut attr = DropGuard::new(&mut attr, |attr| {
        assert_eq!(libc::pthread_attr_destroy(attr.as_mut_ptr()), 0)
    });
    // ...
    let stack_size = cmp::max(stack, min_stack_size(attr.as_ptr()));
    match libc::pthread_attr_setstacksize(attr.as_mut_ptr(), stack_size) {
        0 => {}
        n => {
            assert_eq!(n, libc::EINVAL);
            // Round up to nearest page on alignment failure
            let page_size = sys::pal::conf::page_size();
            let stack_size = (stack_size + page_size - 1) &
                            (-(page_size as isize - 1) as usize - 1);
            // ...
        }
    }
    // ... libc::pthread_create(...)
}

That is the whole story in one function. There is no Rust-specific thread machinery in the kernel path. Rust:

picks a stack size — max(requested, platform minimum), defaulting to DEFAULT_MIN_STACK_SIZE = 2 * 1024 * 1024,
sets it on a pthread_attr_t,
calls pthread_create,
and lets glibc do the mmap + mprotect + clone3 we watched in the trace.

The closure is boxed and handed through as the thread's argument; the JoinHandle wraps the resulting thread handle plus a slot for the return value. join() ends in the futex wait we already saw.

The chain, complete:

std::thread::spawn
  -> std::thread::Builder::spawn_unchecked_
    -> sys::thread::Thread::new
      -> pthread_attr_init / pthread_attr_setstacksize
        -> pthread_create                        (glibc)
          -> mmap(PROT_NONE) + mprotect          (stack + guard page)
            -> clone3(CLONE_VM|CLONE_THREAD|...) (kernel)
              -> new task_struct, scheduled independently

Every layer in that chain is one we can now name, observe, and measure. Nothing in it is magic. And nothing in it is free.

Production Connection

Bringing this back to systems people actually operate.

API servers. Thread-per-request costs about 37 us at p50 and up to 2.7 ms at the tail before your handler runs. Once concurrent requests exceed core count, per-request latency degrades proportionally while throughput and CPU graphs stay flat. This is the specific, measurable reason the industry moved to pools and then to async — not fashion.

Databases and caches. Connection-per-thread designs hit max_map_count and CommitLimit before they hit RAM. A 2 MiB default stack times 10,000 connections is 20 GiB of address space for maybe 100 MiB of actual use. Knowing that VmSize is address space and VmRSS is memory is the difference between capacity planning and guessing.

Message brokers and Kafka-style consumers. The backpressure chain from the introduction — queue growth, memory growth, scheduler pressure, latency spikes — is exactly the oversubscription curve above. When a consumer falls behind and something spawns more workers to catch up, you get the 64-thread row: same throughput, 32x the latency.

Low-latency and trading systems. The tail matters more than the mean, and the tail here is involuntary preemption. 108 preemptions per thread over 12 s is invisible in an average and fatal in a p99.9. This is why such systems pin threads to cores, keep runnable threads at or below core count, and never create threads on the hot path.

Blockchain nodes, proxies, WebSocket servers, vector databases, AI infrastructure. All of them are "many concurrent connections, mostly waiting". That workload is the worst possible fit for thread-per-connection: you pay full thread cost for entities that are idle almost all the time. That mismatch is precisely the gap async runtimes exist to fill — and article 6 will measure whether they actually do.

Everywhere. Name your threads. It costs one Builder::name() call, and it puts a meaningful string into /proc/<tid>/comm, ps, top, perf, and your crash messages.

Conclusion

The hypothesis was: Rust creates a native OS thread that Linux independently tracks and schedules. That held up. The useful part was everything the investigation added around it.

What we established, all of it observed rather than assumed:

There is no thread syscall. clone3 creates a task; the sharing flags — CLONE_VM, CLONE_THREAD, CLONE_FILES, CLONE_SIGHAND — are the entire difference between a thread and a process.
The stack is allocated in userspace by glibc before the kernel is involved: a PROT_NONE mmap of 2 MiB + 4 KiB, then mprotect to open everything except the guard page.
2 MiB is Rust's choice, from DEFAULT_MIN_STACK_SIZE, overridable via RUST_MIN_STACK or Builder::stack_size — confirmed by watching the mmap size change.
join() is a futex wait, armed by CLONE_CHILD_CLEARTID. The kernel clears the child's TID on exit and wakes the waiter.
A thread costs about 2,069 kB of virtual memory and about 10 kB of resident memory. Those two numbers are two orders of magnitude apart, and confusing them is a common operational error.
A thread costs about 37 us to create at p50, with a 2.7 ms tail, at roughly 18,000 threads/sec on two cores.
Beyond core count, throughput holds and latency degrades linearly — 32x for 64 threads on 2 cores — with involuntary context switches scaling to match.
The practical ceiling on thread count comes from max_map_count, pid_max, RLIMIT_NPROC and overcommit, not from RAM. Each thread burns two VMAs.

And one thing I had wrong going in: I assumed the expensive part of thread-per-work was creation. The reuse benchmark says the expensive part is blocking and waking. Creation is merely expensive as well. That distinction is going to shape most of the rest of this series.

Article 2: What Does Arc Actually Do? Atomic Reference Counting Under the Hood. We now have multiple tasks sharing one mm_struct — byte-identical /proc/<tid>/maps, confirmed above. The immediate question is how ownership of anything inside that shared address space is tracked safely. That means looking at what Arc actually contains in memory, what an atomic increment compiles to on x86-64, why the Drop path needs a stronger ordering than the clone path, and what the counter does under contention.

Then article 3 puts a lock around it and measures what contention costs, and article 4 follows the blocking path back to the same futex syscall we already saw here.

Appendix: Environment and Reproducibility

Every number in this article came from this machine:

kernel:  6.18.5-fc-v20 (x86_64)
distro:  Ubuntu 24.04.4 LTS
rustc:   1.95.0 (59807616e 2026-04-14)
glibc:   2.39-0ubuntu8.7
strace:  6.8
cpu:     Intel(R) Xeon(R) Processor @ 2.10GHz, 2 cores
memory:  8216168 kB total
RLIMIT_STACK: 8192 kB

Programs used, all single-file, all buildable with rustc -O:

File	Purpose
`hello_thread.rs`	minimal one-thread program, for the syscall trace
`four_threads.rs`	four named parked threads, for `/proc` inspection
`overflow.rs`	deliberate stack overflow, for the guard page
`cost.rs`	spawn+join latency percentiles and live-thread footprint
`stacksize.rs`	`VmSize`/`VmRSS` per thread across stack sizes
`createcost.rs`	creation-only latency and creation throughput
`reuse.rs`	spawn-per-item vs. channel handoff
`oversub2.rs`	per-thread latency and preemptions vs. thread count

Commands:

$ rustc -O -g hello_thread.rs -o hello_thread
$ strace -f -c ./hello_thread
$ strace -f -o hello_full.strace ./hello_thread
$ strace -f -e trace=none ./overflow          # signals only
$ MALLOC_ARENA_MAX=1 ./cost 200
$ MALLOC_ARENA_MAX=1 ./oversub2

Caveats I want to be explicit about, because they change the numbers:

Two cores. The oversubscription curve is steeper here than on a 32-core host, and absolute spawn latency is higher. The shape transfers; the values do not.
Virtualised environment. Syscall and context-switch costs under a hypervisor differ from bare metal.
glibc-specific. The malloc arena behaviour — those 128 MiB PROT_NONE reservations — is a glibc implementation detail. musl behaves differently, and a Rust binary using a different global allocator will not show it at all.
Instant::now() around spawn includes timing-call overhead. At tens of microseconds that is negligible, but it is not zero.
/proc/self/status counters are per-thread, not per-process. My first oversubscription run read them from the main thread and reported almost no context switches, which is why the final version reads /proc/thread-self/status from inside each worker. Worth knowing before you trust a similar measurement of your own.

If you re-run these on a machine with more cores, I would be interested in where the latency curve bends.

Thanks for Reading

If you got this far — thank you. That was a long way down for one line of Rust.

This series is me working through layers I have used for years without ever inspecting, and writing it up as I go rather than after I have it all figured out. That means the investigations are honest about what surprised me, and it means some of them will be wrong. If you spot something wrong here, I would much rather hear it now than leave it standing while nine more articles get built on top of it.

Next up — Article 2: What Does Arc Actually Do? Atomic Reference Counting Under the Hood.

GitHub: GITHUB_URL
LinkedIn: LINKEDIN_URL

Building a RAG System in Rust with Qdrant, Rig, and gRPC 🦀

Parikalp Bhardwaj — Sat, 30 May 2026 21:01:01 +0000

With Qdrant, Rig, Tonic — and a healthy obsession with what's actually happening underneath.

Why I Built This

A few weeks ago I came across Our First Production-Ready RAG Dev Journey in Pure Rust by the rust-dd team — and something clicked.

Reading it, I realized two things:

Rust is the right language for building AI systems — fast, safe, and built for the kind of infrastructure work AI actually needs.
I'd been wanting to build something like this myself for months and kept finding excuses not to start.

So I started. This post is what came out of it. 🦀

Most RAG tutorials hand you a framework, four function calls, and a working demo. You upload some documents, vectors get generated somewhere, an LLM answers your questions, and you walk away with a chatbot but no real understanding of what just happened.

This post takes the opposite approach. We'll build a small but complete RAG system in Rust using Qdrant for vector search, Rig as the AI framework, and Tonic for gRPC API, and a terminal chat client — and explain why every piece exists. By the end you'll understand:

🧠 Why embeddings work and how semantic retrieval actually functions
🗄️ Why vector databases exist and what problem ANN indexing solves
⚡ Why async runtimes matter for retrieval pipelines
🦀 Why Rust is becoming genuinely interesting for AI infrastructure
🔌 How gRPC fits into modern AI service architectures

The full source is at github.com/Parikalp-Bhardwaj/qrag-rust. Clone it and read along.

🤔 Why RAG exists

LLMs are powerful, but they don't actually know anything about your data. They generate from patterns learned during training, which means:

🚫 Hallucinations when asked about specifics they never saw
📅 Outdated information — anything after the cutoff is invisible
🔒 No access to private knowledge — your docs, your codebase, your wiki
📏 Limited context windows — you can't paste everything in

A model trained in 2023 can't tell you what your internal API does or what changed in last week's release. That's not a model problem — it's a retrieval problem.

RAG fixes it by inserting a retrieval step before generation:

query
  → retrieve relevant context from your data
    → inject context into the prompt
      → generate a grounded response

This shifts the engineering focus. The LLM becomes one component among several, and the quality of the system depends on chunking strategy, embedding quality, retrieval accuracy, indexing, and latency. That's the territory most tutorials skip. We're going to live there.

🧩 The building blocks

Four pieces do most of the work in this project.

🔎 Qdrant — the vector database

In a normal database you search by exact values:

SELECT * FROM documents WHERE title = 'Rust';

That works for keywords. It doesn't work for meaning. Consider:

Query: "How does Rust prevent race conditions?"

Relevant doc: "Rust provides memory safety and fearless concurrency through ownership and borrowing."

Zero word overlap. A SQL LIKE won't find this. But a human reading both knows they're talking about the same thing.

Qdrant stores documents as vectors — numerical fingerprints of meaning. Two semantically similar texts produce two vectors that sit close together in high-dimensional space. Search becomes "find the vectors nearest to my query vector." A Qdrant point looks like:

Point {
  id: "doc-1",
  vector: [0.12, -0.44, 0.91, ...],   // 1536 dimensions
  payload: {
    "file_path": "rust-notes.md",
    "chunk_index": 3,
    "text": "Rust ownership prevents memory bugs..."
  }
}

Why a dedicated database? Naively, retrieval means comparing the query vector against every stored vector — O(n) per query. Fine for 100 documents, a disaster for 10 million. Qdrant uses Approximate Nearest Neighbor (ANN) indexing (HNSW under the hood) to find near matches without scanning everything. You trade a sliver of accuracy for a dramatic speedup. That's the entire reason vector databases exist as a category.

🦀 Rig — the AI application layer

Rig is a Rust framework for LLM apps. It handles the boring, provider-specific parts: embedding APIs, completion APIs, vector store glue, agent construction. Without Rig we'd be hand-writing HTTP clients for OpenRouter.

⚠️ Honest caveat: Rig reduces boilerplate, but it doesn't think for you. Chunking strategy, retrieval ranking, prompt construction — those are still your problem and they're what actually determine answer quality.

🚀 gRPC + Tonic — the service layer

gRPC is a high-performance RPC framework. Instead of POST /chat with JSON, you define services in Protocol Buffers and get strongly-typed clients and servers in any supported language. Tonic is the Rust implementation.

Why gRPC over REST? Two reasons:

Typed contracts. The .proto file is the single source of truth. Client and server can't drift apart.
Backend-to-backend fit. Real AI infrastructure looks like frontend → API gateway → RAG service → vector DB → LLM provider. gRPC is built for the internal hops.

⚡ Tokio — the async runtime

Every network call here is async — Qdrant queries, OpenRouter embeddings, LLM completions. Tokio lets thousands of them run concurrently on a small thread pool without us managing threads by hand. It's the quiet substrate underneath everything else.

🏗️ Architecture at a glance

The system has two phases that share one RagEngine.

Indexing (runs once, or whenever docs change):

./docs → load → chunk → embed (OpenRouter) → store (Qdrant)

Query (runs every time someone asks):

question → embed → search Qdrant → top-k chunks → prompt → LLM → answer

Same embedding model on both sides — that's what makes the geometry work. You can't embed documents with one model and queries with another and expect distances to mean anything.

📂 Project layout

qrag-rust/
├── Cargo.toml               # crate metadata + dependencies
├── build.rs                 # compiles .proto → Rust at build time
├── docker-compose.yaml      # Qdrant container
├── .env.example             # template for secrets
├── proto/
│   └── rag.proto            # gRPC service definition
├── docs/                    # your knowledge base lives here
│   ├── grpc.md
│   ├── rust.md
│   └── tokio.md
|   └── Rust-for-Network-Programming-and-Automation.pdf
└── src/
    ├── main.rs              # boots the gRPC server
    ├── config.rs            # env-var configuration
    ├── document_loader.rs   # read .md, .txt, .pdf
    ├── chunker.rs           # split into ~120-word chunks
    ├── qdrant_store.rs      # embeddings + vector storage
    ├── llm.rs               # prompt + completion
    ├── rag.rs               # orchestration
    ├── grpc_service.rs      # tonic handlers
    └── bin/
        └── chat.rs          # terminal chat client

Each file has one job. That's deliberate — RAG systems get complicated quickly, and clean seams are the only defense.

🛠️ Prerequisites

Rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
rustc --version

System packages

A surprising number of Rust crates compile native C/C++ underneath. On Ubuntu:

sudo apt update
sudo apt install -y \
    build-essential pkg-config libssl-dev \
    clang cmake protobuf-compiler poppler-utils

Package	Why you need it
`build-essential`	GCC/G++ for native compilation
`pkg-config`	Locates system libraries
`libssl-dev`	OpenSSL headers — any HTTPS crate needs them
`clang`	LLVM toolchain (bindgen, ML runtime crates)
`cmake`	Used by several native ML libraries
`protobuf-compiler`	The `protoc` binary that turns `.proto` into Rust
`poppler-utils`	Provides `pdftotext` — we shell out to it for PDFs

Verify protoc:

protoc --version

Docker + Qdrant

sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
docker compose up -d   # uses the docker-compose.yaml in the repo

Port	What it serves
`6333`	REST API + dashboard at `http://localhost:6333/dashboard`
`6334`	gRPC API — what our Rust client talks to

OpenRouter key

We use OpenRouter so the same key serves both the embedding model and the completion model. Create a key, then:

🔐 Setting up `.env` and Qdrant

Before you can run anything, two things need to be in place: a .env file with your API key, and a running Qdrant container.

1. Create the `.env` file

In the project root, create a file named .env (note the leading dot — it's a hidden file):

# Required — your OpenRouter API key
OPENROUTER_API_KEY=sk-or-v1-paste-your-key-here

# Optional — defaults shown
SERVER_ADDR=127.0.0.1
PORT=50051
QDRANT_URL=http://127.0.0.1:6334
QDRANT_COLLECTION=question

Get a key at openrouter.ai/keys. Free tier is enough to test.

What each variable does:

Variable	Default	What it controls
`OPENROUTER_API_KEY`	(required)	Authenticates both embedding and LLM calls
`SERVER_ADDR`	`127.0.0.1`	Host the gRPC server binds to (use `0.0.0.0` for Docker)
`PORT`	`50051`	gRPC port your clients connect to
`QDRANT_URL`	`http://127.0.0.1:6334`	Where Qdrant's gRPC endpoint lives
`QDRANT_COLLECTION`	`question`	Name of the collection that stores your vectors

⚠️ Never commit .env to git. Add it to .gitignore:

echo ".env" >> .gitignore

If you've already committed it once, rotate the key — git history doesn't forget. Ship a .env.example with empty values instead so collaborators know what to set.

2. Start Qdrant with Docker

The repo includes a docker-compose.yaml:

services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: coderag-qdrant
    ports:
      - "6333:6333"   # REST API + dashboard
      - "6334:6334"   # gRPC API (this is what the Rust app uses)
    volumes:
      - qdrant_data:/qdrant/storage

volumes:
  qdrant_data:

Start it:

docker compose up -d

The -d flag runs it in the background. Verify it's up:

docker ps
# you should see coderag-qdrant in the list

curl http://localhost:6333/healthz
# should return: healthz check passed

Open the dashboard in a browser:

📖 Walking through every file

This is the part most tutorials skip. We're going file by file, top to bottom, explaining what each one does and why it's structured the way it is.

`Cargo.toml` — the manifest

[package]
name = "qrag-rust"
version = "0.1.0"
edition = "2024"

[[bin]]
name = "qrag-rust"
path = "src/main.rs"

[[bin]]
name = "chat"
path = "src/bin/chat.rs"

[dependencies]
anyhow = "1"
rig = "0.37.0"
tokio = { version = "1", features = ["full"] }
tonic = "0.12"
prost = "0.13"
rig-qdrant = "0.2"
qdrant-client = "1"
dotenvy = "0.15"
uuid = { version = "1", features = ["v4"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
tracing-subscriber = "0.3"

[build-dependencies]
tonic-build = "0.12"

The two [[bin]] entries are the important part. By default Cargo finds src/main.rs automatically, but the moment you add a binary under src/bin/, Cargo stops auto-discovering and you have to declare both explicitly. One is the server, one is the chat client. Both end up in target/release/ after cargo build --release.

tonic-build is a build-dependency, not a runtime dependency. It only runs at compile time to turn .proto files into Rust code.

`build.rs` — code generation at compile time

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::compile_protos("proto/rag.proto")?;
    Ok(())
}

Three lines, big effect. Every time you run cargo build, this script runs first. It reads proto/rag.proto, generates Rust structs and traits for every message and service, and writes them to Cargo's OUT_DIR. Later, tonic::include_proto!("rag") in main.rs pulls that generated code into our crate.

This is why you never check generated proto code into git — it's regenerated on every build.

`proto/rag.proto` — the service contract

syntax = "proto3";

package rag;

service RagService {
  rpc AskQuestion(AskQuestionRequest) returns (AskQuestionResponse);
  rpc Reindex(ReindexRequest) returns (ReindexResponse);
}

message AskQuestionRequest { string question = 1; }
message AskQuestionResponse {
  string answer = 1;
  repeated Source sources = 2;
}
message Source {
  string file_path = 1;
  uint32 chunks_indexed = 2;
  string preview = 3;
  float score = 4;
}
message ReindexRequest {}
message ReindexResponse {
  uint64 chunks_indexed = 1;
  string message = 2;
}

This file is the single source of truth for the API. Two RPCs: Reindex rebuilds the vector index from ./docs, and AskQuestion is the actual query endpoint. The response carries the answer plus the source chunks that produced it — that's the difference between a RAG system and a black box. Always return your sources so users can verify the grounding.

The numbers (= 1, = 2) are field tags. They're how protobuf identifies fields on the wire. Once assigned, never change them — that's an instant breaking change for every existing client.

`src/config.rs` — environment-driven config

use std::env;

#[derive(Debug, Clone)]
pub struct Config {
    pub addr: String,
    pub port: u16,
    pub qdrant_url: String,
    pub qdrant_collection: String,
}

impl Config {
    pub fn from_env() -> Self {
        Self {
            addr: env::var("SERVER_ADDR")
                .unwrap_or_else(|_| "127.0.0.1".to_string()),
            port: env::var("PORT")
                .unwrap_or_else(|_| "50051".to_string())
                .parse::<u16>()
                .expect("PORT must be a valid number"),
            qdrant_url: env::var("QDRANT_URL")
                .unwrap_or_else(|_| "http://127.0.0.1:6334".to_string()),
            qdrant_collection: env::var("QDRANT_COLLECTION")
                .unwrap_or_else(|_| "question".to_string()),
        }
    }

    pub fn server_addr(&self) -> String {
        format!("{}:{}", self.addr, self.port)
    }
}

Plain config struct, populated from environment variables with sensible defaults. Each setting is documented by its name and its default. SERVER_ADDR and PORT configure where the gRPC server binds. QDRANT_URL points to the Qdrant gRPC port (note: 6334, not the REST 6333). QDRANT_COLLECTION names the collection.

Defaults work out of the box for local dev. Production overrides come from .env or the deployment environment.

`src/document_loader.rs` — reading files from disk

use anyhow::{Context, Result};
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;

#[derive(Debug, Clone)]
pub struct Document {
    pub file_path: String,
    pub content: String,
}

pub fn loader_documents_from_dir<P: AsRef<Path>>(dir: P) -> Result<Vec<Document>> {
    let mut documents = Vec::new();
    let entries = fs::read_dir(&dir)
        .with_context(|| format!("Failed to read directory: {:?}", dir.as_ref()))?;

    for entry in entries {
        let entry = entry?;
        let path: PathBuf = entry.path();
        if !path.is_file() { continue; }
        let Some(extension) = path.extension() else { continue; };
        let extension = extension.to_string_lossy().to_lowercase();

        let content = match extension.as_str() {
            "md" | "text" => fs::read_to_string(&path)
                .with_context(|| format!("Failed to read text file: {:?}", path))?,
            "pdf" => extract_pdf_text(&path)
                .with_context(|| format!("Failed to extract text from PDF: {:?}", path))?,
            _ => continue,
        };

        documents.push(Document {
            file_path: path.to_string_lossy().to_string(),
            content,
        });
    }
    Ok(documents)
}

fn extract_pdf_text(path: &Path) -> Result<String> {
    let output = Command::new("pdftotext")
        .arg(path).arg("-")
        .output()
        .with_context(|| format!("Failed to run pdftotext for {:?}", path))?;

    if !output.status.success() {
        anyhow::bail!("pdftotext failed for {:?}: {}",
            path, String::from_utf8_lossy(&output.stderr));
    }
    Ok(String::from_utf8_lossy(&output.stdout).to_string())
}

This is the on-ramp. The loader walks ./docs, skips directories and unknown extensions, reads each supported file, and returns a Vec<Document> of (path, content) pairs.

Two design choices worth flagging. <P: AsRef<Path>> is a generic over anything that can become a Path — &str, String, PathBuf, &Path. Callers can pass whatever they have without converting. PDF extraction shells out to pdftotext (from poppler-utils) rather than pulling a heavyweight PDF crate. Not glamorous, but it's been battle-tested for years and avoids a 500-line dependency.

`src/chunker.rs` — splitting documents into pieces

use crate::document_loader::Document;

#[derive(Debug, Clone)]
pub struct DocumentChunk {
    pub id: String,
    pub file_path: String,
    pub chunk_index: usize,
    pub text: String,
}

pub fn chunk_retrieve(documents: Vec<Document>, max_word: usize) -> Vec<DocumentChunk> {
    let mut chunks = Vec::new();
    for docs in documents {
        let words: Vec<&str> = docs.content.split_whitespace().collect();
        for (chunk_idx, word_chunk) in words.chunks(max_word).enumerate() {
            let text = word_chunk.join(" ");
            if text.trim().is_empty() { continue; }
            chunks.push(DocumentChunk {
                id: uuid::Uuid::new_v4().to_string(),
                file_path: docs.file_path.clone(),
                chunk_index: chunk_idx,
                text,
            });
        }
    }
    chunks
}

The single most underrated decision in RAG. Too large and retrieval becomes vague (the chunk has the answer plus a lot of noise); too small and meaning fragments across boundaries. We use a simple word-count chunker at 120 words per chunk — a sweet spot for technical docs.

Each chunk gets a fresh UUID (Qdrant uses it as the point ID), the source file path, and a chunk_index so we can show users where the answer came from. split_whitespace() is a small but important choice — it handles tabs, newlines, and runs of spaces correctly, where split(' ') would not.

Production systems often do smarter things: sentence-boundary chunking, semantic chunking (split where meaning shifts), or sliding-window chunks with overlap so context isn't lost at boundaries. All worth experimenting with once your baseline works.

`src/qdrant_store.rs` — the heart of retrieval

This is the longest file in the project. It does three things: create or reset the Qdrant collection, embed chunks and upsert them, and search by question. Let's break it into pieces.

The struct and embedding model:

const EMBEDDING_MODEL: &str = "openai/text-embedding-3-small";

#[derive(Clone)]
pub struct QdrantStore {
    client: Qdrant,
    openrouter_client: OpenRouterAiClient,
    collection_name: String,
}

#[derive(Embed, Clone)]
struct ChunkEmbedding {
    id: String,
    file_path: String,
    chunk_index: usize,
    #[embed]
    text: String,
}

QdrantStore owns two clients: one to Qdrant, one to OpenRouter for embeddings. ChunkEmbedding is a Rig pattern — the #[derive(Embed)] macro plus the #[embed] field attribute tells Rig "embed the text field; everything else is metadata that rides along."

Constructing the store:

pub fn new(qdrant_url: &str, collection_name: &str) -> Result<Self, anyhow::Error> {
    dotenvy::dotenv().ok();

    let client = Qdrant::from_url(qdrant_url)
        .build()
        .context("Failed to create Qdrant client")?;

    let openrouter_client = OpenRouterAiClient::from_env()?;

    Ok(Self {
        client,
        openrouter_client,
        collection_name: collection_name.to_string(),
    })
}

The constructor wires up both clients in one place. Qdrant::from_url(...) builds the gRPC client pointed at http://127.0.0.1:6334 (or wherever QDRANT_URL says). OpenRouterAiClient::from_env() reads OPENROUTER_API_KEY straight from the environment — that's why dotenvy::dotenv().ok() runs first, so values from your .env file are loaded before Rig looks for them.

Both clients are cheap to clone (they wrap connection pools internally), which is why we can hand QdrantStore around freely via #[derive(Clone)] later.

Why pass qdrant_url and collection_name as &str instead of taking a Config struct? Loose coupling. QdrantStore doesn't need to know about the rest of the app's configuration.

Collection creation:

pub async fn ensure_collection(&self) -> Result<()> {
    let exists = self.client.collection_exists(&self.collection_name).await?;
    if exists { return Ok(()); }

    self.client.create_collection(
        CreateCollectionBuilder::new(&self.collection_name)
            .vectors_config(VectorParamsBuilder::new(1536, Distance::Cosine))
    ).await?;
    Ok(())
}

1536 matches openai/text-embedding-3-small. Get this wrong and every upsert fails with a dimension mismatch — a mistake everyone makes once. Cosine distance is the standard choice for normalized text embeddings.

reset_collection (omitted for space — see repo) deletes and recreates. It's what Reindex calls to start fresh.

Embedding and upserting chunks:

pub async fn upsert_chunks(&self, chunks: &[DocumentChunk]) -> Result<usize> {
    if chunks.is_empty() { return Ok(0); }

    let embedding_model = self.openrouter_client.embedding_model(EMBEDDING_MODEL);
    let mut builder = EmbeddingsBuilder::new(embedding_model);

    for chunk in chunks {
        builder = builder.document(ChunkEmbedding {
            id: chunk.id.clone(),
            file_path: chunk.file_path.clone(),
            chunk_index: chunk.chunk_index,
            text: chunk.text.clone(),
        })?;
    }

    let embedded_doc = builder.build().await?;

    let points: Vec<PointStruct> = embedded_doc.into_iter().map(|(doc, embeddings)| {
        let vector: Vec<f32> = embeddings.first().vec
            .iter().map(|&v| v as f32).collect();

        PointStruct::new(
            doc.id.clone(),
            vector,
            Payload::try_from(json!({
                "file_path": doc.file_path,
                "chunk_index": doc.chunk_index,
                "text": doc.text,
            })).expect("valid Qdrant payload"),
        )
    }).collect();

    self.client.upsert_points(
        UpsertPointsBuilder::new(&self.collection_name, points)
    ).await?;

    Ok(chunks.len())
}

The flow: build up a batch of ChunkEmbeddings, hand them to Rig's EmbeddingsBuilder, get back (doc, embeddings) pairs. We cast f64 → f32 because Qdrant expects 32-bit floats. The payload carries the original text and metadata — that's what search will later return to us so we can hand it to the LLM.

Searching:

pub async fn search(&self, question: &str, top_k: usize) -> Result<Vec<RetrievedChunk>> {
    let embedding_model = self.openrouter_client.embedding_model(EMBEDDING_MODEL);

    let query = QueryPointsBuilder::new(&self.collection_name)
        .with_payload(true).limit(top_k as u64).build();

    let vector_store = QdrantVectorStore::new(
        self.client.clone(), embedding_model, query);

    let request = VectorSearchRequest::builder()
        .query(question).samples(top_k as u64).build();

    let results = vector_store.top_n::<serde_json::Value>(request).await?;

    let mut chunk = Vec::new();
    for (score, _id, payload) in results {
        let file_path = payload.get("file_path")
            .and_then(|v| v.as_str()).unwrap_or_default().to_string();
        let chunk_index = payload.get("chunk_index")
            .and_then(|v| v.as_u64()).unwrap_or_default() as usize;
        let text = payload.get("text")
            .and_then(|v| v.as_str()).unwrap_or_default().to_string();

        chunk.push(RetrievedChunk {
            file_path, chunk_index, text, score: score as f32
        });
    }
    Ok(chunk)
}

Rig handles embedding the question internally — that's what vector_store.top_n(...) does under the hood. Results come back as (score, id, payload) tuples. We destructure them and pull file_path, chunk_index, and text out of the JSON payload with safe defaults so a malformed point can't crash the whole query.

`src/llm.rs` — building the prompt and calling the model

#[derive(Clone)]
pub struct LlmService {
    client: openrouter::Client,
}

impl LlmService {
    pub fn new() -> Result<Self> {
        let client = openrouter::Client::from_env()
            .context("Failed to create Rig OpenRouter client from OPENROUTER_API_KEY")?;
        Ok(Self { client })
    }

    pub async fn answer_question(
        &self,
        question: &str,
        chunks: &[RetrievedChunk],
    ) -> Result<String> {
        if chunks.is_empty() {
            return Ok(format!("could not find relevant content for: {}", question));
        }

        let context = build_context(chunks);

        let prompt = format!(r#"
            You are a helpful Rust AI assistant.
            Answer the question using only the provided document context.

            Rules:
            - Be clear and concise.
            - If the context is not enough, say so.
            - Mention the source file when useful.
            - Do not invent facts outside the context.

            Context:
            {}

            Question:
            {}

            Answer:
            "#, context, question);

        let agent = self.client
            .agent("openai/gpt-4o-mini")
            .preamble("You answer questions using retrieved chunks as grounded context.")
            .build();

        let response = agent.prompt(prompt).await
            .context("Failed to generate answer with Rig agent")?;
        Ok(response)
    }
}

fn build_context(chunks: &[RetrievedChunk]) -> String {
    let mut context = String::new();
    for chunk in chunks {
        context.push_str(&format!(
            "\nSource: {}\nChunk: {}\nScore: {:.4}\nText: {}\n",
            chunk.file_path, chunk.chunk_index, chunk.score, chunk.text
        ));
    }
    context
}

This is where retrieval finally meets generation. build_context formats each retrieved chunk into a labeled block so the model can tell them apart and cite them. The main prompt drops that block in, asks the question, and adds explicit grounding rules.

The "Do not invent facts outside the context" line matters more than people think. Without it, models cheerfully fill gaps with training-data plausibilities — which is exactly the hallucination problem RAG is supposed to fix. The "If the context is not enough, say so" rule is equally important: it tells the model that "I don't know" is an acceptable answer.

If retrieval returns nothing (empty chunks), we short-circuit and never call the LLM — saves a token bill and produces a clearer "no relevant content found" message.

`src/rag.rs` — orchestration

#[derive(Clone)]
pub struct RagEngine {
    file_dir: String,
    qdrant_store: QdrantStore,
    llm: LlmService,
}

#[derive(Debug)]
pub struct RagAnswer {
    pub answer: String,
    pub sources: Vec<RetrievedChunk>,
}

impl RagEngine {
    pub fn new(file_dir: impl Into<String>, qdrant_store: QdrantStore, llm: LlmService) -> Self {
        Self { file_dir: file_dir.into(), qdrant_store, llm }
    }

    pub async fn initialize(&self) -> Result<()> {
        self.qdrant_store.ensure_collection().await
    }

    pub async fn reindex_docs(&self) -> Result<usize> {
        let load = loader_documents_from_dir(&self.file_dir)?;
        let chunks = chunk_retrieve(load, 120);
        self.qdrant_store.reset_collection().await?;
        let indexed = self.qdrant_store.upsert_chunks(&chunks).await?;
        Ok(indexed)
    }

    pub async fn ask_question(&self, question: String) -> Result<RagAnswer> {
        let chunk = self.qdrant_store.search(&question, 3).await?;
        let answer = self.llm.answer_question(&question, &chunk).await?;
        Ok(RagAnswer { answer, sources: chunk })
    }
}

RagEngine is the conductor. It doesn't know how PDFs are parsed, how chunks become vectors, or how the LLM is called — it just sequences the pieces. That's the entire point: when you want to swap Qdrant for Weaviate or OpenRouter for a local model, you change one file and everything else is fine.

top_k = 3 is conservative. Bigger k means more context and higher cost; smaller means crisper but riskier. Tune it for your corpus.

#[derive(Clone)] is what makes the engine cheap to hand to the gRPC handler. The expensive things inside (Qdrant client, OpenRouter client) are themselves cheaply cloneable — they're just Arc-wrapped connection pools.

`src/grpc_service.rs` — the network surface

#[derive(Clone)]
pub struct RagGrpcService {
    engine: RagEngine,
}

#[tonic::async_trait]
impl RagService for RagGrpcService {
    async fn reindex(&self, _req: Request<ReindexRequest>)
        -> Result<Response<ReindexResponse>, Status>
    {
        let engine = self.engine.clone();
        let handle = tokio::spawn(async move {
            engine.reindex_docs().await
                .map_err(|err| Status::internal(err.to_string()))
        });
        let chunks_indexed = handle.await
            .map_err(|err| Status::internal(err.to_string()))??;

        Ok(Response::new(ReindexResponse {
            chunks_indexed: chunks_indexed as u64,
            message: format!("Indexed {} chunk into qdrant", chunks_indexed),
        }))
    }

    async fn ask_question(&self, req: Request<AskQuestionRequest>)
        -> Result<Response<AskQuestionResponse>, Status>
    {
        let question = req.into_inner().question;
        if question.trim().is_empty() {
            return Err(Status::invalid_argument("question cannot be empty"));
        }

        let rag_answer = self.engine.ask_question(question).await
            .map_err(|err| Status::internal(err.to_string()))?;

        let sources = rag_answer.sources.into_iter().map(|c| Source {
            file_path: c.file_path,
            chunks_indexed: c.chunk_index as u32,
            preview: c.text.chars().take(180).collect(),
            score: c.score,
        }).collect();

        Ok(Response::new(AskQuestionResponse {
            answer: rag_answer.answer,
            sources,
        }))
    }
}

Tonic handlers are thin — they unpack the request, validate, call the engine, pack the response. Two details worth pointing out.

reindex uses tokio::spawn because indexing can take a while (hundreds of OpenRouter embedding calls). Spawning lets it run on a dedicated task; the handler awaits the join handle and propagates errors. For a production system you'd probably make this fire-and-forget with a separate status endpoint, but for our scale, await-and-wait is fine.

The preview: c.text.chars().take(180).collect() truncates the chunk to 180 characters before sending it over the wire. Full chunks could be hundreds of words; the preview is enough to let the user verify the source without bloating the response.

`src/main.rs` — boot

#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    dotenvy::dotenv().ok();

    tracing_subscriber::fmt()
        .with_target(false).compact().init();

    let config = Config::from_env();
    let addr = config.server_addr();

    let qdrant_store = QdrantStore::new(&config.qdrant_url, &config.qdrant_collection)?;
    let llm = LlmService::new()?;
    let engine = RagEngine::new("./docs", qdrant_store, llm);
    engine.initialize().await?;

    let grpc_service = RagGrpcService::new(engine);
    info!("CodeRAG-rs gRPC server running on {}", addr);

    Server::builder()
        .add_service(RagServiceServer::new(grpc_service))
        .serve(addr.parse()?)
        .await?;

    Ok(())
}

Linear and easy to follow: load .env, set up logging, build config, construct the three services in dependency order (QdrantStore → LlmService → RagEngine), call initialize() to make sure the Qdrant collection exists, then start the tonic server. The #[tokio::main] attribute turns this into an async runtime entry point.

`src/bin/chat.rs` — the terminal client

#[tokio::main]
async fn main() -> Result<()> {
    let addr = std::env::var("RAG_SERVER")
        .unwrap_or_else(|_| "http://127.0.0.1:50051".to_string());

    let mut client = RagServiceClient::connect(addr.clone()).await
        .with_context(|| format!("could not reach RAG server at {addr}"))?;

    println!("Type a question and press Enter. Commands: /reindex, /quit");

    let stdin = io::stdin();
    let mut stdout = io::stdout();
    let mut input = stdin.lock();
    let mut line = String::new();

    loop {
        print!("you > ");
        stdout.flush().ok();
        line.clear();
        if input.read_line(&mut line)? == 0 { break; }   // Ctrl-D

        let q = line.trim();
        if q.is_empty() { continue; }
        if q == "/quit" || q == "/exit" { break; }
        if q == "/reindex" {
            let r = client.reindex(ReindexRequest {}).await?.into_inner();
            println!("indexed {} chunks — {}\n", r.chunks_indexed, r.message);
            continue;
        }

        let r = client.ask_question(AskQuestionRequest {
            question: q.to_string()
        }).await?.into_inner();

        println!("\nbot > {}\n", r.answer);
        for s in r.sources {
            let preview: String = s.preview.chars().take(80).collect();
            println!("  - {} (score {:.3})\n    {}…", s.file_path, s.score, preview);
        }
    }
    Ok(())
}

This is a separate binary that connects to the running server over gRPC. The same tonic::include_proto!("rag") macro that gave the server its types gives the client a strongly-typed RagServiceClient. The loop reads from stdin, dispatches commands (/reindex, /quit), and otherwise sends the input as a question.

Two binaries from one codebase. Production-friendly: you can deploy the server in Docker and ship the chat client to developers' laptops, and they share types automatically.

▶️ Running it end-to-end

Build everything

git clone https://github.com/Parikalp-Bhardwaj/qrag-rust
cd qrag-rust
cargo build

The first build is slow (tonic, qdrant-client, and rig pull a lot of crates). Subsequent builds are quick.

Start Qdrant and the server

docker compose up -d
cargo run --bin qrag-rust

You should see:

INFO CodeRAG-rs gRPC server running on 127.0.0.1:50051

Index your docs (via grpcurl)

grpcurl -plaintext -d '{}' \
  -import-path proto -proto rag.proto \
  127.0.0.1:50051 rag.RagService/Reindex

Expected response:

{
  "chunksIndexed": "386",
  "message": "Indexed 386 chunk into qdrant"
}

Ask a question

grpcurl -plaintext -d '{"question":"What is tokio?"}' \
  -import-path proto -proto rag.proto \
  127.0.0.1:50051 rag.RagService/AskQuestion

Expected response:

{
  "answer": "Tokio is an asynchronous runtime for Rust that enables programs to run many async tasks concurrently. It is useful for background jobs, concurrent I/O, network servers, and worker tasks. The `tokio::spawn` function is utilized to start a new asynchronous task that runs independently within the Tokio runtime. (Source: ./docs/tokio.md)",
  "sources": [
    {
      "filePath": "./docs/tokio.md",
      "preview": "# Tokio Runtime Tokio is an asynchronous runtime for Rust. It allows programs to run many async tasks concurrently. tokio::spawn is used to start a new asynchronous task. The spawn",
      "score": 0.5813062
    },
    {
      "filePath": "./docs/Rust-for-Network-Programming-and-Automation.pdf",
      "chunksIndexed": 230,
      "preview": "resources like sockets and orchestrating the event loop that drives the application. It provides an API for registering interest in I/O events (e.g., data arriving on a socket or a",
      "score": 0.51516765
    },
    {
      "filePath": "./docs/Rust-for-Network-Programming-and-Automation.pdf",
      "chunksIndexed": 233,
      "preview": "With the help of Tokio's abstractions and asynchronous I/O, you can create scalable network apps that can manage any number of connections at once. Hyper: High-Level HTTP Library H",
      "score": 0.48589033
    }
  ]
}

You'll get an answer plus the source chunks that produced it.

Or use the chat CLI

In a second terminal:

cargo run --bin chat

you > what is TCP

bot > TCP, or Transmission Control Protocol, is a core protocol within the TCP/IP suite designed to facilitate reliable data transmission over networks. It ensures that data packets are transmitted accurately and in the correct order by using a process known as the three-way handshake for establishing connections. TCP also incorporates mechanisms for error detection, correction, and retransmission, ensuring data integrity and reliability even in complex network environments. This makes TCP essential for applications that require robust communication.

(Source: ./docs/Rust-for-Network-Programming-and-Automation.pdf)

  sources:
    - ./docs/Rust-for-Network-Programming-and-Automation.pdf  (score 0.546)
      which is then encapsulated in an IP packet, and finally in a link layer frame. T…
    - ./docs/Rust-for-Network-Programming-and-Automation.pdf  (score 0.540)
      requirements. Its robustness ensures reliable data transmission even in complex …
    - ./docs/Rust-for-Network-Programming-and-Automation.pdf  (score 0.526)
      as OSPF (Open Shortest Path First) and BGP (Border Gateway Protocol), help route…

Commands: /reindex to rebuild, /quit to exit.

⚠️ Gotchas worth knowing

A handful of papercuts you'll save time on:

Vector dimension mismatch. If you initialize the collection at the wrong dimension, every upsert fails. Drop the collection (docker compose down -v) and let it recreate. text-embedding-3-small is 1536-dim.
Don't commit .env. If you already did, rotate the key — git history is forever.
OpenRouter for embeddings. OpenRouter is primarily a completion router but does pass through OpenAI's embedding models. If embeddings start erroring, point the embedding client directly at OpenAI and keep OpenRouter for completion.

🔭 Where to go from here

The system you just built is the simplest thing that works. Things to try next, in rough order of payoff:

Better chunking. Sentence-aware splits, sliding-window overlap, or semantic chunking. Probably the single biggest quality lever.
Reranking. After Qdrant returns top-20, use a cross-encoder to rerank down to top-3. Higher precision at the cost of latency.
Hybrid retrieval. Combine vector search with BM25 keyword search. Catches exact-match queries that pure vector search misses.
Streaming responses. gRPC supports server streaming — tokens can land in the client as the LLM produces them.
Per-tenant collections. One Qdrant collection per user or per workspace.
Observability. Trace every step — embedding latency, search latency, LLM latency. RAG systems get slow in surprising places.

If you want a structured course on RAG fundamentals in Rust — chunking strategies, retrieval evaluation, embedding selection — check out Foundations of RAG Systems with Rust on CodeSignal. Pair it with this post: course for the why, this build for the how.

💭 Closing thoughts

Modern AI engineering isn't really about models anymore. The models are a commodity you call over HTTP. The interesting work is everywhere else — splitting documents intelligently, indexing vectors at scale, routing requests with low latency, streaming results without blocking, keeping memory and concurrency under control.

That's systems engineering. And it's exactly the territory Rust is built for. Strong types, async without garbage collection, predictable performance, and a tooling ecosystem (tonic, tokio, rig, qdrant-client) that's quietly catching up to anything Python has for AI infrastructure.

You don't need Rust to build RAG. You need Rust when RAG turns into a real system someone depends on.

Build the thing. Read the source. Break it. That's where the understanding comes from. 🦀

🙌 Thanks for reading

If you made it this far — seriously, thank you. 🙏

I'd love to hear what you're building, what didn't work for you, or what you'd do differently. Feel free to drop a comment, open an issue on the repo, or reach out on LinkedIn — always happy to talk Rust, AI, RAG, or anything in between.

🔗 Links

👤 Connect with me on LinkedIn
📦 Project repo: github.com/Parikalp-Bhardwaj/qrag-rust
📘 Qdrant docs
📘 Rig docs
📘 Tonic guide
📘 OpenRouter

DEV Community: Parikalp Bhardwaj

What Does Arc Actually Do? Atomic Reference Counting Under the Hood

Why Arc Exists

What Is Arc<T>?

A note on vocabulary

Why Not Just Use &T?

Why Not Rc<T>?

What that error is really saying

What "Reference Counting" Actually Means

What "Atomic" Actually Means

Arc Does Not Make T Thread-Safe

The compiler checks this too

Why Arc<Mutex<T>> is everywhere

The Question

The Hypothesis

Experiment 1 — Prove Shared Allocation and Reference Counting

Predict before you run

The output

Experiment 2 — The Same Arc Across OS Threads

Experiment 3 — What Is Actually in That Allocation?

Experiment 4 — What Does Arc::new Allocate?

Experiment 5 — What Does clone Compile To?

Experiment 6 — The Memory Orderings

Experiment 7 — What Does It Cost?

Stressing the Design — More Than One Core

Sharing the counter costs about 4.5x

Read-only sharing is free

The clone private column rises too — and that is article 1's fault

The Mistake That Shows Up in Production

Weak, Cycles, and the Leak Rust Does Not Prevent

What I Could Not Measure Here

Production Connection

Conclusion

Next

Appendix: Environment and Reproducibility

Thanks for Reading

What Actually Happens When Rust Spawns an OS Thread?

Why I'm Starting With Threads

Why Low-Level Knowledge Matters

Performance

Concurrency

Async Systems

Distributed Systems

The Series Roadmap

Series Philosophy

Why Rust for This Series

The Question

The Hypothesis

The Experiment

What the Trace Actually Shows

1. The stack is allocated in userspace, before the kernel is involved

2. Why 2 MiB?

3. clone3 is the actual thread creation

4. join() is a futex, and the kernel does the wake

5. The child does more work than you'd expect before running your closure

What Linux Sees

Confirming CLONE_VM

The stacks, and their guard pages

The guard page, doing its job

What a Thread Actually Costs

Virtual memory

Does stack size actually matter?

Time

But the obvious fix is less obvious than it looks

Stressing the Design: More Threads Than Cores

Where the ceiling actually is

Reading the Source: How the Call Gets There

Production Connection

Conclusion

Next

Appendix: Environment and Reproducibility

Thanks for Reading

Building a RAG System in Rust with Qdrant, Rig, and gRPC 🦀

Why I Built This

🤔 Why RAG exists

🧩 The building blocks

🔎 Qdrant — the vector database

🦀 Rig — the AI application layer

🚀 gRPC + Tonic — the service layer

⚡ Tokio — the async runtime

Why `Arc` Exists

What Is `Arc<T>`?

Why Not Just Use `&T`?

Why Not `Rc<T>`?

`Arc` Does Not Make `T` Thread-Safe

Why `Arc<Mutex<T>>` is everywhere

Experiment 2 — The Same `Arc` Across OS Threads

Experiment 4 — What Does `Arc::new` Allocate?

Experiment 5 — What Does `clone` Compile To?

The `clone private` column rises too — and that is article 1's fault

`Weak`, Cycles, and the Leak Rust Does Not Prevent

3. `clone3` is the actual thread creation

4. `join()` is a futex, and the kernel does the wake

Confirming `CLONE_VM`

🔐 Setting up `.env` and Qdrant

1. Create the `.env` file

`Cargo.toml` — the manifest

`build.rs` — code generation at compile time

`proto/rag.proto` — the service contract

`src/config.rs` — environment-driven config

`src/document_loader.rs` — reading files from disk

`src/chunker.rs` — splitting documents into pieces

`src/qdrant_store.rs` — the heart of retrieval

`src/llm.rs` — building the prompt and calling the model

`src/rag.rs` — orchestration

`src/grpc_service.rs` — the network surface

`src/main.rs` — boot

`src/bin/chat.rs` — the terminal client