What Actually Happens When Rust Spawns an OS Thread?

Following std::thread::spawn from Rust to Linux. Article 1 of **Below the Abstraction* — a series that takes everyday Rust abstractions and follows them down to the kernel. Every number below was measured on a real machine, not recalled.*

---

Why I'm Starting With Threads

I have written a lot of Rust that creates threads. Backend services, gRPC servers, blockchain node components, async applications, background workers, infrastructure glue. thread::spawn is one of those calls that becomes invisible after the first hundred times you type it.

At some point that stopped being acceptable to me.

I had used thread::spawn many times before, but I had never actually verified what Linux saw when the call happened. I knew the shape of the answer — "it makes an OS thread" — but I could not have told you which syscall was issued, how much memory the call reserved, what the kernel scheduler now had to track, or why the same program behaves very differently at 4 threads and at 4,000.

That gap matters more than it looks. Almost every hard problem I have hit in production — tail latency that appears only under load, memory that grows without a leak, a service that goes from fine to unusable somewhere between 500 and 2,000 concurrent connections — turned out to be a question about the layer below the one I was writing in.

There is a second reason this exists. I like low-level work — it is the part of the stack where you can actually check the answer instead of arguing about it. And the fastest way I know to find out whether I really understand something is to try to explain it to someone else. Writing forces the gaps into the open: you can hand-wave your way through a conversation about threads, but you cannot hand-wave a syscall trace. So this series is a little bit me teaching, and mostly me learning in public.

It has one rule, and it is the rule I want to apply to everything:

Don't just use the abstraction. Follow it down until you understand the system underneath it.

Threads come first because everything else in the series stands on them. Arc, Mutex, RwLock, futexes, thread pools, Tokio's scheduler, epoll, async tasks — all of them are, in the end, statements about what an OS thread is, what it costs, and when the kernel takes it away from you. You cannot meaningfully say "a Tokio task is cheaper than a thread" until you have measured what a thread costs.

This article is the measurement.

---

Why Low-Level Knowledge Matters

There is a version of this argument that is just nostalgia — "real engineers know assembly." That is not the argument. The argument is that modern backend and distributed systems fail in ways that are only legible one or two layers down.

Here is the stack this series keeps returning to:

Distributed System
        ↓
Network Communication
        ↓
Sockets
        ↓
Async Runtime / Threads
        ↓
Synchronization
        ↓
Memory
        ↓
System Calls
        ↓
Kernel
        ↓
CPU + Hardware

You write at the top. You get paged at the bottom.

Performance

Every high-level abstraction eventually decomposes into a small set of concrete, countable things:

allocations
syscalls
locks
atomics
cache misses
context switches
network packets

An API surface tells you nothing about how many of each you just bought. Vec::push is amortized O(1) and also, sometimes, an mmap. thread::spawn is one line and also, as we are about to see, a mmap, an mprotect, a clone3, and a new schedulable entity that the kernel must now consider on every scheduling decision for the life of that thread.

You cannot reason about performance only from the API surface.

Concurrency

A line like:

Arc<Mutex<T>>

looks like a type. It is closer to a system:

atomic reference counts
memory ordering
lock state
parking
waking
kernel interaction
cache coherence

At low contention none of that is visible. At high contention it is the only thing that is visible — the difference between a lock that spins briefly in userspace and one that puts a thread to sleep in the kernel shows up as a CPU utilization graph that makes no sense, or a p99 that is 40x the p50. Articles 2, 3 and 4 take that apart.

Async Systems

Most Rust developers write:

tokio::spawn(...)

long before they understand:

Future
Poll
Waker
runtime scheduler
reactor
epoll
kernel readiness notifications

I think async Rust is genuinely confusing if you meet it first. It makes far more sense in the other order: OS threads, blocking, scheduling, and I/O readiness first — then async as a specific answer to a specific cost. That is why this series does not start with Tokio.

Distributed Systems

This is the part I care most about, and it is the part that is usually skipped.

Distributed systems problems become local systems problems, almost always. A latency graph in Grafana is a distributed artifact; the thing producing it is a single machine doing something specific.

Request latency
    ↓
network wait
    ↓
socket readiness
    ↓
runtime scheduling
    ↓
task wakeup
    ↓
lock acquisition
    ↓
cache/database access

Or, the shape I have personally chased more than once:

Kafka consumer slowdown
        ↓
backpressure
        ↓
queue growth
        ↓
memory growth
        ↓
scheduler pressure
        ↓
latency spikes

Every arrow in those chains is a local-machine phenomenon. "The cluster is slow" is a summary; the mechanism is a scheduler, a lock, a page fault, or a socket buffer.

A distributed system is not something separate from operating systems. It is multiple operating systems communicating over an unreliable network. If you do not understand what one machine does under load, a hundred of them will not be easier.

---

The Series Roadmap

Each article is a single investigation, and each one depends on the one before it.

1. What Actually Happens When Rust Spawns an OS Thread? — establish what a thread is, and what it costs.
2. What Does Arc Actually Do? Atomic Reference Counting Under the Hood — now that threads share memory, how is ownership shared safely?
3. Mutex vs RwLock in Rust: Benchmarking Real Contention — shared memory needs mutual exclusion; which primitive, and when?
4. What Happens When a Rust Mutex Blocks? Parking, Waking, and Futexes — the moment a lock stops being userspace-only and calls the kernel.
5. Building a Thread Pool in Rust From Scratch — the direct consequence of article 1's cost measurements.
6. Why Tokio Tasks Aren't Threads: 10,000 Tasks vs OS Threads — the same workload, re-measured against the numbers from article 1.
7. From epoll to Tokio: What Happens When Rust Waits for Network I/O? — where the runtime actually blocks.
8. False Sharing in Rust: CPU Cache Lines and Multithreaded Performance — going below the kernel, into the hardware.
9. Building a Concurrent LRU Cache in Rust — everything above, applied to one realistic data structure.
10. Tracing Rust async/await All the Way Down to epoll — the full path, end to end.

The progression is deliberate: threads → shared state → synchronization → blocking → pooling → async → I/O readiness → hardware → applied → full trace.

---

Series Philosophy

Every article follows the same investigation model:

Question
   ↓
Hypothesis
   ↓
Small Rust Program
   ↓
Run It
   ↓
Inspect Linux
   ↓
Measure
   ↓
Read Source Code
   ↓
Explain Internals
   ↓
Stress the Design
   ↓
Production Connection
   ↓
Conclusion

And one hard constraint: nothing is invented. No benchmark numbers, memory figures, syscall traces, or scheduler behaviour is written from memory or plausibility. Every number and every trace line in this article was produced by running the programs shown, on the machine described in the appendix, and pasted in. Where a result surprised me or contradicted what I expected, I say so rather than smoothing it over — a couple of the measurements below did exactly that.

Your numbers will differ. The shapes should not.

---

Why Rust for This Series

Rust is unusually good for this kind of work, and it is worth being specific about why rather than cheerleading.

What helps:

• Ownership and borrowing make the memory model explicit at the type level, so "who owns this buffer across threads" is a compile-time question rather than a debugging session.
• Send and Sync encode thread-safety as traits. When you study synchronization primitives, having the safety rules written in the type system is a genuine teaching aid.
• No hidden runtime. std::thread::spawn is a thin wrapper over the platform's native threads. There is no green-thread layer or VM between you and clone3, which is exactly what you want when the goal is to see the syscall.
• Explicit atomics with explicit orderings (Relaxed, Acquire, Release, SeqCst) force you to state memory ordering rather than inherit it.
• Zero-cost abstractions mean the assembly usually corresponds to the source in a way you can follow.
• First-class FFI lets you drop to libc and call the raw syscall when you want to compare.

What genuinely hurts:

• Compiler complexity. The borrow checker is a real cost when you are prototyping a data structure whose whole point is aliasing.
• Async type complexity. Deeply nested impl Future types, Pin, and lifetime errors in async code are hard, and error messages in that area are still rough.
• Native dependencies and build friction. Anything touching C libraries brings the usual pain, plus bindgen/cc on top.
• unsafe and FFI boundaries. The moment you go low-level you lose the guarantees that made Rust attractive, and you are back to C-level discipline without C's decades of tooling defaults.
• Ecosystem maturity for systems tooling. C and C++ have a longer tail of profilers, sanitizers, and debugger integrations that "just work". Rust's are good and improving, but not equal.
• Debugging low-level async. A stack trace through a poll chain is much less informative than a stack trace through blocking calls. This is a real regression in observability, and article 10 will have to deal with it head-on.

I am using Rust here because it is the best available lens on these mechanisms, not because it removes them.

---

The Question

What actually happens after this line?

std::thread::spawn(...)

Specifically:

• Which syscall is issued, and with which arguments?
• What memory is reserved before the syscall, and by whom?
• What does Linux create, and how does it name and track it?
• What does it cost — in time, in virtual memory, in resident memory?
• What changes when there are more runnable threads than CPU cores?

The Hypothesis

My starting hypothesis, before running anything:

std::thread::spawn creates a native OS thread. Rust's standard library delegates to pthread_create, which allocates a stack, then asks the kernel to create a new task sharing the caller's address space. Linux tracks that task independently and schedules it independently. The thread is not free: it costs a stack-sized virtual memory reservation and a kernel-visible schedulable entity.

That is a hypothesis, not an answer. Let's check every clause of it.

---

The Experiment

The smallest program that spawns exactly one thread:

// hello_thread.rs
use std::thread;

fn main() {
    let handle = thread::spawn(|| {
        println!("hello from a spawned thread");
    });
    handle.join().unwrap();
    println!("main thread done");
}

Built with rustc -O -g hello_thread.rs -o hello_thread, then traced:

$ strace -f -c ./hello_thread

The full syscall summary for the whole process life is 89 syscalls. The interesting ones:

  0.00    0.000000           0        16           mmap
  0.00    0.000000           0         8           mprotect
  0.00    0.000000           0         5           munmap
  0.00    0.000000           0         2           gettid
  0.00    0.000000           0         1           futex
  0.00    0.000000           0         2           set_robust_list
  0.00    0.000000           0         2           rseq
  0.00    0.000000           0         1           clone3

One clone3. That is the whole thread creation, from the kernel's point of view. Everything else around it is preparation and cleanup.

Note what is not there: there is no thread_create syscall on Linux. There is no separate "thread" object. Threads and processes are the same kernel primitive — a task_struct — created by the same syscall family, differing only in which resources they share. That is the first thing the trace teaches, and it reframes everything that follows.

---

What the Trace Actually Shows

Here is the relevant window from the full trace (strace -f), with the process ID 3616 as the main thread and 3617 as the spawned thread. This is verbatim output, trimmed only to the region around the spawn:

3616  mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7ff8a9bff000
3616  mprotect(0x7ff8a9c00000, 2097152, PROT_READ|PROT_WRITE) = 0
3616  rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
3616  clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM
              |CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
              child_tid=0x7ff8a9dff990, parent_tid=0x7ff8a9dff990, exit_signal=0,
              stack=0x7ff8a9bff000, stack_size=0x1fff40, tls=0x7ff8a9dff6c0}
             => {parent_tid=[3617]}, 88) = 3617
3616  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
3616  futex(0x7ff8a9dff990, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 3617, NULL,
            FUTEX_BITSET_MATCH_ANY <unfinished ...>
3617  rseq(0x7ff8a9dfffe0, 0x20, 0, 0x53053053) = 0
3617  set_robust_list(0x7ff8a9dff9a0, 24) = 0
3617  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
3617  mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff8a1a00000
3617  munmap(0x7ff8a1a00000, 39845888)  = 0
3617  munmap(0x7ff8a8000000, 27262976)  = 0
3617  mprotect(0x7ff8a4000000, 135168, PROT_READ|PROT_WRITE) = 0
3617  gettid()                          = 3617
3617  write(1, "hello from a spawned thread\n", 28) = 28
3617  madvise(0x7ff8a9bff000, 2076672, MADV_DONTNEED) = 0
3617  exit(0)                           = ?
3616  <... futex resumed>)              = 0
3616  write(1, "main thread done\n", 17) = 17
3616  exit_group(0)                     = ?

There is a lot in there. Let's go line by line.

1. The stack is allocated in userspace, before the kernel is involved

mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0)
mprotect(0x7ff8a9c00000, 2097152, PROT_READ|PROT_WRITE)

2101248 bytes is 2 MiB + 4096. The whole region is mapped PROT_NONE — no access at all — and then all of it except the first page is flipped to read/write.

That leftover 4 KiB at the bottom is the guard page. It is a page with no permissions sitting immediately below the stack, so that a stack that grows too far hits an unmapped page and faults instead of silently scribbling over whatever mapping happens to be next in the address space.

This is worth pausing on: the kernel did not allocate the thread stack. glibc did, in userspace, with an ordinary anonymous mmap, before clone3 was ever called. The kernel is handed a pointer to memory that already exists. A "thread stack" is not a kernel concept — it is just anonymous memory that a thread happens to use as a stack.

MAP_NORESERVE is not set here, but the mapping is PROT_NONE until mprotect and untouched afterwards, so no physical pages are committed yet. We will measure exactly that in a moment.

2. Why 2 MiB?

Not because Linux says so. The main thread's stack limit on this machine is 8 MiB:

prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0

The 2 MiB is Rust's choice. In the standard library's Unix thread implementation:

#[cfg(not(any(
    target_os = "l4re",
    target_os = "vxworks",
    target_os = "espidf",
    target_os = "nuttx"
)))]
pub const DEFAULT_MIN_STACK_SIZE: usize = 2 * 1024 * 1024;

We can confirm the plumbing empirically rather than trusting the constant. RUST_MIN_STACK should change the size of that mmap, and it does:

$ RUST_MIN_STACK=1048576 strace -f -e trace=mmap,clone3 ./hello_thread
mmap(NULL, 1052672, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, ...) 
clone3({... stack_size=0xfff40 ...})

$ RUST_MIN_STACK=8388608 strace -f -e trace=mmap,clone3 ./hello_thread
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, ...)
clone3({... stack_size=0x7fff40 ...})

1052672 = 1 MiB + 4096. 8392704 = 8 MiB + 4096. Same guard page, different stack. The request travels from a Rust env var, through Builder, into pthread_attr_setstacksize, and out as the size of an mmap. That is the whole chain, visible in one command.

3. clone3 is the actual thread creation

clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM
        |CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, ...}) = 3617

Each flag is a decision about what the new task shares with the old one:

Flag	Meaning	Consequence
CLONE_VM	share the address space (mm_struct)	this is what makes it a thread and not a process
CLONE_FS	share filesystem info (cwd, umask, root)	chdir in one thread affects all
CLONE_FILES	share the file descriptor table	fd 7 means the same socket in every thread
CLONE_SIGHAND	share signal handlers	one handler table per process
CLONE_THREAD	join the same thread group	same PID/TGID; getpid() matches
CLONE_SYSVSEM	share System V semaphore undo state
CLONE_SETTLS	install the given TLS pointer	thread_local! needs this
CLONE_PARENT_SETTID	write the new TID into the parent's memory	the parent learns the child's TID
CLONE_CHILD_CLEARTID	clear that word on exit and futex-wake it	this is how join() works

exit_signal=0 means the parent process is not sent SIGCHLD when this task dies, because it is a thread, not a child process.

The first four or five flags are the entire difference between fork and "spawn a thread". A process is the same call with fewer sharing flags. Once you have seen this, "threads share memory, processes don't" stops being a rule you memorised and becomes an argument you passed.

4. join() is a futex, and the kernel does the wake

Look at the pairing:

3616  futex(0x7ff8a9dff990, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 3617, ...)
...
3617  exit(0)
3616  <... futex resumed>) = 0

The address 0x7ff8a9dff990 is exactly the child_tid pointer passed to clone3. CLONE_CHILD_CLEARTID told the kernel: when this task exits, zero that word and perform a futex wake on it. The parent then simply waits on that futex with the expected value 3617 (the child's TID).

So handle.join() compiles down to: sleep on a futex until the kernel clears the child's TID. There is no polling. There is no "thread finished" callback. It is one memory word and one kernel wait queue.

That single futex line is the seed of article 4. Mutex, Condvar, join, channel blocking, and Tokio's own parking all end up at the same syscall.

5. The child does more work than you'd expect before running your closure

3617  rseq(0x7ff8a9dfffe0, 0x20, 0, 0x53053053) = 0
3617  set_robust_list(0x7ff8a9dff9a0, 24) = 0
3617  mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff8a1a00000
3617  munmap(0x7ff8a1a00000, 39845888)  = 0
3617  munmap(0x7ff8a8000000, 27262976)  = 0
3617  mprotect(0x7ff8a4000000, 135168, PROT_READ|PROT_WRITE) = 0

rseq registers restartable sequences (used by glibc for fast per-CPU operations). set_robust_list registers the list the kernel walks if this thread dies while holding a robust mutex.

Then something bigger: a 128 MiB PROT_NONE reservation, two munmaps to trim it down to a 64 MiB aligned region, and an mprotect of 132 KiB.

That is glibc's malloc creating a new per-thread arena. The first time a new thread allocates, glibc may give it its own arena to avoid contention on the main arena's lock. The 128 MiB mmap is a reservation of address space, not memory — it is PROT_NONE, so not a single physical page is committed.

I did not expect this to show up in a program whose thread only calls println!. It matters, because it shows up in virtual memory accounting in a way that looks alarming and isn't — as the next section demonstrates.

Also note:

3617  exit(0)
3616  exit_group(0)

The thread exits with exit, which terminates one task. The main thread ends with exit_group, which terminates the whole thread group. Same distinction, visible in the trace.

---

What Linux Sees

A trace shows the transition. /proc shows the steady state. This program spawns four named threads and sleeps:

// four_threads.rs
use std::thread;
use std::time::Duration;

fn main() {
    let mut handles = Vec::new();
    for i in 0..4 {
        let h = thread::Builder::new()
            .name(format!("worker-{i}"))
            .spawn(move || {
                thread::sleep(Duration::from_secs(20));
                i
            })
            .unwrap();
        handles.push(h);
    }
    println!("pid = {}", std::process::id());
    for h in handles {
        h.join().unwrap();
    }
}

While it runs:

$ ls /proc/$PID/task
5481  5483  5484  5485  5486

$ for t in /proc/$PID/task/*; do
>   printf "%s comm=%-12s State=%s\n" "$(basename $t)" "$(cat $t/comm)" \
>     "$(awk '/^State:/{print $2,$3}' $t/status)"
> done
5481  comm=four_threads State=S (sleeping)
5483  comm=worker-0     State=S (sleeping)
5484  comm=worker-1     State=S (sleeping)
5485  comm=worker-2     State=S (sleeping)
5486  comm=worker-3     State=S (sleeping)

Five kernel tasks. Each has its own TID, its own state, its own scheduler accounting. thread::Builder::name() is not just a Rust-side label — it reaches /proc/<tid>/comm, which means it shows up in ps, top, perf, and every kernel-level tool. That is a free observability win that a lot of Rust code leaves on the table.

ps agrees:

$ ps -o pid,tid,psr,pcpu,stat,comm -L -p $PID
  PID   TID PSR %CPU STAT COMMAND
27548 27548   0  0.0 Sl   four_threads
27548 27549   1  0.0 Sl   worker-0
27548 27550   0  0.0 Sl   worker-1
27548 27551   1  0.0 Sl   worker-2
27548 27552   0  0.0 Sl   worker-3

One PID, five TIDs, and PSR shows the kernel has already spread them across both CPUs.

From a worker's own status file:

$ grep -E '^(Name|State|Tgid|Pid|Threads|Cpus_allowed_list)' /proc/$PID/task/27552/status
Name:   worker-3
State:  S (sleeping)
Tgid:   27548
Pid:    27552
Threads:    5
Cpus_allowed_list:  0-1

Pid: 27552, Tgid: 27548. Inside the kernel, "PID" means the task ID and "TGID" means what userspace calls the process ID. getpid() returns the TGID; gettid() returns the PID. This is why the trace showed gettid() — the Rust runtime wants the task identity, not the process identity.

Confirming CLONE_VM

The claim "threads share an address space" is testable:

$ diff <(cat /proc/$PID/task/$PID/maps) <(cat /proc/$PID/task/$TID/maps) \
    && echo "identical"
identical

Byte-identical memory maps for two different tasks. One mm_struct, five tasks pointing at it. That is CLONE_VM, observed rather than asserted.

The stacks, and their guard pages

Dumping the anonymous mappings with their neighbours:

7f3b0abfc000-7f3b0abfd000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0abfd000-7f3b0adfd000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

7f3b0adfd000-7f3b0adfe000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0adfe000-7f3b0affe000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

7f3b0affe000-7f3b0afff000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0afff000-7f3b0b1ff000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

7f3b0b1ff000-7f3b0b200000 ---p 00000000 00:00 0      4 kB ---p   (guard)
7f3b0b200000-7f3b0b400000 rw-p 00000000 00:00 0   2048 kB rw-p   (thread stack)

Four threads, four 2 MiB stacks, four 4 KiB ---p guard pages, laid out back to back. Note the accounting consequence: each thread costs two VMA entries, not one. That will matter when we look at limits.

The guard page, doing its job

A thread with a deliberately small stack and unbounded recursion:

// overflow.rs
use std::thread;

fn recurse(n: u64) -> u64 {
    let pad = [n; 1024];             // 8 KiB of stack per frame
    std::hint::black_box(&pad);
    if n == 0 { 0 } else { recurse(n - 1) + pad[0] }
}

fn main() {
    let h = thread::Builder::new()
        .name("deep".into())
        .stack_size(64 * 1024)       // deliberately small: 64 KiB
        .spawn(|| recurse(1_000_000))
        .unwrap();
    println!("join result: {:?}", h.join().is_err());
}

Running it:

$ ./overflow

thread 'deep' (27182) has overflowed its stack
fatal runtime error: stack overflow, aborting
Aborted
$ echo $?
134

And the signals, from strace:

[pid 27928] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7ff67a257cc0} ---
[pid 27928] --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=27927, si_uid=0} ---
[pid 27928] +++ killed by SIGABRT +++

SEGV_ACCERR — an access permission error, not SEGV_MAPERR. The address is mapped; the thread simply had no permission to touch it. That is the PROT_NONE guard page.

This also explains the sigaltstack calls that appear near the top of the trace for every thread:

mmap(NULL, 16048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0)
mprotect(0x7ff8aa0b9000, 4096, PROT_NONE)
sigaltstack({ss_sp=0x7ff8aa0ba000, ss_flags=0, ss_size=11952}, NULL)
rt_sigaction(SIGSEGV, {sa_handler=0x560dd7ce53a0, ..., sa_flags=...|SA_ONSTACK|SA_SIGINFO}, ...)

Rust installs a SIGSEGV handler with SA_ONSTACK and gives every thread a small alternate signal stack — with its own guard page. It has to: when you overflow a stack you cannot run a signal handler on that stack. Roughly 16 KiB per thread of extra mapping buys you a readable "has overflowed its stack" message instead of an unexplained segfault.

An engineering consequence that costs nothing to apply: the message names the thread. thread 'deep' (27182) has overflowed its stack is dramatically more useful in a production log than thread '<unnamed>'. Name your threads.

---

What a Thread Actually Costs

Now the part that changes how you design systems.

Virtual memory

This program spawns batches of parked threads and reads its own /proc/self/status at each step:

// cost.rs (excerpt)
for target in [0usize, 100, 500, 1000, 2000, 4000] {
    // ... spawn up to `target` live threads, wait until all are parked ...
    println!("{:>8} {:>12} {:>10} {:>8}",
             target, stat("VmSize:"), stat("VmRSS:"), stat("Threads:"));
}

Default run:

live-thread footprint (kB from /proc/self/status)
 threads       VmSize      VmRSS  Threads
       0        70856       2408        1
     100      1193108       3488      101
     500      2020572       7312      501
    1000      3054836      12108     1001
    2000      5123528      21688     2001
    4000      9260740      40868     4001

The jump from 0 to 100 threads is over a gigabyte of virtual memory, which is far more than 100 x 2 MiB. That is the malloc arena behaviour from the trace: glibc creates up to 8 * ncores arenas, each reserving 64 MiB of address space. Capping it isolates the stack cost:

$ MALLOC_ARENA_MAX=1 ./cost 200

 threads       VmSize      VmRSS  Threads
       0         5320       2404        1
     100       210200       3420      101
     500      1038060       7252      501
    1000      2072852      12044     1001
    2000      4142436      21624     2001
    4000      8281508      40784     4001

Now it is clean and linear: roughly 2,069 kB of virtual memory per thread (2 MiB stack + guard page + TLS + thread descriptor). 4,000 threads costs about 7.9 GiB of virtual address space.

The arena arithmetic checks out too. In the default run, 100 threads added 1,122,252 kB of VmSize; 100 x 2,069 kB of stacks accounts for 206,900 kB, leaving 915,352 kB, which is 14 x 65,536 kB. Fourteen 64 MiB arenas, against a ceiling of 8 * 2 cores = 16.

Two lessons, and the second is the one that actually costs people time:

1. Threads are expensive in virtual memory, cheap in resident memory. 4,000 threads cost 7.9 GiB of VmSize but only about 40 MiB of VmRSS — roughly 10 kB resident per thread. The stacks are mapped, not touched. Physical pages arrive on first write.
2. VmSize is not memory usage. If you alert on virtual size, a multi-threaded Rust service on glibc will page you for nothing. Alert on RSS, or on Committed_AS if you care about overcommit headroom.

Does stack size actually matter?

If RSS is what matters and stacks are lazily backed, does stack_size matter at all? Measured, 1,000 threads each, arenas capped:

stack_size=    16384  n=1000  VmSize delta=    49584 kB (   49 kB/thread)  VmRSS delta= 13712 kB ( 13 kB/thread)
stack_size=    65536  n=1000  VmSize delta=    85584 kB (   85 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)
stack_size=   262144  n=1000  VmSize delta=   277584 kB (  277 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)
stack_size=  2097152  n=1000  VmSize delta=  2069584 kB ( 2069 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)
stack_size=  8388608  n=1000  VmSize delta=  8213584 kB ( 8213 kB/thread)  VmRSS delta=  9712 kB (  9 kB/thread)

VmSize tracks the requested stack almost exactly, with a constant overhead of about 21 kB per thread on top (guard page, thread control block, static TLS) — visible in every row from 64 KiB upward: 85 - 64 = 21, 277 - 256 = 21, 2069 - 2048 = 21, 8213 - 8192 = 21. The 16 KiB row breaks the pattern because the request is below the platform minimum and gets raised; that is the cmp::max(stack, min_stack_size(...)) we will see in the source. VmRSS is flat at roughly 9.7 kB per thread regardless of whether you asked for 64 KiB or 8 MiB of stack.

So: shrinking stacks buys you address space, not RAM. On 64-bit that is usually not the constraint you are fighting — which means stack_size tuning is mostly worth doing when you are hitting VMA or overcommit limits, not as a general memory optimisation. That is a different conclusion from the folklore, and it only shows up if you measure both numbers.

Time

Cost of a thread that does nothing at all — spawn immediately followed by join, 1,000 iterations:

spawn+join, 1000 iterations (nanoseconds)
  min    = 11206
  p50    = 47496
  p90    = 119084
  p99    = 230279
  max    = 505619
  mean   = 70174

A round trip of about 47 us at p50, with a p99 of 230 us and a worst case over half a millisecond.

Separating creation from teardown — 10,000 threads created with 64 KiB stacks, all created before any join:

creating 10000 threads (64 KiB stacks)
  spawn() call    p50 =    36569 ns
  spawn() call    p99 =   300113 ns
  spawn() call    max =  2687274 ns
  wall to create all  = 546.666318ms  (18293 threads/sec)
  wall to join all    = 43.088635ms

Roughly 18,000 threads/sec of creation throughput on two cores, with a p50 of 37 us for the spawn() call itself and a tail reaching 2.7 ms.

Put that next to a latency budget. If your API server's p99 target is 10 ms and you spawn a thread per request, creation alone consumes 0.4% of the budget at p50, 3% at p99, and 27% in the worst case observed — before your handler has read a single byte. At 20,000 requests/sec, thread creation alone saturates both cores.

That is the number that justifies article 5.

But the obvious fix is less obvious than it looks

I expected the "reuse a thread instead" comparison to be dramatic. It wasn't:

N = 20000
  spawn+join per item :     46675 ns   (total 933.507137ms)
  channel round-trip  :     37140 ns   (total 742.79602ms)
  ratio               :       1.3x

Handing work to an already-running thread over an mpsc channel and waiting for the reply is only 1.3x faster than creating a whole new OS thread for it.

That result is real, and it is more instructive than the one I was expecting. Both paths are dominated by the same thing: blocking, and being woken by the scheduler. A strict ping-pong handoff costs two context switches per item, and on a 2-core machine those wakeups cost roughly what a clone3 costs. Thread creation is not the expensive part in this shape of workload — blocking is.

Which reframes the lesson. The value of a thread pool is not "spawning is slow". It is that a pool lets many items be in flight across a fixed set of threads without a sleep/wake cycle per item. And it points straight at the async argument: the reason Tokio tasks are cheap is not primarily that they skip clone3 — it is that switching between tasks does not go through the kernel scheduler at all. Article 6 gets to test that claim against these exact numbers.

I am keeping this result in because it is the kind of thing that gets quietly dropped when a benchmark doesn't say what the author wanted it to.

---

Stressing the Design: More Threads Than Cores

This machine has 2 cores. What happens when 64 threads all want CPU?

Each thread does an identical fixed amount of CPU-bound work, and reports its own wall time and its own involuntary context switches, read from /proc/thread-self/status:

// oversub2.rs (excerpt)
let handles: Vec<_> = (0..t).map(|_| thread::spawn(move || {
    let t0 = Instant::now();
    let n0 = self_stat("nonvoluntary_ctxt_switches:");
    std::hint::black_box(burn(WORK));
    (t0.elapsed().as_millis() as u64,
     self_stat("nonvoluntary_ctxt_switches:") - n0)
})).collect();

 threads     min_ms     p50_ms     max_ms     total_ms    nonvol_ctxt
       1        372        372        372          372              3
       2        373        380        380          380             46
       4        654        755        759          760            369
       8       1494       1506       1511         1517            819
      16       2955       2981       3014         3015           1739
      32       5711       5928       6035         6047           3557
      64      11763      12023      12130        12139           6942

Read this carefully, because the interesting result is what didn't happen.

Throughput is preserved. 64 threads each doing 372 ms of work is 23.8 seconds of CPU work, completed in 12.1 s of wall time on 2 cores — a speedup of about 1.96x, which is essentially both cores fully busy. The scheduler did not fall over. There is no throughput collapse from oversubscription here.

Latency is destroyed. The same unit of work takes 372 ms when it has a core to itself and 12,023 ms when it is one of 64 threads sharing two cores. That is 32x. If that unit of work is a request, your p50 just moved by a factor of 32 while your dashboards show healthy CPU utilization and healthy throughput.

Involuntary context switches scale linearly. 6,942 involuntary switches across 64 threads is about 108 preemptions per thread. Each thread consumed roughly 372 ms of actual CPU time, so it was preempted after about 3.4 ms of CPU each time. That is the scheduler doing exactly what it was designed to do — and 6,942 switches is 6,942 register saves, TLB and cache disturbances, and runqueue operations that the 1-thread run did not pay for.

The engineering consequence: oversubscription is a latency problem, not a throughput problem. Adding threads past core count does not get more work done; it spreads the same work over more concurrent, slower units. For a batch job that is harmless. For a request/response service it converts a fast p50 into a uniformly slow one — and it does so without tripping any of the signals people usually watch. Neither CPU utilization nor throughput will tell you. Only per-request latency will.

This is also why "just add more threads" fails for a service that is slow because it is CPU-bound, and works for a service that is slow because it is blocking on I/O. Those two look identical on a throughput graph and are opposite problems.

Where the ceiling actually is

The limits on this box:

threads-max:    64113        # /proc/sys/kernel/threads-max
pid_max:        32768        # /proc/sys/kernel/pid_max
max_map_count:  65530        # /proc/sys/vm/max_map_count
RLIMIT_STACK:   8192 kB
max user processes (-u): 32056
MemTotal:       8216168 kB
CommitLimit:    4108084 kB

The one people forget is max_map_count. We measured that each thread costs two VMAs — stack plus guard page. At 65,530 mappings that caps you near ~32,000 threads from mapping pressure alone, which happens to land in the same range as pid_max. You will typically hit a limit like this, or CommitLimit, well before you run out of RAM.

So "how many threads can I create" has at least four independent answers — threads-max, pid_max, RLIMIT_NPROC, max_map_count — plus overcommit policy. None of them is the number you would guess from RSS.

---

Reading the Source: How the Call Gets There

The trace tells us where we ended up. The source tells us how.

std::thread::spawn is a thin wrapper over Builder, which eventually calls the platform implementation. On Unix that is Thread::new, in the standard library's sys/thread/unix.rs:

pub unsafe fn new(stack: usize, init: Box<ThreadInit>) -> io::Result<Thread> {
    let data = init;
    let mut attr: mem::MaybeUninit<libc::pthread_attr_t> = mem::MaybeUninit::uninit();

    assert_eq!(libc::pthread_attr_init(attr.as_mut_ptr()), 0);
    let mut attr = DropGuard::new(&mut attr, |attr| {
        assert_eq!(libc::pthread_attr_destroy(attr.as_mut_ptr()), 0)
    });
    // ...
    let stack_size = cmp::max(stack, min_stack_size(attr.as_ptr()));
    match libc::pthread_attr_setstacksize(attr.as_mut_ptr(), stack_size) {
        0 => {}
        n => {
            assert_eq!(n, libc::EINVAL);
            // Round up to nearest page on alignment failure
            let page_size = sys::pal::conf::page_size();
            let stack_size = (stack_size + page_size - 1) &
                            (-(page_size as isize - 1) as usize - 1);
            // ...
        }
    }
    // ... libc::pthread_create(...)
}

That is the whole story in one function. There is no Rust-specific thread machinery in the kernel path. Rust:

1. picks a stack size — max(requested, platform minimum), defaulting to DEFAULT_MIN_STACK_SIZE = 2 * 1024 * 1024,
2. sets it on a pthread_attr_t,
3. calls pthread_create,
4. and lets glibc do the mmap + mprotect + clone3 we watched in the trace.

The closure is boxed and handed through as the thread's argument; the JoinHandle wraps the resulting thread handle plus a slot for the return value. join() ends in the futex wait we already saw.

The chain, complete:

std::thread::spawn
  -> std::thread::Builder::spawn_unchecked_
    -> sys::thread::Thread::new
      -> pthread_attr_init / pthread_attr_setstacksize
        -> pthread_create                        (glibc)
          -> mmap(PROT_NONE) + mprotect          (stack + guard page)
            -> clone3(CLONE_VM|CLONE_THREAD|...) (kernel)
              -> new task_struct, scheduled independently

Every layer in that chain is one we can now name, observe, and measure. Nothing in it is magic. And nothing in it is free.

---

Production Connection

Bringing this back to systems people actually operate.

API servers. Thread-per-request costs about 37 us at p50 and up to 2.7 ms at the tail before your handler runs. Once concurrent requests exceed core count, per-request latency degrades proportionally while throughput and CPU graphs stay flat. This is the specific, measurable reason the industry moved to pools and then to async — not fashion.

Databases and caches. Connection-per-thread designs hit max_map_count and CommitLimit before they hit RAM. A 2 MiB default stack times 10,000 connections is 20 GiB of address space for maybe 100 MiB of actual use. Knowing that VmSize is address space and VmRSS is memory is the difference between capacity planning and guessing.

Message brokers and Kafka-style consumers. The backpressure chain from the introduction — queue growth, memory growth, scheduler pressure, latency spikes — is exactly the oversubscription curve above. When a consumer falls behind and something spawns more workers to catch up, you get the 64-thread row: same throughput, 32x the latency.

Low-latency and trading systems. The tail matters more than the mean, and the tail here is involuntary preemption. 108 preemptions per thread over 12 s is invisible in an average and fatal in a p99.9. This is why such systems pin threads to cores, keep runnable threads at or below core count, and never create threads on the hot path.

Blockchain nodes, proxies, WebSocket servers, vector databases, AI infrastructure. All of them are "many concurrent connections, mostly waiting". That workload is the worst possible fit for thread-per-connection: you pay full thread cost for entities that are idle almost all the time. That mismatch is precisely the gap async runtimes exist to fill — and article 6 will measure whether they actually do.

Everywhere. Name your threads. It costs one Builder::name() call, and it puts a meaningful string into /proc/<tid>/comm, ps, top, perf, and your crash messages.

---

Conclusion

The hypothesis was: Rust creates a native OS thread that Linux independently tracks and schedules. That held up. The useful part was everything the investigation added around it.

What we established, all of it observed rather than assumed:

• There is no thread syscall. clone3 creates a task; the sharing flags — CLONE_VM, CLONE_THREAD, CLONE_FILES, CLONE_SIGHAND — are the entire difference between a thread and a process.
• The stack is allocated in userspace by glibc before the kernel is involved: a PROT_NONE mmap of 2 MiB + 4 KiB, then mprotect to open everything except the guard page.
• 2 MiB is Rust's choice, from DEFAULT_MIN_STACK_SIZE, overridable via RUST_MIN_STACK or Builder::stack_size — confirmed by watching the mmap size change.
• join() is a futex wait, armed by CLONE_CHILD_CLEARTID. The kernel clears the child's TID on exit and wakes the waiter.
• A thread costs about 2,069 kB of virtual memory and about 10 kB of resident memory. Those two numbers are two orders of magnitude apart, and confusing them is a common operational error.
• A thread costs about 37 us to create at p50, with a 2.7 ms tail, at roughly 18,000 threads/sec on two cores.
• Beyond core count, throughput holds and latency degrades linearly — 32x for 64 threads on 2 cores — with involuntary context switches scaling to match.
• The practical ceiling on thread count comes from max_map_count, pid_max, RLIMIT_NPROC and overcommit, not from RAM. Each thread burns two VMAs.

And one thing I had wrong going in: I assumed the expensive part of thread-per-work was creation. The reuse benchmark says the expensive part is blocking and waking. Creation is merely expensive as well. That distinction is going to shape most of the rest of this series.

Next

Article 2: What Does Arc Actually Do? Atomic Reference Counting Under the Hood. We now have multiple tasks sharing one mm_struct — byte-identical /proc/<tid>/maps, confirmed above. The immediate question is how ownership of anything inside that shared address space is tracked safely. That means looking at what Arc actually contains in memory, what an atomic increment compiles to on x86-64, why the Drop path needs a stronger ordering than the clone path, and what the counter does under contention.

Then article 3 puts a lock around it and measures what contention costs, and article 4 follows the blocking path back to the same futex syscall we already saw here.

---

Appendix: Environment and Reproducibility

Every number in this article came from this machine:

kernel:  6.18.5-fc-v20 (x86_64)
distro:  Ubuntu 24.04.4 LTS
rustc:   1.95.0 (59807616e 2026-04-14)
glibc:   2.39-0ubuntu8.7
strace:  6.8
cpu:     Intel(R) Xeon(R) Processor @ 2.10GHz, 2 cores
memory:  8216168 kB total
RLIMIT_STACK: 8192 kB

Programs used, all single-file, all buildable with rustc -O:

File	Purpose
hello_thread.rs	minimal one-thread program, for the syscall trace
four_threads.rs	four named parked threads, for /proc inspection
overflow.rs	deliberate stack overflow, for the guard page
cost.rs	spawn+join latency percentiles and live-thread footprint
stacksize.rs	VmSize/VmRSS per thread across stack sizes
createcost.rs	creation-only latency and creation throughput
reuse.rs	spawn-per-item vs. channel handoff
oversub2.rs	per-thread latency and preemptions vs. thread count

Commands:

$ rustc -O -g hello_thread.rs -o hello_thread
$ strace -f -c ./hello_thread
$ strace -f -o hello_full.strace ./hello_thread
$ strace -f -e trace=none ./overflow          # signals only
$ MALLOC_ARENA_MAX=1 ./cost 200
$ MALLOC_ARENA_MAX=1 ./oversub2

Caveats I want to be explicit about, because they change the numbers:

• Two cores. The oversubscription curve is steeper here than on a 32-core host, and absolute spawn latency is higher. The shape transfers; the values do not.
• Virtualised environment. Syscall and context-switch costs under a hypervisor differ from bare metal.
• glibc-specific. The malloc arena behaviour — those 128 MiB PROT_NONE reservations — is a glibc implementation detail. musl behaves differently, and a Rust binary using a different global allocator will not show it at all.
• Instant::now() around spawn includes timing-call overhead. At tens of microseconds that is negligible, but it is not zero.
• /proc/self/status counters are per-thread, not per-process. My first oversubscription run read them from the main thread and reported almost no context switches, which is why the final version reads /proc/thread-self/status from inside each worker. Worth knowing before you trust a similar measurement of your own.

If you re-run these on a machine with more cores, I would be interested in where the latency curve bends.

Thanks for Reading

If you got this far — thank you. That was a long way down for one line of Rust.

This series is me working through layers I have used for years without ever inspecting, and writing it up as I go rather than after I have it all figured out. That means the investigations are honest about what surprised me, and it means some of them will be wrong. If you spot something wrong here, I would much rather hear it now than leave it standing while nine more articles get built on top of it.

Next up — Article 2: What Does Arc Actually Do? Atomic Reference Counting Under the Hood.

• GitHub: GITHUB_URL
• LinkedIn: LINKEDIN_URL