tuning tokio thread pool to cut latency spikes in high‑throughput services

in a recent microservice that ingests 200k requests /s, we saw occasional latency outliers up to 200 ms while the 99th‑percentile stayed under 5 ms. the spike source turned out to be the tokio runtime's default thread‑pool configuration, which silently oversubscribes the CPU and induces cache line contention.

The Bottleneck

tokio builds a work‑stealing scheduler on top of a fixed number of worker threads. by default it uses the number of logical cores, which on a hyper‑threaded 8‑core box means 16 workers. each worker pulls tasks from a shared deque; when many short tasks compete, the deque's head pointer is updated concurrently, causing false sharing on the cache line that holds the queue metadata. the result is a series of cache‑miss stalls that amplify latency under load.

A Minimal Fix

pinning the runtime to a core‑count that matches physical cores and padding the deque structures eliminates the false sharing. the code below shows a tuned builder that also raises the stack size to avoid guard‑page faults when deep recursion occurs.

use tokio::runtime::{Builder, Runtime};

let rt: Runtime = Builder::new_multi_thread()
    .worker_threads(8)               // match physical cores, not logical
    .max_blocking_threads(32)        // bounded blocking pool
    .thread_stack_size(2 * 1024 * 1024) // 2 MiB stacks for heavy recursion
    .enable_all()
    .build()
    .expect("failed to build runtime");

The runtime now creates exactly eight workers; each worker’s local queue resides in its own cache line because the scheduler pads the queue header to a full 64‑byte line. this reduces inter‑worker cache traffic and brings the 99th‑percentile back to sub‑5 ms even under sustained peak load.

Trade‑offs & Edge Cases

• More workers → higher parallelism but also more context switches and larger total stack memory (8 workers × 2 MiB ≈ 16 MiB). on memory‑constrained containers this can push the RSS close to limits and trigger OOM.
• Increasing max_blocking_threads helps when legacy blocking I/O is unavoidable, yet each extra thread adds a kernel scheduler overhead that can negate the latency gains.
• Thread affinity: on Linux you can bind each worker to a distinct physical core via taskset or numactl. without it, the OS may migrate workers across cores, re‑introducing cache line bounce.
• Hyper‑threading: keeping workers equal to physical cores avoids the “two‑threads‑per‑core” contention, but if your workload is I/O bound you might benefit from the extra logical threads; profiling is the only way to decide.

When It Still Fails

if latency spikes persist, look for hidden blocking calls inside async tasks (e.g., std::fs::read_to_string). such calls fall back to the blocking pool, which can starve the worker pool if max_blocking_threads is too low. also verify that the executor’s yield_now is used in long loops to give other workers a chance to steal work.

---

how do you balance thread‑pool size, stack allocation, and cache‑line padding in your high‑throughput async services? share the configuration that gave you the most stable latency curve.