Vincent Tran

Posted on Sep 16 Originally published at 0xgosu.dev on Sep 15

Fast Tokio Is a Scheduling Budget, Not a Bag of Tricks

#ai #productivity #software

An async service can have plenty of spare CPU and still feel slow. The socket becomes readable, the corresponding future is ready, yet the task does not run for another few milliseconds. That gap is where many Tokio performance problems live.

Tokio is not a faster operating system thread. It is a cooperative scheduler that multiplexes many Rust futures over a small set of worker threads. Each future makes progress only when Tokio calls Future::poll; it keeps the worker until it returns Pending, completes, or otherwise yields. Performance therefore depends on the shape of the work between yield points, the number of runnable tasks, and the shared resources those workers must coordinate through.

Russell Cohen’s article on principles for fast Tokio applications captures the central trade-off: split work to improve latency, batch work to improve throughput. This guide develops that idea into a way to reason about production systems, including the places where familiar async advice becomes too simple.

Start with the delay users actually feel

Do not optimize a long poll merely because a tracing tool colored it red. Start from a service objective: request p99, queue time, time to first byte, missed market-data deadline, or another outcome someone cares about. Then move inward until you find the component consuming that budget.

The most useful runtime signal is often scheduling latency : the time from a task becoming ready to the runtime polling it. Tokio can record a schedule-latency histogram when the runtime is built with the relevant metrics enabled. The runtime builder defaults to a compact linear histogram, while a logarithmic configuration can cover a wider range with controlled error.

Scheduling latency is a symptom, not a verdict. A high value can mean one worker spent too long in a poll, all workers blocked on the same lock, queues grew beyond capacity, or the operating system delayed a worker thread. Conversely, a long poll may be harmless when the runtime has spare workers and the service remains within its latency target.

Useful evidence comes in groups:

Compare p50 with p99. A widening tail often points to unfairness or periodic contention.
Correlate schedule latency with request latency rather than reading it alone.
Inspect worker busy time, local and global queue depth, blocking-queue depth, forced yields, and steal operations through Tokio’s runtime metrics.
Measure observability overhead. Histograms, clocks, spans, and locks also consume the budget being studied.

“Dark — Tokio workers prefer local work, periodically check shared sources, and steal when another worker has a backlog. Ready does not mean running immediately.

The scheduler is a set of finite budgets

The current multi-thread runtime normally creates one worker per available CPU. Each worker has a local queue, and the runtime also has a global queue. Work woken from a worker tends to stay local; work scheduled from outside a worker enters the global queue. A worker with no local work can steal half of another worker’s queued tasks.

Tokio’s runtime documentation describes two details that make the trade-off concrete. A local queue currently holds up to 256 tasks before half spill to the global queue. The scheduler checks the global queue on an interval chosen to target roughly 10 milliseconds between checks unless configured otherwise. It also checks for I/O and timer events after a number of scheduler ticks. These are implementation details that may change, but they reveal the underlying costs: checking shared state more often improves responsiveness and requires more synchronization.

Tokio also applies a cooperative operation budget to many of its resources. An always-ready socket or channel cannot consume an unlimited number of Tokio operations in one turn; once the budget is exhausted, those resources report Pending so the task returns to the scheduler. The cooperative scheduling module explains both the safeguard and the dangerous unconstrained escape hatch.

That safeguard cannot interrupt arbitrary user code. A CPU loop, a blocking system call, or a third-party future that never yields can still occupy its worker. Nor does .await automatically imply fairness: if the awaited operation is immediately ready, the future may continue in the same poll.

Split for latency

Imagine a server reading pipelined requests from an in-memory connection buffer. Every read_frame().await is immediately ready. One connection can drain hundreds of frames before another ready connection gets a turn. Total throughput may look healthy while one client’s queue sits behind another client’s batch.

Adding an explicit tokio::task::yield_now().await after every request gives the scheduler more opportunities to rotate tasks. It can dramatically improve tail latency under contention. Yet yield_now intentionally promises less than “another task runs next.” Tokio may poll the same task again, and combinators such as select! can prevent the yield from propagating all the way to the executor. Correctness must never depend on a particular polling order.

The useful pattern is a cooperation budget owned by the application:

let mut requests_since_yield = 0;

loop {
    let frame = connection.read_frame().await?;
    handle(frame).await?;

    requests_since_yield += 1;
    if requests_since_yield == 4 {
        tokio::task::yield_now().await;
        requests_since_yield = 0;
    }
}

Four is illustrative, not universal. Tune the budget with a representative load test. Smaller batches create more scheduling opportunities and typically improve fairness; larger batches amortize scheduler, queue, and cache costs. The correct point depends on the service’s latency target and work per item.

Alice Ryhl’s explanation of blocking in async Rust offers 10–100 microseconds between awaits as a useful scale, not a law. Treat it as a prompt to measure. A 200-microsecond poll may be disastrous in a microsecond-sensitive service and irrelevant in a lightly loaded batch processor.

Batch for throughput

Yielding has a cost. So do spawning a task, transferring ownership through a queue, waking a worker, stealing work, and moving a blocking operation to another thread. A service that creates one task for every ten microseconds of CPU work can spend a surprising fraction of its time managing work instead of doing it.

Filesystem work is a common example. Without a native asynchronous path for a given operation, Tokio’s filesystem APIs use the blocking pool. Sending thousands of tiny metadata operations one by one makes the global handoff visible. If the operations are naturally related, move the whole sensible unit into one blocking closure, or use a dedicated thread for a long-lived synchronous workflow.

The official spawn_blocking documentation makes three constraints explicit:

The blocking pool has a large upper thread limit because it must accommodate blocking I/O.
Once that limit is reached, new blocking jobs wait in a queue.
A running blocking closure cannot simply be aborted; it must finish or cooperate with cancellation itself.

For CPU-heavy jobs, a large blocking pool can oversubscribe the machine. Bound that parallelism with a semaphore or use a CPU-oriented executor such as Rayon. For persistent loops, use a dedicated OS thread. For short synchronous operations that belong together, batch them.

The same reasoning applies to ordinary async tasks: spawning 10,000 tasks does not create 10,000 units of physical parallelism. It creates 10,000 independently scheduled state machines competing for worker time and downstream resources.

Put a limit before the scarce resource

Async makes concurrency cheap enough to become dangerous. A fan-out can open thousands of database or object-store requests long before the remote system signals distress. By then, memory, file descriptors, connection pools, and retry queues may all be expanding.

A Semaphore turns a downstream capacity into an explicit admission boundary. Acquire the permit before spawning or accepting more work when you want the limit to bound task count as well as I/O concurrency. Move an owned permit into the task, and release it when the resource is truly finished.

The permit count is not “number of cores.” It is the concurrency at which the complete system meets its objective. Find it experimentally. Increase concurrency until throughput stops improving or tail latency, errors, or resource use rise sharply; then leave headroom for bursts and failure recovery.

Bounded channels are useful for the same reason. They make pressure travel upstream instead of accumulating as an invisible heap of runnable tasks. A queue should represent a deliberate burst budget, not an alternative to capacity planning.

A mutex can freeze an entire runtime

A synchronous mutex is acceptable inside async code when its critical section is tiny and contention stays low. Tokio’s shared-state guide even prefers std::sync::Mutex for a simple in-memory map because an async mutex costs more and is designed for guards that must survive an .await.

The failure mode is contention. Suppose every request records a metric through one global mutex, while a once-per-minute flush holds that mutex during formatting or I/O. Worker after worker reaches the same lock and blocks at the OS-thread level. Eventually all runtime workers are asleep behind one critical section; there is no free worker left to steal the otherwise runnable tasks.

Switching mechanically to tokio::sync::Mutex avoids blocking the worker while waiting, but it serializes access and adds scheduling overhead. Better options are often:

Copy a small snapshot and release the lock before expensive work.
Shard independent state across several locks.
Give an I/O resource to one task and communicate with it through a bounded channel.
Keep lock-taking inside a short synchronous method so a guard cannot cross .await.

The right primitive follows the protected operation. A few hash-map instructions fit a short synchronous lock. A database connection or stream that performs async I/O often fits an owner task and message passing.

Isolate work when priorities are genuinely different

Tokio can move queued tasks among its workers, but it cannot make the operating system schedule a worker promptly. On an overloaded host, a worker that Tokio unparks may wait milliseconds before the kernel runs it. If the service’s p99 target is itself only a few milliseconds, that delay consumes the budget before application code starts.

Separate latency-sensitive workers from noisy processes and background threads with CPU affinity or cgroup CPU sets where the deployment environment supports them. Reserving a core for logging, compression, or another runtime can be faster than allowing every subsystem to contend for every core.

For stronger application-level separation, build two runtimes: one for request-path work and another for background jobs. Tokio’s builder supports choosing worker counts and running setup in on_thread_start. OS-level priority and affinity still determine whether that logical boundary receives real CPU time.

Multiple runtimes add operational cost. Handles must be passed explicitly, shutdown must cover both, and resources tied to one runtime cannot be treated as universally movable. Use the boundary when workloads have demonstrably different latency classes, not as a default architecture.

At the far edge, briefly spinning instead of yielding can avoid a wake-up delay. It also burns a core, raises power use, steals capacity from neighbors, and can trigger thermal limits. That is a controlled microsecond-latency technique for dedicated hardware, not a general Tokio optimization.

A disciplined tuning loop

The practical sequence is simple, even when the system is not:

Define the user-facing metric and reproduce the bad tail under representative load.
Measure scheduling latency and queue depth with sampling or bounded observability overhead.
Classify the delay: long polls, excessive task granularity, blocking-pool pressure, lock contention, unbounded fan-out, or OS scheduling.
Change one boundary: yield budget, batch size, concurrency permit count, critical section, thread placement, or runtime isolation.
Repeat the same workload and compare throughput, p50, p99, resource use, and failure behavior.

There is no contradiction between “never block the executor” and “a longer poll can be faster.” The first is a safe default for systems whose contention is not yet understood. The second is a measured decision to spend more worker time on one task because the saved coordination outweighs the lost fairness.

Fast Tokio applications do not maximize yielding, batching, task count, or worker count. They assign finite budgets to each: how long a task may hold a worker, how much work crosses a shared queue, how many operations may hit a dependency, and which workloads may compete for a core. Once those budgets are visible, async performance stops looking like folklore and becomes ordinary systems engineering.

DEV Community