Idiomatic Concurrency in Rust: Stop Writing Accidental Microservices

The Problem

A common pattern in Rust async applications looks like this:

• Use the Tokio runtime.
• Break the app into small “microservices” that each run in their own task.
• Give each service a run function that loops over a tokio::select!.
• Add a branch for CancellationToken::cancelled().
• Wire the root token to ctrl_c.
• Let services communicate through channels: mpsc, oneshot, watch, broadcast.
• When one service fails, bubble the error up and let the whole application fail.

At first this feels clean. Each component is isolated. Each has a lifecycle. Each has a cancellation path.

But then the bugs start.

One service returns Err. Nothing cancels its siblings. The sibling is still waiting on a channel receive. The sender is gone. The receiver parks forever. The app does not crash. It does not shut down. It just hangs.

The developer adds more select! branches. More cancellation checks. More boilerplate. The same failure mode appears somewhere else.

The real issue is not Tokio. The real issue is lifecycle ownership.

This is not microservices. It is a modular monolith with in-process tasks. That label matters because it changes the failure model. There is no network partition. There is no independent fate. There is shared fate. If one component dies, the process is still alive, and the rest of the system may be stuck waiting for it.

tokio::spawn detaches. Dropping a JoinHandle does not cancel the task. A failed or panicked task does not automatically cancel its siblings. A select! inside one component does not fix global lifecycle semantics.

The result is exactly the pain described: redundant code, unhandled paths, stopped services, and other services waiting forever for a response that will never come.

The Solution

Treat the application as a supervised set of in-process tasks in a monolith.

Own task lifetimes. Define failure policy. Make every long-lived await cancel-aware. Keep simple things simple.

1. Use structured concurrency

Use JoinSet to own child tasks. Use TaskTracker if you need to track graceful shutdown completion.

Do not detach tasks unless you truly mean fire-and-forget.

Rust and Tokio do not have built-in structured concurrency. JoinSet and TaskTracker approximate it. TaskTracker is for close() + wait() — “everyone is done.” It does not propagate failure. Pair it with JoinSet when you care why things ended.

2. Use one root cancellation token

Create one root CancellationToken from ctrl_c. Give each component a child token.

Child tokens propagate parent → child only. A failing child does not cancel the root. The supervisor must observe the failure via join_next and call root.cancel().

3. Decide failure policy per component

Fail-fast? Restart with backoff? Degrade? Shut down cleanly?

Only restart if the component is safe to restart. Otherwise, bubble the error up and shut down the whole application deliberately.

4. Make every long-lived await cancel-aware

This is the actual fix for “nobody responding.”

CancellationToken does not wake a task parked in rx.recv().await. If a service is blocked on a channel receive that is not racing the token, the drain loop hangs forever.

Every long-lived await must either:

• select! on token.cancelled(), or
• rely on sender-drop so recv() returns Err / None.

Owners should drop their senders during shutdown.

5. Use the right channel for the job

• watch for state/config
• broadcast for events
• mpsc for commands
• oneshot for replies
• bounded channels for backpressure

Unbounded channels hide backpressure bugs.

6. Do not make everything a task

Only spawn when you need independent lifetime, async I/O, or state isolation. If it is just a function call, call it. Message passing everywhere makes simple things harder.

7. Know your runtime defaults

#[tokio::main] is multi-threaded by default. worker_threads defaults to the number of CPUs. #[tokio::test] is current-thread by default.

More worker threads do not make blocking calls safe. Use spawn_blocking for blocking work.

Example Shape

use std::time::Duration;
use tokio::task::JoinSet;
use tokio_util::sync::CancellationToken;

async fn service_a(token: CancellationToken) -> anyhow::Result<()> {
    // Every long-lived await should be cancel-aware.
    loop {
        tokio::select! {
            _ = token.cancelled() => return Ok(()),
            // _ = rx.recv() => { ... }
        }
    }
}

async fn service_b(token: CancellationToken) -> anyhow::Result<()> {
    loop {
        tokio::select! {
            _ = token.cancelled() => return Ok(()),
            // _ = rx.recv() => { ... }
        }
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let shutdown = CancellationToken::new();
    let mut tasks = JoinSet::new();

    tasks.spawn(service_a(shutdown.child_token()));
    tasks.spawn(service_b(shutdown.child_token()));

    let mut first_error: Option<anyhow::Error> = None;

    tokio::select! {
        _ = tokio::signal::ctrl_c() => {}
        res = tasks.join_next() => {
            match res {
                Some(Ok(Ok(()))) => {
                    // Normal exit. For a forever-service this is unexpected.
                    // For finite work, you may want to continue instead of cancelling.
                }
                Some(Ok(Err(e))) => first_error = Some(e),
                Some(Err(join_err)) => first_error = Some(join_err.into()),
                None => {}
            }
        }
    }

    shutdown.cancel();

    // Drain remaining tasks. Don't use `?` here; collect/log instead.
    let drain = async {
        while let Some(res) = tasks.join_next().await {
            match res {
                Ok(Ok(())) => {}
                Ok(Err(e)) => {
                    if first_error.is_none() {
                        first_error = Some(e);
                    }
                }
                Err(join_err) => {
                    if first_error.is_none() {
                        first_error = Some(join_err.into());
                    }
                }
            }
        }
    };

    if tokio::time::timeout(Duration::from_secs(5), drain)
        .await
        .is_err()
    {
        tasks.abort_all();
        while tasks.join_next().await.is_some() {}
    }

    if let Some(e) = first_error {
        return Err(e);
    }

    Ok(())
}

Notes:

• JoinSet forces one output type across all spawns. Heterogeneous services need a common error type like anyhow::Error or a mapped enum.
• After cancellation, a task may legitimately return Err. Log it instead of double-reporting the first error.
• If you need actor semantics, crates like ractor exist. Only reach for them if you actually need actors.

Rules of Thumb

• Rename the mental model: supervised tasks in a monolith, not microservices.
• Own task lifetimes with JoinSet.
• Use one root CancellationToken and child tokens.
• Remember: cancellation flows parent → child, not child → parent.
• Make every long-lived await cancel-aware.
• Define failure policy per component.
• Use bounded channels for backpressure.
• Do not spawn everything.
• Keep plain function calls as plain function calls.
• Heartbeats are for separate processes. In-process, await task completion.
• Coroutines will not solve lifecycle or failure semantics.

One-Liner

Stop writing accidental microservices: treat Tokio tasks as supervised components in a monolith, own their lifetimes with JoinSet, cancel through a root token, and make every await cancel-aware.