Mastering True Parallelism in Rust

Beyond await: Mastering True Parallelism in Rust (and Why It’s Not Python or JavaScript)

We’ve all heard the meme: Rust is “blazingly fast.” But when you audit a real-world, production Rust codebase, you often find a surprising reality. The code uses async/await everywhere, shares a tuned database pool, and avoids obvious blocking calls—yet its throughput ceiling is abysmal.

Why? Because writing await does not automatically make your code parallel.

Recently, we audited a document-processing pipeline (codenamed MenuOps) that was taking 30–90 seconds per document. The codebase had tokio = { features = ["full"] } enabled, but a search revealed a stark truth:

• tokio::join!: 4 uses (expected: ~15)
• tokio::spawn: 6 uses (expected: ~20)
• tokio::task::spawn_blocking: 0 uses
• JoinSet / Semaphore: 0 uses

The worker—the one component that should be maximally parallel—was running fully serially.

If you’re coming from JavaScript or Python, Rust’s async model offers incredible power, but it demands intent. Here’s what we learned about fixing parallelism in Rust, and how it compares to the ecosystems you might already know.

---

1. The CPU-Bound Trap: Don’t Stall the Executor

The Rust Reality

Tokio’s async runtime uses a thread pool (by default, one thread per CPU core). If you run a CPU-bound task on an async worker thread, you block that entire thread. In our audit, Argon2 password hashing was running inline on the async runtime. On an 8-core machine, 8 concurrent logins would block all 8 worker threads for ~150ms each. Every other request—including static assets and /health checks—stalled behind them.

The Fix: Use spawn_blocking.

// Before: Blocks the async worker thread
let password_hash = hasher::encode(&payload.password)?;

// After: Offloads to Tokio's dedicated blocking thread pool
let pw = payload.password.clone();
let password_hash = tokio::task::spawn_blocking(move || hasher::encode(&pw))
    .await
    .map_err(|e| format!("hash task panicked: {e}"))??;

How Python & JS Handle This

• JavaScript: Node.js is strictly single-threaded for JavaScript execution. CPU-bound work (like crypto or image processing) must be offloaded to C++ addons, Web Workers, or worker_threads. If you don’t, you block the entire event loop.
• Python: The Global Interpreter Lock (GIL) means that even with asyncio, CPU-bound Python code will block the event loop. You must use asyncio.to_thread() or a ProcessPoolExecutor to achieve true parallelism.
• The Rust Difference: Rust has no GIL, and its runtime is multi-threaded by default. However, it trusts you completely. It won’t magically move CPU-bound work off the async threads for you. You must explicitly declare it with spawn_blocking.

---

2. The I/O Trap: Fan-Out, Don’t Line-Up

The Rust Reality

The audit found a function, collect_knowledge_text_for_quiz, that fetched up to 5 documents. For each document, it sequentially:

1. Queried the DB for content.
2. If empty, queried for chunks.
3. If empty, queried document versions.
4. If empty, downloaded from R2 and called the Gemini API (8–30s).

Because this was written as a for loop with await inside, 5 documents needing Gemini took ~150 seconds.

The Fix: Collect the futures and execute them concurrently.

let doc_futures: Vec<_> = doc_ids.iter().map(|doc_id| {
    let db = db.clone();
    let r2 = r2.clone();
    async move {
        // ... existing fallback chain for ONE document
    }
});

// Execute all 5 document fetches concurrently
let results = futures::future::join_all(doc_futures).await;

Result: 150s → ~30s.

How Python & JS Handle This

• JavaScript: You’d naturally reach for Promise.all(docIds.map(id => fetchDoc(id))). It’s idiomatic and concurrent.
• Python: You’d use asyncio.gather(*[fetch_doc(doc_id) for doc_id in doc_ids]).
• The Rust Difference: Rust’s join_all or tokio::join! works similarly, but Rust’s strict ownership model means you have to explicitly .clone() your shared state (like db and r2 clients) to move them into the async blocks. This feels verbose at first, but it guarantees no data races at compile time.

---

3. The Concurrency Trap: Unbounded Fan-Out is a Footgun

The Rust Reality

Once you discover join_all or tokio::spawn, the temptation is to use it everywhere. But unbounded concurrency is its own outage. Spawning 500 concurrent Gemini API calls will instantly trigger rate limits (HTTP 429), and 500 concurrent spawn_blocking tasks allocating 19 MiB each will OOM your server (~10 GB).

The Fix: Bound your concurrency with a Semaphore.

use std::sync::Arc;
use tokio::sync::Semaphore;

let sem = Arc::new(Semaphore::new(5)); // Max 5 concurrent digitisation calls

for doc in docs {
    let permit = sem.clone().acquire_owned().await.unwrap();
    let doc = doc.clone();

    tokio::spawn(async move {
        let _permit = permit; // Held for the entire task, released on drop
        // ... perform Gemini digitisation ...
    });
}

How Python & JS Handle This

• JavaScript: Native Promise.all is unbounded. Developers rely on third-party libraries like p-limit or async to constrain concurrency.
• Python: asyncio.Semaphore exists, but it’s often overlooked in favor of naive gather calls, leading to similar rate-limit explosions.
• The Rust Difference: tokio::sync::Semaphore is built into the standard async ecosystem. Because Rust forces you to manage the Permit object (which releases automatically when dropped, even on panic), it makes bounded concurrency safe and ergonomic by default.

---

4. The Worker Loop Fallacy: Serial Policy in a Concurrent World

The Rust Reality

The biggest throughput win in the audit was fixing the background worker. It looked like this:

loop {
    tokio::select! {
        _ = shutdown.cancelled() => return,
        result = runner.run_once() => match result {
            Ok(true) => { tokio::task::yield_now().await; }
            // ...
        }
    }
}

This processed one document at a time, platform-wide. The irony? The database query (FOR UPDATE SKIP LOCKED) was already perfectly designed for concurrent consumers. The serialization was a purely artificial policy choice in the Rust code.

The Fix: Use a JoinSet with a Semaphore to process N jobs concurrently.

pub async fn start_loop(runner: Arc<Self>, shutdown: CancellationToken) {
    const CONCURRENCY: usize = 4;
    let permits = Arc::new(Semaphore::new(CONCURRENCY));

    loop {
        if shutdown.is_cancelled() { return; }

        // Acquire a permit before claiming
        let permit = permits.clone().acquire_owned().await.unwrap();

        let Ok(Some(job)) = runner.claim_next_job(&runner.worker_id).await else {
            drop(permit);
            tokio::time::sleep(Duration::from_secs(2)).await;
            continue;
        };

        let runner = runner.clone();
        tokio::spawn(async move {
            let _permit = permit; // Released when task ends
            let _ = runner.process_claimed(job).await;
        });
    }
}

Note: We picked N=4, not 8, because the worker shares the database connection pool (default 15 connections) with the HTTP server. Parallelism must respect downstream resource limits.

How Python & JS Handle This

• JavaScript: Libraries like BullMQ or Agenda abstract this entirely. You define a worker, set a concurrency number, and the library handles claiming, locking, and parallel execution.
• Python: Celery or RQ handle this via separate worker processes. You don’t write the loop; you write the task, and the broker distributes it.
• The Rust Difference: Rust gives you the primitives (SKIP LOCKED, Semaphore, JoinSet), not the framework. You build the worker loop yourself. This requires more boilerplate, but it means zero hidden magic, no mysterious broker dependencies, and total control over resource allocation.

---

5. Bonus Lesson: Parallelism Multiplies Your Mistakes

When you finally unlock true parallelism, you also unlock the ability to fail much faster and more expensively.

In the MenuOps audit, fanning out Gemini API calls revealed two critical flaws that were hidden by the serial execution:

1. Unbounded Prompts: Concatenating entire documents into prompts without character caps meant that when fanned out, a single 40MB document could trigger massive, billable, and ultimately rejected API calls. Solution: Hard cap context strings before they reach the LLM.
2. Blind Retries: The retry logic was retrying terminal errors (like HTTP 400 or malformed JSON) up to 3 times. Fanned out across 5 documents and 3 batch embeds, a single bad prompt resulted in 18 wasted API calls. Solution: Implement a RetryableError trait that explicitly distinguishes between transient (429, 503) and fatal (400, MAX_TOKENS truncation) errors.

Parallelism doesn’t fix bad logic; it just executes bad logic concurrently.

---

Summary: The Rust Parallelism Mindset

Concept	JavaScript	Python	Rust
Concurrency Model	Single-threaded Event Loop	Asyncio (GIL-bound for CPU)	Multi-threaded Runtime (Tokio)
CPU-Bound Work	worker_threads / C++ Addons	ProcessPoolExecutor / to_thread	tokio::task::spawn_blocking
Fan-Out I/O	Promise.all	asyncio.gather	tokio::join! / JoinSet / join_all
Bounded Concurrency	p-limit (3rd party)	asyncio.Semaphore	tokio::sync::Semaphore (Built-in)
Philosophy	"It just works (until it blocks)"	"Explicit is better than implicit"	"You get exactly what you ask for."

Rust doesn’t give you "free" parallelism like JavaScript promises (which is often a lie due to the single thread), nor does it trap you like Python’s GIL. It gives you a scalpel.

If you write a for loop with await inside, Rust will execute it serially, exactly as you asked. But when you deliberately reach for spawn_blocking, JoinSet, and Semaphore, Rust will unleash the full, terrifying, blazingly fast power of your hardware—safely, predictably, and without data races.