Your Async Python Service Is Lying to You About Concurrency

Your service handles 50 requests/second in staging and collapses at 200 in production. CPU sits at 15%, memory is fine, and asyncio is supposedly giving you thousands of concurrent connections. What actually happened: one synchronous call inside a coroutine blocked the entire event loop, every in-flight request stalled behind it, and your connection pool drained while tasks waited on a lock they'd never acquire. No exception. No traceback. Just p99 latency going vertical and a health check that eventually times out.

These three failure modes—event loop blocking, cancellation that doesn't cancel, and pool exhaustion—account for the majority of async Python incidents that reach production. They're hard to catch because the code looks correct, the tests pass, and the symptoms look like infrastructure problems rather than logic problems.

Event loop blocking: the 8ms that costs you 40 seconds

asyncio is cooperative. The event loop runs one coroutine at a time until that coroutine awaits something. If a coroutine calls blocking code, nothing else runs—not other requests, not the loop's own timers, not cancellation.

Here's the trap. This looks fine:

import asyncio
import hashlib
import json
from fastapi import FastAPI

app = FastAPI()

@app.post("/sign")
async def sign(payload: dict):
    body = json.dumps(payload, sort_keys=True).encode()
    # 200ms of pure-Python CPU work, no awaits
    digest = hashlib.pbkdf2_hmac("sha256", body, b"salt", 600_000)
    return {"sig": digest.hex()}

pbkdf2_hmac at 600k iterations takes roughly 150–250ms on a modern core. During that window the event loop is frozen. At 200 req/s across 4 workers, you don't get 200 concurrent signatures—you get a serialized queue. Each request waits behind every other request's CPU work. Throughput collapses to workers / 0.2s = 20 req/s, and p99 latency scales linearly with queue depth. I've watched this exact pattern turn a 30ms p50 into a 40-second p99 under load.

The fix is to move CPU-bound work off the loop. Two options, with different trade-offs:

import asyncio
from concurrent.futures import ProcessPoolExecutor

# ProcessPoolExecutor, not ThreadPoolExecutor: PBKDF2 is CPU-bound and holds
# the GIL, so threads would still serialize. Processes give real parallelism
# but pay pickling cost per call.
_pool = ProcessPoolExecutor(max_workers=4)

def _derive(body: bytes) -> bytes:
    return hashlib.pbkdf2_hmac("sha256", body, b"salt", 600_000)

@app.post("/sign")
async def sign(payload: dict):
    body = json.dumps(payload, sort_keys=True).encode()
    loop = asyncio.get_running_loop()
    digest = await loop.run_in_executor(_pool, _derive, body)
    return {"sig": digest.hex()}

This is correct but not free. run_in_executor with a process pool pickles arguments and return values, which matters when payloads are large. It also adds a queueing layer: if all 4 workers are busy, the 5th call waits. Size the pool to os.cpu_count() and put a bounded semaphore in front of it if you need backpressure rather than unbounded latency growth.

For I/O-bound blocking—requests, psycopg2, blocking file reads—asyncio.to_thread is usually the right tool:

# to_thread is a thin wrapper over run_in_executor with the default
# ThreadPoolExecutor, whose default max_workers is min(32, cpu+4).
# That default is often too small for I/O fan-out; set it explicitly.
import asyncio
from concurrent.futures import ThreadPoolExecutor

io_pool = ThreadPoolExecutor(max_workers=64, thread_name_prefix="io")

async def fetch_legacy(url: str) -> bytes:
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(io_pool, blocking_fetch, url)

Detection: if you can't explain why p99 is high, instrument loop lag. A 20-line watchdog catches nearly every instance of this bug:

import asyncio, time, logging

async def loop_lag_watchdog(threshold_ms: float = 50.0):
    """Log when the event loop is starved. A healthy loop wakes within ~1ms;
    sustained lag above the threshold means something is blocking."""
    while True:
        start = time.perf_counter()
        await asyncio.sleep(0.01)
        lag_ms = (time.perf_counter() - start - 0.01) * 1000
        if lag_ms > threshold_ms:
            logging.warning("event loop lag: %.1fms", lag_ms)

Export that as a Prometheus histogram and alert on it. It's the single highest-signal metric for async services.

Cancellation: CancelledError is not a cleanup signal

task.cancel() doesn't kill a coroutine. It schedules a CancelledError to be raised at the next await point. If your coroutine is inside a try/except Exception, the CancelledError gets swallowed—and in Python 3.8+ CancelledError inherits from BaseException precisely so that bare except Exception won't catch it. But plenty of code still does this:

async def handler():
    try:
        await do_work()
    except Exception:          # CancelledError is BaseException, so this is ok...
        logger.exception("work failed")
    finally:
        await flush_metrics()  # <-- CancelledError can be raised HERE

The finally block runs during cancellation. If flush_metrics() awaits and the cancellation is re-delivered (which it will be, because the task is still marked cancelled), the await raises CancelledError again, the flush never completes, and the task exits with a partially-flushed state. Worse, if flush_metrics itself catches broadly, you've now suppressed cancellation entirely and the task keeps running after its parent gave up.

The correct pattern is to shield cleanup you must complete, and to re-raise:

async def handler():
    try:
        await do_work()
    except asyncio.CancelledError:
        # Do NOT log this as an error; it's normal control flow.
        raise
    except Exception:
        logger.exception("work failed")
        raise
    finally:
        # shield() prevents *this* await from being cancelled, so cleanup
        # finishes. Bound it with wait_for so a hung flush can't wedge shutdown.
        try:
            await asyncio.wait_for(
                asyncio.shield(flush_metrics()), timeout=2.0
            )
        except (asyncio.TimeoutError, asyncio.CancelledError):
            logger.warning("metric flush skipped during shutdown")

The subtle part: asyncio.shield protects the inner task from outer cancellation, but the outer await still raises CancelledError. That's why it's wrapped. This is the difference between a clean shutdown in 2 seconds and a SIGKILL after your orchestrator's 30-second grace period.

A second cancellation trap: asyncio.gather without return_exceptions. If one child raises, gather propagates immediately but the sibling tasks keep running—they are not cancelled. You leak tasks and their resources:

# Leaks: if fetch_a raises, fetch_b keeps running detached.
results = await asyncio.gather(fetch_a(), fetch_b())

# Correct: cancel siblings on first failure.
async with asyncio.TaskGroup() as tg:
    a = tg.create_task(fetch_a())
    b = tg.create_task(fetch_b())
# TaskGroup (3.11+) cancels remaining tasks and raises ExceptionGroup.

If you're on 3.10 or earlier, use asyncio.gather(..., return_exceptions=True) and inspect results, or manage tasks manually with explicit cancel() calls in a finally.

Connection pool exhaustion: the deadlock that looks like a timeout

This is the one that takes down production at 3am. The pattern:

# asyncpg pool with default min_size=10, max_size=10
pool = await asyncpg.create_pool(dsn, min_size=10, max_size=10)

async def get_user_with_orders(user_id: int):
    async with pool.acquire() as conn:          # holds connection #1
        user = await conn.fetchrow("SELECT * FROM users WHERE id=$1", user_id)
        # Nested acquire inside the outer acquire:
        async with pool.acquire() as conn2:     # waits for a free connection
            orders = await conn2.fetch("SELECT * FROM orders WHERE user_id=$1", user_id)
    return user, orders

Under low load this works—the pool has spare connections. Under high load, all 10 connections are held by outer acquire blocks, each waiting for an 11th connection that will never come. Classic deadlock. The pool's acquire timeout (default 60s in asyncpg) eventually fires, but by then you've queued hundreds of requests behind it and the health check has already failed.

Two fixes, and you want both:

Fix 1: never nest acquires. Use the single connection for both queries:

async def get_user_with_orders(user_id: int):
    async with pool.acquire() as conn:
        # Both queries on the same connection. Sequential, but no deadlock.
        user = await conn.fetchrow("SELECT * FROM users WHERE id=$1", user_id)
        orders = await conn.fetch("SELECT * FROM orders WHERE user_id=$1", user_id)
    return user, orders

If you genuinely need concurrent queries, acquire both connections before awaiting either:

async def get_user_with_orders(user_id: int):
    # Acquire both up front so neither waits on the other's release.
    async with pool.acquire() as c1, pool.acquire() as c2:
        user, orders = await asyncio.gather(
            c1.fetchrow("SELECT * FROM users WHERE id=$1", user_id),
            c2.fetch("SELECT * FROM orders WHERE user_id=$1", user_id),
        )
    return user, orders

Fix 2: set an acquire timeout shorter than your request timeout. The default 60s is almost always wrong. If your p99 target is 500ms, an acquire that waits 60s is a 60s request that will be cancelled anyway:

pool = await asyncpg.create_pool(
    dsn,
    min_size=10,
    max_size=20,
    # Fail fast so the caller can retry or shed load, instead of piling up.
    timeout=2.0,
    command_timeout=5.0,
)

Sizing: max_size should be roughly (peak_concurrent_queries_per_request) × (target_rps) × (avg_query_seconds), capped by what your database can actually handle. Postgres starts thrashing on context switches well before 100 connections per core. If you need more than a few hundred connections total, put PgBouncer in transaction mode in front and keep max_size small (10–20 per app instance).

Detection: asyncpg exposes pool stats via pool.get_size(), pool.get_idle_size(), and pool.get_max_size(). Emit get_size() / get_max_size() as a gauge and alert above 0.8 sustained for 30 seconds. That's your early warning before the deadlock.

When this advice is wrong

run_in_executor with a process pool is the wrong answer if your CPU work is under ~1ms—the pickling and IPC overhead exceeds the work. Profile first; a 50µs hash doesn't need a process.

TaskGroup requires Python 3.11+. On 3.10 you're stuck with gather plus manual cancellation, and the code is uglier.

Shrinking pool max_size and adding acquire timeouts trades throughput for tail latency. If your workload is genuinely bursty and your database has headroom, a larger pool with longer timeouts may be correct—but measure, don't assume.

And if you find yourself fighting the event loop constantly, the honest answer may be that the workload is CPU-bound and you want multiprocessing or a different runtime entirely (Go, Rust, or Python with uvloop and careful separation of sync/async code). Async Python is a tool for I/O concurrency, not a general-purpose performance upgrade.

The three bugs here share a root cause: async/await makes concurrency look like syntax, but it's actually a scheduling contract. Break the contract—block the loop, swallow cancellation, hold a resource across an await—and the failure is silent until it isn't. Instrument loop lag, bound every acquire, and treat CancelledError as control flow. That's most of the battle.