DEV Community

Sergey Shinder
Sergey Shinder

Posted on

One PDF endpoint in our async service made every other endpoint on the worker wait

Our delivery slot service is a FastAPI application under uvicorn with four worker processes. Most of its endpoints are small: is this slot available, reserve it, release it, each answering in a few milliseconds. In August we added one more, which renders a delivery note as a PDF for the depot office, and wrote it as async def like everything else in the file.

From the next morning, slot lookups had a p99 of over two seconds between seven and eight, and normal latency for the rest of the day. The dashboard broke latency down by endpoint and pointed at availability, so we spent a week on its database queries and found nothing wrong with them.

An async def endpoint runs on the worker's event loop, one thread that interleaves every request in flight by switching between them whenever one awaits something. The PDF library is synchronous and pure Python, and a delivery note takes it between one and two seconds of CPU. It never awaits anything, so for those seconds the event loop runs nothing else. Every request already on that worker, and every request routed to it, waits until the PDF is finished. At seven the depot office prints the whole day's notes, three hundred of them, so a quarter of all slot lookups sat behind a renderer.

The latency belonged to the PDF and was recorded against everything else. We confirmed it by running one pod with PYTHONASYNCIODEBUG=1 for an hour. In debug mode asyncio logs any step that holds the loop for more than a tenth of a second, and every warning named the PDF endpoint.

Rendering now runs in a separate process pool through run_in_executor, so the event loop only awaits the result. A plain def endpoint would have gone to FastAPI's thread pool, which is better, but threads running pure Python still compete for the interpreter lock, and a process does not. Each worker also measures event loop lag, the delay between when a timer was due and when it ran, and we alert on that rather than on any single endpoint's latency.

In an async service, every handler on a worker shares one thread. A slow handler is never slow on its own.

– Sergey Shinder

Top comments (0)