Most latency targets for AI features arrive as one number on a ticket: "under 200 ms." Nobody owns 200 ms. Someone owns the vector search, someone owns the model server, someone owns the network in between, and each of them assumes they get most of it.
Below is the budget I'd start from for a retrieval-backed AI endpoint. A query comes in, relevant items are retrieved, a model ranks or scores them, and the result goes back while the user waits. Semantic search, recommendations and RAG-style lookups all have roughly this shape. Copy it, adjust it to your stack, and argue about it in the design review, not the incident review.
The budget
| Bucket | Line item | p99 budget | What usually blows it |
|---|---|---|---|
| Network | Client to edge, round trip | 25 ms | Fresh TLS handshakes, distant regions |
| Network | Internal hops (gateway, app, model) | 10 ms | Sidecars and services added over time |
| Network | Response payload | 5 ms | Returning full documents instead of IDs |
| Retrieval | Vector search (ANN) | 20 ms | Index outgrows memory, filters applied after search |
| Retrieval | Metadata and feature fetch | 25 ms | One call per item instead of one batched call |
| Retrieval | Permission and business filters | 15 ms | Filtering in application code after retrieval |
| Inference | Query embedding | 10 ms | Cold model, silent CPU fallback |
| Inference | Queue and batching wait | 10 ms | Batch window tuned for throughput |
| Inference | Reranker over top candidates | 45 ms | Too many candidates, input length creep |
| Inference | Post-processing | 5 ms | Formatting in slow loops |
| Reserve | Headroom | 30 ms | Spent on day one if nobody guards it |
| Total | 200 ms |
How to read it
Every number is a p99, not an average. Budgeting each hop at p99 is deliberately conservative, because every hop rarely hits its tail on the same request. The one place it isn't conservative enough is anything that fans out, where the request waits on the slowest of many calls. That's why the metadata line gets more than the vector search itself.
Inference is the biggest bucket, and the reranker is the biggest single line. A cross-encoder's cost grows with the number of candidates and their length, so cutting the rerank set from 50 to 20 usually buys back more time than any amount of model tuning.
Network looks generous at 40 ms. With users spread across continents it isn't, so serve from more regions or rebalance the table before launch.
The headroom is 15% of the total. It covers deploys; garbage collection pauses, noisy neighbors and the hop someone adds next quarter. If you can't find 15% spare on paper, you won't find it in production.
When the numbers don't fit
Cut in this order.
Do less work. Re-rank fewer candidates, return IDs instead of documents, and push filters into the index query so you don't retrieve things you'll throw away.
Move work off the request path. Cache embeddings for frequent queries, keep metadata in a store the request can hit once, and refresh features when events arrive rather than when users ask.
Then buy speed. Faster hardware or a smaller, quantized model. It works, but it's the most expensive fix, and the one teams reach for first.
Once the table balances, put a p99 alert on each line, not just on the total. When the total breaks, you want to know which line broke it.
First, check that 200 ms is the right number
The table assumes 200 ms is the right target, and sometimes it isn't. A feature that can answer within a second, or show up after the page loads, deserves a cheaper design. If that decision isn't settled yet, this walkthrough of real-time AI architecture, from latency tiers to the serving layer, is worth reading before you defend a single line above.
Top comments (0)