Bilgin Ibryam

Posted on Aug 29 Originally published at generativeprogrammer.com

Latency Patterns for Faster AI Applications

#architecture #ai #distributedsystems #performance

A practical map for shortening the path from user intent to a useful AI response.

When you type a question into ChatGPT, latency is the time until the first useful response appears. The same applies when you click on a dashboard and wait for the first useful chart. That interval may include browser work, authentication, an API gateway, application logic, database queries, an LLM call, and the final rendering. The user waits for all the required work on this critical path, not only for one isolated component.

This is why a fast database query or LLM call does not prove that the application is fast. We need to measure the full path as a distribution and identify what controls the percentile we care about. Once we know what dominates the critical path, we can bring requests and data closer, reduce work, execute independent work concurrently, or anticipate predictable work. These are general distributed-system patterns that apply equally to AI applications. The map below adapts latency optimization ideas from Pekka Enberg’s Latency book into four practical pattern categories.

An agentic request may cross RAG retrieval, databases, a model gateway, one or more LLMs, MCP or other tool servers, and an orchestration loop. A large part of the latency often comes from the LLM calls, but the model is still only one segment of the path. Model-serving optimizations include quantization, decoding, and server scheduling. I cover those in this talk and this post. In this post, I explicitly focus on non-model latencies across the application path.

Each category explains the patterns through traditional application examples, then compares how they apply in an AI application. Taken together, I see 19 latency patterns across four categories.

Locality Patterns

The first thing to check is whether the request and the data it needs are far apart. That distance may be geographical, between services, between processes on one host, or even between a CPU core and memory. There are four common patterns for shortening this part of the path.

1. Colocation Pattern

Colocation shortens or removes a boundary between the source and the target. The scale can vary significantly. A CDN can serve the dashboard shell closer to the user. Application logic can run in the same region as its database instead of crossing a wide-area network. On one host, CPU affinity and NUMA-aware placement can keep hot work and memory near the cores that use them. The mechanism is the same even when the distances are very different.

When to use. Use colocation when the distance between components that frequently interact is a measurable part of the critical path.

The main trade-off. Tighter placement can reduce scheduling flexibility and make failover or scaling across locations more difficult.

2. Replication Pattern

Replication places a readable copy in more than one location. A nearby replica can shorten the read path, but only when the application is allowed to read from it. For example, a dashboard can read tenant data from the nearest regional replica instead of crossing regions for every query.

When to use. Use replication for read-heavy paths that can tolerate a clearly defined level of staleness.

The main trade-off. Synchronous replication may keep the write waiting for required acknowledgements, while asynchronous replication can respond earlier but allow lag. The application must decide which reads can tolerate older data, how they are routed, and what should happen during a failure.

3. Partitioning Pattern

Partitioning divides data or work into units that can be placed and processed independently. It does not make anything closer by itself, but a good partition key can keep a tenant’s query and data together. For example, a dashboard can partition metrics by tenant so that most queries touch one shard rather than several.

When to use. Use partitioning when requests and data share a stable key that can keep related work within one partition or location.

The main trade-off. A poor key can create a hot partition, while a request that spans several partitions brings routing and coordination back onto the critical path.

4. Caching Pattern

Caching creates a shortcut for repeated reads or computation. The hit ratio tells us how often the shortcut works, while the miss penalty tells us how painful the original path remains. For example, a dashboard can cache common aggregates close to the application instead of recalculating them for every page load.

When to use. Use caching when reads or computations repeat and their results can be safely reused.

The main trade-off. A high hit ratio can still produce a bad tail when misses are expensive. Freshness, invalidation, and ownership determine whether the cached result can be trusted.

Choosing What to Bring Closer

In a traditional application, start with the kind of distance that creates the delay. Use colocation when two components communicate frequently, replication when reads are far from the source, partitioning when related data can be kept together, and caching when the same data or computation is requested repeatedly. The patterns can be combined, but each addresses a different reason for travelling across the system.

In an AI application, apply the same test to the orchestrator, model gateway, retrieval store, state store, and tools. Colocate dependencies that communicate frequently, replicate read-heavy knowledge where residency rules allow it, partition state by tenant or session, and cache repeated retrieval, schema discovery, session summarization, policy lookup, or model prefill work. Cache keys must include every input that can change the meaning of a result, including tenant and authorization scope.

Work Reduction Patterns

If the latency trace points at computation as the bottleneck, the next move is to remove work. Work that no longer exists cannot delay the request, consume capacity, or create another failure point.

5. Algorithmic Work Reduction Pattern

An index over tenant and time can replace a full scan. A hash lookup can replace a linear search. Filtering before an expensive join or sort can reduce the dataset before the costly operation begins. These changes reduce the amount of work rather than making the same work slightly faster.

When to use. Use this pattern when query or computation cost dominates and a better algorithm, index, or data structure can reduce the work.

The main trade-off. A faster access path may require additional indexes, memory, preprocessing, or implementation complexity.

6. Selective Data Processing Pattern

A dashboard does not need the full object graph when the first chart uses five fields. Building large intermediate objects, copying them between layers, serializing them, and then discarding most of the result consumes time without changing what the user sees. A lean response should carry only what the next step needs.

When to use. Use selective data processing when the next step needs only a subset of the available data.

The main trade-off. Removing too much data can trigger another request or prevent a later step from completing.

7. Setup Reuse Pattern

Reusing an established connection, a parsed schema, or validated orchestration state avoids repeating the same setup on every request.

When to use. Use setup reuse when the same initialization cost is paid repeatedly and the resulting state can be retained safely.

The main trade-off. Reused state needs explicit lifecycle, freshness, isolation, and failure handling.

8. Request Coalescing Pattern

Combining several known reads into one request can remove repeated serialization, scheduling, and round trips. This is where request batching can help latency. For example, a dashboard backend can fetch the summary, alerts, and recent events in one call instead of making three sequential requests.

When to use. Use request coalescing when several predictable calls cross the same boundary and can be safely combined.

The main trade-off. A combined request can make one slow operation hold back all the others. It should be measured against the user-visible latency target, not assumed to be faster.

9. Runtime Tuning Pattern

Allocation, garbage collection, demand paging, scheduling, and context switches can all delay useful work. Profile the critical path first, then reduce only the runtime costs that measurably affect it. For example, a service may reduce short-lived allocations or tune its heap after profiles tie p99 spikes to garbage-collection pauses. These costs are measurable suspects, not default explanations for every slow service.

When to use. Use runtime tuning when profiles show that runtime overhead affects the percentile you are trying to improve.

The main trade-off. Runtime tuning can increase implementation complexity and may depend on a specific runtime, operating system, or hardware configuration.

Choosing What Work to Remove

In a traditional application, first identify what kind of work dominates the trace. Algorithmic work reduction changes how the result is found. Selective data processing reduces how much data moves through the path. Setup reuse avoids repeating initialization, while request coalescing removes repeated boundary crossings. Runtime tuning comes last because it is useful only when profiles point to the runtime itself.

In an AI application, remove model and tool calls before optimizing them. Use deterministic code or indexed search for direct lookup, filter and rank evidence before it enters the prompt, reuse model and tool connections where safe, and combine calls that always happen together. Route simple tasks to a smaller model when it meets the same quality target, stop generation when the useful answer is complete, and bound tool output before it becomes prompt input. Smaller prompts help only when they remain sufficient. If missing context causes another retrieval and model turn, the end-to-end path may become longer.

Once the remaining work is necessary, the next question is whether it really has to run in sequence.

Concurrent Execution Patterns

Some work is necessary but does not need to happen in sequence. Suppose authentication must finish first, but four dashboard panels are then independent. Running them serially makes the user wait for roughly the sum of their durations. Starting them together can bring the wait closer to the slowest required panel, plus scheduling and coordination overhead.

10. Synchronization Avoidance Pattern

A contended lock can make otherwise short operations line up behind one owner. Immutable state may remove the shared dependency, while a single-owner design may replace lock contention with an explicit queue. The important question is whether another operation must finish or release something before this one can continue.

When to use. Use synchronization avoidance when coordination over shared state is a measurable source of waiting and the state can be made immutable, partitioned, or owned by one component.

The main trade-off. Removing a lock may move the waiting elsewhere or require a different state-management model.

11. Independent Concurrency Pattern

Concurrency lets several tasks make progress, while parallelism executes work at the same time. For the dashboard, separate panel reads can run concurrently. In an agentic application, a research agent can query independent document stores and market-data sources in parallel, then join the results before synthesis. If one tool needs an identifier returned by another, that dependency remains serial.

When to use. Use independent concurrency when required tasks have no data dependency and the system has enough capacity to run them together.

The main trade-off. Concurrency does not help when one task genuinely needs the result of another, and it can make latency worse when the fan-out exhausts a shared dependency.

12. Progressive Response Pattern

An event loop can use resources more efficiently while external I/O is pending, but it does not make the database, model, or remote service intrinsically faster. The visible latency improves when an AI application streams useful model output or an independently useful tool result instead of holding everything until the full task completes.

When to use. Use a progressive response when a partial result is independently useful and time to first useful result matters to the user.

The main trade-off. This pattern shortens time to first useful result, not full completion, and the client must handle partial or provisional state.

13. Concurrency Budget Pattern

A queue stores waiting; it does not remove it. If arrivals exceed capacity, the queue grows and tail latency follows. Set a concurrency budget, propagate deadlines and cancellation, apply backpressure, and stop work when the user no longer needs it. For example, a dashboard can allow only four panel queries to run at once rather than sending an unbounded fan-out to the database.

When to use. Use a concurrency budget when one request can fan out across services, databases, models, or tools and overload a shared dependency.

The main trade-off. A budget that is too low leaves capacity unused, while one that is too high moves the queue into a downstream dependency.

14. Hedged Requests Pattern

Request hedging reduces exposure to a straggler by sending equivalent copies and accepting the first valid response. For example, after a short delay, a service can send the same read to a second replica and use whichever valid response arrives first.

When to use. Use hedged requests when rare stragglers dominate tail latency, spare capacity exists, and the operation is idempotent or side-effect-free.

The main trade-off. Hedging spends extra capacity and can worsen overload if it is used without a strict budget.

Choosing What Can Run Together

In a traditional application, synchronization avoidance removes unnecessary waiting, while independent concurrency overlaps work that has no dependency. A progressive response changes when useful output becomes visible, not how fast the full operation completes. A concurrency budget protects shared capacity, while hedged requests spend additional capacity to reduce exposure to rare stragglers.

In an AI application, use concurrency for independent model, tool, and retrieval calls, but keep dependent calls serial. A progressive response is a separate delivery decision: it changes when useful output becomes visible, not which tools can run in parallel. Apply one concurrency budget across models, subagents, tools, and retrieval, and hedge only side-effect-free or idempotent operations. The goal is a shorter dependency chain, not the largest possible fan-out.

Anticipation Patterns

If a remaining delay cannot be removed or safely overlapped after the request arrives, we can anticipate likely work and move it earlier.

15. Predictive Prefetching Pattern

Once authentication identifies the user, the application may fetch the default dashboard before the browser asks for its first panel. This is a prediction about a future read.

When to use. Use predictive prefetching when the next read is predictable enough to justify occasionally wasted work.

The main trade-off. Prefetching helps only when the data arrives in time and is still useful when the application needs it. A wrong prediction consumes resources without shortening the request.

16. Optimistic Update Pattern

A filter can appear selected while persistence continues, as long as the interface shows that the change is pending and can roll it back after a failure. This improves perceived latency after the user acts. It does not make durable completion happen earlier.

When to use. Use optimistic updates when success is common, failure is visible, and the operation can be safely reversed.

The main trade-off. A failed operation requires rollback, and provisional state must never be presented as completed work.

17. Speculative Execution Pattern

Speculative execution runs a likely branch before the actual choice arrives. For example, a dashboard can start calculating the most common time range while the user is still opening the time-range filter.

When to use. Use speculative execution when a small number of likely branches can run safely, independently, and be cancelled.

The main trade-off. The work must be isolated and cancellable because a wrong prediction consumes resources without helping the request.

18. Precomputation Pattern

Maintaining a tenant aggregate as events arrive can replace repeated foreground computation with a read. Precomputation moves known work from the foreground request to the point where the source data changes.

When to use. Use precomputation when an expensive result can be refreshed as its source changes rather than calculated for every request.

The main trade-off. Precomputed results consume storage and refresh capacity, and they may be stale when the source changes faster than they can be updated.

19. Prewarming Pattern

A small worker pool, connection pool, or execution environment can be initialized before traffic arrives. Prewarming moves predictable setup cost off the foreground critical path.

When to use. Use prewarming when cold-start or setup cost is predictable and keeping limited capacity ready is affordable.

The main trade-off. Warm resources consume capacity even when no request uses them.

Choosing What to Do Earlier

In a traditional application, predictive prefetching prepares a likely read, optimistic updates show likely success before durable completion, and speculative execution starts a likely branch. Precomputation prepares reusable results when source data changes, while prewarming prepares resources before requests arrive. The difference is what is moved earlier: data, feedback, a branch, a result, or capacity.

In an AI application, prefetch likely context once intent is known, run safe read-only searches for likely branches, precompute embeddings, indexes, summaries, and policy metadata, and prewarm agent workers or model and tool connections. An optimistic interface may show a tool action as pending before it commits, but never as completed.

For all of these patterns, track whether the early work was used, whether it completed in time, how much critical-path time it saved, how much work was discarded, and whether optimistic state had to roll back. Cancel early work when it is no longer useful. A prediction that delays normal traffic is not an optimization.

Find the Bottleneck, Choose the Pattern

Do not start by choosing a pattern. Start by tracing the full request and locating where delay enters the critical path: context retrieval, model access, tools, agent coordination, or verification and delivery.

Read the matrix in two moves. First, choose the column where the delay appears. Then move down that column and select the intervention that matches its cause:

Locality Patterns: The request, data, or dependency is too far away.
Work Reduction Patterns: The application performs unnecessary computation, data processing, or calls.
Concurrent Execution Patterns: Independent work is waiting in sequence.
Anticipation Patterns: Predictable work remains on the foreground path.

The cells are starting points, not a checklist. Pick the smallest pattern that changes the dominant part of the path. Before applying it, check whether it will amplify the tail or trade away quality, freshness, privacy, cost, or correctness. Then measure the same latency clock at the target percentile together with task success. If the bottleneck moves, return to the matrix.

DEV Community