For years, much of the AI industry's attention has been on building larger models, buying more GPUs, and expanding data centers.
But model size and hardware capacity are only part of the equation. What matters just as much is how efficiently those resources are used.
Running a large language model at scale involves more than loading weights into GPU memory. There's request scheduling, memory management, communication between devices, caching, and the constant challenge of keeping expensive hardware busy.
DeepSeek's approach to large-scale inference is a good example of how much performance can be gained through better infrastructure design. Techniques such as disaggregated inference, mixture-of-experts parallelism, dynamic load balancing, and speculative decoding can significantly reduce the cost of serving large models.
These techniques aren't exclusive to DeepSeek, and several have been around for years. What's interesting is how they work together in a production inference system.
1. The Real Cost of Running Large Language Models
Training a large language model requires enormous computing resources. Serving that model to users presents a different set of problems.
Once deployed, the model needs to handle thousands or millions of requests, often with very different input lengths and latency requirements.
Inference typically consists of two phases.
Prefill: Processing the Input
During prefill, the model processes the input prompt and builds the internal representations needed to start generating a response.
Consider an AI coding assistant receiving:
- A repository containing thousands of source files
- Documentation describing the application architecture
- A request to identify a concurrency bug
- Previous conversation history
Before generating an answer, the model needs to process the supplied context.
Prefill is generally compute-intensive because many input tokens can be processed in parallel. Longer prompts require more computation and can significantly increase the time before the first output token appears.
Decode: Generating the Output
Decode works differently.
The model generates tokens autoregressively, meaning each new token depends on previously generated tokens.
Unlike prefill, decoding is sequential across generation steps. The GPU repeatedly accesses model weights and attention-cache data, which can make memory bandwidth a major bottleneck.
This distinction matters because hardware optimized for prefill isn't necessarily the best configuration for decoding.
Many inference deployments run both phases on the same GPU resources. That's simpler to manage, but it can leave hardware underutilized, particularly when traffic contains a mixture of long prompts and short responses.
2. Disaggregated Inference: Separate Reading From Writing
One way to address this is to separate prefill and decode into different execution pools.
Rather than handling the entire inference process on the same GPUs, the system assigns resources according to the requirements of each phase.
A simplified architecture looks like this:
Incoming Request
|
v
API Gateway
|
v
Request Router
|
v
Prefill Cluster
Compute Optimized
|
v
KV Cache Transfer
|
v
Decode Cluster
Memory Optimized
|
v
Token Streaming
|
v
Client
This design has several practical advantages.
Independent scaling: If incoming requests contain very long prompts, additional prefill capacity can be allocated without expanding the decode cluster.
Hardware specialization: Different GPU configurations can be used depending on whether compute throughput or memory bandwidth is more important.
Better scheduling: Prefill operations don't have to compete directly with decode workloads for the same resources.
Improved utilization: The infrastructure can allocate resources based on actual traffic instead of maintaining a fixed configuration.
There is a trade-off, however.
The KV cache generated during prefill needs to be transferred to the decode workers. Depending on context length, network bandwidth, and cache placement, this transfer can introduce considerable overhead.
Disaggregated inference works best when the performance gains from specialization are greater than the additional communication costs.
3. Mixture of Experts Changes the Economics of Large Models
Traditional dense transformers activate most of their parameters during each forward pass.
Mixture-of-Experts (MoE) models use a different approach.
Instead of relying on a single feed-forward network in every transformer block, an MoE layer contains multiple expert networks. A routing mechanism determines which experts process each token.
Conceptually:
Input Token
|
v
MoE Router
|
+-----------+-----------+
| | |
v v v
Expert 1 Expert 2 Expert N
| | |
+-----------+-----------+
|
v
Combined Output
The important distinction is between total parameters and active parameters.
A model can contain hundreds of billions of parameters while using only a fraction of them to process an individual token.
Compared with a dense model of similar total size, this can substantially reduce computation.
However, the inactive parameters don't disappear. Their weights still need to be stored in memory.
At scale, running an MoE model becomes a matter of distributing experts across devices, routing tokens efficiently, and avoiding unnecessary communication.
Scaling MoE Models With Expert Parallelism
Suppose a model contains hundreds of experts distributed across a large GPU cluster.
Instead of storing every expert on every GPU, the inference system places different experts on different devices.
When a token needs a particular expert, the computation is routed to the GPU holding that expert.
This reduces the model-weight memory required on each device, although other model components may still need to be replicated or partitioned.
The system has to balance three competing requirements:
- Minimize unnecessary duplication of expert weights.
- Keep communication overhead under control.
- Distribute work evenly across available GPUs.
The right configuration depends on the model architecture, interconnect bandwidth, batch size, and incoming traffic.
Wide expert parallelism can be especially useful in large deployments with high-bandwidth GPU interconnects.
But adding more GPUs doesn't automatically improve throughput. If communication between devices becomes too expensive, the additional hardware may provide little benefit.
4. Dynamic Expert Load Balancing
Distributing experts across GPUs introduces another challenge: uneven workloads.
Not every expert receives the same amount of traffic. Some may be selected far more frequently than others, depending on the model's routing behavior and the requests being processed.
Consider a cluster with 128 GPUs.
Most devices complete their assigned work in 5 milliseconds, but one heavily loaded GPU takes 50 milliseconds.
If the operation requires synchronization, the other devices may have to wait for the slowest one.
That single overloaded GPU can become a bottleneck for the entire cluster.
Replicating Popular Experts
An Expert Parallelism Load Balancer (EPLB) can help by monitoring expert utilization and adjusting where experts are placed.
When an expert becomes overloaded, the system can create additional replicas on GPUs with available capacity.
Incoming tokens can then be distributed across those replicas.
A simplified implementation might look like this:
def rebalance_experts(experts, gpus):
for expert in experts:
if expert.load > OVERLOAD_THRESHOLD:
target = find_available_gpu(gpus)
if target is not None:
replicate_expert(expert, target)
update_routing_table(expert, target)
This is illustrative pseudocode rather than a production implementation.
In practice, expert replication requires available memory, synchronization, and updates to routing information. Moving expert weights between devices also consumes bandwidth.
Rebalancing too frequently can make performance worse rather than better.
A well-designed load balancer needs to consider how long an imbalance is likely to persist and whether replication is worth the cost.
The underlying problem is familiar from distributed databases and microservice infrastructure. The difference is that the system is distributing neural-network components rather than conventional application services.
5. Speculative Decoding: Generating More Tokens Per Verification Step
Autoregressive generation has an inherent limitation: tokens are generated sequentially.
Even when the next several tokens are predictable, a conventional decoder still processes them one step at a time.
Speculative decoding attempts to reduce this overhead.
The idea is to use a smaller, faster draft model, or another lightweight prediction mechanism, to propose several tokens ahead of time.
The larger target model then verifies those candidates in parallel.
The process works as follows:
- A lightweight drafter proposes several candidate tokens.
- The target model evaluates those candidates in parallel.
- Accepted tokens are added to the output.
- Generation continues from the first rejected position.
For example:
Draft model proposes:
"The database connection is now ready"
Target model verifies:
"The" Accepted
"database" Accepted
"connection" Accepted
"is" Accepted
"now" Rejected
Result:
Four draft tokens accepted
Generation continues from rejection
Instead of performing a separate target-model decoding step for every accepted token, the system can verify multiple candidates together.
This can improve throughput, particularly when the draft model is inexpensive and its predictions are frequently accepted.
Confidence-Aware Drafting
A further optimization is to adjust the number of drafted tokens dynamically.
Traditional implementations may generate a fixed number of candidates for every verification step.
But if the drafter is unlikely to predict the next token correctly, generating additional candidates may waste computation.
A confidence-aware approach estimates the likelihood of acceptance and stops drafting when confidence falls below a threshold.
Conceptually:
def draft_tokens(model, context, max_tokens=8):
candidates = []
for _ in range(max_tokens):
token, confidence = model.predict(context)
if confidence < 0.75:
break
candidates.append(token)
context.append(token)
return candidates
This example illustrates the idea, not the complete algorithm. Real implementations need properly calibrated confidence estimates and verification procedures.
The actual performance improvement depends on:
- Draft-model overhead
- Acceptance rate
- Target-model verification efficiency
- Sequence length
- Hardware utilization
With an appropriate verification algorithm, speculative decoding can preserve the target model's output distribution.
The main challenge is ensuring that the time saved during verification exceeds the additional cost of drafting.
6. KV Cache Management Is Becoming a First-Class Infrastructure Problem
Long-context inference consumes a considerable amount of memory, and much of that comes from the key-value cache.
During generation, transformer attention layers store key and value representations associated with previously processed tokens.
These cached representations allow the model to generate new tokens without recomputing the entire attention history at every step.
The drawback is memory consumption.
For a conventional full-attention transformer, a simplified estimate is:
KV Cache Bytes =
2
× Number of Layers
× Number of KV Heads
× Head Dimension
× Sequence Length
× Bytes per Element
This doesn't account for architectural optimizations such as compressed KV representations or other attention mechanisms, but it shows why context length matters.
As the number of concurrent requests increases, KV cache memory can become a major constraint.
The Problem With Idle Conversations
Imagine a coding assistant working with a large repository.
The user has a long conversation with the model, then leaves for several hours.
Keeping that conversation's entire KV cache in GPU memory is expensive, especially when thousands of other sessions are active.
Discarding the cache frees memory, but the next request may require processing the conversation history again.
The inference system has to decide whether retaining the cache is more economical than recomputing it later.
A Hierarchical Caching Strategy
One approach is to store cached state across multiple memory tiers.
Active Conversation
|
v
GPU KV Cache
Lowest Latency
|
v
System RAM
Warm Sessions
|
v
Persistent Storage
Cold Sessions
|
v
Cache Restoration
or Recomputation
Frequently accessed sessions remain in fast memory, while less active sessions can be moved to slower, less expensive storage.
When a session resumes, the system can restore cached data or recompute the necessary context.
The decision depends on cache size, transfer costs, available memory, and the likelihood that the session will be used again.
Some model architectures also support more compact forms of reusable state, reducing the amount of computation needed after a cache miss.
However, an ordinary transformer cannot reconstruct an arbitrary long conversation from only its last few tokens without additional retained information.
Cache management is therefore closely tied to both model architecture and serving infrastructure.
For long-context applications, it can have as much impact on operating costs as improvements to the model's compute kernels.
7. The API Gateway as an Inference Bottleneck
GPU optimization gets most of the attention, but an inference request passes through several CPU-side operations before reaching the model.
Depending on the serving architecture, the gateway may handle:
- Authentication and authorization
- Request parsing and validation
- Rate limiting
- Tokenization
- Workload scheduling
- Response streaming
At high request volumes, these operations can become bottlenecks of their own.
Parallel Tokenization
Tokenization converts raw text into the numerical token IDs used by language models.
For large prompts, tokenization can contribute noticeable latency before inference begins.
Using multiple CPU cores can help process independent portions of the input in parallel.
However, tokenization cannot always be parallelized by splitting text at arbitrary positions.
Many tokenizers depend on surrounding characters and boundary rules. Splitting a string incorrectly can change the resulting token sequence.
A parallel implementation must preserve those rules to produce the same output as sequential tokenization.
Rust for High-Performance Gateways
Rust is well suited to this type of infrastructure.
It provides memory safety without a garbage collector, supports concurrent execution, and gives developers control over memory allocation and resource management.
A Rust-based gateway can efficiently handle request parsing, tokenization, scheduling, and streaming.
That doesn't mean Rust automatically makes an application faster.
Poor algorithms, unnecessary memory copies, excessive allocation, and lock contention can still cause performance problems.
The implementation language helps, but the overall design matters more.
Security at the Gateway
The gateway also plays an important role in maintaining trust boundaries.
Untrusted user input should never be interpreted as privileged system instructions simply because it appears inside a request.
A secure implementation should validate message structures, preserve role boundaries, and enforce authorization outside the language model.
Correctly encoding user input as ordinary text helps prevent structural manipulation of requests.
However, it doesn't eliminate prompt injection.
A malicious document can still contain instructions intended to influence model behavior, even when the document is correctly passed as untrusted content.
Preventing those attacks requires application-level controls, restricted tool permissions, and careful handling of external data.
8. Hardware Independence Is Becoming Strategically Important
A significant part of today's inference infrastructure is built around specific GPU architectures and software ecosystems.
CUDA, hardware-specific kernels, collective communication libraries, and memory-management techniques are deeply integrated into many inference engines.
This creates several dependencies:
- Hardware availability and supply constraints
- Vendor-specific software optimizations
- High migration costs
- Limited deployment flexibility
- Geographic and regulatory restrictions
Supporting multiple accelerator architectures can reduce some of these risks.
But portability is more complicated than compiling the same code for a different device.
An inference engine optimized for NVIDIA hardware may depend on CUDA-specific operations, communication libraries, and memory layouts.
Running efficiently on another accelerator requires adapting or replacing those components.
For MoE models, communication performance is particularly important because tokens may need to move between experts distributed across many devices.
An alternative accelerator needs efficient matrix operations, memory management, collective communication, and synchronization.
Hardware portability is therefore not just about compatibility.
It's about maintaining acceptable performance across different execution environments.
For organizations operating large inference clusters, that flexibility can become a significant infrastructure advantage.
9. Understanding the Economics of AI Inference
Tokens per second is an important metric, but it doesn't tell us everything about inference efficiency.
A cluster might achieve impressive throughput under ideal benchmark conditions while performing less efficiently with real production traffic.
What matters is how much useful work the infrastructure delivers relative to its total cost.
A simplified calculation is:
Cost per Million Tokens =
Total Infrastructure Cost
------------------------- × 1,000,000
Total Billable Tokens
Even this calculation requires some care.
Input tokens, output tokens, and cached input tokens can have very different computational costs.
Traffic patterns also matter. A system processing large batches of short requests behaves differently from one handling long conversations with low concurrency.
A useful evaluation should include several metrics.
| Metric | What it measures |
|---|---|
| Tokens per second | Overall generation throughput |
| Time to first token | Initial response latency |
| Inter-token latency | Streaming responsiveness |
| GPU utilization | How effectively hardware is used |
| Cache hit rate | Amount of recomputation avoided |
| Energy consumption | Operating costs |
| Network overhead | Cost of distributed execution |
| Hardware depreciation | Long-term infrastructure expense |
| Revenue per GPU-hour | Commercial productivity |
High revenue per GPU-hour doesn't necessarily mean high profit.
Actual margins depend on hardware acquisition costs, depreciation, power consumption, networking, maintenance, and sustained utilization.
The same caution applies to benchmark results.
Peak throughput measured under controlled conditions shouldn't be confused with average performance under production workloads.
A fair comparison needs to account for latency requirements, model quality, request patterns, and total infrastructure costs.
10. Practical Inference Optimizations for Developers
Most developers won't be operating clusters with hundreds of GPUs.
Still, the principles behind large-scale inference optimization are useful for smaller deployments.
Separate Different Types of Workloads
Interactive requests, batch processing, embeddings, and long-context analysis have different performance requirements.
Putting everything into the same queue can create unnecessary latency.
Even a relatively small deployment can benefit from separating latency-sensitive requests from background processing.
Optimize Cache Reuse
If an application repeatedly sends the same instructions, documents, or conversation prefixes, prompt caching can reduce redundant computation.
For long-context applications, avoiding repeated prefill work may provide a bigger improvement than upgrading hardware.
Measure Actual Bottlenecks
Before optimizing GPU execution, check where the application is spending its time.
Tokenization, scheduling, network transfers, storage access, and batching can all contribute to latency.
A faster GPU won't solve a bottleneck elsewhere in the request pipeline.
Use Quantization Carefully
Lower-precision weights can reduce memory requirements and sometimes improve throughput.
However, the results depend on hardware support, quantization format, model architecture, and acceptable quality trade-offs.
Quantization should be evaluated against the actual workload rather than assumed to improve performance in every case.
Evaluate Speculative Decoding
Speculative decoding can improve output generation speed when draft acceptance rates are high enough.
But it introduces additional computation and implementation complexity.
The improvement should be measured under realistic traffic conditions.
Optimize for Useful Throughput
Generating more tokens per second isn't always the same as delivering a better service.
A system with higher throughput but worse latency may be unsuitable for interactive applications.
Likewise, reducing infrastructure costs is not particularly useful if model quality falls below application requirements.
The goal should
Top comments (0)