High throughput inference is rarely limited by raw compute. Instead, it hits a wall on memory bandwidth, KV cache capacity, and scheduling overhead. For teams running agentic loops or long-context RAG, the standard advice to quantize, batch, and scale horizontally only works if the serving layer can absorb the load without hidden latency taxes. This article breaks down the mechanics of high-throughput LLM serving, then shows how to apply them against a live API where predictable pricing and warm infrastructure change the optimization equation.
The Throughput Bottleneck
Transformer inference has two distinct phases. Prefill is compute-intensive and highly parallel. Decode is memory-bandwidth-bound because each new token requires loading the full model weights and the growing KV cache across the GPU memory bus. As batch size or sequence length increases, the KV cache can dominate available VRAM, forcing smaller batches and lower throughput. For long-context models, this effect is nonlinear. A 100K context window can consume tens of gigabytes of cache per sequence, which means batching more than a few concurrent requests becomes impossible without aggressive compression or offloading.
Batching and Scheduling
The most impactful server-side optimization is continuous batching, sometimes called in-flight batching. Rather than waiting for every request in a static batch to finish decoding, the scheduler swaps new requests into the GPU as soon as others complete prefill or decode. This keeps tensor cores occupied and minimizes pipeline bubbles. Dynamic batching adds another layer by grouping requests with similar sequence lengths or target outputs, though this is harder to control from the client side. One often overlooked factor is cold-start latency. If the inference backend must spin up a container or reload weights for each burst, the batching window collapses. Oxlo.ai serves popular models with no cold starts, so the batching scheduler can maintain steady-state throughput without warmup penalties.
Quantization and Speculative Decoding
When memory is the constraint, quantization is the lever. Weight-only quantization to INT4 or INT8, and more importantly, KV cache quantization to FP8, directly increases the number of concurrent sequences that fit in VRAM. The tradeoff is a small accuracy degradation that is usually negligible for retrieval and agentic tool use. Speculative decoding offers a different path. A small draft model generates candidate tokens, and the larger target model verifies them in parallel. If the draft model is well aligned, this can reduce latency significantly without changing the final distribution. Oxlo.ai hosts several architectures that are naturally efficient at scale, including DeepSeek V4 Flash, an MoE model with a 1 million token context window, and DeepSeek R1 671B MoE for deep reasoning. For code generation workloads, Qwen 3 Coder 30B and Oxlo.ai Coder Fast are available with the same flat request pricing, making them practical for high-volume CI pipelines.
Request Economics and Predictable Scaling
Throughput optimization is not only a technical exercise. It is an economic one. Under token-based pricing, long prompts and high concurrency create a cost surface that is hard to model. Every extra token in the context window increases the bill, so squeezing more throughput out of a long-context pipeline can actually raise costs faster than it improves latency. Oxlo.ai uses request-based pricing. You pay one flat cost per API request regardless of prompt length or output tokens. This means that for long-context and agentic workloads, where inputs are large and requests are frequent, the cost per unit of work stays constant as you scale up batch sizes or context windows. You can optimize for pure throughput without watching a meter run on every token. For exact plan details, see the Oxlo.ai pricing page.
Client-Side Optimization
Even with an optimized backend, client behavior determines realized throughput. Blocking synchronous calls waste network round
Top comments (0)