Optimizing LLM Inference for High Throughput

#aiinfrastructure #oxlo #ai

High-throughput LLM inference is rarely limited by raw compute. In production, throughput, measured in requests per second or aggregate tokens per second, is usually constrained by memory bandwidth, KV cache growth, and scheduling overhead. Optimizing it requires co-design across the model runtime, the serving infrastructure, and your client request patterns. The sections below break down the technical levers that actually move the needle, and how your provider's pricing model determines which levers are practical to pull.

Continuous Batching and Memory-Aware Scheduling

Static batching is inefficient for generative models because sequence lengths vary. If one request in a batch generates 2,000 tokens and another generates 200, the shorter request idles GPU slots while the longer one finishes. Continuous batching, also called inflight batching, solves this by allowing the inference engine to evict completed sequences and admit new requests into the active batch on every forward pass. Frameworks like vLLM and TensorRT-LLM implement this at the serving layer.

For developers, the practical implication is simple: prefer endpoints that expose continuous batching natively rather than building manual client-side batching logic. Client-side batching increases latency for individual users and complicates error handling, whereas server-side continuous batching improves both throughput and tail latency transparently.

KV Cache Management and Paged Attention

During autoregressive decoding, the KV cache often consumes more GPU memory than the model weights themselves, especially at long context lengths and high batch sizes. Traditional allocation reserves a contiguous block for each request's maximum sequence length, which wastes memory and caps batch size.

PagedAttention fragments the KV cache into fixed-size blocks that are allocated non-contiguously, similar to an operating system's virtual memory. This reduces fragmentation and allows the scheduler to fit more concurrent requests on the same GPU. On the client side, you should set tight but safe max_tokens limits and take advantage of prefix caching if your provider supports it, so shared system prompts or document contexts are computed once and reused across requests.

Quantization and Weight Streaming

At small batch sizes, inference is memory-bandwidth bound: the time to read weights from HBM dominates the time to compute matmuls. Quantization to INT8, FP8, or 4-bit formats reduces bandwidth pressure and increases the effective batch size you can run before hitting memory limits.

Trade-offs exist. Aggressive quantization can degrade reasoning or code-generation quality. The correct approach is to evaluate perplexity and task accuracy on your own data, then deploy the smallest viable precision. For high-throughput services, even a modest reduction in precision can unlock significantly higher request concurrency with no additional hardware.

Client-Side Concurrency and Connection Reuse

The network path between your application and the inference endpoint is often overlooked. Establishing a new TCP connection and TLS handshake for every request adds tens to hundreds of milliseconds of latency and burns CPU on both sides.

Use an async HTTP client with HTTP/2 or persistent keep-alive connections. Stream responses to improve time-to-first-token and to apply backpressure if your downstream consumer slows. Limit outbound concurrency with a semaphore tuned to the provider's queue depth. If you burst thousands of requests simultaneously without server-side flow control, you will hit rate limits or cause queueing that negates any runtime optimization.

How Pricing Architecture Affects Throughput Decisions

Token-based pricing creates a direct tension between throughput and cost. Every optimization that increases context length, batch size, or output length also increases the token count and therefore the bill. This incentivizes teams to truncate prompts, split long documents into smaller chunks, or avoid agentic loops, all of which add engineering complexity and can reduce model accuracy.

Oxlo.ai uses request-based pricing: one flat cost per API call regardless of input length or output tokens. This removes the conflict between throughput optimizations and cost control. You can pack large contexts into a single request, run multi-turn agent workflows, or generate long outputs without marginal cost scaling. For high-throughput systems, this predictability simplifies capacity planning and makes long-context and agentic workloads economically viable.

Oxlo.ai also provides no cold starts on popular models and is fully compatible with the OpenAI SDK, so you can drop it into existing Python or Node.js pipelines without rewriting client code. You can explore plans and details at https://oxlo.ai/pricing.

Implementation: High-Throughput Async Client

The following pattern uses Python asyncio with the OpenAI SDK pointed at Oxlo.ai. It streams responses, reuses the underlying HTTP connection, and bounds concurrency with a semaphore to avoid overwhelming the endpoint.

import asyncio
from openai import AsyncOpenAI

# Oxlo.ai is a drop-in replacement for the OpenAI client
client = AsyncOpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_API_KEY",
    max_retries=2,
    timeout=60.0,
)

async def generate(prompt: str, semaphore: asyncio.Semaphore):
    async with semaphore:
        response = await client.chat.completions.create(
            model="qwen3-32b",
            messages=[{"role": "user", "content": prompt}],
            stream=True,
            max_tokens=512,
        )

        content = ""
        async for chunk in response:
            delta = chunk.choices[0].delta.content
            if delta:
                content += delta
        return content

async def main(prompts: list[str]):
    # Tune concurrency to your workload and endpoint capacity
    semaphore = asyncio.Semaphore(32)
    tasks = [generate(p, semaphore) for p in prompts]
    return await asyncio.gather(*tasks)

if __name__ == "__main__":
    prompts = ["Summarize the key benefits of async I/O"] * 100
    results = asyncio.run(main(prompts))
    print(f"Completed {len(results)} requests")

Key details in this pattern: the AsyncOpenAI client maintains a connection pool automatically, stream=True reduces time-to-first-byte, and the semaphore prevents unbounded concurrency that would degrade tail latency.

Model Selection for Throughput

Not all models saturate hardware equally. Smaller dense models or efficient Mixture-of-Experts architectures generally deliver higher requests-per-second than massive monolithic models. Oxlo.ai offers 45+ models across categories, which lets you align model size to latency requirements without changing integration code.

For throughput-sensitive paths, consider efficient options like DeepSeek V4 Flash (MoE architecture, 1M context window) or Qwen 3 32B (strong multilingual performance with lower memory pressure). For complex reasoning where per-request latency is less critical than accuracy, DeepSeek R1 671B or GLM 5 are available on the same endpoint structure. Because Oxlo.ai charges per request, switching between models for different pipeline stages does not complicate cost forecasting.

Conclusion

Optimizing LLM throughput is a stack-wide problem. Continuous batching, PagedAttention, quantization, and async client patterns each address a different bottleneck. However, the economics of your inference provider determine which optimizations are practical to deploy at scale. Token-based meters discourage the very techniques, like large-context batching and agentic loops, that improve system-level throughput.

Oxlo.ai’s request-based pricing removes that constraint. Combined with no cold starts, OpenAI SDK compatibility, and a broad model catalog, it is a strong option for teams building high-throughput, long-context, or agentic production systems.

DEV Community