Optimizing LLM Performance for High Throughput

High-throughput LLM serving is one of the hardest infrastructure problems in production AI. Throughput is usually measured in requests per second or total tokens generated per second, but optimizing it requires balancing latency budgets, cost constraints, and model accuracy. This article walks through practical techniques you can use today to maximize throughput, from client-side concurrency to provider-level architecture, with concrete code examples you can run immediately.

Client Concurrency and Connection Pooling

Your client is often the first bottleneck. Opening a new TCP connection for every request adds TLS handshake and TCP slow-start latency. For high throughput, reuse connections and limit concurrency to match what the server can absorb.

With the OpenAI SDK, you can configure the underlying HTTP client. If you are using Oxlo.ai, the setup is identical because the platform is fully OpenAI SDK compatible. You only need to change the base URL and API key.

import asyncio
import openai
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY",
    max_retries=2,
    timeout=60.0,
)

async def generate(prompt: str, semaphore: asyncio.Semaphore):
    async with semaphore:
        response = await client.chat.completions.create(
            model="deepseek-v4-flash",
            messages=[{"role": "user", "content": prompt}],
            stream=False,  # non-streaming maximizes throughput per connection
        )
        return response.choices[0].message.content

async def main(prompts: list[str]):
    # Limit concurrency to avoid overwhelming the network stack
    semaphore = asyncio.Semaphore(20)
    tasks = [generate(p, semaphore) for p in prompts]
    return await asyncio.gather(*tasks)

if __name__ == "__main__":
    prompts = ["Explain concurrency"] * 100
    results = asyncio.run(main(prompts))

The semaphore prevents thundering-herd behavior. Setting stream=False reduces per-request overhead and lets the server batch completions more effectively.

Server-Side Batching and Dynamic Scheduling

Inference engines use continuous, or in-flight, batching to keep GPU tensor cores saturated. Instead of waiting for the entire batch to finish before starting a new one, the scheduler swaps in new requests as soon as others complete their generation. This is invisible to the client, but you can design payloads that help the scheduler.

• Keep prompts roughly similar in length. Wildly different input sizes force padding or uneven KV-cache allocation.
• Set max_tokens to a realistic ceiling. Unbounded generation prevents the scheduler from reclaiming KV-cache blocks early.
• Use JSON mode or structured outputs only when necessary. Constrained decoding adds grammar overhead on each forward pass.

Oxlo.ai runs popular models without cold starts, so the batching scheduler is already warm and ready to accept traffic spikes.

Model Selection and Quantization

Not every task requires the largest model. Mixture-of-Experts architectures like DeepSeek V4 Flash route each token through only a subset of parameters, delivering near-state-of-the-art reasoning at higher throughput than dense models of comparable size. For coding and agentic tool use, Minimax M2.5 and DeepSeek V3.2 offer strong accuracy with lower latency profiles.

If you control your own inference stack, quantization to FP8 or INT8 can nearly double throughput with minimal accuracy loss on most tasks. When you use a hosted provider, look for platforms that offer quantized variants of flagship models so you do not have to manage calibration yourself.

Caching and Request Deduplication

High-throughput systems often serve repetitive prompts. A two-tier cache can eliminate redundant inference entirely.

Tier 1 is an exact-match cache. Hash the prompt and system message, then store the completion in Redis or Valkey with a TTL that matches your freshness requirements.

import hashlib
import json
import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def get_cache_key(messages, model):
    payload = f"{model}:{json.dumps(messages, sort_keys=True)}"
    return hashlib.sha256(payload.encode()).hexdigest()

async def cached_generate(messages, model="llama-3.3-70b"):
    key = get_cache_key(messages, model)
    cached = r.get(key)
    if cached:
        return cached

    # client is the AsyncOpenAI instance from earlier
    response = await client.chat.completions.create(
        model=model,
        messages=messages,
    )
    content = response.choices[0].message.content
    r.setex(key, 3600, content)  # 1-hour TTL
    return content

Tier 2 is a semantic cache. Use an embedding model to index prompts by vector similarity. If a new prompt is within a cosine-similarity threshold of a cached prompt, return the stored answer. Oxlo.ai provides embedding endpoints through BGE-Large and E5-Large, so you can run the entire pipeline on the same API.

Streaming, Latency, and Throughput

Streaming improves time-to-first-byte, which is great for chat UIs, but it can reduce total throughput. Each streamed chunk carries HTTP overhead and prevents the server from compressing the full response. For batch or agentic workloads where latency is less critical than cost and volume, disable streaming.

If you need both low latency and high throughput, split traffic. Use streaming for user-facing paths and non-streaming batch jobs for background agents. Oxlo.ai supports both modes on all chat models, so you can point different services at the same endpoint with different payload flags.

Choosing a Pricing Model for Scale

Most inference providers bill by the token. Under token-based pricing, throughput optimizations that increase context length or conversation history directly raise your cost. This creates a perverse incentive: every technique that improves accuracy, such as including more retrieved documents or maintaining longer multi-turn context, makes the workload more expensive.

Oxlo.ai uses flat per-request pricing. The cost of an API call does not scale with prompt length, so you can send long contexts, agent traces, and full document suites without watching the meter run. For high-throughput agentic and long-context workloads, this model is often significantly cheaper than token-based alternatives. You can verify current plans on the Oxlo.ai pricing page.

The platform offers 45+ models across seven categories, from reasoning and code to vision and audio, all through a single OpenAI-compatible endpoint. There are no cold starts on popular models, which means your throughput does not collapse during traffic spikes due to container warmup.

Conclusion

Maximizing LLM throughput is a stack-wide problem. Tune your client with connection reuse and bounded concurrency. Design prompts for efficient server-side batching. Cache aggressively. And choose a provider whose pricing model rewards the optimizations you implement rather than penalizing them.

Oxlo.ai is built for this pattern. Its request-based pricing removes the cost penalty for long prompts, the OpenAI-compatible API lets you switch base URLs without rewriting code, and the absence of cold starts keeps latency predictable under load