Debugging LLM Inference Performance: Best Practices and Tools

Slow inference is rarely a random failure. It is usually the result of a specific bottleneck in the serving pipeline: prefill computation, decode memory bandwidth, KV cache pressure, or suboptimal batching. To debug effectively, you must treat the LLM API as a measurable system with distinct phases rather than an opaque text generator. This article walks through the metrics, tools, and architectural decisions that let you isolate latency and throughput issues, and shows how Oxlo.ai removes common infrastructure variables so your profiling reflects model behavior, not platform noise.

Measuring What Matters: TTFT and TPOT

The two most important client-side metrics are Time To First Token (TTFT) and Time Per Output Token (TPOT), sometimes measured as inter-token latency. TTFT captures the prefill phase, where the model processes your prompt and builds the KV cache. TPOT captures the decode phase, where each new token is generated sequentially. A high TTFT with a low TPOT means your prompt is the problem. A low TTFT with a high TPOT means the model is memory-bandwidth bound during generation.

Because Oxlo.ai is fully OpenAI SDK compatible, you can reuse existing telemetry scripts by changing only the base URL. The following Python snippet records both TTFT and TPOT for any chat model:

import os
import time
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY")
)

messages = [{"role": "user", "content": "Write a detailed analysis of cache eviction policies."}]

start = time.perf_counter()
stream = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=messages,
    stream=True,
    max_tokens=400
)

ttft = None
token_times = []

for chunk in stream:
    now = time.perf_counter()
    if ttft is None:
        ttft = now - start
        print(f"TTFT: {ttft:.3f}s")
    else:
        token_times.append(now - last)
    last = now

avg_tpot = sum(token_times) / len(token_times) if token_times else 0
print(f"Average TPOT: {avg_tpot:.3f}s")

Store these measurements in your observability stack. Look for outliers at the p99, not just the median, because tail latency is what breaks user experience.

Client-Side Telemetry and Distributed Tracing

If you self-host, you can instrument the inference engine directly with Prometheus and Grafana. With managed providers, the server stack is a black box, so client-side telemetry is the only signal you fully control. Wrap every API call with OpenTelemetry spans that capture TTFT, TPOT, total request duration, prompt length, and completion length.

Structured logging gives you the data to correlate latency spikes with specific prompts or model versions. When you switch to Oxlo.ai, the OpenAI SDK compatibility means you do not need to rewrite your instrumentation layer. Your existing spans, retries, and timeout logic continue to work because the response schema and streaming format are identical.

import time
import logging
from opentelemetry import trace

tracer = trace.get_tracer("llm.inference")

def traced_completion(client, model, messages, max_tokens):
    with tracer.start_as_current_span("llm.chat") as span:
        start = time.perf_counter()
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            stream=True,
            max_tokens=max_tokens
        )
        # ... stream handling and metric recording ...
        span.set_attribute("model", model)
        span.set_attribute("ttft", ttft)
        span.set_attribute("tpot", avg_tpot)
        return response

Use these traces to answer whether a slowdown is reproducible on a specific model, a specific prompt shape, or a specific time of day.

Separating Prefill from Decode Problems

Prefill latency grows with input length. Long system prompts, few-shot examples, and agentic tool trajectories all expand the prefill phase. If your TTFT is high, try shrinking the context window, switching to a model with a more efficient attention implementation, or enabling prompt caching if the provider supports it.

Decode latency is harder to fix because it is often memory-bandwidth bound. A larger model or a higher quantization level usually improves TPOT, but at the cost of accuracy. If your application requires long outputs, consider whether a smaller, faster model can produce an acceptable draft that a larger model refines