Debugging LLM Inference: Best Practices and Techniques

#engineering #oxlo #ai

I shipped an internal debugging agent that ingests raw inference traces and tells us exactly why a prompt is failing or producing garbage. In this tutorial, we will build that same tool, a small Python script that diagnoses LLM inference issues and suggests concrete fixes. It runs entirely against Oxlo.ai's API, so you get flat per-request pricing while iterating on long system prompts and context windows.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Scaffold the client and trace loader

We start with a minimal script that reads a JSON trace file and initializes the Oxlo.ai client. I keep traces in a flat schema: model, messages, temperature, max_tokens, and the raw response.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def load_trace(path: str) -> dict:
    with open(path) as f:
        return json.load(f)

trace = load_trace("trace.json")
print(f"Loaded trace for model {trace['model']}")

Step 2: Write the debugger system prompt

The agent needs to act like a senior ML engineer. I give it a strict system prompt that forces structured reasoning about common failure modes.

SYSTEM_PROMPT = """You are an LLM inference debugger. Analyze the provided trace and identify exactly one issue from this list:
- truncated_output (max_tokens too low)
- temperature_randomness (temp too high for deterministic task)
- context_overflow (input near or over context limit)
- instruction_override (system prompt ignored or jailbroken)
- schema_mismatch (JSON mode or function calling misused)
- no_issue (trace looks correct)

Respond in JSON with keys: issue, confidence (0-1), explanation, fix_suggestion.
Be concise. Do not guess."""

Step 3: Format the trace and diagnose

We pack the trace metadata into a single user message so the model has full context. I use Oxlo.ai's JSON mode to guarantee parseable output.

def build_diagnostic_prompt(trace: dict) -> str:
    return json.dumps({
        "model_used": trace["model"],
        "temperature": trace.get("temperature", 1.0),
        "max_tokens": trace.get("max_tokens", -1),
        "num_messages": len(trace["messages"]),
        "response_text": trace["response"]["choices"][0]["message"]["content"],
        "finish_reason": trace["response"]["choices"][0]["finish_reason"],
    }, indent=2)

user_message = build_diagnostic_prompt(trace)

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    response_format={"type": "json_object"},
    temperature=0.1,
)

diagnosis = json.loads(response.choices[0].message.content)
print(json.dumps(diagnosis, indent=2))

Step 4: Apply the fix and replay

Now we wire up a replay function that applies the debugger's suggestion and re-runs the conversation. This validates the fix without burning tokens on guesswork. Because Oxlo.ai uses request-based pricing, replaying a long-context trace costs the same flat rate as the first call.

def apply_fix(trace: dict, diagnosis: dict) -> dict:
    fixed = trace.copy()
    suggestion = diagnosis.get("fix_suggestion", "").lower()

    if "max_tokens" in suggestion and "increase" in suggestion:
        fixed["max_tokens"] = trace.get("max_tokens", 256) * 4
    if "temperature" in suggestion and "lower" in suggestion:
        fixed["temperature"] = 0.0
    if "json" in suggestion and "schema" in suggestion:
        fixed["response_format"] = {"type": "json_object"}

    return fixed

fixed_trace = apply_fix(trace, diagnosis)

replay = client.chat.completions.create(
    model=fixed_trace["model"],
    messages=fixed_trace["messages"],
    max_tokens=fixed_trace.get("max_tokens"),
    temperature=fixed_trace.get("temperature"),
)

print("Replay finish_reason:", replay.choices[0].finish_reason)
print("Replay output:", replay.choices[0].message.content[:200])

Step 5: Wrap it in a CLI

Finally, we add argument parsing so we can run python debugger.py trace.json against any trace. This is the finished tool.

import argparse

def main():
    parser = argparse.ArgumentParser(description="Debug an LLM inference trace")
    parser.add_argument("trace_file")
    args = parser.parse_args()

    trace = load_trace(args.trace_file)
    user_message = build_diagnostic_prompt(trace)

    diag_resp = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    diagnosis = json.loads(diag_resp.choices[0].message.content)

    if diagnosis["issue"] == "no_issue":
        print("No issue detected.")
        return

    print(f"Issue: {diagnosis['issue']} (confidence: {diagnosis['confidence']})")
    print(f"Explanation: {diagnosis['explanation']}")

    fixed_trace = apply_fix(trace, diagnosis)
    replay = client.chat.completions.create(
        model=fixed_trace["model"],
        messages=fixed_trace["messages"],
        max_tokens=fixed_trace.get("max_tokens"),
        temperature=fixed_trace.get("temperature"),
    )
    print(f"Replay finish_reason: {replay.choices[0].finish_reason}")

if __name__ == "__main__":
    main()

Run it

Create a file named trace.json with a deliberately broken trace. Here the model is asked for a detailed summary but max_tokens is set to 50, forcing a truncation.

{
  "model": "llama-3.3-70b",
  "temperature": 1.2,
  "max_tokens": 50,
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a detailed summary of quantum computing."}
  ],
  "response": {
    "choices": [
      {
        "message": {"content": "Quantum computing is a type of computation that harnesses..."},
        "finish_reason": "length"
      }
    ]
  }
}

Run the debugger:

$ python debugger.py trace.json
Issue: truncated_output (confidence: 0.95)
Explanation: finish_reason is 'length' and max_tokens is only 50 for a detailed summary request.
Replay finish_reason: stop

The tool correctly flags the truncation, bumps max_tokens, and verifies that the replay completes.

Next steps

Plug this into your CI pipeline to automatically flag regressions whenever a trace returns an unexpected finish reason. You can also extend the script to batch-test fixes across multiple Oxlo.ai models, such as DeepSeek V3.2 and Qwen 3 32B, because flat per-request pricing makes A/B testing on long contexts cheap.

DEV Community