Debugging LLM Model Performance: Best Practices and Techniques

#engineering #oxlo #ai

I spend too much time staring at malformed JSON and vague model outputs trying to guess what went wrong. In this tutorial we will build a small trace debugger that captures a failed LLM call, diagnoses the root cause with a reasoning model, and validates a rewritten prompt. We will run every step against Oxlo.ai so you can iterate on long traces without token costs scaling with context length.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Capture a trace

I use a dataclass that records exactly what happened during a bad call. I store the model name, raw prompt, raw response, latency, and any parser error so the debugger has full context.

import json
from dataclasses import dataclass

@dataclass
class Trace:
    model: str
    prompt: str
    response: str
    latency_ms: float
    error: str | None = None

# A realistic failure: generic model asked for structured JSON, returns prose
bad_trace = Trace(
    model="generic-7b",
    prompt="Extract all liabilities from the 10-K as JSON.",
    response="The company has several liabilities including long-term debt...",
    latency_ms=850.0,
    error="JSONDecodeError: Expecting value: line 1 column 1",
)

Step 2: Craft the debugger prompt

I treat the system prompt as the entire brain of the tool. It constrains the model to classify the failure and return a predictable JSON schema. I keep it strict so I do not have to guess at the output format.

SYSTEM_PROMPT = """You are an LLM performance debugger.
Analyze the trace below and classify the root cause into exactly one category:
- model mismatch (wrong model size or family for the task)
- prompt ambiguity (missing schema, examples, or output format)
- output parsing (response cannot be parsed into the expected structure)
- latency (unnecessary overhead for a simple task)

Respond with a JSON object containing:
- issue: the category
- explanation: one concise sentence
- recommended_model: one of llama-3.3-70b, qwen-3-32b, deepseek-v3.2, kimi-k2.6
- rewritten_prompt: a clearer prompt that fixes the issue
- confidence: high, medium, or low
"""

Step 3: Diagnose with a reasoning model

I send the trace to DeepSeek R1 671B on Oxlo.ai because deep reasoning catches subtle prompt engineering mistakes. Oxlo.ai's request-based pricing is useful here: I can paste in long prompts and stack traces without the cost scaling by token count, which keeps iterative debugging predictable. The endpoint is fully OpenAI compatible, so my client code is a drop-in replacement.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def diagnose(trace: Trace) -> dict:
    user_message = f"""Model: {trace.model}
Latency: {trace.latency_ms}ms
Error: {trace.error}
Prompt: {trace.prompt}
Response: {trace.response}"""

    response = client.chat.completions.create(
        model="deepseek-r1-671b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

Step 4: Validate the fix

I do not trust a diagnosis until I validate it. I take the recommended model and rewritten prompt from the JSON, then call Oxlo.ai again to verify the fix actually works. I often see the debugger recommend Llama 3.3 70B for general extraction, or Kimi K2.6 when the task mixes reasoning and coding.

def validate_fix(trace: Trace, diagnosis: dict) -> str:
    print(f"Detected issue: {diagnosis['issue']}")
    print(f"Explanation: {diagnosis['explanation']}")
    print(f"Recommended model: {diagnosis['recommended_model']}")

    response = client.chat.completions.create(
        model=diagnosis["recommended_model"],
        messages=[
            {"role": "system", "content": "Follow the user's instructions exactly."},
            {"role": "user", "content": diagnosis["rewritten_prompt"]},
        ],
    )
    return response.choices[0].message.content

Run it

This main block ties the pieces together. It diagnoses the bad trace, prints the structured report, and runs the corrected prompt.

if __name__ == "__main__":
    diagnosis = diagnose(bad_trace)
    print(json.dumps(diagnosis, indent=2))

    fixed_output = validate_fix(bad_trace, diagnosis)
    print("\n--- Fixed output ---")
    print(fixed_output)

Example output:

Detected issue: prompt ambiguity
Explanation: The prompt requests JSON but provides no schema or example.
Recommended model: llama-3.3-70b
{
  "issue": "prompt ambiguity",
  "explanation": "The prompt requests JSON but provides no schema or example.",
  "recommended_model": "llama-3.3-70b",
  "rewritten_prompt": "Extract all liabilities from the 10-K. Return valid JSON with keys: short_term, long_term, total. Example: {\"short_term\": 1000000, \"long_term\": 5000000, \"total\": 6000000}",
  "confidence": "high"
}

--- Fixed output ---
{
  "short_term": 1200000,
  "long_term": 4800000,
  "total": 6000000
}

Next steps

Wire this script into your CI pipeline so it runs automatically whenever a parser error is detected in staging. You could also extend the validator to A/B test the old prompt against the rewritten one across several Oxlo.ai models and pick the winner by latency and accuracy. If you want predictable pricing while running these experiments, check the request-based plans at https://oxlo.ai/pricing.

DEV Community