When you are shipping an agent to production, the hardest part is figuring out why a model fails on edge cases. In this guide, we are building an automated LLM performance debugger that runs prompts against multiple Oxlo.ai models, scores the outputs, and generates a diagnosis report. It is designed for teams that need to validate reasoning quality before deploying to users.
What you'll need
- Python 3.10 or higher
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Build the evaluation harness
We need a lightweight framework that sends a prompt to a model, captures the response, and records latency. Oxlo.ai's flat per-request pricing makes this cheap even when we send long system prompts or few-shot examples.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def run_test(model, prompt):
start = time.time()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0.2,
)
latency = time.time() - start
return {
"model": model,
"output": response.choices[0].message.content,
"latency": round(latency, 2),
}
prompt = "Explain the trade-offs between vector search and keyword search for legal document retrieval."
result = run_test("llama-3.3-70b", prompt)
print(result)Step 2: Run multi-model benchmarks
We will test the same prompt against three Oxlo.ai models with different strengths. Llama 3.3 70B handles general reasoning, Qwen 3 32B covers multilingual tasks, and DeepSeek V3.2 targets coding workloads. This surfaces which architecture handles your specific workload best.
models = ["llama-3.3-70b", "qwen-3-32b", "deepseek-v3.2"]
prompt = "Write a Python function that safely parses a nested JSON string up to 5 levels deep, returning None on any malformed input."
results = []
for model in models:
print(f"Testing {model}...")
results.append(run_test(model, prompt))
for r in results:
print(f"{r['model']}: {r['latency']}s")Step 3: Add an automated judge
Raw outputs are hard to compare at scale, so we will use Kimi K2.6 to score each response on correctness, instruction following, and clarity. The judge runs on Oxlo.ai as well, keeping everything on one bill and one API format.
Here is the system prompt we will use for the judge:
JUDGE_SYSTEM_PROMPT = """You are an expert evaluator. Score the assistant response below on a scale of 1 to 10 for:
1. Correctness (facts and logic)
2. Instruction following (did it do what was asked)
3. Clarity (is it readable and well structured)
Respond in this exact JSON format:
{
"correctness": int,
"instruction_following": int,
"clarity": int,
"summary": "one sentence verdict"
}
"""import json
def judge_response(model_output, original_prompt):
evaluation_prompt = f"Original prompt: {original_prompt}\n\nAssistant response:\n{model_output}"
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": evaluation_prompt},
],
response_format={"type": "json_object"},
temperature=0.1,
)
return response.choices[0].message.content
for r in results:
r["scores"] = judge_response(r["output"], prompt)
print(f"Scores for {r['model']}: {r['scores']}")Step 4: Build the failure analyzer
When the judge flags a low score, we need to know why. We will add a final analysis step that feeds the original prompt, the weak response, and the judge's critique into DeepSeek V3.2 to generate a concrete fix recommendation.
ANALYZER_TEMPLATE = """You are a debugging specialist. A language model produced a weak response to a user prompt.
User prompt:
{prompt}
Model response:
{output}
Judge feedback:
{feedback}
Diagnose the root cause. Choose one of: reasoning error, instruction drift, hallucination, format violation, or insufficient detail. Then give a one-paragraph fix recommendation.
"""
def analyze_failure(model_output, original_prompt, feedback):
filled = ANALYZER_TEMPLATE.format(
prompt=original_prompt,
output=model_output,
feedback=feedback,
)
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[{"role": "user", "content": filled}],
temperature=0.3,
)
return response.choices[0].message.content
for r in results:
scores = json.loads(r["scores"])
if scores["correctness"] < 7:
diagnosis = analyze_failure(r["output"], prompt, r["scores"])
print(f"Diagnosis for {r['model']}:\n{diagnosis}\n")Run it
The script below ties everything together. It runs the benchmark, scores each model, and prints a diagnosis for any response that scores below 7 on correctness.
if __name__ == "__main__":
test_prompt = "Write a Python function that safely parses a nested JSON string up to 5 levels deep, returning None on any malformed input."
models = ["llama-3.3-70b", "qwen-3-32b", "deepseek-v3.2"]
results = []
for m in models:
results.append(run_test(m, test_prompt))
for r in results:
r["scores"] = judge_response(r["output"], test_prompt)
for r in results:
scores = json.loads(r["scores"])
print(f"{r['model']} | Latency: {r['latency']}s | Correctness: {scores['correctness']}/10")
if scores["correctness"] < 7:
diag = analyze_failure(r["output"], test_prompt, r["scores"])
print(f" Issue detected: {diag}")Example output:
llama-3.3-70b | Latency: 1.24s | Correctness: 9/10
qwen-3-32b | Latency: 0.98s | Correctness: 8/10
deepseek-v3.2 | Latency: 1.45s | Correctness: 10/10Wrap-up
You now have a repeatable debugger that compares models and explains failures. Two concrete next steps: integrate this into your CI pipeline to catch regressions on every deploy, or extend the harness with vision tasks using Kimi VL A3B to debug multimodal prompts. For pricing details on running these evaluations at scale, see https://oxlo.ai/pricing.
Top comments (0)