Debugging LLM Models

#engineering #oxlo #ai

I built a small CLI tool that debugs bad LLM outputs by treating another LLM as a critic. It takes a broken prompt-response pair, diagnoses the failure, and rewrites the system prompt. Because I run it on Oxlo.ai, iterating on long debugging traces costs the same flat rate per request regardless of how much context I stuff into the call.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Capture a bad response

First we need a realistic failure. I will use a lazy system prompt that asks for JSON but forgets to forbid markdown fences and filler text. We call Oxlo.ai's Llama 3.3 70B and store the output.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = (
    "You are a helpful assistant. Reply with JSON."
)

user_message = "Summarize the benefits of request-based pricing."

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

bad_response = response.choices[0].message.content
print("=== BAD RESPONSE ===")
print(bad_response)

Step 2: Write the debugger prompt

The debugger is just another chat completion with a carefully written system prompt. It must return structured JSON, so I tell it exactly what keys to use and ban markdown wrapping.

DEBUGGER_SYSTEM_PROMPT = """
You are a prompt debugger. Analyze the following interaction and identify why the assistant failed to follow instructions.

Return strictly a JSON object with no markdown formatting and no extra commentary. Use this exact structure:
{
  "diagnosis": "One sentence describing the root cause.",
  "issues": ["List of specific problems."],
  "fixed_system_prompt": "A rewritten system prompt that would fix the issues."
}

Here is the data to analyze:
- Original system prompt: {original_system}
- User message: {user_message}
- Actual assistant output: {actual_output}
"""

Step 3: Build the debugger client

Now we feed the bad interaction into the debugger. I use Kimi K2.6 on Oxlo.ai because its reasoning and coding strengths handle meta-analysis well. We parse the JSON output after stripping any accidental markdown fences.

import json

debugger_user_message = f"""Original system prompt:
{SYSTEM_PROMPT}

User message:
{user_message}

Actual assistant output:
{bad_response}"""

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": DEBUGGER_SYSTEM_PROMPT},
        {"role": "user", "content": debugger_user_message},
    ],
)

raw = response.choices[0].message.content.strip()

if raw.startswith("

```"):
    raw = raw.split("```

")[1]
    if raw.startswith("json"):
        raw = raw[4:]

result = json.loads(raw.strip())
print(json.dumps(result, indent=2))

Step 4: Apply the fix and re-run

Finally we swap in the fixed system prompt and rerun the original user message against Llama 3.3 70B. If the debugger did its job, the new output should follow instructions without filler.

SYSTEM_PROMPT = result["fixed_system_prompt"]

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

good_response = response.choices[0].message.content
print("=== FIXED RESPONSE ===")
print(good_response)

Run it

Save the pieces above as debug_agent.py and run it. My last execution produced this output.

$ python debug_agent.py
=== BAD RESPONSE ===
Sure, here is the JSON you requested:

{
  "summary": "Request-based pricing helps you predict costs."
}

=== DEBUGGER OUTPUT ===
{
  "diagnosis": "The system prompt did not forbid conversational filler or specify a JSON schema.",
  "issues": [
    "No explicit instruction to output only JSON.",
    "No example or schema provided."
  ],
  "fixed_system_prompt": "You are a helpful assistant. Output ONLY valid JSON. Do not include markdown fences, explanations, or filler text. The JSON must contain a single key 'summary' with a string value."
}

=== FIXED RESPONSE ===
{"summary": "Request-based pricing offers predictable, flat costs per API call, making budgeting simpler and eliminating surprise token overages."}

Next steps

Integrate this debugger into your evaluation pipeline. Every time a prompt fails a unit test, pass the trace to Oxlo.ai and automatically open a PR with the fixed system prompt.

Point the debugger at Kimi K2.6 on Oxlo.ai when you need to diagnose long agent traces. Its 131K context window lets you feed entire conversation histories or codebase chunks into a single flat-rate request.

DEV Community