Debugging LLM inference in production means dealing with silent failures, malformed JSON, and context overflows. I recently shipped a small debugging harness for a customer support agent that catches these issues before they reach users. In this tutorial, we will build that harness step by step on Oxlo.ai, using its OpenAI-compatible API and flat per-request pricing to iterate without watching token counters. See https://oxlo.ai/pricing for plan details.
What you'll need
- Python 3.10 or newer
pip install openai- An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Scaffold the client and make a raw call
Before adding logic, confirm that you can hit Oxlo.ai and inspect the raw response shape. I start every project with a minimal script that prints the full response object.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Why is my API call slow?"},
],
)
print(response.model_dump_json(indent=2))
Step 2: Define the agent and its system prompt
We need a concrete workload to debug. I use a customer support intent classifier that must return strict JSON. Keeping the system prompt in its own constant makes it easy to iterate without touching business logic.
SYSTEM_PROMPT = """You are a support intent classifier. Extract the user's issue category, urgency, and required action.
Rules:
- category must be one of: billing, technical, account, general.
- urgency must be one of: low, medium, high, critical.
- action must be a single sentence describing the next step.
- Output ONLY valid JSON. Do not wrap it in markdown fences."""
SAMPLE_USER_MESSAGE = "I was charged twice this month and I need a refund immediately."
Step 3: Add request/response logging
Most inference bugs are invisible until you log the exact payload that left your client. I wrap the Oxlo.ai call in a small helper that records model name, finish_reason, and the full response content to stdout.
import json
import time
def debug_chat_completion(client, model, messages, temperature=0.2):
start = time.time()
response = client.chat.completions.create(
model=model,
messages=messages,
temperature=temperature,
)
latency = time.time() - start
message = response.choices[0].message
finish_reason = response.choices[0].finish_reason
log = {
"model": model,
"latency_ms": round(latency * 1000, 2),
"finish_reason": finish_reason,
"content": message.content,
}
print(json.dumps(log, indent=2))
return response
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": SAMPLE_USER_MESSAGE},
]
debug_chat_completion(client, "llama-3.3-70b", messages)
Step 4: Enforce JSON mode and validate output
A common inference failure is a model ignoring JSON instructions and returning markdown. We can add response_format and a validation loop. If parsing fails, we log the raw text and retry once with a stronger model.
import json
from json import JSONDecodeError
def safe_json_completion(client, model, messages, max_retries=1):
for attempt in range(max_retries + 1):
response = client.chat.completions.create(
model=model,
messages=messages,
response_format={"type": "json_object"},
temperature=0.1,
)
raw = response.choices[0].message.content
try:
parsed = json.loads(raw)
print(f"Attempt {attempt + 1}: parsed OK")
return parsed
except JSONDecodeError as e:
print(f"Attempt {attempt + 1}: JSON parse failed: {e}")
print(f"Raw output: {raw[:500]}")
if attempt == 0:
model = "qwen-3-32b"
print("Falling back to qwen-3-32b for retry.")
else:
raise
result = safe_json_completion(client, "deepseek-v3.2", messages)
print(json.dumps(result, indent=2))
Step 5: Handle context length with model fallback
When you pass long transcripts or logs, you can hit token limits. Instead of crashing, catch the error and switch to a model with a larger context window. Oxlo.ai carries Kimi K2.6 with 131K context, which is useful for this exact case.
from openai import BadRequestError
def resilient_chat_completion(client, messages, long_context_model="kimi-k2.6"):
try:
return client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
temperature=0.2,
)
except BadRequestError as e:
error_msg = str(e).lower()
if "context length" in error_msg or "too long" in error_msg:
print(f"Context limit hit on llama-3.3-70b. Switching to {long_context_model}.")
return client.chat.completions.create(
model=long_context_model,
messages=messages,
temperature=0.2,
)
raise
# Example long input
long_input = SAMPLE_USER_MESSAGE + " Here is the full log: " + ("x " * 3000)
long_messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": long_input},
]
response = resilient_chat_completion(client, long_messages)
print(response.choices[0].message.content[:200])
Step 6: Build an interactive debugging REPL
The fastest way to debug a prompt is to talk to it interactively. I add a small loop that prints the parsed JSON, raw finish_reason, and lets me swap models on the fly.
import json
from json import JSONDecodeError
def debug_repl(client, default_model="llama-3.3-70b"):
print("Inference debugger REPL. Type 'quit' to exit, 'model <name>' to switch.")
current_model = default_model
while True:
user_input = input("\nUser> ").strip()
if user_input.lower() == "quit":
break
if user_input.lower().startswith("model "):
current_model = user_input.split(" ", 1)[1]
print(f"Switched to {current_model}")
continue
msgs = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_input},
]
try:
resp = client.chat.completions.create(
model=current_model,
messages=msgs,
response_format={"type": "json_object"},
temperature=0.1,
)
msg = resp.choices[0].message
print(f"Model: {current_model}")
print(f"Finish reason: {resp.choices[0].finish_reason}")
try:
parsed = json.loads(msg.content)
print("Parsed JSON:")
print(json.dumps(parsed, indent=2))
except JSONDecodeError:
print("Raw output (invalid JSON):")
print(msg.content)
except Exception as e:
print(f"Error: {e}")
if __name__ == "__main__":
debug_repl(client)
Run it
Paste the pieces together into debug_agent.py, set your OXLO_API_KEY, and run python debug_agent.py. Here is a sample session using Oxlo.ai flat request pricing to test multiple models without token anxiety.
$ export OXLO_API_KEY=oxlo_...
$ python debug_agent.py
Inference debugger REPL. Type 'quit' to exit, 'model <name>' to switch.
User> I was charged twice this month and I need a refund immediately.
Model: llama-3.3-70b
Finish reason: stop
Parsed JSON:
{
"category": "billing",
"urgency": "high",
"action": "Process a refund for the duplicate charge and confirm with the user."
}
User> model qwen-3-32b
Switched to qwen-3-32b
User> My account is locked after too many password attempts.
Model: qwen-3-32b
Finish reason: stop
Parsed JSON:
{
"category": "account",
"urgency": "medium",
"action": "Unlock the account and prompt the user to reset their password."
}
Next steps
You now have a small debugger that logs every Oxlo.ai call, validates JSON, and falls back gracefully on context limits. Wire this into your FastAPI middleware so every production request is captured, or add Pydantic schemas so validation failures return typed errors instead of raw strings. Both are easier when your provider charges per request, because iterating on long prompts with Kimi K2.6 or DeepSeek R1 does not inflate your bill.
Top comments (0)