Debugging large language models is not like debugging a traditional REST endpoint. When a response drifts, a tool call misfires, or a JSON schema collapses into free text, the culprit is rarely a stack trace. It is usually hidden inside a system prompt, a temperature setting, or a context window that silently truncated your few-shot examples. This guide covers practical techniques for isolating and fixing failures in production LLM pipelines, with concrete code you can run today.
Lock Down Randomness and Context
The first rule of LLM debugging is to make the failure reproducible. Start by pinning temperature to 0.0 and setting a fixed seed if the model supports it. Non-determinism is useful for creative tasks, but it destroys your ability to bisect a bug.
When you use Oxlo.ai, you interact with a fully OpenAI SDK compatible API. Changing the base URL is enough to start debugging against any model in the catalog, from Llama 3.3 70B to DeepSeek R1 671B MoE. Here is a minimal client configured for reproducibility:
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"]
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Explain recursion."}],
temperature=0.0,
# seed=42, # uncomment if the model supports it
)
print(response.choices[0].message.content)
Keep the message list immutable between runs. Even a single newline change can shift token probabilities and alter the output. If you need to test across multiple architectures to see whether the bug is model-specific or prompt-specific, swap the model string and rerun. Because Oxlo.ai exposes more than 45 models through one endpoint, you do not need to rewire your client.
Trace the Full Prompt
Most production bugs are prompt bugs. The version of the prompt you think you sent is rarely the version the model receives. Middleware can inject guardrails, retrieval pipelines can append stale context, and user inputs can contain adversarial whitespace.
Log the exact message array and system prompt before every API call. If you are hitting context limits, log the length in characters as a proxy for token count. On token-based providers, long traces inflate costs linearly, which discourages thorough logging. Oxlo.ai uses flat per-request pricing, so sending a fully expanded prompt with retrieved documents and few-shot examples costs the same whether it is 500 tokens or 50,000 tokens. That removes the cost penalty for verbose debugging traces during incident response.
Validate Structured Output
When you ask a model for JSON, you are really asking for a grammar-constrained sampler. If the output is malformed, the failure is often in the prompt, not the parser. Explicitly request JSON mode and supply a schema in the system prompt.
import json
response = client.chat.completions.create(
model="qwen3-32b",
messages=[
{"role": "system", "content": "You are a helpful assistant. Respond with valid JSON matching this schema: {\"confidence\": float, \"answer\": string}"},
{"role": "user", "content": "What is the capital of France?"}
],
response_format={"type": "json_object"},
temperature=0.0,
)
raw = response.choices[0].message.content
data = json.loads(raw)
Oxlo.ai supports JSON mode and streaming across its chat models. If you are debugging streaming parsers, remember that the final chunk may contain finish-reason metadata. Validate the assembled string before parsing, and fail loudly if the model emits trailing prose after the closing brace.
Debug Tool Use and Agent Loops
Function calling introduces a new class of bugs: hallucinated function names, arguments that violate schemas, and infinite loops where the model keeps calling the same tool with mutated parameters. The best defense is a strict dispatcher.
AVAILABLE_TOOLS = {"search", "calculator", "fetch_user"}
def dispatch_tool(name: str, args: dict) -> str:
if name not in AVAILABLE_TOOLS:
raise ValueError(f"Hallucinated tool: {name}")
# ... run tool ...
Log every turn of an agentic loop, including the model's tool_calls, the tool responses, and the assistant messages that follow. Long tool descriptions and multi-turn conversations can rack up token counts quickly on token-based platforms. Oxlo.ai's request-based pricing is designed for this: one flat cost per request regardless of how many tools are described or how long the conversation history grows. That makes it practical to iterate on agentic workflows without surprise bills.
Models such as Qwen 3 32B, GLM 5, and Minimax M2.5 on Oxlo.ai are specifically strong at agentic tool use, so if your current model misfires on function boundaries, switching architectures is a single parameter change.
Inspect Context Windows
Silent truncation is one of the hardest bugs to catch. If your instructions sit at the start of a long prompt and the model ignores them, the context window may have swallowed the tail. Some providers truncate from the middle; others truncate from the beginning. You need to know which strategy your backend uses.
Oxlo.ai hosts long-context models such as DeepSeek V4 Flash with a 1 million token context window and Kimi K2.6 with 131K tokens. When you have the room, place your core instructions at both the top and the bottom of the prompt, or use explicit delimiters to mark sections. Because Oxlo.ai does not charge per token, filling a long context window with retrieval-augmented generation or full codebase context does not scale your debugging costs. You pay per request, which lets you test extreme context lengths to verify whether truncation is the root cause.
Log Requests and Responses
Always capture the request ID, timestamp, and raw response body. If you are using streaming, accumulate chunks in a buffer and log the complete message only after finish_reason is received. This preserves the exact bytes that your parser sees.
stream = client.chat.completions.create(
model="deepseek-v3.2",
messages=messages,
stream=True,
)
buffer = ""
for chunk in stream:
delta = chunk.choices[0].delta.content or ""
buffer += delta
# optional: log chunk for latency analysis
print(buffer)
Oxlo.ai returns streaming responses with no cold starts on popular models, so the first chunk arrives immediately. When you are debugging latency versus content quality, that fast start matters. It lets you distinguish between a slow model and a stalled connection.
Swap Models to Isolate Regressions
If a prompt that worked yesterday now produces garbage, the problem may be a model version update, a quantization change, or a shift in your own retrieval pipeline. The fastest way to bisect is to hold the prompt constant and swap the model.
With Oxlo.ai, the same OpenAI SDK call works across the entire catalog. You can move from a generalist like Llama 3.3 70B to a reasoning specialist like DeepSeek R1 671B MoE, or to an agent-oriented model like Kimi K2.6, without rewriting client code. Because pricing is request-based rather than token-based, testing a large-parameter model does not trigger a disproportionate charge. You pay one flat cost per request, so you can afford to run A/B tests across multiple architectures to find the one that respects your prompt structure.
Conclusion
Debugging LLMs is an exercise in controlling variables: lock the seed, freeze the prompt, log the full trace, and swap one model at a time. The biggest friction in that loop is usually cost. Every long context dump, every agentic turn, and every test against a larger model inflates the bill on token-based platforms.
Oxlo.ai removes that friction with flat per-request pricing. Whether you are sending 1,000 tokens or 100,000 tokens, the cost is the same, so you can debug long-context and agentic workloads without budget anxiety. With more than 45 models, full OpenAI SDK compatibility, and no cold starts, you can focus on fixing the bug instead of calculating the bill. See the pricing page to compare plans, or point your existing client to https://api.oxlo.ai/v1 and start debugging now.
Top comments (0)