Production LLMs fail in ways that rarely appear on benchmark leaderboards. When a summarization task silently drops key entities, or an agent loops endlessly on a malformed tool call, you need a diagnostic framework rather than a new model card. This field guide covers the most common production failure modes, with concrete reproduction steps and mitigations you can apply today.
Context Window Failures
The most common silent failure is context truncation. A model may appear to ignore instructions buried in the middle of a long prompt, or it may skip entire sections of retrieved documentation. This is often the lost in the middle effect combined with accidental truncation.
Diagnosis. Log the exact token count of your prompt and compare it against the model's context limit. If you are near the boundary, some providers truncate the input without raising an error. Move system instructions to the start or end of the context window, and keep critical constraints within the first or last 25% of the prompt.
Mitigation. Chunk long documents and use map-reduce patterns. If your workflow requires monolithic context, switch to a model with a larger window. On Oxlo.ai, DeepSeek V4 Flash supports a 1M context window, and Kimi K2.6 offers 131K tokens for advanced reasoning and vision workloads. Because Oxlo.ai uses request-based pricing, sending a long prompt for debugging or retrieval does not scale your cost with input length. See https://oxlo.ai/pricing for details.
Non-Determinism and Reproducibility
Even with temperature set to zero, different runs can yield different outputs depending on logit sampling, hardware nondeterminism, or provider-side load balancing. This makes regression testing difficult.
Diagnosis. Fix your seed, lock temperature and top_p, and log the full payload including the exact model version. Compare hashes of the output across multiple runs.
Mitigation. If your provider supports it, set a deterministic seed. Implement a semantic cache for identical prompts so you do not pay for repeated inferences. For integration tests, snapshot acceptable output ranges rather than exact strings.
On Oxlo.ai, you can pass standard OpenAI SDK parameters such as seed and temperature. Because the platform charges per request rather than per token, running a regression suite with dozens of prompt variants does not produce unpredictable token bills. This makes it practical to test for nondeterminism continuously.
Tool Use and Function Calling
Function calling failures usually fall into three categories: hallucinated parameter names, incorrect tool selection, and ignoring the tool entirely in favor of plain text.
Diagnosis. Inspect the schema you are sending. Overly nested objects, vague descriptions, or twenty available tools at once will confuse any model. Reduce the tool set to only what is needed for the current turn.
Mitigation. Add one-shot or few-shot examples inside the tool description. Use tool_choice to force a specific function when the workflow stage is known. Validate arguments with a JSON schema validator before executing.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_API_KEY"
)
response = client.chat.completions.create(
model="qwen3-32b",
messages=[{"role": "user", "content": "What is the weather in Paris?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather. Example: {\"location\": \"Paris\", \"unit\": \"celsius\"}",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
},
"required": ["location"]
}
}
}],
tool_choice={"type": "function", "function": {"name": "get_weather"}}
)Oxlo.ai supports function calling across its catalog, including agentic models such as Qwen 3 32B, GLM 5, and Minimax M2.5. The API is fully OpenAI SDK compatible, so your existing tool definitions port without client-side changes.
Latency and Throughput Bottlenecks
High time-to-first-token (TTFT) or slow inter-token latency breaks user experience. The root cause is often a mismatch between model size and task complexity.
Diagnosis. Measure TTFT separately from total generation time. If TTFT is high but tokens arrive quickly afterward, you are likely queueing on the provider side or sending an oversized prompt. If every token is slow, the model is simply too large for your latency budget.
Mitigation. Enable streaming so users see partial results immediately. For routing or classification, use a smaller model. Reserve large reasoning models for steps that actually need deep inference.
Oxlo.ai offers no cold starts on popular models and supports streaming responses. For latency-sensitive stages, route to Oxlo.ai Coder Fast or DeepSeek V3.2. For heavy reasoning, use DeepSeek R1 671B MoE or Kimi K2 Thinking. The flat per-request pricing means you can cache routine calls and only pay for the hard ones, without input-length penalties.
Structured Output and JSON Mode
JSON mode failures are easy to spot, the output is unparseable, but the cause is often subtle. Streaming and JSON mode can conflict on some providers. Complex nested schemas with optional unions also increase failure rates.
Diagnosis. Disable streaming temporarily and test with response_format={"type": "json_object"}. Simplify the schema to flat required fields only. If the model still emits markdown fences, add a system instruction that forbids them.
Mitigation. Request smaller objects and assemble them client side. If you need a nested structure, ask for an array of flat records first, then remap.
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant. Output valid JSON only, with no markdown fences."},
{"role": "user", "content": "Extract the name and email from: Contact us at support@oxlo.ai"}
],
response_format={"type": "json_object"}
)If one model struggles with a complex schema, Oxlo.ai lets you switch to another model, such as Llama 3.3 70B or Kimi K2.5, without rewriting client code. The API remains fully OpenAI compatible across all 45+ models.
When to Swap Models
Sometimes the issue is not the prompt or the code, but the model itself. A model may plateau on a specific reasoning pattern, or its latency and cost structure may no longer fit the task.
Decision framework. Classify the task by complexity and latency needs. Simple classification or extraction does not require a 400B parameter model. Long-horizon agentic tasks, however, need robust chain-of-thought and tool adherence.
Oxlo.ai hosts 45+ models across seven categories, from embeddings and vision to code and audio. Because the platform uses request-based pricing, swapping from a lightweight model to a heavy reasoning model like GLM 5 or DeepSeek R1 does not require rearchitecting your cost model around token counts. You pay per request regardless of prompt length, which makes A/B testing models in production financially predictable. For current plan details, visit https://oxlo.ai/pricing.
Top comments (0)