Three silent failure modes that standard APM misses, and the instrumentation layer that catches them before your users do.
Your AI feature passes load testing. Latency is under two seconds, error rate is under one percent, the product demo runs clean. You ship it. Three weeks later, a customer reports that the AI-powered output "stopped making sense." Your dashboard shows nothing wrong. The endpoint is returning 200. The logs are quiet.
This is the observability gap in production AI backends. It is different from anything traditional monitoring tools were built to catch.
Why Standard APM Fails for LLM-Backed Services
HTTP response codes, p95 latency, and error rates describe the transport layer. For a REST endpoint serving cached data or querying a relational database, that is enough. For an LLM-backed feature, a 200 response tells you almost nothing about whether the system is working correctly.
A 200 from OpenAI or Anthropic can mean: the model returned a valid response. It can also mean: the model silently truncated your input because it exceeded the context window and returned a coherent-but-incomplete answer. Or: the response is syntactically correct but output quality has shifted because someone updated the system prompt two weeks ago without an evaluation run.
The failure modes that hit AI-native backends in production do not show up as error spikes on a Grafana dashboard. They show up as a slow erosion of output quality that accumulates over days until a user complains.
The Three Failure Modes
Silent Context Truncation
Most LLM clients truncate inputs silently when they hit the model's context limit, or throw an exception that gets swallowed by a generic catch block. Your feature continues returning 200s, but the model is working from a partial view of the data. Depending on what gets cut, the output degrades in ways subtle enough to pass automated validation but wrong enough that attentive users notice. The truncation is systematic, predictable, and completely invisible to standard monitoring.
Rate Limit Cascades
Token-per-minute limits are enforced per API key. At low traffic you will never hit them. At 200 concurrent users, a single high-traffic hour can push you into 429s that your retry logic converts into a thread-blocking cascade. If you are not tracking cumulative token consumption per minute per model, you have no signal before you hit the wall. By the time 429s appear in your error logs, you are already degraded.
Prompt Regression
Output quality is a function of three things: your prompt, your model version, and your input data distribution. All three change over time. Model providers update default behavior in minor versions. Engineers edit prompts without evaluation runs. Real user traffic exposes edge cases that your staging environment never did. Without a structured evaluation pipeline, prompt regressions surface as customer complaints rather than CI failures.
What We've Seen
At one client running an AI-powered document processing feature, the backend was handling several hundred LLM calls per day across two providers. Standard monitoring showed nothing unusual. A code review surfaced a context overflow condition that had been silently truncating inputs for eleven days. No user had seen enough degraded outputs to raise a ticket, but the truncation was systematic for documents over a certain length.
The fix took two hours: a tokenizer check before each API call, plus a hard rejection of requests that would overflow the context window. What was missing before the fix was a dashboard showing the distribution of input token counts per endpoint. That would have flagged the problem on day one.
A second pattern: a provider-side model version bump. One team was pinned to gpt-4-turbo by alias, not by exact identifier. The provider's new default version had different formatting behavior for a structured JSON output schema the feature depended on. Schema validation failures started appearing in the retry logs. It took four hours to diagnose because there was no span attribute recording which exact model version had handled each call.
The Instrumentation Stack
Standard APM covers infrastructure observability. AI-native backends need a second layer on top of it. Four components cover the surface area.
1. Structured OpenTelemetry Spans on Every LLM Call
Every call to an LLM provider should emit an OpenTelemetry span recording: exact model identifier, input token count, output token count, latency, and provider HTTP status. This is the raw data for everything else.
// NestJS — LLM call wrapped in an OpenTelemetry span
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('llm-service');
async function callLLM(prompt: string, model: string): Promise<string> {
return tracer.startActiveSpan('llm.completion', async (span) => {
span.setAttributes({
'llm.model': model,
'llm.prompt_tokens': estimateTokens(prompt),
});
try {
const res = await openai.chat.completions.create({
model,
messages: [{ role: 'user', content: prompt }],
});
span.setAttributes({
'llm.completion_tokens': res.usage?.completion_tokens ?? 0,
'llm.total_tokens': res.usage?.total_tokens ?? 0,
});
span.setStatus({ code: SpanStatusCode.OK });
return res.choices[0].message.content ?? '';
} catch (err) {
span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
throw err;
} finally {
span.end();
}
});
}
Token count and exact model version per call flow into Prometheus via an OTEL collector. From there, you have the foundation for rate limit alerting and model version attribution.
2. Token Budget Alerts
Set a Prometheus alert at 70% of your per-key TPM limit. That gives you a two-minute window to queue requests, shed load, or page on-call before hitting 429s.
# Prometheus alert rule
- alert: LLMTokenBudgetHigh
expr: |
sum(rate(llm_tokens_total[1m])) by (api_key, model)
> 0.70 * on(api_key) llm_tpm_limit
for: 2m
annotations:
summary: "Token budget at {{ $value | humanizePercentage }} for {{ $labels.model }}"
The 70% threshold is not arbitrary. At 70%, you have runway to respond before degradation starts. At 90%, you are already mid-cascade.
3. Exact Model Version Pinning
Pin to exact model identifiers, not aliases. gpt-4o-2024-08-06 not gpt-4o. When you upgrade, do it in a PR with a structured evaluation run in CI. Treat model upgrades like dependency upgrades: deliberate, reviewed, gated on test results.
This is the lowest-effort, highest-leverage change most AI-native teams can make this week. Add one line to your model config, and the four-hour diagnosis of "why did output quality shift" becomes a one-minute log query.
4. Evaluation Pipeline in CI
For any feature where output quality matters — which is most LLM features — run structured evaluations against a held-out test set as part of your deployment pipeline. Langfuse integrates directly into most AI backends and can gate deployments on score thresholds.
// Langfuse trace for evaluation tracking (TypeScript)
const langfuse = new Langfuse({ publicKey, secretKey, baseUrl });
const trace = langfuse.trace({ name: 'document-summary' });
const generation = trace.generation({
name: 'summarize',
model: 'gpt-4o-2024-08-06',
input: prompt,
});
generation.end({ output: result });
Your evaluation script queries Langfuse for the latest generation traces against your test set and fails the deployment if quality metrics drop more than 5% from baseline. Prompt regressions become CI failures, not support tickets.
The evaluation test set does not need to be large. Twenty to thirty representative inputs with labeled expected outputs catches the vast majority of regressions in our experience. The effort to build it is one day. The effort to maintain it is twenty minutes per sprint, when new edge cases surface from production traffic and get added to the set.
The Engineering Cost
Getting this stack in place takes one backend engineer two to three days, assuming they have shipped LLM features in production before. OpenTelemetry setup is the longest part. Prometheus alerts and Langfuse integration each take a few hours once the spans are flowing.
The alternative is discovering these failure modes after they have been running silently for a week. That is a four-hour diagnosis, a customer apology, and a post-mortem. Every team we have worked with that shipped this observability stack before their first production incident has been glad they did. Every team that shipped it after their first production incident has said the same thing.
Key Takeaways
- Standard APM covers transport-layer health. The three AI-specific failure modes — silent context truncation, rate limit cascades, and prompt regression — each require dedicated instrumentation.
- OpenTelemetry spans on every LLM call are the foundation. Token count, exact model version, and latency per call give you the raw data for everything else.
- Pin to exact model identifiers, not aliases. One config change. Eliminates a class of four-hour debugging sessions.
- Evaluation pipelines should be in CI before your first production incident, not after. The cost is two to three engineer-days. The cost of not having it is a week of silent degradation followed by a customer complaint.
SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog
Top comments (0)