I learned this the hard way. A pipeline processing 10,000 job listings daily, scoring each one with GPT-4 function calling, and it was a black box. When something broke, I had no idea what broke, why it broke, or how much it cost while it was breaking.
That's not a production system. That's a prototype running on production infrastructure.
The difference between a demo and a reliable AI agent isn't better prompts or a fancier model. It's observability. Logging, cost tracking, error monitoring, and latency metrics turned that fragile pipeline into something I could trust. Here's exactly what I did and why.
The Problem: An LLM Pipeline With No Dashboard
The system scored 10,000+ job listings daily using GPT-4 function calling. Each listing went through a multi-step pipeline: fetch from source, normalize the data, call the LLM for scoring, parse the structured output, and store the result.
For weeks it ran fine. Then one day it didn't.
A model update changed the output format silently. The function call returned valid JSON but with a different key structure. The pipeline kept running, kept returning 200s, and kept storing null scores. I didn't notice for three days. Three days of bad data, wasted API costs, and a backlog of unscored listings.
That's when I stopped treating observability as optional.
What to Log: The Minimum Viable Trace
You don't need Datadog or a full OpenTelemetry setup to start. You need three things: a trace ID per request, structured logs for every LLM call, and a way to query them.
Here's the pattern I settled on after trying a few approaches:
interface LLMTrace {
traceId: string;
model: string;
promptTokens: number;
completionTokens: number;
latencyMs: number;
success: boolean;
error?: string;
inputPreview: string; // first 200 chars
outputPreview: string; // first 200 chars
timestamp: Date;
}
Every LLM call gets one of these. The trace ID ties it back to the original job listing. The token counts let me track cost per job. The success flag catches silent failures where the API returned 200 but the output was garbage.
I log this to a dedicated llm_traces collection. It's separate from application logs because LLM calls have their own failure modes and cost implications. Mixing them with general app logs makes both harder to query.
What I Learned From the First Week of Logging
The first week of structured logging told me things I didn't want to hear.
First, latency was all over the map. Some calls completed in 400ms. Others took 12 seconds for the same model and prompt structure. No pattern I could see without the trace data. Once I had it, I found the culprit: prompt length variance. Listings with long descriptions pushed token counts past a threshold where the model's response time doubled. I added a prompt truncation step and latency flattened.
Second, cost. I had no idea what a single scoring call cost until I logged token counts per trace. The average was $0.003 per listing. At 10,000 daily listings, that's $30 a day. Not a problem. But the tail was ugly. Some listings with huge descriptions cost 10x the average. I added a max-token cap and saved about 18% on monthly API spend.
Third, silent failures. The most dangerous kind. The LLM returned valid JSON with a score of 0 for every listing in a batch. No error. No exception. Just useless data. The success flag in my trace caught it because the output didn't match the expected schema. Without that check, I'd have shipped bad scores for days.
How to Structure Agent Traces
A single LLM call is easy to trace. An agent that makes multiple calls, retries, and conditional branches is harder. You need a span model.
Each trace becomes a parent with child spans. The parent is the overall job. Each child span is one LLM call, one tool invocation, or one decision point. They share the trace ID and carry their own timing and status.
interface AgentSpan {
traceId: string;
spanId: string;
parentSpanId?: string;
spanType: 'llm_call' | 'tool_use' | 'decision' | 'retry';
model?: string;
inputTokens: number;
outputTokens: number;
durationMs: number;
status: 'success' | 'error' | 'timeout';
errorMessage?: string;
timestamp: Date;
}
This structure lets me answer questions like: which step in the pipeline is slowest? Which model calls fail most often? Are retries actually succeeding or just burning money?
I found that retries on timeout errors had a 40% success rate. Retries on rate limit errors had a 90% success rate. That changed how I handled each case. Timeout retries got a longer cooldown. Rate limit retries just needed a short backoff.
Catching Failures Before They Compound
The most expensive bug I caught with observability wasn't a crash. It was a silent schema drift.
The LLM had been returning scores as integers between 1 and 100. Then an API update changed the output format. Scores came back as strings like "85.0" instead of 85. The pipeline accepted them, stored them, and the frontend displayed them. Everything looked fine until someone noticed the sorting was wrong. String comparison sorts "9" after "85". Listings with single-digit scores were ranking above high-scoring ones.
A simple type check in the trace handler caught it:
function validateScore(output: unknown): number | null {
if (typeof output === 'number' && output >= 1 && output <= 100) {
return output;
}
if (typeof output === 'string') {
const parsed = parseFloat(output);
if (!isNaN(parsed) && parsed >= 1 && parsed <= 100) {
return parsed;
}
}
return null; // triggers alert
}
When the score is null, the trace logs a warning and the pipeline pauses that batch. No bad data propagates. No silent corruption.
The Metrics That Matter for AI Agents
Not all metrics are equal. Here's what I track and why.
Token cost per job. This is your unit economics. If you don't know what one AI operation costs, you can't price your product or catch regressions. A prompt change that adds 200 tokens might not feel like much until you multiply it by 10,000 daily jobs. That's 2 million extra tokens a day. At GPT-4o pricing, that's about $6 a day. Over a month, $180. Over a year, over $2,000. For one prompt change.
Error rate by error type. Timeout errors and content filter errors need different responses. Timeouts need retry with backoff. Content filter errors need prompt adjustment. If you lump them together, you can't tune either.
Latency p50, p95, p99. The average hides the problem. A p50 of 800ms looks fine until you see the p99 is 14 seconds. That tail is what kills user experience. I set an alert when p95 exceeds 3 seconds. It's fired twice. Both times the cause was a prompt that had drifted to include too much context.
Cost per model. If you're routing between models (GPT-4o for complex scoring, GPT-4o-mini for simple classification), you need to know the actual cost split. I found that 30% of my calls were hitting the expensive model when the cheap one would have worked. A routing rule fixed it.
The Trust Loop
Observability does something more important than catching bugs. It builds trust.
When a founder asks "is the AI working right now?" you need to answer with data, not vibes. A dashboard showing the last 100 traces, their status, latency, and cost is worth more than any testing suite. It proves the system is doing what you think it's doing.
I built a simple status page that shows the last hour of pipeline activity. Green traces for success, red for failures, yellow for retries. The founder checks it once a day. When it's all green, they don't worry about the AI. That trust lets me iterate faster. I can deploy a prompt change, watch the traces for 10 minutes, and know immediately if it broke something.
Without observability, every deployment is a leap of faith. With it, every deployment is a measured risk with a rollback trigger.
The Real Cost of Skipping Observability
I see teams skip observability because it feels like overhead. Another system to maintain. Another dashboard to check. Another thing that can break.
The real overhead is debugging a black box. Three days of bad data. A $500 API bill you can't explain. A founder who's lost confidence in the AI feature.
Observability is the cheapest insurance you can buy for an AI system. A few extra fields in your database. A simple status endpoint. A weekly cost report. That's it. You don't need a PhD in distributed tracing. You need a trace ID, a success flag, and a timestamp.
If your team is shipping AI features and finding yourself unable to answer basic questions about cost, latency, or failure rates, that's the kind of thing I help with. Happy to compare notes on what's worth tracking and what's noise.
Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.
Top comments (0)