I spent weeks perfecting a system prompt for a job description rewrite pipeline. The prompt had persona blocks, banned-word lists, structured output schemas, the works. It was beautiful.
Then I deployed it to production and watched it rewrite "Senior Software Engineer" into "a visionary digital architect who orchestrates cross-functional synergies." The prompt was fine. The problem was I had no idea what the model was actually doing until users told me.
That's when I learned the hard truth about production AI: your prompt is a guess. Your logs are proof.
The Hallucination That Cost Me a Pipeline
The job board platform I built processes over 10,000 listings daily. Each one goes through an LLM scoring pipeline that ranks relevance, extracts skills, and normalizes job titles. Early on, I trusted the prompt to handle everything.
One day the pipeline started tagging "Entry Level" positions as "Executive" roles. The system prompt hadn't changed. The model version hadn't changed. But the output quality had silently degraded.
I spent three days tweaking prompts before I thought to check the logs. The issue was obvious once I looked: the token usage per request had dropped by 40%. The model was truncating its reasoning because a cost-saving change I'd made to the temperature setting was causing shorter completions. The prompt was fine. The configuration was broken.
Here's what I should have had from day one:
interface LLMLogEntry {
requestId: string;
model: string;
systemPrompt: string;
userPrompt: string;
completion: string;
tokensUsed: number;
latencyMs: number;
temperature: number;
timestamp: Date;
score?: number; // quality score from downstream validation
}
Every single LLM call gets logged with full context. Not just the input and output, but the configuration, the timing, and the downstream quality signal. Without this, you're debugging blind.
The Three Signals You Can't Ignore
After that incident, I built a structured logging layer around every AI call in the system. Three metrics matter more than anything else.
Latency drift. When your average response time jumps from 800ms to 2.4 seconds, something changed. It could be API congestion, a model update, or a prompt that's growing too long. I catch this with Sentry performance monitoring on the API layer. A sudden latency spike is usually the first sign of trouble, hours before users complain.
Token consumption per task. The job description rewrite pipeline was shut down because GPT-4.1 costs made it uneconomical at 1M+ listings. I knew the per-request cost, but I didn't have a dashboard showing total weekly spend trending upward. Now I log token counts per request and aggregate them by endpoint. When a single prompt starts costing 30% more per completion, I see it immediately.
Output structure failures. The most dangerous failure mode is when the model returns valid JSON with wrong data. A hallucinated skill, a fabricated company name, a salary range that doesn't match the listing. I validate every structured output against the original input data. If the model claims a job requires "10+ years of React" and the source listing says "2 years preferred," that's a log entry and a retry.
Why Prompt Engineering Alone Won't Save You
I see teams pour weeks into prompt optimization and skip observability entirely. They treat the prompt like a configuration file you can perfect once and forget. That's wrong for two reasons.
First, the model changes under you. OpenAI, Anthropic, Google, they all update their models silently. A prompt that worked perfectly in March produces gibberish in June. Without logs, you can't tell if the prompt broke or the model changed. With logs, you compare last week's completions to this week's and see the drift immediately.
Second, production data is messier than your test set. Your test prompts are clean, well-formed, and representative. Real user input is garbled, truncated, and malicious. LogRocket session replays showed me a user pasting a job description with Unicode characters that caused the LLM to return empty responses. The prompt handled it fine. The input was the problem. Without session replay, I'd still be rewriting system prompts.
Here's the logging middleware I use now:
async function logLLMCall<T>(
model: string,
prompt: string,
executor: () => Promise<T>,
context: Record<string, unknown>
): Promise<T> {
const start = Date.now();
try {
const result = await executor();
const latency = Date.now() - start;
await db.llmLogs.create({
data: {
model,
prompt,
completion: JSON.stringify(result),
latencyMs: latency,
success: true,
context,
timestamp: new Date()
}
});
return result;
} catch (error) {
const latency = Date.now() - start;
await db.llmLogs.create({
data: {
model,
prompt,
completion: null,
latencyMs: latency,
success: false,
error: error.message,
context,
timestamp: new Date()
}
});
throw error;
}
}
Every call gets logged. Every failure gets captured. Every latency spike gets recorded. This pattern costs almost nothing to implement and saves hours of debugging.
The Dashboard That Changed How I Ship AI
I built a simple dashboard that shows three things: average latency by model, total tokens consumed per day, and error rate by endpoint. That's it. Three charts.
The first time I saw it in production, I noticed the error rate for one endpoint was 12%. The endpoint was the job scoring pipeline. Twelve percent of all scoring requests were failing silently and returning default scores. Users saw accurate results for 88% of jobs and random results for the other 12%. They couldn't tell. The system didn't alert.
The root cause was a race condition in the batch processing queue. Two concurrent requests would overwrite each other's state. The logs showed the pattern immediately: paired failures with identical timestamps. Without that dashboard, I might have blamed the model and wasted weeks on prompt engineering.
I use Sentry for error tracking and LogRocket for session replay on the frontend. But the LLM-specific logging lives in the application database. Sentry catches crashes. The structured logs catch silent failures.
When Your AI Agent Needs a Doctor, Not a Coach
If your team is deploying AI features and shipping slower because you can't tell if a failure is the prompt, the model, or the data, that's the kind of thing I help with. I build production observability into AI pipelines from day one, so you catch hallucinations, cost spikes, and latency issues before they reach your users. Happy to compare notes on what's worked for your setup.
Written by Abdul Rehman, full-stack AI engineer building production SaaS, MVPs, and AI automation. More at PrimeStrides.
Top comments (0)