I spent last week debugging why our AI-powered customer support bot was giving increasingly strange answers. The model hadn't changed. The prompts were identical. The infrastructure was stable.
So what was different?
I checked the logs. I checked the embeddings. I even re-ran the evaluation suite — everything passed. But real users were complaining about hallucinated product recommendations.
The breakthrough came when I stopped looking at the model and started looking at something nobody measures: context drift.
The metric nobody tracks
Every AI observability tool I've used focuses on the same three things:
- Latency (p50, p95, p99)
- Token usage and cost
- Error rates and timeouts
These are all infrastructure metrics. They tell you whether the system is working, not whether it's producing good outputs.
Our bot was responding in 800ms, using 200 tokens, with zero errors. By every metric that mattered, it was performing perfectly.
Yet it was slowly becoming useless.
What I found
I started tracking something simple: the semantic similarity between current prompts and the training corpus. Over three weeks, the average similarity dropped from 0.87 to 0.62.
Users were asking about products we'd launched, features we'd added, and edge cases we'd never anticipated. The model was doing its best — but its best was calibrated for a different world.
The observability stack saw zero anomalies because nothing broke. The system was functioning exactly as designed. It was just designing for a moving target.
The pattern I built
I created a simple monitoring layer that tracks three new signals:
Output variance over time. Not variance within a single request, but variance across days. If your model's output distribution shifts significantly (measured by embedding distance), that's your early warning.
Prompt-embedding drift. Every incoming prompt gets embedded and compared against a rolling window of historical prompts. When the average distance crosses a threshold, you know the user base is evolving.
Feedback signal lag. Most systems collect user feedback (thumbs up/down, corrections). But that feedback arrives hours or days after the problematic output. I built a pipeline that correlates feedback signals with the prompt-drift metric — and it turned out drift preceded bad feedback by an average of 4 days.
What this means for your stack
If you're building AI applications in 2026, here's what I'd add to your observability:
- Embedding-based output monitoring: Sample 1% of outputs daily, embed them, and track distribution shifts. A simple PCA projection over time reveals when your model's "personality" is drifting.
- Prompt similarity windows: Maintain a rolling buffer of the last 10,000 prompts. Compare new prompts against this buffer. Alert when similarity drops below a threshold.
- Correlation dashboards: Plot drift metrics alongside business metrics (conversion, retention, CSAT). You'll often find that model quality degradation shows up in business numbers before it shows up in error rates.
- Automated re-calibration triggers: When drift exceeds a threshold, automatically trigger a re-evaluation of your prompts and, if necessary, a fine-tuning cycle.
The uncomfortable truth
Most AI observability tools solve the wrong problem. They help you detect when your model crashes, not when your model becomes incrementally worse.
Incremental degradation is harder to detect because it doesn't trigger alerts. It doesn't show up as an error. It's a slow bleed that only reveals itself when you're measuring the right things.
I'm still iterating on this approach. Next up: building an automated system that detects drift patterns and suggests which prompts need updating.
What metrics do you track that others don't? I'd love to hear what you've discovered.
Top comments (0)