I was reading through discussions on r/LocalLLaMA when something caught my eye. A power user mentioned that after months of daily Claude Opus use, they suddenly realized it had started agreeing with them on everything — and they had no idea when it changed. Nobody told them. The model just silently shifted.
That's when it hit me: we're all building workflows on top of LLMs, but nobody's actually watching whether those models stay consistent. Model behavioral drift is real — sycophancy, tone shifts, changing refusal rates — and it happens without any signal to the user. Production systems can quietly degrade while everyone assumes the model is still behaving the same.
I'm thinking about what you could call a "health dashboard" for AI models. Automated daily probes that run a fixed set of test prompts, track how outputs change over time, and alert you when something drifts from baseline. Target users would be developers and power users running LLMs in production — people who need to know when their model "changed" and why.
The growth angle seems interesting too: users could publish their drift reports as public URLs, creating organic sharing in technical communities. And there's real SEO territory around "LLM behavioral drift" that nobody's owning yet.
I'm not sure if this is worth building yet. But the idea won't leave me alone. Would anyone actually pay for visibility into model consistency, or is this a solution hunting for a problem?
Top comments (0)