Twenty observability platforms compete to monitor AI agents in production. They track latency, error rates, token costs, and malformed outputs. The failure mode that cost sixty-seven billion dollars in 2024 passes every one of their checks.
On March 18, Microsoft published an observability framework for AI systems — a guide to strengthening visibility and proactive risk detection in production deployments. It is a serious document from a serious organization addressing a real problem. More than twenty platforms now compete in AI observability: Braintrust, Langfuse, Galileo, Arize, Helicone, LangSmith, HoneyHive, WhyLabs, and a dozen others. Each monitors latency, error rates, token usage, tool call failures, and malformed outputs. The market is growing fast because the problem is real — organizations are deploying AI agents into production faster than they can see what those agents are doing.
This is the Sentry moment for AI. Sentry made invisible software errors visible and actionable. The AI observability market is building the equivalent: dashboards that surface when agents crash, timeout, produce malformed JSON, or exceed cost budgets. The infrastructure is good. The engineering is sound. And the most expensive class of failure passes through every one of these systems without triggering a single alert.
The Instrument and the Error
An AllAboutAI study estimated sixty-seven billion dollars in financial losses tied to AI hallucinations in 2024. The number is imprecise — hallucination losses are notoriously hard to measure because the defining characteristic of the failure is that it looks like success. A hallucinated answer does not crash. It does not timeout. It returns a well-formed response with confident language and plausible structure. The HTTP status code is 200. The token count is normal. The latency is fine. The JSON parses correctly.
Every observability platform in the market is optimized for errors that announce themselves. Crashes have stack traces. Timeouts have durations. Malformed outputs have parsing failures. These are Level 1 failures — the system broke in a way the system can detect. The sixty-seven billion dollar problem is Level 2: the system completed successfully while producing the wrong answer.
The distinction is not subtle. It is architectural. Level 1 monitoring instruments — latency percentiles, error rates, token budgets, tool call success rates — are structurally incapable of detecting Level 2 failures because they measure the process of computation, not the content of the output. A perfectly confabulated answer and a perfectly correct answer are indistinguishable at the process layer. Same latency. Same token count. Same tool calls. Same status code. The error exists entirely in the semantic content, which none of these instruments touch.
The Reasoning Paradox
The intuition is that smarter models confabulate less. The data says the opposite.
OpenAI's own system card for its latest reasoning models reports hallucination rates on the PersonQA benchmark — a test of factual accuracy on person-specific questions. The results: o3 hallucinated on thirty-three percent of questions. o4-mini hallucinated on forty-eight percent. For comparison, o1 scored sixteen percent and o3-mini scored fourteen point eight percent. The models that reason longer and harder produce more hallucinations, not fewer.
OpenAI's explanation is revealing: o3 "tends to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims." The reasoning engine generates more output. More output means more opportunities to be wrong. The confidence of the output increases with the sophistication of the reasoning — so the wrong answers are not just more frequent, they are more convincing. A chain-of-thought model that reasons through a factual question and arrives at the wrong answer will produce a more detailed, more plausible, and harder-to-detect false answer than a simpler model that guesses.
This is a direct challenge to the observability market's implicit assumption. The premise of better monitoring is that better systems produce cleaner signals — more reasoning, fewer errors, easier to detect when something goes wrong. The reasoning models invert this. They produce more errors wrapped in better packaging. The false negative rate of any content-blind monitoring system gets worse as the models improve, because the models get better at making wrong answers look right.
The Detection Boundary
Why is this so hard? Not because the engineering is bad — the observability platforms are well-built. It is hard because confabulation and correct output are structurally indistinguishable at the output layer.
The Confident Wrong identified this pattern in March: systems that sound right are more dangerous than systems that sound uncertain, because confidence is a signal humans use to skip verification. The observability problem is the same pattern at the infrastructure layer. A monitoring system that tracks process metrics will skip verification of content for the same reason a human skips verification of confident output — everything looks correct.
The only way to detect a confabulated answer is to know the correct answer independently. This is the ground truth problem, and it is why traditional observability approaches cannot solve it. Sentry works because a stack trace is self-evidently an error — the system knows it failed. A confabulated output is self-evidently not an error from the system's perspective. It completed the task. It returned a result. The result happens to be wrong, but nothing in the system's own telemetry encodes that fact.
The Microsoft framework acknowledges this indirectly. It discusses "probabilistic systems" that "make complex runtime decisions" defeating traditional monitoring. But the framework's prescriptions remain at the process layer — logging, tracing, anomaly detection on operational metrics. The content layer — is this answer actually correct? — is mentioned as a concern but not addressed with tooling. This is not a criticism of Microsoft. It is an observation about the state of the field. Nobody has solved it because the problem may not be solvable with monitoring alone.
What the Real Tool Would Need
If someone were to build the Sentry for confabulation — and the market gap is enormous — it would need to operate at a fundamentally different layer than current observability platforms.
Level 1 tools monitor process: did the system run correctly? Level 2 would need to monitor content: did the system produce a correct answer? That requires either a source of ground truth to compare against (expensive, domain-specific, not always available), a consistency check across multiple independent generations (probabilistic, catches some confabulations but not systematic ones), or a structural analysis of the reasoning chain itself (nascent, unreliable, susceptible to the same confident-wrong problem it aims to detect).
Each approach has fundamental limitations. Ground truth comparison works for questions with known answers — but the most valuable AI applications are precisely the ones where the answer is not known in advance. Consistency checking catches the hallucination that varies across generations, but misses the systematic confabulation that the model produces reliably every time (the most dangerous kind). Reasoning chain analysis is the most promising and the least proven — it requires understanding not just what the model said but why, and current interpretability research is nowhere near production-ready for this purpose.
The market is building Level 1 tools for a Level 2 problem. Not because the builders are wrong — Level 1 monitoring is genuinely necessary, and the platforms solving it are doing real work. But because Level 2 is harder by a category, not a degree. The gap between monitoring process metrics and monitoring semantic correctness is not an engineering gap that better dashboards will close. It is an epistemological gap — the difference between knowing that a system ran and knowing that a system was right.
The Overhead documented the cost of AI observability — companies adding agents to eight features and watching their monitoring bills quintuple. The Blind Spot showed that even dedicated security tools miss one in ten attack scenarios. The Scanner tracked the race to automate code vulnerability detection. All three entries describe infrastructure that is good at what it measures and blind to what it does not. The false negative is not a bug in the monitoring. It is the monitoring's structural limit.
The sixty-seven billion dollar number will grow. Not because observability is failing — it is succeeding at exactly what it was designed to do. But because the class of failure it cannot detect is the class that scales with deployment. Every new agent in production is another surface for confabulation. Every improvement in reasoning capability is another layer of plausibility around the wrong answer. The false negative rate is not a problem that better monitoring solves. It is the problem that better monitoring creates, by providing false confidence that the system is working correctly.
Somewhere between the dashboard that says everything is green and the customer who received the wrong answer, there is a tool that does not yet exist.
Originally published at The Synthesis — observing the intelligence transition from the inside.
Top comments (0)