I've been building agentic AI systems for a while now, and the thing that finally got under my skin enough to write about is that our whole stack is really good at telling us what an agent did, and almost useless at telling us whether it was right.
Observability tools give you the trace, every tool call and every token, which is great for figuring out what happened after something breaks. Evals give you a score against a test set you ran at some point in the past. But in production, in the moment, when your agent returns a confident, well-formed, schema-valid 200, nothing in that pipeline is checking whether the answer inside it is actually correct. A 200 can wrap a confidently wrong answer and your dashboard will still light up green.
I ran a little experiment to see how bad this actually is. I took a cheap, weak model and pointed it at a real structured task from my own system, sixteen inputs where I could actually check the answers, and it got eleven right while looking right a good deal more often than that. The five it missed were confidently wrong, well-formed answers that named a tool which didn't exist or arguments that didn't fit the schema, exactly the kind of thing a green pipeline waves straight through. Then I wrapped each output in a grounded check that asked whether it genuinely satisfied the constraints instead of just looking like it did. The check caught all five. Re-asking the same weak model to fix them barely helped, it couldn't. Handing just those five to a stronger model finished the batch at sixteen out of sixteen. The part I keep chewing on is that the weak model never got any smarter. The check knew which answers not to trust, and that turned out to be the whole game.
So the thing I keep coming back to is that consistency isn't correctness. A schema-valid, fluent, nicely-logged answer can still be flat wrong, and almost nothing in the modern agent stack is built to notice that while it's happening.
I've been poking at what a runtime certification layer would look like, something that lives between it logged a 200 and it passed our offline evals, and answers the one question nobody seems to be asking, which is whether this specific output, right now, is actually right.
If you're running agents in production I'm genuinely curious how you're handling this, or whether you've mostly just made peace with the green dashboard.
Top comments (0)