When people ask what is hardest about running a network of AI agents, they expect me to say accuracy. It isn't. A wrong answer that looks wrong is cheap. Someone reads it, frowns, and moves on. The expensive failure is the wrong answer that looks completely right. The agent is calm, the formatting is clean, the reasoning reads like it knows exactly what it's doing, and it is wrong in a way nobody catches until the result is already in production or already paid for.
I have watched this play out enough times to stop trusting confidence as a signal. An agent that says "done" with no hedging is not more reliable than one that flags a doubt. Often it is less reliable, because the ones that hedge have at least noticed the edge of their own knowledge. The calm ones sailed right past it.
The first instinct most people have is to fix this with a better model. Throw a smarter agent at it and the confident-wrong answers go away. They don't. A stronger model is wrong less often, but when it is wrong it is wrong with even more poise, which makes the bad answer harder to catch instead of easier. You have not removed the failure, you have dressed it better.
The thing that actually helped was changing what the agent has to produce. Not just the answer, but the evidence for the answer. If the task is "clean this data," the agent also has to show the rows it changed and why. If the task is "draft this reply," it shows the source it pulled each claim from. The output stops being a verdict you have to trust and becomes a diff you can scan in ten seconds. A confident-wrong answer survives a verdict. It rarely survives having to show its work.
The other thing I had to learn was to stop treating agreement as proof. When several agents weigh in and they all land in the same place, the easy read is "great, high confidence." But a panel of similar agents agreeing usually just means they share the same blind spot. The agreement is correlation, not signal. The disagreement is where the real information lives, because a model only changes its answer after reading another one when something actually made it reconsider. Silent unanimous agreement is the case I now flag for a human, not the case I wave through.
The part that took me longest to accept is that you cannot test your way out of this entirely. There is no ground truth sitting in the room to end the argument. You can make the wrong answer more expensive to produce and easier to catch, but you cannot make it impossible. So the honest design goal is not "never wrong." It is "wrong in a way a person notices before it costs anything."
That is also why I tie quality to the moment of payment rather than the moment of output. An agent can claim it finished. What it cannot fake is a result good enough that the person looking at it decides to pay for it. The check that matters is not the agent grading itself. It is a human, holding the actual output, deciding it was worth it. Confidence is free. That decision is not.
None of this is solved. I still get confident-wrong answers every week. But I have stopped trying to make the agents sure of themselves, and started making them show me why I should be.
Top comments (0)