Every team shipping an agent has the same meeting. Someone asks "is it ready?" and the room splits. One person saw a great demo. Another watched it invent a refund policy an hour ago. The argument runs in circles because nobody agreed what "ready" means, so the loudest opinion wins and the agent ships on a vibe.
Making an AI agent production-ready is not a moment of confidence. It is a bar you write down before you build, then measure against. This post is about that bar: why agents need a different one than the services you already ship, and how to define it so "is it ready?" becomes a number instead of an argument.
Why "production-ready" breaks for agents
For a normal service, "production-ready" is settled. Correct output for valid input, handles errors, meets a latency target, has tests and a rollback. You know the shape of done. An agent breaks three of those assumptions at once:
- It is non-deterministic. The same input can produce different output, so "correct" becomes "acceptably right, often enough."
- Its failure surface is open-ended. A function fails in ways you enumerated; an agent fails in ways you never imagined, because it composes language, tools, and judgment on the fly.
- Its worst case is not a 500 error. It is a confident wrong answer that looks right, which is far more expensive than a crash, because a crash at least tells you it failed.
So the honest question is not "is the agent correct." It is "is the agent acceptably wrong, safely, within budget, and repeatably enough to trust." That question has four parts, and each is a line on your bar.
The four lines of the bar
Write these down before you build. If you cannot fill them in, you do not have a spec, you have a wish.
- Task success. On a fixed set of real tasks, not the happy-path demo, what fraction must the agent complete correctly? Pick the number. 85 percent means one in seven users gets a wrong answer. Acceptable for this job, or fireable? Decide on purpose.
- Failure acceptability. Not all wrong is equal. An agent that says "I'm not sure, here's a human" failed gracefully. One that invents a policy and states it with confidence failed dangerously. Your bar names which failures are allowed and which are disqualifying.
- Cost and latency ceiling. An agent that is right but takes forty seconds and three dollars a call is not ready for a chat box. The budget is part of correctness, not a footnote.
- Containment. When it is wrong, and it will be, what stops the damage? Can it move money, delete data, or email a customer with no human in the loop? The bar defines the blast radius you accept.
Not one of these is "the demo worked." A demo is a single sample with a friendly operator. The bar is what you hold the agent to on inputs you did not hand-pick.
Define it first, on purpose
A bar written afterward is rigged. Once you have a working agent, every threshold quietly slides to whatever it already scores. Eighty-two percent becomes "eighty is fine." The scary failure mode becomes "edge case." You will rationalize the thing you built into the thing you meant to build. Set the numbers while you still have the nerve to fail them.
None of this tells you how to measure the bar yet. That is Part 2, where the eval set stops being an afterthought and becomes the product you are actually building. For now the win is smaller and larger at once: next time someone asks "is it ready?", you do not argue. You point at the bar.
A demo is an anecdote. A bar is a contract. You cannot ship what you refuse to define.
Top comments (0)