Every AI agent demo works. You wire up a model, point it at a handful of documents, and it answers questions in a way that makes the room nod. Then it goes near real work: real enquiries, real payments, real customers, and it starts failing in ways the demo never showed. I build AI agents for businesses through neu Intelligence, and the gap between demo and dependable is where almost all of the actual engineering lives. This is the checklist I wish more teams ran before they call a prototype a product.
1. The data layer, not the model, decides the answer quality
Demos get a curated folder of perfect PDFs. Production gets the CRM with duplicate contacts, the spreadsheet two people edit at once, and the WhatsApp thread where the real booking details live. Before touching prompts, map every system the agent must read from and write to, and decide which one is authoritative for each fact. If two systems disagree and the agent has no rule to pick, you have built a confident rumor machine.
2. Give it a job description, not a feature toggle
An agent should have a written contract: the tasks it owns end to end, the systems it may touch, the actions it may take alone, and the ones that always wait for a person. "Handle parent enquiries" is a feature. "Draft replies to enquiry messages; never quote prices outside this list; hand anything about refunds to a human" is an operable role. Most agent failures described as hallucinations are really scope failures: nobody wrote down what was allowed.
3. Ship in shadow mode first
The safest first release is one where the agent does everything except send. It drafts, a human approves, and you log both. Two or three weeks of that gives you a measured error rate on real inputs, which is the only honest basis for widening permissions. Least privilege stops being a guess and becomes a dial you turn against evidence.
4. The escalation path is the product
Sooner or later the agent meets something it has never seen. What happens then is not an edge case, it is the core design decision: does it freeze, guess, or hand off cleanly with the full context attached? Design the hand-back first, then the autonomy. An agent with a good escalation path is production ready long before a fully autonomous one.
5. Watch the cost curve, not the demo bill
One pilot query costs nothing. The interesting number is cost per completed task as volume grows: retries, long contexts, and chatty tool loops scale badly if nobody designed for them. Measure per task from week one, and treat cost reduction as a standing feature rather than an emergency project in month four.
6. Ownership decides whether you can fix any of this
When the agent misbehaves at 9pm on a Sunday, can someone read its logs, edit its prompts, and change its tools without a vendor's permission? Code, prompts, schemas and integration details should live in systems you control, with a runbook that ships with the build. Ask any agency what you keep when the contract ends; the answer tells you what you are actually buying.
None of this is exotic. It is the same discipline operations engineers have applied to services for decades: clear boundaries, observability, gradual rollout, an owner. The models are new; the reliability playbook is not. If you are evaluating an agent build: I build these at https://neu.ie (disclosure), and the assess, prototype, build, run sequence we use is written up at https://neu.ie/process/. The short version: diagnose the real operation before quoting anything, and let week one produce a working session on your actual workflow, not a requirements document.
Top comments (1)
The shadow mode point deserves more emphasis — it's the cheapest way to buy real-world eval data without production consequences. One thing I'd add to point 2: make the job description machine-readable too, not just a document for humans. When the agent can read its own contract and check "is this within my tasks, systems, and outcomes?" before acting, the contract becomes a runtime guardrail rather than documentation. The escalation path point is the one we see fail most: teams design for the happy path, and the agent's first truly novel situation becomes an incident.