Score AI agent vendors on how they handle failure, not on their demo. A polished demo tells you the agent can handle the five inputs the sales team rehearsed. It tells you nothing about what happens on input six, when a customer types something unexpected, uploads a bad file, or asks the agent to do something it was never scoped for. The vendors worth hiring can answer that question in detail. The ones who can't will hand you a maintenance bill later.
Here's the checklist we'd use if we were on the buying side.
Architecture and design
1. Why did you choose an agent instead of a simpler workflow?
Agentic loops with multiple LLM calls and tool-calling chains are slower, more expensive, and harder to debug than a single well-prompted call. We learned this the direct way building our own outreach tool: our first pass chained four steps (plan, extract, draft, revise), and it cost more per email than the single-call version we shipped instead, for worse copy. A vendor should be able to give you that same kind of before/after reasoning for their own architecture, not default to more agents because "agent" sells better than "script." Read up on the tradeoff in AI agents vs. workflows before the first call, so you can push back if the answer is vague.
2. Can you show me the simplest version of this that would work?
In our own outreach tooling, we scrape each prospect's site with a self-hosted Firecrawl instance and a local model, then draft the email in a single call that extracts the relevant facts and writes the pitch at the same time. We tried a multi-step chain first, plan then extract then draft then revise, and it cost more per email and produced worse copy, because each hop introduced its own chance to drift off the source material. If a vendor's default answer to every problem is "more agents, more steps," that's a signal they're optimizing for billable complexity, not for your outcome. Compare notes with single-call vs. agent chains.
3. What tools and data does the agent have access to, and why?
Every tool the agent can call is a potential blast radius. A support agent that can issue refunds needs tighter guardrails than one that can only draft a reply for human approval. Ask the vendor to walk through the permission model, not just the feature list.
Quality and evaluation
4. How do you measure whether the agent is actually working?
"We test it before launch" is not an answer. You want a repeatable eval set: a fixed collection of real (or realistic) inputs the vendor runs the agent against, with pass/fail criteria you both agree on. Without this, you're trusting vibes, and vibes don't hold up when the model provider ships a silent update. This should be a contract line item, not a verbal promise, per evals in AI vendor contracts.
5. What happens when the agent is confident and wrong?
Every LLM-based system produces confident, fluent, wrong answers sometimes. Ask specifically how the vendor's design catches this, whether that's a confidence threshold that routes to a human, a second model checking the first, or hard-coded guardrails on high-stakes actions. If the answer is "the model is pretty accurate," keep asking.
6. Can I see it fail on purpose?
Ask the vendor to run three inputs you choose, live, on the spot, including at least one that should be rejected or escalated rather than answered. This takes ten minutes and tells you more than a week of reading proposals.
Cost and ownership
7. What's the unit cost per interaction, and how does it scale?
Token costs, tool calls, and any third-party API fees (speech-to-text, search, embeddings) should roll up into a per-conversation or per-task number, broken out by step, not just a blended average. That breakdown is exactly what surfaced the problem in our own outreach tool: once we costed the chain step by step, the extra plan/extract/revise hops were visibly adding cost per email without adding quality, which is what pushed us back to a single call. A vendor who can't hand you that same kind of step-by-step number hasn't actually load-tested their own system.
8. Who owns the code, prompts, and evals when the engagement ends?
This should be explicit in the contract. If the vendor keeps the prompts, the eval set, or the fine-tuned weights, you're locked into paying them for every future change, no matter how the relationship is going.
9. What does month two look like, after launch?
Models get updated, edge cases surface, and prompts drift out of sync with your product. Ask what ongoing maintenance costs, who's on call for it, and what's included versus billed separately. We wrote a full breakdown in what AI agent maintenance actually costs if you want the specifics before you negotiate.
10. Fixed price or time and materials, and why for this project?
Neither is universally safer. Fixed price works when the task is well-defined and the failure modes are known. Time and materials fits better when you're still discovering what "working" means. A vendor who pushes one model regardless of the project shape is optimizing for their own risk, not yours. More on the tradeoff in fixed-price vs. time and materials.
Security and data handling
11. Where does customer data go, and who else can see it?
If the agent calls a third-party model API, ask exactly what's sent, whether it's used for training, and where it's logged. This matters even more for anything touching health records, financial data, or PII. Run through the fuller list in the AI agent security checklist.
12. What's the rollback plan if this goes wrong in production?
A kill switch, a way to route everything to a human, a versioned prompt history you can revert. If the vendor hasn't thought about this, you'll be the one improvising it during an incident.
Reading the proposal itself
Watch for the same red flags that show up in any software project: vague scope, no mention of who tests what, timelines that don't account for iteration. AI agent projects amplify these risks because the output is probabilistic, so ambiguity in the contract becomes ambiguity in production behavior.
Three proposals is usually enough to compare architecture choices and pricing philosophy without turning this into a part-time procurement job. Ask the same twelve questions of each one, and pay closer attention to the vendor who says "it depends" and explains why than the one with the slickest deck.
If you're evaluating vendors for an AI agent build and want a second opinion on a proposal, let's talk.
Originally published on the Pykero blog.
Top comments (0)