If your AI vendor can't show you a scored test set that proves their system works on cases it hasn't seen before, you're not buying a product — you're buying a demo. Ask for the evals before you sign, not after the invoice.
Founders and CTOs evaluating AI vendors have gotten good at spotting a slick demo. What most haven't learned yet is how to tell the difference between a demo that works because it was rehearsed and a system that works because it was tested. The gap between those two is exactly what evals are for, and it's the single most useful question you can ask in a vendor selection process.
What an eval actually is
An eval is a fixed set of inputs — real questions, real documents, real edge cases from your domain — paired with graded expected outputs, run automatically every time the underlying prompt, model, or retrieval logic changes. Instead of "it seemed to work when I tried it," you get a number: 87% of cases pass, up from 74% last sprint.
This isn't exotic tooling. OpenAI open-sourced a basic framework for exactly this purpose back in 2023 (openai/evals), and every serious AI shop has since built some version of it, whether custom or off-the-shelf. The concept predates LLMs entirely — it's just automated testing, applied to a system whose failure modes are probabilistic instead of deterministic.
The reason it matters more for AI than for a typical web app: a REST endpoint either returns the right JSON or it doesn't. An LLM feature can return a plausible-sounding wrong answer 15% of the time, and a demo of ten hand-picked prompts will never surface that. Evals are the only way to see the failure rate before your customers do.
Why this is a buyer problem, not just an engineering one
If you're hiring for a custom chatbot, a RAG-based support system, or an agentic workflow, the vendor's pitch deck will show you the happy path every time. That's not dishonest — it's just not information. The questions that actually de-risk the purchase are:
- What's your eval set size, and where did the cases come from?
- What's the current pass rate, and what's the trend over the last month?
- What happens to the eval score when you swap models or change the prompt?
- Do you re-run evals before every deploy, or only when something breaks?
A vendor who has this instrumented will answer in numbers. A vendor who doesn't will answer in adjectives — "it's really solid," "we've tested it a lot." That's the tell.
What to put in the contract
Treat evals the way you'd treat test coverage in a normal software SOW — as a deliverable, not a courtesy:
- Minimum eval set size and composition. For most single-feature AI products, 50-150 cases pulled from real or realistic data is a reasonable floor. A support bot needs cases drawn from actual past tickets, not synthetic questions the vendor invented.
- A pass threshold, agreed before build starts. 80% might be fine for an internal tool; a customer-facing feature or anything touching healthcare or compliance-sensitive data needs a higher bar and human review of the failures, not just the score.
- A re-run requirement. Every model version bump, prompt edit, or retrieval change should trigger the same eval set. Silent regressions are the most common way "it worked in the demo" becomes "it broke in production three weeks after launch."
- Ownership of the eval set at handoff. You should leave the engagement owning the test cases and the scoring logic, not just the shipped feature. If the vendor won't hand that over, that's worth noting during scoping.
None of this needs to bloat the contract. A short exhibit — eval set size, pass threshold, re-run trigger, and who owns the harness at the end — covers it.
What it costs, and why it's worth it
Building an initial eval suite typically adds something like 10-15% to the cost of the AI feature itself: time spent collecting representative cases, writing the grading rubric (or hooking up an LLM-as-judge for open-ended outputs), and wiring the harness into CI so it runs automatically. That's a real line item, and a vendor quoting a suspiciously fast, suspiciously cheap AI build is often quietly skipping it.
It's cheap compared to the alternative. The true cost of an AI MVP isn't just the build — it's the cost of finding out post-launch that the system fails on 1 in 5 real customer questions, then re-architecting under pressure while customers are already seeing bad answers. Evals move that discovery to before launch, when it's a code review conversation instead of an incident.
The red flag version of this, in one sentence
If a vendor's answer to "show me your evals" is a demo instead of a dashboard, they don't have a tested system — they have a system they haven't tested yet, and you'd be the one who finds out where it breaks. That's the same instinct behind most other red flags in a software engagement: ask for evidence, not confidence.
Evaluating AI vendors isn't fundamentally different from evaluating any other software partner — you're looking for proof of rigor, not just proof of output. Evals are simply where that rigor shows up for AI work specifically, and they're cheap enough to ask for that there's no good reason a serious vendor should push back.
If you're scoping an AI feature and want a partner who builds the eval suite alongside the feature, not after something breaks, let's talk.
Originally published on the Pykero blog.
Top comments (2)
One of the biggest shifts in AI development is treating prompts and models like code—with testing, regression checks, and measurable quality. A polished demo is easy to build; a system that consistently performs across hundreds of real-world cases is much harder. Evals should be a standard deliverable in every serious AI project, not an afterthought.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.