When a vendor shows you an AI feature in action, you're watching a performance. The inputs are hand-picked, the outputs are polished, and nothing in that demo resembles the queue of messy, ambiguous, incomplete cases your team handles every day.
The problem isn't that vendors are dishonest. The problem is that demos are structurally unsuited to tell you whether a system will perform on your data. You need a repeatable, documented process that produces an answer you can put in a contract.
Build Your Own Evaluation Set
Before the vendor sees a line of your data, your team should assemble a set of real historical cases — inputs you already have answers for. Draw from normal operations, but deliberately include the hard ones: requests with missing fields, poor-quality scans, ambiguous instructions, edge cases your best people have argued about.
When two experienced team members genuinely disagree on the correct answer, leave that case out. That disagreement is a process problem, not a model problem. Including it just muddies your measurement.
Label every case with the answer a competent person would give. Once you have a working set, hold back roughly a third of them. Share the rest with the vendor for development. The holdout set is what you actually test against at sign-off — if a vendor has only seen a subset of your data, you get a cleaner read on whether the system generalizes.
Aim for enough cases to get a stable measurement. Very small sets produce noisy results; a larger and more varied set gives you a number you can trust and re-run.
Set the Threshold Before You See the Score
There is no universal accuracy threshold for AI features. The right number depends entirely on what happens when the system is wrong.
A tool that drafts responses for a human reviewer to approve before anything goes out has a very different risk profile than one that autonomously posts to a financial ledger or triggers a customer-facing communication. The tolerable error rate for the first might be generous; for the second, you may want something close to airtight.
Work through the error economics before you run the test. If the system produces a wrong answer, what does it cost in time, money, or customer trust to catch and correct it? That math gives you a defensible threshold. Write it down and get the vendor to agree to it in writing — before testing begins, not after you see a number you don't like.
Test the Edges, Not Just the Averages
Raw accuracy on typical cases is only part of the picture. Three categories of behavior often go untested until they cause problems:
Refusals. What does the system do when the input is genuinely outside its scope? It should decline clearly and route the case elsewhere, not produce garbage with false confidence.
Escalation. When should it hand off to a human? Does it actually do that, or does it push through with low confidence and no flag?
Hostile inputs. Someone will eventually try to manipulate the system — contradictory instructions, adversarial prompts, or attempts to extract behavior it wasn't designed for. You want to know what happens before your customers find out.
Stability is also worth checking: run identical cases on different days and look for consistency. Some variation is inherent in large language models, but a system that gives wildly different outputs on the same inputs is one you can't rely on.
Lock the Evaluation Set Into the Contract
An evaluation set that lives only in a shared spreadsheet is easy to ignore once the ink is dry. The better approach: attach it as a formal exhibit to the vendor agreement.
That contract language should require written notice before the vendor makes any changes that affect the underlying system — model updates, prompt changes, data sources, integration layers. It should specify that re-testing is required after such changes, with results delivered within a defined window.
The evaluation data should belong to you, not the vendor. That matters if you ever need to switch providers: a proprietary test set controlled by the current vendor is leverage you don't want them to have.
After go-live, treat the evaluation set as a living regression suite rather than a one-time gate. Business inputs drift; your test set should be updated periodically to reflect how your operations actually look today. Vendors update underlying models on their own schedules — automated changes can shift behavior without anyone alerting you.
This guide originally appeared on agentpalisade.com. Agent Palisade helps small and mid-sized businesses put AI to work inside the tools they already use — practical automation, internal assistants, and AI security reviews. Book a free 30-minute call.
Top comments (0)