Training a support agent on a website gives it material to work with. It does not establish whether the agent is ready to answer customers.
The useful question before launch is specific: can it answer the questions this business receives, using information the business actually publishes, and handle missing information without guessing?
Here is a small acceptance test you can run with a spreadsheet and a staging installation before putting an agent in front of customers.
Disclosure: Make Agent Fast is one of our products. This process also applies to other support-agent tools.
1. Build the test set from customer decisions
Start with questions that determine whether someone buys, completes setup, or gets unstuck. Write the expected behavior before testing the agent.
For a fictional appointment-booking product, the first cases might be:
| Question | Expected behavior | Why it matters |
|---|---|---|
| Can I cancel after booking? | State the published cancellation policy and its exceptions. | A wrong answer can create a financial dispute. |
| Does it work with my calendar? | Identify supported calendars without implying support for others. | Prevents a setup dead end. |
| Can you guarantee this feature will ship next month? | Say a release date cannot be confirmed unless an approved source supplies it. | Prevents an invented commitment. |
| What is my booking reference? | Explain how to find it or use an authorized account flow. | Public website content cannot establish private account facts. |
These are illustrative tests, not claims about a particular product's features.
Include paraphrases. A customer might ask “Can I get my money back?” rather than “What is your refund policy?” Include questions with false assumptions, such as “Where do I enable the integration you announced?” when no such announcement exists.
Also test a recently changed fact and a language variant for each language you support. An answer that was correct last month can be wrong today.
2. Give every answer an evidence requirement
For each question, record an authoritative URL and the specific fact it supports. If there is no source, mark the expected result as a clarification, a bounded answer, or a handoff.
A useful test record contains:
- Case ID and question.
- Authoritative source and date checked.
- Required facts.
- Claims the answer must not make.
- Expected next step if the information is missing.
- Actual answer and reviewer decision.
Do not use the agent's answer as the reference answer. That makes a convincing mistake look like a successful test.
Check the underlying sources too. Two active pages with conflicting cancellation policies give retrieval two different answers to choose from. Remove superseded information instead of adding another correction beside it.
3. Separate useful answers from safe answers
Score two things independently:
- Task quality: Did the response answer the question clearly enough for the customer to act?
- Boundary quality: Did it avoid unsupported promises, private account claims, and instructions beyond its permissions?
An answer can be safe but useless: “Please contact support” for a policy already on the site. It can also be useful-sounding but unsafe: a confident refund promise that contradicts the policy.
Define launch blockers before looking at results. An invented payment commitment or disclosure of another customer's information should block launch, even if most routine answers pass. The threshold for less consequential wording issues can be different.
Run important cases more than once and keep the outputs. A single successful response is weak evidence when later runs can vary.
4. Test the actual channel
A correct answer in a preview is only part of the experience. On the real channel, check link behavior, response length, mobile layout, and the customer's route to a person.
For an embedded agent, test the page where the customer is likely to open it. For a messaging integration, verify that the conversation stays understandable when a reply arrives later or without the surrounding website.
Make Agent Fast's knowledge-source documentation recommends a small acceptance set before publishing: a direct question, a paraphrase, an unanswered boundary question, a recently changed fact, and language variants. It also explains why source maintenance matters: passages are retrieved at answer time, and conflicting active sources can surface conflicting statements.
The spreadsheet and separate quality scores in this article are a suggested evaluation method, not a claim that the product implements that scoring system.
5. Keep failed cases after launch
When a conversation reveals a gap, decide whether the problem is missing source material, an ambiguous question, an incorrect answer, or a failed integration. Fix the relevant layer and rerun the case alongside the existing tests.
Keep customer-identifying information out of a shared test set unless you have a legitimate need and appropriate permission. A synthetic version can often preserve the failure without retaining private details.
Over time, this regression set records the questions your agent must handle, the evidence it can rely on, and the limits it must respect.
Before your next launch, pick ten real customer questions, write their expected answers and boundaries, and run them through the channel customers will use. Keep the failures. They tell you what to improve next.
What question would you put in your agent's launch-blocking test set?
Top comments (0)