Let’s face it, no software is perfect at launch. If you waited until it was, you’d likely miss the boat completely. And, if we’re being realistic, most programs behave beautifully during testing. Test data is usually tidy and APIs tend to behave. The real world is a lot messier, because you can’t plan for everything customers do.
Clients might miss half the order number or switch topics three times. They might want your bot to do something it can’t do or overwhelm it. You can’t plan for everything, but you can be more prepared, and this is what this checklist is all about.
Trace What Happens To Every Request
What happens step-by-step when there’s a bad conversation? You need to make sure that your developer can follow it through the system to quickly highlight any issues. A typical request can pass through:
- Authentication
- An API gateway
- Conversation storage
- Retrieval
- A model provider
- Several internal services
If these systems don’t share requests or trace IDs, it’s hard to debug any issues. We’re not saying you should log every bit of data. That can come back to bite you by creating privacy issues when customer conversations contain sensitive information. What you need is for your developer to be able to reconstruct what happened.
They need to see what model and prompt version were used, whether retrieval ran, which tools were called, whether anything timed out and how long each stage took.
Useful fields include:
- Request and trace IDs across services.
- Model and prompt versions.
- Retrieval results or document identifiers where appropriate.
- Tool names, validation results and execution times.
- Input and output token counts.
- Retry and timeout events.
- Fallback and escalation events.
- Application and provider errors.
You’ll need to be more careful with the conversation logs. Support chats can contain names, addresses, account information and payment details. Decide what information you need to store before launch, so that you don’t have a lot of extraneous details you need to clean up later.
Build A Test Set That Looks Like Real Support
You know what you want the bot to do, and you have an idea of how to get it to do that. But you’re technically minded whereas most clients won’t be. People make mistakes and they may ask the same thing in different ways.
For example, “How do I get my money back if I bought something last month?” This is the same as asking what the refund policy is, and most humans will understand that. A bot, however, might find it confusing if it wasn’t trained on this data. The same applies when people make spelling mistakes or make repeated requests.
If you can, draw on real support requests to get an idea of what clients actually ask. And then build your evaluation set from that. Think about what the average person might ask and how they might ask it, and test that.
Have you planned for multi-turn conversations? See what happens when the customer corrects themselves, introduces a second problem or refers to something mentioned ten messages earlier.
You also need to train your model on questions that don’t have enough information to answer. Those are much more useful when it comes to troubleshooting than the easy one.
A useful test set includes:
- Common questions with several natural phrasings.
- Misspelled or incomplete requests.
- Multi-turn conversations.
- Questions with missing information.
- Questions outside the bot’s scope.
- Requests that require a tool call.
- Requests that should go to a human.
- Attempts to access information the customer shouldn’t see.
- Prompt injection attempts.
- Backend failures and timeouts.
Do keep the set somewhere you can version and rerun it. Otherwise, every prompt or model change starts another round of guessing about whether the system actually improved.
Separate Retrieval Problems From Model Problems
It’s tempting to blame the model when the chatbot gives the wrong answer. But it’s not always that simple. Maybe the retrieval supplied the wrong document and the bot was working with bad information. Changing the system prompt won’t fix that.
You should always test retrieval separately. Give the bot known questions and make sure it brings up the expected documents. Then also test the metadata filters too. You need to bring up the right data for the right area. For example, you don’t want the bot to bring up the information for the wrong country because both documents contain the key word.
You also need to make sure the system doesn’t pull up out-of-date documents. That means cleaning up your knowledge base before testing.
It’s important to distinguish between a bot retrieving the wrong document and misunderstanding the right one. Did the bot get bad information or did it make something up? Those are all very different bugs.
Don’t Forget Conversation State
Long conversations expose problems that short test chats won’t. How does your model handle:
- Context limits
- Summarization
- Truncation
Let’s say an order number disappears from the context after a few turns, what happens if your customer asks the model to check that again?
You also need the state management to handle corrections. A customer might give the wrong order number the first time. The model needs to adjust to this rather than focusing on the first number it stored.
You should always test long conversations rather than assuming the model will figure everything out. Do keep an eye on token usage too. It can work out expensive if you have to send back the entire conversation back when you’re working through a lot of them. It’ll also become time-consuming.
Treat Tool Calls As Code, Not Conversation
Your engineering risk changes when your bot can do something instead of just being able to talk. Maybe the model decides to issue a refund in error.
The application needs to:
- Validate the request
- Check permissions
- Decide whether the operation is allowed
You should always have a safeguard. The model should never be the only thing between a customer and a state-changing API. Lots of things need to be tested, including:
- Missing parameters
- Invalid parameters
- Expired authentication
- Duplicate requests
- Slow downstream services
- Retries
Say the chatbot sends a refund request and the payment service completes it, but the response times out. If the application blindly retries, the system might process the refund twice. That’s where the API should step in to prevent the duplication.
You’ll need to know:
- What happens when arguments fail validation.
- What happens when the service times out.
- What happens when the service returns an error.
- Whether retries are safe.
- Whether the operation needs idempotency protection.
- What the customer sees when the tool can’t complete the request.
Break The Dependencies On Purpose
This is the fun part. You want to make the model fail. Here are some ideas:
- Turn off retrieval
- Make an internal API return a 500
- Force a timeout
- Simulate a model provider error
- Return malformed data from a dependency
It’s not the error that’s interesting but what the customer sees afterwards. Your bot shouldn’t say the order is fine when lookup fails because it needs to give an answer. If it can’t verify something, it should say as much.
You also have to watch retry behavior. One retry might be sensible. Three different services all retrying independently can turn a small outage into something much bigger.
Write down the expected behaviour for important failure cases and test against it. Otherwise, different parts of the application can end up making different assumptions when something breaks.
Measure Where The Time Goes
“Average response time” doesn’t tell you much. You need to break the request into pieces. Ask:
- How long did retrieval take?
- How long did the database call take?
- How long did the model take?
- Was there a tool call in the middle?
You should also consider the percentiles overall. An average wait time of two seconds means little if some customers wait 15 seconds.
For streaming responses, you’ll need to measure time to first token separately from total response time. Those numbers point to different problems. A model that starts responding quickly but takes 20 seconds to finish has a different issue from one that sits there for ten seconds before showing anything.
Once you have the numbers, you can set sensible limits for the workflows that matter. It’s fine for a complicated account query to take longer than a simple FAQ. You’ll need to know what normal looks like to work this out.
Try To Make It Hallucinate
Don’t only test questions the chatbot is supposed to answer. Ask it something it won’t know about, like an outdated product name. While bots aren’t sentient, they are trained to provide answers, so they may make things up to do so.
Ask for policies that don’t exist or create two similar documents to see if it chooses the wrong one. Then look at what it says.
The obvious fabrications can seem alarming, but they’re not the real problems. The answers that are problematic are those that are mostly right but have one invented detail at the end. These are the ones that sound plausible to customers and that can cause support issues.
For anything involving account status, pricing, refunds, eligibility or another piece of information that needs to be current, you’ll need to make sure the chatbot actually consults the relevant system.
There should be a clear difference between “I don’t have enough information to confirm that” and a confident guess.
Try To Break The Security Boundaries
Prompt injection belongs in the test plan, but it’s only one part of security testing.
Ask the chatbot to reveal system instructions. Tell it to ignore previous instructions. Put instructions inside retrieved content. See whether customer-controlled text can influence tool behaviour.
Then test the application around the model.
Change identifiers. See whether one customer can access another customer’s order. Check whether knowing an email address or order number is enough to retrieve information that should require authentication.
Permission checks belong in the application layer. A prompt telling the model not to disclose private information is not an access-control system.
If the chatbot can retrieve internal documents or call privileged tools, map out exactly what it can reach. A model doesn’t need access to something just because the underlying application can access it.
Test The Guardrails Both Ways
A safety rule can be too weak. It can also be too aggressive.
Create ordinary customer requests that should pass, including some that are awkwardly worded. Add requests that genuinely need to be blocked.
This becomes especially important after changing a model or prompt. A guardrail that worked last month can start blocking harmless requests when the surrounding behaviour changes.
Keep track of why requests were blocked. If developers can only see that “the safety check failed,” it’s difficult to tell whether the rule is doing what it should.
When a rule can be handled reliably in application code, consider doing that instead of asking the model to make the decision every time. Code is easier to test than a sentence buried in a system prompt.
Watch Token Use And Cost
LLM bills can get interesting once real conversations arrive.
Track input tokens, output tokens, model selection, retrieval volume, tool calls and retries. Long conversations can become particularly expensive when the application keeps sending a large context window back to the model.
Test the expensive cases before launch, not just the average ones.
Look for loops too. A failed tool call that triggers another model response, which triggers another tool call, can rack up requests surprisingly fast.
Cost monitoring doesn’t need to be complicated. The useful part is being able to see when a particular workflow starts costing much more than expected.
Test Concurrency, Limits And Queues
Load testing shouldn’t stop at “500 requests completed successfully.”
Try bursts. Try sustained traffic. Run lots of long conversations at the same time. Deliberately hit provider rate limits.
Watch queues and connection pools as traffic increases. A chatbot can have perfectly healthy model response times while the application around it is struggling.
You also need to know what happens when capacity runs out. Does the system reject requests cleanly? Does it queue them? Does it fall back to another path?
A clean failure is better than letting a backlog spread through every service connected to the chatbot.
Make Sure The Logs Actually Help
Before release, take a deliberately broken conversation and give it to someone who didn’t build that feature.
Ask them to work out what happened.
Can they find the trace? Can they tell which model responded? Can they see the retrieved documents? Can they tell whether a tool call failed? Can they see whether the application retried the request?
If answering those questions means searching five systems manually, the observability setup needs work.
Metrics, logs and traces should point back to the same request or conversation identifiers. The information can technically exist and still be almost useless during an incident.
Keep Versions And Rollbacks Simple
The model isn’t the only thing that changes chatbot behaviour.
Prompts change. Retrieval settings change. Chunking changes. Tool definitions change. Application code changes. A model provider can also change the model sitting behind an API.
Record the versions involved in each request.
When a new release is ready, run the evaluation set against the current and proposed versions. Don’t rely on one overall score. Look for regressions in the parts of the system that matter.
If the new version handles general questions better but starts making incorrect tool calls, that’s a regression worth catching before release.
A feature flag can make this much easier. So can a switch that disables one particular tool or sends chatbot traffic back to the existing support path.
Test the switch before you need it.
Decide What Gets An Alert
Not every metric needs to wake someone up at 3 a.m.
Pick a small number of signals that point to a real production problem. That might be a sharp rise in model errors, tool failures, latency, unexpected token consumption or conversations being handed to agents much more often than usual.
Change detection can be more useful than fixed thresholds. If tool failures normally sit at 1% and suddenly hit 8%, something has probably changed, even if 8% wasn’t previously defined as an outage.
Someone needs to own those alerts. An alert without an obvious next step quickly becomes background noise.
Have A Kill Switch
This sounds obvious, but it’s worth testing.
If the chatbot starts giving bad answers after a deployment, how quickly can you disable it? Can one problematic tool be turned off without taking the whole chatbot offline? Can you send the traffic back to the existing support channel?
The answer shouldn’t involve waiting for the next full application release. And, if you work with SaaS customer support outsourcing, they should understand who to alert about the subsequent increase in queries.
A support chatbot is still a software system. It has dependencies, state, permissions, failure modes and deployment risks, with an LLM sitting in the middle of some of them. The pre-launch work needs to cover all of that.
The real test isn’t whether the chatbot gets the happy-path questions right. It’s whether the engineering team can see what happened when it gets one wrong, stop it from taking an unsafe action and shut down the problematic part before the issue spreads.
The chatbot doesn’t need to be perfect when it ships. The team needs to know where it can fail, what those failures look like and how to deal with them.
Top comments (0)