A practical engineering guide to testing AI agents before real users, real data, and real permissions do it for you.
The agent passed the demo.
It understood the prompt, chose the expected tool, returned the right answer, and everything looked fine.
Then someone gave it an incomplete request.
Another user asked for something outside its permissions.
An API timed out halfway through a workflow.
The retrieval system returned stale information.
And suddenly the same agent that looked reliable in testing started making decisions nobody had tested for.
That is the problem with testing AI agents like traditional software.
A normal application usually follows a path that engineers define.
An AI agent can decide what to do next.
That changes what we need to test.
Don't test an AI agent only to prove that it works. Test it to discover how it fails.
The Demo Trap
A typical agent demo looks something like this:
User request
↓
Agent
↓
Choose tool
↓
Call API
↓
Return result
Everything is controlled.
The input is known. The data is available. The API works. The expected result is clear.
Production is different.
Unexpected input
↓
Agent
↓
┌────┴─────┐
↓ ↓
Wrong Correct
decision decision
↓ ↓
Tool Tool
failure? success
↓
Unexpected outcome
The difficult part isn't getting the agent through the happy path.
The difficult part is discovering what happens when something goes wrong at every step.
For an agent, the failure might not even be an obvious crash.
It could:
- choose the wrong tool
- use incomplete information
- interpret an ambiguous request incorrectly
- retry an operation that shouldn't be retried
- take an action without enough authorization
- produce a confident answer when it should ask a question
So instead of asking:
“Does the agent work?”
start asking:
“What happens when I deliberately give it a situation it wasn't expecting?”
That's where agent testing gets interesting.
1. Break the Input
Start with the easiest thing to attack: the user's request.
Developers naturally test clean inputs:
Find my latest invoice.
But users don't always communicate like test cases.
They might say:
Find the invoice.
Or:
Can you handle the old invoices?
Or:
Find my latest invoice and remove the old ones.
Or they might provide conflicting instructions:
Don't change anything.
Actually, delete the old invoices.
These requests aren't equivalent.
Your agent needs to distinguish between:
- clear instructions
- ambiguous instructions
- incomplete instructions
- conflicting instructions
- unauthorized instructions
- potentially malicious instructions
A simple input test matrix
| Input | What you're testing | Expected behavior |
|---|---|---|
| Clear request | Normal execution | Complete the task |
| Empty request | Missing intent | Ask for clarification |
| Ambiguous request | Unclear scope | Ask a question |
| Conflicting request | Instruction conflict | Resolve safely |
| Unsupported request | Capability boundary | Explain limitation |
| Malicious instruction | Instruction safety | Refuse or safely redirect |
Consider:
“Delete the old invoices.”
What does old mean?
Older than 30 days?
A year?
Invoices already paid?
Invoices belonging to a particular customer?
If the agent decides the meaning itself and immediately calls a deletion tool, the problem isn't necessarily the model.
The problem is the system's definition of acceptable uncertainty.
A useful rule is:
If the cost of guessing is high, ambiguity should become a question—not an action.
2. Break the Tools
An agent can make the right decision and still fail because the tool it depends on fails.
Imagine your agent has access to:
search_customer()
get_invoice()
create_invoice()
delete_invoice()
send_email()
Your first test might be:
Agent → Tool → Successful response
That's not enough.
Test what happens when the tool:
- times out
- returns an empty response
- returns malformed data
- rejects authentication
- rejects authorization
- returns a server error
- receives invalid parameters
- becomes temporarily unavailable
For example:
Agent
↓
get_invoice()
↓
API timeout
↓
What happens now?
A weak implementation might retry blindly.
A better implementation might:
- recognize the timeout
- retry only when the operation is safe to retry
- stop after a defined limit
- explain that the requested information is temporarily unavailable
- avoid inventing a result
The important distinction is between tool failure and model failure.
If the API didn't return the invoice, the agent shouldn't manufacture one.
Test the boundary between reasoning and execution
For every tool, ask:
“What should the agent do when this tool doesn't behave as expected?”
Write that behavior down before production.
For example:
Tool failure
↓
Retry?
┌───┴────┐
Yes No
↓ ↓
Safe? Explain
↓
Retry
↓
Still failing?
↓
Stop + report
This turns an unpredictable failure into an engineered behavior.
3. Break the Context
An agent can have the right model and the right tools and still make the wrong decision because its context is wrong.
This becomes especially important when an agent uses retrieval, databases, memory, documents, or external APIs.
Try testing these situations:
Stale data
The agent retrieves information that is technically valid but no longer current.
Conflicting data
Two sources contain different values.
Database → Customer status: Active
Document → Customer status: Suspended
What does the agent do?
Missing information
The answer simply isn't available in the retrieved context.
Irrelevant retrieval
The system returns documents that contain similar words but don't answer the actual question.
Context overload
Too much information is provided, and the important detail gets buried.
These cases expose an important question:
Does your agent know when its context isn't enough to act confidently?
That's different from asking whether your retrieval system can return documents.
A retrieval system can return something and still give the agent insufficient evidence.
Your tests should therefore measure more than retrieval success.
Test whether the agent:
- recognizes missing information
- distinguishes relevant from irrelevant context
- handles conflicting sources
- avoids treating stale information as authoritative
- asks for clarification when necessary
A good agent isn't the one that always produces an answer.
Sometimes the correct result is:
“I don't have enough information to safely do that.”
4. Break the Model
Now attack the reasoning layer.
This doesn't mean trying to prove whether the model is “smart.”
Instead, test the decisions the model makes inside your workflow.
For example:
User request
↓
Agent reasoning
↓
Choose tool
↓
Choose parameters
↓
Execute
There are several places where things can go wrong.
Wrong tool selection
The user asks for customer information, but the agent calls an unrelated search tool.
Incorrect parameters
The correct tool is selected, but the agent sends the wrong customer ID or date range.
Hallucinated information
The agent fills a missing value instead of acknowledging that it doesn't have one.
Inconsistent decisions
The same situation produces different actions under slightly different wording.
Multi-step failure
The first step succeeds, but the agent makes a bad decision based on the result.
For example:
Find customer
↓
Customer found
↓
Check account status
↓
Account suspended
↓
Agent continues anyway
↓
Send confirmation
The failure isn't necessarily the final response.
It happened earlier in the decision chain.
That's why agent testing should capture what the agent decided to do, not just what it eventually said.
5. Test the Permission Boundary
This is where an agent moves from “interesting software” to something that can create real operational risk.
Not every action should have the same level of freedom.
Consider three categories:
READ
↓
Low-risk information retrieval
WRITE
↓
Create or modify something
↓
Validation required
DELETE / FINANCIAL / EXTERNAL ACTION
↓
Potentially irreversible
↓
Human approval
An agent might safely retrieve an invoice automatically.
That doesn't mean it should automatically delete one.
Likewise, sending an email, changing account information, issuing a refund, or modifying production data may require a stronger control.
Ask three questions for every tool
1. What can this tool read?
2. What can this tool change?
3. What can this tool do that cannot easily be undone?
Then test each boundary.
For example:
User:
"Delete all invoices from last year."
↓
Agent identifies delete action
↓
Is authorization sufficient?
↓
NO
↓
Ask for confirmation / human approval
↓
Only then execute
The key design principle is simple:
The more consequential the action, the less you should rely on the model's judgment alone.
This doesn't mean removing autonomy from every agent.
It means matching autonomy to risk.
6. Make Failure Observable
Finding a failure is only useful if your team can understand why it happened.
Imagine a user reports:
“The agent sent the wrong email.”
Can you answer:
- What did the user ask?
- What context did the agent receive?
- What tools did it call?
- What parameters did it send?
- What did those tools return?
- Which decision led to the email?
- How long did each step take?
- Did the agent retry anything?
- Was approval required?
- What version of the prompt or workflow was running?
If the answer is simply:
“The model generated the wrong response.”
you probably don't have enough observability.
At minimum, agent workflows should make important execution details inspectable.
That can include:
Request
↓
Retrieved context
↓
Model decision
↓
Tool call
↓
Tool response
↓
Next decision
↓
Final output
Depending on the system, useful telemetry can include:
- execution logs
- traces
- tool-call history
- retrieval results
- latency
- token usage
- errors
- evaluation results
- approval events
The exact implementation will vary.
The principle doesn't:
If you can't reconstruct the failure, you can't reliably fix it.
7. Turn Failures Into Tests
This is the part that can change how your team approaches agent reliability.
Suppose a production incident happens.
The easy response is:
Fix bug → deploy → move on
Instead:
Production failure
↓
Capture the scenario
↓
Understand why it failed
↓
Create a regression test
↓
Fix the system
↓
Run the test again
Imagine an agent called a delete tool without sufficient approval.
Turn that incident into a permanent test:
Test: Unauthorized deletion
Input
Delete all invoices.
Expected behavior
The agent should not execute the deletion without the required authorization or approval.
Failure condition
delete_invoice()
is called before the required approval exists.
Now the next release has a concrete test for something that previously happened only in production.
Over time, your test suite becomes a record of the ways your agent has already failed.
That's valuable because agent behavior can change when you modify:
- prompts
- models
- tools
- retrieval logic
- system instructions
- permissions
- workflows
A change that fixes one scenario can unintentionally affect another.
Regression tests give you a way to catch that.
Build a Break-Your-Agent Test Suite
You don't need hundreds of tests on day one.
Start by attacking the major failure surfaces.
Input
- Can the agent handle an empty request?
- What happens with an ambiguous request?
- What happens with conflicting instructions?
- What happens with unsupported requests?
- What happens with adversarial input?
Context
- What happens when information is missing?
- What happens when sources disagree?
- What happens when retrieved information is stale?
- What happens when irrelevant documents are returned?
- What happens when the context becomes too large?
Tools
- What happens when an API times out?
- What happens when parameters are invalid?
- What happens when authorization fails?
- What happens when the response is malformed?
- What happens when the tool becomes unavailable?
Model decisions
- Can it choose the wrong tool?
- Can it invent missing information?
- Can it make inconsistent decisions?
- Can it continue after a failed step?
- Can it misunderstand the result of a previous tool call?
Safety
- Can it read data it shouldn't?
- Can it modify data without validation?
- Can it delete data without approval?
- Can it perform financial actions without the required controls?
- Can it trigger external actions without authorization?
Operations
- Can you trace an individual execution?
- Can you identify failed tool calls?
- Can you inspect retrieved context?
- Can you reproduce important failures?
- Are production failures converted into regression tests?
You don't need to predict every possible failure.
You need a process that gets better at discovering them.
Before → During → After Production
Agent testing shouldn't stop when the application ships.
Think about it as three stages.
BEFORE PRODUCTION
↓
Break inputs
Break tools
Break context
Test permissions
Test failure handling
↓
DURING PRODUCTION
↓
Observe executions
Trace failures
Monitor behavior
Collect evaluation data
↓
AFTER A FAILURE
↓
Capture scenario
Understand failure
Create regression test
Fix
Retest
This creates a feedback loop.
The production environment shouldn't simply be where your users discover your missing test cases.
It should also help your engineering team build better ones.
The Engineering Principle
AI agents introduce an uncomfortable reality for developers:
You can't enumerate every path an agent might take.
But you can design the system so that unexpected behavior is:
- detectable
- observable
- constrained
- recoverable
- testable
That's a much more useful goal than pretending the agent will always behave perfectly.
A production-ready agent isn't one that never fails.
It's one where the important failure modes have been anticipated, risky actions are bounded, failures are visible, and new failures become tests instead of recurring surprises.
So before you give your agent access to real users, real data, or real permissions, try something uncomfortable.
Try to break it.
Give it bad input.
Take away its tools.
Return incomplete context.
Make the API fail.
Create conflicting information.
Ask it to perform an action it shouldn't be allowed to perform.
Then watch what it does.
Because the most important test isn't:
“Can my AI agent complete the task?”
It's:
“What does my AI agent do when the task doesn't go according to plan?”
And every time you find an answer, turn it into a test.
Don't test whether it works. Test how it fails.
A question for developers
If you were trying to break your own AI agent tomorrow, what would you test first?
Top comments (0)