Testing Production AI Agents: A Practical Framework for Graph-Based Agent Systems

Table of Contents

1. Introduction
2. Why Testing AI Agents Is Different
3. The Anatomy of a Production AI Agent
4. Why Traditional Unit Testing Becomes Difficult
5. Testing at Different Levels 5.1 Deterministic Unit Testing 5.2 Tool-Level Testing 5.3 Graph / Orchestration Testing 5.4 End-to-End Scenario Testing
6. Separating Deterministic and Probabilistic Components
7. Testing Individual Tools Without LLM Calls
8. Designing an End-to-End Agent Test Set
9. Testing Complex Business Requirements
10. Testing Graph Routing and State Transitions
11. Testing Tool Selection and Tool Arguments
12. Testing Retrieval and Validation Loops
13. Testing Failure and Recovery Behaviour
14. Testing Structured LLM Outputs
15. Measuring Agent Quality 15.1 Accuracy 15.2 Tool-Call Accuracy 15.3 Routing Accuracy 15.4 Retrieval Quality 15.5 Reliability and Stability 15.6 Token and Latency Efficiency
16. Regression Testing for Agents
17. Building an Agent Evaluation Dataset
18. Mocking vs Real Dependencies
19. Controlling LLM Variability During Testing
20. Common Testing Mistakes
21. A Practical Testing Architecture
22. Lessons Learned from Production Agent Development
23. Conclusion

1. Introduction

AI agents are often presented as applications that simply connect an LLM to a collection of tools.

In production, the reality is considerably more complicated.

A serious agent may contain planners, routers, state management, retrieval systems, database tools, validation logic, retry mechanisms, and multiple execution paths. The LLM introduces another layer of uncertainty because the same input does not necessarily produce exactly the same reasoning or tool usage.

As a result, testing an agent requires a different mindset from testing a conventional backend service.

The central idea of this write-up is to separate testing into two categories:

deterministic software testing and probabilistic agent evaluation.

Deterministic components should be tested directly. LLM-driven behaviour should be evaluated through realistic scenarios.

2. Why Testing AI Agents Is Different

A traditional function might behave like:

input → function → output

An agent can behave like:

user question → planner → graph routing → tool selection → retrieval → validation → retry → synthesis → response

The number of possible execution paths can grow quickly.

A single user question may result in:

• one tool call
• several tool calls
• no tool calls
• a retry
• a different retrieval strategy
• a clarification question
• a completely different graph branch

This makes exhaustive unit testing impractical.

The difficulty increases further when business requirements are highly contextual.

3. The Anatomy of a Production AI Agent

A useful way to understand testing requirements is to decompose the agent into layers.

Layer 1: Deterministic application logic

Examples:

• database queries
• data transformation
• validation functions
• state updates
• graph execution
• retry logic
• serialization

These components can usually be tested using conventional unit tests.

Layer 2: Tools

Examples:

• project search
• semantic search
• attachment retrieval
• location lookup
• database queries

Tools usually have deterministic contracts even when the LLM decides when to call them.

Layer 3: LLM reasoning

Examples:

• intent classification
• planning
• tool selection
• query construction
• response generation

This layer is inherently probabilistic.

Layer 4: Agent behaviour

This is the overall combination of the previous layers.

The most important question becomes:

Given a realistic user request, does the agent eventually produce the correct behaviour?

4. Why Traditional Unit Testing Becomes Difficult

Suppose an agent has ten graph nodes and each node can potentially branch into several paths.

Testing every possible combination can quickly become unmanageable.

It is also possible to have a test pass even though the overall agent is broken.

For example:

Tool A works correctly.

Tool B works correctly.

Planner works correctly.

Router works correctly.

But the graph may still send the planner output to the wrong branch.

This is why testing only individual components is insufficient.

5. Testing at Different Levels

A practical architecture uses multiple testing layers rather than one large test suite.

5.1 Deterministic Unit Testing

Test ordinary software directly.

Examples:

• parser behaviour
• database query construction
• state transformations
• validation rules
• graph edge conditions

These tests should be fast and inexpensive.

5.2 Tool-Level Testing

Test tools without involving an LLM.

For example:

project_name → search_project() → database → expected project

This allows database and retrieval behaviour to be validated independently.

5.3 Graph / Orchestration Testing

Test whether the graph transitions correctly.

For example:

Planner → ProjectQueryDecision → GetProjectName → ProjectQuery → Reply

The LLM output can be replaced with a deterministic fixture so that graph behaviour can be verified independently.

5.4 End-to-End Scenario Testing

Provide realistic user questions to the complete agent.

The test evaluates the final behaviour rather than an individual function.

6. Separating Deterministic and Probabilistic Components

One of the most useful design principles is:

Do not use an LLM to test functionality that can be tested deterministically.

Suppose a project search tool costs an LLM call every time it is tested indirectly.

That means thousands of tokens might be spent validating a function that is ultimately just querying a database.

Instead:

Input → Tool → Expected Result

can be tested directly.

Then:

User Question → LLM → Tool Selection

can be tested separately.

This gives each component the appropriate testing strategy.

7. Testing Individual Tools Without LLM Calls

Before exposing a tool to an LLM, validate the tool itself.

For example, a project search tool can have tests for:

• exact project names
• partial project names
• spelling variations
• nonexistent projects
• multiple matching projects
• empty input
• malformed input
• database failures

The output should have a deterministic expectation.

Once those tests pass, the tool becomes a trusted component of the agent.

This significantly reduces debugging complexity.

If a production test fails later, the investigation can focus on the LLM's decision-making rather than immediately suspecting the underlying tool.

8. Designing an End-to-End Agent Test Set

For the complete application, a scenario-based dataset is more useful than hundreds of artificial unit cases.

The dataset should represent real user behaviour.

For a property-information agent, scenarios might include:

• identify a project from an ambiguous name
• search for projects matching a location
• compare two projects
• ask for detailed project information
• ask about attachments
• ask a follow-up question
• provide incomplete requirements
• refer to a previously mentioned project
• request information unavailable in the database
• intentionally provide ambiguous information

The expected result does not necessarily have to be an exact text match.

It can instead define expected behaviour.

For example:

Expected:
- identify project
- retrieve project information
- retrieve attachments
- answer user

Not acceptable:
- choose unrelated project
- skip required retrieval
- invent project information

9. Testing Complex Business Requirements

Business requirements are often more difficult than technical requirements.

A user might ask:

"Tell me about the project."

That simple sentence could require very different behaviour depending on conversation history.

The agent may need to determine:

• which project?
• what information?
• whether attachments are required?
• whether the user is asking about a property or project?
• whether additional clarification is necessary?

Therefore, test cases should include conversational context rather than isolated questions.

This is where scenario-based evaluation becomes particularly valuable.

10. Testing Graph Routing and State Transitions

Graph-based agents introduce another category of failures.

The nodes themselves may work correctly while the transitions are incorrect.

Testing should therefore verify:

state + node output → expected next node

For example:

Intent = project_search
        ↓
ProjectDecision
        ↓
GetProjectName
        ↓
ProjectQuery

Tests should verify both:

1. the state produced by a node
2. the graph transition triggered by that state

This makes graph errors much easier to isolate.

11. Testing Tool Selection and Tool Arguments

It is not enough for the agent to call the correct tool.

The tool arguments must also be correct.

For example:

Tool: get_project
Expected:
{
    "project_name": "ABC Residence"
}

The agent could fail by:

• calling the wrong tool
• using the wrong project name
• passing irrelevant context
• omitting required parameters
• making unnecessary calls

Therefore, an evaluation framework should record tool calls as part of the agent trace.

12. Testing Retrieval and Validation Loops

Production agents frequently contain feedback loops.

For example:

retrieve → validate → re-query → validate

This creates another testing dimension.

You should test:

• successful retrieval
• partially relevant retrieval
• completely irrelevant retrieval
• validation failure
• retry behaviour
• maximum retry limit
• successful recovery

The goal is not only to verify that the happy path works.

The recovery path must also be predictable.

13. Testing Failure and Recovery Behaviour

Reliable agents need explicit failure testing.

Examples include:

• LLM returns malformed JSON
• tool returns no results
• database query fails
• retrieval produces irrelevant documents
• required information is missing
• tool timeout occurs
• LLM attempts an invalid tool call

A mature agent should fail in controlled ways rather than simply producing an incorrect answer.

14. Testing Structured LLM Outputs

Whenever possible, intermediate LLM outputs should use a schema.

For example:

{
  "intent": "project_search",
  "requires_tool": true,
  "tool": "get_project",
  "reason": "Project identity must be resolved first."
}

Testing becomes easier because the evaluator can validate:

• schema correctness
• allowed values
• required fields
• consistency between fields
• tool validity

This is considerably easier to evaluate than unconstrained natural language.

15. Measuring Agent Quality

Agent evaluation should go beyond final-answer accuracy.

Useful metrics include:

Accuracy

Did the agent provide the correct answer?

Tool-call accuracy

Did the agent call the correct tools?

Routing accuracy

Did it traverse the correct graph path?

Retrieval quality

Did it retrieve relevant information?

Reliability

Does it behave consistently across repeated runs?

Efficiency

How many LLM calls and tokens were required?

Latency

How long did the full execution take?

This allows an agent to be evaluated as a software system rather than merely as a chatbot.

16. Regression Testing for Agents

Every production failure can become a future test case.

For example:

Production incident
        ↓
Identify failure mode
        ↓
Create regression scenario
        ↓
Add to evaluation dataset
        ↓
Prevent recurrence

Over time, the scenario dataset becomes a practical representation of the application's business requirements.

This is particularly valuable because agent behaviour can change significantly when:

• prompts change
• models change
• tools change
• graph logic changes
• retrieval indexes change

17. Building an Agent Evaluation Dataset

A useful evaluation dataset should contain:

Field	Purpose
User question	Original scenario
Conversation context	Relevant history
Expected intent	Required interpretation
Expected tools	Valid tool usage
Expected entities	Project/property/etc.
Expected outcome	Required behaviour
Failure conditions	Unacceptable behaviour
Evaluation result	Pass/fail or score

This makes agent testing repeatable rather than dependent on manually inspecting conversations.

18. Mocking vs Real Dependencies

Different tests require different levels of realism.

For unit tests, mock external dependencies aggressively.

For integration tests, use real databases and real tools where practical.

For end-to-end evaluation, test the complete production-like pipeline.

A useful principle is:

The lower the test level, the more deterministic it should be.

The higher the test level, the more realistic it should be.

19. Controlling LLM Variability During Testing

LLM behaviour introduces variance.

For stable evaluation, control variables such as:

• model version
• temperature
• system prompt
• tool definitions
• retrieval dataset
• structured output schema

Even then, exact output matching is usually inappropriate.

Evaluation should focus on semantic correctness and required behaviour.

20. Common Testing Mistakes

Several approaches tend to fail in production.

Testing everything through the LLM

This is expensive and makes failures difficult to diagnose.

Testing only individual functions

This misses orchestration and graph-level failures.

Using only exact answer matching

A correct answer can be expressed in many valid ways.

Testing only happy paths

Production failures frequently occur in ambiguous and incomplete requests.

Ignoring tool-call traces

The final answer may look reasonable even though the agent reached it through an invalid process.

21. A Practical Testing Architecture

A practical architecture can therefore look like:

                ┌──────────────────────┐
                │  Unit Tests          │
                │  Deterministic Code  │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Tool Tests           │
                │  DB / Retrieval       │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Graph Tests          │
                │  Routing / State      │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Agent Evaluation     │
                │  Scenario Dataset     │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Production Feedback  │
                │  Regression Cases     │
                └──────────────────────┘

The important point is that no single testing strategy is sufficient.

22. Lessons Learned from Production Agent Development

The main lesson is that testing an agent is fundamentally an exercise in decomposition.

Trying to test the entire system through end-to-end LLM calls is expensive.

Trying to test the entire system through traditional unit tests is insufficient.

The more practical approach is to divide the system into deterministic and probabilistic boundaries.

Deterministic logic should be tested directly.

Tools should be tested independently.

Graph execution should be tested with controlled inputs.

LLM reasoning should be evaluated through realistic scenarios.

End-to-end tests should verify whether the complete system satisfies business requirements.

23. Conclusion

Production AI agents sit somewhere between software engineering and probabilistic systems engineering.

The challenge is not simply making an LLM produce a good answer.

The challenge is making the entire system predictable enough to operate reliably:

LLM + tools + graph + retrieval + business logic + state

That changes how testing needs to be designed.

Rather than asking:

"Can I unit test this agent?"

A better question is:

"Which parts of this agent are deterministic, which parts are probabilistic, and what is the correct testing strategy for each?"

That distinction has become one of the most useful principles in building stable production agents.