Originally published on AIdeazz — cross-posted here with canonical link.
My first production AI agent shipped with 131 tests in its evaluation harness. This wasn't a luxury; it was a non-negotiable step after a $0.03/run silent failure nearly cost me a client. I built the harness before writing new features, a decision driven by the fundamental limitations of traditional unit tests in multi-agent systems. Without it, I would have pushed a "working" agent that consistently failed to meet user intent under specific, common conditions, burning compute and trust.
The $0.03 Silent Failure
The agent's job was simple: process a user's natural language request for a specific report, query a database, and return the data. My initial unit tests covered the LLM's parsing of intent, the database query generation, and the final data formatting. All green. The agent worked perfectly in my dev environment with my curated test cases.
Then, a user asked for "last month's sales figures for product X." The agent returned "no data found." My logs showed the LLM correctly identified "product X" and "last month." The database query looked correct. But the user was seeing an empty result.
The problem wasn't in the individual components. The LLM correctly extracted the date range as "last month." My database query generator correctly translated "last month" into BETWEEN '2023-10-01' AND '2023-10-31'. The silent failure was in the interaction between the LLM's interpretation of "last month" and the database's actual data schema. My sales data was stored with a report_date field that represented the end of the reporting period. So "last month's sales" meant data with report_date = '2023-10-31'. The agent was querying for a range when it should have been querying for a specific date.
This wasn't a bug in my code, nor in the LLM's logic, nor in the database. It was a mismatch in semantic interpretation across system boundaries, a class of error traditional unit tests are blind to. Each component passed its individual test, but the system as a whole failed to deliver the correct outcome for a common user request. Each failed run cost $0.03 in Groq inference and Oracle Cloud database queries. Small individually, but compounding rapidly with user interaction.
Why Unit Tests Fail Multi-Agent Systems
Unit tests validate isolated functions or methods. They are excellent for ensuring add(a, b) returns a + b. They can even validate that an LLM call with a specific prompt returns a JSON object matching a schema.
What unit tests cannot do for AI agents:
- Validate emergent behavior: The core of an AI agent is its ability to reason and adapt. This emergent behavior, especially in multi-agent systems where agents interact, is not reducible to individual function calls. A unit test cannot predict how Agent A's output, slightly perturbed by a new LLM version, will affect Agent B's subsequent action.
- Test for semantic drift: As seen with my "last month" example, an LLM's interpretation of a phrase might subtly shift, or a downstream system's interpretation might differ. Unit tests don't capture these semantic mismatches across component boundaries.
- Cover the long tail of user intent: Users are unpredictable. They will phrase requests in ways you never anticipated. Unit tests are typically written for happy paths and known edge cases. An eval harness, especially one driven by synthetic data generation, can explore a much wider range of inputs.
- Measure user experience metrics: Was the response helpful? Was it fast enough? Was it accurate from the user's perspective? Unit tests only check internal correctness, not external utility.
My 4-Layer AI Agent Evaluation Harness
My current harness for a production agent on Oracle Cloud, routing between Groq and Claude, consists of 131 tests across four distinct layers. This structure allows me to catch failures at different levels of abstraction and interaction.
Layer 1: Component-Level LLM Output Validation (38 tests)
These are the closest to traditional unit tests, but specifically for LLM outputs. They ensure that individual LLM calls adhere to expected formats and basic content constraints.
- Schema validation: Does the LLM's JSON output conform to the Pydantic schema? (e.g.,
assert pydantic_model.parse_raw(llm_output)) - Keyword presence: Does the output contain expected keywords for a given intent? (e.g.,
assert "SELECT" in sql_query_output) - Basic content checks: Is a date present if a date was requested? Is a product ID present? (e.g.,
assert re.search(r'\d{4}-\d{2}-\d{2}', date_string)) - Tool call validation: If the LLM is supposed to call a tool, does the tool name and arguments match the expected format? (e.g.,
assert tool_call.name == "get_report" and "product_id" in tool_call.args)
These tests run quickly, often in milliseconds, and are crucial for catching regressions in prompt engineering or LLM model updates. I run them with Groq's Llama 3 8B for speed and cost-efficiency ($0.00000027/token for output).
Layer 2: Agent Step-by-Step Logic (52 tests)
This layer tests the internal decision-making and state transitions of the agent. It ensures that given a specific input, the agent progresses through the correct sequence of actions.
- Intent routing: If the user asks for X, does the agent correctly identify intent Y and route to the corresponding sub-agent or tool? (e.g.,
assert agent_state.current_intent == "REPORT_GENERATION") - Contextual awareness: If the user asks a follow-up question, does the agent correctly use previous turns' context? (e.g.,
assert "previous_product_id" in agent_state.context) - Error handling paths: If a tool call fails (simulated), does the agent correctly inform the user or attempt a retry? (e.g.,
assert "error_message" in agent_response and "try again" in agent_response.text) - Conditional logic: If a certain condition is met (e.g., missing required parameter), does the agent ask for clarification? (e.g.,
assert "missing_parameter" in agent_response.text)
These tests often involve mocking external services (database, APIs) to isolate the agent's internal logic. They are still relatively fast, typically under 100ms per test.
Layer 3: End-to-End Functional Scenarios (35 tests)
This is where the eval harness truly shines. These tests simulate real user interactions from start to finish, including all external integrations (database, APIs, other agents). This is where my "last month" bug would have been caught.
- Full user journeys: "Show me sales for product X last month." -> Agent -> DB -> Data -> Formatted Response. The test asserts the final output matches the expected data.
- Complex multi-turn conversations: "What's the weather?" -> "In London." -> "Tomorrow." The test asserts the final weather forecast is correct for London, tomorrow.
- Edge cases with real data: Querying for non-existent products, dates far in the past/future, malformed inputs.
- Performance under load (basic): A small subset of these tests are run concurrently to check for basic race conditions or resource contention.
These tests are more expensive, as they involve full LLM calls and database interactions. I typically run them with Claude 3 Haiku or Opus depending on complexity, costing $0.00025/token for Haiku input and $0.00125/token for output. A full run of this layer can cost $0.01-$0.03 per test. This is why I keep the number of tests here focused on critical paths and high-impact scenarios.
Layer 4: User Experience & Latency (6 tests)
This layer is less about correctness and more about the qualitative aspects of the agent.
- Response time: Is the agent responding within acceptable latency thresholds? (e.g.,
assert response_time < 5.0seconds for a complex query). This is critical for Telegram/WhatsApp agents. - Tone and helpfulness: Does the agent maintain a consistent, helpful tone? This is often a subjective evaluation, but I use a separate LLM (e.g., Claude 3 Haiku) to evaluate the agent's response against a rubric. (e.g., "Rate the helpfulness of this response on a scale of 1-5, explaining your reasoning.")
- Conciseness: Is the agent providing just the necessary information, or is it verbose? (e.g.,
assert len(response_text.split()) < 200for a simple query).
These tests are often the most expensive due to the LLM-as-a-judge component, but they are crucial for ensuring the agent is not just correct, but usable.
The Cost of Rigor: $0.03/Run
Running all 131 tests costs approximately $0.03 per full harness execution. This breaks down roughly as:
- Layer 1 (Groq): ~$0.0005
- Layer 2 (Mocked): ~$0.0001
- Layer 3 (Claude Haiku/Opus + Oracle DB): ~$0.025
- Layer 4 (Claude Haiku as judge): ~$0.004
This cost is negligible compared to the potential loss of client trust or wasted development time debugging production issues. I run the full harness on every significant code commit and before every deployment. It's integrated into my CI/CD pipeline on Oracle Cloud Infrastructure.
Building this harness took time – about two weeks initially. But it has paid for itself multiple times over by catching subtle interaction bugs and semantic mismatches that no amount of manual testing or traditional unit tests would have found. It's the only way I can confidently ship AI agents into production without constant fear of silent, costly failures.
Frequently Asked Questions
Q: How do you generate the test cases for Layer 3 (End-to-End Functional Scenarios)?
A: I start with real user queries from logs, then use an LLM (Claude 3 Opus) to generate variations and edge cases based on those. I also manually craft critical path scenarios.
Q: What's your strategy for mocking external services in Layer 2?
A: I use Python's unittest.mock library extensively. For database interactions, I mock the ORM layer or the database client directly to return predefined datasets, ensuring the agent's logic is tested independently of actual DB state.
Q: How do you handle non-deterministic LLM outputs in your tests?
A: For Layer 1, I use schema validation and keyword presence, which are robust to minor phrasing variations. For Layer 3, I often use fuzzy string matching or LLM-as-a-judge (another LLM) to evaluate the correctness of the final output, rather than exact string matching.
Q: What's the biggest challenge in maintaining such a large test suite?
A: Keeping the test data and expected outputs up-to-date as the agent's capabilities evolve. I automate this where possible by regenerating expected outputs for certain tests, but it still requires periodic manual review.
Q: Why Oracle Cloud for infrastructure?
A: Oracle Cloud offers competitive pricing for compute and database services, especially for smaller operations like mine without VC funding. Their free tier and always-free services are also a significant advantage for bootstrapping.
Top comments (0)