Originally published on AIdeazz — cross-posted here with canonical link.
My AI agents failed silently for 17 hours. The user experience was a dead end: "I'm sorry, I can't fulfill this request." No error logs, no exceptions, just a polite refusal. This wasn't a bug in my Python code; it was a subtle, systemic drift in agent behavior. I would have shipped this broken experience directly to users without my evaluation harness. It caught the regression across 131 tests, costing me $0.03 per full run, before I wrote a single new feature.
The problem with AI agents isn't just about writing correct code; it's about ensuring correct behavior across a dynamic, non-deterministic system. Unit tests catch IndexError or KeyError. They don't catch an agent deciding that "create a marketing plan for a new SaaS product" is now out of scope, even though it was a core capability last week.
The Silent Killer: Behavioral Drift
I run a multi-agent system on Oracle Cloud, routing requests between Groq and Claude, serving users on Telegram and WhatsApp. My agents handle everything from content generation to data analysis. The core challenge isn't just orchestrating these LLMs; it's maintaining their intent.
A recent update to my prompt template for the MarketingStrategist agent introduced a new constraint: "Focus on B2B SaaS only." This was intended to refine its output. What it actually did was make the agent refuse any request that didn't explicitly mention "B2B SaaS." A user asking for "a marketing plan for a new coffee shop" would get the polite refusal. My existing unit tests, which checked function calls and output formats, passed with flying colors. The agent's behavior had regressed.
This behavioral drift is insidious. It doesn't throw an exception. It doesn't log a critical error. It simply changes the agent's interpretation of its mandate. Without an evaluation harness, I'd be relying on manual spot-checks or, worse, user complaints.
The 4-Layer Evaluation Harness Architecture
I built my evaluation harness with four distinct layers, designed to catch different types of regressions:
- Input Validation & Routing (Layer 1): Does the initial router agent correctly identify the intent and route to the right specialized agent? Does it handle malformed inputs gracefully?
- Agent Goal Fulfillment (Layer 2): Does the specialized agent actually achieve its stated goal? This is the most critical layer for behavioral drift.
- Output Quality & Format (Layer 3): Is the output structured correctly (e.g., JSON, Markdown)? Is the content coherent, relevant, and free of hallucinations?
- Tool Usage & External API Integration (Layer 4): If the agent uses external tools (e.g., a search API, a database query tool), are those tools called correctly, and is their output integrated properly?
Each layer has its own set of tests. For example, Layer 1 includes tests like "route 'create a marketing plan' to MarketingStrategist" and "route 'analyze sales data' to DataAnalyst." Layer 2 for MarketingStrategist includes tests like "generate a 3-point marketing plan for a new coffee shop" and "create a content calendar for a B2B SaaS product." The coffee shop test is what caught the regression.
My harness is a Python script that iterates through a YAML file of test cases. Each test case defines:
-
input: The user prompt. -
expected_agent: (Optional, for Layer 1) The agent expected to handle the request. -
expected_output_keywords: (Optional, for Layer 2 & 3) Keywords expected in the final output. -
expected_tool_calls: (Optional, for Layer 4) Regex patterns for tool calls. -
negative_keywords: (Optional, for Layer 2 & 3) Keywords not expected. -
max_tokens: (Optional) To control LLM cost.
The script then sends the input to my main router agent, captures the full interaction trace, and asserts against the defined expectations.
The 131 Tests and $0.03/Run Cost
My current suite has 131 tests. This includes 28 tests for Layer 1 (routing), 65 for Layer 2 (goal fulfillment across 5 specialized agents), 25 for Layer 3 (output quality), and 13 for Layer 4 (tool usage).
Running the full suite takes approximately 3 minutes and costs, on average, $0.03. This cost is primarily from LLM inference. I use Groq for its speed and low cost for initial routing and simpler tasks, and Claude 3 Haiku/Sonnet for more complex generation. The harness itself uses a small, fast LLM (often Groq Mixtral) for output evaluation (e.g., "does this output contain a marketing plan?"). This meta-LLM evaluation is crucial for Layer 2 and 3, as simple keyword matching isn't robust enough.
Here's a breakdown of a typical test run cost:
- Initial routing (Groq Mixtral): ~0.000001 tokens/char * 100 chars input * 131 tests = negligible.
- Specialized agent inference (Groq Mixtral/Claude Haiku): Varies, but averages 2000 tokens/test (input + output).
- Groq Mixtral: $0.50/M tokens. 2000 tokens * 65 tests (Layer 2) = 130,000 tokens. Cost: $0.065.
- Claude Haiku: $0.25/M tokens. 2000 tokens * 65 tests = 130,000 tokens. Cost: $0.0325.
- Evaluation LLM (Groq Mixtral): ~500 tokens/test for evaluation prompt + response. 500 tokens * 131 tests = 65,500 tokens. Cost: $0.03275.
My actual cost per run is lower because not all tests hit the most expensive models, and many are shorter. The $0.03 figure is a conservative average. This is a trivial cost compared to the engineering hours lost debugging production issues or the reputational damage from shipping broken features.
What AI Agent Tests Catch That Unit Tests Cannot
Unit tests verify the correctness of individual functions or methods. They ensure add(2, 3) returns 5. They check if a parser correctly extracts data from a string.
AI agent tests, specifically behavioral tests, verify the emergent properties of the system. They answer questions like:
- Intent Preservation: Does the agent understand and act upon the user's original intent, even with ambiguous phrasing?
- Contextual Awareness: Does the agent correctly use information from previous turns in a conversation?
- Constraint Adherence: Does the agent respect negative constraints (e.g., "do not use external tools," "keep it under 100 words")?
- Tool Orchestration: Does the agent correctly decide when to use a tool, which tool to use, and how to use its output?
- Bias & Safety: Does the agent avoid generating harmful or biased content? (This is a harder problem, but the harness can catch obvious regressions.)
My coffee shop example perfectly illustrates this. The Python code for the MarketingStrategist agent was perfectly functional. The prompt change, however, altered its interpretation of its role, leading to a behavioral regression that a unit test would never flag. The agent was still "working" in the sense that it was processing input and producing output, but the output was a refusal, not a marketing plan.
Integrating with CI/CD (Oracle DevOps)
I've integrated this harness into my Oracle DevOps CI/CD pipeline. Every git push to main triggers a full evaluation run. If any of the 131 tests fail, the pipeline breaks, preventing deployment. This forces me to address behavioral regressions before they ever reach a staging environment, let alone production.
The output of the harness is a detailed report, showing which tests failed, the input, the expected output, and the actual output. This makes debugging straightforward. Instead of sifting through logs, I get a direct comparison of expected vs. actual behavior.
Building this harness was a significant upfront investment. It took me about two weeks of focused effort. But it has paid for itself many times over by catching silent failures and giving me confidence to iterate rapidly on agent capabilities. It's the only way I can ship production AI agents with zero VC funding and still maintain quality.
Frequently Asked Questions
Q: How do you handle non-deterministic LLM outputs in your assertions?
A: I use a combination of keyword matching, negative keyword matching, and a small, fast LLM (Groq Mixtral) to evaluate the semantic content of the output. For example, instead of asserting "output == 'Hello world'", I assert "output contains 'Hello' and 'world'" or "LLM confirms output is a greeting."
Q: What if the evaluation LLM itself hallucinates or makes a mistake?
A: This is a valid concern. I mitigate this by keeping the evaluation prompts very simple and direct, using a highly reliable model (Groq Mixtral for speed/cost, sometimes Claude 3 Haiku for higher stakes), and having human review for any new evaluation prompts. The evaluation LLM is not generating content, only classifying it.
Q: How do you manage the test data (the 131 tests)?
A: All test cases are stored in a version-controlled YAML file. This allows for easy additions, modifications, and review. I categorize tests by agent and by the type of behavior they target.
Q: What's the biggest challenge in maintaining the harness?
A: Keeping the tests relevant as agent capabilities evolve. As I add new features or refine agent prompts, I often need to update existing tests or add new ones. This requires discipline, but it's a necessary overhead for stable agent development.
Q: Why not use an existing open-source eval framework?
A: Many frameworks are either too generic (focused on LLM benchmarks) or too opinionated for my specific multi-agent, multi-LLM, Oracle Cloud setup. Building my own gave me precise control over the 4-layer architecture, cost optimization, and direct integration with my existing CI/CD and monitoring tools.
Top comments (0)