Originally published on AIdeazz — cross-posted here with canonical link.
I shipped a silent failure. My production AI agent, designed to extract structured data from user messages, started returning null for a critical field. Not an error, not an exception, just null. It passed all 27 unit tests. My CI/CD pipeline was green. The agent was "working" according to every metric I had. It took a manual spot check of 10 user interactions to find the problem. This was the moment I stopped building new features and started building an AI agent evaluation harness.
My current harness runs 131 tests across four distinct layers for every agent change. Each full run costs $0.03 on average, leveraging Groq for speed and Claude 3.5 Sonnet for complex evaluations. This setup catches issues that unit tests fundamentally cannot, because AI agent failures often manifest as semantic drift or performance degradation rather than outright code breaks.
The Problem: Semantic Drift and Silent Failures
Traditional unit tests verify deterministic code. add(2, 2) must always return 4. An AI agent, however, operates within a probabilistic domain. A prompt change, a new system message, or even a different LLM version can subtly alter its output without breaking the code structure.
My initial failure was a perfect example. The agent's job was to parse user requests for custom AI agent builds. One field was target_platform (e.g., "Telegram", "WhatsApp", "Web"). I updated the system prompt to emphasize "clarity and conciseness" in the output JSON. This seemingly innocuous change, intended to reduce verbosity, caused the LLM to sometimes omit target_platform if it felt the platform was "obvious" from context, even though the schema explicitly required it. The agent's Python code still called json.loads() and accessed data['target_platform']. When the key was missing, it defaulted to None or null depending on the parsing logic. No KeyError, no TypeError. Just null.
This is semantic drift. The agent's interpretation of the task changed, not its code execution. Unit tests, focused on function inputs and outputs, cannot catch this. They verify the parser works, not that the LLM always produces the key.
The Four Layers of My Evaluation Harness
My AI agent evaluation harness 131 tests production readiness by focusing on the agent's behavior and output quality across different scenarios. I run these tests on Oracle Cloud Infrastructure (OCI) using custom Python scripts and a PostgreSQL database for test case management.
Layer 1: Input Robustness (38 tests)
This layer focuses on how the agent handles variations in user input. It's about ensuring the agent doesn't break or produce garbage when faced with common real-world deviations.
- Variations in phrasing: 15 tests with different ways of asking the same question (e.g., "build a Telegram bot", "I need a bot for TG", "Telegram agent please").
- Missing information: 10 tests where critical fields are omitted, expecting the agent to either ask clarifying questions or return a specific "incomplete" status.
- Ambiguous requests: 8 tests with vague or contradictory instructions, checking for graceful handling or clarification.
- Edge cases: 5 tests for maximum length inputs, special characters, or empty messages.
For example, an agent designed to book appointments should, when given "book an appointment", respond with "What date and time?" not "Here's your appointment for null on null."
Layer 2: Output Fidelity (52 tests)
This is the core of catching semantic drift. It verifies that the agent's output matches the intended meaning and structure, not just the syntactic correctness.
- Schema adherence: 20 tests verifying that the output JSON strictly follows the Pydantic schema, including all required fields and correct data types. This uses
pydantic.validate_jsonand custom checks for nested structures. - Content accuracy: 15 tests where a reference LLM (Claude 3.5 Sonnet, for its reasoning capabilities) evaluates the agent's output against the original prompt and a human-defined "gold standard" answer. For instance, if the agent is supposed to summarize, the reference LLM checks if the summary captures the main points.
- Completeness: 10 tests ensuring all expected pieces of information are present in the output. This caught my
target_platformnullissue. - Conciseness/Verbosity: 7 tests evaluating if the output meets specific length or detail requirements. An agent designed for quick answers shouldn't write an essay.
The content accuracy tests are crucial. I use a prompt like: "Given the user's request: '{user_input}', and the agent's output: '{agent_output}', and the expected output characteristics: '{expected_characteristics}', evaluate if the agent's output is accurate and complete. Respond 'PASS' or 'FAIL' followed by a brief explanation." This is where the $0.03/run cost comes in, as these evaluations are often run on Claude.
Layer 3: Tool/Function Call Correctness (27 tests)
Many of my agents are multi-agent systems, routing requests to specialized sub-agents or calling external APIs. This layer verifies these interactions.
- Correct tool selection: 10 tests ensuring the agent calls the right tool for a given intent (e.g., "schedule meeting" calls
calendar_tool.schedule, notemail_tool.send). - Correct arguments: 12 tests verifying that the arguments passed to the tool are correctly extracted from the user's request and formatted according to the tool's schema. This is a common failure point for LLMs.
- Error handling: 5 tests simulating tool failures (e.g., API timeouts, invalid credentials) and checking if the agent gracefully handles them, informs the user, or retries.
I mock external APIs during these tests to control the environment and ensure deterministic outcomes.
Layer 4: Performance and Cost (14 tests)
While not directly about correctness, these are critical for production.
- Latency benchmarks: 5 tests measuring the end-to-end response time for various input complexities. I track p50, p90, and p99 latency. My target is <2 seconds for simple requests on Groq.
- Token usage: 5 tests logging input/output token counts to monitor cost per interaction. This helps identify prompt bloat or verbose LLM responses.
- Rate limit handling: 4 tests simulating hitting LLM provider rate limits and verifying the agent's retry logic or fallback mechanisms.
My current setup uses Groq for the initial LLM call in most agents due to its speed and cost-effectiveness for simple parsing and routing. More complex reasoning or evaluation tasks are routed to Claude 3.5 Sonnet. This hybrid approach keeps the average cost per agent interaction low while maintaining high quality where it matters.
Building the Harness: Infrastructure and Workflow
My harness is a Python application. Test cases are stored in a PostgreSQL database, allowing for easy management and versioning. Each test case includes:
-
test_id -
agent_id -
layer(e.g., 'input_robustness') -
test_name -
user_input -
expected_output_characteristics(a natural language description for LLM evaluation) -
expected_json_schema(for strict schema validation) -
expected_tool_call(for tool call verification) -
status(Pass/Fail) -
details(error messages, LLM evaluation rationale)
When I make a change to an agent (e.g., a prompt update, a new tool definition), I trigger a full harness run. The script iterates through the test cases, invokes the agent, and then runs the appropriate evaluation logic for each layer.
For LLM-based evaluations (Layer 2), I use a dedicated EvaluationAgent (itself an AI agent) that takes the user input, the agent's output, and the expected characteristics, then returns a PASS/FAIL and a reason. This EvaluationAgent is typically powered by Claude 3.5 Sonnet due to its strong reasoning capabilities.
The results are logged back into the database and presented through a simple web dashboard (built with Streamlit for internal use). A failed test immediately blocks deployment.
The Cost of Not Testing
The null target_platform issue cost me several hours of debugging, a frustrated user, and a loss of trust. If I had caught it earlier, before it hit production, the impact would have been minimal. The $0.03 per run for 131 tests is a trivial cost compared to the engineering time spent debugging production issues or the potential revenue loss from a malfunctioning agent.
My initial investment in building the harness took about two weeks. This included designing the test layers, setting up the database, and writing the evaluation logic. It was time taken away from shipping new features, but it was absolutely necessary. It's a foundational piece of infrastructure for any serious AI agent development. Without it, you're shipping blind, hoping your probabilistic systems don't drift into silent failure.
Frequently Asked Questions
Q: How do you handle non-deterministic LLM outputs in your evaluations?
A: For non-deterministic outputs like summaries or creative text, I use a reference LLM (Claude 3.5 Sonnet) to evaluate the quality and accuracy against a set of human-defined criteria, rather than exact string matching. The prompt for the evaluation LLM is carefully crafted to focus on semantic correctness and completeness.
Q: What if the evaluation LLM itself makes a mistake?
A: This is a valid concern. I mitigate this by using a highly capable LLM (Claude 3.5 Sonnet) for evaluation and by having a human review a sample of the evaluation LLM's judgments, especially for new test cases or when an agent fails unexpectedly. The evaluation prompt is also designed to be explicit and minimize ambiguity.
Q: How do you manage the test data for 131 tests? Is it all manual?
A: Initially, many test cases were manual. However, I've started automating test case generation for variations (e.g., paraphrasing tools, data augmentation). For critical business logic, human-curated test cases remain essential. The PostgreSQL database allows for easy querying and updating of test cases.
Q: Why not just use an existing eval framework?
A: I found existing frameworks either too generic (requiring significant customization for multi-agent, tool-using systems) or too opinionated towards specific LLM providers or use cases. Building my own allowed for deep integration with my specific agent architecture, Oracle Cloud infrastructure, and custom evaluation logic tailored to my agents' unique failure modes.
Q: How do you keep the $0.03/run cost low with 131 tests?
A: The key is intelligent routing. Most initial agent interactions and simple checks use Groq, which is extremely fast and cost-effective. Only complex semantic evaluations (e.g., Layer 2 content accuracy) are routed to more expensive, but more capable, models like Claude 3.5 Sonnet. Many tests also involve local schema validation or mocked tool calls, incurring minimal LLM cost.
Top comments (0)