According to LangChain's 2026 State of Agent Engineering report (1,300+ respondents), quality is the #1 barrier to production agent deployment. 32% of teams cite it as their primary blocker.
And yet, only 52% of teams have any evaluation system in place.
This is the testing gap. Agents are non-deterministic, multi-step systems that make traditional unit testing nearly useless. But that doesn't mean we can't test them at all.
What Can Be Tested Deterministically?
Before reaching for LLM-as-judge (expensive, non-deterministic), there's a surprising amount you can verify with plain assertions:
1. Tool Call Correctness
Did the agent call the right tools? In the right order? With the right arguments?
from agent_eval import Trace, assert_tool_called, assert_tool_call_order
trace = Trace.from_jsonl("weather_agent_run.jsonl")
assert_tool_called(trace, "get_weather", args={"city": "SF"})
assert_tool_not_called(trace, "delete_user") # safety check
assert_tool_call_order(trace, ["search", "read", "summarize"])
This catches a huge class of regressions: prompt changes that make the agent forget to use a tool, or use tools in the wrong order.
2. Loop Detection
Agents love getting stuck. Same tool, same args, over and over.
assert_no_loop(trace, max_repeats=3) # fail if any tool called 3+ times consecutively
assert_max_steps(trace, 10) # budget control
3. Output Sanity
assert_final_answer_contains(trace, "San Francisco")
assert_final_answer_matches(trace, r"\\d+°F") # must include temperature
assert_no_empty_response(trace)
assert_no_repetition(trace, threshold=0.85) # no copy-paste answers
4. Regression Detection
The killer feature for CI/CD: compare a baseline trace against a new one.
from agent_eval import diff_traces
baseline = Trace.from_jsonl("baseline.jsonl")
current = Trace.from_jsonl("current.jsonl")
diff = diff_traces(baseline, current)
print(diff.summary())
# ❌ Tool removed: get_weather
# 🐢 Latency increased: 800ms → 5000ms (6.3x)
# 📝 Final answer changed (similarity: 42%)
assert not diff.is_regression # fails if tools removed or latency >2x
Run this in CI after every prompt/model change. Catch regressions before they hit production.
The Three-Layer Testing Pyramid
I think about agent testing in three layers:
Layer 1: Deterministic assertions (fast, free, reliable)
- Tool calls, control flow, output patterns, performance bounds
- Zero API calls, zero cost, millisecond execution
- This is where 80% of your test value comes from
Layer 2: Statistical metrics (fast, free, approximate)
- Similarity scoring, drift detection, efficiency metrics
- Still no API calls, runs locally
Layer 3: LLM-as-Judge (slow, costly, powerful)
- Hallucination detection, goal completion, reasoning quality
- Use sparingly — for things that can't be checked deterministically
Most teams jump straight to Layer 3. That's like writing only integration tests and no unit tests. Start from the bottom.
The Tooling Gap
Current options:
- LangSmith/Braintrust/Arize — enterprise SaaS, great but heavy
- DeepEval — open source but 40+ dependencies (includes PyTorch)
- agentevals — needs OpenAI API for every evaluation
- promptfoo — Node.js, not Python
I built agent-eval to fill the gap: zero dependencies, local-first, framework-agnostic. Layer 1 and 2 assertions that run anywhere Python runs, with no API keys, no accounts, no uploads.
pip install agent-eval-lite
The data is clear: quality kills agent deployments. The fix isn't more powerful models — it's better testing. Start with deterministic assertions. You'll be surprised how much they catch.
References:
Top comments (0)