When building AI agents, it is easy to test randomly.
You change a prompt.
You run the agent once.
The answer looks better.
So you think the system improved.
But LLM-based systems do not work that simply.
A change that improves one example can quietly break another one.
That is why agent evaluation needs structure.
Instead of asking:
Does this one output look better?
A better question is:
Did this change improve the agent across the same set of test cases?
That is the idea behind evaluation-driven development.
A simple evaluation loop
A basic evaluation loop looks like this:
Curate dataset
→ make one change
→ run experiment
→ evaluate outputs
→ compare results
→ fix regressions
The goal is to stop relying only on vibes.
You want to know what changed, what improved, and what broke.
Step 1: Curate a dataset
Start with a small dataset of test cases.
It does not need to be huge.
Even 10 to 25 strong examples can be useful at the start.
For an agent, a test case can include:
- user input
- expected tool
- expected parameters
- expected output
- edge cases
- failure cases
Example:
Input:
"Can you check the status of order #1234?"
Expected tool:
order_status_check
Expected parameter:
order_number = 1234
This gives you a stable baseline.
Without a dataset, every test becomes subjective.
Step 2: Track each change as an experiment
Once you have test cases, treat every change like an experiment.
You may change:
the system prompt
the router prompt
tool descriptions
model
memory strategy
retrieval logic
skill structure
Then run the same dataset again.
This helps answer:
Did the change actually improve the agent?
Not just:
Did one output look better?
Step 3: Use the right evaluator
Different parts of the agent need different evaluators.
Code-based evals
Code-based evals are useful when the rule is clear.
Examples:
is the JSON valid?
did the agent call the correct tool?
did it extract the correct parameter?
did the SQL stay read-only?
did generated code run?
did the output match the expected value?
These are good because they are more deterministic.
If code can check it reliably, use code first.
*LLM-as-a-judge: *
LLM-as-a-judge is useful when the output needs quality judgment.
Examples:
is the summary clear?
is the answer grounded?
did the response hallucinate?
did it answer the actual question?
did it identify the right entity?
is the final answer useful to the user?
This helps when simple rules are not enough.
But LLM judges are not perfect. They can also be wrong.
So use clear labels when possible:
correct / incorrect
grounded / not grounded
useful / not useful
safe / unsafe
Avoid vague scores when you do not need them.
Human evaluation
Human evaluation still matters when the task is:
high-stakes
subjective
domain-specific
safety-sensitive
tied to user trust
Sometimes the best signal is still a human reviewer or user feedback.
Human labels can also become part of your future eval dataset.
Step 4: Look for regressions
One of the most useful things evals can catch is regression.
A regression means something that worked before is now broken.
Example:
You improve the final response style.
But now the router chooses the wrong tool more often.
Without evals, you may only notice the better writing.
With evals, you can say:
Response quality improved.
Router accuracy dropped on 3 test cases.
That is useful.
It tells you what to fix next.
Step 5: Evaluate your evaluator too
This part is easy to miss.
Your evaluator also needs evaluation.
For example, suppose you use LLM-as-a-judge to check whether the router chose the correct tool.
If you already have ground truth for the correct tool, you can compare the LLM judge against that.
That helps answer:
Is my LLM judge actually reliable?
For clear cases, code-based evals can act as a reference point.
Then you can check if the LLM judge agrees.
This matters because LLM-as-a-judge is useful, but it is not 100% correct.
What a good experiment should show
A good eval result should show more than one score.
It should show:
what changed
which dataset was used
which evaluator was used
which test cases passed
which test cases failed
where regressions happened
what to fix next
Example:
Experiment:
Updated router prompt
Result:
Tool choice improved from 70% to 85%
Regression:
Parameter extraction failed on 3 older examples
This is much better than saying:
The agent feels better.
Why this matters
Agents can fail in many small ways.
A prompt update may improve answer quality but break tool choice.
A model change may improve reasoning but increase latency.
A new tool description may reduce hallucination but make parameter extraction worse.
Without structured experiments, these changes are hard to track.
With structured evals, you can understand the failure more clearly.
Instead of saying:
The agent is bad.
You can say:
The router chose the wrong tool.
or:
The skill generated invalid SQL.
or:
The summary was not grounded.
That is where evals become useful.
They turn vague feedback into specific fixes.
Final thought
Evaluation-driven development means treating agent changes like experiments.
Create test cases.
Make one change.
Run the same dataset.
Use the right evaluator.
Compare results.
Catch regressions.
Then improve with evidence.
Try this before your next agent change:
Save 10 test prompts.
Run them before and after the change.
Track tool choice, final answer quality, step count, and failures.
Small eval sets are better than vibes.
Top comments (0)