The people who know your AI agent best may be the people least able to see all of its flaws.
Not because they are bad engineers.
Because they built it.
Years ago, when I was taking art classes, my teacher told me something I've never forgotten:
“Sara, you can't judge your own art.”
I remember thinking, of course I can. 😂
Then she explained.
After spending hours looking at the same piece, your eyes get filled with it. You stop seeing what is actually there. You see what you expect to see.
I've used that lesson everywhere since.
And I think AI agents have the same problem.
You designed the requirements.
You designed the system.
You know why every decision was made.
Then you design the evaluation and ask:
“Does my agent actually work?”
That's where the blind spot can appear.
Your evaluation may end up testing the system according to the same assumptions that created it.
The evaluator can inherit the system's assumptions
Consider a simple requirement:
“The agent should answer customer questions accurately.”
Seems reasonable.
So the team creates an evaluation set with questions that have clear intent and well-defined answers.
The agent performs beautifully.
94%.
Green dashboard. 🎉
But an external evaluator might ask a different question:
What happens when the customer's request has two plausible interpretations?
Now you have a different test:
“Can I change my billing address?”
Does the agent answer immediately?
Does it ask which account or address the customer means?
Does it make an assumption?
The original evaluation may have been technically correct.
It just never tested the ambiguity.
That is the blind spot.
Internal evaluation is still essential
This isn't an argument that internal teams shouldn't evaluate their own systems.
They absolutely should.
The people who built the system understand its requirements, architecture, constraints, tools, and intended behavior better than anyone.
That knowledge is extremely valuable when designing evaluations.
But it can also create an invisible constraint:
You know what the system is supposed to do, so you naturally tend to test within the boundaries you already understand.
An independent evaluator brings a different mental model.
Not necessarily better technical knowledge.
A different set of assumptions.
Someone who can ask:
- What did we assume here?
- What happens if the requirement is ambiguous?
- What happens at the edge?
- What did we forget to test?
- Which behaviors are we treating as acceptable without actually defining why?
- What if the system is doing exactly what we designed, but what we designed was wrong?
That last question is the uncomfortable one.
The dangerous evaluation is the one that confirms everything you already believe
An evaluation isn't only a collection of tests.
It is also a model of what “working” means.
If the same people define the requirements, design the system, choose the test cases, define the rubric, and interpret the results, there is a risk that the entire evaluation inherits the same assumptions.
Everything can look internally consistent.
And still be wrong.
This is why I think evaluation independence deserves more attention as AI agents become more capable.
You don't necessarily need an external evaluator for every test.
But you do need some mechanism that is independent of the assumptions being evaluated.
That could mean an external evaluator.
It could mean a separate team.
It could mean adversarial test design.
It could mean deliberately asking someone unfamiliar with the implementation to construct edge cases.
The mechanism can vary.
The principle doesn't:
The evaluation should be capable of challenging the assumptions behind the system, not merely confirming that the system behaves according to them.
Sometimes the hardest failure to find is the one everyone involved has learned not to see.
Internal testing is necessary.
But sometimes, you need someone who hasn't spent hours staring at the same painting.
Top comments (0)