Most engineering teams are building agent eval the way they built QA — pass/fail checks, CI gates, a green badge. That model is structurally wrong for agents. Agent failures don't come from the input distribution your tests cover. They come from the adversarial distribution your tests don't.
The right mental model is the security audit: rotational, adversarial, owned by people whose job is to find what breaks rather than to confirm what works.
Here is what changes when you accept that.
What everyone gets wrong
Open the docs of any popular agent eval framework — Promptfoo, DeepEval, LangSmith, Confident AI. The shape is the same.
A YAML of test cases. A runner that produces pass/fail counts. A CI integration that surfaces a green check. The framing is borrowed wholesale from unit testing: declare expected behavior, assert reality matches, gate the deploy. Vendor copy reads "test your LLM application like any other software."
It isn't like any other software.
The premise of unit testing is that the input distribution is stable and the failure modes are knowable in advance. Both premises break for agents. Inputs are arbitrary natural language, arbitrary fetched web pages, arbitrary tool outputs. Failure modes — prompt injection, tool exfiltration, context-window poisoning, multi-step misuse — have all been discovered after deployment, by adversaries, not by test authors.
The other popular view is to outsource the question. "The model card says it's safe." That is a category error. A frontier-model eval tells you whether the model produces unsafe outputs in the lab's harness. It does not tell you whether your agent, with your tools, against your data sources, in your threat model, is safe.
The third version is the audit-as-a-checkpoint mindset. Hire a red-team firm. One-week engagement. PDF in, file the PDF, ship. This is closer to the right idea but compresses a continuous practice into a discrete event. Agents drift. Inputs drift. Tools drift. A point-in-time audit ages the moment it is filed.
The reframe
Treat agent evaluation as you would treat a security program for a high-value system. The differences are not cosmetic — they cascade.
Test sets are static; adversarial inputs evolve. A regression suite measures whether your agent still does what it did last week on a fixed set of inputs. That is a stability measurement, not a safety one. Stability is necessary; it is not sufficient. The OWASP LLM Top 10 v2 publishes ten attack categories — none of them are detected by a regression suite that only checks task success.
Pass rates hide tail risk. A 99 % safe agent fails 1 % of the time. For QA, the question is whether 1 % is tolerable. For security, the question is which 1 %. A 99 % task-success rate that includes 1 % "leaks customer data when asked nicely in a base64-encoded prompt" is not a 99-grade agent. It is unshippable.
Reporting agent reliability as a single percentage is the same category error as reporting a web app's security posture as "97 % of unit tests pass." The right shape is per-threat-class: prompt-injection success rate, tool-misuse rate, exfiltration rate, capability-escape rate. Each gets its own threshold.
Evaluation is a clock, not a CI gate. CI gates assume the system under test changes and the test set is fixed. For agents, the test set is the part that should change.
In our work running ATHelper agents in production across two quarterly red-team rotations, the pattern was consistent: regression coverage stayed flat between rotations, and each new rotation surfaced 3-5 issues the regression suite would never have found — because regression tests known scenarios while rotation tests adversarial ones. The cost of running both was roughly 1.4× the cost of running regression alone. That is far below the cost of a single production prompt-injection incident.
Cadence matters more than depth. A thin monthly rotation outperforms a deep annual audit because drift compounds.
Ownership decides incentive. If the eval team reports into engineering productivity, they optimize for ship velocity — coverage becomes a number to grow, false positives become a number to shrink, the implicit goal is keeping the green light on. If they report into security or risk, they optimize for catching what slipped.
The same headcount, the same tools, the same eval suite, different reporting line — different findings. This is not a hypothesis. It is the same dynamic that moved AppSec teams out from under engineering productivity at most mature software companies a decade ago.
What this means for CTO / VP Eng / Head of AI
Four moves, in priority order, for next quarter's roadmap.
- Move the eval owner's reporting line.
Whoever is accountable for agent eval should report through security, risk, or a dedicated AI safety function — not through eng productivity, platform, or DX. The headcount can stay where it is for execution; the reporting line is what shifts incentive. If you don't have a security-aligned home for AI eval yet, this is the higher-leverage org change in 2026 than any tooling decision.
- Replace the CI eval gate with a release-bound red-team rotation.
Keep your existing eval framework running on every commit for regression — that is still useful. But add a separate gate: no agent capability ships to production until it has cleared a red-team rotation against the current adversarial probe set. Rotations run on a fixed cadence (every 2-4 weeks), not on demand, so they cannot be skipped under deadline pressure. The rotation produces a written report; the report goes to the eval owner's reporting line, not to engineering management.
- Reclassify eval failures as incidents.
A regression test failure goes to the engineer who wrote the code. A red-team finding goes to the incident response process — same severity classification, same SLA, same postmortem expectation as a production security incident. This sounds heavy. It is the right weight. Treating an agent prompt-injection finding as "a test that needs fixing" is what produces the kind of "we knew about it for six months" disclosure that ends careers.
- Convert one-time audit spend into recurring red-team capacity.
If your 2026 budget contains a line item for "AI security audit, one-time, $40-80K," redirect it to either recurring vendor red-team capacity at roughly the same annual spend, or headcount for an internal AI red-team function if your scale supports it. The audit produces a snapshot. The recurring capacity produces a function. You need the function.
What I'm not saying
I'm not saying QA is irrelevant for agents. Task success, step accuracy, tool-call accuracy, recovery rate — all matter. The argument is that those numbers, by themselves, do not answer "is this safe to ship."
I'm not saying every team needs a dedicated AI red team. The argument is about reporting line and incentive, not headcount. A single eval owner reporting into security is meaningfully different from the same person reporting into eng productivity.
I'm not saying you can outsource this. External red-team firms don't know your domain, your data, your tool surface, or your threat model. They are useful for periodic external validation, the same way external pen-testers are. They are not a substitute for an internal function.
I'm not saying current eval frameworks are useless. DeepEval, Promptfoo, garak, LangSmith are necessary infrastructure. They are not sufficient on their own, the same way unit-test frameworks are not sufficient on their own to constitute a software security program.
The shift is not which tools you use. It is what category of work you think you are doing.
If this resonated with how you're thinking about agent reliability — or if it sharpened a disagreement worth pushing back on — I'd genuinely like to hear it in the comments.
Veyon Solutions runs ATHelper, a reliability and security platform for AI agents. The full version of this argument, with references to OWASP LLM Top 10 v2, NIST AI RMF, MITRE ATLAS, and the eval frameworks named above, lives at https://www.at-helper.com/blog.
Top comments (0)