We know when an AI agent gets a response from a tool, getting a response back doesn’t necessarily mean that response should be trusted.
It can lose context, become repetitive, grow less confident, fill gaps with agreeable language, or return something that looks usable while creating problems downstream.
I built ReliAgent to look for reliability signals like these in agent tool calls.
Then I built Basanos, a validation program designed to test it adversarially.
Across Basanos-2 and Basanos-3, I ran 4,200 trials. Some of the most useful results weren’t the successful ones.
The benchmark found real problems
I didn’t want Basanos to be a demonstration designed to produce good numbers.
Testing uncovered genuine product issues that were corrected before publication. Later experiments also exposed problems in the experimental design itself and required corrective runs.
Those corrections are part of the research record.
Then I changed the models
Basanos-2 consisted of 1,800 trials, but it had an important limitation: it used a single model, Anthropic’s Sonnet 5.
So Basanos-3 added 2,400 trials across 12 experiments using three additional model families:
- GPT-5.6 Luna
- GPT-5.6 Terra
- Claude Haiku 4.5
That’s when I ran into an interesting problem with how benchmark results can be interpreted.
One sycophancy experiment produced a much lower observed detection rate on Claude Haiku 4.5 than on the other models.
At first glance, that looked like detector failure.
But when I inspected the responses, Haiku frequently refused to behave sycophantically in the first place. It pushed back rather than agreeing with the premise presented to it.
There wasn’t always a failure behavior for the detector to find.
That distinction matters.
If you’re testing a system that detects model behavior, there are really two questions:
Did the model produce the failure condition?
If it did, did the detector recognize it?
I learned that treating those as the same question can produce a misleading benchmark result.
Some results barely changed across models
The language-dependent detectors showed variation between models.
The two metadata-driven detectors evaluated in Basanos-3 — confidence collapse and context degradation — behaved differently. Both produced 1.00 TPR and 0.00 FPR across all three tested model families.
Cross-provider testing also surfaced output differences and a configuration dependency that needed to be accounted for in deployment.
Those are findings I wouldn’t have gotten from testing one model alone.
What 4,200 trials changed for me
A few principles came out of the work that I’ll carry into future benchmarks:
Test mechanisms, not just metrics. Know what condition you’re actually creating before counting whether something detected it.
Keep clean controls. A system that catches everything by flagging everything isn’t useful.
Separate model behavior from detector behavior. A model failing to exhibit an expected failure mode isn’t automatically a detector false negative.
Test across model families. A single-model benchmark tells you what happened in that environment, not necessarily what generalizes.
Keep the mistakes. Superseded experiments and corrective runs are part of the research record too.
And perhaps the biggest lesson:
Design benchmarks you’re willing to lose.
If the only acceptable outcome is proving your product works, you’re not really testing it.
If you want to dig into the results
Basanos is the validation program for ReliAgent, which I’ve been developing through HDGForge.
I’ve made the study reports and non-reconstructive results public for anyone interested in looking more closely at the methodology, results, corrections, and limitations.
I’ll share more from the experiments as I continue the work.
Disclosure: I used AI assistance to help edit this article. The research, experiments, results, and conclusions described here are my own work.
Top comments (2)
The two-question distinction — did the model produce the failure, then did the detector catch it — is the part most eval harnesses quietly get wrong. We hit the exact same thing: a "detection rate" number collapses the moment a model refuses to produce the failure condition at all, so a safer model looks like a detector regression. The fix that worked for us was conditioning TPR on the subset of trials where the failure actually occurred, and reporting the base rate of the failure condition per model alongside it — otherwise the two are hopelessly entangled. Your result that the metadata-driven detectors (confidence collapse, context degradation) held 1.00/0.00 across all three families while the language-dependent ones drifted matches our experience too: behavioral/linguistic signals are model-coupled, structural ones travel. Do you track the per-model base rate of each failure condition explicitly, or infer it from the response inspection after the fact?
In Basanos 3, inferred after the fact. The per model base rate is reconstructed using response inspection of missed trials. Its expressed as a limitation in section 6 of the technical report. In future studies, a fix such as the one you used will be applied. Thank you for sharing your experience.