I Ran 4,200 Trials Testing LLM Agent Reliability. Here’s What Broke.

Gregory Harris — Sat, 15 Aug 2026 01:11:38 +0000

We know when an AI agent gets a response from a tool, getting a response back doesn’t necessarily mean that response should be trusted.

It can lose context, become repetitive, grow less confident, fill gaps with agreeable language, or return something that looks usable while creating problems downstream.

I built ReliAgent to look for reliability signals like these in agent tool calls.

Then I built Basanos, a validation program designed to test it adversarially.

Across Basanos-2 and Basanos-3, I ran 4,200 trials. Some of the most useful results weren’t the successful ones.

The benchmark found real problems

I didn’t want Basanos to be a demonstration designed to produce good numbers.

Testing uncovered genuine product issues that were corrected before publication. Later experiments also exposed problems in the experimental design itself and required corrective runs.

Those corrections are part of the research record.

Then I changed the models

Basanos-2 consisted of 1,800 trials, but it had an important limitation: it used a single model, Anthropic’s Sonnet 5.

So Basanos-3 added 2,400 trials across 12 experiments using three additional model families:

GPT-5.6 Luna
GPT-5.6 Terra
Claude Haiku 4.5

That’s when I ran into an interesting problem with how benchmark results can be interpreted.

One sycophancy experiment produced a much lower observed detection rate on Claude Haiku 4.5 than on the other models.

At first glance, that looked like detector failure.

But when I inspected the responses, Haiku frequently refused to behave sycophantically in the first place. It pushed back rather than agreeing with the premise presented to it.

There wasn’t always a failure behavior for the detector to find.

That distinction matters.

If you’re testing a system that detects model behavior, there are really two questions:

Did the model produce the failure condition?

If it did, did the detector recognize it?

I learned that treating those as the same question can produce a misleading benchmark result.

Some results barely changed across models

The language-dependent detectors showed variation between models.

The two metadata-driven detectors evaluated in Basanos-3 — confidence collapse and context degradation — behaved differently. Both produced 1.00 TPR and 0.00 FPR across all three tested model families.

Cross-provider testing also surfaced output differences and a configuration dependency that needed to be accounted for in deployment.

Those are findings I wouldn’t have gotten from testing one model alone.

What 4,200 trials changed for me

A few principles came out of the work that I’ll carry into future benchmarks:

Test mechanisms, not just metrics. Know what condition you’re actually creating before counting whether something detected it.

Keep clean controls. A system that catches everything by flagging everything isn’t useful.

Separate model behavior from detector behavior. A model failing to exhibit an expected failure mode isn’t automatically a detector false negative.

Test across model families. A single-model benchmark tells you what happened in that environment, not necessarily what generalizes.

Keep the mistakes. Superseded experiments and corrective runs are part of the research record too.

And perhaps the biggest lesson:

Design benchmarks you’re willing to lose.

If the only acceptable outcome is proving your product works, you’re not really testing it.

If you want to dig into the results

Basanos is the validation program for ReliAgent, which I’ve been developing through HDGForge.

I’ve made the study reports and non-reconstructive results public for anyone interested in looking more closely at the methodology, results, corrections, and limitations.

Basanos research

ReliAgent

HDGForge