DEV Community: Gregory Harris

I Freaking Love Breaking My Own Software

Gregory Harris — Sun, 30 Aug 2026 20:23:23 +0000

Validation testing is one of my favorite parts of the application lifecycle.

I love, I mean I freaking love hammering away at code in search of bugs. There's a strange rush that comes from finding something you didn't expect, figuring out why it happened, fixing it, and then trying to break it all over again.

I probably spend too much time in the evenings and weekends making apps and hammering away at them.

I think the obsession started with a job I had as an application analyst.

We had implemented a new PACS system, and users were reporting performance problems. They knew something wasn't right, but all they had was anecdotal evidence.

The system had no built-in timers. Apparently, the makers simply hadn't thought that would be a thing we'd need.

So I became the timer.

For hours every day, broken into multiple sessions at different times of day and on different days of the week, I sat like a stone.

One hand on the mouse. The other hovering above a stopwatch.

Select an image. Start the stopwatch. Wait for it to load. Stop. Record.

Again.

And again.

I shifted the testing pattern by one day each week over the course of months. I was trying to document what the users could only describe through experience.

Before moving into application support, I'd spent a dozen years as a clinician. Maybe that had something to do with the sense of ownership I felt when I was essentially asked:

Is there actually a performance problem? And if there is, can you find the pattern?

I found the performance degradation.

I didn't find the pattern.

And that second result was just as important.

There was a measurable problem. But the evidence didn't support pretending we knew why or when it would happen.

Those days are long gone, but the rush I get when I break an application before an end user does isn't.

I learned firsthand what happens when software reaches users with problems that should have been found earlier.

So now I hammer away at everything I release. Do I find every bug in every possible real-world scenario?

Of course not.

Try as I may, there's always another edge case.

What has changed dramatically is the number of tools I have available to look for those edge cases, including AI.

Today I ask models from two different companies to come up with scenarios I missed. I can take those scenarios, turn them into controlled trials, run them against an application, document the results, make changes, and then run the tests again.

I'm happy to do it.

In fact, at this point, if I ship without AI assisting in validation, I feel like I've shipped a product without using all the tools at my disposal.

But there's an important distinction there.

I'm not asking an AI model:

"Hey, is my application good?"

That's not validation.

I want it helping me find ways the application could fail.

Give me the ugly inputs. The weird sequence of events. The thing I didn't think a user would ever do. Challenge the assumptions I made while building it.

Then test what actually happens.

That philosophy became especially important as I started building software for AI agents.

An ordinary application can fail because of a bug in the application.

An agent system has another problem: sometimes every part appear to have worked as designed.

The API responded, the call returned successfully. The agent got its answer.

Nothing crashed.

And yet the result still shouldn't be trusted.

That question eventually became interesting enough that I built an application specifically around it: ReliAgent.

ReliAgent sits after an agent's tool call and examines the call, its parameters, response and metadata for reliability problems before the result continues downstream.

But building the tool created another problem for me.

How do I know the reliability tool is reliable?

So the same instinct that once had me sitting in front of a PACS workstation with a stopwatch eventually turned into thousands of controlled trials against the software I was building.

The tools have changed considerably.

The obsession hasn't.

I still want to break it before you do.

Author's note: I built ReliAgent through HDGForge. The validation work referenced above is documented publicly in the Basanos research project.

I Ran 4,200 Trials Testing LLM Agent Reliability. Here’s What Broke.

Gregory Harris — Sat, 15 Aug 2026 01:11:38 +0000

We know when an AI agent gets a response from a tool, getting a response back doesn’t necessarily mean that response should be trusted.

It can lose context, become repetitive, grow less confident, fill gaps with agreeable language, or return something that looks usable while creating problems downstream.

I built ReliAgent to look for reliability signals like these in agent tool calls.

Then I built Basanos, a validation program designed to test it adversarially.

Across Basanos-2 and Basanos-3, I ran 4,200 trials. Some of the most useful results weren’t the successful ones.

The benchmark found real problems

I didn’t want Basanos to be a demonstration designed to produce good numbers.

Testing uncovered genuine product issues that were corrected before publication. Later experiments also exposed problems in the experimental design itself and required corrective runs.

Those corrections are part of the research record.

Then I changed the models

Basanos-2 consisted of 1,800 trials, but it had an important limitation: it used a single model, Anthropic’s Sonnet 5.

So Basanos-3 added 2,400 trials across 12 experiments using three additional model families:

GPT-5.6 Luna
GPT-5.6 Terra
Claude Haiku 4.5

That’s when I ran into an interesting problem with how benchmark results can be interpreted.

One sycophancy experiment produced a much lower observed detection rate on Claude Haiku 4.5 than on the other models.

At first glance, that looked like detector failure.

But when I inspected the responses, Haiku frequently refused to behave sycophantically in the first place. It pushed back rather than agreeing with the premise presented to it.

There wasn’t always a failure behavior for the detector to find.

That distinction matters.

If you’re testing a system that detects model behavior, there are really two questions:

Did the model produce the failure condition?

If it did, did the detector recognize it?

I learned that treating those as the same question can produce a misleading benchmark result.

Some results barely changed across models

The language-dependent detectors showed variation between models.

The two metadata-driven detectors evaluated in Basanos-3 — confidence collapse and context degradation — behaved differently. Both produced 1.00 TPR and 0.00 FPR across all three tested model families.

Cross-provider testing also surfaced output differences and a configuration dependency that needed to be accounted for in deployment.

Those are findings I wouldn’t have gotten from testing one model alone.

What 4,200 trials changed for me

A few principles came out of the work that I’ll carry into future benchmarks:

Test mechanisms, not just metrics. Know what condition you’re actually creating before counting whether something detected it.

Keep clean controls. A system that catches everything by flagging everything isn’t useful.

Separate model behavior from detector behavior. A model failing to exhibit an expected failure mode isn’t automatically a detector false negative.

Test across model families. A single-model benchmark tells you what happened in that environment, not necessarily what generalizes.

Keep the mistakes. Superseded experiments and corrective runs are part of the research record too.

And perhaps the biggest lesson:

Design benchmarks you’re willing to lose.

If the only acceptable outcome is proving your product works, you’re not really testing it.

If you want to dig into the results

Basanos is the validation program for ReliAgent, which I’ve been developing through HDGForge.

I’ve made the study reports and non-reconstructive results public for anyone interested in looking more closely at the methodology, results, corrections, and limitations.

Basanos research

ReliAgent

HDGForge

I’ll share more from the experiments as I continue the work.

Disclosure: I used AI assistance to help edit this article. The research, experiments, results, and conclusions described here are my own work.