Your AI Coding Agent Passed the Tests. Did the Tests Actually Run?

Cecelia Turnbeau — Tue, 20 Oct 2026 22:28:42 +0000

Your AI Coding Agent Passed the Tests. Did the Tests Actually Run?

A coding agent finishes a task, runs pytest, and reports success.

Exit code: 0.

Everything is green.

But here's a question I've become increasingly interested in:

What exactly did that green result prove?

Because there's a difference between a test suite passing and the intended verification actually taking place.

And that distinction is becoming increasingly important as we delegate more software development to autonomous agents.

I tried changing the evidence, not just the code

While investigating coding-agent verification, I ran a small controlled experiment against a repository after a clean test run.

I introduced five changes to see which ones the test suite would catch.

Four of the five left the suite passing.

Among the changes I examined:

A weakened assertion in a test file.
A new conftest.py at the repository root.
A modified pytest.ini that altered test collection.
An implementation change that genuinely broke behavior.

The implementation failure turned the suite red.

But several changes affecting what the tests asserted or executed still produced successful exit codes.

That experiment wasn't a measurement of how frequently coding agents behave this way. These were deliberate changes designed to probe the verification boundary.

What interested me was something more fundamental.

A successful test command is not automatically evidence that the intended test obligations were satisfied.

The test files aren't the entire verification system

Developers often protect their acceptance tests from modification.

That's sensible.

But with pytest, the files in tests/ aren't the only things influencing what executes.

Configuration and collection behavior matter too.

For example, a repository-level conftest.py can participate in pytest's collection process. A pytest.ini can alter discovery settings. Plugins and environment configuration can also influence execution.

Depending on the setup, those mechanisms can change what gets evaluated without modifying a protected test assertion.

The important distinction is between:

Protecting the tests

and

Protecting the authority that determines which tests execute.

Those aren't necessarily equivalent.

And even protecting collection configuration isn't a complete solution. The interpreter, dependencies, environment variables, and loaded plugins can introduce additional assumptions.

Why an exit code isn't enough

An exit code of zero tells you that the command completed successfully according to the rules of that execution.

It doesn't, by itself, establish:

That the expected tests were collected.
That required tests actually executed rather than being skipped.
That the expected assertions remained intact.
That the execution environment matched the approved baseline.
That the tests were sufficient to establish the requested behavior.

Those are separate questions.

And some require different kinds of evidence.

For example, comparing collected test identifiers may catch missing tests, but identifiers alone won't reveal that a required test was skipped.

Similarly, independently rerunning an unchanged test suite doesn't establish that its assertions represent the correct product requirements.

Execution integrity and test adequacy are different problems.

We shouldn't confuse them.

Independent verification has its own trust boundary

A common response is to rerun the tests somewhere else.

That's often the stronger approach.

An independent CI runner or clean environment that the coding agent cannot modify can remove important opportunities for interference.

But the boundary still needs to be defined carefully.

Which repository revision gets evaluated?

Which test definitions are authoritative?

Can the agent change the configuration governing collection?

Which dependencies and plugins are trusted?

What happens when a legitimate task actually requires modifying the tests?

Simply moving execution into another process doesn't answer every question. What matters is whether the verification authority is genuinely independent of the work being judged.

And no matter how well that boundary is constructed, passing tests remain limited by what those tests were designed to detect.

There's also a human problem

The more I've explored this, the more interested I've become in what happens after a verification system refuses to clear a change.

Because a gate that catches interference but repeatedly demands unnecessary human investigation can create another bottleneck.

Consider two situations.

Situation A: A required test was silently removed from execution.

Situation B: A legitimate, approved configuration update changed how tests are collected.

Both may require withholding clearance until the verification basis is understood.

But the developer shouldn't have to spend twenty minutes reconstructing the entire run to determine which situation occurred.

The useful output isn't merely a red light.

It's actionable evidence: what changed, what executed, what couldn't be established, and what requires approval.

And this creates a second question worth measuring:

Can independent verification reduce the human effort needed to accept agent-generated work, without weakening the checks that justify acceptance?

That's a harder question than detecting one successful test-tampering example.

It's also arguably the more commercially important one.

What I'm building around this question

I'm Cecelia Turnbeau, an attorney by professional background and the founder of ClaraGate.

I've spent the past several months developing an independent verification approach for autonomous coding-agent workflows.

The current alpha is deliberately narrow: it targets a defined OpenCode and Python/pytest workflow, independently reruns a designated test command using a declared interpreter, and checks a frozen verification surface.

It does not guarantee execution in a clean, agent-inaccessible clone. It does not certify test adequacy or prove code correctness.

Those boundaries are important.

The larger idea I'm exploring is that an agent's declaration of completion and the authority to accept its work should be separate things.

But an architectural argument alone isn't sufficient to establish product value.

That has to come from actual developer use.

What I want to hear from developers

I'm particularly interested in people using coding agents on real repositories, especially Python/pytest projects.

Not what an ideal verification system should do.

What you're actually doing today.

When your coding agent reports that a task is complete:

Do you independently rerun its tests, inspect the diff, rely on CI, or use another reviewer?
Have you encountered a passing result where the wrong tests ran, tests were weakened, or important behavior wasn't checked?
What part of verifying agent-generated work still consumes the most human attention?

And perhaps the most interesting question:

If an independent verifier could establish that the designated checks ran against an intact verification basis, would that actually change your review workflow, or would it duplicate evidence you already trust?

I'm interested in both answers.

If your existing setup already solves this effectively, I'd genuinely like to understand how.

If it doesn't, I'd like to understand where the remaining burden lives.

Because the goal shouldn't be to introduce another gate simply because we can.

The goal should be to make autonomous work more reliably acceptable, with evidence that justifies the decision.

I'm documenting this investigation through ClaraGate Insights. I'm particularly interested in candid technical criticism and firsthand development workflows.

DEV Community: Cecelia Turnbeau

Your AI Coding Agent Passed the Tests. Did the Tests Actually Run?

Your AI Coding Agent Passed the Tests. Did the Tests Actually Run?

I tried changing the evidence, not just the code

The test files aren't the entire verification system

Why an exit code isn't enough

Independent verification has its own trust boundary

There's also a human problem

What I'm building around this question

What I want to hear from developers