DEV Community

Viktoria
Viktoria

Posted on Edited on

Stop Treating Green Tests as Proof That Your AI Agent Wrote the Right Code

Your AI agent wrote the code, wrote the tests, and reported a green build. What evidence came from outside its own interpretation?

A green pipeline tells you that the implementation agrees with the tests. It does not tell you that either one agrees with the requirement.

This distinction matters when the same agent reads the task, writes the implementation, chooses the expected values, and runs the suite. One interpretation drives both the code and the evidence used to approve it. If that interpretation is wrong, the pull request can be coherent, polished, and green.

At Explyt, we build an AI agent for JetBrains IDEs. The agent can inspect project structure, resolved usages, dependencies, inspections, tests, run configurations, coverage, and debugger data. Those facts help verify what the project does. They do not replace an independent definition of what the project should do.

TL;DR
A passing test proves that one observed result matched one expectation for one set of inputs.
A test written from the same mistaken interpretation as the implementation can confirm the mistake.
A defect regression test should fail on the old implementation and pass after the fix.
New business logic needs acceptance criteria that exist outside the generated code.
IDE, build, and runtime evidence can challenge the agent's account of the project, but a human still owns the business decision.
One wrong assumption can pass every check
Consider this pricing rule:

Apply a loyalty discount only to items that do not already have a promotion.

The agent misreads the rule and discounts the whole order. Then it writes this test:

@test
void appliesLoyaltyDiscount() {
var order = orderWithRegularAndPromotionalItems();

var total = pricingService.calculate(order, loyalCustomer());

assertThat(total).isEqualTo(new BigDecimal("810.00"));
Enter fullscreen mode Exit fullscreen mode

}
The implementation and 810.00 come from the same flawed interpretation.

The code is wrong. The test expects the wrong result. CI confirms that they agree. Nothing in that loop checks the business rule.

A passing test therefore has a narrower meaning than teams often give it. It proves that one observed result matched one encoded expectation for one set of inputs. It does not prove that the expectation came from the right source.

Make the test capable of rejecting the code
For a defect fix, run the new regression test against the old implementation:

Disable or revert the fix.
Run the new test.
Confirm that it fails because the original defect is present.
Restore the fix.
Run the test again and confirm that it passes.
If the test stays green on the defective version, it does not provide evidence for the fix. It may hit another branch, use the wrong fixture, or assert an unrelated side effect.

For new functionality, introduce a controlled violation. Reverse the condition, remove the boundary check, or return the wrong status. The relevant test should fail.

Coverage does not settle this. A line can execute while the assertions remain blind to the behavior that matters.

Before reading the implementation, ask:

What specific wrong implementation would this test reject?

Then check where the expected result came from.

not null, a non-empty collection, and no exception are weak signals when the requirement concerns pricing, permissions, state transitions, or data integrity. Mocks need the same pressure. If the agent programs a mock to return the response its implementation expects, the test may validate an invented dependency contract.

Expected values need scrutiny too. Copying the production formula into the test gives you two copies of one error.

Separate confirmation from verification
Confirmation starts with the generated solution:

Does it compile?
Do its tests pass?
Is the diff internally consistent?
Verification starts outside the generated solution:

What behavior was required?
What failure should be observable?
What project evidence can contradict the agent's claim?
Your team needs both modes.

Start with acceptance criteria written independently from the implementation plan. "Handle duplicate requests correctly" leaves the agent room to define its own success. "A repeated request with the same idempotency key must not create a second operation" defines an observable result.

Keep the change small enough to review. If one run changes the domain model, migration, API, error handling, and dozens of tests, the reviewer must reconstruct the entire task before checking a single assumption.

Ask for counterevidence, not another summary:

Find an input that violates the criterion.
Identify a changed branch the tests do not exercise.
Compare mocks with the installed dependency contract.
Propose a minimal mutation each test must detect.
A second review agent can help, but changing the model does not create independence by itself. If both agents receive the same wrong expected behavior, the second can repeat the first error with different wording. Independent criteria matter more than independent prose.

Let the project challenge the model
A language model produces a plausible account of the codebase. The project contains the facts.

Compile the affected modules. Run the relevant unit, integration, and contract tests. Use IDE inspections. Resolve actual symbol usages, implementations, overrides, dependency versions, and available APIs. Inspect coverage of changed branches when the risk justifies it.

This is where Explyt changes the verification model. Explyt grounds agent work in evidence available through JetBrains IDEs: code structure, resolved usages, dependencies, inspections, tests, coverage, run configurations, and debugger data.

The agent can test a claim against the project instead of relying only on text search, generated explanations, or long terminal logs.

Project evidence still has a boundary:

The IDE can show which implementation is called. It cannot decide which business rule the company intended.
The debugger can show the actual transaction state. It cannot decide whether that state is acceptable.
The test runner can show that every selected test passed. It cannot prove that the selected tests encode the right behavior.
Tools establish what happened. Your team defines what should happen.

Match the evidence to the risk
Different changes require different evidence.

A local refactoring with no contract change may need a focused diff, IDE refactoring, inspections, existing tests, and human review.

A defect fix needs a regression test that fails on the previous implementation.

New business logic needs acceptance scenarios derived independently from the generated code.

A public API change needs contract and compatibility checks.

A data migration needs an isolated run on a realistic schema, invariant checks, and a tested rollback path.

Concurrent or distributed behavior needs repeatable scenarios and runtime observation. One successful run says little when event order can change the result.

When runtime state matters, reproduce the scenario in the application. Stop on the changed branch. Inspect the actual inputs and intermediate values. Follow the call stack. Compare the observed result with the acceptance criterion.

When the requirement is contradictory, the business rule has no owner, the test is unstable, or no safe environment exists, stop the automation. More generated code will not resolve a missing source of truth.

Before merging agent-written code
Your team should be able to show:

acceptance criteria that exist outside the implementation;
a test that fails on a concrete wrong version;
assertions tied to required outcomes;
mocks checked against real dependency contracts;
relevant boundary and failure paths;
actual IDE, build, and runtime evidence;
explicit unverified risks;
a human who understands the diff and accepts the decision.
Green tests are valuable. They show that the code passed the checks you wrote.

They do not show that you wrote the right checks.

When your agent hands you a clean pull request, a polished report, and a green pipeline, ask one question:

What evidence in this review did not come from the agent's own interpretation?

Top comments (0)