A coding agent says, “Done.” The test suite is green. The diff looks plausible.
But which requirement did the tests actually verify?
This is a question I keep returning to while working on AI-assisted coding workflows. The risk is not only that an AI assistant generates incorrect code. It is that we can mistakenly turn a narrow observation—one command passed—into a much broader claim—the requested behavior is correct.
A green test can be completely irrelevant
Imagine the requirement is:
Reject invalid email addresses during signup.
The application currently contains:
def signup(email: str) -> bool:
return True
And the test suite contains:
def test_smoke():
assert 1 + 1 == 2
Running pytest -q will report success. It should: that assertion is true. But the test has not exercised signup, let alone invalid input.
Add a test that actually checks the requirement:
def test_signup_rejects_invalid_email():
assert signup("not-an-email") is False
That test fails against the current implementation. Nothing mysterious happened: the first green result was real, but the conclusion drawn from it was too strong.
This matters when a coding agent summarizes a run as “all tests pass” and we silently interpret that as “your requested feature works.” Those are different statements.
Five questions worth keeping separate
When reviewing AI-generated changes, I find it useful to separate at least five evidence roles:
| Role | Question | What it does not prove |
|---|---|---|
| Intent | What behavior did the user request? | That it was implemented |
| Change | Which files and revision changed? | That the change was correct |
| Verification | Which checks actually ran? | That they covered every requirement |
| Environment | Where did those checks run? | That another environment behaves identically |
| Boundary | What remains untested, stale, or unknown? | That nothing else can fail |
A coding-agent transcript can provide helpful context, but its statement that a task was completed is still a claim. A test result is an observation. A signed CI statement may authenticate provenance. None of these can replace the others.
Making an evidence gap visible
I've been experimenting with this idea in an open-source Python CLI called RAEE-Lite. Its narrow goal is to keep the requirement and its supporting evidence separate rather than automatically turning a passing test into a verified feature.
The current public source is a v0.10.0rc2 release candidate, available for review—not a stable or independently security-certified release. If you want to try it, inspect the source candidate and install it in a disposable Python environment.
Then, in a new temporary project directory:
raee-lite init
raee-lite add "Signup rejects invalid email addresses"
raee-lite expect --for R1 --kind unit
raee-lite status
raee-lite check
The intended result is that requirement R1 remains UNTESTED, with an evidence-gap report. raee-lite status intentionally exits with status 1 while the requirement lacks evidence; the default check operation doesn't execute your project's tests.
This is deliberately not an automatic proof of code correctness. RAEE-Lite does not semantically certify that a test covers an entire requirement. It makes the absence of documented evidence visible and leaves the judgment open to human review.
The second problem: stale evidence
Even a relevant test result should be tied to a particular source revision. If a file changes after a test run, the earlier result does not automatically apply to the new code.
The same distinction applies to environments. A check executed locally is not identical to a check executed on a protected CI branch, and a signed provenance statement authenticates who made a claim—not whether its test design was sufficient.
This is why I prefer reports that say “untested” or “stale” rather than projecting certainty they haven't earned.
What remains unverified
RAEE-Lite is still in early access. Although it has local test results and synthetic security regression cases, the release candidate has not yet completed the protected-main Linux/Windows/macOS acceptance matrix, independent GitHub Artifact Attestation verification, or third-party security review. It's also not a substitute for comprehensive test coverage, a security audit, or rigorous code review.
I'd value developer feedback on the evidence model itself: What do your AI coding tools claim after a green test? Which kinds of missing evidence would help you catch false confidence? A sanitized, minimal reproduction is more useful than a star or promotional repost.
The public project, including Chinese, English, German, and Spanish documentation, is at github.com/jinxinche/RAEE.
Research context: The idea of keeping evidence roles distinct was motivated by a separate oncology AI preprint: Hu et al. (2026), Restricted Multimodal Oncology AI: Role-Aware Evidence Organisation and Cross-Domain Transfer Boundaries, DOI:10.21203/rs.3.rs-10194336/v1. That preprint did not evaluate or validate this coding tool.
Writing disclosure: This article was drafted with AI writing assistance based on my project's documentation and test records. I am responsible for reviewing the technical claims before publication.
Top comments (0)