DEV Community

Cover image for The agent wrote the code. The tests are green. The linter is clean. You can't explain what changed.
Viktoria
Viktoria

Posted on

The agent wrote the code. The tests are green. The linter is clean. You can't explain what changed.

#ai #programming #javascript #python

There is a moment in a code review where a developer is asked why the code does what it does, and the answer is: "Claude wrote it." The code works. The tests pass. The reviewer approves. Nobody on the call can explain the architecture of the PR, including the person who submitted it.

At Explyt, where we build an AI agent for JetBrains IDEs, two independent customer development interviews reported the same timeline: a month of agent-driven development is enough to lose the mental model of the code. The tests are green, the linter is clean, and the developer who shipped the PR cannot answer a question about it in review. The code works. The model is gone.

TL;DR

  • Two independent custdevs reported the same timeline: a month of agent-driven development, and the developer can no longer explain the code they are shipping.
  • Compilation, green tests, and auto-LGTM verify the artifact. They do not verify that the author understands the invariant the code protects. A PR can pass all three on code nobody in the room can explain.
  • The community calls this comprehension debt. It shows up before review metrics move. The first signal is a question the developer can't answer.
  • The IDE keeps the running program. The chat transcript keeps the text. Understanding lives in the gap between them, and the debugger is where you close it.

The one-month timeline

Two customer development interviews, independent of each other, converged on the same number. A month of agent-driven development — where the agent writes the code and the developer reviews the diff — is enough to lose the mental model.

The pattern is the same in both reports. The developer starts by reviewing every line. Within a week, they review the summary and skim the diff. Within two weeks, they approve based on the green tests and the agent's description. Within a month, they are asked a question about the code in a review and cannot answer it. The code works. The model is gone.

What makes this worth reporting is the convergence: two people who never spoke to each other named the same timeline.

What green tests don't prove

Compilation, green tests, and auto-LGTM — the three signals most pipelines use to accept a change — can all pass on code nobody understands.

Signal What it checks What it doesn't check
Compiler Types match Intent is correct
Test suite Code matches its own assumptions Assumptions are correct
Auto-LGTM Pipeline ran Author can explain the result

The compiler checks types, not intent. A method that compiles can still violate the invariant it was written to protect. The test suite checks the code against its own assumptions, and if the assumptions are wrong, the tests pass and the bug ships. Auto-LGTM checks that the pipeline ran, not that the author can explain the result.

All three verify the artifact. None of them verify the author's model of the artifact. A developer who cannot explain why the code does what it does can still ship a PR that compiles, passes tests, and gets an auto-merge.

The day development stops

There is a moment where the agent fails to fix the next bug, and the developer cannot step in.

The developer who reviewed every line can step in. The developer who approved based on green tests for a month cannot — they don't know where the invariant lives, what the side effects are, or which module owns the state. Development stops because the human ran out of context, not because the agent ran out of capability. The tool is still there. The understanding is not.

Reddit practitioners describe this as the limit of "vibe coding" — the practice of accepting agent output without rebuilding a mental model. The practice works until the agent fails, and then the developer who was vibing is stuck. They can't debug what they don't understand, and they can't understand what they never read.

Explain it in your own words

A practice reported from custdev: before accepting a diff, explain the change in your own words, without looking at the agent's summary.

Compilation and green tests don't prove the author understands the invariant. A verbal explanation does. If you can close the diff, describe what changed, why it changed, and what breaks if it reverts — you own the change. If you can't, the diff owns you.

The check takes a minute. Close the diff. Close the agent's summary. Open a blank document. Write: "This change does X because Y. If reverted, Z breaks." If you can't fill in X, Y, and Z from memory, you don't understand the change, and you shouldn't merge it.

A Reddit team practice adds a social version: every PR needs a named engineer who can later explain the change from memory. LLM use is allowed, but one person must be able to answer for the result and the breakage. The rule does not ask the agent to explain itself — it asks the human to.

One day a week

Another practice from custdev: one day a week, stop feature work. Spend the day simplifying architecture, reviewing code, and regaining understanding.

The developer who reported this practice described it as maintenance — of the mental model, not of the code. The code is maintained by the agent. The model is maintained by the human. The model decays faster than the code, because the agent generates code faster than the human can rebuild the model, and the gap widens every week.

One day a week is a practice one developer described, not a recommendation. What it tells you is that ownership is a recurring cost, not a one-time investment. You don't buy it by reading the code once. You pay it continuously, or you lose it.

What the IDE keeps that the chat doesn't

The chat transcript keeps the text. The IDE keeps the running program. Understanding lives in the gap between them, and the debugger is where you close it.

Run configurations let the agent run the test through the IDE and get back console output, test results, and compilation errors as a structured result. The developer sees which tests ran, which failed, and what the failure was — a structured result, not a line that says "tests passed."

Run configurations: the IDE runs the test and returns the result. The developer sees what ran, not just that it passed.

Debug mode asks the agent to set a breakpoint, read the variable values, inspect the call stack — before proposing a fix. The variable value at the breakpoint is an observation the chat cannot give you. It is also the moment where understanding gets rebuilt: you see the state, you see the decision point, you see why the code does what it does.

Debug mode: breakpoint, variable values, call stack. The moment where the mental model gets rebuilt.

Agent Changes keeps the diff in a scoped panel separate from the chat. Each change is individually accepted or rejected. The diff is where the developer rebuilds the model — by reading the code that changed and asking whether it matches the intent, not by reading the agent's summary of the intent.

None of these replace the human in the loop. They give the human something to loop on that is more than text. The chat says "the code does X." The IDE says "the variable at line 47 is null, and here is the call stack that got you there." The first is a claim. The second is a fact. Understanding starts from the second.

Anthropic's own guidance points the same way. Claude Code best practices: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop." The check the IDE provides — a test result with names and failures, a debugger with variable values — is a check the chat cannot give you, and it is also where the mental model gets rebuilt. Verification and understanding come from the same instrument.

Try this on your next PR

Copy this rule into your AGENTS.md or team checklist. Every line is a suggestion, not a claim — adapt it to your project.

## Comprehension check (before merge)

1. Close the diff and the agent's summary.
2. Open a blank document. Write:
   "This change does X because Y. If reverted, Z breaks."
3. If you can't fill in X, Y, and Z from memory:
   - Reopen the diff.
   - Set a breakpoint at the decision point.
   - Watch the variable. Read the call stack.
   - Try again.
4. If you still can't explain it after step 3, don't merge it.
Enter fullscreen mode Exit fullscreen mode

The check takes a minute. The cost of skipping it shows up in a month.

FAQ

Isn't this just a skills problem? Get better at reading code and it goes away.

Reading code is the floor. The custdevs who reported the one-month timeline were experienced developers. The model decayed because the agent generates code faster than the human can read it, not because the human can't read. A faster reader buys you more time; the gap still widens.

If the agent can fix bugs on its own, why does my understanding matter?

Because the agent can't fix every bug. The moment it fails — the edge case, the production-only issue, the interaction between three modules — you are the one who steps in. If you can't, development stops. Two custdevs described the same month-long window after which they could no longer step in.

Won't better tests solve this?

Tests check the code against its own assumptions. If the assumptions are wrong, the tests pass and the bug ships. Tests verify the artifact, not the author's model of it. You can have 100% coverage and zero understanding.

Is this against using AI agents for code?

The "explain it in your own words" check and the "one day a week" practice come from developers who use agents daily. They are not arguing against agents. They are describing what they do to keep owning the code the agent writes. The question is not whether to use the agent. The question is what you do alongside it.

Conclusion

The agent writes the code. The tests are green. The linter is clean. The question is not whether the code works — the question is whether you can still explain it. If you can't, the code owns you, not the other way around.

The next time the agent produces a diff, close the chat. Open the diff. Explain the change in your own words: what changed, why, and what breaks if it reverts. If you can't, set a breakpoint at the decision point and watch the variable. That is one fact the chat cannot give you, and it is where understanding starts.

If you work in a JetBrains IDE, Explyt on the Marketplace gives the agent the IDE run and the debugger. The rest is on you.

Related reading

Sources

Top comments (0)