DEV Community

Altair
Altair

Posted on

Ask, Don't Just Read: Keeping Up With Code an AI Wrote

#claudecode #ai #codereview #productivity

I'm the person in charge of digital transformation at a Japanese company. I build internal tools and workflow automations, and these days an AI coding agent (Claude Code) writes almost all of the code.

I thought my problem was "code review can't keep up." When I looked closer, it was something else.

The bottleneck wasn't review. It was understanding.

To be honest, for most of this work I no longer read diffs line by line. What I was actually stuck on was:

  • I couldn't tell whether the change did what I intended.
  • I couldn't explain it to anyone else.

Tools that make diff review faster don't help if you've already stopped reading diffs. I needed a way to say "I understand this" without reading everything.

It turns out everyone is hitting the same wall

  • Anthropic's own Claude Code team has said the bottleneck has moved from writing code to reviewing it.
  • Margaret-Anne Storey (2026) splits the problem into three kinds of debt: technical debt in the code, cognitive debt (the team's understanding erodes), and intent debt (the reasons and constraints never get written down). It's a conceptual paper, not an experiment.
  • In a 2026 survey in Japan, 86.3% of engineers who review AI-generated code said the burden had increased, and the top problem (49.5%) was "the submitter can't explain the code."

The study that changed how I approached it was Anthropic's randomized trial from January 2026. 52 mostly junior engineers learned a new library. The group using AI scored 50% on a comprehension quiz versus 67% for the hand-coding group, and the biggest gap was in debugging.

But the result depended on how people used the AI. People who fully delegated to it scored under 40%. People who asked conceptual questions and fixed errors themselves scored 65% or higher. (The authors are clear that this is correlation, not causation.)

So maybe reading an AI's explanation isn't enough. Maybe what matters is the human asking the question.

What I do now: an intent check

Spec-driven development plus automated review is great for a big team, but it's too heavy for one person. I cut it down to four things.

1. Write the intent before building

One line of purpose, then 3–5 success criteria in the form "when X, Y happens." The AI drafts them, I confirm. One round-trip.

This is where mismatches between the goal and the approach show up — much cheaper than finding them afterward.

2. Judge the result with a table, not the diff

# Success criterion Result Evidence
1 … ✓ / ✗ / unverified test run, actual output, screen, counted numbers

Evidence must come from actually running it. "The code says so" doesn't count as ✓. The last row is always: "Do the comments and design notes still match what it does now?"

3. Leave one or two questions — without answers

Under the table, the AI writes one or two "questions worth asking to understand this better." It doesn't write the answers. I decide whether to ask. If I don't, that's fine.

If the study above is right, asking beats reading. This just leaves the door open.

4. Keep four lines for explaining it to others 

- What changed: (from the user's point of view)
- Why:
- How to check it:
- What it doesn't do:
Enter fullscreen mode Exit fullscreen mode

These go into the design notes. The table is for me to judge "did it do what I meant." These four lines are for explaining it to someone else.

A one-line change, two surprises

The first change I tried this on was a single line: keep 3 daily backups instead of 7. The goal was to free up disk space.

The table showed:

  • ✓ It keeps the latest 3 plus one per week for 4 weeks.
  • ✗ It barely frees any space. Old backups go to the Recycle Bin, so the disk space doesn't come back until the bin is emptied.
  • ✗ The comment at the top of the file still said "keep 7."

Neither problem was visible from the diff. They showed up because I wrote down the purpose first.

What it is not

  • Not a replacement for automated code review. That finds defects; this finds gaps against intent.
  • Not a replacement for having the actual users check the output against real data.
  • When something is ✗, the AI doesn't quietly fix it. A human decides whether the intent or the implementation was wrong.

What I don't know yet

I've only just started. To avoid judging it by how it feels, I log four things each time:

  1. Number of ✗ results
  2. Time spent on the up-front back-and-forth
  3. Whether I asked the questions
  4. Later, when someone asked me about it, whether I could answer without asking the AI again

In one study, developers felt 20% faster with AI while actually being 19% slower (METR, 2025). I'll write a follow-up once I have a few months of data.

As AI writes more of our code, the human job seems to be shifting from "read everything" to "decide what to trust." If so, the key to keeping our understanding might be not how much we read, but how often we ask.

References

Top comments (2)

Collapse
 
eventuallyconsistentcode profile image
John Reed •

"The code says so doesn't count as ✓" is the sentence I'd frame, because it's the whole disease in nine words…

I build cairn, a work harness for coding agents (github.com/eventually-consistent-c...), and your four steps map almost one to one onto the gates it runs, so let me show where they land and where you're ahead of me.

Your step 1, intent before building: every task in a cairn plan carries a verify: <command> clause written at plan time, before any code exists, and the planner refuses to proceed if a task is missing one (plan.md). Same reason you give: criteria written afterward describe what got built, not what was meant. Yours is a sentence per criterion, mine is a command per task, and I think yours is the better interface for the human and mine is the better one for the m
achine, so both probably belong.

Your step 2, the evidence table: when an agent closes an issue it has to pass evidence: { command, result } from the run it just did, and the server posts that as the close comment and appends a ledger line, it doesn't accept a summary (work.md). Then a separate pass re-runs the declared check and flags evidence-missing or observed-mismatch when the close claim and the re-run disagree (check.ts). Phase verification is explicitly goal-backward, the verb text says to check the codebase delivers what the phase promised, "not merely that tasks closed" (verify.md). Your "keep 7" comment that nobody updated is what docs_drift is for, it's the last thing verify reports, and your table's last row says the same thing in plainer words.

Your "when something is ✗, the AI doesn't quietly fix it" is also a hard rule here: a failed verification has to open a trace session with the failing output as its first entry, never patch-and-rerun inline, because the fix without the record is how the same bug comes back in a month wi
th no history.

Step 4, the four lines for someone else: that's what the close comment plus brief is for, it regenerates a briefing from the ledger and the decision cards for "someone who wasn't there" (brief.md). Storey's intent debt is the one I worry about most, so decisions and constraints live as memory cards with a confidence grade and provenance commits, and retro re-grades them against what actually happened (retro.md). Nothing decays on its own, but nothing stays high-confidence without being re-checked either.

And the honest part: step 3 I don't have. Cairn produces evidence for the human to read, and reading is exactly what the Anthropic study says isn't enough. Two unanswered questions under the table, that the human chooses whether to ask, is a cheap mechanism and I'm stealing it, probably
as a block the close comment carries. Thanks for that one, and please do write the follow-up with the numbers, the "could I answer without asking the AI again" metric is the one nobody measures…

Collapse
 
altair_alter profile image
Altair •

Thanks, John. Seeing the four steps mapped onto gates that actually run is the best review this post could get.

I agree with the split: a sentence per criterion for the human, a command per task for the machine. In my table the sentence comes first, and the command ends up in the evidence column, so maybe they can live in the same row.

The re-run that flags observed-mismatch is the part I don't have. Right now I trust the evidence the AI pastes into the table. A second pass that runs the check again would catch evidence that has gone stale.

Please do take step 3. Skipping the questions is allowed on purpose, so they stay cheap.

The follow-up with numbers will come once I have a few months of data.