DEV Community

Discussion on: Your AI's code is "almost right". Here are 7 ways it fools you, and the 30-second check for each

Collapse
 
stackmasteraliza profile image
Aliza Ali •

2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking git diff --stat first is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?

Collapse
 
stackmasteraliza profile image
Aliza Ali •

Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.

Thread Thread
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, that's a fair point. Each edit passes review on its own, so the only place the extra changes show up together is the final diff. That makes --stat useful even with step-by-step review, not only with auto-accept.