DEV Community

Discussion on: Your AI's code is "almost right". Here are 7 ways it fools you, and the 30-second check for each

Collapse
 
mudassirworks profile image
Mudassir Khan •

the check on number 2 is the one i would make automatic, because a green run where the test file also changed tells you nothing. we added a rule where any diff touching a test file needs a one line reason in the PR, and a test that got skipped or loosened to toBeDefined fails the review bot. the failure that still slips through for us is number 5, a catch that returns an empty array, since it passes every test written by the same agent. do you have a way to catch swallowed errors besides grepping the diff for catch?