DEV Community

Sarah
Sarah

Posted on

AI-Generated Code Passes the Demo and Fails the Review. Here's the Checklist I Use

#ai #codequality #programming #webdev

TL;DR

  • AI-generated code usually runs. The problem is that it's plausible, and plausible code is harder to review than code that's obviously broken.
  • The failures are predictable: hallucinated APIs, happy-path-only logic, silent error swallowing, stale dependencies, duplicated helpers, weak types, and security shortcuts.
  • Don't review AI code the way you review human code. Use a checklist aimed at the failures AI actually makes, and put most of it into CI.
  • Below are the 7 checks I run, with code you can copy.

Why "it works" is the wrong bar

When a junior dev writes a bug, the code usually looks uncertain. There are odd names, leftover console logs, and a TODO comment. Those are signals that tell you where to slow down.

AI code has none of those signals. It's well formatted, confidently named, and commented. Reviewers relax because it looks senior. That's the trap.

My rule is to treat AI output as a first draft from a fast intern who has read every Stack Overflow answer but never been on-call.

1. Hallucinated or outdated APIs

The model remembers an API as it existed in its training data, or makes one up that sounds right.

// AI wrote this. Looks fine.
const res = await fetch(url, { timeout: 5000 });
Enter fullscreen mode Exit fullscreen mode

fetch has no timeout option. This compiles in loose setups and silently does nothing. Here is the fix:

const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 5000);
try {
  const res = await fetch(url, { signal: controller.signal });
  return await res.json();
} finally {
  clearTimeout(timer);
}
// or, in modern runtimes:
await fetch(url, { signal: AbortSignal.timeout(5000) });
Enter fullscreen mode Exit fullscreen mode

Check: Turn on strict TypeScript ("strict": true, "noUncheckedIndexedAccess": true). Strict types catch a large share of invented options at compile time.

2. Happy-path-only logic

Ask for "a function that parses user age," and you'll get something like this:

function parseAge(input) {
  return parseInt(input);
}
Enter fullscreen mode Exit fullscreen mode

parseAge("12abc") returns 12. parseAge("") returns NaN. parseAge("-5") returns -5. None of these fail loudly. Here is a version that does:

function parseAge(input) {
  const n = Number(input);
  if (!Number.isInteger(n) || n < 0 || n > 150) {
    throw new RangeError(`Invalid age: ${input}`);
  }
  return n;
}
Enter fullscreen mode Exit fullscreen mode

Check: For every AI-written function, write the 3 nastiest inputs before reading the implementation. Property-based testing is a good fit here:

import fc from "fast-check";

test("parseAge never returns garbage", () => {
  fc.assert(
    fc.property(fc.string(), (s) => {
      try {
        const age = parseAge(s);
        return Number.isInteger(age) && age >= 0 && age <= 150;
      } catch {
        return true; // throwing is acceptable, returning junk is not
      }
    })
  );
});
Enter fullscreen mode Exit fullscreen mode

3. Silent error swallowing

AI loves the try/catch that makes errors disappear:

try {
  await saveOrder(order);
} catch (e) {
  console.log(e);
}
Enter fullscreen mode Exit fullscreen mode

The order isn't saved, the user sees success, and you find out from a refund request.

Check: Add a lint rule that bans empty or log-only catches.

// .eslintrc.json
{
  "rules": {
    "no-empty": ["error", { "allowEmptyCatch": false }],
    "@typescript-eslint/no-floating-promises": "error"
  }
}
Enter fullscreen mode Exit fullscreen mode

The no-floating-promises rule is underrated. AI frequently forgets await, and this rule catches it.

4. Dependency drift and phantom packages

AI suggests npm install for packages that are deprecated, unmaintained, or don't exist. That last case is a real supply-chain risk, because attackers register the package names models tend to hallucinate.

Check: Before installing anything an AI suggested, run this:

npm view <package> time.modified repository.url
npm audit --omit=dev
Enter fullscreen mode Exit fullscreen mode

If the package was last modified years ago or has no repo, don't install it.

5. Duplicated helpers

Each prompt is a fresh context. Ask for a date formatter on Monday and another on Thursday, and you'll end up with formatDate, formatDateString, and toDisplayDate scattered across three folders, each handling time zones differently.

Check: Run a duplicate detector in CI.

npx jscpd src --min-lines 8 --threshold 3
Enter fullscreen mode Exit fullscreen mode

Also give the AI context. Paste your existing utils/ file into the prompt and tell it to reuse what's there.

6. Weak types disguised as strong ones 

const data = (await res.json()) as User[];
Enter fullscreen mode Exit fullscreen mode

The as cast is a promise you made to the compiler, not a check. If the API returns { error: "..." }, the type system will happily lie to you for the rest of the request.

Check: Validate at the boundaries.

import { z } from "zod";

const User = z.object({ id: z.string(), email: z.string().email() });
const data = z.array(User).parse(await res.json());
Enter fullscreen mode Exit fullscreen mode

Add a lint rule to flag as casts on any or unknown values, and review every one.

7. Security shortcuts

These are the classic ones I keep seeing in AI output:

// SQL via string interpolation
db.query(`SELECT * FROM users WHERE email = '${email}'`);

// Wide-open CORS "to fix the error"
app.use(cors({ origin: "*", credentials: true }));

// Secrets inline "for testing"
const API_KEY = "sk-live-...";
Enter fullscreen mode Exit fullscreen mode

Check: Run static analysis plus secret scanning on every PR.

npx semgrep --config auto
npx gitleaks detect --source .
Enter fullscreen mode Exit fullscreen mode

Put it in CI, not in your head

Reviewers get tired. CI doesn't. Here is a minimal GitHub Actions job that covers most of the list:

name: ai-code-guard
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - run: npx tsc --noEmit
      - run: npx eslint . --max-warnings 0
      - run: npx jscpd src --min-lines 8 --threshold 3
      - run: npm test -- --coverage
      - run: npx semgrep --config auto --error
      - uses: gitleaks/gitleaks-action@v2
Enter fullscreen mode Exit fullscreen mode

That leaves human review for the parts tools can't judge: whether the logic matches the business rule, and whether this code should exist at all.

Where generation tools fit

Not all AI codegen is equal. A general chat assistant writes code with no knowledge of your project. Tools that generate within a fixed, opinionated stack produce more consistent output because there are fewer ways to go wrong. For example, RapidNative generates React Native screens on a known Expo and NativeWind setup. That cuts down the duplicated-helper and phantom-dependency problems, though you still need checks 2, 3, and 6.

Constraining the generator and enforcing the checks is the combination that works. Either one alone leaks.

The review mindset, in one line

Don't ask "does this work?" Ask "what input makes this lie?"

What's the worst AI-generated bug that made it past your review? Drop it in the comments.

Top comments (0)