The AI code review slot machine stops when someone decides what counts

On 11 October, Lucas (@lucas_builds) posted a short complaint that was viewed 1.2 million times:

Asking Claude to review your code is a slot machine.

Every review finds brand new bugs. Review again: more. Review a tenth time: it still finds something.

It ends with a question: how do you decide your code is actually done?

Lucas later said the Claude Code team replied that fixes are landing, so part of it is a product problem. But the question is the interesting part, because most of the complaint isn't about a bad model. A reviewer asked to "find bugs" will always find something, because there is always something that could be different. A missing edge case, a name that could be clearer, a check that could be stricter. None of that is false. It just never ends.

The five rules from 920 replies

The next day Lucas collected the answers. Out of 920 replies, the same five rules kept coming up:

1. Define done before the review: tests pass, spec met.
2. Only fix critical and high. Everything else goes on a later list.
3. No failing test, no bug.
4. Let a different model review than the one that wrote it.
5. Cap it at 3 rounds.

Addy Osmani, quoting the original the day before, had already made the same point as the first rule: "Clearly define 'done' before code review starts." Without it, he wrote, a reviewer, human or agent, "may still find something that could be improved". His fix is to give the agent checks it can run (commands, tests, clicking through the app in a browser) and to merge when nothing blocks.

These are good rules. Read them again, though, and most of them are the same rule in different clothes.

Every rule is a stopping condition

Rule 1 says what done is. Rule 3 says a finding only counts if a test can show it. Rule 5 stops after three rounds whatever happens. Rule 2 stops at a severity line. Each one draws a line the reviewer can't move.

Rule 4 is the odd one out. A different model catches different things, which is useful, but it doesn't tell either model when to stop. Two slot machines are still slot machines.

What the rules have in common is that the line is drawn before the review, by someone other than the reviewer. That's the whole trick. The review never ends when the thing deciding what counts is also the thing asked to find more.

Tests draw the line for behavior, not for everything

"Tests pass, spec met" is the strongest version, and where you can write the test, it's the right one. A failing test is a finding nobody can argue with.

But a lot of what a review catches isn't behavior a test pins down. The retry that will double-charge under a race you didn't think to test. The function that works and that nobody on the team will understand in a month. The new dependency going the wrong way across a boundary. The spec says nothing about those, and a test only exists after someone has noticed the problem.

Rule 2 tries to cover that ground with severity, but look at who assigns it. "Only fix critical and high" usually means the reviewer labels its own findings, and then you act on its labels. The line moved back to the reviewer.

The line is a reading

So for the part tests don't cover, someone has to decide what counts, and the only one who can is the person who'll answer for the code. That means reading it. Not every line with the same attention, but enough to say: this, this and this have to change; the rest is fine, or later.

That decision is the definition of done for the round. Written down, it's a short list of comments on exact lines. The agent's job then has an end: resolve those, and nothing else. A second pass checks the list, not the whole file again. When the list is empty, you stop.

This is what I mean by the inverted code review: the human reads first and the agent commits. It isn't a rejection of AI review. An agent is very good at proposing findings. It just shouldn't be the one deciding which ones count.

How Reado holds that line

I build Reado around that loop, so here's what it does today, and where it doesn't help.

When you read in Reado, you comment on a line range, and each comment is either a task for the agent or a note for the next reader. Tasks are the list. Send review hands only the open tasks to the agent running in Reado's terminal (Claude Code, Codex, Copilot, Gemini, OpenCode or Cursor). Notes stay out of its way. The agent resolves each task or fails it with a reason. A fix it can't prove with a passing check stays on your list as unverified until you look at it; a failed task goes back to open, and after three failures it stops and waits for you. The list is the stopping condition, and you wrote it.

When you want the agent to look first, the guided review keeps the same split. The agent walks the files in scope and proposes comments: bug, refactor, performance, question, or "needs context" when it can't judge. Nothing it proposes becomes a task until you accept it. A second-opinion pass asks for a challenge of the findings already on the file, looking for false positives and over-statement, and those challenges are proposals too.

That's also where rule 4 fits. Reado's terminal has tabs and side-by-side panes, and you can run a different agent in each one. Reado sends to the one you last clicked, as long as an agent is running in it, and Send review lets you pick the terminal when more than one is open. So you can let Claude Code do the first pass in one, click over to Codex in another, and ask it for the second opinion. Two models, but still one list, and you're the one who accepts what goes on it.

Reado doesn't make the slot machine stop on its own. If you accept everything, it's the same machine with extra clicks. What it changes is where the decision sits: in your list of tasks, on the lines you read, and in your repository as plain Markdown in .reado/, so next week you can still see why the line looks the way it does.

What I'd take from the thread

If you want one habit out of those 920 replies, it's Addy's: decide what done means before the review starts. Write the test where you can. Where you can't, read the code and write the list yourself, on the lines that matter, before the agent touches it again.

The agent will always find another bug. You decide which ones are worth fixing, and then you stop pulling the lever.

The Reado app is free and MIT-licensed, if you want to try that loop: download it here.