Every PR description is a claim. "No consumers of this API were changed." "Safe to merge." Most review tools take that claim at face value, because checking it would mean looking outside the diff.
I wanted to know what happens when a review tool doesn't take the claim at face value. So I built two connected services. I made a change that looked completely safe in isolation. I wrote a PR description that explicitly said nothing downstream would break. Then I let Qodo decide whether to believe me.
It didn't.
the setup
Two small repos, deliberately simple:
-
orders-api— a Cloudflare Worker.GET /orders/:idreturns{ id, status, total, currency }. -
orders-client— a consumer that fetches an order and fulfills it iforder.status === 'paid'.
Baseline works end to end. Fetch a paid order, it fulfills. Fetch a pending order, it skips. Nothing exotic.
Then I opened a PR on orders-api that changes status from a plain string to a nested object: { value, updatedAt }. The reasoning in the PR description was real: I wanted an audit trail for when an order's status last changed. The description also said this explicitly:
"No consumers of this API were changed as part of this PR — response shape update only."
That sentence is true and misleading at the same time. No consumer's code changed. But orders-client still does order.status === 'paid', and that comparison (a string against an object) silently returns false forever. Paid orders stop being fulfilled. No crash, no error, no log line that looks wrong. Just orders that quietly never ship.
orders-api's own type checker sees none of this. Nothing in that repo references orders-client. The break is real. It's completely invisible from inside the repo where it originates.
running qodo
I commented /agentic_review on the PR. Ninety seconds later:
Action required: Existing clients misread order statuses 🐞 Bug · Correctness
"Order.status changes from the existing string union to an OrderStatus object while the same route continues serializing the order without versioning or negotiation. Every successful order request now returns an incompatible field to existing consumers, including the documented orders client that this PR explicitly leaves unchanged."
That last clause is quoting my own PR description back at me, the part where I said nothing downstream was affected, and pointing out that it's wrong. Qodo didn't guess this. It checked orders-client, the actual repo, and found the line that breaks.
proving it wasn't a lucky guess
One finding could just mean Qodo read my PR description carefully and got suspicious of the wording, not that it checked the other repo at all. I wanted to rule that out.
So I ran a control. Same exact code change. Same PR description, word for word. The only variable: I removed orders-client from Qodo's connected repositories first.
The result, on an otherwise identical PR:
Great, no issues found!
"Qodo reviewed your code and found no material issues that require review."
Connected: one action-required bug, correctly named, correctly explained. Disconnected: a clean pass. Same code, same description, same claim about consumers being unaffected. The only thing that changed was whether Qodo could see the repo the claim was about.
That rules out coincidence. The connection is causal.
what surprised me
The "no issues found" screen is the one that stuck with me, not the bug it missed.
It's polished. Little Qodo mascot, clean green checkmark. If you saw that screen in isolation, it reads like an endorsement: ship it, you're clear. It isn't lying, exactly. Qodo found no issues in what it could see. But "no issues found" and "no issues exist" are different claims, and the UI doesn't visually distinguish between them.
Every review tool has this problem, not just Qodo: a clean bill of health only means as much as the tool's field of view. A confident, cheerful "all clear" from a tool that can't see half the picture is more dangerous than an honest "I don't know," because nothing about the interaction tells you to doubt it.
The fix is exactly what connecting orders-client did: widen what the tool can see before it renders a verdict at all.
single-repo vs. connected
orders-api's own review pipeline (tests, types, linting) has no way to catch this. It was never going to. The bug doesn't live in orders-api. It lives in the gap between two repos that don't know about each other.
This is why cross-repo context matters here: it catches a false claim of safety that a single repo has no way to verify. Most bugs in most PRs are contained to the diff. This one specifically wasn't, and the failure mode when you get it wrong is the quiet kind: nothing breaks loudly, revenue just doesn't fulfill, and someone eventually finds out from a support ticket, not a stack trace.
Single-repo tools aren't wrong to focus on the diff. They're just answering a narrower question than the one that mattered here.
takeaway
Your PR description is a claim, not a fact. "Nothing downstream changed" is something you believe until the thing checking your PR can actually see downstream.
I wrote a true sentence that was still wrong. Qodo caught the wrong part. Then I proved it wasn't luck by taking the same claim, removing the tool's ability to check it, and watching it get fooled by the exact same sentence.
If your review tool can't see the repos your change touches, it isn't reviewing your change. It's reviewing your diff and trusting your description for everything else.
Repos: orders-api · orders-client. Qodo runs on the free tier for public repos — qodo.ai.
Top comments (0)