I pre-registered a small study, pointed a 12-reviewer pipeline at three merged Copilot PRs in major .NET organisations, and the failure mode I foun...
For further actions, you may consider blocking this person and/or reporting abuse
The “ignored repo memory” framing matches the failure I keep running into with coding agents. The cheap guard I like is forcing the reviewer to name the older file or helper that should have changed its answer. If it can’t point to one, it’s probably just grading the diff in isolation.
That guard matches the pilot data almost exactly. In case 1, every finding that would have changed the merge decision cited an unchanged file. The sharpest one pointed at a guard in another module whose comment named the exact failure mode the new code reintroduced. The findings that only referenced the diff were the ignorable ones.
review-pro encodes the per-finding version of your rule: a finding without located evidence is forbidden by rubric, and "located" usually means outside the diff. What it doesn't have yet is your aggregate version — if an entire review of a substantive change contains zero references to unchanged files, the review itself is suspect. That's cheap to check at the synthesis stage, and I'm adding it: github.com/tufantunc/review-pro/is...
One boundary I'd put on it: it can't be a hard per-finding gate, since some real defects live entirely inside new code (the off-by-one variety). As a review-level tripwire, though, I think you're right and it costs nothing. 🙌
Doesn't a corpus of merged PRs filter hallucinations out by construction? An invented API usually dies in CI or human review, so finding zero of them in merged code is what you'd expect even if agents invent things constantly. Running the same protocol on closed or abandoned agent PRs would show whether the invention failure mode just gets caught earlier in the funnel.
This is a really interesting distinction. It makes me think that “context retrieval” for coding agents isn't just about finding semantically similar code — it's about finding the existing decisions, conventions, constraints, and failure knowledge that should influence the change.
An agent can produce perfectly plausible code and still be wrong simply because it missed something the repository already knew.
I wonder if “what the codebase already knows but the agent failed to retrieve” could become its own useful category of agent failure.
"Finding the existing decisions, constraints, and failure knowledge not semantically similar code" is the sharpest framing of why this fails that I've seen. The artifacts that mattered in the pilot were all dissimilar to the new code: a catch clause in a version-calculation module looks nothing like ancestor-parsing logic; a deserializer's nullability looks nothing like
the client code that deleted a workaround. Embedding-style retrieval would never rank them. What links them to the change is causal; caller graphs, contracts, an upstream issue number in a deleted comment not similarity.
On making it a category: the pilot effectively operationalized the review-side version as
ignored-convention, and it carried every finding that would have changed a merge decision (classification rules are in the pre-registration). The property you'd have to accept for it as a first-class agent-failure category is that it's counterfactual. You can only label "failed to retrieveX" after someone locates X. Which is exactly why the review has to leave the diff, and why located evidence is a hard rule here rather than a nicety.
If you ever formalize the retrieval-side version of this, I'd read it.
Yeah, the counterfactual part is what makes this especially difficult to evaluate. You can’t really measure “the agent missed this” until the evidence is found and shown to have mattered. I also like the causal-vs-semantic distinction — it suggests retrieval quality needs to be evaluated against relationships and decision impact, not just similarity or recall.
The distinction between reviewing a diff and reviewing the full change is easy to miss. In our work at IT Path Solutions, we've seen that AI-assisted reviews become much more reliable when the agent considers callers, existing guards, shared utilities, and contracts, not just the lines that changed. The repository often contains the context needed to catch a seemingly reasonable change that would otherwise slip through review.
That four-item list;
If production use at your end has surfaced a context category beyond those four, I'd genuinely like to hear it. The scoping policy gets calibrated from exactly that kind of report.
One category I'd add is configuration and runtime state. A change can look correct when you trace callers, guards, utilities, and contracts, but still behave differently because of feature flags, environment variables, deployment configuration, or runtime assumptions that aren't obvious from the code path. That feels like an important boundary for agents too, especially when trying to explain why something works in one environment but breaks in another.
Configuration and runtime state is a real fifth category and the pilot accidentally supports you. The highest-severity finding in case 1 was environment-conditional: the same code works on a full clone and breaks on a shallow CI clone, and nothing in the diff hints that the failure needs
--depth 1to exist. Case 3's entire finding cluster was an env varapplied at some call sites and not others including two sites where the variable never applies at all, so the "same" timeout means different things in different invocation contexts.
The tool covers slices of this today (environment-conditional behaviour and feature-gate leaks sit in the correctness rubric; invented config keys are the inverse check), and the in-repo part is tractable — CI workflows, deployment manifests, and .env examples are all searchable evidence. But your category is also where my "the repository already knows" framing hits its honest edge: flag state and deployment config often live outside the repo entirely in the flag service, the deploy platform, an ops runbook. A repo-grounded reviewer can flag that behaviour is environment-dependent; it can't always tell you which environment is the broken one.
That boundary — repo-knowable vs. environment-knowable — might be the most useful refinement this thread has produced.
github.com/tufantunc/review-pro/is...
The framing of ignored what the repo already knew versus hallucination is the more useful bug report, because it's fixable in a way hallucination isn't. Hallucination is a model-capability problem. Ignoring known context is a retrieval and enforcement problem, and it's the one that actually scales with better models rather than getting solved by them.
This matches the review failure I keep seeing. The diff gives the reviewer a neat boundary, but the bug usually lives in the caller graph or an old guardrail outside that boundary. Green CI only proves the sampled path survived.
"Green CI only proves the sampled path survived" — I'll be borrowing that. Case 2 of the pilot was one notch worse: the sampling frame couldn't reach the risk at all. Every fixture omitted the field; the hazard was an explicit
"field": nulla different deserializer path. The suite passedidentically with the bug fixed or not.
Both of your locations showed up in the pilot: the top finding was literally an old guardrail in another module, and its severity only surfaced by walking the caller graph. That instinct is getting an aggregate check now a review of a substantive change that cites zero out-of-diff files is itself suspect: github.com/tufantunc/review-pro/is...