GitHub published one of my favorite kinds of engineering posts this month: the kind where the first attempt made things worse.
The story is simple in the uncomfortable way. Copilot code review had its own code exploration tools. GitHub tried replacing them with better maintained shared tools from the Copilot CLI world: grep, glob, and view. This sounds obviously good. Fewer duplicated tool implementations. Shared improvements. Very clean architecture diagram. Maybe a tasteful arrow.
Then the benchmarks said no.
Review cost went up. Fewer useful issues were caught.
This is the part I like. Not because failure is funny, although it often has excellent timing. I like it because it exposes a real truth about agents:
Tools are not neutral plumbing.
For an agent, the tool surface is part of the behavior. It changes what the model notices, how it searches, how much context it drags around, when it gives up, and what kind of evidence it thinks it has.
That is not an implementation detail.
That is product design with a terminal attached.
a better hammer still needs the right job
The tempting interpretation is "better tools should produce better agents."
This is almost right, which is the most expensive kind of wrong.
The shared tools GitHub tried were not bad. The problem was the workflow implied by those tools and their instructions. A CLI coding assistant often needs to explore. The user might ask it to understand a repo, plan a change, modify files, run tests, and keep going after the first attempt burns down in a normal software way.
Broad exploration makes sense there.
Code review is different.
A reviewer starts from a diff. A reviewer asks small questions. Did this changed helper affect a caller? Is this config value used somewhere else? Is this new authorization path actually covered? What is the smallest nearby piece of evidence that proves or disproves the concern?
The reviewer does not usually open half the repository to "build context."
That is not review. That is tourism with line numbers.
When Copilot code review got a more general set of exploration tools, it started behaving more like a browsing assistant. It searched broadly, read broadly, carried extra context forward, and became less focused on the pull request evidence.
In human terms, it became the reviewer who says, "I just need to understand the whole system first," and then disappears into services/legacy_v2_final/ for forty minutes.
We have all worked with this person. Sometimes this person is us. Nobody is proud.
context is not free just because it is text
One reason this matters is that agents do not treat tool output like a disposable terminal printout.
If a human runs grep and sees twenty irrelevant lines, we can ignore them. We are good at pretending we did not see things.
An agent carries tool output into its working context. Extra files, nearby ranges, broad search results, wrong paths, and speculative reads can all become part of the reasoning soup.
That has two costs.
First, the literal cost. More tokens. More calls. More time.
Second, the attention cost. The agent may spend reasoning budget on context that does not answer the review question. It may find a nearby pattern and chase it. It may comment on something technically interesting and practically useless, which is already a known genre of human code review.
This is why "just give the agent more context" is not a strategy.
Sometimes more context helps. Sometimes it turns the task into a haunted attic.
Good agent systems need context discipline. Not because we are trying to starve the model, but because the task has a shape. A code review agent anchored to a diff should not behave like a repo onboarding assistant.
Same model, same tools, different job.
Different job means different instructions.
tool instructions are api docs for the model
GitHub fixed the regression by changing the instructions around the tools.
The review agent had to start from the diff, form concrete review questions, narrow with cheap discovery, and only read focused code ranges when it knew why. If a search failed, it should retry simply. If a path was wrong, it should use glob instead of guessing neighboring paths like a tired engineer at 1 AM.
That sounds small.
It is not small.
Tool instructions are API documentation for the model.
We already know bad API docs can make humans use a good API badly. They call the wrong endpoint. They pass the wrong shape. They retry the same broken request. They build a wrapper named doThing2 and now everyone has to live with it.
Agents do the same thing, just faster and with more confidence.
A tool called view is not enough. View what? When? How much? After which kind of search? For which kind of evidence?
The model needs the workflow, not only the affordance.
This is where many internal agent projects will hurt themselves. A platform team will expose a nice shared tool layer: search, browser, tickets, logs, deploy status, maybe a few admin actions if everyone is feeling spicy. Then each product team will assume the tool layer is done.
It is not done.
The tool layer is raw capability. The agent still needs job-shaped guidance.
evals are useful when they show the path
The other good lesson from the GitHub post is that benchmarks were useful because they showed traces, not only scores.
A score saying "review quality dropped" is useful, but it does not tell you where the agent got lost. A trace can show the behavior: broad searches, wrong reads, repeated loops, and not enough anchoring to the diff.
That is the difference between measuring the outcome and debugging the system.
Agent evals that only return a grade will turn into executive dashboards. You know, the place nuance goes to become a pie chart.
Useful evals need to preserve the path: what tools were called, what came back, where the agent widened instead of narrowing, whether it read before it had a reason, and whether the final answer matched the task.
This is not only for GitHub-scale products. Any company building internal agents needs this.
If your support agent has a CRM lookup tool, you need to know whether it checks the customer record before answering. If your incident agent has log access, you need to know whether it follows the alert or goes sightseeing in unrelated services.
The trace is where the truth leaks out.
shared tools still matter
None of this means shared tools are a bad idea.
They are probably necessary. Duplicated agent tools become a maintenance swamp very quickly. Every team invents its own search wrapper, browser helper, log reader, and "safe" shell abstraction.
Shared tools are good.
But shared tools scale only when the workflow around them matches the job.
The same grep, glob, and view can support a CLI assistant, a code review agent, a security fixer, and a migration helper. But each one needs different instincts. The CLI assistant may explore broadly. The reviewer should narrow from the diff.
If we pretend one generic prompt can cover all of that, we are back to corporate magic. Very expensive. Comes with a dashboard.
the punchline
The lesson is not "AI code review is unreliable."
The lesson is more annoying and more useful:
Agent behavior lives in the combination of model, tools, instructions, context, evals, and product workflow.
Change one part and the system can improve. Or it can get worse in a way that looks irrational until you inspect the trace.
That is exactly why serious agent work is becoming ordinary engineering work. You need benchmarks. You need traces. You need job-specific instructions. You need cost visibility.
Better tools are not enough.
The agent has to know what kind of work it is doing.
For code review, that means start from the diff, ask narrow questions, gather focused evidence, and stop before the investigation becomes a hobby.
Honestly, that is not bad advice for humans either.
references
- GitHub Blog: Better tools made Copilot code review worse. Here's how we actually improved it.
- OpenAI: Running Codex safely at OpenAI
To test my projects, I use Railway. If you want $20 USD to get started, use this link.
Top comments (0)