Oussama Khorchany

Posted on Nov 7

Do Coding Agents Really Fail at Business Logic? What the Evidence Shows

#ai #discuss #programming #productivity

Why I started this

I set out to test a suspicion, not to sell a tool: coding agents fail at bug fixing because they do not understand the business behind the code. After a day of digging, my answer is that the problem looks real but narrower than I assumed, and the obvious fixes are weaker than they sound.

I use Claude Code with multiple agents and subagents, and I have designed my own software factory, EGM. Bug fixing is where I get the most frustrated. Agents misread the real cause of a bug. They ship patches that are technically valid but break a business rule. They miss what a change does elsewhere in the app. And context gets lost when work is delegated to subagents.

One honest note: I could not recall a specific bug to use as a case study, so I will not invent one. Everything below is what other people measured or reported, plus what I could and could not verify. I did the research with two AI assistants, Claude and ChatGPT, and asked them to challenge each other's findings.

What the large studies show

The big studies agree that agents do worse on bug fixes, but none of them shows that missing business logic is the reason.

Merge rates by task type. Where Do AI Coding Agents Fail? (MSR 2026) studied about 33,000 pull requests from five agents: Codex, Copilot, Devin, Cursor and Claude Code. Documentation and CI tasks merged most often. Performance and bug-fix tasks merged least. A third-party summary of the paper puts bug fixes at 64% and documentation at 84%.
Why PRs were rejected. In a sample of 600 rejected PRs, the most common cause was reviewer abandonment (38%), then duplicate or unwanted PRs. Problems on the agent's own side were about 2%. These figures also come from the same third-party summary, so check them against the paper.
Fixes only. A second study looked at bug-fix PRs alone and found that 46.41% of fixes from Copilot, Devin, Cursor and Claude were rejected. It sorts the reasons into four groups: wrong implementation, failing CI, agent capability limits and low priority.
After merging. A study of 1,210 merged bug-fix PRs ran SonarQube before and after each merge. Code smells dominated. Real bugs were rarer but often severe, and the most common one was a function called with the wrong number of arguments. The authors conclude that merge success is not a reliable quality signal. Caveats: no human baseline, and 949 of the 1,210 PRs came from Codex but only 15 from Claude Code.

What none of these measure is whether a fix violated a business rule. The datasets label PRs by task type, CI result and review outcome, and there is no business-rule category. That is not proof the problem is small. It means the evidence is indirect, and I did not find a benchmark built to measure it (my search could have missed one).

What developers report

Practitioners describe the same pattern again and again, but these are anecdotes, not measurements.

One commenter in a Hacker News thread says an agent understands a repo functionally, not from the business side.
In Ask HN: I hate coding agents, a developer who writes docs, AGENTS.md files and skills says the agent does about 80% right, and its wrong assumptions make the last 20% hard to fix.
Another thread describes agents fixing each bug while breaking other things, because nobody asked why the code looked the way it did.
A third says coding assistants bury requirement gaps inside large diffs instead of escalating them to a person.

There are counterpoints. One commenter finds agents very good at debugging complex bugs that span many files, and weaker at deciding whether a change belongs in the architecture or only locally. Others blame long, degraded context, as in this Claude Code issue.

These comments are self-selected and unmeasured. I treat them as hypotheses to test, not as proof.

What existing tools cover

Existing tools handle navigation, impact analysis and workflow well, but none I found measures whether a patch respects business rules.

Tool	What it does	Business-rule angle	Adoption signal
Repo Mind	Repository understanding through retrieval and graph summaries; a GitHub Next research project, listed as completed	None	Its own tests: small gain (see below)
GroundGraph	Rust CLI and MCP server linking requirements, docs, tests and code; AI proposes business-logic candidates, a human confirms	Closest design: human-confirmed rules	Very early; the repo page showed 0 stars when I looked
GitNexus	Code knowledge graph with blast-radius analysis over MCP	None, structural only	About 42,000 stars (June 2026); license reported as PolyForm Noncommercial
Serena	Symbol-level search and editing over language servers	None	About 25,000 stars, 170+ contributors, MIT
Unblocked	Commercial engine over code, PRs, Slack, Jira, Confluence	Cross-source decision context	Vendor claims only
CodeAlive	Commercial graph-based context engine over MCP	Code only	Vendor and comparison-site numbers
Superpowers	Skills-based workflow: spec, plan, tests, subagent review including a spec-compliance check	Process, not knowledge	Reported at hundreds of thousands of stars; I have not verified it

The Repo Mind results are the most useful data point here. On SWE-bench Pro its context raised the resolution rate from 44.97% to 46.09%, a gain of about one point. The gain was larger for consistency: pass^3 rose by 6.7 points. But the agents called its structural tools in only about 8% of Pro tasks and 18% of Verified tasks, and wrapping those tools in a subagent did not help. Good context does not help if the agent does not use it.

The narrowest gap I can name

The one thing I could not find a tool for is capturing business rules that exist only in people's heads, in a form a patch can be checked against.

More context alone does not seem to be the answer. A controlled preprint (single author, 70 multi-file SWE-bench Verified tasks, Python, single-shot, so a narrow test) found that natural-language summaries of code answered 4 of 45 behavioral questions, against 27 of 45 for the source itself. Compact structural views of the surrounding code resolved no more issues than dropping that code. Instruction files are no sure fix either: in a study of 15,549 agentic PRs from 148 projects, 27.7% of projects raised their merge rate by 20% or more after adding them, while 26.35% saw it fall.

If a rule is not in the code, the tests or the docs, retrieval cannot surface it. GroundGraph's design, where the AI proposes candidate rules and a human confirms them, is the closest I found. Even so, its own documentation notes that test verification is declared, not proven.

My hypothesis, and it is only that: a rule captured as an executable check at the moment a human discovers it, for example during a bug fix, may work better than prose docs. The evidence against it is real. Agents rarely call external tools, and people have to write and confirm the rules.

What I could not verify, and what would change my mind

My current position is that the problem exists, but the evidence that a separate tool would solve it is insufficient.

What I could not verify:

Several leads from my starting notes did not turn up under those titles, so I left them out: a Google post on agent distraction, a paper called "Finding Needles in Code Haystacks", and specific Codex issue numbers.
I did not review Reddit, and I did not read GroundGraph's own benchmark results or its issue tracker.
Star counts and licenses come from third-party pages and may be out of date.

What would change my mind:

A log of real failures showing that business-rule violations are a small share next to wrong root causes. Then the gap is not worth a tool.
A benchmark showing that an external rule store raises fix correctness without adding review work. Then it is worth building.
Evidence that developers will not write or confirm rules. Then adoption is the blocker.

Help me test this

If you have a real case where an agent's bug fix was technically right but wrong for the business, I would like to hear it. Reply with:

What the bug was.
What the agent changed.
Which rule the fix violated.
Where that rule lived: code, a test, a doc, or nowhere.
How you found out.
Which agent and model you used.

If enough cases come in, I will share a tally of where the missing rules were.

DEV Community