The PR looked great.
The agent had pulled a 40-line function out of a 1200-line file, split it into three small helpers, named the variables better, and added docstrings. Every existing test passed. The diff was +80/-35. I clicked approve and went to lunch.
That afternoon, p95 on the main endpoint went from 320ms to 610ms.
The function wasn't hot in the obvious path. It was hot in the not-so-obvious one: it ran once per item in a 40k-item batch. The refactor had turned a single pass over the array into three passes — one to filter, one to map, one to reduce. The math looked the same. The constant factor didn't.
This is the refactor trap that AI coding tools walk into every single time. They optimize for what the diff looks like in GitHub, not for what the function costs at 4 a.m.
Why the tests didn't catch it
Existing tests assert behavior, not cost. They say "given input X, output equals Y." They don't say "and it takes under 2ms." A refactor that changes runtime by 2x but preserves every output value is a green CI run.
The agent doesn't know how often the function is called. It doesn't see the 40k-loop somewhere upstream. It reads the function in isolation and does what any of us would do if we forgot the call frequency: make it pretty.
The cheap fix that stuck
We started asking the agent one thing before any refactor of a non-trivial function:
"How many times is this called per request, and what's the rough input size?"
Sometimes it knows (the caller is in the same file). Sometimes it doesn't — and when it doesn't, that's the signal to pause. If the agent can't answer the call-frequency question, the refactor is a guess.
The other change: we kept a 5-line benchmark in the repo for the five hottest paths. Not a full load test — just a bench script that runs the function 10k times and prints median. It takes 200ms to run. The agent is told to run it before and after any refactor touching those paths. If the median moves more than 20%, the PR gets rejected automatically.
The bigger pattern
Performance regressions from AI refactors are silent because the tool has no model of cost. It has a model of readability, naming, structure, and test-passing. It does not have a model of "this runs 40k times per request."
You can fix that by giving it the missing context — call counts, hot paths, the one benchmark file — or by putting a cheap guardrail after the diff. Either works. Both is better.
The diff that looks the cleanest is often the one that's quietly costing you.
Top comments (0)