I keep seeing this in agent-written PRs.
The ticket said the feature was broken for some users. The agent “fixed” it. CI is green. The demo path works on the author’s machine. Then you read the diff and the dynamic lookup is gone: a config key, a feature flag, a tenant setting, or a branch that used to choose behavior is now a literal. The happy path always runs. The broken case never happens in the suite because the suite never reaches it anymore.
You did not ask for a demo that always wins. You asked for the real branch to work.
The 60-second check I run now
When an agent PR claims to fix a config, flag, or environment-sensitive bug:
- Diff for literals that replaced a lookup, flag, or env read. Ask why the dynamic path had to die.
- Re-run the failing case with the original config/flag values. Does the bug still exist under the hood?
- If the only change is “always take the happy path,” I bounce it or ask for a fix that keeps the real branch and adds a test for the previously broken setting.
This is one of the stops on a short bar I use before I merge agent PRs. Free one-pager if you want the rest: https://chopragunji.gumroad.com/l/zpnmdn
If you'd rather someone else do a hard pass on one scary AI PR, I still have founding slots at $49 (normally $99): https://chopragunji.gumroad.com/l/byoyi/FOUNDING
Have you merged an agent “fix” that just hardcoded the path that always worked in the demo?
Top comments (0)