I keep seeing this in agent-written PRs.
CI was red on a flaky step. The agent “fixed” it. The pipeline is green again. Then you read the diff and nothing about the flake changed. The real change is a retry wrapper: rerun the test three times, retry the HTTP call until it succeeds, or loop the job until it passes. Green theater with a loop.
You did not ask for patience. You asked for a stable signal.
The 60-second check I run now
When an agent PR claims to fix flakes, timeouts, or intermittent CI:
- Diff the “fix.” Did it remove shared state, fix the race, or harden the assertion — or only add retries / maxAttempts / expect.poll loops around the same failure?
- Ask what fails on the first attempt now. If the PR cannot name a root cause and only raises attempt count, bounce it.
- If the only change is “retry until green,” I reject it until the first attempt is trustworthy or the flake has a real owner.
This is one of the stops on a short bar I use before I merge agent PRs. Free one-pager if you want the rest: https://chopragunji.gumroad.com/l/zpnmdn
If you'd rather someone else do a hard pass on one scary AI PR, I still have founding slots at $49 (normally $99): https://chopragunji.gumroad.com/l/byoyi/FOUNDING
Have you merged an agent “flake fix” that was really just more retries?
Top comments (0)