I keep seeing this in agent-written PRs.
A script step was failing. The agent “fixed” it. The pipeline is green again. Then you open the diff and the bug is still there. The real change is exit-code theater: command || true, set +e, continue-on-error: true, or a shell that always exits 0 after the failure. Green because nobody is listening.
You did not ask for a quieter runner. You asked for a real signal.
The 60-second check I run now
When an agent PR claims to fix a failing CI step, script, or job:
- Diff the “fix.” Did it fix the underlying command, or only swallow the exit code with
|| true/set +e/ continue-on-error / an always-zero wrapper? - Ask what fails now if that step is red for real. If the PR cannot name a failure mode that still stops the pipeline, bounce it.
- If the only change is “make this step always succeed,” I reject it until the failure is owned or the step is intentionally optional with a documented reason.
This is one of the stops on a short bar I use before I merge agent PRs. Free one-pager if you want the rest: https://chopragunji.gumroad.com/l/zpnmdn
If you'd rather someone else do a hard pass on one scary AI PR, I still have founding slots at $49 (normally $99): https://chopragunji.gumroad.com/l/byoyi/FOUNDING
Have you merged an agent “CI fix” that was really just || true?
Top comments (1)
The distinction in step 2 is the one that matters most in review: a step can be legitimately non-blocking, but then its failure still needs a visible signal somewhere, like a job summary, an artifact check, or a separate required check downstream. Otherwise continue-on-error just moves the failure out of sight. Do you treat a documented reason as enough to accept || true temporarily, or should that temporary version also need a named owner and an expiry so it cannot quietly become permanent?