Two green CI runs told me the release worked. Both were no-ops.
A green check means one thing: the job exited 0. That is all it means. It does not tell you the step you cared about actually did the work you think it did. I learned this the slow way, twice in a row, on the same release pipeline.
The setup
I had wired up automated releases with python-semantic-release. The idea is clean: you write conventional commits, the tool reads them, decides the next version, tags it, and pushes. Because the release job has to push a version bump and a changelog back to a protected branch, I gave it a dedicated RELEASE_TOKEN with the rights to do that.
The wiring looked healthy. Push, pipeline runs, green check, move on.
The first no-op
python-semantic-release only cuts a release when the commits since the last tag actually warrant one. Conventional-commit types map to version bumps roughly like this:
| Commit type | Release effect |
|---|---|
feat: |
minor bump |
fix:, perf:
|
patch bump |
docs:, ci:, chore:, style:, refactor:, test:
|
no release |
My two green runs landed on a docs: commit and a ci: commit. python-semantic-release looked at them, decided there was nothing to release, and short-circuited. It exited 0 well before it ever reached the push path. The RELEASE_TOKEN, the protected-branch push, the tag, none of it ran. The job was green because it correctly did nothing.
So the part of the pipeline I most wanted to verify, that the token worked and the push to the protected branch succeeded, stayed untested. It stayed pending until the next feat: or fix: commit came along and gave the tool an actual reason to reach that code. The green check told me the release worked. What it meant was "there was nothing to release, so I stopped early."
The second no-op
The same pipeline had a build_command that produced build artifacts before the release. The bug was subtle: build_command runs your command, but it does not automatically stage the output that command produces. If your release step then commits and pushes, the artifacts the build just generated are not in that commit unless you stage them yourself.
Every per-task run went green. The build command exited 0. The release step exited 0. Nothing in a single job's view was wrong. The gap only became visible at a whole-branch review, looking at the actual diff that would ship, where the missing artifacts stood out against everything else.
What a green check actually is
Both failures share one root. A check reports the exit code of the command you told it to run. It cannot report the thing you assumed that command would do. Coverage is not correctness. A pipeline verifies the paths you encoded, on the inputs you happened to feed it. The path that breaks is usually the one that never got exercised: the integration you mocked, the config that only exists in prod, the branch of logic that this particular commit did not trigger.
"All checks passed" and "the release works" are two different statements. The pipeline can only ever prove the first one.
What I check now
I read the log, not the badge. Before I trust a green release, I confirm the specific path ran:
- Did the version actually bump, or did the tool decide there was nothing to release?
- Did the push to the protected branch happen, with the token, on this run?
- Did the staged diff contain the artifacts the build produced?
Concretely:
- Assert the effect, not the exit code. If a job is supposed to push a tag, have something downstream check the tag exists. Exit 0 is necessary, it is not sufficient.
-
Force the real path in a test. A release dry-run on a synthetic
feat:commit exercises the push path even when today's real commits would not. Then you learn the token works before you need it. - Keep whole-branch review. Per-task green catches a lot. The staging gap and the no-op release both survived every per-task check and only showed up when someone looked at the whole branch as one diff. That review is not redundant with CI. It catches the class of bug CI structurally cannot see.
None of this means the pipeline is useless. It means a green check is where the checking begins.
Has a green run ever lied to you like this?
Top comments (0)