The check that could not fail

Context

There is one cheap way to ask python-semantic-release whether it will write your changelog. It answers would have written your changelog to CHANGELOG.md.

It answers that for a setup that writes nothing. Mine wrote nothing for thirty releases.

A few weeks ago I wrote Green CI ≠ working software. The argument there was that a green badge proves the job exited 0 and nothing more, so you should stop reading badges and start asserting effects.

This post is what happened next. I went and looked at the checks I was relying on to tell me an effect had happened. Four of them were pointed at the wrong thing. I had built two of them myself.

TL;DR: the first post was about the badge. This one is about the layer above it. You build a real check, its scope is wrong, and its silence still means nothing. Before you trust any quiet signal, ask what failure would make it warn you. If you cannot answer that, you do not have a check. You have a ritual.

They all said it worked

Four of them, in my own systems. A release tool, a shell pipeline, a monitor, and a security dashboard. Different tools, different years, same shape.

Each one reported success. Each one was correct about something. None of them was correct about the thing I needed.

That gap is easy to miss, because a check that passed and a check that never looked produce the same output. Both are silent. Both look like good news.

Case 1: thirty releases, no changelog

CHANGELOG.md had 6 entries. The newest was v0.3.0, from July 6. After the fix it had 36.

So thirty releases wrote nothing. That is v0.4.0 through v0.14.6.

The rest of the release worked fine. Tags were pushed, GitHub Releases were created, PyPI got the files. Only the changelog step produced nothing.

The cause is one missing line. python-semantic-release v10 writes the changelog in update mode. It inserts new text after a marker line inside CHANGELOG.md. The marker is called the insertion_flag. My CHANGELOG.md did not have that line: the file was last written under v9, before an upgrade changed how the marker was configured. So v10 had nowhere to insert, and it inserted nothing.

It printed no warning. It exited 0.

The fix is this config plus the same marker line pasted into CHANGELOG.md. The string in the config and the line in the file are a matched pair:

[tool.semantic_release.changelog]
mode = "update"
insertion_flag = "<!-- version list -->"

Back to the check from the opening. It is the only cheap way to ask this tool whether it will write your changelog:

semantic-release --noop changelog

It prints would have written your changelog to CHANGELOG.md.

It prints that for a setup that writes nothing. The message goes out before any insert is attempted, and the insert is the step that breaks. A dry run never reaches it. I reproduced this against version 10.5.3.

That check could never say no. It ran thirty times and was never once useful. A person asking for the release notes found the bug.

Case 2: three gates, none of them able to stop anything

While fixing the above, my own verify step looked like this:

uv run ruff check | tail -1 && uv run pytest | tail -1 && ...

A pipe returns the exit code of its last command. Here that is tail, and tail exits 0 no matter what the command before it did.

So ruff exited 1, nobody saw it, and the chain moved on and committed.

Three gates in a row. Not one of them could stop anything. The repair is small: drop the | tail -1 from each gate, or run the chain under set -o pipefail. I kept that repair as its own commit instead of hiding it in another one, because the near miss is worth keeping.

Case 3: the monitor that watched the wrong tools

This one is a monitor, not a CI job. That makes it harder to spot. A CI job shows you a green tick you can click. A monitor shows you nothing at all, and nothing is also what success looks like.

A monitor that never alerts is a check that always passes.

My AI agent setup has a cost monitor. After every tool call it measures the size of the result, adds it to a running total, and warns me when the total gets high. I built it because large context builds up slowly from many small results, so only a running total can catch it.

It never warned me. That was not good news.

The monitor was wired to three tool types: Agent, Task and Workflow. Nothing else. So it measured sub-agent results and never saw Read, Bash or Grep.

I already knew where the size came from. In the expensive session that made me build the monitor, Bash returned about 400K characters over 623 calls and Read about 258K. Sub-agent output was about 53K. The monitor watched the 53K.

The number was never wrong. It measured everything it could see. Meanwhile the comment right above the code said the monitor covers "all tools". The comment described what I meant. The config described what ran. Nobody compared them.

I found it by asking one blunt question about the code: which events is this wired to, and what can it actually see? I asked for line numbers instead of a summary. The answer took one line.

Case 4: the dashboard that read the wrong branch

The first three cost only me time. This one cost a team.

A security compliance dashboard showed gaps. Test coverage missing for several services. Scan results that did not match what we knew. The first guess was a broken scanner. The scanner was fine.

The uploads ran from feature branches. The dashboard filtered on main. So whole categories of results were simply not shown. Empty and unknown branch names slipped through the filters too, and dropped more rows.

The scan results were right the whole time. The branch label that routed them was wrong. Nobody was checking that label.

The fix: make the upload step report the main branch name, add unknown to the branch filters, and stop writing empty branch columns.

If a compliance dashboard looks wrong, check the filters before you check the scanner. Empty and unknown values are a common way to lose rows quietly.

The one question to ask

Fixing my changelog fixes one repo once. So the same commit added a changelog-check job. It runs after a release and checks that the release commit really touched CHANGELOG.md.

That job runs after the artifact exists, and it looks at the artifact. A dry run can never do this, because a dry run happens before the artifact exists. It has only the plan to report on, so the plan is what it reports.

So here is the question to ask about any check you trust. Does it look at the result, or only at the plan for the result? The cheap check failed me because it could only ever see the plan.

Three habits come out of these four cases:

• Ask what would have to go wrong for this check to fail. No answer means no check.
• Read the filter, not just the metric. Every monitor has one. A matcher, a label selector, a log filter, a branch filter, a sampling rule. The filter decides what the number means, and it is the part nobody reviews.
• Write one test that proves the monitor can see its subject. Not that it fires at the limit. That it sees the category at all. Mine would have failed on day one.

One last thing

Writing this up, I got my own number wrong. I said eleven releases. The tags say thirty.

That wrong number travelled through a retro page and two follow-up notes before I counted the tags. It is also still sitting in the comment above the config that fixes the bug, in the repo, right now, as I publish this.

The write-up of a counting failure contained a counting failure. My memory reported the plan too.