The commit that fixed my only security advisory failed its security audit. The audit never ran.

On 4 September I cut two releases of safari-mcp ten minutes apart. The second one exists because of a mistake in the first, and its CI run went red for a reason that had nothing to do with either.

Here is the sequence, because the order is the point.

Release one: shipped with the alerts open

v2.18.0 was a real release: receipts that survive a reconnect, screenshot downscaling, a queue for parked extension workers. CI was green, 227 tests passed locally, the changelog was written. I tagged it.

What I had not done was look at the Dependabot tab. There was one open PR there, bumping qs from 6.15.3 to 6.16.0 to close two moderate advisories. Lockfile-only, CI green, sitting there for a day. I released around it. So the fresh version that landed on npm carried two advisories that anyone running npm install would see in the summary line.

I noticed within minutes, merged the bump, waited for CI on main, and cut v2.18.1 with a one-line ### Security entry. The release workflow ran, npm got the new tarball, the registry picked it up. Fine. Lesson written down: merge the green Dependabot PRs before cutting, not after.

Release two: red CI on the commit whose only job was security

Then the CI run for that release commit failed.

The workflow runs the test matrix on three Node versions. One of them failed at this step:

- name: Audit production dependencies (fail on high+)
  run: npm audit --audit-level=high --omit=dev

The log:

npm warn audit 503 Service Unavailable - POST https://registry.npmjs.org/-/npm/v1/security/audits/quick
npm error audit endpoint returned an error

The npm ci before it had taken seven minutes. npm was having a bad morning. The audit endpoint was unreachable, npm audit exited non-zero, the step failed, and because the matrix has fail-fast on by default, GitHub cancelled the other job that was still running. Final tally on the commit that fixed my only security advisory: one job passed, one failed, one cancelled. Red.

The same thing happened again that afternoon on the next commit, on a different Node version. Two days later I re-ran both without changing a line. All green.

What the gate actually measured

I put npm audit in CI so that a release cannot go out with a known high-severity vulnerability in production dependencies. That is a reasonable thing to want. But look at what the step can and cannot tell me:

• Exit 0: the audit ran and found nothing at high or above.
• Exit 1: the audit ran and found something — or the audit could not run at all.

Those two are not the same finding, and the step collapsed them into one red X. On 4 September the repository had zero open Dependabot alerts. There was nothing to find. The gate went red anyway, because it is a network call to a third party, and the third party was down.

A check that fails when it cannot run is not a gate. It is a coin flip weighted by someone else's uptime. And the weighting is not small: npm ci plus npm audit are two round trips to the same registry, in the same job, with no retry between them.

fail-fast then made it worse in the specific way that amplifiers do. One transient error on one Node version became three red jobs, two of which had no opinion about anything. The whole point of a matrix is to learn something per version. Cancelling the siblings on the first failure throws that information away exactly when it would have been useful, because the one thing a green sibling would have told me is "this is not your code".

Why nothing was stopped

Here is the part I am least comfortable with. The red CI did not block the release. The release workflow triggers on the tag, not on CI status, and I had cut the tag after checking that the previous commit was green. So v2.18.1 was on npm before its own CI had finished, and the CI then went red on a commit that was already published.

I had thought of CI as a gate. It was a lagging indicator. A gate that runs after the door has closed is documentation, and documentation that is wrong is worse than none — for two days the commit history said "this release failed CI", and it had not failed anything.

The fix

Two changes, both small, both shipped this morning:

strategy:
  fail-fast: false

so each Node version reports its own result, and the audit step now tells the two failure modes apart:

set -o pipefail
for attempt in 1 2 3; do
  if npm audit --audit-level=high --omit=dev 2>&1 | tee audit.log; then exit 0; fi
  grep -q "audit endpoint returned an error" audit.log || exit 1
  echo "npm audit endpoint unavailable (attempt $attempt/3) — retrying in 30s"
  sleep 30
done
exit 1

A real finding still fails immediately. An endpoint error retries three times, then fails — which is correct, because after ninety seconds of no answer I genuinely do not know whether the release is clean, and "I don't know" should be red. What I have removed is the case where a single 503 pretends to be a vulnerability.

The thing I have not fixed is the ordering problem: the release still does not wait for CI on its own commit. That is a bigger change to a workflow that has finally been reliable for two weeks, and I am not going to touch it on the same morning I am writing about touching things too fast.

The question

The two mistakes on 4 September are opposites. The first was a gate I forgot to look at (Dependabot). The second was a gate that looked at nothing and reported anyway (npm audit during an outage). I over-trusted one and under-checked the other, and the release that came out of it was fine — which is the uncomfortable bit, because "it was fine" is how these habits survive.

So: when a CI check depends on an external service, do you fail the build when that service is down, or skip the check and mark it? I chose "retry, then fail", on the argument that an unknown security state should be red. I can see the case for a yellow.