Silent failure
For three weeks nobody could install your VS Code extension. The GitHub Actions publish job was green every single day. You find out when a user asks why the version is so old.
Three weeks green, zero installs
This happened to us. The publish job for our JetBrains plugin reported success on every run since the middle of July. The plugin was not in the marketplace at all.
The registry API answered with a 404. A search for the product name returned nothing. Meanwhile the build was green, the release notes were written, and the changelog was up to date.
Nobody noticed. Not the pipeline, not the dashboard, not us. The gap between the last good release and the discovery was three weeks.
Your pipeline is not lying to you
This is the part worth understanding, because it is why the same thing is probably waiting in your repository too.
A release pipeline has one job: get the artifact somewhere a stranger can install it. Almost every pipeline checks something else. It checks that the upload command exited zero.
Those two questions agree nearly always. Our publish step went further and did the sensible thing. It caught the failure, compared the error text against a list of known-harmless cases, and exited zero for those.
One of those cases was pending moderation. A new version sits in review before it becomes visible. Failing the build for that would be noise, so it was allowed through.
Here is the trap. A harmless transient state and a permanent block produce the same message. Once the plugin was stuck, every later run matched the same friendly pattern and reported success. The pipeline answered its question correctly. It was the wrong question.
The check that catches it, in about two minutes
Ask the store, not the pipeline. That is the whole idea, and you can add it today without changing anything else.
One: after publishing, fetch the public listing the way a stranger would. No credentials, no internal API, no authenticated client. Seeing what an outsider sees is the entire point.
Two: read the version from that response and compare it to the version in your manifest. Not to the version you just built, and not to a variable in the job. To the file that is checked in.
Three: print one line per place you publish to, and say which one you could not reach. Reached, not reached, or stale. Never a bare zero.
That last part matters more than it looks. A source that errors out and gets counted as zero results looks exactly like a quiet day.
Make it fail on purpose, before you trust it
A watcher nobody has seen go red is not a watcher. It is a hope with a log file.
Give it a fixture where the listing is stale and assert that it exits non-zero. Run it once and look at the red. It takes a few minutes and it is the only thing that separates a real check from a comfortable one.
We skipped this step once on a different watcher. It sat there for weeks reporting nothing, and reporting nothing was indistinguishable from everything being fine.
Once you see this shape, you see it everywhere
A cache file written into a workspace that the next run wipes clean. A daily report whose delta is computed against a seeded example row. A source that returns an HTTP error and gets counted as zero.
Every one of them is green. Every one of them delivers nothing.
The missing ingredient is always the same: a third state. Systems model success and failure and forget not measured. Without it, an outage is indistinguishable from a quiet day, and people act on the number anyway.
What changes for you
Before: you find out from a user, three weeks late. After: you find out the next morning, from a check you wrote in two minutes.
It also catches the one case no exit code can reach. Sometimes the upload really did work and the store still shows the old version. Your pipeline has no way to see that. A stranger looking at the page does.
Ours is about forty lines. It opens three public pages, one for each store we publish to. It reads the version shown on each page and compares it to the version number in the file we ship. No login, no API key, no access to anything private.
The day we switched it on, it went red. It had been red for three weeks and nobody knew.
I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.
Free tier, hosted in the EU: cachly.dev
Top comments (0)