Your health check is lying: three ways a green pipeline stays broken

A scraper pipeline I maintain stopped publishing for five weeks. Every morning the report said the same thing: logged in, ready to capture. Nothing was captured.

It was not broken in any dramatic way. It was broken the way most automation breaks: a check answered a question slightly different from the one that mattered, and the answer was always yes.

Here are the three shapes I found, and the probe rules I now apply everywhere.

1. "Logged in" measured the presence of a file

The check ran auth status --offline. That flag does exactly what it says: read the credential file, make no network request. The session behind it had been rejected with a 401 nineteen days earlier. The file was still there, so the verdict was "logged in, you can capture now".

With a cache in front of reality, liveness and existence become different questions. An offline check is a cache check.

The replacement: exactly one request to an identity endpoint, judged by status code.

GET /api/v4/me
  200 -> session is real; print the username
  401 -> "credentials expired"; print the re-auth instruction
  5xx -> "unverified"; never green

Three outcomes, never two. A check that can only say green or red will invent a green the moment the network hiccups.

2. A moved environment broke a path, and the break was silent

The CLI package lived on the C: drive. A toolchain migration moved the package root to D: and set an environment variable pointing at it. Our wrapper still resolved the old absolute path.

What kept anyone from noticing is the interesting part: the wrapper caught the launch error and logged it at debug level, so the daily job reported a successful run that produced zero items. Nothing failed. Nothing worked.

Two changes fixed it. Resolve the path from the environment variable, documenting the legacy default as a fallback. And make "binary missing" a hard failure instead of a zero-row success. Zero rows is a result; a missing binary is not.

3. The gate was measuring the instrument, not the channel

Our channel gate called fetch('https://dev.to/api/users/me') and the call failed. The credential was fine. The API was fine. The host simply does not resolve from this network without the local proxy — and Node's built-in fetch ignores system proxy settings unless you wire a dispatcher. A one-line curl -x proved reachability in under a second.

So the gate printed "API channel failed": an accurate observation about the instrument and a wrong conclusion about the world. Run on a schedule, it would have blocked every publish for a month, honestly.

The rule now: a probe must name the layer it failed at. DNS, TCP, TLS, HTTP, auth, parse — six boxes, and only the ones that actually answered get printed. "Failed" is not a diagnosis.

The pattern: probes need positive evidence

All three bugs are one bug: a success criterion that can be satisfied without doing the thing it claims to verify.

• presence of a credential is not a valid session
• a zero-row success is not a working pipeline
• a reachability failure is not a channel failure

The probe rules that survived:

1. Every probe touches reality: one real request or one real DOM read per claim.
2. Verdicts are three-valued: pass, fail-with-reason, unverified. Unverified never renders as pass.
3. A probe reports the layer: what answered and what did not.
4. Failures name the next action: which human, which window, which click.
5. Store the smallest evidence that proves the verdict — a username, a status code, a byte count — not the whole response body.
6. When a write depends on a probe, the write stays blocked until the probe passes. "Materials ready" is not "executable".

None of this is exotic. It is the difference between asking "does the file exist" and asking "do I actually work right now".

A pipeline that reports green for five weeks is more dangerous than one that crashes on day one. The crash gets fixed the same morning; the green one gets trusted.