DEV Community

Bcrypto
Bcrypto

Posted on

My monitor said "all clear" while it was completely blind

#bugsmash #monitoring #javascript #reliability

I have a small watcher script. Its whole job is to notice when one specific web
page changes — a rewards program that, when it announces, opens a claim window
that closes in about 90 days. Miss the announcement, miss the window, miss the
money. So the watcher runs daily and tells me: changed, or no action needed.

One morning I read the logs and found this, the day before:

INCOME WATCHERS — no change
No action needed.
Enter fullscreen mode Exit fullscreen mode

Reassuring. Except the line above it, from a different run, was:

⚠ fetch failed — fetch failed
⚠ fetch failed — fetch failed
Enter fullscreen mode Exit fullscreen mode

Both of the watcher's fetches had failed that day. And it had still printed
"No action needed" and exited 0.

Why that's the worst possible bug for this program

A monitor has exactly one promise: if something happened, I'll tell you. The
one output it must never produce is a confident all-clear while blind.

Mine did. "I checked and nothing changed" and "I couldn't check at all" came out
identical — same message, same exit code, same green. If the announcement I'm
watching for had landed on a day my flaky connection dropped two requests, the
log would have said "no action needed" and the 90-day clock would have started
without me.

Here's the code that did it:

let html;
try {
  html = await fetchText(w.url);
} catch (e) {
  lines.push(`  ⚠ ${w.label}: fetch failed — ${e.message}`);
  continue;   // <-- skip this surface and carry on
}
Enter fullscreen mode Exit fullscreen mode

continue skips a failed surface, and the run ends by reporting on the surfaces
it did read. Zero surfaces read, zero changes found, "no action needed." The
logic is technically correct and completely useless.

The fix

Two ideas. First, a transient failure isn't a verdict — retry before believing
it. Second, and the real point: "could not check" is a distinct state from
"nothing changed," and must never render as the second.

// Retry: transient DNS/TLS blips must not be mistaken for "nothing changed".
let html, lastErr;
for (let attempt = 1; attempt <= 3; attempt++) {
  try { html = await fetchText(w.url); lastErr = null; break; }
  catch (e) { lastErr = e; if (attempt < 3) await sleep(attempt * 3000); }
}

if (lastErr) {
  const fails = (state[w.id]?.consecutiveFailures || 0) + 1;
  state[w.id] = { ...state[w.id], consecutiveFailures: fails };
  failures.push({ label: w.label, error: lastErr.message, consecutive: fails });
  lines.push(`  ⚠ ${w.label}: COULD NOT CHECK (${fails}x) — ${lastErr.message}`);
  continue;
}
Enter fullscreen mode Exit fullscreen mode

And the output now refuses to say all-clear when any surface went unchecked —
even in --quiet mode, because silence there is the whole bug:

if (failures.length) {
  console.log('\n  ⚠ DEGRADED — could not check ' + failures.length + ' surface(s).');
  console.log('    This is NOT "nothing changed". An announcement could have landed unseen.');
  const worst = Math.max(...failures.map(f => f.consecutive));
  if (worst >= 3) {
    console.log('    🚨 ' + worst + ' runs in a row failed. The watcher is effectively OFF.');
  }
  process.exitCode = 20;   // distinct from 10 (real change) so a cron can tell them apart
}
Enter fullscreen mode Exit fullscreen mode

Three states now, three exit codes: 0 checked-and-clear, 10 something
changed, 20 couldn't check. A consecutive-failure counter means a watcher
that's been quietly dead for days announces it instead of blending into the
green.

The lesson

Absence of an alert can mean two very different things: nothing happened, or
I'm not watching anymore. Any monitor that renders both as the same calm
"all clear" isn't reassuring — it's the most dangerous kind of broken, because
it fails in the one direction you'll never think to check. Make "I couldn't
look" loud, and never let it wear the same face as "I looked, we're fine."

From the little automation stack behind
a project of mine. The watcher is boring.
Boring is the point — right up until it lies to you.

Top comments (0)