My health check passed for weeks while reading nothing

One of my monitoring checks reported "all clear" on every run for weeks. It was reading an empty list and calling that good news.

Here's the whole bug:

// The alarm: is any real customer on our email provider's suppression list?
const res = await fetch(
  "https://api.provider.com/v3/suppressions?limit=500",
  { headers: { "api-key": key } },
);
const body = await res.json();
const blocked = body.contacts ?? [];        // <-- the bug lives here

The provider caps limit at 100. Ask for 500 and you get HTTP 400 with {"code":"out_of_range"}. There's no contacts key on an error body, so ?? [] turned a failed request into an empty list, and the step went off and reported:

ok | suppression list holds no real customer — 0 blocked

Zero blocked. Beautifully green. It had never successfully read the list in its life.

The thing that gets me is that it would have gone green in exactly the same way if every single customer had been suppressed. The output had nothing to do with the thing it was watching, and no way to say so.

That's worse than having no check at all, because the green tick is doing active harm — it's answering a question you now think you've covered.

These are hard to spot. They never fail, so they never show up in an incident. They never flap, so they look like your most stable check. And the code reads fine — ?? [] is idiomatic, it looks defensive, it's the kind of line you skim past approvingly. Reading the diff would not have caught this.

What caught it was a second check disagreeing.

I'd just added a different step that picks a genuinely suppressed address off the live list and asserts our login form refuses it with a reason. It ran in the same fifteen-minute cycle and found one. The older step, same run, said zero. One check saying "here's one" next to another saying "there are none" is what made it visible.

The fix is two lines and slightly embarrassing. Cap the limit at 100 and paginate. And treat a failed read as a failure:

if (res.status !== 200) {
  return {
    ok: false,
    detail: `provider would not return the list (HTTP ${res.status})`,
  };
}

An alarm that can't read its input has to say so. "I don't know" and "nothing is wrong" are different answers and only one of them is safe to paint green.

The other thing I'd do differently: test the check against something you know is broken. Every one of these I've found had never been run against a true positive. If your alarm has never once fired on purpose, you don't actually know it can.

I made that exact mistake twice in one day, incidentally. The replacement check I wrote first looked up a single address and treated HTTP 200 as suppressed — except that path 404s for suppressed and healthy addresses alike, because it only exists for DELETE. Would have reported "fine" for every customer, forever, without ever failing. I only found out because I pointed it at an address I already knew was on the list.

If you want to go looking tonight, the shape to grep for is anywhere a failure can be read as an empty or zero result. ?? [] on a parsed response. data?.rows?.length ?? 0 where 0 means healthy. A try/catch returning a default. Any count-based alert where "we got nothing back" and "there's nothing to report" produce the same number.