DEV Community

tbalcewicz
tbalcewicz

Posted on

The silent zero: the worst bug your monitoring system can have

#python #automation #devops

A crash is a good outcome. Somebody gets paged, somebody fixes it, everyone moves on.

The bad outcome is the one where your system finishes cleanly, reports nothing, and is wrong. No exception. No alert. A perfectly normal-looking run that says "0 new items" - when the truth is "I could not reach the source and I have no idea what is out there."

I call it the silent zero, and it is the most expensive bug I know of in any system whose job is to watch something.

Why it is so easy to write

Nobody sets out to build this. It appears through reasonable-looking defensive coding:

def fetch_notices():
    try:
        return api.get_notices()
    except Exception:
        return []          # here
Enter fullscreen mode Exit fullscreen mode

Every instinct says this is the responsible version. It does not crash. It degrades gracefully. The report still goes out.

But look at what the caller sees. fetch_notices() returned an empty list. There are exactly two situations that produce an empty list:

  1. the source is up, and genuinely published nothing today
  2. the source is down, unreachable, rate-limiting you, or changed its response format

These are opposite situations and the code cannot distinguish them. One means "all clear", the other means "you are blind". They arrive as the same value.

How it actually plays out

A monitoring system is trusted in proportion to how boring it is. After a few quiet weeks, "nothing today" stops being read as information and starts being read as absence of information. That is exactly when it becomes dangerous.

The failure has a shape. A portal changes its HTML, or rotates a certificate, or starts requiring a header you do not send. Your collector throws. Your except swallows it. The report says zero. It says zero the next day too, and the day after.

Nobody investigates a quiet inbox. Weeks later someone asks why you missed something obvious, and you discover the source has been dark since a date you have to reconstruct from git log.

The cost is not the outage. It is that you cannot say when it started, so you cannot say what you missed.

The rule

A failure and an empty result must never be representable by the same value.

Three things follow.

1. Count failures explicitly, and make them travel 

def collect(sources):
    notices, failures = [], []
    for source in sources:
        try:
            notices.extend(source.fetch())
        except (URLError, TimeoutError, JSONDecodeError) as exc:
            failures.append((source.name, repr(exc)))
    return notices, failures
Enter fullscreen mode Exit fullscreen mode

Two return values. The caller cannot ignore the second without deciding to.

2. Put the failure in the output humans read

Not in a log file nobody opens. In the artefact itself - the email subject, the top of the spreadsheet, the first line of the report:

subject = f"Tender report - {len(rows)} new"
if failures:
    subject = f"[PARTIAL - {len(failures)} source(s) down] " + subject
Enter fullscreen mode Exit fullscreen mode

A recipient who sees [PARTIAL - 3 sources down] in their inbox knows to distrust the zero. That is the whole goal.

3. Catch narrow, and let the unknown crash 

except Exception:          # hides bugs in your own code
except (URLError, TimeoutError, JSONDecodeError):   # only expected failures
Enter fullscreen mode Exit fullscreen mode

Broad except catches your KeyError from a renamed field just as happily as a network timeout. A schema change silently becomes "no results", which is the same silent zero wearing a different hat. If you did not anticipate it, you want the crash.

4. Isolate per source, not per run

Wrapping the whole collection loop in one try/catch means the first failing source kills the other nine. Wrap each one. A run with seven of ten sources plus a loud warning is genuinely useful. A run that aborted at source one is not.

Alert on silence, not just on errors

The strongest version of this does not wait for a failure to be detected at all. It treats prolonged silence as a signal in its own right:

if days_since_last_result(source) > source.expected_quiet_days * 2:
    warn(f"{source.name}: nothing in {days} days - verify it still works")
Enter fullscreen mode Exit fullscreen mode

Sources have a normal rhythm. If one usually yields something every couple of days and has been silent for two weeks, that is worth a human glance - regardless of whether anything technically threw. Sometimes the answer is "quiet season". Sometimes it is "we have been scraping a 404 page since March".

The check that costs nothing

Open the last report your monitoring produced and ask one question:

If every single source had been down, would this report look different?

If the answer is no, you do not have a monitoring system. You have a system that sends emails.

I build and maintain automation and monitoring systems - mostly Python and UiPath, against public data sources that were never designed to be read by machines.

Top comments (0)