My monitoring tool lied to me and secretly deleted all past data integrity issues

Hey there, it's your resident grandpa coder here. I'm 38 and I spend my evenings and weekends tinkering with AI agents and automated trading bots.

Today, I want to talk about how my homemade monitoring tool betrayed me. And I mean, a pretty catastrophic betrayal. My watchdog, designed to detect data integrity issues in my bots, quietly switched sides, reported "everything's fine!", and silently wiped out all records of past, unresolved issues.

These kinds of "silent failures" are genuinely terrifying. By the time you notice, it's often too late. This is a record of how overconfidence in my own code led to my monitoring tool actively concealing the very problems it was supposed to monitor—a truly gnarly self-contradiction.

The Discovery: Slack Went Quiet

Each of my bots collects and stores specific data daily. Gaps in this data throw off all downstream analysis and trading logic. So, I had a run_watch script constantly running, designed to instantly notify me via Slack if any data gaps were detected.

This script was quite effective. When I first implemented it, I'd frequently get notifications like "Data missing for 2023-10-26!". This helped me quickly catch API changes or my own bugs.

But then, at some point, the notifications just… stopped.

"Oh, my bots have been stable lately. Looks like I've fixed all the data integrity issues," I thought, blissfully unaware. I even started to wonder if my coding skills had improved.

Then one day, I happened to peek directly into the database. And there they were: data gaps, plain as day. Some were from a week ago, others much older, all unresolved. And Slack? Not a peep.

This was bad. It was like my watchdog wasn't just neglecting its duties, but actively colluding with the burglars.

The Cause: Self-Incriminating Logic in the Monitoring Tool

I immediately dug into the run_watch code. Its logic was roughly like this:

1. Scan each bot's data store to get a list of missing dates (missing_dates).
2. Load a list of previously notified but unresolved missing data issues, saved in unresolved_issues.json.
3. Compare the current scan results (missing_dates) with the unresolved_issues.json list.
4. If new missing data is found that's not in unresolved_issues.json, notify Slack and add it to the list.
5. Conversely, if an issue is in unresolved_issues.json but not found in the current scan, assume it's "resolved" and remove it from the list.

The problem lay in logic point #5 and a specific function supporting it.

Right at the heart of the problem was this _oldest_expected() function:

// Problematic code: Supposed to return the 'oldest expected date', but returned the 'oldest currently missing date'
def _oldest_expected(report, bot):
    ms = e.get("missing") or []
    return min(ms) if ms else "0000-00-00"

Based on its name, this function should return the oldest date for which a bot is expected to have data. For instance, if a bot started collecting data from 2022-01-01, it should return that date.

But look at the implementation. All it does is e.get("missing"), which means it's returning the oldest date from the list of missing data found in the current scan. If no missing data is found at all, it returns "0000-00-00".

This discrepancy between the name and the implementation created a critical bug.

If, for some reason, the bot's data source API was temporarily unstable, or the network was flaky, the scan process might return "zero missing items."

At that moment, run_watch would interpret it like this:

"Oh, no missing data found in this scan. That means all those past, unresolved issues stacked up in unresolved_issues.json must be resolved! Alright, I'll clear them all out!"

And just like that, records of past data gaps, still very much present in the DB, were completely wiped from the monitoring list. The monitoring tool was actively deleting its own evidence. The tool meant to prevent silent failures was creating the most silent and insidious kind of failure. Irony, indeed.

The Fix: When in Doubt, Do Nothing (Fail Safe)

The fix was simple.

I changed the responsibility of the scan process itself to return information about "how far back it actually scanned." The monitoring tool then uses this "scan range" to determine if an issue is resolved.

Specifically, each bot's scan function scan() now returns expected_first (the earliest date it actually scanned) along with the list of missing data. If the scan range can't be confirmed due to API issues or similar, it returns a future date like "9999-99-99".

The monitoring tool now checks this expected_first. Only if an unresolved issue date in unresolved_issues.json is newer than expected_first does it consider it "resolved."

If expected_first is a future date (meaning the scan range is uncertain), it resolves nothing. It defaults to the safe side. This simple change prevents temporary scan failures from wiping out all historical logs.

Lessons Learned: Don't Over-Trust Your Own Tools

I learned three things from this incident:

1. Function names must perfectly match their implementation. _oldest_missing_found_in_this_scan would have been better than _oldest_expected, even if it's a ridiculously long name. When a name lies, you end up deceiving yourself.
2. A monitoring tool is itself a single point of failure. If it goes silent, everything goes dark. Perhaps I need a meta-monitoring system that considers "no notifications" an anomaly. Something like a DIY Dead Man's Snitch.
3. Proving a negative is hard. A report of "no missing data" needs to distinguish between "there genuinely is no missing data" and "we failed to detect any missing data." A fail-safe design that "does nothing" when uncertain is especially critical for monitoring systems like this.

Ultimately, the person you should trust least with the code you write is yourself. This incident was a stark reminder that when things feel "stable lately," that's precisely when you should be most suspicious that some silent time bomb is ticking away.

---

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*