Your cron job exited 0. That doesn't mean it worked.

My trading bot liquidated an entire position because of a number
that was technically valid.

The exchange API returned a balance of 0. Not an error, not a
timeout — a clean 200 with a well-formed body. My bot read it,
calculated a 100% drawdown, and did exactly what I had told it to
do in that situation: close everything.

Nothing errored. No exception, no non-zero exit, clean logs. My
monitoring said the job ran on schedule.

It just ran on garbage.

The gap between "ran" and "worked"

Most job monitoring answers one question: did the process report
back? That catches the loud failures — crashes, timeouts, dead
containers. Those are the easy ones, because something screams.

It does not catch:

• A query that returned 0 rows because an upstream table was empty
• A backup that wrote a truncated file and uploaded it successfully
• A parser that silently skipped every row after a schema change
• An API returning a valid-looking default instead of real data

Every one of these exits 0. Every one pings your monitor. Every one
is wrong.

The absence of an error is not the presence of success. We conflate
them constantly because, most of the time, they line up.

What I do now

I stopped asking "did it run" and started asking "does the result
look like a result." At the end of each run I emit a few numbers
and compare them to previous runs:

• Row count against a baseline — not a fixed threshold, a comparison to the recent average. A job that normally writes 15,000 rows and suddenly writes 400 is interesting even though 400 is a legal number. Fixed thresholds rot; nobody re-tunes them.
• Value ranges — a balance of exactly 0, a price of 0, a date in 1970. Valid types, implausible values.
• Schema hash — if the column names or order change, I want to know before the data does something strange, not after.
• Output hash vs. the previous run — if today's output is byte-identical to yesterday's, either nothing happened or I'm reading a stale source.

No warehouse, no pipeline framework. It's a dictionary of numbers
at the end of a script.

Where I'm still stuck

Where the check belongs. Right now it lives inside each script,
so every script re-implements a slightly different version of the
same logic — and the check dies with the process it's supposed to
be checking.

Proving it afterward. Knowing a run was fine is one thing.
Demonstrating later that a specific output came from a specific run
and hasn't been modified since is another. Log lines are trivially
editable.

What I'd like to know

If you run scheduled jobs in production:

1. Do you verify the output, or only that the job finished?
2. Where does that check live — in the script, a separate step, or your monitoring tool?
3. If you've hit a "succeeded but wrong" case, how long did it take you to notice?

I'm still patching this together myself, so I'm more curious what
people actually do than what the ideal setup looks like.