DEV Community

Hive80-lab
Hive80-lab

Posted on

The 2 a.m. cron job nobody watches (and the one-line fix)

#devops #cron #monitoring #smallbusiness

Sunday, 2 a.m., the disk filled

Disk full at 2 a.m. on a Sunday. Not because nobody fixed it — because nobody KNEW. The cleanup job had been rm: no space left on device-ing itself into a log for three weeks. Every alerting philosophy debate in the world is useless if your cron jobs fail silently.

Here's the cheap, boring fix I install everywhere: make every cron job say something, out loud, every single run.

The one-line pattern

Every cron entry becomes:

job.sh >> /var/log/job.log 2>&1 && echo "OK $(date)" >> /var/log/job.log || echo "FAIL $(date)" >> /var/log/job.log
Enter fullscreen mode Exit fullscreen mode

Ugly. Works. But logs nobody reads are still decoration, so the real rule is:

  1. Every job appends a PASS/FAIL line with a timestamp. One line. No stack traces needed.
  2. One job reads all the logs and prints the verdict: 3 PASS, 1 FAIL, last FAIL 6h ago.
  3. That verdict goes somewhere a human sees — an email, a chat webhook, a status page, whatever costs you zero setup time.

The insight isn't clever tooling. It's that a cron job without a heartbeat is a rumor, and a rumor doesn't page you at 2 a.m.

The three silent-killer patterns

  • The job that only runs when data arrives. It passes when idle, and "passing" means nothing. Add a canary: assert the output file changed today.
  • The job whose failure email goes to a mailbox nobody reads. Test it: break the job on purpose, confirm a human notices within the hour. If not, you don't have alerting.
  • The chain where step 3 depends on step 1 and nobody checks step 1. Print the whole chain state in one line: step1 OK (2m), step2 SKIPPED, step3 SKIPPED — the word SKIPPED is the tell.

What this looks like as a product

I kept rebuilding the same three files for clients — the heartbeat wrapper, the log scanner, the one-page morning report — so I packaged them:

Even if you never buy anything: pick your most important cron job and make it print one PASS/FAIL line tonight. The first week it fires, you'll find a job that's been broken longer than you've worked there.

I run ops for small teams and write about the boring automations that keep them out of trouble. More at https://hive80lab.gumroad.com

Top comments (0)