There's a Cron Job in Production That Nobody Remembers Writing.

It runs at 3:07 AM, reconciles our invoices, and its author left in 2022. The crontab séance: inventory every scheduled job, give each an owner + purpose + loud failure or a funeral, crontab-as-code, staleness alerts, and ephemeral job boxes.

Every company has ghosts like this. October is a fine month to introduce yourself.

Scheduled jobs are employees without HR files

A cron job is a colleague: it has a shift, a responsibility, credentials, access. Unlike every human on payroll, it never got an onboarding (what does it need and why), a performance review (does it still succeed, still matter), an offboarding (its author left; its badge stayed active), or a manager (when it fails at 3 AM, who gets paged? nobody — the silence compounds).

Our invoice ghost had a valid API credential embedded in the script, a hardcoded path that survived two migrations by luck, and a failure mode that surfaces as accounting confusion in three weeks, not an alert in three minutes.

The séance
Print every crontab on every box — including the "temporary" ones — and hold one rule over each line: every job gets an owner, a purpose, and a loud failure — or a funeral.

#!/bin/bash
# cron-seance.sh — run per host; collect the output, do the arithmetic, feel the chill
set -euo pipefail
echo "== user crontabs =="
for u in $(cut -d: -f1 /etc/passwd); do
  c=$(crontab -l -u "$u" 2>/dev/null || true)
  [ -n "$c" ] && { echo "--- $u ---"; echo "$c" | grep -vE '^\s*#|^\s*$'; }
done
echo "== system dirs =="
grep -rHvE '^\s*#|^\s*$' /etc/cron.d /etc/cron.daily /etc/cron.hourly /etc/crontab 2>/dev/null
echo "== systemd timers =="
systemctl list-timers --all --no-pager | awk 'NF>6'

Two pages of output, eleven engineers. Then per line: who owns it, what happens when it fails, who knows why it exists. Lines that can't answer are ghosts.

Adopt the load-bearing ones, bury the rest
The invoice job didn't get deleted — it got hired: named owner, repo with tests, credentials out of the script and into scoped runtime injection, and alerts on both outcomes:

# the job's last line — a heartbeat:
touch /var/run/heartbeat/invoice-recon

# checker, on the scheduler host (outside the box it watches):
*/30 * * * *  find /var/run/heartbeat -name invoice-recon -mmin +1500 | grep . \
    && alert "invoice-recon: silent too long"     # "it didn't run" is an incident too

Six jobs got funerals — and funeral meant the line, the script, its credentials, and the downstream thing it fed. Half-buried cron is how ghosts get sequels.

Crontab-as-code, or the ghosts return
"I'll just add it on the server" is the exact sentence that spawns ghosts. Now the box's crontab is generated from a repo file, and CI is the only writer:

# infra/cron/invoice-recon.cron — reviewed like any colleague's PR
SHELL=/bin/bash
MAILTO=""                          # silence is not monitoring
07 3 * * * root /opt/jobs/invoice-recon.sh >> /var/log/jobs/invoice-recon.log 2>&1

# the only path into /etc/cron.d
krova ssh jobs-1 -- 'sudo install -m 644 /srv/infra/cron/*.cron /etc/cron.d/'

Where the jobs live matters too
Some scheduled work moved to ephemeral boxes — spin up, do the thing, die (same pattern as our CI):

CUBE="job-invoice-$(date +%F)"
krova cubes create "$CUBE" --cpu 1 --ram 2 --disk 20 --image ubuntu-24.04
krova ssh "$CUBE" -- 'INVOICE_KEY="'"$INVOICE_KEY"'" /opt/invoice-recon.sh'  # creds at runtime
krova cubes delete "$CUBE"        # leaves a bill line and a log, not a mystery

An ephemeral job can't accumulate drift or host a forgotten second job. Resident crons that remain live behind the usual walls: no public IP, own kernel, default-deny inbound. Ghosts are an ownership problem first, a hosting problem second — but right hosting makes ownership visible.

The honest part

• Some ghosts are the business. Deleting the invoice job would have been the incident. The audit finds load-bearing ghosts before they fail; it doesn't ban ghosts.
• Silence is the real vulnerability. A job that fails loudly is a bug; one that fails silently is debt with interest. Staleness alerts caught two more ghosts we'd otherwise have met during an accounting argument.
• Culture takes a week to feel heavy. Then someone proposed a cron change with a test, an owner, and a rollback note — in a review, like a normal colleague — and the habit had a heartbeat.

Meet your ghosts before they meet you
Tonight, on every box you own: open the crontab, read each line out loud, ask the three questions. Every line that can't answer is a ghost. Some are harmless. One of them, somewhere, is your invoice reconciliation.