DEV Community

lastmileops
lastmileops

Posted on

Run records and missed-run checks for scheduled scripts

#devops #linux #monitoring #bash

Tested on Linux with bash, jq and GNU date.

A quiet Slack channel can mean two things: nothing went wrong, or the job never ran. Most small scheduled scripts can't
tell those apart, because the only trace they leave is the alert they send when something looks bad.

Two small additions fix that: write a record for every run, including the boring successful ones, and check separately
that each job actually finished recently.

1. One line per run 

#!/usr/bin/env bash
# run-logged.sh JOB_NAME COMMAND... — run a job and append one JSON line per run to $RUN_LOG
set -uo pipefail
job="$1"; shift
log="${RUN_LOG:-/var/log/run-records.jsonl}"
rev=$(sha256sum "$(command -v "$1")" 2>/dev/null | cut -c1-12)
start=$(date -u +%FT%TZ)
out=$("$@" 2>&1); status=$?
end=$(date -u +%FT%TZ)
jq -nc --arg job "$job" --arg rev "${rev:-unknown}" --arg start "$start" --arg end "$end" \
      --argjson status "$status" --arg out "$(printf "%s" "$out" | tail -c 2000)" \
      '{job:$job, rev:$rev, start:$start, end:$end, status:$status, output:$out}' >> "$log"
exit "$status"
Enter fullscreen mode Exit fullscreen mode

Use it in cron instead of calling the script directly:

0 6 * * * RUN_LOG=/var/log/run-records.jsonl /opt/tools/run-logged.sh nightly-report /opt/tools/nightly-report.sh
Enter fullscreen mode Exit fullscreen mode

Every run now leaves a line with the job name, a short hash of the program that ran, start and end time, exit status and
the tail of its output — whether it found something or not. When someone asks "did it run on Tuesday, and which version?",
the answer is one jq away.

2. A separate check for missed runs 

#!/usr/bin/env bash
# check-missed.sh JOB_NAME MAX_AGE_MINUTES — fail if the job has no finished, successful run recently
set -uo pipefail
job="$1"; max="$2"; log="${RUN_LOG:-/var/log/run-records.jsonl}"
last=$(jq -r --arg job "$job" 'select(.job==$job and .status==0) | .end' "$log" 2>/dev/null | tail -1)
if [[ -z "$last" ]]; then echo "MISSED: $job has no successful run on record"; exit 2; fi
age=$(( ( $(date -u +%s) - $(date -u -d "$last" +%s) ) / 60 ))
if (( age > max )); then echo "MISSED: $job last succeeded ${age} min ago (limit ${max})"; exit 2; fi
echo "OK: $job succeeded ${age} min ago"
Enter fullscreen mode Exit fullscreen mode

Schedule it after the job's deadline and send its output wherever alerts go:

30 7 * * * RUN_LOG=/var/log/run-records.jsonl /opt/tools/check-missed.sh nightly-report 120 || notify-oncall
Enter fullscreen mode Exit fullscreen mode

It complains in three cases: the job never succeeded, it last succeeded too long ago, or it keeps failing. A failed data
fetch shows up as a failed check instead of a clean report.

What this doesn't do

It doesn't make the script correct, and it doesn't tell you why a run was skipped — only that it was. It's also still a
file on one machine: if several people depend on the job, they need somewhere to see these records that isn't your shell.

This article was created with the help of AI. The code was run on Linux before publishing.

Top comments (0)