DEV Community: Anguishe

I Killed My Report Script Mid-Write. The Output File Never Noticed.

Anguishe — Thu, 24 Sep 2026 13:00:00 +0000

The failure this pattern prevents is a quiet one. A script writes its output straight to the path a consumer reads, dies halfway, and the consumer loads the torn file as though it were complete. Nobody sees an error, because the only process that failed is gone, and everything downstream of it exited 0 on partial data. Meanwhile /tmp collects the working files of every run that never reached its cleanup line at the bottom.

I did not want to trust the fix on paper, so I tested it the unfriendly way on my own machine, bash 5.3.9. A small generator writes a CSV, with a sleep 30 planted between its first and second row to stand in for a slow API. I started it, waited one second, and sent it SIGTERM from outside.

This is the shape it runs in:

#!/bin/bash
set -euo pipefail
FINAL_PATH="${1:?usage: report.sh /path/to/output.csv}"

TMP_FILE=$(mktemp)             # unique path, created 0600
cleanup() {
  local code=$?                # first line: capture before anything overwrites it
  rm -f "$TMP_FILE"
  exit "$code"
}
trap cleanup EXIT              # registered on the line after mktemp

generate_rows > "$TMP_FILE"    # every write goes to the temp path
mv "$TMP_FILE" "$FINAL_PATH"   # atomic on one filesystem

What the kill left behind: exit status 143, which is 128 plus signal 15. The temp path mktemp had handed out came back No such file or directory, so the trap ran on the way down. And out.csv still held the previous run's three complete lines. A consumer reading it at that moment gets yesterday's data, whole. Stale is something monitoring can see. Torn is not.

Every way out is covered

Walk the exits. The generator fails halfway: set -e aborts, the EXIT trap removes the partial temp file, the published file is untouched. A signal arrives — Ctrl-C, a shutdown's SIGTERM, my kill — and the same trap runs, as the run above shows. The script succeeds: mv has already moved the temp file, so rm -f finds nothing, complains about nothing, and the status passes through.

EXIT is the reason this is four lines instead of twenty. It is a bash pseudo-signal that fires on normal completion, an explicit exit 1, a set -e abort, and after the handling of a real signal. Trap INT or TERM separately only when you want signal-specific behaviour, such as logging who killed you.

Two scoping rules each save a confused hour. A trap set inside $( ) or a ( ) group belongs to that subshell and fires when the subshell exits, so cleanup traps go at the top level. And a second trap … EXIT replaces the first — traps do not stack — so when two resources need releasing, a temp file and a background ssh -L tunnel for instance, both go in one handler.

The ordering that launders a failure

The comment on the local code=$? line is the part people move. I ran four versions of the handler against a script that ends in exit 3, and bash turned out more forgiving than the folklore:

A trap that removes the file and never calls exit: the script exits 3. Bash keeps the original status.
A trap ending in a bare exit: 3. Inside an EXIT trap, a bare exit reuses the status the script was leaving with.
Capture first, then rm -f, then exit "$code": 3.
rm -f first, then local code=$?, then exit "$code": 0.

The one version that reports success after a failure is the one that looks most careful. It captures $? — after rm has succeeded, so it captures rm's zero and exits with it. Cron sees success, alerting sees success, and the cleanup has removed the evidence too. Capture on the first line, clean second, exit last.

Why mktemp and not /tmp/myscript.$$

A predictable temp path is two bugs. Overlapping runs, retried jobs and helpers can collide on the same name and interleave their writes, which produces exactly the garbage this pattern is meant to prevent. And /tmp is world-writable, so a guessable name invites something else on the box to plant a file or a symlink where you are about to write. mktemp returns a unique path, created atomically, readable only by you. When a script needs several working files, make one directory with mktemp -d and remove it whole in the same handler; every intermediate file inherits the cleanup.

What trap cannot do, and why the design survives it

kill -9 cannot be caught. No trap runs, and power loss is no different. That is not an argument against the trap; it is why the rest of the design looks the way it does. Orphans from an uncatchable death land in /tmp, where reboot or tmpfiles ageing clears them, rather than in a data directory. And mv means even an uncatchable death cannot publish a torn file, because being half-written and being at the published path are never true at the same moment.

One cheap gate is worth adding wherever a consumer trusts your output: before the mv, check the temp file's size with wc -c against a floor, and exit non-zero instead of publishing an implausibly small result. An upstream that "succeeded" with an empty body then pages a human instead of reaching the dashboard.

A cleanup path you have never executed is a cleanup path you are guessing about. Plant a sleep, kill it from another terminal, and check three things: /tmp is clean, the output still holds the old complete file, and the exit code is not zero. It takes about a minute.

The complete script, the FAQ on subshell scoping and signal lists, and the line-by-line breakdown: https://bashsnippets.xyz/snippets/bash-trap-cleanup

The same survival kit continues with bash error handling, so failures stop the script and the trap has something honest to report, the timeout command, so a hang dies and cleans up instead of blocking forever, and the Cron Wrapper Generator, which assembles the lot around any command. The rest of the library is at https://bashsnippets.xyz

set -euo pipefail Is Missing a Letter. My ERR Trap Stayed Silent Until I Added -E.

Anguishe — Tue, 22 Sep 2026 13:00:00 +0000

Here is a nine-line script that does everything the strict-mode articles tell you to. set -euo pipefail on line 2. On line 3, an ERR trap whose entire job is to print the failing command and its line number: trap 'echo "✗ failed: ${BASH_COMMAND} (line ${LINENO})" >&2' ERR. Then a helper function that runs mkdir /proc/export, a directory that cannot be created, standing in for a full disk or a missing mount.

I ran it on my machine, bash 5.3.9. It printed start, then mkdir's own complaint, then exited 1. The trap said nothing. No failing command, no line number — the two facts it was registered to report. From a nightly job, that is the log you get: a start banner, one stderr line if you are lucky, and an exit code that tells you something broke but not what.

Then I changed one character, set -Eeuo pipefail, and ran it again. Same mkdir error, followed by ✗ failed: mkdir /proc/export (line 5), and exit 1. The trap had been fine all along. Nobody had told it to follow the script into the function, and a function is where most real failures happen.

Three flags, three promises, one missing

set -euo pipefail gets pasted as a single incantation. It is three flags making three separate promises, and each has its own list of places where it quietly declines to keep them.

-e says exit when a command fails and nobody is checking. Bash defines "checking" generously: the condition of an if, anything left of && or ||, anything under ! — and, the part that surprises people, everything inside a function that was called from one of those positions. A careful helper becomes unguarded the moment somebody wraps it in if helper; then, and nothing warns you.

-u says an unset variable is an error rather than an empty string — the difference between rm -rf "$BUILD_DIR/" and rm -rf / when the variable is spelled BUILDDIR in one place.

-o pipefail says a pipeline fails if any stage fails, not only the last one. Without it, curl … | jq … proceeds happily when curl dies and jq parses the empty result into something harmless.

None of those is the letter I was missing.

Why the trap said nothing

trap … ERR fires on the same conditions that make -e exit, so it looks like the natural place to print a diagnostic. But by default the ERR trap is not inherited by functions, command substitutions, or subshells. My failing mkdir was one level down, inside a function. Errexit did its job and killed the script. The trap, registered in the top-level shell, never saw a failure that happened in the function's scope. Exit 1, and silence — which is precisely the log entry that tells you nothing at 03:00.

The fix is -E, errtrace, which makes the ERR trap follow you into functions and subshells. The handler I run now names the command, not only the line, because bash keeps the pieces in BASH_COMMAND and BASH_LINENO:

#!/bin/bash
set -Eeuo pipefail
CROSS="✗"

on_err() {
  local code=$?     # first line, before anything else overwrites it
  echo "$CROSS failed: '${BASH_COMMAND}' (line ${BASH_LINENO[0]}, exit ${code})" >&2
}
trap on_err ERR

With that at the top, a failure anywhere in the script logs the command, the line and the exit status instead of nothing. -E is the flag missing from nearly every strict-mode line on the internet. If you take one thing from this, take the E.

The other silent spot: local

The second silent spot is a line most of us have written dozens of times: local out=$(some_command). On the same box, a function running local out=$(false) under set -euo pipefail carried on to its next line and the script exited 0. That line never triggers errexit, and no trap fires for it, because local is a command in its own right and its exit status — success, it declared the variable — is the one bash sees. The command substitution's failure is discarded before anyone looks. Declare on one line, assign on the next, and the failure is yours again. It is two lines instead of one everywhere you capture output into a local, and it is the bug behind ShellCheck's masked-return-value warning, which is not pedantry.

One caveat so you do not chase a ghost: with -E set, a failure inside an explicit ( subshell ) fires the trap twice — once in the subshell, once in the parent as the non-zero status propagates. That is expected. If duplicate alerts matter, guard on a flag or move the work out of the subshell.

Cleanup on every path, and when to leave it all off

The ERR trap tells you what broke; the EXIT trap is what stops the script leaving a half-written file behind for the next stage to load as if it were complete. It runs on every termination, and the rule that makes or breaks it is the same one as above: capture $? on the first line of the handler. Move local code=$? below the rm -f "$TMP_FILE" and it reports rm's status — zero, forever — and the script exits clean after a failure. That is how a script with strict mode at the top ends up reporting success: I ran that ordering against an exit 3 on bash 5.3.9 and the script exited 0.

Strict mode is also not the right default everywhere. A health checker that runs twenty probes and expects some to fail will spend more || true than logic fighting -e. .bashrc should never set it, because one failed command would close your terminal. And grep -q used as a test returns non-zero as data, not as an error. The guide covers where to leave it off, rather than pretending the flags are free.

What the log says now

The template I start from now opens with set -Eeuo pipefail, both traps registered before any work happens, $? captured first in each handler, locals declared and assigned on separate lines. When it fails, the log says which line and which command. That is the whole difference between a script that exits 1 and a script that tells you why.

The full template with every behaviour checked against bash 5.3 — where errexit goes silent, the nounset defaults for variables that are only unbound in staging, and the assembled skeleton: https://bashsnippets.xyz/guides/safe-bash-script-template

If you would rather not type it, the Bash Boilerplate Generator emits this shape with your script name filled in, trap cleanup on exit works the mktemp-and-atomic-mv half in full, and the rest of the library is at https://bashsnippets.xyz

For Months My Dev Server Came Up on 3001. Tonight I Found Out Who Had 3000.

Anguishe — Wed, 16 Sep 2026 17:21:09 +0000

A listener nobody remembered starting held port 3000 on my own machine since spring. One ss flag named the owner without root — and showed why 'list open ports' is three different questions with three different commands.

Every time I started the dev server for this site, the terminal said Port 3000 is in use, trying 3001 instead, and every time I read it and moved on. Next.js does not complain; it steps up one port and carries on, and so did I, for the better part of a season. There is a particular flavour of embarrassment in being the person who writes about port audits while a mystery listener sits on your own laptop that you have been silently working around since spring.

Tonight I asked. ss -ltn 'sport = :3000' came back with two rows — 0.0.0.0:3000 and [::]:3000, every interface, both address families — and a Process column that was blank. Not because nothing was there. Because I was not root, and ss -p only names sockets owned by your own user. Something on my machine was accepting connections from the whole network on 3000, and the tool I reached for shrugged.

The flag that answers without sudo

The snippet answer is sudo ss -ltnp. It works. But I wanted to know what thing this was more than I wanted a PID, and there is a flag that answers that for anyone:

ss -ltnpe 'sport = :3000'
# LISTEN 0 4096 0.0.0.0:3000 0.0.0.0:* ino:20816 sk:2002 cgroup:/system.slice/docker.service <->

-e prints the socket's owning cgroup, and on a systemd box the cgroup path ends in the unit that created it. Port 3000 belonged to docker.service. One docker ps later: a docker-proxy publishing an Open WebUI container's internal 8080 as host port 3000, started in the spring to try something, never stopped, exposed on every interface the machine has. No root needed to learn that. The kernel knows the cgroup of every socket, and ss will tell whoever asks.

Why the question is three questions

What I took from the evening is that "list open ports" is not one question, and most of the wrong answers online come from answering a different one than the person asked.

What is listening, and on which address, is ss -ltun and needs no privilege. The address column matters more than the port: 127.0.0.1:9050 is reachable from that machine and nowhere else whatever the firewall says, 0.0.0.0 is the whole network, and 127.0.0.53%lo is systemd-resolved pinned to loopback. If you have been piping ss through grep, the built-in filter syntax — ss -Hltn 'sport = :3000', with -H dropping the header so the output goes straight into awk — is the thing to switch to.

Who owns it is the question that needs root for other users' PIDs, and the one where -e gets you most of the way for free. lsof -nP -iTCP -sTCP:LISTEN reads the same facts from the process side and resolves the user name, which ss does not; fuser 3000/tcp is the terse one, and it exits 1 in silence when you run it unprivileged against someone else's socket, which looks exactly like "nothing there."

Whether it is reachable is a different question again. A port can be listening and unreachable — bound to loopback, or behind a firewall rule — and reachable with nothing of yours listening, when something upstream forwards it. nc -zv host 3000 asks the socket directly; Connection refused means the packet arrived and the kernel said no, while a timeout means a firewall ate it. On a box with no nc, bash opens TCP sockets on its own through /dev/tcp, and the timeout 2 in front of that is not optional, because a filtered port will hang the connect for two minutes.

What docker-proxy hides

My listener was the general case, not a curiosity. When a container publishes a port, Docker starts a small userspace proxy that binds the host port and forwards into the container's network namespace. From the host, ss sees the proxy. It never sees the application. So the host-side list answers "which host ports are exposed" and cannot answer "what is behind them" — that is docker ps and its PORTS column.

It cuts the other way too. A container port that is not published does not appear in the host's ss output at all; it lives in a different namespace. A host-level audit that ignores this will report a database as not listening while it serves an entire compose stack over an internal bridge. To see those, you enter the namespace with nsenter, or run ss inside the container if the image has it.

And when there is no ss, no netstat, nothing at all — a scratch container — the kernel still publishes the socket table as text in /proc/net/tcp, hex addresses and all. 0BB8 is 3000. 0A is LISTEN. One line of awk reads it, and the uid column is right there without root; the PID is the one thing that file cannot give you.

The diff is the point

Knowing the list tonight is worth less than knowing when it changes. The guide ends in a script that writes one CSV line per listening socket — protocol, address, port, service name, owner — using -e so it degrades gracefully without root, and with --diff compares against the previous run and exits 3 with a report naming what appeared and what vanished. While testing it I opened a Python http.server on 8099 between runs; the third run named it. Had that been in root's crontab in April, the docker-proxy on 3000 would have been a one-line email the night it appeared, not a season of a dev server politely stepping aside.

The full guide — every command run on this box with the output pasted as it came out, the netstat-to-ss translation table, the /proc/net/tcp decoder, and the audit script: https://bashsnippets.xyz/guides/open-ports-linux

Once you have the PID, Kill a Process on a Port covers the SIGTERM-then-SIGKILL escalation without reaching for -9 first, the one-script version this guide grew out of is List All Open Ports on Linux, and the rest of the library is at https://bashsnippets.xyz

Our Status Dashboard Was Green for 61 Hours While the API Was Down

Anguishe — Thu, 30 Jul 2026 01:01:09 +0000

Our status dashboard scraped a partner's health API every two minutes and lit up green the entire weekend the partner was down. By Monday morning they'd been offline for 61 hours and our board had been reporting "all systems operational" the whole time. Nobody paged. Nothing logged an error. The graph was a flat, confident green line over a three-day outage.

I wrote that scrape. I'd tested it against a live 200, watched it pull the status field, and shipped it, because the happy path worked and the happy path is the only path you ever see in a demo. Three separate bugs were sitting in the code the whole time, and every one of them would have passed a code review. The part I'm not proud of is that finding them took most of a Monday, and the fix for all three fit in about fifteen lines I'd skipped the first time.

Calling an API from a shell script is three problems wearing one trenchcoat: making the request, reading the response, and knowing when either one failed. Get any of the three wrong and the failure is silent — which is the worst kind, because silent failures get discovered by your users, not your tooling.

Bug one: curl exits 0 when the API returns a 500

curl reports on the transport, not the HTTP result. If it reached the server and got a complete response back — any response, including a 503 maintenance page — it exits 0. That's why set -euo pipefail doesn't save you here: from bash's point of view, nothing failed. The command ran, it got bytes back, everyone's happy.

The scrape had been faithfully saving a 503 HTML error page every two minutes and treating it as data. The fix is to make the status code something the script actually looks at:

response=$(curl -sS --connect-timeout 5 --max-time 30 -w $'\n%{http_code}' "$url")
http_code="${response##*$'\n'}"   # last line is the status
body="${response%$'\n'*}"         # everything before it is the body

Now 2xx is success, 429 and 5xx are transient and worth a retry, and any other 4xx is your own broken request and should fail loudly instead of being retried into oblivion. The two timeouts matter as much as the status check: a short --connect-timeout so a dead host fails fast, and a hard --max-time so a server that accepts your connection and then hangs can't leave a cron job wedged until the next run piles up behind it.

Bug two: grep on JSON doesn't error, it lies

Once you have a verified 2xx body, the next trap is reading fields out of it. Our script pulled the status with grep '"status"' | cut, and on the 503 error page grep matched a different "status" string that happened to appear in the HTML and read it as "ok".

grep, cut, and sed are line-oriented, and JSON has no meaningful lines. The same object can be minified onto one line or pretty-printed across twenty and it's identical data — but every line-based pattern you write depends on one specific layout the API is free to change without telling you. When it changes, your pattern doesn't throw an error. It quietly matches the wrong bytes.

jq parses the document into a real structure and addresses it by path. Three flags carry most of the weight:

# -e sets the exit code from the result, so "missing" is a branch, not an empty string
healthy=$(jq -er '.healthy' <<<"$body") || { echo "field missing"; exit 1; }

# -r prints the raw value — without it "web-01" keeps its quotes and breaks comparisons
region=$(jq -r '.region.name' <<<"$body")

# // supplies a real default, so a missing key becomes 0 instead of the literal null
stars=$(jq -r '.stargazers_count // 0' <<<"$body")

-r, //, and -e are the difference between a parse that survives an API reformat and one that breaks the next time someone on the other end runs a linter. Building select() and projection filters by hand is fiddly and a wrong quote silently matches nothing, which is exactly what the jq Filter Builder is for — paste a real response, click the fields, and it evaluates the filter live against your JSON before you wire it into anything.

Bug three: the failure nobody was told about

The third bug wasn't a bug in the code at all — it was the absence of one. The scrape could fail and nothing would say so. There was an email alert configured, technically; it went to an address forwarding to a distribution list everyone had muted in 2023. An alert nobody reads is decoration.

A Slack incoming webhook puts the failure where the team is already looking. The mechanism is a single POST of a JSON payload to a secret URL, so there are exactly two ways to get it wrong — send malformed JSON, or leak the URL. Build the payload with jq so an error message full of quotes and newlines can't break it, and wire it to a trap so you never have to remember to call it:

: "${SLACK_WEBHOOK_URL:?set it in the environment, never in the script}"

slack_alert() {
  local payload
  payload=$(jq -n --arg text ":rotating_light: $1" '{text: $text}')
  curl -sS --max-time 10 -X POST -H 'Content-Type: application/json' \
    -d "$payload" "$SLACK_WEBHOOK_URL" >/dev/null
}

trap 'slack_alert "scrape failed at line $LINENO (exit $?)"' ERR

With set -e, any unhandled non-zero exit fires the trap and posts the failure — with the line number — before the script dies. You set the trap once at the top and the whole script is covered.

Any one of the three would have caught it on minute one

That's the part worth sitting with. The 503 would have tripped the status check. The reshaped error page would have tripped jq -e. And the scrape falling over would have posted to Slack. Three independent guards, each closing one gap, and the outage needed all three to be missing to stay invisible for 61 hours. Wire the pattern together — request that fails on real failures, parse that doesn't lie, alert when either breaks — and you hear about a problem the first time it happens, not the third day.

Full fetch → parse → alert walkthrough with the complete script: https://bashsnippets.xyz/guides/shell-scripts-that-talk-to-apis

The three pieces each have their own deep dive — making the request safely with curl, parsing the response with jq, and alerting to Slack on failure — and the rest of the library is at https://bashsnippets.xyz

Your Cron Job Works. That's Not the Same as Surviving Cron

Anguishe — Wed, 08 Jul 2026 23:42:05 +0000

There's a gap between a script that works when you run it and a script that survives running unattended for a year. The first only has to succeed once, with you watching, on a good day. The second has to keep working through slow disks, hung sockets, recovering services, and reboots — with nobody watching, because the entire point of cron is that nobody is watching.

Almost every script I've seen take down a server was fine in a manual run and fell apart the first time conditions weren't ideal and there was no human there to notice. I've hit three versions of this personally: a sync that stacked copies until the box hit load 41, a backup that hung and didn't run for nine days without a peep, and a deploy that raced a database's boot and paged me at 11pm. Three commands, three failures, one underlying truth — "works on my machine" and "survives cron" are different engineering problems.

Cron jobs die quietly in exactly three ways, and there's a fourth problem underneath all of them that keeps the first three invisible.

The invisible problem first: cron eats your output

When cron runs a job, anything it prints to stdout or stderr goes nowhere unless you've configured otherwise. No log, no record, no trace. So a job can fail every night for a month and the only signal is the absence of whatever it was supposed to produce — and you notice that when something downstream breaks, not when the job fails.

Two things turn cron from silent to legible. MAILTO at the top of the crontab mails you any output a job produces. More reliably, redirect every job's output to a log file you control:

*/5 * * * * /usr/local/bin/safe-sync.sh >> /var/log/sync.log 2>&1

The 2>&1 is the part people forget — it folds stderr into the same stream as stdout, so your errors land in the log instead of evaporating. Everything below assumes you've done this. A locked, time-bounded, retried job that still logs nothing is a job whose failures you'll discover by accident.

Overlap: lock it to one instance

This is the failure that turns a transient slow patch into a self-inflicted outage. A sync scheduled every minute normally finishes in twenty seconds; one slow-disk afternoon it starts taking ninety, and cron launches a new copy every minute on top of the last, until a half-dozen copies are fighting over the same disk and the load climbs into the dozens.

The guard is flock: a kernel-held lock on an open file descriptor that lets exactly one copy run and makes the rest skip. Unlike a PID file, there's nothing to clean up — the kernel releases the lock automatically when the process exits, crashes, or is killed, so you never inherit a stale lock from a run that died badly. But a lock has a sharp edge: if the locked job hangs, it holds the lock forever and every future run skips. The job stops running entirely, silently. Which is the next problem.

Hang: bound the runtime

A hung command is worse than a failed one, because a failed command at least exits and frees its lock. A backup wedged on a held database lock, a curl against a dead socket, an ssh into a black hole — these never return. Under cron, "never returns" means the slot is jammed, the lock is held, and the job produces zero signal because it never gets far enough to log anything.

timeout is the outside bound the command can't set on itself:

# SIGTERM at 5 minutes; if it's ignored (process stuck in I/O), SIGKILL 20s later
timeout -k 20s 5m mysqldump --single-transaction mydb > /backup/mydb.sql.partial

It exits 124 when it had to step in and 137 when the command ignored the polite SIGTERM and had to be force-killed — codes worth branching on, because "slow" and "wedged" are different problems. The -k grace matters specifically for processes stuck in uninterruptible I/O, which can't act on SIGTERM at all. Timeout is what makes a lock safe: with both in place, a hang gets killed on a deadline, the lock always releases on time, and the failure becomes a loud 124 instead of a silent gap.

Transient failure: retry with backoff

Not every failure deserves to kill the run. A fresh database container that isn't accepting connections for six seconds, a 503 during a rolling restart, a 429 explicitly asking you to back off — these are transient. Treating the first one as fatal is how a normal six-second boot delay becomes a failed deploy at eleven at night.

The guard is a bounded retry with exponential backoff: try the command, and on failure wait a delay that doubles each round, with jitter so parallel callers don't retry in lockstep, capped so a genuinely dead dependency fails fast instead of looping forever.

# Wait for the port to be ready before doing the work that depends on it
retry 6 nc -z -w 2 db.internal 5432

The discipline that makes retries safe rather than dangerous: retry transient failures only. A 404 fails identically on every attempt, so retrying it just delays the real error and buries it under retry noise.

Composing all three — and knowing which job needs which

Locking, bounding, and retrying are independent guards, and the real value is stacking them: a single run that can't overlap, can't hang, and rides out a blip — and tells you, by mail or log, when it finally does give up. That composition is mechanical enough to generate, which is exactly what the Hardened Cron Wrapper Generator does: paste the command, toggle the guards, get a ShellCheck-clean wrapper and the crontab line.

But not every job needs all of it — wrapping a one-line date echo in flock and timeout is ceremony. Add a lock to anything that mutates shared state and can outrun its interval. Add a timeout to anything that touches the network or a database. Add retries to anything that depends on something which boots or recovers on its own clock. Log everything, always. The full decision table — guard by guard, when to add it and when to skip it — is in the guide.

Read the whole thing, including the "know when it broke" alerting layer and the decision matrix: https://bashsnippets.xyz/guides/bash-scripts-that-survive-cron

The three guards as standalone snippets: flock, timeout, retry with backoff. To write a new hardened script from a blank slate rather than wrap an existing command, the Bash Boilerplate Generator and the error-handling snippet cover the skeleton. The rest of the library is at https://bashsnippets.xyz

Three Lines of Bash Stand Between Your Cron Job and a Silent Outage

Anguishe — Sun, 05 Jul 2026 17:32:17 +0000

Three separate incidents taught me the same lesson over about a year. A */1 rsync stacked six copies on a slow disk and drove a box to load 41. A mysqldump hung on a database lock and didn't run for nine days without a single alert. A deploy raced a database's boot and paged me at 11pm for a migration that wasn't broken. Different commands, different failures — but the same root shape every time: a script that was fine when I ran it by hand fell over the first time conditions weren't ideal and nobody was watching.

The fixes are individually small. A cron job that can't take down a server needs three guards: a lock so it can't overlap itself, a timeout so it can't hang forever, and a bounded retry so a transient blip doesn't kill the run. The problem isn't knowing that. It's that composing the three correctly by hand — getting the flock file descriptor right, the timeout SIGKILL grace right, the retry quoting right, and not mangling a pipeline in the process — is exactly the kind of fiddly that produces a wrapper that looks right and silently isn't.

So I built a generator that does the composition: the Hardened Cron Wrapper Generator.

What it actually does

You paste in the command your cron job runs, toggle the guards you want, set the schedule, and it emits two things: a complete wrapper script and the crontab line that calls it. The generated bash follows a strict standard — set -euo pipefail, CHECK/CROSS defined before use, comments that explain why each line is there — and it's ShellCheck-clean, which is the part that matters when you're pasting something into a root crontab.

The toggles map exactly to the three failure modes:

Lock (flock) stops overlap. You pick the behavior: Skip if busy (-n) for frequent jobs where a missed run is harmless because the next one catches up, or Wait then give up (-w) for jobs that must eventually run but can tolerate a short queue. The tool puts the lock file under /run/lock — a tmpfs cleared on reboot — not /tmp, where temp-cleaners can delete a lock mid-run and let a second copy through. That single path choice is a bug people ship constantly.

Bound the runtime (timeout) stops hangs. You set the max runtime and a SIGKILL grace period. The grace exists because timeout first sends a polite SIGTERM, and a process wedged in uninterruptible I/O — a dead NFS mount, a held database lock — physically can't act on SIGTERM. The grace is how long to wait before sending SIGKILL, which the kernel enforces unconditionally. The tool sets a sane default; you tune it for jobs you know touch slow storage.

Retry transient failures rides out the blip. You set max attempts and a base delay, and the backoff doubles each round up to a 30-second cap, with jitter so parallel callers don't retry in lockstep and re-overwhelm a recovering service.

The part that makes the output trustworthy

Two design decisions in the tool are worth calling out because they're the ones I'd get wrong by hand at 11pm.

First, the crontab line changes shape depending on what you enable. Lock and timeout are simple enough to inline directly on the crontab line — flock and timeout in front of the command, a one-liner. But retry and email-on-failure need a bash function and a mailx call, which can't live on a crontab line. So when you turn either of those on, the output switches from an inline crontab one-liner to a full wrapper script plus a crontab line that calls it. The tool makes that decision for you instead of generating something that won't run.

Second, it's explicit about pipelines. timeout and the retry loop bound the single process they launch — so a raw cmd1 | cmd2 would only be partly guarded. The tool surfaces this: for a pipeline or multiple commands, you wrap them in bash -c '...' yourself, and it tells you so, rather than silently generating a wrapper that bounds half your command.

It also handles the thing underneath all three guards that people forget: logging. Cron throws stdout and stderr away by default, so a job can fail every night for a month and the only signal is the absence of whatever it was supposed to produce. The tool's timestamped-logging toggle redirects output to a file you control, and the email-alert toggle sends a mailx failure notice after all retries are exhausted — so the job fails loudly instead of disappearing.

Why generate it instead of memorizing it

Because the failure mode of hand-writing these is that the wrapper looks correct and isn't. The flock that used /tmp. The timeout with no -k that left the wedged process alive. The retry loop that retried a 404 forever. Each of those is a one-character or one-decision mistake, and each one only shows up the night it matters. A generator that bakes the right defaults in — and explains them inline in the output so you can read what you're shipping — removes that whole class of error.

Build your wrapper here: https://bashsnippets.xyz/tools/cron-wrapper-generator

The three snippets it composes, if you want the full reasoning behind each guard, are flock, timeout, and retry with backoff. Build the schedule itself in the Cron Job Builder, and read the whole decision in Bash Scripts That Survive Cron. The rest of the library is at https://bashsnippets.xyz

I Spent 40 Minutes at 11pm Debugging a Deploy That Wasn't Broken

Anguishe — Thu, 02 Jul 2026 03:21:37 +0000

I once spent forty minutes at eleven at night debugging a deploy that wasn't broken. The release script ran the database migration, the migration threw connection refused, the script exited non-zero, the deploy rolled itself back, and I got paged.

So I did the things you do. I read the migration. I read the logs. I checked the database — it was up, it was healthy, it accepted my connection instantly. I re-ran the deploy and it worked. I chalked it up to gremlins and went to bed, which is the part I'm not proud of, because it happened again two days later. That time I watched the timing: the script brought up a fresh database container and started the migration about six seconds before Postgres finished initializing and began accepting connections. The migration was racing the database's boot. Most of the time it won. The times it lost, I lost forty minutes.

The script wasn't wrong about anything except one assumption: that a dependency is ready the instant you ask for it.

In production, dependencies are eventually ready

That's the mental model shift. Networks blip. A service you call returns a 503 for the two seconds it takes to finish a rolling restart. An API rate-limits you with a 429 it fully expects you to retry. A fresh container's database isn't accepting connections for its first few seconds. Treating the first failure as fatal turns every one of these normal, transient conditions into a paged engineer — and the script that handles them isn't smarter — it declines to give up on the first try.

But retrying naively is its own trap. Retry instantly and you hammer a recovering service into staying down. Retry forever and a genuinely dead dependency hangs your script indefinitely. Retry a 404 and you wait a minute to confirm what you already knew. Good retries are bounded, backed off, and selective.

A retry function you can reuse anywhere

#!/bin/bash
# Purpose: survive transient failures instead of dying on the first error
set -euo pipefail

CHECK="✓"
CROSS="✗"

# retry <max_attempts> <command> [args...]
retry() {
    local max_attempts="$1"; shift
    local attempt=1
    local delay=1            # base delay — doubles each round
    local max_delay=30       # cap so the backoff never runs away

    until "$@"; do
        if (( attempt >= max_attempts )); then
            echo "$CROSS '$*' failed after $attempt attempts" >&2
            return 1
        fi
        # 0–2s of jitter so parallel callers don't all retry on the same beat
        local pause=$(( delay + RANDOM % 3 ))
        echo "$CROSS attempt $attempt failed — retrying in ${pause}s" >&2
        sleep "$pause"
        attempt=$(( attempt + 1 ))
        delay=$(( delay * 2 ))
        (( delay > max_delay )) && delay=$max_delay
    done

    echo "$CHECK '$*' succeeded on attempt $attempt"
}

# The actual fix for my 11pm deploy: wait for Postgres to accept connections.
retry 6 nc -z -w 2 db.internal 5432
echo "$CHECK database reachable — running migration"

The whole engine is until "$@"; do ... done. until runs the command and executes the loop body only when it fails, exiting the instant it succeeds. Passing the command as "$@" (after shift-ing past the attempt count) means the function retries anything — a curl, an ssh, a port check, your own script — without caring what it is.

The backoff is the three lines at the bottom of the loop: sleep for the current delay plus a little jitter, then double the delay, capped at max_delay. That gives you 1s, 2s, 4s, 8s, 16s, 30s, 30s…

The jitter is not decoration

RANDOM % 3 looks trivial, and it's the line people delete to "clean up." Don't. Without jitter, a fleet of machines that all failed at the same instant — because the same service went down — will all retry at the same instant, and the same instant after that, producing a synchronized thundering herd that knocks the recovering service straight back over on every round. A few hundred milliseconds of randomness per client spreads the retries out so the service actually gets room to recover. At one machine it does nothing; at fifty it's the difference between recovery and a retry storm.

The mistake that makes retries dangerous

# Good: a transient failure that retrying can fix
retry 6 nc -z -w 2 db.internal 5432

# Bad: retrying a deterministic failure just delays the error 30 seconds
retry 6 curl -fsS https://api.example.com/v1/thing-that-returns-404

A retry loop is only as smart as what you point it at. A port check belongs in a loop because the answer changes — "no" until the database boots, then "yes." A request that returns 404 returns 404 on attempt one and attempt six; the loop just postpones the failure and buries the real status under retry noise. Retry transient failures — timeouts, connection-refused, 429, 5xx, DNS hiccups. Don't retry deterministic ones — a 404, a 401, a syntax error, a missing file. When you can, branch on the exit code or HTTP status and loop only on the codes worth looping on.

For plain curl, its built-in --retry 5 --retry-delay 2 does most of this and is simpler; reach for the function when the thing you're retrying isn't curl, or when you want one backoff policy across a database probe, an ssh call, and a download at once.

Back to 11pm

That deploy never paged me again once the migration waited for the port instead of assuming it. The database still took its six seconds to boot, the network still blipped occasionally — retrying didn't make the dependencies faster. It stopped a normal, transient slowness from being treated as a fatal error, which is most of what "production-ready" means for a script.

Full function with the wait-for-port pattern and the guidance on which failures to retry: https://bashsnippets.xyz/snippets/bash-retry-with-backoff

Retries are the third leg of an unattended job: flock stops overlap, timeout stops hangs, retry rides out the blip. The Hardened Cron Wrapper Generator wires all three into one wrapper, Bash Scripts That Survive Cron is the end-to-end version, and the rest of the library is at https://bashsnippets.xyz

My Backup Hadn't Run in 9 Days and Nothing Told Me

Anguishe — Fri, 26 Jun 2026 03:39:56 +0000

The backup ran fine every night for fourteen months, and then it didn't run for nine days, and nothing told me. No error in the log. No failed-job alert. No bounced cron mail. The nightly mysqldump had hung — the database had a long-held lock from a runaway analytics query, the dump opened its transaction and sat there waiting for it, forever.

Cron launched it at 2am, it never exited, and here's the cruel part: because I'd been smart enough to wrap it in a lock so two dumps couldn't run at once, every subsequent night's run saw the lock still held by the zombie from the 9th and skipped quietly. The clever lock turned a one-night hang into a nine-day outage. I found it when I went to restore a table and discovered my newest "backup" was a mysqldump process that had been running since the previous Tuesday.

That's nine days I'd have lost if anything had gone wrong with the live database. The ten minutes of feeling foolish when I traced it back was nothing next to that.

A hung command is worse than a failed one

This is the lesson worth internalizing, because it's counterintuitive. A failed command exits, frees its lock, and the next run tries again — the system self-heals. A hung command exits never. It holds resources, blocks its own future runs, and produces exactly zero signal because it never gets far enough to log anything. Failures are loud. Hangs are silent, and silence is what kills you in unattended automation.

You can't rely on a command to bound its own runtime, either. The whole problem is that it's wedged somewhere it can't time itself out of — blocked in the kernel waiting on a lock, or on a dead socket that will never send the FIN it's waiting for. So you bound it from the outside.

Bounding the runtime with timeout

#!/bin/bash
# Script: bounded-dump.sh
# Purpose: Stop a hung command from running forever and jamming the cron slot
set -euo pipefail

CHECK="✓"
CROSS="✗"

MAX_RUNTIME="5m"   # longer than the normal worst case, well under the interval
KILL_GRACE="20s"   # after SIGTERM, wait this long, then SIGKILL
DEST="/backup/mydb.sql"

# In an `if` so set -e doesn't abort before we read the exit code.
# Write to a .partial file so a timed-out run never leaves a corrupt "backup".
if timeout -k "$KILL_GRACE" "$MAX_RUNTIME" \
        mysqldump --single-transaction mydb > "$DEST.partial"; then
    mv "$DEST.partial" "$DEST"
    echo "$CHECK dump completed within $MAX_RUNTIME"
else
    code=$?
    rm -f "$DEST.partial"
    case "$code" in
        124) echo "$CROSS dump exceeded $MAX_RUNTIME — terminated (SIGTERM)" >&2 ;;
        137) echo "$CROSS dump ignored SIGTERM — force-killed (SIGKILL)" >&2 ;;
        *)   echo "$CROSS dump failed with exit code $code" >&2 ;;
    esac
    exit "$code"
fi

timeout -k "$KILL_GRACE" "$MAX_RUNTIME" mysqldump ... is the entire mechanism. At five minutes, timeout sends the dump a SIGTERM. A well-behaved program treats SIGTERM as "wrap up and exit." But the dump from my outage wasn't misbehaving — it was blocked in the kernel waiting on a lock, and a process in that state physically cannot act on SIGTERM. That's what -k 20s is for: twenty seconds after the polite signal, timeout sends SIGKILL, which the kernel enforces unconditionally. Nothing survives SIGKILL.

Read the exit code — it's the difference between a log and a mystery

124  # still running at the deadline — SIGTERM was sent
137  # 128 + 9 — ignored SIGTERM, had to be force-killed
*    # anything else is the command's own failure

Collapsing all three into "backup failed" throws away the one piece of information that tells you whether you have a slow database, a wedged one, or a broken dump command. A 124 says "this is taking too long — investigate the query." A 137 says "this is wedged in I/O — investigate the lock or the mount." They point at different problems. (If you ever blank on which code is which, bashsnippets.xyz/tools/bash-exit-code-lookup decodes 124 and 137 directly.)

The `.partial` trick matters more than it looks

If you redirect straight to the real backup file and the command times out mid-write, you've just replaced last night's good backup with a half-written, unrestorable file — and you won't know until the night you need it. Writing to a .partial path and mv-ing into place only on a clean exit means a failed or timed-out run leaves the previous good backup untouched. A mv on the same filesystem is atomic; the redirect is not.

For commands that talk to the network, layer the tool's own timeout underneath — curl --max-time, ssh -o ConnectTimeout, a net_read_timeout on the dump. Those fire first and fail cleanly. timeout is the outer hard stop for the night the inner one doesn't.

Back to the nine days

A timeout is what makes a lock safe. Locking a job to a single instance stops overlap, but a hang inside the locked job holds that lock forever — which is precisely how my nine-day gap happened. Bound the runtime and the lock always gets released, on a deadline, every time, and the failure becomes a loud 124 in the log instead of a silent gap you discover during a restore.

Full script with the exit-code branching and the FAQ on timing out a whole pipeline: https://bashsnippets.xyz/snippets/bash-timeout-command

The lock that made my hang invisible is flock, and the third guard is retry with backoff. The Hardened Cron Wrapper Generator composes all three, Bash Scripts That Survive Cron walks the whole decision, and the rest of the library is at https://bashsnippets.xyz

A Cron Job Took Our Server to Load 41 by Attacking Itself

Anguishe — Tue, 23 Jun 2026 00:18:35 +0000

A */1 rsync took our staging box to a load average of 41 one afternoon, and it took me longer than I want to admit to work out why. The sync normally finished in about twenty seconds. That day the backup target's NFS mount went sluggish, the sync started taking ninety seconds, and cron — which does not know or care whether the last run is still going — launched a fresh copy every single minute on top of it.

Inside ten minutes there were a half-dozen rsyncs all reading the same tree off the same slow disk, each one making the disk slower, each new minute adding another. The box wasn't under attack. It was attacking itself, one polite copy at a time. The thing that stung was that nothing was broken — every individual rsync was correct, the disk eventually recovered on its own, and the only reason it became an outage is that cron has no concept of "the last one is still running."

That's the trap with scheduled jobs: a command that's perfectly fine when you run it by hand can take down a server the first time it runs longer than its interval with nobody watching.

The fix everyone reaches for first is the wrong one

The instinct is a PID file: write $$ to /var/run/job.pid on start, check whether that file exists on the next run, bail if it does. It almost works. Then one run gets kill -9'd, or the box reboots mid-job, and the PID file is left behind pointing at a process that died on Tuesday. Now every future run sees a "lock" owned by a PID that no longer exists, and the job never runs again — the opposite failure, just as silent.

There's also a race between the check and the write, and the times you most need the lock to be clean are exactly the times cleanup didn't happen, because the process died before it could clean up.

flock has none of that. The lock isn't a file you create and delete — it's a lock the kernel holds on an open file descriptor, and the kernel releases it automatically the instant that descriptor closes. The process exiting closes it. So does crashing. So does kill -9. There is no state to leave behind, which is the entire reason it survives the failure modes a PID file can't.

The single-instance pattern

#!/bin/bash
# Script: backup-with-lock.sh
# Purpose: Stop a cron job from overlapping itself when one run runs long
set -euo pipefail

CHECK="✓"
CROSS="✗"

# /run/lock is tmpfs, cleared cleanly on reboot. Never /tmp — temp-cleaners
# delete files there, and a deleted lock mid-run lets a second copy run.
LOCK_FILE="/run/lock/$(basename "$0").lock"

# The > opens (and creates) the lock file on fd 200 and holds it open for the
# whole script. The lock lives on this descriptor, not on the file existing.
exec 200>"$LOCK_FILE"

# -n = non-blocking: if a previous run still holds the lock, give up now
# instead of queueing another copy behind it.
if ! flock -n 200; then
    echo "$CROSS $(date '+%F %T') previous run still active — skipping" >&2
    exit 0
fi

echo "$CHECK $(date '+%F %T') lock acquired — starting"
rsync -a --delete /data/ /mnt/backup/data/
echo "$CHECK $(date '+%F %T') finished — kernel releases the lock on exit"

The two lines doing the work are exec 200>"$LOCK_FILE" and flock -n 200. The first opens the lock file on a descriptor that stays open for the life of the process. The second tries to grab the lock without waiting; if a sibling process already holds it, flock returns non-zero, we log it and exit 0 — a skipped run is normal, not an error, so we don't want it lighting up cron's mail.

Notice there is no cleanup. No trap to remove a PID file, no rm at the end. When this script exits for any reason, fd 200 closes and the lock is gone. That "for any reason" is the whole point.

You can lock a job without editing it at all

If the misbehaving job is already deployed and you don't want to touch it, wrap it from the crontab line:

# Skip the run if the last one is still going
*/1 * * * * /usr/bin/flock -n /run/lock/sync.lock /usr/local/bin/sync.sh

flock runs sync.sh only if it can grab the lock; if last minute's run is still holding it, this minute's run exits immediately and does nothing. It's the fastest retrofit for a job that's already on fire — no redeploy.

One thing worth burning into memory: -n skips, -w 30 waits up to thirty seconds then gives up, and a bare flock with neither blocks forever. On a fast cron schedule that bare form turns your "skipped" runs into a pile of stuck processes — the exact thing you were trying to prevent.

The part that actually mattered

The load-41 afternoon ended the moment I wrapped that rsync in flock -n. The slow NFS mount was still slow, but now exactly one sync ran at a time and the extras skipped harmlessly until the disk recovered. Locking didn't fix the slow disk — it stopped a transient slow disk from becoming a self-inflicted outage. That's the difference between a script that works when you run it and one that survives unattended.

A lock alone isn't the whole story, though. If the locked job itself hangs, it holds the lock forever and every future run skips — so the job silently stops running and you find out days later. That's why locking pairs with bounding runtime with timeout and retrying transient failures.

Full script, the -n vs -w decision, and the FAQ on where the lock file should live: https://bashsnippets.xyz/snippets/bash-flock-single-instance

If you're hardening a cron job, the next two guards are timeout and retry with backoff; the Hardened Cron Wrapper Generator stitches all three into one wrapper, and the full reasoning is in Bash Scripts That Survive Cron. The rest of the library is at https://bashsnippets.xyz

A Function Without local Overwrote My Variable and rm -rf Deleted the Wrong Directory

Anguishe — Mon, 22 Jun 2026 02:46:38 +0000

The deploy script had been running in production for four months without a problem. It built releases into a temp directory, ran some validation, and then cleaned up by removing whatever $target pointed at. $target was set near the top of the script to the current release directory — the one the running application was serving from. A helper function called prepare() also used a variable named target, because the person who wrote it (me, four months earlier) did not think about scope.

prepare() built the new release into a temp directory. On a good run, it set target to the temp path, did its work, and returned. The main script then did its cleanup at the end and removed $target — which, after calling prepare(), was the temp directory. That worked correctly for four months.

Then a deploy failed partway through prepare(). The temp directory was half-built. target was now pointing at the half-built temp path. The main script caught the failure, started its cleanup, and ran rm -rf "$target". It removed the half-built temp directory, which was correct. But then it kept going — there was a second cleanup step that also used $target and expected it to still be the release directory. By the time the script finished, it had removed the running application's release directory. The application restarted and found nothing to serve.

The users noticed before I did. The restart loop was filling logs, the application was returning 502, and I was sitting in the deploy output trying to figure out what had gone wrong in a failure path I had tested against a stub.

Variables in bash are global by default

This is the single most surprising thing about bash functions if you have written code in almost any other language. In Python, a variable assigned inside a function is local to that function unless you explicitly declare it global. In bash, it is the opposite. A variable assigned inside a function is visible — and writable — everywhere in the current shell unless you declare it local.

target="/srv/release/current"

prepare() {
  target=$(mktemp -d)    # No local — this overwrites the global $target
  echo "building in $target"
}

prepare
echo "target is now: $target"   # Prints the temp dir, not /srv/release/current

The function does exactly what it looks like — it sets target. The problem is that it sets target everywhere, not just inside itself. The caller's target is gone.

local confines the assignment to the function scope:

prepare() {
  local target              # confined to this function
  target=$(mktemp -d)
  echo "$target"            # hand the value out via stdout
}

build_dir=$(prepare)        # capture what the function echoed
echo "built in: $build_dir"
echo "release dir still: $target"   # unchanged

local target means: this variable exists only inside this function. When the function returns, the variable and its value vanish. The caller's target is never touched. The habit I now enforce on every bash function I write: local for every variable the function introduces, not just the ones I think might conflict. The conflict I do not predict is the one that deletes the wrong directory.

return is a status, not a value

After the incident, I audited every function in the deploy script. I found a second bug:

count_pending() {
  local n
  n=$(find "$QUEUE_DIR" -type f | wc -l)
  return "$n"      # WRONG if n > 255
}

count_pending
if [[ $? -gt 0 ]]; then
  echo "queue has items"
fi

return sets an exit status. Exit statuses are a single byte: 0 to 255. return 300 wraps to 44. For months the queue count had been above 255 on busy days, and the $? check was comparing against a wrapped value. The logic had been wrong for months and had accidentally worked because the wrapped values still triggered the gt 0 condition. But any script making real decisions based on the actual count — how many workers to spin up, whether to page someone — would have been working with garbage.

The correct pattern is to echo the value and capture it:

count_pending() {
  local n
  n=$(find "$QUEUE_DIR" -type f | wc -l)
  echo "$n"         # data goes to stdout
  return 0          # status: success
}

pending=$(count_pending)
echo "queue depth: $pending"

return answers "did this succeed." echo plus command substitution answers "what is the value." These are two different questions and bash gives you two different mechanisms for a reason. Mixing them is how a function that counts 300 items makes the caller think it counted 44.

Arguments and the $@ quoting rule

Inside a function, arguments arrive as positional parameters: $1, $2, all of them as "$@", the count as $#. Quoting "$@" is what keeps multi-word arguments intact:

process_hosts() {
  echo "processing $# hosts"
  for host in "$@"; do
    echo "  checking: $host"
  done
}

process_hosts "web-01" "db primary" "cache-02"

Without the quotes around "$@", db primary splits into two loop iterations and you are back to the word-splitting problem from a different angle. The quoted "$@" is the way bash passes an array of arguments through a function call with each element preserved.

This matters most when you are writing wrapper functions — functions that receive arguments and pass them to another command:

run_with_retry() {
  local retries="${1:?}"
  shift
  local attempt=0
  while ((attempt < retries)); do
    "$@" && return 0
    ((attempt++))
    echo "retry $attempt/$retries"
    sleep 2
  done
  return 1
}

run_with_retry 3 rsync -av "source dir/" remote:/dest/

"$@" after the shift is everything after the retry count — the command and all its arguments, each preserved as a separate item even if they contain spaces. Without the quotes, "source dir/" splits and rsync receives the wrong arguments.

getopts for anything with flags

For one or two fixed positional arguments, reading $1 and $2 directly is fine. The moment a function or script takes optional flags in any order, do not parse them by hand:

verbose=0
output_dir=""

while getopts "vd:" opt; do
  case "$opt" in
    v) verbose=1 ;;
    d) output_dir="$OPTARG" ;;
    *) echo "usage: $0 [-v] [-d dir]"; exit 1 ;;
  esac
done
shift $((OPTIND - 1))   # move past the flags to positional args

The colon after d marks it as requiring an argument, which arrives in $OPTARG. getopts handles flag bundling (-vd dir), missing arguments (-d with no path generates an error automatically), and unknown flags. Hand-rolled $1 parsing gets all of these wrong in subtle ways — it accepts -d at the end without a value, it does not handle -vd dir, and it requires the flags in a specific order.

The deploy script that caused the incident had hand-rolled argument parsing. Among other things, it silently accepted a --target flag with no value and proceeded with an empty string, which caused a different class of problem I had also not fully traced before the bigger incident made the whole thing visible.

What the deploy script looks like now

Every function declares local for every variable. Values that need to cross function boundaries go through echo and command substitution. Exit statuses communicate success or failure. getopts handles the flags. There is a trap on EXIT that cleans up the temp directory using a local variable that only the cleanup function can see:

cleanup() {
  local tmp_dir="${TEMP_BUILD_DIR:-}"   # local ref to the temp dir
  [[ -n "$tmp_dir" && -d "$tmp_dir" ]] && rm -rf "$tmp_dir"
}
trap cleanup EXIT

The $target variable in the main script is set once at the top and never touched by any function. Functions that need a temp directory create one, store it in a local variable, use it, and the cleanup trap handles removal. The variable naming conflict that caused the incident cannot happen because the pattern prevents it structurally.

The application has been running cleanly since then. The deploy script has hit the failure path twice since the fix — different failures, unrelated causes — and both times the cleanup ran correctly and left the running release untouched.

Full version with the local-scope fix, the echo-for-values pattern, the return-as-status trap, and a getopts template: https://bashsnippets.xyz/snippets/bash-functions-arguments

A function that fails should fail loudly — wrap the script in set -euo pipefail — and the bash boilerplate generator can scaffold all of this with the right traps and argument parsing wired in from the start. The rest is at https://bashsnippets.xyz

The Alert Never Fired Because the Loop Skipped the Last Line of the File

Anguishe — Fri, 19 Jun 2026 17:58:09 +0000

We kept a plaintext file of hostnames, one per line, and a monitoring script read the file and pinged each host every five minutes. When a host failed to respond, the script sent an email alert. The system had been running for months and it worked — we had caught three actual outages with it, which gave us real confidence in the setup.

app-07 was added to the list on a Thursday afternoon. The engineer who added it was using VS Code on a Mac, and VS Code by default does not add a trailing newline to a file when you append to it using certain editing workflows. The file had ended in a newline before the edit. After the edit, the last line — app-07 — had no trailing newline.

app-07 went down the following Sunday afternoon at 2:17pm. The monitoring script ran at 2:20, 2:25, 2:30, all the way through the evening. No alert ever fired. The on-call engineer found out at 8pm when a client emailed. The system had been down for almost six hours.

When I looked at the script, the bug was immediately obvious once I knew what to look for. But I had written that script, I had tested it, and I had been looking at the monitoring confirmation emails for months without ever noticing. The confirmation email listed the hosts it checked. app-07 was never in the list. I had been reading those emails without actually counting the hosts. I just scanned for the OK lines and moved on.

Why read drops the last line

read returns a success exit status when it reads a line and finds the newline that terminates it. When the file does not end in a newline, read still populates the variable with the final line's content, but it returns a non-zero (failure) exit status because it hit end-of-file before finding a terminator. A while read host loop checks the return status to decide whether to execute the loop body. On the final, newline-less line, read puts app-07 into host and then returns failure. The while loop sees failure and exits without running the body. The content is there. The variable is populated. The loop throws it away.

This behavior is documented in the POSIX spec for read. It is not a bash quirk. Any POSIX shell handles the missing-final-newline case this way. A plain while read line loop is incorrect for any file you do not personally control the formatting of, which in practice means nearly any file.

The fix is one extra clause:

#!/bin/bash
# Script: check-hosts.sh
# Purpose: ping every host in a file, including a newline-less final line
# Usage: ./check-hosts.sh hosts.txt
set -euo pipefail

CHECK="✓"
CROSS="✗"
HOST_FILE="${1:?Usage: check-hosts.sh <host-file>}"

while IFS= read -r host || [[ -n "$host" ]]; do
  [[ -z "$host" || "$host" == \#* ]] && continue
  if ping -c1 -W2 "$host" >/dev/null 2>&1; then
    echo "$CHECK up:   $host"
  else
    echo "$CROSS down: $host"
  fi
done < "$HOST_FILE"

|| [[ -n "$host" ]] says: if read returned failure but the variable is non-empty, run the loop body anyway. That is precisely the leftover-final-line case. read failed because it hit end-of-file, but it populated host with app-07 before returning. The || catches it. app-07 gets pinged.

What IFS= and -r actually do

You see while IFS= read -r line written in every correct read-loop example, and it is worth being specific about what each piece prevents because both have their own failure mode.

IFS= sets the field separator to empty for the duration of the read command. Without it, read strips leading and trailing whitespace from each line. A hostname like app-07 (with leading spaces, which some editors produce) becomes app-07, which might be correct. An indented config value, a Python-style YAML string, a log line that starts with spaces for alignment — all of these are silently modified. Setting IFS= tells read to take the line exactly as it appears.

-r prevents read from interpreting backslash sequences. Without -r, a line like C:\temp\logs has its backslashes consumed as escape characters and arrives as C:templogs. This matters less for hostname files and enormously for any script that processes Windows paths, config files that use backslash as a line-continuation character, or log files from mixed-OS environments. The -r flag is essentially free protection; there is no reason not to include it.

A bare read line without either flag silently mangles both whitespace and backslashes. The script works correctly on clean input and produces wrong output on input with edge cases. The wrong output does not produce an error. You find out when the data that mattered was the indented or backslash-containing kind.

The subshell trap that kills your counters

This is the one that is most likely to make you question your sanity:

# This looks correct. It is not.
fails=0
cat hosts.txt | while IFS= read -r host; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done
echo "Total failures: $fails"   # Always prints 0

The pipe creates a subshell for the right side. The while loop runs inside that subshell. fails increments correctly inside the subshell. When the subshell exits, the parent shell's fails is still 0, because the increment happened in a different process. The parent echo sees the original value.

This catches people because the loop body itself works — the pings happen, the increment logic is correct — but any state the loop was supposed to accumulate for later use is silently discarded. I spent forty minutes on a version of this problem before I remembered that pipes create subshells. It is the kind of thing that feels like a bash bug until you understand that it is behaving exactly as documented.

The fix is to redirect the file into the loop instead of piping into it:

fails=0
while IFS= read -r host || [[ -n "$host" ]]; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done < hosts.txt
echo "Total failures: $fails"   # Now correct

done < hosts.txt feeds the file to the loop's stdin without a pipe. The loop runs in the current shell. fails accumulates in the current shell. The echo sees the real count.

Why for loop is wrong for this

# Never do this — iterates words, not lines
for line in $(cat hosts.txt); do
  ping -c1 "$line"
done

$(cat hosts.txt) is command substitution. Bash captures the text output and word-splits it on IFS — spaces, tabs, newlines. For a file with one hostname per line and no spaces in the hostnames, this accidentally produces the right behavior. For any file with spaces — log lines, config values, paths with spaces, anything a non-developer might have generated — it splits lines into fragments and each fragment becomes a loop iteration.

There is no version of for line in $(cat file) that is correct for reading lines. The right tool is always while IFS= read -r line || [[ -n "$line" ]]; do ... done < file. The for loop is the right tool for iterating a known list that you control directly, not for reading file content.

The monitoring system, after the fix

After the app-07 incident we made three changes. The obvious one was fixing the read loop with the || [[ -n "$host" ]] guard. The second was adding a sanity check at the top of the script that counted the lines in the hosts file and compared it to the number of hosts the loop actually processed — a mismatch meant the file was malformed or something else was wrong. The third was adding a nightly email that included the count of hosts checked, not just the status of each one, so a future addition to the file that somehow got lost would show up as "expected 12 hosts, checked 11."

The confirmation email count was something I should have had from the start. If I had been looking at "checked 11/12 hosts" instead of a list of OK lines, I would have noticed app-07 missing on the first night. The monitoring was working. The observability of the monitoring was not.

Full version with CSV-field parsing, comment-skipping, and the subshell-safe redirect form: https://bashsnippets.xyz/snippets/bash-read-file-line-by-line

To iterate a list of files rather than a file's contents, reach for a for loop instead, and wrap anything that acts on what it reads in set -euo pipefail. More at https://bashsnippets.xyz

A For Loop Skipped Every File With a Space and Called the Backup a Success

Anguishe — Thu, 18 Jun 2026 19:29:49 +0000

The nightly backup looped over for f in $(ls /data/exports) and copied each file to a backup volume. It exited clean every night. For three weeks, green exit codes, no errors, nothing in the logs to suggest anything was wrong. The backup script had been written by someone who left the company six months before I started, and it had never been tested against files with spaces in their names because the original export directory only ever had files like Q3.xlsx and report-final.xlsx.

Then someone generated Q3 final.xlsx.

It took three weeks because that file was generated once a quarter and the next time someone needed it was a quarter later. The person who needed it was the CFO. The CFO does not particularly enjoy being told that the backup system that was supposed to protect the quarterly export has been silently failing for an indeterminate period of time and we are not sure which other files it missed.

I know the specific backup had been running for three weeks because that was the date the file appeared in the exports directory. Every night since then, the loop had been splitting Q3 final.xlsx into two items — Q3 and final.xlsx — trying to copy two paths that did not exist, logging two harmless "no such file or directory" lines that nobody read, and moving on. Every file without a space in its name backed up fine. The script looked correct because most of the time it was.

Why $(ls) word-splits

When you write for f in $(ls /data/exports), bash runs ls, captures its text output, and splits it on IFS — the internal field separator, which defaults to spaces, tabs, and newlines. Filenames are just text in the output of ls. A file named Q3 final.xlsx is one filename, but ls outputs it as the string Q3 final.xlsx, and bash splits that string on the space into two separate items before the loop ever starts.

This is not a bug in ls. This is not a bug in bash. This is exactly what $( ) does to any command's output — it captures text and bash processes it as text. The problem is that filenames are not reliably text-safe; they can contain any character except null and the path separator. Spaces are common. Tabs are less common but legal. Newlines are technically legal. Treating command output as a filename list breaks the moment any of those show up.

The fix is to stop treating command output as a filename list and let bash build the list from the filesystem directly:

#!/bin/bash
# Script: backup-exports.sh
# Purpose: copy export files without losing ones with spaces in their names
# Usage: ./backup-exports.sh
set -euo pipefail

CHECK="✓"
CROSS="✗"
SRC_DIR="/data/exports"
DEST_DIR="/backup/exports"
shopt -s nullglob   # empty glob expands to nothing, not the literal pattern

for f in "$SRC_DIR"/*.xlsx; do
  if cp -- "$f" "$DEST_DIR/"; then
    echo "$CHECK backed up: $(basename "$f")"
  else
    echo "$CROSS failed: $(basename "$f")"
  fi
done

for f in "$SRC_DIR"/*.xlsx asks bash to expand the glob. Bash talks directly to the filesystem and gets back a properly-separated list of matching paths. No command output, no text splitting, no ambiguity about what the separator is. A file named Q3 final.xlsx stays one item because it was never turned into text and split back apart. The -- before "$f" tells cp to stop reading flags, so a filename that starts with a dash does not get interpreted as a cp option.

The second half: quoting on use

The glob fixes the loop header. Quoting fixes the point of use. These are separate.

for f in "$SRC_DIR"/*.xlsx; do
  cp $f "$DEST_DIR/"    # WRONG — $f re-splits here
  cp "$f" "$DEST_DIR/"  # RIGHT — the quotes prevent the split
done

Even with a correct glob, an unquoted $f in the cp command re-splits on spaces. Bash has already expanded the variable to Q3 final.xlsx, but when you use it unquoted, the shell processes word splitting again on that value and cp receives two arguments: Q3 and final.xlsx. The quotes around "$f" tell bash to pass the entire value as a single argument. The rule is: glob over parse to build the list, quote on use to keep each item intact.

I have seen this mistake repeated in scripts at three different companies. In each case the scripts had been running for months or years and the problem was invisible because the majority of files had no spaces. The ones that did — client names, quarterly reports, anything a non-technical person named — were being silently skipped or mishandled.

Ranges and counters: where the brace trap hides

n=5

# This does NOT produce a range — it produces the literal string {1..5}
for i in {1..$n}; do echo "attempt $i"; done

# This works — C-style, evaluates variables at runtime
for ((i = 1; i <= n; i++)); do echo "attempt $i of $n"; done

Brace expansion happens before variable expansion in bash's order of operations. By the time $n is replaced with its value, the brace expansion step has already passed and {1..$n} is just a string. This is the kind of thing that fails silently in a test environment where the count is small and hardcoded, and causes weird output in production where it is a variable.

The C-style loop is the correct form any time the bound is a variable. It evaluates at runtime and handles arithmetic naturally. If the count is genuinely fixed at write time, {1..10} works fine. If it is ever going to be a variable, use for (( )).

Arrays: the original bug wearing a different hat

servers=("web-01" "db primary" "cache-02")

# WRONG — word-splits "db primary" into two iterations
for s in ${servers[@]}; do
  ping -c1 "$s"
done

# RIGHT — quotes keep each element intact
for s in "${servers[@]}"; do
  ping -c1 "$s"
done

"${servers[@]}" with the quotes and [@] is the form that preserves each element as a single item regardless of what is in it. Without the quotes, bash word-splits the array expansion and db primary becomes two separate loop iterations — neither of which is a real hostname. This is exactly the same word-splitting mechanism as the $(ls) problem, just manifesting in arrays instead of command output.

Once you internalize that word-splitting happens wherever an unquoted variable or expansion appears, the rule becomes one rule instead of several: always quote expansions. The specific context changes; the mechanism does not.

The nullglob case

shopt -s nullglob
for f in /data/exports/*.xlsx; do
  echo "$f"
done

Without shopt -s nullglob, if no .xlsx files exist, the glob *.xlsx does not expand — it stays as the literal string *.xlsx. The loop runs once with f set to the literal string /data/exports/*.xlsx. Your script then tries to process a file with that exact path, which does not exist, and produces an error or silently does nothing depending on what you do with it.

Setting nullglob tells bash to expand a non-matching glob to nothing (an empty list), so the loop simply does not run. This is almost always the right behavior when you are iterating files that might not exist.

What happened to the Q3 export

We recovered it from the CFO's local machine, where she had downloaded it before the backup was supposed to preserve it. The fix to the script took four minutes. The conversation about why the backup system had been failing silently for three weeks took longer. The monitoring that we added afterwards — a nightly check that the backup directory has at least as many files as the source directory — took another twenty minutes.

The monitoring should have been there from the start. So should the glob. So should the set -euo pipefail that would have made the copy failures loud instead of silent. These are things you add before something breaks, and the only reason to know you need them is to have seen, or caused, or read about what happens when they are missing.

Full examples with the safe glob, counter, C-style loop, array form, and nullglob guard: https://bashsnippets.xyz/snippets/bash-for-loop-examples

For reading a file's lines one at a time, a for loop is the wrong tool — use while IFS= read -r — and wrap any loop that touches real files in set -euo pipefail. The rest is at https://bashsnippets.xyz