DEV Community: Anguishe

I Spent 40 Minutes at 11pm Debugging a Deploy That Wasn't Broken

Anguishe — Thu, 02 Jul 2026 03:21:37 +0000

I once spent forty minutes at eleven at night debugging a deploy that wasn't broken. The release script ran the database migration, the migration threw connection refused, the script exited non-zero, the deploy rolled itself back, and I got paged.

So I did the things you do. I read the migration. I read the logs. I checked the database — it was up, it was healthy, it accepted my connection instantly. I re-ran the deploy and it worked. I chalked it up to gremlins and went to bed, which is the part I'm not proud of, because it happened again two days later. That time I watched the timing: the script brought up a fresh database container and started the migration about six seconds before Postgres finished initializing and began accepting connections. The migration was racing the database's boot. Most of the time it won. The times it lost, I lost forty minutes.

The script wasn't wrong about anything except one assumption: that a dependency is ready the instant you ask for it.

In production, dependencies are eventually ready

That's the mental model shift. Networks blip. A service you call returns a 503 for the two seconds it takes to finish a rolling restart. An API rate-limits you with a 429 it fully expects you to retry. A fresh container's database isn't accepting connections for its first few seconds. Treating the first failure as fatal turns every one of these normal, transient conditions into a paged engineer — and the script that handles them isn't smarter — it declines to give up on the first try.

But retrying naively is its own trap. Retry instantly and you hammer a recovering service into staying down. Retry forever and a genuinely dead dependency hangs your script indefinitely. Retry a 404 and you wait a minute to confirm what you already knew. Good retries are bounded, backed off, and selective.

A retry function you can reuse anywhere

#!/bin/bash
# Purpose: survive transient failures instead of dying on the first error
set -euo pipefail

CHECK="✓"
CROSS="✗"

# retry <max_attempts> <command> [args...]
retry() {
    local max_attempts="$1"; shift
    local attempt=1
    local delay=1            # base delay — doubles each round
    local max_delay=30       # cap so the backoff never runs away

    until "$@"; do
        if (( attempt >= max_attempts )); then
            echo "$CROSS '$*' failed after $attempt attempts" >&2
            return 1
        fi
        # 0–2s of jitter so parallel callers don't all retry on the same beat
        local pause=$(( delay + RANDOM % 3 ))
        echo "$CROSS attempt $attempt failed — retrying in ${pause}s" >&2
        sleep "$pause"
        attempt=$(( attempt + 1 ))
        delay=$(( delay * 2 ))
        (( delay > max_delay )) && delay=$max_delay
    done

    echo "$CHECK '$*' succeeded on attempt $attempt"
}

# The actual fix for my 11pm deploy: wait for Postgres to accept connections.
retry 6 nc -z -w 2 db.internal 5432
echo "$CHECK database reachable — running migration"

The whole engine is until "$@"; do ... done. until runs the command and executes the loop body only when it fails, exiting the instant it succeeds. Passing the command as "$@" (after shift-ing past the attempt count) means the function retries anything — a curl, an ssh, a port check, your own script — without caring what it is.

The backoff is the three lines at the bottom of the loop: sleep for the current delay plus a little jitter, then double the delay, capped at max_delay. That gives you 1s, 2s, 4s, 8s, 16s, 30s, 30s…

The jitter is not decoration

RANDOM % 3 looks trivial, and it's the line people delete to "clean up." Don't. Without jitter, a fleet of machines that all failed at the same instant — because the same service went down — will all retry at the same instant, and the same instant after that, producing a synchronized thundering herd that knocks the recovering service straight back over on every round. A few hundred milliseconds of randomness per client spreads the retries out so the service actually gets room to recover. At one machine it does nothing; at fifty it's the difference between recovery and a retry storm.

The mistake that makes retries dangerous

# Good: a transient failure that retrying can fix
retry 6 nc -z -w 2 db.internal 5432

# Bad: retrying a deterministic failure just delays the error 30 seconds
retry 6 curl -fsS https://api.example.com/v1/thing-that-returns-404

A retry loop is only as smart as what you point it at. A port check belongs in a loop because the answer changes — "no" until the database boots, then "yes." A request that returns 404 returns 404 on attempt one and attempt six; the loop just postpones the failure and buries the real status under retry noise. Retry transient failures — timeouts, connection-refused, 429, 5xx, DNS hiccups. Don't retry deterministic ones — a 404, a 401, a syntax error, a missing file. When you can, branch on the exit code or HTTP status and loop only on the codes worth looping on.

For plain curl, its built-in --retry 5 --retry-delay 2 does most of this and is simpler; reach for the function when the thing you're retrying isn't curl, or when you want one backoff policy across a database probe, an ssh call, and a download at once.

Back to 11pm

That deploy never paged me again once the migration waited for the port instead of assuming it. The database still took its six seconds to boot, the network still blipped occasionally — retrying didn't make the dependencies faster. It stopped a normal, transient slowness from being treated as a fatal error, which is most of what "production-ready" means for a script.

Full function with the wait-for-port pattern and the guidance on which failures to retry: https://bashsnippets.xyz/snippets/bash-retry-with-backoff

Retries are the third leg of an unattended job: flock stops overlap, timeout stops hangs, retry rides out the blip. The Hardened Cron Wrapper Generator wires all three into one wrapper, Bash Scripts That Survive Cron is the end-to-end version, and the rest of the library is at https://bashsnippets.xyz

My Backup Hadn't Run in 9 Days and Nothing Told Me

Anguishe — Fri, 26 Jun 2026 03:39:56 +0000

The backup ran fine every night for fourteen months, and then it didn't run for nine days, and nothing told me. No error in the log. No failed-job alert. No bounced cron mail. The nightly mysqldump had hung — the database had a long-held lock from a runaway analytics query, the dump opened its transaction and sat there waiting for it, forever.

Cron launched it at 2am, it never exited, and here's the cruel part: because I'd been smart enough to wrap it in a lock so two dumps couldn't run at once, every subsequent night's run saw the lock still held by the zombie from the 9th and skipped quietly. The clever lock turned a one-night hang into a nine-day outage. I found it when I went to restore a table and discovered my newest "backup" was a mysqldump process that had been running since the previous Tuesday.

That's nine days I'd have lost if anything had gone wrong with the live database. The ten minutes of feeling foolish when I traced it back was nothing next to that.

A hung command is worse than a failed one

This is the lesson worth internalizing, because it's counterintuitive. A failed command exits, frees its lock, and the next run tries again — the system self-heals. A hung command exits never. It holds resources, blocks its own future runs, and produces exactly zero signal because it never gets far enough to log anything. Failures are loud. Hangs are silent, and silence is what kills you in unattended automation.

You can't rely on a command to bound its own runtime, either. The whole problem is that it's wedged somewhere it can't time itself out of — blocked in the kernel waiting on a lock, or on a dead socket that will never send the FIN it's waiting for. So you bound it from the outside.

Bounding the runtime with timeout

#!/bin/bash
# Script: bounded-dump.sh
# Purpose: Stop a hung command from running forever and jamming the cron slot
set -euo pipefail

CHECK="✓"
CROSS="✗"

MAX_RUNTIME="5m"   # longer than the normal worst case, well under the interval
KILL_GRACE="20s"   # after SIGTERM, wait this long, then SIGKILL
DEST="/backup/mydb.sql"

# In an `if` so set -e doesn't abort before we read the exit code.
# Write to a .partial file so a timed-out run never leaves a corrupt "backup".
if timeout -k "$KILL_GRACE" "$MAX_RUNTIME" \
        mysqldump --single-transaction mydb > "$DEST.partial"; then
    mv "$DEST.partial" "$DEST"
    echo "$CHECK dump completed within $MAX_RUNTIME"
else
    code=$?
    rm -f "$DEST.partial"
    case "$code" in
        124) echo "$CROSS dump exceeded $MAX_RUNTIME — terminated (SIGTERM)" >&2 ;;
        137) echo "$CROSS dump ignored SIGTERM — force-killed (SIGKILL)" >&2 ;;
        *)   echo "$CROSS dump failed with exit code $code" >&2 ;;
    esac
    exit "$code"
fi

timeout -k "$KILL_GRACE" "$MAX_RUNTIME" mysqldump ... is the entire mechanism. At five minutes, timeout sends the dump a SIGTERM. A well-behaved program treats SIGTERM as "wrap up and exit." But the dump from my outage wasn't misbehaving — it was blocked in the kernel waiting on a lock, and a process in that state physically cannot act on SIGTERM. That's what -k 20s is for: twenty seconds after the polite signal, timeout sends SIGKILL, which the kernel enforces unconditionally. Nothing survives SIGKILL.

Read the exit code — it's the difference between a log and a mystery

124  # still running at the deadline — SIGTERM was sent
137  # 128 + 9 — ignored SIGTERM, had to be force-killed
*    # anything else is the command's own failure

Collapsing all three into "backup failed" throws away the one piece of information that tells you whether you have a slow database, a wedged one, or a broken dump command. A 124 says "this is taking too long — investigate the query." A 137 says "this is wedged in I/O — investigate the lock or the mount." They point at different problems. (If you ever blank on which code is which, bashsnippets.xyz/tools/bash-exit-code-lookup decodes 124 and 137 directly.)

The `.partial` trick matters more than it looks

If you redirect straight to the real backup file and the command times out mid-write, you've just replaced last night's good backup with a half-written, unrestorable file — and you won't know until the night you need it. Writing to a .partial path and mv-ing into place only on a clean exit means a failed or timed-out run leaves the previous good backup untouched. A mv on the same filesystem is atomic; the redirect is not.

For commands that talk to the network, layer the tool's own timeout underneath — curl --max-time, ssh -o ConnectTimeout, a net_read_timeout on the dump. Those fire first and fail cleanly. timeout is the outer hard stop for the night the inner one doesn't.

Back to the nine days

A timeout is what makes a lock safe. Locking a job to a single instance stops overlap, but a hang inside the locked job holds that lock forever — which is precisely how my nine-day gap happened. Bound the runtime and the lock always gets released, on a deadline, every time, and the failure becomes a loud 124 in the log instead of a silent gap you discover during a restore.

Full script with the exit-code branching and the FAQ on timing out a whole pipeline: https://bashsnippets.xyz/snippets/bash-timeout-command

The lock that made my hang invisible is flock, and the third guard is retry with backoff. The Hardened Cron Wrapper Generator composes all three, Bash Scripts That Survive Cron walks the whole decision, and the rest of the library is at https://bashsnippets.xyz

A Cron Job Took Our Server to Load 41 by Attacking Itself

Anguishe — Tue, 23 Jun 2026 00:18:35 +0000

A */1 rsync took our staging box to a load average of 41 one afternoon, and it took me longer than I want to admit to work out why. The sync normally finished in about twenty seconds. That day the backup target's NFS mount went sluggish, the sync started taking ninety seconds, and cron — which does not know or care whether the last run is still going — launched a fresh copy every single minute on top of it.

Inside ten minutes there were a half-dozen rsyncs all reading the same tree off the same slow disk, each one making the disk slower, each new minute adding another. The box wasn't under attack. It was attacking itself, one polite copy at a time. The thing that stung was that nothing was broken — every individual rsync was correct, the disk eventually recovered on its own, and the only reason it became an outage is that cron has no concept of "the last one is still running."

That's the trap with scheduled jobs: a command that's perfectly fine when you run it by hand can take down a server the first time it runs longer than its interval with nobody watching.

The fix everyone reaches for first is the wrong one

The instinct is a PID file: write $$ to /var/run/job.pid on start, check whether that file exists on the next run, bail if it does. It almost works. Then one run gets kill -9'd, or the box reboots mid-job, and the PID file is left behind pointing at a process that died on Tuesday. Now every future run sees a "lock" owned by a PID that no longer exists, and the job never runs again — the opposite failure, just as silent.

There's also a race between the check and the write, and the times you most need the lock to be clean are exactly the times cleanup didn't happen, because the process died before it could clean up.

flock has none of that. The lock isn't a file you create and delete — it's a lock the kernel holds on an open file descriptor, and the kernel releases it automatically the instant that descriptor closes. The process exiting closes it. So does crashing. So does kill -9. There is no state to leave behind, which is the entire reason it survives the failure modes a PID file can't.

The single-instance pattern

#!/bin/bash
# Script: backup-with-lock.sh
# Purpose: Stop a cron job from overlapping itself when one run runs long
set -euo pipefail

CHECK="✓"
CROSS="✗"

# /run/lock is tmpfs, cleared cleanly on reboot. Never /tmp — temp-cleaners
# delete files there, and a deleted lock mid-run lets a second copy run.
LOCK_FILE="/run/lock/$(basename "$0").lock"

# The > opens (and creates) the lock file on fd 200 and holds it open for the
# whole script. The lock lives on this descriptor, not on the file existing.
exec 200>"$LOCK_FILE"

# -n = non-blocking: if a previous run still holds the lock, give up now
# instead of queueing another copy behind it.
if ! flock -n 200; then
    echo "$CROSS $(date '+%F %T') previous run still active — skipping" >&2
    exit 0
fi

echo "$CHECK $(date '+%F %T') lock acquired — starting"
rsync -a --delete /data/ /mnt/backup/data/
echo "$CHECK $(date '+%F %T') finished — kernel releases the lock on exit"

The two lines doing the work are exec 200>"$LOCK_FILE" and flock -n 200. The first opens the lock file on a descriptor that stays open for the life of the process. The second tries to grab the lock without waiting; if a sibling process already holds it, flock returns non-zero, we log it and exit 0 — a skipped run is normal, not an error, so we don't want it lighting up cron's mail.

Notice there is no cleanup. No trap to remove a PID file, no rm at the end. When this script exits for any reason, fd 200 closes and the lock is gone. That "for any reason" is the whole point.

You can lock a job without editing it at all

If the misbehaving job is already deployed and you don't want to touch it, wrap it from the crontab line:

# Skip the run if the last one is still going
*/1 * * * * /usr/bin/flock -n /run/lock/sync.lock /usr/local/bin/sync.sh

flock runs sync.sh only if it can grab the lock; if last minute's run is still holding it, this minute's run exits immediately and does nothing. It's the fastest retrofit for a job that's already on fire — no redeploy.

One thing worth burning into memory: -n skips, -w 30 waits up to thirty seconds then gives up, and a bare flock with neither blocks forever. On a fast cron schedule that bare form turns your "skipped" runs into a pile of stuck processes — the exact thing you were trying to prevent.

The part that actually mattered

The load-41 afternoon ended the moment I wrapped that rsync in flock -n. The slow NFS mount was still slow, but now exactly one sync ran at a time and the extras skipped harmlessly until the disk recovered. Locking didn't fix the slow disk — it stopped a transient slow disk from becoming a self-inflicted outage. That's the difference between a script that works when you run it and one that survives unattended.

A lock alone isn't the whole story, though. If the locked job itself hangs, it holds the lock forever and every future run skips — so the job silently stops running and you find out days later. That's why locking pairs with bounding runtime with timeout and retrying transient failures.

Full script, the -n vs -w decision, and the FAQ on where the lock file should live: https://bashsnippets.xyz/snippets/bash-flock-single-instance

If you're hardening a cron job, the next two guards are timeout and retry with backoff; the Hardened Cron Wrapper Generator stitches all three into one wrapper, and the full reasoning is in Bash Scripts That Survive Cron. The rest of the library is at https://bashsnippets.xyz

A Function Without local Overwrote My Variable and rm -rf Deleted the Wrong Directory

Anguishe — Mon, 22 Jun 2026 02:46:38 +0000

The deploy script had been running in production for four months without a problem. It built releases into a temp directory, ran some validation, and then cleaned up by removing whatever $target pointed at. $target was set near the top of the script to the current release directory — the one the running application was serving from. A helper function called prepare() also used a variable named target, because the person who wrote it (me, four months earlier) did not think about scope.

prepare() built the new release into a temp directory. On a good run, it set target to the temp path, did its work, and returned. The main script then did its cleanup at the end and removed $target — which, after calling prepare(), was the temp directory. That worked correctly for four months.

Then a deploy failed partway through prepare(). The temp directory was half-built. target was now pointing at the half-built temp path. The main script caught the failure, started its cleanup, and ran rm -rf "$target". It removed the half-built temp directory, which was correct. But then it kept going — there was a second cleanup step that also used $target and expected it to still be the release directory. By the time the script finished, it had removed the running application's release directory. The application restarted and found nothing to serve.

The users noticed before I did. The restart loop was filling logs, the application was returning 502, and I was sitting in the deploy output trying to figure out what had gone wrong in a failure path I had tested against a stub.

Variables in bash are global by default

This is the single most surprising thing about bash functions if you have written code in almost any other language. In Python, a variable assigned inside a function is local to that function unless you explicitly declare it global. In bash, it is the opposite. A variable assigned inside a function is visible — and writable — everywhere in the current shell unless you declare it local.

target="/srv/release/current"

prepare() {
  target=$(mktemp -d)    # No local — this overwrites the global $target
  echo "building in $target"
}

prepare
echo "target is now: $target"   # Prints the temp dir, not /srv/release/current

The function does exactly what it looks like — it sets target. The problem is that it sets target everywhere, not just inside itself. The caller's target is gone.

local confines the assignment to the function scope:

prepare() {
  local target              # confined to this function
  target=$(mktemp -d)
  echo "$target"            # hand the value out via stdout
}

build_dir=$(prepare)        # capture what the function echoed
echo "built in: $build_dir"
echo "release dir still: $target"   # unchanged

local target means: this variable exists only inside this function. When the function returns, the variable and its value vanish. The caller's target is never touched. The habit I now enforce on every bash function I write: local for every variable the function introduces, not just the ones I think might conflict. The conflict I do not predict is the one that deletes the wrong directory.

return is a status, not a value

After the incident, I audited every function in the deploy script. I found a second bug:

count_pending() {
  local n
  n=$(find "$QUEUE_DIR" -type f | wc -l)
  return "$n"      # WRONG if n > 255
}

count_pending
if [[ $? -gt 0 ]]; then
  echo "queue has items"
fi

return sets an exit status. Exit statuses are a single byte: 0 to 255. return 300 wraps to 44. For months the queue count had been above 255 on busy days, and the $? check was comparing against a wrapped value. The logic had been wrong for months and had accidentally worked because the wrapped values still triggered the gt 0 condition. But any script making real decisions based on the actual count — how many workers to spin up, whether to page someone — would have been working with garbage.

The correct pattern is to echo the value and capture it:

count_pending() {
  local n
  n=$(find "$QUEUE_DIR" -type f | wc -l)
  echo "$n"         # data goes to stdout
  return 0          # status: success
}

pending=$(count_pending)
echo "queue depth: $pending"

return answers "did this succeed." echo plus command substitution answers "what is the value." These are two different questions and bash gives you two different mechanisms for a reason. Mixing them is how a function that counts 300 items makes the caller think it counted 44.

Arguments and the $@ quoting rule

Inside a function, arguments arrive as positional parameters: $1, $2, all of them as "$@", the count as $#. Quoting "$@" is what keeps multi-word arguments intact:

process_hosts() {
  echo "processing $# hosts"
  for host in "$@"; do
    echo "  checking: $host"
  done
}

process_hosts "web-01" "db primary" "cache-02"

Without the quotes around "$@", db primary splits into two loop iterations and you are back to the word-splitting problem from a different angle. The quoted "$@" is the way bash passes an array of arguments through a function call with each element preserved.

This matters most when you are writing wrapper functions — functions that receive arguments and pass them to another command:

run_with_retry() {
  local retries="${1:?}"
  shift
  local attempt=0
  while ((attempt < retries)); do
    "$@" && return 0
    ((attempt++))
    echo "retry $attempt/$retries"
    sleep 2
  done
  return 1
}

run_with_retry 3 rsync -av "source dir/" remote:/dest/

"$@" after the shift is everything after the retry count — the command and all its arguments, each preserved as a separate item even if they contain spaces. Without the quotes, "source dir/" splits and rsync receives the wrong arguments.

getopts for anything with flags

For one or two fixed positional arguments, reading $1 and $2 directly is fine. The moment a function or script takes optional flags in any order, do not parse them by hand:

verbose=0
output_dir=""

while getopts "vd:" opt; do
  case "$opt" in
    v) verbose=1 ;;
    d) output_dir="$OPTARG" ;;
    *) echo "usage: $0 [-v] [-d dir]"; exit 1 ;;
  esac
done
shift $((OPTIND - 1))   # move past the flags to positional args

The colon after d marks it as requiring an argument, which arrives in $OPTARG. getopts handles flag bundling (-vd dir), missing arguments (-d with no path generates an error automatically), and unknown flags. Hand-rolled $1 parsing gets all of these wrong in subtle ways — it accepts -d at the end without a value, it does not handle -vd dir, and it requires the flags in a specific order.

The deploy script that caused the incident had hand-rolled argument parsing. Among other things, it silently accepted a --target flag with no value and proceeded with an empty string, which caused a different class of problem I had also not fully traced before the bigger incident made the whole thing visible.

What the deploy script looks like now

Every function declares local for every variable. Values that need to cross function boundaries go through echo and command substitution. Exit statuses communicate success or failure. getopts handles the flags. There is a trap on EXIT that cleans up the temp directory using a local variable that only the cleanup function can see:

cleanup() {
  local tmp_dir="${TEMP_BUILD_DIR:-}"   # local ref to the temp dir
  [[ -n "$tmp_dir" && -d "$tmp_dir" ]] && rm -rf "$tmp_dir"
}
trap cleanup EXIT

The $target variable in the main script is set once at the top and never touched by any function. Functions that need a temp directory create one, store it in a local variable, use it, and the cleanup trap handles removal. The variable naming conflict that caused the incident cannot happen because the pattern prevents it structurally.

The application has been running cleanly since then. The deploy script has hit the failure path twice since the fix — different failures, unrelated causes — and both times the cleanup ran correctly and left the running release untouched.

Full version with the local-scope fix, the echo-for-values pattern, the return-as-status trap, and a getopts template: https://bashsnippets.xyz/snippets/bash-functions-arguments

A function that fails should fail loudly — wrap the script in set -euo pipefail — and the bash boilerplate generator can scaffold all of this with the right traps and argument parsing wired in from the start. The rest is at https://bashsnippets.xyz

The Alert Never Fired Because the Loop Skipped the Last Line of the File

Anguishe — Fri, 19 Jun 2026 17:58:09 +0000

We kept a plaintext file of hostnames, one per line, and a monitoring script read the file and pinged each host every five minutes. When a host failed to respond, the script sent an email alert. The system had been running for months and it worked — we had caught three actual outages with it, which gave us real confidence in the setup.

app-07 was added to the list on a Thursday afternoon. The engineer who added it was using VS Code on a Mac, and VS Code by default does not add a trailing newline to a file when you append to it using certain editing workflows. The file had ended in a newline before the edit. After the edit, the last line — app-07 — had no trailing newline.

app-07 went down the following Sunday afternoon at 2:17pm. The monitoring script ran at 2:20, 2:25, 2:30, all the way through the evening. No alert ever fired. The on-call engineer found out at 8pm when a client emailed. The system had been down for almost six hours.

When I looked at the script, the bug was immediately obvious once I knew what to look for. But I had written that script, I had tested it, and I had been looking at the monitoring confirmation emails for months without ever noticing. The confirmation email listed the hosts it checked. app-07 was never in the list. I had been reading those emails without actually counting the hosts. I just scanned for the OK lines and moved on.

Why read drops the last line

read returns a success exit status when it reads a line and finds the newline that terminates it. When the file does not end in a newline, read still populates the variable with the final line's content, but it returns a non-zero (failure) exit status because it hit end-of-file before finding a terminator. A while read host loop checks the return status to decide whether to execute the loop body. On the final, newline-less line, read puts app-07 into host and then returns failure. The while loop sees failure and exits without running the body. The content is there. The variable is populated. The loop throws it away.

This behavior is documented in the POSIX spec for read. It is not a bash quirk. Any POSIX shell handles the missing-final-newline case this way. A plain while read line loop is incorrect for any file you do not personally control the formatting of, which in practice means nearly any file.

The fix is one extra clause:

#!/bin/bash
# Script: check-hosts.sh
# Purpose: ping every host in a file, including a newline-less final line
# Usage: ./check-hosts.sh hosts.txt
set -euo pipefail

CHECK="✓"
CROSS="✗"
HOST_FILE="${1:?Usage: check-hosts.sh <host-file>}"

while IFS= read -r host || [[ -n "$host" ]]; do
  [[ -z "$host" || "$host" == \#* ]] && continue
  if ping -c1 -W2 "$host" >/dev/null 2>&1; then
    echo "$CHECK up:   $host"
  else
    echo "$CROSS down: $host"
  fi
done < "$HOST_FILE"

|| [[ -n "$host" ]] says: if read returned failure but the variable is non-empty, run the loop body anyway. That is precisely the leftover-final-line case. read failed because it hit end-of-file, but it populated host with app-07 before returning. The || catches it. app-07 gets pinged.

What IFS= and -r actually do

You see while IFS= read -r line written in every correct read-loop example, and it is worth being specific about what each piece prevents because both have their own failure mode.

IFS= sets the field separator to empty for the duration of the read command. Without it, read strips leading and trailing whitespace from each line. A hostname like app-07 (with leading spaces, which some editors produce) becomes app-07, which might be correct. An indented config value, a Python-style YAML string, a log line that starts with spaces for alignment — all of these are silently modified. Setting IFS= tells read to take the line exactly as it appears.

-r prevents read from interpreting backslash sequences. Without -r, a line like C:\temp\logs has its backslashes consumed as escape characters and arrives as C:templogs. This matters less for hostname files and enormously for any script that processes Windows paths, config files that use backslash as a line-continuation character, or log files from mixed-OS environments. The -r flag is essentially free protection; there is no reason not to include it.

A bare read line without either flag silently mangles both whitespace and backslashes. The script works correctly on clean input and produces wrong output on input with edge cases. The wrong output does not produce an error. You find out when the data that mattered was the indented or backslash-containing kind.

The subshell trap that kills your counters

This is the one that is most likely to make you question your sanity:

# This looks correct. It is not.
fails=0
cat hosts.txt | while IFS= read -r host; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done
echo "Total failures: $fails"   # Always prints 0

The pipe creates a subshell for the right side. The while loop runs inside that subshell. fails increments correctly inside the subshell. When the subshell exits, the parent shell's fails is still 0, because the increment happened in a different process. The parent echo sees the original value.

This catches people because the loop body itself works — the pings happen, the increment logic is correct — but any state the loop was supposed to accumulate for later use is silently discarded. I spent forty minutes on a version of this problem before I remembered that pipes create subshells. It is the kind of thing that feels like a bash bug until you understand that it is behaving exactly as documented.

The fix is to redirect the file into the loop instead of piping into it:

fails=0
while IFS= read -r host || [[ -n "$host" ]]; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done < hosts.txt
echo "Total failures: $fails"   # Now correct

done < hosts.txt feeds the file to the loop's stdin without a pipe. The loop runs in the current shell. fails accumulates in the current shell. The echo sees the real count.

Why for loop is wrong for this

# Never do this — iterates words, not lines
for line in $(cat hosts.txt); do
  ping -c1 "$line"
done

$(cat hosts.txt) is command substitution. Bash captures the text output and word-splits it on IFS — spaces, tabs, newlines. For a file with one hostname per line and no spaces in the hostnames, this accidentally produces the right behavior. For any file with spaces — log lines, config values, paths with spaces, anything a non-developer might have generated — it splits lines into fragments and each fragment becomes a loop iteration.

There is no version of for line in $(cat file) that is correct for reading lines. The right tool is always while IFS= read -r line || [[ -n "$line" ]]; do ... done < file. The for loop is the right tool for iterating a known list that you control directly, not for reading file content.

The monitoring system, after the fix

After the app-07 incident we made three changes. The obvious one was fixing the read loop with the || [[ -n "$host" ]] guard. The second was adding a sanity check at the top of the script that counted the lines in the hosts file and compared it to the number of hosts the loop actually processed — a mismatch meant the file was malformed or something else was wrong. The third was adding a nightly email that included the count of hosts checked, not just the status of each one, so a future addition to the file that somehow got lost would show up as "expected 12 hosts, checked 11."

The confirmation email count was something I should have had from the start. If I had been looking at "checked 11/12 hosts" instead of a list of OK lines, I would have noticed app-07 missing on the first night. The monitoring was working. The observability of the monitoring was not.

Full version with CSV-field parsing, comment-skipping, and the subshell-safe redirect form: https://bashsnippets.xyz/snippets/bash-read-file-line-by-line

To iterate a list of files rather than a file's contents, reach for a for loop instead, and wrap anything that acts on what it reads in set -euo pipefail. More at https://bashsnippets.xyz

A For Loop Skipped Every File With a Space and Called the Backup a Success

Anguishe — Thu, 18 Jun 2026 19:29:49 +0000

The nightly backup looped over for f in $(ls /data/exports) and copied each file to a backup volume. It exited clean every night. For three weeks, green exit codes, no errors, nothing in the logs to suggest anything was wrong. The backup script had been written by someone who left the company six months before I started, and it had never been tested against files with spaces in their names because the original export directory only ever had files like Q3.xlsx and report-final.xlsx.

Then someone generated Q3 final.xlsx.

It took three weeks because that file was generated once a quarter and the next time someone needed it was a quarter later. The person who needed it was the CFO. The CFO does not particularly enjoy being told that the backup system that was supposed to protect the quarterly export has been silently failing for an indeterminate period of time and we are not sure which other files it missed.

I know the specific backup had been running for three weeks because that was the date the file appeared in the exports directory. Every night since then, the loop had been splitting Q3 final.xlsx into two items — Q3 and final.xlsx — trying to copy two paths that did not exist, logging two harmless "no such file or directory" lines that nobody read, and moving on. Every file without a space in its name backed up fine. The script looked correct because most of the time it was.

Why $(ls) word-splits

When you write for f in $(ls /data/exports), bash runs ls, captures its text output, and splits it on IFS — the internal field separator, which defaults to spaces, tabs, and newlines. Filenames are just text in the output of ls. A file named Q3 final.xlsx is one filename, but ls outputs it as the string Q3 final.xlsx, and bash splits that string on the space into two separate items before the loop ever starts.

This is not a bug in ls. This is not a bug in bash. This is exactly what $( ) does to any command's output — it captures text and bash processes it as text. The problem is that filenames are not reliably text-safe; they can contain any character except null and the path separator. Spaces are common. Tabs are less common but legal. Newlines are technically legal. Treating command output as a filename list breaks the moment any of those show up.

The fix is to stop treating command output as a filename list and let bash build the list from the filesystem directly:

#!/bin/bash
# Script: backup-exports.sh
# Purpose: copy export files without losing ones with spaces in their names
# Usage: ./backup-exports.sh
set -euo pipefail

CHECK="✓"
CROSS="✗"
SRC_DIR="/data/exports"
DEST_DIR="/backup/exports"
shopt -s nullglob   # empty glob expands to nothing, not the literal pattern

for f in "$SRC_DIR"/*.xlsx; do
  if cp -- "$f" "$DEST_DIR/"; then
    echo "$CHECK backed up: $(basename "$f")"
  else
    echo "$CROSS failed: $(basename "$f")"
  fi
done

for f in "$SRC_DIR"/*.xlsx asks bash to expand the glob. Bash talks directly to the filesystem and gets back a properly-separated list of matching paths. No command output, no text splitting, no ambiguity about what the separator is. A file named Q3 final.xlsx stays one item because it was never turned into text and split back apart. The -- before "$f" tells cp to stop reading flags, so a filename that starts with a dash does not get interpreted as a cp option.

The second half: quoting on use

The glob fixes the loop header. Quoting fixes the point of use. These are separate.

for f in "$SRC_DIR"/*.xlsx; do
  cp $f "$DEST_DIR/"    # WRONG — $f re-splits here
  cp "$f" "$DEST_DIR/"  # RIGHT — the quotes prevent the split
done

Even with a correct glob, an unquoted $f in the cp command re-splits on spaces. Bash has already expanded the variable to Q3 final.xlsx, but when you use it unquoted, the shell processes word splitting again on that value and cp receives two arguments: Q3 and final.xlsx. The quotes around "$f" tell bash to pass the entire value as a single argument. The rule is: glob over parse to build the list, quote on use to keep each item intact.

I have seen this mistake repeated in scripts at three different companies. In each case the scripts had been running for months or years and the problem was invisible because the majority of files had no spaces. The ones that did — client names, quarterly reports, anything a non-technical person named — were being silently skipped or mishandled.

Ranges and counters: where the brace trap hides

n=5

# This does NOT produce a range — it produces the literal string {1..5}
for i in {1..$n}; do echo "attempt $i"; done

# This works — C-style, evaluates variables at runtime
for ((i = 1; i <= n; i++)); do echo "attempt $i of $n"; done

Brace expansion happens before variable expansion in bash's order of operations. By the time $n is replaced with its value, the brace expansion step has already passed and {1..$n} is just a string. This is the kind of thing that fails silently in a test environment where the count is small and hardcoded, and causes weird output in production where it is a variable.

The C-style loop is the correct form any time the bound is a variable. It evaluates at runtime and handles arithmetic naturally. If the count is genuinely fixed at write time, {1..10} works fine. If it is ever going to be a variable, use for (( )).

Arrays: the original bug wearing a different hat

servers=("web-01" "db primary" "cache-02")

# WRONG — word-splits "db primary" into two iterations
for s in ${servers[@]}; do
  ping -c1 "$s"
done

# RIGHT — quotes keep each element intact
for s in "${servers[@]}"; do
  ping -c1 "$s"
done

"${servers[@]}" with the quotes and [@] is the form that preserves each element as a single item regardless of what is in it. Without the quotes, bash word-splits the array expansion and db primary becomes two separate loop iterations — neither of which is a real hostname. This is exactly the same word-splitting mechanism as the $(ls) problem, just manifesting in arrays instead of command output.

Once you internalize that word-splitting happens wherever an unquoted variable or expansion appears, the rule becomes one rule instead of several: always quote expansions. The specific context changes; the mechanism does not.

The nullglob case

shopt -s nullglob
for f in /data/exports/*.xlsx; do
  echo "$f"
done

Without shopt -s nullglob, if no .xlsx files exist, the glob *.xlsx does not expand — it stays as the literal string *.xlsx. The loop runs once with f set to the literal string /data/exports/*.xlsx. Your script then tries to process a file with that exact path, which does not exist, and produces an error or silently does nothing depending on what you do with it.

Setting nullglob tells bash to expand a non-matching glob to nothing (an empty list), so the loop simply does not run. This is almost always the right behavior when you are iterating files that might not exist.

What happened to the Q3 export

We recovered it from the CFO's local machine, where she had downloaded it before the backup was supposed to preserve it. The fix to the script took four minutes. The conversation about why the backup system had been failing silently for three weeks took longer. The monitoring that we added afterwards — a nightly check that the backup directory has at least as many files as the source directory — took another twenty minutes.

The monitoring should have been there from the start. So should the glob. So should the set -euo pipefail that would have made the copy failures loud instead of silent. These are things you add before something breaks, and the only reason to know you need them is to have seen, or caused, or read about what happens when they are missing.

Full examples with the safe glob, counter, C-style loop, array form, and nullglob guard: https://bashsnippets.xyz/snippets/bash-for-loop-examples

For reading a file's lines one at a time, a for loop is the wrong tool — use while IFS= read -r — and wrap any loop that touches real files in set -euo pipefail. The rest is at https://bashsnippets.xyz

find . -delete Ran Before the Filter and Emptied the Whole Tree

Anguishe — Wed, 17 Jun 2026 23:59:08 +0000

I meant to delete the .cache files under a data directory. The server had been running for two months and the cache layer had grown to about 14GB. The application team told me it was safe to purge it — they'd rebuilt the cache logic and the old files were just dead weight. I typed find /data -delete -name "*.cache" because I was moving fast and I figured the order of arguments to find did not matter. It does. find evaluates its expression left to right, and -delete is not a filter — it is an action. It fired on every path find walked, starting at /data itself, and -name "*.cache" never got a chance to narrow anything. By the time I hit Ctrl-C the tree was roughly 80% gone.

That was not a test server.

The restore from backup took forty minutes. The application was down for those forty minutes. The postmortem was a forty-five minute conversation with people who did not particularly enjoy having it. I have run hundreds of find commands since then and I verify the expression order before every single one that has any destructive action attached to it — not because I've forgotten the rule, but because the cost of forgetting it once is not recoverable with an apology.

Why the order is the program

find does not have a flag parser that groups tests and actions separately. It walks a directory tree and evaluates its arguments as a logical expression, left to right, short-circuiting on false. When you write:

find /data -name "*.cache" -delete

it evaluates -name "*.cache" first on each path. If the name does not match, it short-circuits and -delete never runs on that path. When you write:

find /data -delete -name "*.cache"

it evaluates -delete first. -delete always succeeds — it removes the path and returns true, which means the expression continues to -name. The name check runs after the deletion, on a file that no longer exists, which is meaningless. The effect is that everything gets deleted and nothing is filtered.

This is not a bug. It is exactly how the man page says find works. It is just not how anyone instinctively reads a command the first few times they use it.

The rule is: tests filter, actions act, and actions must come after the tests that are supposed to narrow them. Write it in that order every time.

The quoting trap, right behind the ordering one

Here is the one that is even more subtle, because it causes the wrong behavior and produces no error at all:

# WRONG — the shell expands *.cache before find ever sees it
find /data -name *.cache

# RIGHT — quote the pattern so find does the matching
find /data -name "*.cache"

If there happens to be a single .cache file in your current working directory when you run this, the shell expands *.cache to that one filename and passes it to find -name as a literal string. find then searches for files named exactly that, everywhere under /data. It finds some, it finds none, but it is definitely not doing a wildcard search. If there are multiple .cache files in your current directory, find receives too many arguments for -name and errors out with something confusing about paths needing to precede the expression.

Either way you did not get what you intended, and if you added -delete, you deleted the wrong things silently.

Quoting the pattern is the fix. The quotes prevent the shell from expanding the glob so find receives the literal *.cache pattern and handles the wildcard itself, across the directory tree you pointed it at.

The age sign that everyone reverses at least once

find /var/log -name "*.log" -mtime +30   # older than 30 days
find /var/log -name "*.log" -mtime -1    # modified within the last day
find /var/log -name "*.log" -mtime 30    # exactly day 30 (almost never what you want)

+30 is older than thirty days. -1 is within the last day. A bare 30 means precisely thirty days ago, which is almost never the thing you're trying to match. The sign convention is the opposite of what feels natural — you want files "older than" a threshold and the intuitive symbol for "bigger number" is +, but a lot of people read +30 as "in the last thirty days" the first time they see it.

I have reversed this twice in production. Once I kept the wrong logs. Once I deleted logs I needed for an audit the following week. Neither was catastrophic but both were embarrassing, and both happened under the kind of mild time pressure where double-checking the man page feels slower than it actually is. The builder I built for this labels the output as "older than" or "within the last" in plain English next to the value, which removes the one decision in a find -delete job most likely to go wrong when you are already stressed.

The -exec batching difference nobody explains

# Runs the command once per file — slower, one PID per file
find /data -name "*.cache" -exec rm {} \;

# Batches files into one invocation — faster, one rm for many files
find /data -name "*.cache" -exec rm {} +

\; runs the command once per file. + collects as many paths as it can and passes them all to one invocation of the command, the same way xargs does. For something like rm, which accepts multiple arguments, the + form is faster and produces less process overhead. For something like a custom script that must process exactly one file at a time, \; is correct.

Most resources either do not mention this difference or mention it once in a reference table. The practical consequence is real — on a directory with ten thousand files, \; spawns ten thousand processes. The + form spawns a handful. On a cleanup job that runs in cron, the difference shows up in CPU load.

The preview step I skipped the day I broke things

The most reliable way to avoid the ordering and quoting mistakes is to build the command in two steps. First, run it with -print instead of -delete:

find /data -name "*.cache" -print

Read every line of that output. Confirm the list is what you intended. Then, and only then, swap -print for -delete. This adds maybe thirty seconds to the workflow. It would have saved me forty minutes and a postmortem. I skipped it because I was confident. Confidence is not a useful substitute for verification when the operation is irreversible.

The find command builder enforces this by showing a warning whenever you select -delete or -exec as the action, and offering to generate the -print version of the command first. It is the kind of nudge I would have appreciated having that day.

What the builder actually does

It assembles the expression in the legally correct order — tests first, action last — so you cannot accidentally replicate the mistake I made. You set the starting path, add tests in whatever order feels natural to you (name, type, age, size, exclude path), and pick an action. The output command always has the tests before the action, regardless of the order you clicked things in.

Every active flag gets a plain-English description inline. -mtime +30 reads as "modified more than 30 days ago." -name "*.cache" reads as "name matches the glob *.cache, quoted so find handles the wildcard." The -exec {} + form is the default when you pick -exec, with a note explaining why it is faster.

The whole point is to get from "I need to find and delete files matching these conditions" to a verified, copy-paste command without the thirty-second loop of man-page reading and second-guessing that I used to do and, on one bad morning, skipped.

What I upgraded while I was at it

The same discipline applies to the other tools. The rsync command builder now has presets for the three setups people build most often — local backup, push to remote, mirror — because those cover maybe ninety percent of rsync jobs and getting the flags right from scratch every time is where the dangerous ones like --delete get misapplied. The cron builder previews the next five run times after you build an expression, because a cron job I once deployed ran at 3am UTC instead of 3am local time and I did not notice until it fired on the wrong schedule for a week. You can also paste an existing crontab line and get the human-readable schedule back. The chmod builder now accepts an octal you paste in and sets the checkboxes — two-directional, because reading a file's permissions and understanding what they mean is just as common a task as setting them from scratch.

These are not features I planned in advance. They are all things I needed at 2am and did not have. That is the pattern most of this site runs on.

Build a find command with tests ordered before actions and every flag explained: https://bashsnippets.xyz/tools/find-command-builder

If you are scoping files before searching or transforming them, the whole pipeline is documented in Bash Text Processing: find, grep, sed, and awk. The rest of the free tools are at https://bashsnippets.xyz/tools

A for Loop Skipped 23 Files and Called It a Successful Backup

Anguishe — Sun, 14 Jun 2026 17:26:03 +0000

The backup ran every night at 2am and emailed me a green "847 files archived" summary. I'd built it, tested it against my own home directory where every file was named like report_2024.csv, watched it sail through, and shipped it. For weeks the summary said everything was fine.

Then a coworker asked me to restore a file. Q3 forecast.xlsx. It wasn't in the archive. Neither was Annual Review FINAL.pdf, or meeting notes (draft).md, or any of the other 23 files someone had named the way normal humans name files — with spaces in them. The backup had been quietly skipping the most important files on the share for a week, and the nightly email had been telling me the whole time that nothing was wrong.

Here's the line that did it:

for file in $(ls "$DIR"); do
  echo "Processing $file"
done

The problem is word splitting, and it's invisible until the day it isn't. An unquoted $(ls ...) splits its output on every space, so Q3 forecast.xlsx doesn't arrive as one filename — it arrives as two iterations, Q3 and forecast.xlsx. Neither one exists on disk. The loop tries both, finds nothing, shrugs, and moves on without a single error. The script "succeeds" because from its point of view it did exactly what it was told.

I've now got two habits burned in, and I don't write a loop without both. The first: glob the directory, never parse ls.

for file in "$DIR"/*; do
  [[ -e "$file" ]] || continue   # the glob yields a literal '*' on an empty dir
  echo "Processing $file"
done

"$DIR"/* hands you each real path as a single unit. No subshell, no string to re-split, no ls output to misread. The [[ -e "$file" ]] || continue guard covers the one quirk of globbing: when a directory is empty, * expands to the literal character *, and without the guard you'd try to process a file named *.

The second habit: quote every expansion, every time. "$file", never $file. The day a filename has a space in it, the unquoted version splits into two arguments and your command operates on paths that were never there.

Arrays follow the exact same rule, and the distinction that matters is [@] versus [*]:

servers=("web-01" "db-prod 02" "cache-03")

for host in "${servers[@]}"; do
  echo "Connecting to $host"   # three clean iterations, the space survives
done

"${servers[@]}" in double quotes gives you one word per element — db-prod 02 stays whole. "${servers[*]}" joins everything into a single string and is almost never what you want in a loop. Drop the quotes on either and you're back to the bug that ate my backup.

When you genuinely need a counter — walking app.log.1 through app.log.9, numbering batches, counting retries — use the C-style form. Do not reach for for i in {1..$n}; brace expansion runs before variable expansion, so $n never gets substituted and you loop once over the literal text {1..$n}. Ask me how I know.

for ((i = 1; i <= MAX_ROTATED; i++)); do
  log="$LOG_DIR/app.log.$i"
  [[ -f "$log" ]] || continue
  echo "Scanning $log"
done

And reading a file line by line — for line in $(cat file) is wrong in two directions at once. It splits on whitespace instead of newlines, so a line with spaces becomes several iterations and blank lines vanish, and it globs, so a line containing * expands to filenames in your current directory. The form that survives real files:

while IFS= read -r line; do
  echo "Line: $line"
done < "$INPUT_FILE"

IFS= stops bash from trimming leading and trailing whitespace. -r stops it from eating backslashes. One line per iteration, exactly as written.

The production script I keep on the site ties all of this together — globs the directory, quotes every use, counts successes and failures separately, and exits non-zero if anything failed. That last part is the one people skip, and it's the one that matters: a loop that processes files but always exits 0 is how you end up with a nightly email that says 847 when the real number is 824.

The cost of my mistake was a week of bad backups and the specific discomfort of a coworker finding the gap before my own tooling did. I'm not interested in repeating that, so the quoting is muscle memory now. Yours might as well be too.

Full script with all four loop forms — files, arrays, counters, and line-by-line reads — production-ready and ShellCheck-clean: https://bashsnippets.xyz/snippets/bash-for-loop-examples

If your loops are the thing crashing scripts, the Bash Error Handling snippet pairs with this one, and the rest of the library is at https://bashsnippets.xyz

The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

Anguishe — Fri, 12 Jun 2026 16:27:11 +0000

For three weeks a deployment pipeline reported every step green and shipped a build that had failed to compile on every single run. The build step ended in npm run build | tee build.log so the output could be archived. That pipe is the whole story: bash returns the exit status of the last command in a pipeline, which was tee, and tee always succeeds at copying text. The compiler's non-zero exit got thrown away the instant the pipe handed off. The error was sitting right there in build.log. GitHub Actions saw exit code 0, painted the step green, and deployed the broken artifact. Nobody read the log, because the checkmark said there was nothing to read.

That's the defining property of bash in CI, and it's why I treat pipeline scripts differently from anything I run in a terminal: a silent failure can present as success. On a server you watch a command fail in front of you. In a pipeline, a swallowed exit code produces a green checkmark over broken code, and the gap between "the logs show an error" and "the pipeline reports failure" is exactly where outages are born. I wrote the full guide because I've now been burned by every variation of this, and there's a consistent set of habits that close the gap.

There are four failure modes that are specific to CI and barely ever bite you at an interactive prompt. Exit codes swallowed by a pipe — the story above, any command | tee, command | grep, command | sort. Shell provisioning differences — ubuntu-latest gives you bash 5.x, macos-latest gives you bash 3.2 from 2007, and a script using associative arrays or ${var,,} passes on one runner and throws a syntax error on the other in the same workflow. Environment variable gaps — CI sets variables you don't control and omits ones you assume exist, and without set -u a missing $DEPLOY_TARGET becomes an empty string and does something quietly wrong. Interactive-shell assumptions — CI runs a non-interactive, non-login shell that never sources your .bashrc, so a command that works when you type it dies with command not found because the thing that defined it was never loaded.

The header that closes most of these is short, and every line earns its place:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

set -e exits the moment any command fails, so the step exits non-zero and the workflow actually registers a failure. set -u treats an unset variable as an error, so a typo'd $DPLOY_TARGET dies immediately instead of expanding to nothing and corrupting a path. set -o pipefail makes a pipeline return the first non-zero exit among its commands rather than only the last — that one flag is the direct fix for the | tee bug that ran green for three weeks.

Secrets deserve their own paragraph because CI hands you env: values and secrets: values identically — the shell can't tell them apart, the only difference is that Actions masks the secret's literal string in the log. The trap is that the moment you transform a secret (base64-decode it, slice it, interpolate it), the transformed value no longer matches the mask and prints in clear text. Validate required values up front with ${VAR:?} so a missing secret fails at startup with a clear message instead of on line 47 with a cryptic permission denied, and be very careful with set -x in any step that touches a secret.

The pipe-exit-code problem is worth one concrete tool beyond pipefail: PIPESTATUS is an array holding the exit code of every command in the last pipeline, read immediately after:

npm run build | tee build.log
build_rc="${PIPESTATUS[0]}"   # npm's code, not tee's
[[ "$build_rc" -eq 0 ]] || { echo "build failed: $build_rc" >&2; exit "$build_rc"; }

pipefail has one well-known false positive — grep returns 1 when it finds no matches, which is often fine, and under pipefail plus set -e that aborts the script. Absorb it deliberately with || true only where a non-match is genuinely acceptable, and nowhere else, because blanketing every command in || true just reinvents the silent-success problem you're trying to kill.

Docker has its own landmine: every entrypoint script must end with exec "$@". Without it, your script stays PID 1 and your app runs as a child, so when the orchestrator sends SIGTERM on docker stop or a rolling deploy, the signal hits the script, which doesn't forward it, and after the grace period the orchestrator escalates to SIGKILL — abrupt termination, dropped connections, lost in-flight work. exec "$@" replaces the shell with your app so it becomes PID 1 and receives signals directly. The guide pairs this with a wait_for dependency-check pattern and a trap, and the Bash trap & Signal Handler Builder generates the exact signal block an entrypoint needs.

Deploys get the same treatment: deploy into a fresh timestamped directory, flip a current symlink atomically with ln -sfn so traffic never sees a half-written release, keep the last several releases so rollback is just re-pointing the symlink, run a health check after the swap and fail the deploy if it doesn't pass, and stamp the git SHA into the release so "what's running right now" always has an answer. And when a step fails and the logs won't say why, set -x around just the suspect section shows you each command with its variables expanded — a doubled slash or an empty segment in the trace is usually your bug standing in plain sight.

The full guide is the field manual version of all of this — the four failure modes, the safe header, secret validation with a validate_env function, PIPESTATUS and pipefail, Docker entrypoints, the atomic-symlink deploy script with rollback and health check, debugging with set -x, and a production-ready checklist at the end: https://bashsnippets.xyz/guides/bash-scripting-for-ci-cd-pipelines

If your pipeline scripts are dying on the small stuff first — unquoted loops, bad argument parsing, missing traps — the snippet library that feeds into this guide is at https://bashsnippets.xyz

My Script Crashed and Left a Lock File Behind. Every Run After That Refused to Start.

Anguishe — Thu, 11 Jun 2026 16:56:45 +0000

A backup script of mine created a lock file on startup so two copies couldn't run at once — sensible. Then one night it hit an error partway through, set -e killed it on the spot, and it died without ever reaching the line that removes the lock. The lock file sat there. Every scheduled run for the next three days started up, saw the lock, printed "already running," and exited immediately. No backups ran. The cron job was firing perfectly on time and doing nothing, and the only symptom was an absence — backups that simply weren't there — until I went looking and found a stale lock from Tuesday.

The fix is a trap. A trap registers a cleanup handler that runs when the script exits for any reason — clean finish, set -e failure, Ctrl+C, kill. Put the lock removal in an EXIT trap and it runs no matter how the script dies. The lock would have been gone the instant that backup crashed, and the next run would have started fine.

So why did I write the script without one? Because I could never remember the syntax cold. Single quotes or double? Which signals? Does EXIT fire on exit 1 or only on a clean finish? How do I get the exit code inside the handler? Every time I needed a trap I ended up with three browser tabs open, reading the same Stack Overflow answers, second-guessing the quoting. Under pressure, in the middle of fixing something else, that friction is exactly when people skip the trap entirely — which is how I ended up with the stale lock in the first place.

So I built the thing I kept wishing existed: a Bash trap & Signal Handler Builder. You pick the signals you want to handle, check off the cleanup actions you need, and it writes a correct, ShellCheck-clean trap block you paste into your script. No tabs, no second-guessing the quoting.

It covers the signals that actually come up. EXIT — fires on every exit, the one that should carry your cleanup. ERR — fires when a command fails under set -e, the one that logs the exact failing line with $LINENO. INT — Ctrl+C. TERM — what kill, systemctl stop, and docker stop send. HUP — terminal closed or SSH dropped. PIPE — writing to a closed pipe. Each one has a one-line reminder of when it fires and what it's good for, because half the battle is just remembering that TERM is the one Docker sends.

The cleanup actions are the things people forget until a crash makes them care: remove temp files (with the TMPFILE=$(mktemp) declaration wired in up top), remove a lock file — the exact failure that bit me — stop background jobs the script started, log the exit reason with the code, and restore the terminal cursor. Tick the ones you need and they land in the handler.

The generated code isn't a toy snippet. It single-quotes the trap so the handler resolves when the signal fires instead of at definition time — the SC2064 gotcha most hand-written traps get wrong. It includes an idempotency guard so that when ERR and EXIT both fire on the same failure, your cleanup runs exactly once instead of twice. The ERR handler captures $LINENO so you find out which line actually blew up. I ran the generator's output through ShellCheck across a dozen different configurations while building it, and every one comes back clean — the whole point was that you can paste it and trust it, not paste it and then go debug the thing that was supposed to save you debugging.

There are two copy buttons, because there are two situations. "Copy trap block only" when you've already got a script and just want to drop the traps in. "Copy complete script header" when you're starting fresh and want the shebang, set -euo pipefail, the CHECK/CROSS vars, the resource declarations, and the handler all assembled in the right order. It only declares the variables the generated code actually uses, so you never paste in a CROSS you never reference and get a ShellCheck warning for your trouble.

The lock-file incident cost me three days of silently missing backups and ten minutes of feeling foolish when I found the cause. The trap that would have prevented it is four lines. The reason I didn't have those four lines was pure friction — I couldn't recall the syntax fast enough to be bothered in the moment. This tool removes the friction, which is the only thing that was ever standing between me and a correct script.

Build your trap block here — pick signals, pick cleanup actions, copy clean code: https://bashsnippets.xyz/tools/bash-trap-builder

If you want the full picture of where traps fit — strict mode, cleanup, the failure modes that make them necessary — the Bash Error Handling snippet is the companion read, and the rest of the tools are at https://bashsnippets.xyz/tools

I Packaged the Scripts I Copy to Every New Server Into a $9 Toolkit. Here's What's In It and Why.

Anguishe — Tue, 09 Jun 2026 15:44:19 +0000

Every time I provision a new server — whether it's a $5 DigitalOcean droplet for a side project or a client's production box — there's a set of scripts I copy to /opt/scripts before I do anything else.

Not after the app is deployed. Not after the first incident. Before I touch the application at all. Before I configure nginx. Before I set up the database. The monitoring layer goes in first because the first time you need it, you needed it yesterday.

Disk monitoring that fires before the outage. A backup pipeline with automatic retention. SSL certificate checks that run daily at 8am so I'm not finding out from a user's email. A service watchdog that restarts nginx or Postgres within 60 seconds of a crash instead of six hours later when someone notices the site is down.

These scripts took me about two years of production incidents to build. Not because they're complicated — they're not. Each one is 15-50 lines. Because each one was built in response to a specific failure where I didn't have the thing I needed and had to build it under pressure at a bad time of day.

I open-sourced the basic versions on BashSnippets.xyz. Those are free and they'll stay free. But the versions I actually run in production are different from the tutorial versions in ways that matter — and that gap is what I packaged into the toolkit.

What's Different Between the Free Snippets and the Toolkit

The free snippets on the site are single-purpose scripts that each solve one problem. They're complete. They work. If all you need is a disk space check, the free version does that.

The toolkit versions are built as a system.

There's a shared library — bashlib.sh — with 31 functions that every script sources. Logging, color output, email alerts, error handling, lock file management, threshold checks, dry-run support. Instead of each script reimplementing its own log() function and its own error handling and its own email logic, they all call bashlib.sh and get consistent behavior across the board.

That means when I change how logging works, it changes everywhere. When I add Slack webhook support to the alert function, every script that calls alert() gets Slack notifications without any changes to the script itself. The shared library is the infrastructure layer that turns six standalone scripts into a cohesive system.

The free snippets don't have this because a shared library adds a dependency — bashlib.sh has to exist at a known path, the scripts have to source it at startup, and if someone downloads one script without the library it breaks. That's fine for a toolkit you install as a package. It's bad for a tutorial page where someone wants to copy-paste one script and have it work immediately. Both approaches are correct for their context.

What's in the Box

6 production scripts:

Each one follows the same structure — set -euo pipefail, sourcing bashlib.sh, named variables for every threshold and path (no magic numbers buried in command pipelines), comments explaining not just what each line does but why it exists, and explicit non-zero exits on every failure path.

The scripts cover disk space monitoring, database backup with retention, SSL certificate expiry checking across multiple domains, service watchdog with automatic restart, log rotation and cleanup, and system health reporting.

bashlib.sh — the shared library (31 functions):

This is the part I'm most particular about. Functions for timestamped logging with severity levels. Color-coded terminal output that degrades gracefully when piped to a file (no escape codes in your log files). Email and webhook alerting. Lock file acquisition with stale-lock detection. Dry-run mode that every function respects. PID file management. Threshold comparison helpers. Configuration file loading.

Every function is documented with a usage comment. Every function handles its own error cases. The library passes ShellCheck with zero warnings at every severity level.

template.sh:

A starter template that sources bashlib.sh, sets up traps, parses arguments, and has placeholder sections for your own logic. When I need a new script on a server, I copy this template, fill in the business logic, and the error handling and logging are already done.

52-page field guide (PDF):

Not an API reference. Not a man page reformatted as a PDF. A field guide — structured as the things you need to know in the order you need to know them when you're setting up automation on a new server.

Covers the why behind every pattern in the scripts. Why set -euo pipefail and what each flag actually prevents. Why traps on EXIT instead of just INT. Why lock files need stale detection. Why backup retention has to be a separate step from the backup itself. Why SSL monitoring should be independent of your renewal tool.

Each section includes the real failure scenario that motivated the pattern, because "best practice" without the consequence attached is advice that gets skipped.

Why $9

I thought about this for a while. The scripts are worth more than $9 to anyone who's going to use them — preventing one 4am incident pays for the toolkit immediately. But I also know what it's like to be the person running a $5 VPS on a budget, and I wanted the price to be low enough that buying it doesn't require approval from anyone or a second thought.

$9. MIT license. Unlimited personal and commercial use. You can deploy these on client servers, modify them however you want, include them in your own automation. No subscription. No upsell. No "starter tier" with premium features behind another paywall.

The free snippets on the site are not going away. They're not a demo. They're complete, working scripts that I actively maintain. The toolkit is for people who want the production layer — the shared library, the integrated system, the field guide that ties it together — and want it in one download instead of building it themselves.

Who This Is For

If you're managing one or more Linux servers and you don't already have automated monitoring, backups, and alerting set up — this is the fastest path to having all three. Copy the scripts to the server, edit the config variables at the top of each file, add the cron entries from the guide, and you have a monitoring layer that didn't exist 10 minutes ago.

If you already have these things set up and you built them yourself, you probably don't need this. You might find something useful in bashlib.sh — the stale lock detection or the dry-run mode — but the scripts themselves won't tell you anything new.

If you're learning bash and want to see how production scripts are structured differently from tutorial scripts, the field guide is probably the most useful part. The "why" sections explain patterns that most bash tutorials skip because they're not relevant to a single-file script running on a laptop.

Every Script Passes ShellCheck at Zero Warnings

This one matters to me. ShellCheck is the static analysis tool for bash. It catches the bugs that work on happy-path input and break on edge cases — unquoted variables that split on whitespace, pipelines that swallow exit codes, deprecated syntax that newer bash versions handle differently.

Every script in the toolkit, including bashlib.sh, passes ShellCheck at the strictest severity level with zero warnings. Not "a few style notes we decided were fine." Zero. I treat ShellCheck warnings the same way I treat compiler warnings in C — they are bugs I haven't hit yet, and ignoring them is technical debt with a guaranteed due date.

If you run ShellCheck on these files and get output, something went wrong and I want to know about it.

The Link

→ bashsnippets.xyz/starter-kit

$9. Instant download. 6 production scripts + bashlib.sh shared library (31 functions) + template.sh + 52-page field guide. ShellCheck-clean. MIT license.

Already have scripts and need somewhere to run them? DigitalOcean droplets start at $4/month and you can get $200 in free credit to start:

→ Get $200 free credit — DigitalOcean

The free snippet library with 17+ scripts and 7 interactive tools is at:

→ bashsnippets.xyz

6 small things we shipped across the BashSnippets tools this week

Anguishe — Tue, 09 Jun 2026 03:49:07 +0000

Nobody announces small features. You ship them, they're in there, and the people who find them either notice or they don't. I want to start documenting these because some of them are the kind of thing that makes a tool actually worth using day-to-day instead of being something you visit once and close.

Six updates across six tools this week. None of them are headline features. All of them are fixes for specific annoyances that came up in my own usage, which is the only real source I trust for "does this actually help."

1. Bash Boilerplate Generator — Trap & Cleanup Handler

There's a now a toggle for trap handling. When on, the generated template includes:

cleanup() {
  # Remove temp files, release locks, undo partial changes
  rm -f "$TMPFILE"
}

trap cleanup EXIT INT TERM

Why this matters: trap cleanup EXIT means the cleanup() function runs no matter how the script exits. Normal completion. Ctrl+C. An uncaught error. A kill signal. The cleanup function runs. Every time.

Without this pattern, scripts that create temp files leave them behind when they're interrupted. Scripts that acquire lock files leave them locked. Scripts that make partial changes — creating a directory, writing part of a config — leave the system in an inconsistent state because the cleanup code at the end of the script never ran.

The trap-on-exit pattern is not optional for anything running unattended. It's the thing that separates "runs fine when nothing goes wrong" from "also recovers gracefully when something does." I've seen this pattern left out of boilerplate generators enough times that I wanted to make it explicit and easy to include.

The generated stub includes a TMPFILE=$(mktemp) line as a concrete example of something that needs cleanup. Replace it with whatever state your script actually manages.

→ bashsnippets.xyz/tools/bash-boilerplate-generator

2. Bash Exit Code Lookup — Export as `case` Statement

After looking up an exit code, there's a button that generates a ready-to-paste case $? in ... esac block with the explanation inline as a comment.

Before this, the lookup gave you the meaning of the exit code and you had to write the case handler yourself. That's not hard — the case syntax is simple — but it's friction. You looked up what 126 means ("command found but not executable — check permissions"), now you have to translate that into:

case $? in
  0)   echo "Success" ;;
  1)   echo "General error" ;;
  126) echo "Command found but not executable — check file permissions" ;;  # ← the one you just looked up
  127) echo "Command not found — check PATH or typo" ;;
  *)   echo "Unknown exit code: $?" ;;
esac

The export button generates that block for you, with the specific code you looked up highlighted in the right place and the explanation preserved as a comment. Paste it directly into your script. No rewriting the syntax from scratch.

The case template includes the four most common exit codes (0, 1, 126, 127) plus the wildcard catch-all. Delete the ones you don't need.

→ bashsnippets.xyz/tools/bash-exit-code-lookup

3. Cron Job Builder — Next 5 Run Times

This is the one I wanted for myself.

Enter any cron expression — say, 0 3 * * 1-5 — and it now immediately shows the next five times it will fire:

Next run times for: 0 3 * * 1-5

1.  Mon Jun 09 2026  03:00:00
2.  Tue Jun 10 2026  03:00:00
3.  Wed Jun 11 2026  03:00:00
4.  Thu Jun 12 2026  03:00:00
5.  Fri Jun 13 2026  03:00:00

The problem this solves: cron expressions are easy to get subtly wrong. 0 3 * * * fires at 3am. But 3am what — your server's local time or UTC? And is your server set to UTC? And does */6 mean every 6 hours starting at midnight, or starting at the first minute it's defined? And does 0 9 * * 1 fire on Monday or Sunday, depending on which cron implementation counts week starts?

Without something that shows you the actual fire times, you add the job, wait until the next day, and find out whether your assumptions were right or wrong. If they were wrong, you change it and wait another day. It can take three or four days to confirm a cron expression fires when you want it to, purely because of iteration time.

Showing the next five run times eliminates that cycle entirely. You know immediately whether your expression does what you think it does. Change it, verify it, add it to your crontab — all in under a minute.

→ bashsnippets.xyz/tools/cron-job-builder

4. Chmod Calculator — World-Writable Warning and Live Symbolic Mirror

Two updates here bundled together because they're related.

World-writable warning:

Enabling any world-write bit (the w in the "others" column) now shows a warning banner:

⚠️ World-writable permissions mean any user on the system can modify this file or directory. This is rarely correct. Verify this is intentional.

This is the chmod 777 trap. I've seen chmod 777 applied as a "quick fix" for permission errors more times than I can count. It works immediately, which is why people do it. The fact that it means "literally every user and process on this system can write to this file" gets lost in the urgency of making the error go away.

The warning doesn't prevent you from setting world-writable permissions. It just makes sure you've seen the words "any user on the system can modify this" before you click confirm. If you intended that, the warning is just noise. If you didn't, it might save you.

Live symbolic ↔ octal mirror:

Every permission you configure now shows both forms simultaneously:

Octal:    chmod 755
Symbolic: chmod u=rwx,go=rx

Both update in real time as you toggle permissions. This is useful for two reasons: you see the octal code for when you need to type it quickly in a terminal, and you see the symbolic form for when you need to express the same permission in a script where the explicit form is easier to read and maintain. Neither is "more correct" — they're the same thing expressed two ways, and having both removes the mental step of converting between them.

→ bashsnippets.xyz/tools/chmod-permissions-builder

5. PATH Debugger — Duplicate Entry Detector

Duplicate entries in $PATH accumulate over years. Every time a tool's installer adds itself to your PATH, every time you add an entry to .bashrc, every time a package manager prepends its bin directory to your environment — the list grows. On machines that have been around for a while, or on developer laptops where tools get installed and removed and reinstalled, you can end up with /usr/local/bin listed four times.

Duplicate entries don't usually cause obvious breakage. The correct binary still runs. It just runs with slightly more PATH resolution overhead on every command, and the duplicate entries make it harder to reason about which version of a tool will actually be picked when you have multiple installed. If /usr/local/bin appears before /usr/bin, the local install wins. When it appears four times, you've lost track of the actual resolution order.

The debugger now flags repeated entries and generates a one-liner to deduplicate and export a clean PATH:

# Generated dedup command:
export PATH=$(echo "$PATH" | tr ':' '\n' | awk '!seen[$0]++' | tr '\n' ':' | sed 's/:$//')

That pipeline: converts PATH from colon-separated to newline-separated, uses awk to keep only the first occurrence of each entry, converts back to colon-separated, and strips the trailing colon. The resulting PATH has the same resolution order as the original but with all duplicates removed.

Add that line to the bottom of your .bashrc and your PATH will deduplicate itself on every new shell session.

→ bashsnippets.xyz/tools/path-debugger

6. ShellCheck Error Decoder — Inline Script Scanner

ShellCheck is the best static analysis tool for bash scripts. If you're writing bash and you're not running your scripts through ShellCheck, you're missing real bugs. I'll say that plainly.

The problem is that ShellCheck error codes (SC2086, SC2046, SC1091, etc.) are meaningful if you know what they mean and opaque if you don't. When ShellCheck tells you SC2086: Double quote to prevent globbing and word splitting, that makes sense. When it just gives you SC2086 in isolation, you have to look it up.

The inline script scanner adds a paste box where you put your bash script directly. The tool maps recognized SC error codes to the lines in your script where they appear — not in the abstract documentation, but in your specific code. You see the line, the SC code, and the explanation together.

Important caveat I want to be direct about: this is not a replacement for running actual ShellCheck. The real ShellCheck tool parses bash properly, handles edge cases, and catches issues that a pattern-matcher won't. If you're writing scripts that matter, install ShellCheck and run it directly:

# Install
apt install shellcheck       # Debian/Ubuntu
brew install shellcheck      # macOS

# Run
shellcheck yourscript.sh

What the decoder in BashSnippets tools is useful for: understanding what a specific SC code means in the context of your own code rather than in a reference page. It's a learning tool and a quick lookup, not a substitute for the real thing.

→ bashsnippets.xyz/tools/shellcheck-error-decoder

All tools are free, no login, no account required. The full tools index is at:

→ bashsnippets.xyz/tools