DEV Community: Anguishe

grep "error|failed" Found Nothing in a Log That Said 'backup failed'

Anguishe — Thu, 22 Oct 2026 13:00:00 +0000

I wrote a three-line backup log on my own machine: a start line, tar: /var/www/uploads: Cannot open: Permission denied, and backup failed (exit 2). Then I ran the check a lot of people keep as a weekly ritual, the one that looks like it asks "did anything go wrong?": grep "error|failed" backup.log.

GNU grep 3.12 printed nothing and exited 1. The last line of the log says failed in plain text.

grep -E "error|failed" backup.log printed that line and exited 0.

Nothing about the first command is malformed, so there is no error message. Exit 1 means "no lines matched", which is exactly what grep returns for a genuinely clean log. A monitoring check wired to that exit code reports all clear on a log full of failures, every single run, and the empty output does not start an investigation. It ends one.

The pipe that isn't a pipe

grep's default engine is Basic Regular Expressions, and in BRE the pipe is not alternation. It is a literal character. error|failed does not mean "error or failed"; it means the twelve-character string error|failed, pipe included, which no logger has ever written. The same goes for +, ? and parentheses: in BRE they are ordinary characters unless you backslash-escape them. Alternation the way every other regex engine behaves needs -E, Extended Regular Expressions — the mode egrep has been shorthand for since long before most of us started typing it.

# BRE (default): the pipe is a literal character — matches nothing, exits 1
grep "error|failed" backup.log

# ERE: the pipe means OR — the search you actually meant
grep -E "error|failed" backup.log

GNU grep also accepts grep "error\|failed" in BRE mode — on my machine that matched and exited 0 — but the escaped pipe is a GNU extension that POSIX does not define, so it is a habit that stops working on the systems that do not carry GNU grep. -E is the portable spelling.

The deeper trap is in the exit codes. grep exits 0 on a match, 1 on no match and 2 on an actual error; pointed at a file that did not exist, it printed No such file or directory and exited 2. An impossible pattern is not an error. "No matching lines" and "you searched for a string that cannot exist" produce identical output: none, with the same status. A false negative is the worst failure a search tool has, because nobody goes looking for a problem that grep has already said is not there.

The alias is a crutch

The tempting fix is alias grep='grep -E' in your shell rc file, and it does make the interactive ritual behave. It also does nothing anywhere that matters. Aliases apply only to interactive shells, so every cron job and script on the same box still runs BRE grep. The alias does not exist on any machine you have not configured, which is most of the machines you will ever SSH into. And it trains your fingers to type patterns that silently degrade the moment it is absent. The durable fix is knowing which engine a grep runs under and what its silence means before you trust it — and writing -E into the script itself.

So the builder shows the engine and the empty result

The grep Pattern Builder assembles the command live as you type: pattern, path, and an optional comma-separated list of file types that it turns into --include flags, so a log search does not wade through binaries. Toggles cover the flags that earn their keep — case-insensitive, recursive, line numbers, invert, count, filenames-only, whole-word, quiet and -B/-A context lines — and every combination produces a plain-English sentence saying what the command will do before you run it.

The engine picker sits front and centre: BRE, -E and -P, each with a note about what is literal where. The BRE note says it outright: + ? | ( ) are literal unless escaped. The -P note warns that Perl-compatible regex is GNU-only and fails on macOS and BSD grep, the portability cousin of the same trap: a \d pattern that works on a Debian server errors out on a Mac.

There are guardrails for mistakes people ship. It turns on -r when the path looks like a directory, because plain grep pointed at a directory refuses with "Is a directory", and in a script with stderr redirected that refusal is invisible. It drops -n when -c, -l or -q is active, because those flags change the output mode and line numbers stop meaning anything.

The panel that makes the ritual trustworthy is the live tester. Paste a few sample lines, including one real failure line from your actual log, and it highlights which lines the current pattern matches, with a count. In BRE mode it treats a bare pipe literally, the way grep does, so error|failed against a pasted backup failed line highlights nothing: zero out of one, a number that makes you look up. Switch to -E and the line lights. The preview runs on JavaScript's regex engine, which is close to ERE and PCRE for everyday patterns — treat it as a check on your pattern, not a replacement for running grep.

One unescaped character turned a failure search into a search for a string that cannot exist, and grep answered that question with perfect accuracy. The command was valid, ran clean, exited 1, and reported nothing wrong.

Build the command and read the explanation before you trust it: https://bashsnippets.xyz/tools/grep-pattern-builder

The Search Files for Text snippet covers the grep patterns worth keeping, Delete Old Log Files handles the other half of log hygiene, and the rest of the library is at https://bashsnippets.xyz

I Moved a Script to Another Directory on PATH. bash Said 'No such file or directory' and Exited 127.

Anguishe — Wed, 21 Oct 2026 13:00:00 +0000

One bash process, one small demo script called report, and a mv between two directories that are both on PATH. I set it up in a scratch tmpfs mounted over /opt inside unshare -rm, with PATH starting /opt/jobs/bin:/opt/jobs/newbin, and ran it as a script. Inside the namespace I appear as root, which is why ls says root root:

$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ ls -l /opt/jobs/newbin/report
-rwxrwxr-x 1 root root 42 Oct  3 18:26 /opt/jobs/newbin/report
$ report
session.sh: line 5: /opt/jobs/bin/report: No such file or directory
exit=127
$ type -a report
report is /opt/jobs/newbin/report
$ hash -t report
/opt/jobs/bin/report
$ command -v report
/opt/jobs/bin/report
$ hash -r
$ report
report 2.1: 14 jobs ok
exit=0

The file exists, it is executable, and its new directory is on PATH. bash ran the old path anyway, said No such file or directory, and exited 127, the same code a typo gets. I ran the same sequence through an interactive bash -i to be sure it was not a script quirk, and got bash: /opt/jobs/bin/report: No such file or directory, exit 127.

What makes it hard to trust your own eyes is the middle of that output. type -a names the new location. command -v, the check most scripts use to ask "is this installed", names the old one.

bash remembers where it found things

The first time bash runs a command through a PATH search, it stores the full path in a hash table, so the next call can skip the search. That is what hash -t report prints. When the file moves, bash does not search again; it executes the remembered path, the kernel answers ENOENT, and bash reports No such file or directory with the stale path in the message.

type -a searches PATH fresh every time, because its job is to list every match. command -v reports what bash would run, and bash would run the hashed path. So the two most common "where is it" checks disagree, and the one that agrees with the error is the one that looks wrong.

hash -r empties the table. A new terminal starts with an empty one. And there is a shell option made for exactly this:

$ shopt -s checkhash
$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ report
report 2.1: 14 jobs ok
exit=0
$ hash -t report
/opt/jobs/newbin/report

With checkhash set, bash checks that a hashed path still exists before running it, searches PATH again when it does not, and updates the table. In the wild this shows up after an upgrade moves a binary between /usr/local/bin and /usr/bin, or after swapping a pip-installed tool for the distro package, both of which the page describes.

Six causes, one exit code

command not found and its 127 cover more than a stale hash. The page reproduces the rest: a typo, a program installed outside PATH, a script in the current directory run without ./, an alias a script cannot see, a found file whose shebang names a missing interpreter (a CRLF line ending does that), and the cron job whose PATH is shorter than your terminal's.

The cron one is worth reproducing, because it never fails while you are watching. Below, a demo tool in a scratch directory on my interactive PATH, then the same call with cron's two-directory PATH, which env -i reproduces without waiting for the schedule. After that, the page's diagnosis script, why-command-not-found.sh, run in the namespace with a home directory of its own, shown as ~, and the same tool in ~/.local/bin:

$ nightly-sync
nightly-sync 0.4
exit=0
$ env -i PATH=/usr/bin:/bin bash -c 'nightly-sync'
bash: line 1: nightly-sync: command not found
exit=127

$ ./why-command-not-found.sh nightly-sync
✓ nightly-sync resolves to ~/.local/bin/nightly-sync
✗ cron's default PATH (/usr/bin:/bin) will NOT find it: use ~/.local/bin/nightly-sync in the crontab, or set PATH= at its top
  if your terminal still says 'not found' or 'No such file', its hash table is stale: run hash -r
exit=0
$ ./why-command-not-found.sh gti
✗ gti is not on PATH
  did you mean git? (/usr/bin/git)
exit=1

It uses type -P, which searches PATH for files only and ignores aliases and functions, so it sees what any other script would see. It repeats the lookup with PATH=/usr/bin:/bin to catch the cron case, reads the first line of the file for a carriage return, checks ./NAME, looks in the usual places installers put binaries without touching PATH (~/.local/bin, ~/.cargo/bin, /usr/sbin, /opt/*/bin), and tries every adjacent-letter swap of the name, which is how gti became git. It exits 1 for every cause it finds, so ./why-command-not-found.sh rsync || exit 1 can gate a deploy script.

The order that separates them

type -a NAME for everything bash can see. hash -t NAME when the message says No such file or directory about a path. echo "$PATH" | tr : '\n' when the answer is "nothing". And env -i PATH=/usr/bin:/bin bash -c NAME before anything goes in a crontab.

The stale hash is the one cause where everything you check says the tool is fine: ls and type -a both show it right where it should be. One hash -r, or one new terminal, is the whole fix.

All six causes reproduced, with the diagnosis script and what each check does: https://bashsnippets.xyz/snippets/bash-command-not-found

If 127 is all a log gives you, the Bash Exit Code Lookup explains it next to 126, and the Bash $PATH Debugger checks a pasted PATH for missing directories and duplicates. The rest of the library is at https://bashsnippets.xyz

I Pointed rsync --delete at a Nearly Empty Directory. It Deleted 3,800 Files and Exited 0.

Anguishe — Tue, 20 Oct 2026 13:00:00 +0000

The rsync accident that ends backups is not an error. It is rsync doing exactly what it was told, successfully, to the wrong directory. I wanted to see how quiet that is, so I rebuilt it in a scratch directory on my own machine with rsync 3.5.0.

The destination held 3,800 files, standing in for a nightly mirror. The source held one file, index.nginx-debian.html — what Debian's nginx package leaves behind in /var/www/html after you move a site's web root somewhere else and forget that the backup script still points at the old path. Then the mirror command anyone would have in that script: rsync -a --delete src/ dst/.

It exited 0. Afterwards the destination held one file: index.nginx-debian.html. Three thousand eight hundred files gone, in a run that any cron wrapper, healthcheck ping or ✓ backup complete log line would have recorded as a success. Run that nightly and every green checkmark in the dashboard is a record of the deletion.

Exit 0 doesn't mean what your monitoring thinks

The bug is not in rsync. It is in what we assume its exit code promises. rsync reserves non-zero codes for failures on its own terms: an unreachable host, a protocol error, an I/O failure, source files vanishing mid-transfer (code 24), a partial transfer (code 23). Making a destination match a near-empty source is none of those. It is the requested operation, delivered without incident. A monitor that checks the exit code answers "did rsync finish?" and most of us have labelled that answer "is the data safe?" Those are different questions.

rsync's sharp edges fail by succeeding

Once you see that shape, you find it all over the command. A trailing slash on the source means "copy the contents"; no trailing slash means "copy the directory itself", which nests a second directory inside your destination and, on a --delete mirror, changes what gets compared and therefore what gets removed. Both versions run clean. Drop -a and rsync stops preserving permissions, symlinks and timestamps, and because its quick check decides "already transferred" by size and modification time, a destination full of transfer-time mtimes never matches again, so every run quietly becomes a full re-copy that still exits 0. Hand a comma-separated list to a single --exclude and it is treated as one pattern with a comma in it, so it matches nothing and node_modules rides along every night. None of these produce an error. Each completes a different operation from the one in your head.

The flag that shows the damage first

The habit that prevents the whole class is previewing every destructive sync. Same scratch directories, same command, with --dry-run and -v:

$ rsync -a --delete --dry-run -v src/ dst/
sending incremental file list
deleting photo-999.jpg
deleting photo-998.jpg
deleting photo-997.jpg
…

Thousands of deleting lines, and afterwards the destination still held all 3,800 files. A wall of deletions in a preview you are reading is impossible to miss. The same wall executed in a log nobody opens is a green checkmark.

One detail the documentation does not shout: the -v is not optional. I ran --dry-run without it against a second scratch directory and it printed nothing at all and exited 0. A dry run you cannot see is not a preview. Use -v, or --itemize-changes if you want one line per file with a change code.

So the builder makes the preview the default

The Rsync Command Builder assembles the command in a live preview while you toggle the real options: archive mode, verbose output, compression, resume for unreliable links (--partial --progress), SSH transport, a bandwidth cap for syncs that share a link with people trying to use it, and exclude patterns. You type the excludes comma-separated and it emits one individually quoted --exclude flag per pattern, so the shell never expands your glob and the comma mistake cannot happen.

The opinions built into it are the point. Turn on --delete without dry-run and a red warning appears: those deletions are permanent, preview them first. Turn dry-run on and it switches to a green note telling you to remove --dry-run only after you have read the output. The Mirror preset ships with dry-run already enabled — mirroring is the one mode where you switch the safety off on purpose instead of remembering to switch it on. The whole thing runs in your browser; nothing you type leaves the page.

The scratch run took a fraction of a second to delete 3,800 files and report success. The preview took the same fraction of a second to list every one of them. The only difference was a flag.

Build the command with the guardrails on: https://bashsnippets.xyz/tools/rsync-command-builder

For the script around it — cron scheduling, SSH keys, a hardened nightly job — the Rsync Remote Backup snippet pairs with this tool, Automated File Backup covers the local variant, and the rest of the library is at https://bashsnippets.xyz

My Machine Has No mail Command. That's Where Most Disk Alerts Die.

Anguishe — Thu, 15 Oct 2026 13:00:00 +0000

I recently found that the disk check on my own Linux machine had been printing WARNING: Disk at 88% into nothing: cron's journal said No MTA installed, discarding output beside every run. So I checked what it would take for that warning to reach me by email. mail: not installed. mailx: not installed. sendmail: not installed. msmtp: not installed.

That is normal for a desktop install and common on minimal server images. It also means every monitoring script that ends in | mail -s "…" you@example.com is a script whose last line fails. Run it by hand and you would at least see command not found. Run it from cron and the error goes the same way as the output: cron tries to mail it to you, has no mail agent to do it with, and discards it. Two layers of silence, and the check itself is working perfectly.

Detection and delivery are different jobs

The detection logic is the part everybody polishes: read the number, compare it, decide. Telling a person is a separate job, and a monitor that only appends to a log file or prints to stdout has not done it — it has relocated the problem. Somebody still has to poll that output, and that somebody is a person with better things to do at 2am. The fix is to push the news to the place you already check compulsively, which for most of us is an inbox.

The core is small enough to memorize:

#!/bin/bash
set -euo pipefail

USAGE=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -gt 80 ]; then
  printf 'Disk on %s hit %s%% at %s\n' "$(hostname)" "$USAGE" "$(date '+%F %T')" \
    | mail -s "[ALERT] disk space on $(hostname)" you@example.com
fi

Read the number, compare it, and when it is over the line, pipe a message into mail with a subject that names the machine — because once the same check runs on several hosts, an alert that does not say which server is on fire is a puzzle, not an alert.

The alert you "sent" that never arrived

Installing mail fixes the first layer and exposes the next one. mail exiting 0 does not mean your alert reached anyone. It means the message was handed to whatever mail transfer agent lives on the box; delivery is the MTA's problem, and the exit code you can observe stops at the handoff. On a server with no MTA configured for the outside world, that handoff often drops the message into a local mailbox under /var/mail — a file on the same machine that no human reads. The script reports success, and the alert sits a few directories away from the log it was supposed to replace.

Cron adds one more trap on top. Jobs run with a minimal PATH, so a mail that resolves in your login shell can be command not found under cron. The test that proves anything is not running the script by hand and receiving an email. It is letting cron run it and receiving an email, then reading the MTA's log to see what it actually did.

Real delivery means giving the box a real path out. On Debian or Ubuntu, mailutils pulls in an MTA, and the "Internet Site" option covers a machine that is allowed to send directly. Plenty are not — outbound mail from cloud IP ranges gets filtered or spam-binned constantly — so relay through a provider instead: msmtp pointed at a Gmail account with an app password is a few lines of config, and curl can speak SMTP to a relay on boxes where you would rather not install an MTA at all. Either turns "handed to the MTA" into "accepted by a server whose whole job is delivering".

The opposite failure: the alert that cries wolf

Once delivery works you meet the inverse problem. A threshold does not trip once; it stays tripped. Disk at 81 % at nine o'clock is still 81 % at ten, and an hourly cron job will mail you the same fact twenty-four times a day until you fix the disk or stop reading, which comes first. An alert channel you have learned to skim is a silent failure with extra steps.

The guard is a dedupe sentinel: on send, touch a marker file; before sending, check the marker's age with find "$MARKER" -mmin -1440. I ran that logic three times on my machine. The first call sent. The second, a moment later, was suppressed. Then I backdated the marker 25 hours with touch -d and the third call sent again. One email per incident per day; the condition is still measured every hour, only the nagging is suppressed.

Delivery is the whole difference

The check on my machine had the right threshold and the right schedule-shaped intentions. What it lacked was any route from the machine to me, and nothing in its own output could say so, because its output had nowhere to go either. Before you trust an alert, send one on purpose, from cron, and watch it arrive.

The full script — the alert body with the top disk consumers included, the msmtp config for a Gmail relay, the once-per-24-hours dedupe guard, and the cron traps that make alerts vanish — is here: https://bashsnippets.xyz/snippets/bash-send-email-alert

An email alert is the last guard an unattended job needs, not the first: the Cron Wrapper Generator wires alert-on-failure in alongside the lock, timeout and retry it generates, and Bash Scripts That Survive Cron is the full playbook for jobs nobody watches. The rest of the library is at https://bashsnippets.xyz

Bash Said '-eq: unary operator expected'. The Script Still Exited 0.

Anguishe — Wed, 14 Oct 2026 13:00:00 +0000

Four lines of bash, one empty variable, two tests of it. I wrote it on purpose, in a scratch directory, to capture the error for a guide. The output is not what I expected to capture:

$ cat queue-check.sh
#!/bin/bash
count=""
if [ $count -eq 0 ]; then echo "queue empty"; fi
if [[ $count -eq 0 ]]; then echo "queue empty ([[ ]])"; fi

$ bash queue-check.sh
queue-check.sh: line 3: [: -eq: unary operator expected
queue empty ([[ ]])
exit=0

The first test printed an error. The second printed an answer, and the answer is wrong: count is not zero, it is empty. Then the script exited 0, so anything watching the exit code, a cron wrapper or a CI step, saw a clean run.

That is three separate lies from one bug, and only one of them is on stderr. The part I keep coming back to: if I had only written the [[ ]] line, the way most modern bash guides tell you to, there would have been no message at all. I would have trusted "queue empty".

The message names the check, not the cause

unary operator expected points at -eq. The bug is the variable before it. $count is unquoted, and an empty unquoted variable is removed by word splitting before [ ever runs. So [ received -eq 0 instead of "" -eq 0, tried to read -eq as a one-argument test like -f, and gave up with exit status 2. Inside an if, status 2 counts as "not true", so the branch was skipped and the script carried on. [[ ]] does no word splitting, so it saw an empty operand, and in an arithmetic comparison an empty string evaluates to 0. True.

The fix is to quote it, [ "$count" -eq 0 ], which turns the error into an honest integer expected, or to give it a default with "${count:-0}" when empty really does mean zero.

That pattern, a message that names the check that failed rather than the reason, holds for almost every bash error I reproduced for the guide. Here is the syntax-error version, from a script missing one semicolon:

$ bash missing-then.sh
missing-then.sh: line 4: syntax error near unexpected token `fi'
missing-then.sh: line 4: `fi'
exit=2

$ shellcheck missing-then.sh

In missing-then.sh line 2:
if [ -f /etc/hostname ] then
                        ^-- SC1010 (warning): Use semicolon or linefeed before 'then' (or quote to make it literal).

bash blames line 4. The mistake is on line 2, where then without a ; before it became one more argument to [. bash parses a whole if ... fi block before running it, and it reports the first token that cannot fit, which is the fi with no then open. The line number is where bash gave up, not where you went wrong. ShellCheck pointed at the right line on the first try.

Permission denied, after chmod +x

The one that wastes the most time is Permission denied when the execute bit is already set. I reproduced it on a tmpfs mounted noexec, inside a throwaway mount namespace so nothing real was touched:

$ chmod +x backup.sh
$ ./backup.sh
bash: line 1: ./backup.sh: Permission denied
exit=126
$ findmnt -no OPTIONS --target backup.sh
rw,noexec,relatime,uid=1000,gid=1000,inode64

chmod succeeded, the mode says executable, and the kernel still refused, because the filesystem forbids executing anything on it, root included. Exit 126 means "found it, could not run it". The other causes chmod cannot reach are a parent directory missing its x bit and a file owned by someone else, and namei -l on the full path shows both in one listing.

The rest of the nine

The guide covers nine messages, each reproduced on my machine with the output pasted as it came out. command not found has six causes behind one exit code, 127, and type -a plus a look at cron's two-directory PATH separates them. bad interpreter: No such file or directory complains about a file that plainly exists, because the shebang ends in a Windows carriage return. No space left on device can fire on a filesystem df -h reports as 0% used, when the inodes ran out instead of the blocks. Connection refused can come from a service that is listening, on 127.0.0.1 only. And Argument list too long means rm never started, so nothing was deleted.

The order I read any error in now

The message first, word for word, with the variable parts removed. Then the exit code: 127 not found, 126 found but not executable, 2 usually a syntax or usage error, 128+N killed by signal N. Then bash -x, which prints each command after expansion, so you see the [ -eq 0 ] bash actually ran instead of the [ $count -eq 0 ] you wrote. Then ShellCheck, which flags the unquoted $count as SC2086 before the script runs at all.

Then validate the value before you test it. I re-ran the same check under set -euo pipefail, and strict mode did nothing: count="" is set, so set -u stays quiet, and set -e ignores a failing command inside an if condition by design. The script printed the same error and then still running. What stopped it was one line that says what the variable must look like, [[ $count =~ ^[0-9]+$ ]] || { echo "count is not a number: '$count'" >&2; exit 1; }, which exited 1 and printed count is not a number: ''. A quoted variable makes the error honest. A validated one makes the script stop.

Full guide with all nine messages, the one check that tells each one's causes apart, and links to the fix for every case: https://bashsnippets.xyz/guides/bash-error-messages

If an exit code is all you have, the Bash Exit Code Lookup decodes it, and the ShellCheck Error Decoder explains the common SC codes ShellCheck prints. The rest of the library is at https://bashsnippets.xyz

grep Found 5 Matches for a Hostname I Expected 3 Times. sed Would Have Rewritten All 5.

Anguishe — Tue, 13 Oct 2026 13:00:00 +0000

To see how a routine rename goes wrong, I built a small config tree on my own machine. conf.d/app.conf points an upstream at api.internal:8080. conf.d/health.conf has a health-check URL on the same host. samples/example.conf is a sample shipped as documentation and should never change. conf.d/docs.conf sets doc_host = apixinternal.example.com, a different host that happens to look similar. And cache.bin is a binary file that contains the string. The job: move the live config from api.internal to api-v2.internal.

The reflex is one line: sed -i "s/api.internal/api-v2.internal/g" $(grep -rl "api.internal" myapp). It exited 0. Afterwards the two live files were correct — and docs.conf now read doc_host = api-v2.internal.example.com, the sample had been rewritten, and the binary cache had been edited in place. Five files touched, two intended, and nothing in the output to say so.

Two mechanisms, one bad afternoon

The shallow one is regex. In api.internal that dot is not punctuation; it is a wildcard matching any single character, in grep and sed alike. So the pattern does not mean "this hostname", it means "this substring, with anything in the middle, wherever it appears", which is how apixinternal matched. When you mean a literal dot, escape it (api\.internal); when you mean a literal string, grep -F says so outright.

The deep one is order. sed -i writes over the original file the instant the pattern matches — no preview, no confirmation, no undo unless you supply one. Text processing on a live tree is a pipeline with a sequence: find decides which files may be touched, grep shows what will actually change, sed or awk makes the change, and a diff earns your trust afterwards. The one-liner started at step three and pointed it at everything grep could see.

grep: count the matches before you rewrite them

The search that should come first is the same one the one-liner hid inside $( ). Run on its own, grep -rn "api.internal" myapp printed five hits: the two live lines, the sample, the apixinternal line in docs.conf, and binary file matches for the cache. I expected three. That mismatch is the entire value of the step: the extra two lines are the damage, caught at read time instead of write time.

With the dot escaped and the search limited to config files — grep -rn 'api\.internal' myapp --include="*.conf" — the count dropped to three: the two live lines and the sample. Closer, and still one file too many.

find: print the blast radius

Every bulk edit begins with a file list, and the default list, "everything under this directory, including whatever I forgot lives there", is the dangerous one. find myapp -type f -name "*.conf" -not -path "*/samples/*" printed three candidates: app.conf, docs.conf and health.conf. The -not -path is the single flag that keeps documentation out of a rename. Read that output like a checklist; anything destructive should only ever receive filenames that appeared on it.

sed: transform with an undo you did not have to build

-i.bak edits in place and leaves a .bak copy of every original: a full rollback for the price of four characters. It is also the portable spelling. GNU sed treats the backup suffix as optional, but BSD and macOS sed require one, so a bare sed -i 's/old/new/' that works on Linux swallows your expression as the suffix on a Mac.

Composed, on a fresh copy of the same tree:

# Scope with find, confirm with grep, transform with a per-file undo
find myapp -type f -name "*.conf" -not -path "*/samples/*" \
  -exec grep -l 'api\.internal' {} + \
  | xargs sed -i.bak 's/api\.internal/api-v2.internal/g'

It edited app.conf and health.conf and left a .bak beside each. docs.conf was a candidate from find but grep -l with the escaped dot never selected it, so apixinternal.example.com survived. The sample was never a candidate. The binary cache was never a candidate. Two files changed, the two I meant. (If the tree can hold filenames with spaces, grep -lZ into xargs -0 closes the word-splitting gap.) Then diff one file against its .bak before believing the run.

awk: when the question is a column, not a line

grep and sed think in lines. The moment your question is about a field — total bytes served from column 10 of an access log, requests grouped by status code — that is awk, which splits every line into $1 through $NF and runs a small program per line. Its END block fires once after the last line, which is how a million-line log becomes a single number. Chain all four stages and you get the command worth keeping for incidents: find scopes to today's logs, grep pulls the error lines, awk strips the timestamp fields so identical errors collapse together, and sort | uniq -c | sort -rn | head ranks them.

The list worth printing

The unscoped one-liner did nothing wrong by its own rules: pattern found, substitution made, exit 0, five times. The difference between five files and two was an explicit file list, an escaped dot, and one grep read before the write. Scope, confirm, transform, verify. In that order this work is boring, and boring is the goal.

The full guide, with each stage's flags and the incident-triage pipeline in copy-paste form: https://bashsnippets.xyz/guides/bash-text-processing

If you would rather assemble the flags than memorize them, the find command builder and grep pattern builder construct and explain the exact invocation before you run it. The rest of the library is at https://bashsnippets.xyz

find -mtime +30 Kept a File That Was 30.5 Days Old. I Checked With touch -d.

Anguishe — Thu, 08 Oct 2026 13:00:00 +0000

Nothing on my Linux machine deletes anything by age unless I tell it to. ~/.cache alone holds 47 GB right now — 13 GB of pip downloads, 11 GB of uv, 5 GB of browser cache — each one a program doing exactly its job, none of them with a retention policy. Application logs have the same shape: a writer that never stops and no reader. That is how a disk goes from fine to full without a single thing malfunctioning.

The standard fix is an age-based cleanup with find. It is also a command that deletes files, so before trusting it with anything real, I tested its three sharp edges on files I backdated with touch -d.

Why a full disk lies to you

The reason prevention beats diagnosis here is what a full disk looks like when it arrives. The moment the filesystem runs out of blocks, every process that tries to write gets ENOSPC back from the kernel, and almost no application reports that as "the disk is full". You get a failed save in one program, a crashed build in another and a database complaint in a third, all worded differently, while the process that filled the disk goes quiet because it cannot write either. By the time symptoms appear they point everywhere except the cause.

Aging files out with find

The tool has shipped with every Linux and macOS box for decades:

LOG_DIR="/var/log/myapp"
DAYS=30

# Preview pass — read this list before letting anything delete
find "$LOG_DIR" -type f -name "*.log" -mtime +"$DAYS" -print

Run that, read the output, and when the list holds exactly what you expect, swap -print for -delete and run it again. The three runs below are why the preview is not optional.

Edge one: -mtime +30 does not mean 30 days

I created four logs dated 29 days, exactly 30 days, 30 and a half days, and 31 days back. find . -name "*.log" -mtime +30 -print printed one of them: the 31-day file. The 30.5-day file survived.

find measures age in whole 24-hour periods and throws the fraction away, so 30.5 days old counts as 30, and the + means strictly greater than. -mtime +29 printed three files: 30, 30.5 and 31 days. If your policy says "keep 30 days", +29 is the flag that enforces it. The off-by-one is invisible in a quick test and surfaces a month later as "why is that file still there".

Edge two: -delete goes last, always

On a copy of the same directory I put the predicates in the wrong order: find . -delete -name "*.log". Afterwards the directory had zero entries. Every log, the non-log keep.txt, everything.

find evaluates its expression strictly left to right, and -delete is an action that runs and returns true. Put it first and it executes on every file find walks, before -name ever gets a vote. There is no warning and no prompt. The fix is order: tests first, -delete at the very end.

Edge three: never pipe filenames to rm

The reflex is find … | xargs rm. I made four files — app v2.log, app, v2.log and keep.txt — and ran find . -name "app v2.log" | xargs rm. When it finished, app and v2.log were gone. app v2.log, the one file the command was aimed at, was still there.

xargs split the filename on the space and handed rm two arguments, both of which happened to exist. That is the worst possible outcome: the target survives, two innocent files die, and rm reports nothing unusual. -delete never leaves find — no pipe, no word splitting — so it cannot do that. If you genuinely need another command, -print0 | xargs -0 keeps each name in one piece.

The blast-radius controls

-type f keeps directories and symlinks out of the match. And the path should name one application's log directory — /var/log/nginx, /var/log/myapp — never /var/log wholesale, which holds logs the OS still needs and files that must never be deleted by age. Retention for a database transaction log is a backup problem, not a find problem.

Then take yourself out of the loop

The version that actually prevents a full disk is the one that runs without you: the path and the age limit in named variables, a crontab line running it weekly, and the cleanup's own output appended to a log with a timestamp, so every run leaves a record. Whatever grows without bound gets aged out on a schedule, long before it matters.

Three runs on backdated files took less time than reading this post. Each one would have been an incident on a real directory.

Full script with the multi-directory loop, the .gz variant for rotated logs, a before/after disk-usage report and ready-made cron lines: https://bashsnippets.xyz/snippets/delete-old-log-files

Deleting old logs is the recovery; hearing about a filling disk before it hits 100 % is the upgrade — the disk space warning script does that, and when the space thief is not a log at all, find large files names it. The rest of the library is at https://bashsnippets.xyz

My Disk Check Prints WARNING at 88%. Cron Throws It Away Every Monday.

Anguishe — Tue, 06 Oct 2026 13:00:00 +0000

There is a small disk check in my home directory that I wrote months ago. It reads / with df, strips the percent sign, and prints WARNING: Disk at N% once usage passes 80. A crontab line runs it on Mondays. Today I ran it by hand for the first time in a while, and it answered WARNING: Disk at 88%.

Then I went looking for where that warning goes when cron runs it, and the answer was nowhere. The log file sitting next to the script was last written on May 2, and its newest line says OK: Disk at 30% — those were manual runs, because the crontab line never redirected anything into it. Cron's own journal filled in the rest. Last Monday it ran the script 54 times between 05:00 and 05:59, because the schedule reads * 5 * * 1, every minute of hour five, where I meant once. And beside those runs, the same journal says No MTA installed, discarding output. Every warning that check has produced under cron went into a mail system this machine does not have.

Nothing about that is exotic. It is the most common way a monitoring script fails: it runs, it detects the condition, and the channel between it and a person does not exist.

A full disk never introduces itself

The reason that channel matters is what the failure looks like when it arrives. When a filesystem hits 100 %, nothing says "the disk is full." Every process that needs a write fails in its own vocabulary: the database complains about its own files, the app dies persisting a session, and every error points at the component that surfaced it rather than the cause underneath. One root cause, several convincing disguises.

Logging is a write too, so the logs stop at the moment the disk fills, and the recorded evidence ends right before the incident begins. And ext4 holds back about 5 % of blocks for root, so a root shell keeps working on a disk where every non-root service is starving. The machine feels healthy to the person debugging it.

There is a second trap on the way back down, and this box taught me that one too. On August 28 I deleted a 44.6 GB video file here. du dropped by 44 GB; df moved by 5. XFCE's thumbnailer daemon, tumblerd, had the file open, and the kernel does not release blocks while any process still holds a descriptor on the deleted inode. pkill -x tumblerd gave the space back instantly. Judge a cleanup by df, never by du: du counts names that are gone, df counts blocks that are actually free.

The check, done so a person sees it

# 80% is a calendar item; 100% is an outage
df -h | awk 'NR>1 && $5+0 >= 80 {print $0}'

NR>1 skips df's header row. $5+0 is the part worth stealing: the Use% column is a string like 82%, and awk's numeric coercion reads the leading digits and drops the rest, so adding zero turns it into 82 with no cut or sed. It prints only the filesystems over the line, which means its output is the alert.

That is also where mine broke, so check the last hop before you trust any of it. | mail -s only works on a machine with a working mail agent; test it once by hand. If there is no mail agent, send the output to a webhook or at least to a log line with a timestamp. And read the schedule field twice: 0 5 * * 1 is once a week, * 5 * * 1 is sixty times.

That one-liner is the first of the 25 scripts in the guide this post is drawn from, and the shape repeats across all of them: a small check, run on cron, that turns a silent slow failure into a loud early one.

The other 24 aim the same idea at different failures

The guide groups them into the five ways servers actually go down: disk growth, backup gaps, service crashes, network problems and security drift.

The disk section continues from the warning. du -ah | sort -rh | head -20 names what grew, and a find-based pruner clears the predictable offenders. One flag in it deserves a label: -mtime +30 does not mean "30 days or older" — the + means strictly more than 30 twenty-four-hour periods, and misreading it changes what gets deleted.

The backup section checks the thing a timestamp cannot. A dump that has been failing quietly since a credential rotation is not a backup, it is confidence with a date on it. The guide's mysqldump script checks that the output file is non-empty before it finishes, so a broken backup fails loudly on night one instead of on restore day.

The service section caps undetected downtime: systemctl is-active --quiet nginx || systemctl restart nginx on a five-minute cron means a crash costs five minutes. On one condition — log every restart. A watchdog that does not log will re-heal a memory leak every few hours and hide it by suppressing its only symptom.

The SSL check reads the certificate the server is actually presenting over the wire, not the file on disk, which catches the renewed-but-never-reloaded case. And ss -tlnp compared against a known baseline turns "unexpected listener on a public port" from a post-incident discovery into a line item.

The closing section wires everything into /etc/cron.d with staggered minutes and one shared log, and that log turns out to be the real product: after a week you know your normal. A deviation from a known baseline takes seconds to spot. My own disk check had a threshold and a schedule and no way to reach anyone, which is the gap the cron section exists to close.

All 25 scripts, grouped by failure category, each with its core command and a link to the hardened version with thresholds, alerting and cron lines: https://bashsnippets.xyz/guides/bash-scripts-every-sysadmin-needs

Once they are scheduled, the jobs themselves have to survive running unattended. Bash Scripts That Survive Cron covers the locks, timeouts and retries, and the Cron Wrapper Generator wraps those guards around any command you paste in. The rest of the library is at https://bashsnippets.xyz

sed 's/port/listen_port/g' Changed Four Lines. I Wanted One.

Anguishe — Thu, 01 Oct 2026 13:00:00 +0000

Renaming a config key is the job sed exists for, so I gave it the smallest honest test I could build on my own machine: a four-line YAML file and the rename most people type first. The file held port: 8080, support_email: ops@example.com, transport: tcp and report_dir: /var/reports. The command was sed -i 's/port/listen_port/g' app.yml.

GNU sed 4.9 returned immediately, exit 0, no output. -i edits in place and prints nothing, so there was nothing to read until I opened the file:

listen_port: 8080
suplisten_port_email: ops@example.com
translisten_port: tcp
relisten_port_dir: /var/relisten_ports

One line was meant to change. All four did. The email key is now a word no program will look up, the transport key is gone, and report_dir points at /var/relisten_ports, a directory that does not exist. Nothing failed. sed did what the command said, and the command said something different from what I meant. Across a service tree of forty config files that is not four bad lines; it is every file that happens to contain the letters, and you find out from whatever reads them next.

sed matches characters, not words

s/port/listen_port/g means: wherever the four characters p-o-r-t appear, replace them. Inside support. Inside transport. Inside report, and twice on the report_dir line because reports carries them again. sed has no idea what a word is unless the pattern tells it.

Three defaults stack into the failure, and none of them is a sed bug. An unanchored pattern that matches more than you pictured. -i, so there is no output to review. And a glob like *.yml or a find | xargs that reaches more files than you have opened. Each one is a default you have to refuse on purpose.

Fix one: tell sed where the word ends

GNU sed understands \b as a word boundary, a zero-width match between a word character and a non-word character. On the same original file, sed -i 's/\bport\b/listen_port/g' app.yml changed exactly one line, listen_port: 8080, and left the other three alone. In support, the p of port sits right after another word character, so \b refuses to match there.

Two details matter before you trust it. Underscore counts as a word character, so \bport\b will not touch the port inside db_port either — usually what you want, occasionally not. And \b is a GNU extension: BSD sed on macOS documents [[:<:]] and [[:>:]] for word start and end instead. A script that has to run on both platforms should not assume either spelling.

Fix two: read the diff before anything is written

The habit that matters more than any regex is refusing to let the first run be the in-place one. Drop -i and pipe sed's output into diff against the original: sed 's/port/listen_port/g' app.yml | diff app.yml -. On my test file that printed all four lines as changed, suplisten_port_email included, while the file on disk stayed intact. The whole disaster, on screen, costing nothing.

diff exits 1 when it finds differences, so the check works in a script as well as at a prompt. For more than one file, the shape that holds up is three steps. Build the file list with grep -rl first, so sed only ever touches files that contain the pattern and everything else keeps its modification time — build caches and rsync care about that. Show a diff per file. Then require an explicit --apply before a single byte is written. The default fails toward reading, not writing. If you forget the flag, the worst outcome is a diff you did not want to see.

Three more places the same command bites

The replacement side has its own special character. An unescaped & in the replacement means "the entire match", so s/CEO/Smith & Sons/ turns CEO into Smith CEO Sons. Write \& when you mean an ampersand.

The delimiter is yours to choose. s|/var/www/html|/srv/www|g is the same command as the version with every slash escaped, and it is the one you can still read under pressure. Pick a character that appears in neither side.

When the pattern comes from a shell variable, it is still a regex. OLD=staging.internal matches stagingXinternal too, because the dot means any character, and a | or & inside the value breaks the command outright. For values you control, a delimiter the value cannot contain is enough. For arbitrary input — filenames, anything a user typed — escape the variable first or use a tool that does literal replacement.

And the macOS trap that eventually costs every mixed team an afternoon: GNU -i takes an optional suffix, BSD -i requires one. On a Mac, sed -i 's/old/new/g' file treats the script as the backup suffix and then tries to run the filename as the program. The BSD spelling is sed -i '' 's/old/new/g' file.

Where sed stops

sed works one line at a time, so a plain s/// never matches across a newline. For multi-line surgery, awk or perl -0pe are the honest tools. And if the file is JSON or YAML with real structure, stop doing regex on it; a field-aware tool such as jq parses instead of pattern-matching, which is the entire reason it exists.

The rename I wanted needed two characters of anchoring and one pipe to diff. The version without them exited 0 and rewrote four lines out of four.

The full bulk-replace script — grep -rl file list, per-file diff, --apply gate, GNU/BSD -i detection — with the line-by-line breakdown and the FAQ on delimiters and multi-line matches: https://bashsnippets.xyz/snippets/bash-sed-find-replace

For the discovery half of the job, searching files with grep covers finding the text before you replace it, and the Grep Pattern Builder builds anchored patterns interactively. The rest of the library is at https://bashsnippets.xyz

SC2086 Is ShellCheck's Lowest-Severity Warning. It Let rm Delete Two Files I Never Named.

Anguishe — Tue, 29 Sep 2026 13:00:00 +0000

ShellCheck files SC2086 at info, the lowest severity it shows by default, in the same tier as remarks about echo flags. It fires on every unquoted $var, so on a script that has never been linted it is often most of the output, and it is the rule people reach for first when they start a .shellcheckrc. It reads like a style nag. I wanted to know what treating it like one actually costs, so I built the smallest case I could on my own machine: three files and a four-line script.

The directory held quarterly, report.txt and quarterly report.txt. The script set report="quarterly report.txt" and ran rm $report, with set -euo pipefail at the top.

It deleted quarterly. It deleted report.txt. It left quarterly report.txt, the one file it was asked to remove, exactly where it was. And it exited 0, because as far as rm could tell it had been handed two names and both existed. Strict mode had nothing to catch; nothing failed. In a real cleanup job that is a wrong-file deletion with a green exit code, and the first sign is the next job looking for a file that is no longer there.

What bash does before rm ever runs

The mechanism is that an unquoted $report is not handed to rm as one thing. Bash expands the variable, splits the result on whitespace, expands any * or ? it finds against the current directory, and only then builds the argument list. rm $report with a space in the value has become rm quarterly report.txt — two arguments — by the time rm wakes up. Double quotes switch both steps off: rm "$report" is one argument, no globbing, done.

Then the lint run on the same four lines, ShellCheck 0.11.0 and bash 5.3.9:

$ shellcheck before.sh

In before.sh line 4:
rm $report
   ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
rm "$report"

Info. The same severity ShellCheck gives a remark about echo flags. There is no severity level for "deletes files you did not name and reports success," so this is where it lives, and that label is the reason it is the first rule people exclude.

Where the same bug hides

The glob half is quieter. pattern="*.log" followed by echo Looking for $pattern prints the names of the log files in the directory instead of the pattern, because the star was expanded before echo saw it. Hand that unquoted pattern to find -name and it breaks the same way: one matching file and find searches for that literal name, two and it dies with paths must precede expression. ShellCheck flags it as SC2086 too; its separate code, SC2061, is for a glob typed straight into the command, find . -name *.log.

Inside [ ] the failure changes shape. [ is a command, so [ $name = "root" ] with an empty $name gives test two arguments instead of three. It complains about a unary operator on stderr, the condition evaluates false, and the script keeps walking. The branch you were guarding is skipped without an exit. Inside [[ ]] bash does not split at all, which is why ShellCheck stays quiet there and why "switch to double brackets" is a legitimate fix rather than a dodge.

And for f in $files — the one place where splitting is usually on purpose — ShellCheck 0.11.0 does not flag at all. It assumes you meant it. The loop still breaks on the first element containing a space. What you need is a list that can hold spaces, which in bash means an array, and an unquoted array expansion gets a different, louder code: SC2068, at error severity.

The honest case for turning it off

Here is the nuance a blanket disable throws away. There are two real situations where the split is the point: a flag string read from a config file — opts="-r -n" then grep $opts pattern . — and arguments passed through a thin wrapper. ShellCheck cannot know the split is intentional, so it warns. You have two honest exits. Make it an array, opts=(-r -n) and grep "${opts[@]}" pattern ., which ShellCheck accepts without comment. Or keep the string and put # shellcheck disable=SC2086 on the line above, with the reason after a second #, so the next reader knows it was a decision and not a leftover.

That directive covers one command. The repo-wide .shellcheckrc covers every command you will ever write in that repo, including the rm. That is the difference between disabling a check and disabling your own judgement.

There is also the opposite move, which most people who disable the rule never find: shellcheck --include=SC2086 script.sh reports this rule and nothing else, and still exits 1 when it finds one. That is how you enforce quoting as a CI gate on a repo that is still working through its other forty findings — the exact situation that makes people reach for disable= in the first place.

What it costs to filter by severity

Quoted, rm "$report" on the same directory deletes one file, the right one. ShellCheck's lowest severity holds the failure worth caring about most, and filtering by severity is exactly how it gets thrown away.

The full write-up, with every run pasted from ShellCheck 0.11.0 — the [ ] case, the for-loop that lints clean and still breaks, and the one-line, one-file and one-project ways to disable it: https://bashsnippets.xyz/shellcheck/sc2086

If the code on your screen is not SC2086, the ShellCheck Error Decoder explains any SC code with a before/after fix, SC2046 is the same trap for $(command) output, and the rest of the library is at https://bashsnippets.xyz

I Killed My Report Script Mid-Write. The Output File Never Noticed.

Anguishe — Thu, 24 Sep 2026 13:00:00 +0000

The failure this pattern prevents is a quiet one. A script writes its output straight to the path a consumer reads, dies halfway, and the consumer loads the torn file as though it were complete. Nobody sees an error, because the only process that failed is gone, and everything downstream of it exited 0 on partial data. Meanwhile /tmp collects the working files of every run that never reached its cleanup line at the bottom.

I did not want to trust the fix on paper, so I tested it the unfriendly way on my own machine, bash 5.3.9. A small generator writes a CSV, with a sleep 30 planted between its first and second row to stand in for a slow API. I started it, waited one second, and sent it SIGTERM from outside.

This is the shape it runs in:

#!/bin/bash
set -euo pipefail
FINAL_PATH="${1:?usage: report.sh /path/to/output.csv}"

TMP_FILE=$(mktemp)             # unique path, created 0600
cleanup() {
  local code=$?                # first line: capture before anything overwrites it
  rm -f "$TMP_FILE"
  exit "$code"
}
trap cleanup EXIT              # registered on the line after mktemp

generate_rows > "$TMP_FILE"    # every write goes to the temp path
mv "$TMP_FILE" "$FINAL_PATH"   # atomic on one filesystem

What the kill left behind: exit status 143, which is 128 plus signal 15. The temp path mktemp had handed out came back No such file or directory, so the trap ran on the way down. And out.csv still held the previous run's three complete lines. A consumer reading it at that moment gets yesterday's data, whole. Stale is something monitoring can see. Torn is not.

Every way out is covered

Walk the exits. The generator fails halfway: set -e aborts, the EXIT trap removes the partial temp file, the published file is untouched. A signal arrives — Ctrl-C, a shutdown's SIGTERM, my kill — and the same trap runs, as the run above shows. The script succeeds: mv has already moved the temp file, so rm -f finds nothing, complains about nothing, and the status passes through.

EXIT is the reason this is four lines instead of twenty. It is a bash pseudo-signal that fires on normal completion, an explicit exit 1, a set -e abort, and after the handling of a real signal. Trap INT or TERM separately only when you want signal-specific behaviour, such as logging who killed you.

Two scoping rules each save a confused hour. A trap set inside $( ) or a ( ) group belongs to that subshell and fires when the subshell exits, so cleanup traps go at the top level. And a second trap … EXIT replaces the first — traps do not stack — so when two resources need releasing, a temp file and a background ssh -L tunnel for instance, both go in one handler.

The ordering that launders a failure

The comment on the local code=$? line is the part people move. I ran four versions of the handler against a script that ends in exit 3, and bash turned out more forgiving than the folklore:

A trap that removes the file and never calls exit: the script exits 3. Bash keeps the original status.
A trap ending in a bare exit: 3. Inside an EXIT trap, a bare exit reuses the status the script was leaving with.
Capture first, then rm -f, then exit "$code": 3.
rm -f first, then local code=$?, then exit "$code": 0.

The one version that reports success after a failure is the one that looks most careful. It captures $? — after rm has succeeded, so it captures rm's zero and exits with it. Cron sees success, alerting sees success, and the cleanup has removed the evidence too. Capture on the first line, clean second, exit last.

Why mktemp and not /tmp/myscript.$$

A predictable temp path is two bugs. Overlapping runs, retried jobs and helpers can collide on the same name and interleave their writes, which produces exactly the garbage this pattern is meant to prevent. And /tmp is world-writable, so a guessable name invites something else on the box to plant a file or a symlink where you are about to write. mktemp returns a unique path, created atomically, readable only by you. When a script needs several working files, make one directory with mktemp -d and remove it whole in the same handler; every intermediate file inherits the cleanup.

What trap cannot do, and why the design survives it

kill -9 cannot be caught. No trap runs, and power loss is no different. That is not an argument against the trap; it is why the rest of the design looks the way it does. Orphans from an uncatchable death land in /tmp, where reboot or tmpfiles ageing clears them, rather than in a data directory. And mv means even an uncatchable death cannot publish a torn file, because being half-written and being at the published path are never true at the same moment.

One cheap gate is worth adding wherever a consumer trusts your output: before the mv, check the temp file's size with wc -c against a floor, and exit non-zero instead of publishing an implausibly small result. An upstream that "succeeded" with an empty body then pages a human instead of reaching the dashboard.

A cleanup path you have never executed is a cleanup path you are guessing about. Plant a sleep, kill it from another terminal, and check three things: /tmp is clean, the output still holds the old complete file, and the exit code is not zero. It takes about a minute.

The complete script, the FAQ on subshell scoping and signal lists, and the line-by-line breakdown: https://bashsnippets.xyz/snippets/bash-trap-cleanup

The same survival kit continues with bash error handling, so failures stop the script and the trap has something honest to report, the timeout command, so a hang dies and cleans up instead of blocking forever, and the Cron Wrapper Generator, which assembles the lot around any command. The rest of the library is at https://bashsnippets.xyz

set -euo pipefail Is Missing a Letter. My ERR Trap Stayed Silent Until I Added -E.

Anguishe — Tue, 22 Sep 2026 13:00:00 +0000

Here is a nine-line script that does everything the strict-mode articles tell you to. set -euo pipefail on line 2. On line 3, an ERR trap whose entire job is to print the failing command and its line number: trap 'echo "✗ failed: ${BASH_COMMAND} (line ${LINENO})" >&2' ERR. Then a helper function that runs mkdir /proc/export, a directory that cannot be created, standing in for a full disk or a missing mount.

I ran it on my machine, bash 5.3.9. It printed start, then mkdir's own complaint, then exited 1. The trap said nothing. No failing command, no line number — the two facts it was registered to report. From a nightly job, that is the log you get: a start banner, one stderr line if you are lucky, and an exit code that tells you something broke but not what.

Then I changed one character, set -Eeuo pipefail, and ran it again. Same mkdir error, followed by ✗ failed: mkdir /proc/export (line 5), and exit 1. The trap had been fine all along. Nobody had told it to follow the script into the function, and a function is where most real failures happen.

Three flags, three promises, one missing

set -euo pipefail gets pasted as a single incantation. It is three flags making three separate promises, and each has its own list of places where it quietly declines to keep them.

-e says exit when a command fails and nobody is checking. Bash defines "checking" generously: the condition of an if, anything left of && or ||, anything under ! — and, the part that surprises people, everything inside a function that was called from one of those positions. A careful helper becomes unguarded the moment somebody wraps it in if helper; then, and nothing warns you.

-u says an unset variable is an error rather than an empty string — the difference between rm -rf "$BUILD_DIR/" and rm -rf / when the variable is spelled BUILDDIR in one place.

-o pipefail says a pipeline fails if any stage fails, not only the last one. Without it, curl … | jq … proceeds happily when curl dies and jq parses the empty result into something harmless.

None of those is the letter I was missing.

Why the trap said nothing

trap … ERR fires on the same conditions that make -e exit, so it looks like the natural place to print a diagnostic. But by default the ERR trap is not inherited by functions, command substitutions, or subshells. My failing mkdir was one level down, inside a function. Errexit did its job and killed the script. The trap, registered in the top-level shell, never saw a failure that happened in the function's scope. Exit 1, and silence — which is precisely the log entry that tells you nothing at 03:00.

The fix is -E, errtrace, which makes the ERR trap follow you into functions and subshells. The handler I run now names the command, not only the line, because bash keeps the pieces in BASH_COMMAND and BASH_LINENO:

#!/bin/bash
set -Eeuo pipefail
CROSS="✗"

on_err() {
  local code=$?     # first line, before anything else overwrites it
  echo "$CROSS failed: '${BASH_COMMAND}' (line ${BASH_LINENO[0]}, exit ${code})" >&2
}
trap on_err ERR

With that at the top, a failure anywhere in the script logs the command, the line and the exit status instead of nothing. -E is the flag missing from nearly every strict-mode line on the internet. If you take one thing from this, take the E.

The other silent spot: local

The second silent spot is a line most of us have written dozens of times: local out=$(some_command). On the same box, a function running local out=$(false) under set -euo pipefail carried on to its next line and the script exited 0. That line never triggers errexit, and no trap fires for it, because local is a command in its own right and its exit status — success, it declared the variable — is the one bash sees. The command substitution's failure is discarded before anyone looks. Declare on one line, assign on the next, and the failure is yours again. It is two lines instead of one everywhere you capture output into a local, and it is the bug behind ShellCheck's masked-return-value warning, which is not pedantry.

One caveat so you do not chase a ghost: with -E set, a failure inside an explicit ( subshell ) fires the trap twice — once in the subshell, once in the parent as the non-zero status propagates. That is expected. If duplicate alerts matter, guard on a flag or move the work out of the subshell.

Cleanup on every path, and when to leave it all off

The ERR trap tells you what broke; the EXIT trap is what stops the script leaving a half-written file behind for the next stage to load as if it were complete. It runs on every termination, and the rule that makes or breaks it is the same one as above: capture $? on the first line of the handler. Move local code=$? below the rm -f "$TMP_FILE" and it reports rm's status — zero, forever — and the script exits clean after a failure. That is how a script with strict mode at the top ends up reporting success: I ran that ordering against an exit 3 on bash 5.3.9 and the script exited 0.

Strict mode is also not the right default everywhere. A health checker that runs twenty probes and expects some to fail will spend more || true than logic fighting -e. .bashrc should never set it, because one failed command would close your terminal. And grep -q used as a test returns non-zero as data, not as an error. The guide covers where to leave it off, rather than pretending the flags are free.

What the log says now

The template I start from now opens with set -Eeuo pipefail, both traps registered before any work happens, $? captured first in each handler, locals declared and assigned on separate lines. When it fails, the log says which line and which command. That is the whole difference between a script that exits 1 and a script that tells you why.

The full template with every behaviour checked against bash 5.3 — where errexit goes silent, the nounset defaults for variables that are only unbound in staging, and the assembled skeleton: https://bashsnippets.xyz/guides/safe-bash-script-template

If you would rather not type it, the Bash Boilerplate Generator emits this shape with your script name filled in, trap cleanup on exit works the mktemp-and-atomic-mv half in full, and the rest of the library is at https://bashsnippets.xyz