DEV Community: Anguishe

Restart=always Did Nothing. The Service Never Exited.

Anguishe — Thu, 12 Nov 2026 14:00:00 +0000

On August 6 my laptop booted to a blank terminal. The display manager, lightdm, has Restart=always in its unit, and systemctl status lightdm said active (running). As far as systemd was concerned, nothing was down.

What had happened was a race. Xorg entered the NVIDIA driver's init routine about 140 ms before the /dev/nvidia* device nodes existed, because another GPU service, also Restart=always and started one second earlier, was competing for the card. Xorg never came back. Fourteen hours later it was still there in state S, one thread, 20 ms of CPU used in total, its log file cut off mid-sentence with no error line.

I had trusted that restart policy without reading what it promises. Restart=always restarts a process after it exits. Mine never exited. A cron job checking systemctl is-active lightdm every minute would have said "fine" all night. sudo systemctl restart lightdm brought the desktop back in three seconds, and the real fix turned out to be ordering, nvidia-persistenced so the device nodes exist before anything asks for them, not a restart policy at all.

"Down" is three different states

The mistake is treating "down" as one condition. It is at least three, and they look identical from a chair across the room. A process that crashed or was killed shows failed, and a restart is correct. A service someone stopped on purpose shows inactive with Result=success, and a restart fights the human doing maintenance. A process that is running but not answering shows active (running), and no restart ever happens, because nobody asked. My August 6 was the third kind, and the standard "restart it if it's down" watchdog only handles the first.

Read the state before you touch it

Most watchdogs start from systemctl is-active. I re-ran it on the same laptop today, systemd 261, with a deliberate typo in the third name:

$ for u in cron lightdm crno; do
>   printf '%-8s is-active=%-9s rc=' "$u" "$(systemctl is-active "$u")"
>   systemctl is-active --quiet "$u"; echo $?
> done
cron     is-active=active    rc=0
lightdm  is-active=active    rc=0
crno     is-active=inactive  rc=4

Exit code 0 is running. Exit code 3 is a real unit that is not running. Exit code 4 is a unit that does not exist, and is-active prints the same word, inactive, for 3 and 4. A watchdog written as if ! systemctl is-active --quiet "$SERVICE" treats the typo as an outage, tries to start crno every minute, fails every minute, and alerts every minute until someone reads the message closely enough to spot the swapped letters.

The command a script should use is systemctl show, because each field comes back as one clean line with no colour and no localisation:

$ systemctl show lightdm -p Restart,RestartUSec,NRestarts,ActiveState,SubState,Result,StartLimitBurst,StartLimitIntervalUSec
Restart=always
ActiveState=active
SubState=running
StartLimitIntervalUSec=10s
StartLimitBurst=5
Restart=always
RestartUSec=100ms
Result=success
NRestarts=0

That is my lightdm today: Restart=always, 100 ms between attempts, zero restarts since boot. LoadState separates the typo from the outage. Result tells you why a unit is in its state, and NRestarts climbing means systemd is already doing the job and your watchdog is late.

The last two lines are the start-limit trap. A unit started more than five times inside ten seconds, which is what a short RestartSec does to a daemon that crashes on startup, goes to failed with Result=start-limit-hit, and from then on systemctl start refuses until someone runs systemctl reset-failed. A watchdog that does not know this reports "restart failed" forever. One that resets it unconditionally every minute has switched off systemd's crash-loop protection and replaced it with its own crash loop, with an email each cycle. The guide's version resets a bounded number of times per outage, then stops and tells a human.

When it is running but not answering

That leaves the case I actually had. No state check sees it, because the state is correct: the process exists and is running. The only way to know a service answers is to ask it the way a client would, with a timeout on the question, for example curl -fsS --max-time 5 http://127.0.0.1/ >/dev/null for a web server, or ss -Hltn 'sport = :5432' | grep -q . for anything that should be listening on a port. A probe without a timeout is one more hung process on the box, and the next cron tick starts another.

When the probe fails on an active unit, systemctl restart is the treatment, with two costs. Restart is stop plus start, so a process that ignores SIGTERM holds it for TimeoutStopSec, 90 seconds by default. And the restart destroys the evidence of what the process was stuck on. A hung process is the only witness to what it was waiting on, so capture its state and its wchan before anything is killed.

Fourteen hours of active (running) and a restart policy that never fired: the policy was working as documented. I had asked it to cover a failure it cannot see.

Full guide, with the three-state table, the Restart= values and when each fires, the start-limit trap, and a cron watchdog that probes, honours a maintenance flag and alerts once per outage: https://bashsnippets.xyz/guides/auto-restart-linux-service

The watchdog script on its own is Service Watchdog, and what to capture before the restart wipes it out is in Diagnosing a Hung Process. The rest of the library is at https://bashsnippets.xyz

ss Showed 0.0.0.0:3000 and No Process Name. It Was a Docker Container on Every Interface.

Anguishe — Wed, 11 Nov 2026 14:00:00 +0000

The dev server for my site came up on port 3001 for months. Next.js wants 3000, finds it taken, moves up one, and says nothing worth reading. When I finally asked who had 3000, this is what ss told me as a normal user. It is the opening run of the open-ports guide on the site, published 2026-09-01:

$ ss -ltnp 'sport = :3000'
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      4096         0.0.0.0:3000       0.0.0.0:*
LISTEN 0      4096            [::]:3000          [::]:*

Two sockets, every IPv4 interface and every IPv6 interface, and an empty Process column. One more flag, ss -ltnpe, printed the socket's cgroup, cgroup:/system.slice/docker.service, and one docker ps later the owner had a name: a docker-proxy publishing an Open WebUI container's port 8080 as host port 3000. It had been there since spring.

The container is not the part that bothers me. I had read output like that before, seen no name, and moved on. Both answers were on screen. The address column said who could reach it, and the empty column said I was not allowed to see the owner. I read neither.

ss prints facts, not verdicts

The Local Address column is the reach. 127.0.0.1 and [::1] are this machine only. 0.0.0.0 is every IPv4 address the host has, LAN, VPN and Docker bridge included, and [::] is the IPv6 twin. An empty Process column without root does not mean "no process". It means the socket belongs to another user, and ss only names your own. The -e flag prints the owning cgroup instead, which is the systemd unit, and it needs no root.

A published container port has one more twist, and this part is documented Docker behaviour rather than something I tested here: Docker routes published ports through its own iptables chains before the host's ufw rules see the packet, so a ufw deny 3000 does not cover them. Docker's documentation covers it under Docker and ufw. The guide's fix was to publish on loopback, -p 127.0.0.1:3000:8080.

None of that is hard once you know it. Holding all of it in your head while reading a long ss listing is where it goes wrong, so I built a page that does the reading.

What the explainer does with a paste

The Open Ports Explainer takes the output of ss -tulpn, ss -ltnpe, netstat -tulpn or lsof -i -P -n and works out which one it is on its own. It parses in the browser, never connects to a port, and has Redact addresses ticked by default, so LAN and public addresses are masked on screen and in the share link. A Load example button fills it with a real ss -tulpn listing if you want to see it work first.

Every listener becomes a card: the address, the reach scope in words, the owner (from the Process column, or the systemd unit when you ran ss -e without root), what usually lives on that port, flags ranked by severity, and the next command to run. A share link carries only the normalised rows in the URL hash, with PIDs dropped, and Copy baseline CSV writes the listing in the shape the ports-audit script diffs against.

Tonight I ran the guide's two lines back through the explainer's own parser, the same TypeScript the page runs, with a short Node wrapper that prints each card as text:

0.0.0.0:3000  +  [::]:3000
  All interfaces, IPv4 and IPv6 · owner not shown
  dev server / Grafana. Node, Next.js or Rails dev server by default; Grafana and Open WebUI also use 3000.
  WARN A development server reachable from your network. Dev servers are not built to face other machines.
  INFO No owner shown. Run ss -ltnpe to see the systemd unit, which also tells you whether this is a container.
  NOTE Two sockets, one per address family. Either can close without the other.
  next: ss -ltnp 'sport = :3000'

That INFO line is the step I skipped for months. Given the -e line instead, the owner column became docker.service, a second WARN appeared ("A published container port. Docker's own iptables rules route this traffic to the container before ufw's rules see it"), and the next command changed to docker ps --format '{{.Names}}\t{{.Ports}}'.

A fresh run, nothing real listening

To check the ranking I made three listeners inside a private network namespace (unshare -rn, no sudo, gone when the shell exits): Python's http.server on 0.0.0.0:3000, a bare socket on 0.0.0.0:5432 standing in for a database, and one on 127.0.0.1:8099. As root inside the namespace, ss named every owner. Below it, the explainer's summary line and the first of its three cards:

$ ss -tulpn
Netid State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
tcp   LISTEN 0      128        127.0.0.1:8099      0.0.0.0:*    users:(("python3",pid=612058,fd=3))
tcp   LISTEN 0      5            0.0.0.0:3000      0.0.0.0:*    users:(("python3",pid=612055,fd=3))
tcp   LISTEN 0      128          0.0.0.0:5432      0.0.0.0:*    users:(("python3",pid=612056,fd=3))

3 listeners (3 sockets) · 2 reachable from the network · 2 flagged

0.0.0.0:5432
  All IPv4 interfaces · python3
  PostgreSQL. PostgreSQL server, often a local dev database container.
  CRITICAL A database or cache is listening on every interface. Anyone on your network can try to log in to it.
  next: ss -ltnpe 'sport = :5432'

ss printed the database row last. The explainer put it first, because a database on a wildcard address outranks a dev server, and the loopback socket got a green OK. It names a port by its usual tenant, so a Python socket on 5432 reads as PostgreSQL. That is the right guess for a scan, and the reason every card ends with a command that checks the owner.

What it cannot see is your firewall, NAT or cloud security groups, and the page says so under the input box. It tells you who could reach a socket from where it is bound; whether a packet gets there is a question for nc from another machine.

The months of 3001 would have been one paste. The card said "owner not shown" and handed me the command that names the owner.

Paste your own ss -tulpn into the Open Ports Explainer: https://bashsnippets.xyz/tools/open-ports-explainer

For the commands behind each verdict, including the -e owner trick and what docker-proxy hides, the open ports guide pairs with it, and the ports-audit script turns the baseline CSV into a nightly alert. The rest of the library is at https://bashsnippets.xyz

I Deleted a 44.6 GB File and df Gave Me Back 5 GB. Something Still Had It Open.

Anguishe — Tue, 10 Nov 2026 14:00:00 +0000

On August 28 my laptop's disk sat at 88 %, and I went looking for something big to delete. I found it quickly: a 44.6 GB video intermediate, a rotate-to-vertical export I could rebuild from its source at any time. I deleted it. du on that folder dropped by 44 GB. df on the filesystem moved by 5.

For a few minutes I doubted the filesystem. The real answer was a process. XFCE's thumbnailer daemon, tumblerd, had opened the file to generate a preview and never closed it. pkill -x tumblerd gave the space back instantly, and the daemon respawns on its own the next time something needs a thumbnail, so killing it cost nothing.

That gap between du and df is the whole lesson, and it is the one people hit on servers with log files.

Why deleting a file can free nothing

rm does not delete data. It removes a directory entry that points at an inode. The blocks go back to the filesystem only when the inode's link count reaches zero and no process still has the file open. While one descriptor is open, the file is invisible to ls and du and still fully allocated as far as df is concerned. du counts names that are gone; df counts blocks that are actually free. When they disagree, the difference is deleted-but-open files.

lsof +L1 lists them: open files with a link count below one. Without root it shows only your own processes; on my machine today it shows nothing more alarming than a few of PipeWire's memory-backed buffers. On a server, run it as root and the culprit usually has a name, a PID and a size.

There is a second reason df can refuse to move: hard links. Removing one name of a file that has another name frees nothing, because the link count is still above zero. stat -c %h file prints the count before you rely on the delete.

And on a server, when the file is a log that a live service is still writing, rm is the wrong verb entirely. Truncate it in place with : > access.log: the inode and every open descriptor survive, the length drops to zero, and the space comes back without restarting anything.

Finding what ate the disk

df tells you which filesystem is full. It cannot tell you why, because it reads totals from the filesystem's own bookkeeping. The question "what filled it" is two commands:

# The 20 largest files and directories, biggest first
du -ah / --exclude=/proc --exclude=/sys --exclude=/dev 2>/dev/null \
  | sort -rh | head -n 20

# Individual files over 500 MB, wherever they hide
find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

Run against my home directory without root, the first one answered in under two minutes: 645 G in total, 430 G of it in one projects folder, and the top of the list was two video projects' raw-footage folders. No mystery left, only decisions about what to keep.

The one character that reorders the list

sort -rh is doing more work than it looks. I fed the same four du-style lines to both variants. sort -rn returned 850K, 500M, 13G, 10G: it reads the leading digits and stops, so 850 beats 500 beats 13, and the smallest item is ranked first. sort -rh returned 13G, 10G, 500M, 850K, because -h understands the size suffixes du prints. One letter separates a ranking from a shuffle, and the wrong one fails quietly on exactly the list you are reading under pressure.

Why the excludes are not optional

ls -l /proc/kcore on my laptop reports a file of 140,737,471,590,400 bytes, which is 128 TiB on a machine with an 868 GB disk. It is not a file; it is the kernel's address space presented as one, and /proc and /sys are full of synthesized entries like it. Any size scan that walks them either wastes minutes or reports nonsense, and the find above adds -xdev so it stays on one filesystem and never crosses into them or into network mounts.

The order that works under pressure is df -h to find the full filesystem, the ranked du to find what grew, find -size for the single monster file, and lsof +L1 the moment df and du disagree. My 44.6 GB would have taken ten seconds with that last step instead of ten minutes of doubting the filesystem.

Full script with the exclude patterns, thresholds, and the recent-growth variation: https://bashsnippets.xyz/snippets/find-large-files-linux

Getting the space back is the recovery half. Never seeing 100 % again is the Disk Space Warning snippet's job — it alerts at a threshold you pick instead of letting the disk do the alerting — and Delete Old Log Files keeps the logs from regrowing behind you. The rest of the library is at https://bashsnippets.xyz

I Killed the Parent Process. Its Child Kept the Port.

Anguishe — Thu, 05 Nov 2026 14:00:00 +0000

EADDRINUSE after you are sure you stopped the server has a common cause, and I wanted to watch it happen rather than describe it. On my own machine I started a wrapper the way tools like npm run dev do — a shell that launches the real server as a child and waits on it: bash -c 'python3 -m http.server 8098 & wait'.

Two processes appeared: the parent bash and the child python3. lsof -ti :8098 returned one PID, and it was the child's. I killed the parent, the process I had started and the one most people would find first in ps.

The port stayed taken. lsof -ti :8098 still returned the same PID, now an orphan that had been reparented and was listening as if nothing had happened. Killing the thing you launched does not free a port when the thing you launched is not the thing holding it.

Ask the socket table, not the process table

The mistake is asking the process table a question that belongs to the socket table. Process names are ambiguous: on a developer machine, ps aux | grep node returns a crowd of editor helpers, language servers and the grep itself, none of them labelled with a port. The socket table has exactly one answer per listening port, because a bound port belongs to one process until it lets go.

lsof -ti :PORT asks that directly. -i filters by internet address and -t prints only the PID, because it exists to be piped into something. The first thing to pipe it into is not kill:

PID=$(lsof -ti :8098 | head -n 1)
ps -p "$PID" -o comm=                        # look at what it is before you kill it
kill "$PID"                                  # SIGTERM — a request, not an execution
sleep 5
kill -0 "$PID" 2>/dev/null && kill -9 "$PID" # still alive? force it

On my run, ps -p named the orphan python3, and the plain kill was enough: one second later kill -0 found nothing, the port was free, and SIGKILL was never needed.

The ps -p line is there because discovery answers "which PID", not "safe to kill". On a laptop it is usually your own crashed dev server. On a shared box, the process on 5432 might be a database someone is migrating. Two seconds of reading the name is cheap.

Why SIGTERM comes first

Plain kill sends SIGTERM, which the process may catch: flush its buffers, finish in-flight requests, remove its pidfile, close the socket properly. kill -9 sends SIGKILL, which the process never sees; the kernel removes it mid-write if that is where it happened to be. SIGTERM, a pause, and SIGKILL only if something is still alive is the order that loses nothing in the common case and still works in the stubborn one.

When lsof is not installed — minimal containers and small VPS images often lack it — ss -ltnp gives the same answer from iproute2, which modern distributions ship everywhere. Without root it names only your own processes; its -e flag still shows the owning systemd unit for anyone's socket, which is how I once found a forgotten docker-proxy squatting on port 3000 of my own laptop.

When the port is still busy after the kill

Sometimes you kill the right process, lsof comes back empty, and the bind still fails. Nothing is holding the port; the kernel is. A closed TCP connection lingers in TIME_WAIT for up to a minute to absorb packets still in flight for the old conversation — on my machine at that moment, ss -Htan state time-wait | wc -l counted two. There is no PID to kill. Servers that set SO_REUSEADDR on their listening socket can rebind straight away; for those that do not, the fix is to wait a minute, not to reach for a bigger hammer.

The whole failure comes down to asking the right table. The parent is what you remember starting; the socket knows who is actually there.

Full script with the port validation, the ss fallback, and the SIGTERM→SIGKILL escalation loop: https://bashsnippets.xyz/snippets/kill-process-on-port

When the thing that needs to die has a name instead of a port number, Kill a Process covers the pgrep/pkill side, and once the port is free, Check If Website Is Up confirms whatever replaced it is actually answering. The rest of the library is at https://bashsnippets.xyz

One ssh Flag Tells You Whether a Server Still Accepts Passwords

Anguishe — Tue, 03 Nov 2026 14:00:00 +0000

Setting up SSH keys produces a row of green lights. ssh-keygen prints a fingerprint, ssh-copy-id reports Number of key(s) added: 1 and exits 0, and the next login skips the password prompt. None of those answers the question the setup was for: does this server still accept passwords?

There is a one-flag way to ask. ssh -o PreferredAuthentications=none user@host offers no credentials at all, so the server has to refuse, and its refusal lists every method it would have accepted. I pointed it at a Mac on my home network that has Remote Login switched on, from my Kali laptop. The answer was Permission denied (publickey,password,keyboard-interactive). Passwords are open there. The same kind of question put to GitHub's SSH endpoint comes back with publickey and nothing else. No login attempt, no guessing, two seconds.

That probe answers what the setup evening's green lights never could, because those lights were measuring something else.

What the exit 0 actually promised

ssh-copy-id succeeding means precisely this: it authenticated to the server — using your password, note — appended your public key to ~/.ssh/authorized_keys, and exited. Its exit code is a claim about the contents of a file, not about how future logins behave. The passwordless login that follows proves "some authentication method succeeded", nothing sharper. Key auth working and password auth being closed are two independent switches. Flipping the first tells you nothing about the second, and OpenSSH ships with the second on: on this laptop, /etc/ssh/sshd_config still has #PasswordAuthentication yes commented out, which means the default, which means yes.

SSH hides its refusals from the side you are watching

Authentication is a negotiation. The server offers an ordered list of methods, the client walks down it, and when one method is refused the client moves quietly to the next. The refusal is recorded once, in the server's auth log, which is the one place nobody looks during setup.

I have watched that negotiation go wrong on my own machine with no useful message at all. ssh from this Kali laptop to that same Mac printed Permission denied three times, then Too many authentication failures. The real cause appeared nowhere in the output: KDE's password helper, ksshaskpass, had intercepted the prompt, failed to render it, and handed ssh an empty password each time. On top of that, the client offered every key it had first, and each offer spent one of the server's authentication attempts before the password was ever tried. SSH_ASKPASS_REQUIRE=never and -o PubkeyAuthentication=no got through. The point is not the KDE bug. It is that from the client side, a broken method and a skipped method look the same.

Keys get refused for reasons with no client-side symptom at all. The classic one is permissions: sshd's StrictModes checks the remote ~/.ssh, authorized_keys and your home directory, and if any of them is group-writable it ignores authorized_keys entirely. Your login still works through password fallback, so everything looks fine until the day you disable passwords and lock yourself out.

That leaves two invisible states that look identical from your terminal: key auth broken but masked by password fallback, and key auth fine with the password door wide open. In both, you are logged in.

The core, plus the one test that does not lie

ssh-keygen -t ed25519 -C "$(whoami)@$(hostname)-$(date +%Y%m%d)" -f ~/.ssh/id_ed25519 -N ""
chmod 700 ~/.ssh && chmod 600 ~/.ssh/id_ed25519
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server
ssh -o PasswordAuthentication=no user@server   # proves the KEY works — not "something worked"

-N "" gives the key an empty passphrase so generation runs without a prompt; leave it out of an automated run and the script sits waiting for input. The chmod lines are not ceremony: SSH refuses a private key other users can read, with a message people reliably misread as a passphrase problem.

The last line is the one that matters. Forcing PasswordAuthentication=no for a single connection removes the fallback, so success can only mean the key authenticated. If that command prompts or fails, your key is being refused — usually by server-side permissions — and you find out while the other way in still exists.

Then the actual finish line, on the server: PasswordAuthentication no in /etc/ssh/sshd_config, reload sshd, and run the PreferredAuthentications=none probe again. password should be gone from the list. Do it with your current session held open and test from a second terminal, so a silently refused key costs you a fix and not a locked door.

Why the setup is a script

Because doing it from memory fails in small ways: the wrong key type one time, the wrong output path another, -N forgotten so the run hangs. The full version on the page refuses to overwrite an existing key pair, sets directory and key permissions on every run, prints the public half ready to paste into a cloud control panel, and runs ssh-copy-id for you when you give it a target, with a loop variant for pushing one key to several servers.

Every public IPv4 address gets probed for SSH passwords around the clock. The green lights from key setup do not tell you whether that matters to your server. One refusal message does.

Full script with the existing-key guard, permission handling and the multi-server loop: https://bashsnippets.xyz/snippets/ssh-key-setup-script

If permissions are the part that has bitten you, file permissions and security covers the modes SSH insists on, list open ports shows what else your box is answering on, and the rest of the library is at https://bashsnippets.xyz

find -perm 777 Missed a World-Writable File in My Test Tree. -perm -o+w Didn't.

Anguishe — Thu, 29 Oct 2026 13:00:00 +0000

Most homegrown permission audits start with the search everybody remembers: find /var/www -type f -perm 777. I built a three-file scratch tree on my own machine to see what that search actually covers. index.html at a normal 644, w666.txt at 666, and w777.sh at 777 — the classic leftover from a chmod 777 thrown at an error to make it go away.

find . -type f -perm 777 found one file: w777.sh. It did not find w666.txt, which any account on the machine can write to.

find . -type f -perm -o+w found both.

That one-character difference is the whole audit. Without a leading dash, -perm matches files whose mode is exactly the one you typed and nothing else; a world-writable file that is not also executable sits at 666 and is invisible to it. With the dash, -perm switches to mask semantics: -perm -o+w, or -perm -002, matches any file with the other-write bit set, whatever the remaining bits say. Reach for the exact form when you are hunting your own 777 leftovers specifically. Reach for the mask when the question is "what can other people write to", because that question does not care about the other eight bits.

Why the other-write bit is the one that matters

A Unix permission check sorts every process into owner, group or other, and a world-writable file hands write access to the last bucket: every account on the machine, including the ones you forgot exist. On a web server that includes the account the server itself runs as. A compromised plugin executing as www-data needs no root and no stolen password — a world-writable file in the web root is one it can rewrite at will, whether that means defacing a page or planting a backdoor in a cache directory. There is no exploit chain in that story. There is an open write bit and something hostile enough to use it.

The fix that locks you out

The tempting cleanup after an audit like that is one recursive command: chmod -R 644 across the whole tree. I ran it on a copy of the scratch tree.

chmod complained before it finished — cannot access 'site2/index.html': Permission denied, once per file — because it stripped the directory's execute bit first and then could not get back in to reach the files inside. Afterwards ls site2 still listed all three names, and cat site2/index.html failed with Permission denied. The files exist, have sane modes, and cannot be reached.

On a directory, the execute bit does not mean "run this". It means "you may enter and traverse this". Take it away recursively and every directory becomes a wall; on a web server that shows up as a wave of 403s for content that is sitting right there on disk. Files and directories need different modes, which means two passes:

find /var/www -type f -perm -o+w 2>/dev/null    # any world-writable file
find /var/www -type f -exec chmod 644 {} \;     # files: rw-r--r--
find /var/www -type d -exec chmod 755 {} \;     # dirs keep x, or nobody gets in

The first line is the audit, the second and third are the repair. Files go to 644, directories to 755, and nothing is walled off.

Keys go further, and ssh enforces it

Secrets go down to 600, and for SSH private keys that is not advice — the tools enforce it. I generated a throwaway ed25519 key in the scratch directory, set it to 644, and asked ssh-keygen -y to read it. It printed the WARNING: UNPROTECTED PRIVATE KEY FILE! banner, Permissions 0644 for 'testkey' are too open, then This private key will be ignored, and exited 255. At 600 the same command loaded the key without comment.

The ssh client makes the same check when it loads a key during a login. It ignores the key and carries on with whatever other methods it has, so what you see depends on what else was available: the banner followed by a password prompt, or the banner followed by Permission denied. Either way the answer is in ls -l, four characters wide, and the fix is chmod 600.

Let the scan remember

None of these three runs involved a malfunction. The exact-mode search did exactly what it says, chmod did exactly what it was told, and ssh refusing a readable private key is the permission model working. What fails is memory: which 777 did I throw at which error, and did I ever close it? A scan with the mask form of -perm answers that in under a second, and a saved report makes the answer comparable from one week to the next.

The full audit script — with the saved report file, the SUID scan for a deeper security pass, and the recommended-modes reference for files, directories, keys and private dirs — is at https://bashsnippets.xyz/snippets/file-permissions-security

If your next move after fixing a key is generating one properly, the SSH key setup script covers that end to end, and when you need a mode you do not have memorized, the Chmod Permissions Builder turns checkboxes into the octal value and the exact chmod command. The rest of the library is at https://bashsnippets.xyz

I Deleted a 48M Log to Free Space. The Next Write Still Hit 'No space left on device'.

Anguishe — Wed, 28 Oct 2026 13:00:00 +0000

A 64M filesystem, a worker that appends to a log and keeps it open the way a daemon does, and the reflex everyone has when a disk fills up: delete the log. I built it on a tmpfs mounted on /srv inside unshare -rm, a private mount namespace with no sudo, where I show up as root. The worker is a bash process that writes 48M to worker.log through file descriptor 3 and then turns into sleep, still holding the descriptor:

$ df -h /srv
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            64M   48M   16M  75% /srv
$ du -sh /srv/app
48M /srv/app
$ rm /srv/app/worker.log
$ du -sh /srv/app
0   /srv/app
$ df -h /srv
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            64M   48M   16M  75% /srv
$ head -c 32M /dev/zero > /srv/app/export.bin
head: error writing 'standard output': No space left on device
exit=1
$ du -sh /srv/app
16M /srv/app
$ df -h /srv
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            64M   64M     0 100% /srv

du went to zero the moment the log was gone. df did not move. Then a 32M write failed at 16M, and the filesystem reported itself 100% full while du could only find the 16M half-file from the write that failed.

Every check you would normally run after a cleanup says the cleanup worked. ls shows no log. du shows almost nothing. Only df, the one number that matters to the next write, disagrees, and it does not say why.

rm removes a name, not the data

A file is an inode, and a directory entry is a name pointing at it. rm removes the name. The kernel frees the blocks only when the inode has no names left and no process has it open. The worker still had descriptor 3 open, so 48M stayed allocated to a file nobody can see. du walks names, so it stopped counting the moment the name went. df asks the filesystem how many blocks are in use, so it kept counting.

lsof +L1 lists open files with a link count below one, which is the definition of deleted-but-open:

$ lsof -w +L1 /srv
COMMAND    PID USER FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep   608657 root 3w   REG   0,78 50331648     0    3 /srv/app/worker.log (deleted)
exit=0
$ ls -l /proc/$PID/fd | grep deleted
l-wx------ 1 root root 64 Oct  3 18:18 3 -> /srv/app/worker.log (deleted)
$ : > /proc/$PID/fd/3
$ df -h /srv
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            64M   16M   48M  25% /srv
$ ps -o pid,stat,cmd -p $PID
    PID STAT CMD
 608657 S    sleep 600

NLINK 0, (deleted), 50,331,648 bytes (48M exactly), held open for writing (3w) by PID 608657. /proc/PID/fd/3 is a live handle on the deleted file, so : > /proc/$PID/fd/3 truncates it in place. df dropped by 48M and the process kept running with its descriptor open, now pointing at an empty file. For a service, restarting it is the cleaner fix; truncation is for when you cannot, and it throws away whatever was in the file, so aim it at a runaway log and never at a database.

One thing I did not expect: lsof exited 0 here. On a second demo mount holding another deleted file, lsof -w +L1 data, run from the parent directory, and lsof -w +L1 /srv/demo/data printed the same row. The relative path exited 1, the absolute one 0. Same lsof 4.99.4, same file. Read its output, never its exit status.

The next time: truncate, and know how the log was opened

The fix for next time is to empty a live log instead of deleting it: : > /srv/app/worker.log keeps the inode and every open descriptor, and the space comes back at once. There is a catch that depends on how the writer opened the file, so I ran two writers side by side, one opened with >> (O_APPEND) and one with >, each wrote 16M, I truncated both, and each wrote 1M more:

$ : > /srv/app/append.log; : > /srv/app/plain.log
$ ls -lsh /srv/app
total 2.0M
1.0M -rw-rw-r-- 1 root root 1.0M Oct  3 18:18 append.log
1.0M -rw-rw-r-- 1 root root  17M Oct  3 18:18 plain.log

The append writer started again at the new end of the file: 1.0M on disk, 1.0M long. The other writer kept its own offset, 16M, and wrote its next megabyte there. plain.log now claims to be 17M long with only 1.0M allocated, a sparse file whose first 16M is a hole. It costs no disk, but ls -l reports 17M, anything that copies it naively gets 16M of zeros, and anyone sizing logs by ls -l will chase a problem that is not there. That is why logrotate's copytruncate suits append-mode logs (its man page warns it can still lose the lines written between the copy and the truncate), and why for anything else the right move is to make the program reopen its log, usually with SIGHUP or a restart.

The script that does the looking

On a real machine lsof +L1 prints dozens of rows, and a deleted file shared by many processes appears once per process. The page's deleted-open-files.sh groups by inode, skips anything under 1M, sorts biggest first and totals it. On the same demo it printed ✗ 48M /srv/app/worker.log (deleted), held by: sleep[620003] fd 3, a total of 48M and the two ways to free it, and exited 1. After the truncate it printed ✓ no deleted-but-open files over 1.0M on /srv (as root) and exited 0, which makes it usable as a check in a disk alert.

The 48M log was never gone. It had lost its name and kept its blocks, and lsof +L1 was the one command that could see both.

lsof for deleted files, ports and busy mounts, with the grouping script: https://bashsnippets.xyz/snippets/lsof-command-examples

To hear about a filling disk before a write fails, the disk space warning script alerts at a threshold you pick, and Find Large Files on Linux covers what is still on the disk with a name. The rest of the library is at https://bashsnippets.xyz

I Tested a jq Alert Against a Down Status. It Printed 'no alert'.

Anguishe — Tue, 27 Oct 2026 13:00:00 +0000

The shape is everywhere in monitoring scripts: curl a status endpoint, pull one field out with jq, compare it, fire an alert on a match. I wrote the smallest honest version of it on my own machine, jq 1.8.1, against a response that said the service was down:

resp='{"status":"down"}'
status=$(echo "$resp" | jq '.status')
echo "[$status]"                                                 # ["down"]
if [ "$status" = "down" ]; then echo ALERT; else echo "no alert"; fi   # no alert

The service was down. The script said no alert, and exited 0. The echo "[$status]" line shows why: the variable holds "down", quote characters included. jq handed bash a six-character string, bash compared it byte for byte against the four characters in the test, and they never match. Put that in a five-minute cron job and it stays quiet through every outage it exists to report.

Adding two characters, jq -r, changed the output to [down] and the script to ALERT.

jq speaks JSON, bash speaks bytes

jq's default output format is JSON, and a JSON string includes its own quotes, so printing "down" is jq being correct. Bash's = has no idea that quotes might be markup; it compares the bytes it was given. -r tells jq to emit the string's contents rather than its JSON representation, which is why nearly every value headed into a shell variable, a filename or a test wants it.

That is the first of three ways a jq one-liner fails while looking right. The second: asking for a key the response does not have is not an error. On the same response, jq -r '.region' printed null and exited 0. In bash that is the literal four-character string null, which passes [ -n ] checks and walks straight into filenames and log lines. The // operator is the guard: jq -r '.region // "unknown"' printed unknown.

The third is select(), and here the folklore is half wrong. Forget the double quotes around a string inside the filter — select(.name == redis) — and jq does not quietly match nothing. It refuses to compile, prints redis/0 is not defined, and exits 3. That one is loud, and set -e will catch it. The silent version is a value that is spelled right but does not exist: select(.name == "nginx") against a list with no nginx in it printed nothing and exited 0. So did a case mismatch. Your script then treats that silence however it treats an empty string.

Three traps, and two of them share one property: plausible output, zero exit code.

Build the filter against the response you already have

The usual way to debug all three is trial and error against the live API: tweak the filter, re-run the curl, squint, repeat, often against a rate-limited endpoint. The jq Filter Builder turns that around, because you already have the JSON.

Paste a real response, or load one of the bundled samples, and it renders the parsed structure as a clickable tree. Click a field and the path is built for you, however deep it is nested. Arrays get the full treatment: switch on per-element iteration, attach a select() whose field name comes from a dropdown filled with the keys that actually exist on the elements, so a misspelled key stops being possible, and project one field out of each match. The comparison value is written the way jq reads literals — true, false and numbers bare, everything else double-quoted for you inside the single-quoted program. The quoting decision is made mechanically.

Below that sit two copyable outputs: the bare filter for a script, and the full curl -s … | jq command with your URL in place, -r included when you ask for raw output. And a live preview, evaluated in the browser against the JSON you pasted, on every change. Leave -r off and the quotes are right there in the preview. When the result is empty, the preview says why — No elements matched (jq would output nothing), Path not found, or that a // empty default is deliberately producing nothing — so empty output stops being ambiguous.

It runs client-side: nothing is uploaded and no remote jq is invoked, so a response from a production system stays in your browser. Scope, honestly: it builds the extraction patterns that cover most API scripting — nested access, array indexing, iteration with select() and projection, // defaults, raw output. Reduce, string interpolation and arithmetic still belong to the manual.

The four-line check at the top of this post could not have shipped from the builder. The quotes show up in the preview before the script exists.

Build a filter against your own API response: https://bashsnippets.xyz/tools/jq-filter-builder

The parse JSON with jq snippet covers these patterns in script form with the hardening around them, curl for API requests is the other half of the same pipeline, and the rest of the library is at https://bashsnippets.xyz

grep "error|failed" Found Nothing in a Log That Said 'backup failed'

Anguishe — Thu, 22 Oct 2026 13:00:00 +0000

I wrote a three-line backup log on my own machine: a start line, tar: /var/www/uploads: Cannot open: Permission denied, and backup failed (exit 2). Then I ran the check a lot of people keep as a weekly ritual, the one that looks like it asks "did anything go wrong?": grep "error|failed" backup.log.

GNU grep 3.12 printed nothing and exited 1. The last line of the log says failed in plain text.

grep -E "error|failed" backup.log printed that line and exited 0.

Nothing about the first command is malformed, so there is no error message. Exit 1 means "no lines matched", which is exactly what grep returns for a genuinely clean log. A monitoring check wired to that exit code reports all clear on a log full of failures, every single run, and the empty output does not start an investigation. It ends one.

The pipe that isn't a pipe

grep's default engine is Basic Regular Expressions, and in BRE the pipe is not alternation. It is a literal character. error|failed does not mean "error or failed"; it means the twelve-character string error|failed, pipe included, which no logger has ever written. The same goes for +, ? and parentheses: in BRE they are ordinary characters unless you backslash-escape them. Alternation the way every other regex engine behaves needs -E, Extended Regular Expressions — the mode egrep has been shorthand for since long before most of us started typing it.

# BRE (default): the pipe is a literal character — matches nothing, exits 1
grep "error|failed" backup.log

# ERE: the pipe means OR — the search you actually meant
grep -E "error|failed" backup.log

GNU grep also accepts grep "error\|failed" in BRE mode — on my machine that matched and exited 0 — but the escaped pipe is a GNU extension that POSIX does not define, so it is a habit that stops working on the systems that do not carry GNU grep. -E is the portable spelling.

The deeper trap is in the exit codes. grep exits 0 on a match, 1 on no match and 2 on an actual error; pointed at a file that did not exist, it printed No such file or directory and exited 2. An impossible pattern is not an error. "No matching lines" and "you searched for a string that cannot exist" produce identical output: none, with the same status. A false negative is the worst failure a search tool has, because nobody goes looking for a problem that grep has already said is not there.

The alias is a crutch

The tempting fix is alias grep='grep -E' in your shell rc file, and it does make the interactive ritual behave. It also does nothing anywhere that matters. Aliases apply only to interactive shells, so every cron job and script on the same box still runs BRE grep. The alias does not exist on any machine you have not configured, which is most of the machines you will ever SSH into. And it trains your fingers to type patterns that silently degrade the moment it is absent. The durable fix is knowing which engine a grep runs under and what its silence means before you trust it — and writing -E into the script itself.

So the builder shows the engine and the empty result

The grep Pattern Builder assembles the command live as you type: pattern, path, and an optional comma-separated list of file types that it turns into --include flags, so a log search does not wade through binaries. Toggles cover the flags that earn their keep — case-insensitive, recursive, line numbers, invert, count, filenames-only, whole-word, quiet and -B/-A context lines — and every combination produces a plain-English sentence saying what the command will do before you run it.

The engine picker sits front and centre: BRE, -E and -P, each with a note about what is literal where. The BRE note says it outright: + ? | ( ) are literal unless escaped. The -P note warns that Perl-compatible regex is GNU-only and fails on macOS and BSD grep, the portability cousin of the same trap: a \d pattern that works on a Debian server errors out on a Mac.

There are guardrails for mistakes people ship. It turns on -r when the path looks like a directory, because plain grep pointed at a directory refuses with "Is a directory", and in a script with stderr redirected that refusal is invisible. It drops -n when -c, -l or -q is active, because those flags change the output mode and line numbers stop meaning anything.

The panel that makes the ritual trustworthy is the live tester. Paste a few sample lines, including one real failure line from your actual log, and it highlights which lines the current pattern matches, with a count. In BRE mode it treats a bare pipe literally, the way grep does, so error|failed against a pasted backup failed line highlights nothing: zero out of one, a number that makes you look up. Switch to -E and the line lights. The preview runs on JavaScript's regex engine, which is close to ERE and PCRE for everyday patterns — treat it as a check on your pattern, not a replacement for running grep.

One unescaped character turned a failure search into a search for a string that cannot exist, and grep answered that question with perfect accuracy. The command was valid, ran clean, exited 1, and reported nothing wrong.

Build the command and read the explanation before you trust it: https://bashsnippets.xyz/tools/grep-pattern-builder

The Search Files for Text snippet covers the grep patterns worth keeping, Delete Old Log Files handles the other half of log hygiene, and the rest of the library is at https://bashsnippets.xyz

I Moved a Script to Another Directory on PATH. bash Said 'No such file or directory' and Exited 127.

Anguishe — Wed, 21 Oct 2026 13:00:00 +0000

One bash process, one small demo script called report, and a mv between two directories that are both on PATH. I set it up in a scratch tmpfs mounted over /opt inside unshare -rm, with PATH starting /opt/jobs/bin:/opt/jobs/newbin, and ran it as a script. Inside the namespace I appear as root, which is why ls says root root:

$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ ls -l /opt/jobs/newbin/report
-rwxrwxr-x 1 root root 42 Oct  3 18:26 /opt/jobs/newbin/report
$ report
session.sh: line 5: /opt/jobs/bin/report: No such file or directory
exit=127
$ type -a report
report is /opt/jobs/newbin/report
$ hash -t report
/opt/jobs/bin/report
$ command -v report
/opt/jobs/bin/report
$ hash -r
$ report
report 2.1: 14 jobs ok
exit=0

The file exists, it is executable, and its new directory is on PATH. bash ran the old path anyway, said No such file or directory, and exited 127, the same code a typo gets. I ran the same sequence through an interactive bash -i to be sure it was not a script quirk, and got bash: /opt/jobs/bin/report: No such file or directory, exit 127.

What makes it hard to trust your own eyes is the middle of that output. type -a names the new location. command -v, the check most scripts use to ask "is this installed", names the old one.

bash remembers where it found things

The first time bash runs a command through a PATH search, it stores the full path in a hash table, so the next call can skip the search. That is what hash -t report prints. When the file moves, bash does not search again; it executes the remembered path, the kernel answers ENOENT, and bash reports No such file or directory with the stale path in the message.

type -a searches PATH fresh every time, because its job is to list every match. command -v reports what bash would run, and bash would run the hashed path. So the two most common "where is it" checks disagree, and the one that agrees with the error is the one that looks wrong.

hash -r empties the table. A new terminal starts with an empty one. And there is a shell option made for exactly this:

$ shopt -s checkhash
$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ report
report 2.1: 14 jobs ok
exit=0
$ hash -t report
/opt/jobs/newbin/report

With checkhash set, bash checks that a hashed path still exists before running it, searches PATH again when it does not, and updates the table. In the wild this shows up after an upgrade moves a binary between /usr/local/bin and /usr/bin, or after swapping a pip-installed tool for the distro package, both of which the page describes.

Six causes, one exit code

command not found and its 127 cover more than a stale hash. The page reproduces the rest: a typo, a program installed outside PATH, a script in the current directory run without ./, an alias a script cannot see, a found file whose shebang names a missing interpreter (a CRLF line ending does that), and the cron job whose PATH is shorter than your terminal's.

The cron one is worth reproducing, because it never fails while you are watching. Below, a demo tool in a scratch directory on my interactive PATH, then the same call with cron's two-directory PATH, which env -i reproduces without waiting for the schedule. After that, the page's diagnosis script, why-command-not-found.sh, run in the namespace with a home directory of its own, shown as ~, and the same tool in ~/.local/bin:

$ nightly-sync
nightly-sync 0.4
exit=0
$ env -i PATH=/usr/bin:/bin bash -c 'nightly-sync'
bash: line 1: nightly-sync: command not found
exit=127

$ ./why-command-not-found.sh nightly-sync
✓ nightly-sync resolves to ~/.local/bin/nightly-sync
✗ cron's default PATH (/usr/bin:/bin) will NOT find it: use ~/.local/bin/nightly-sync in the crontab, or set PATH= at its top
  if your terminal still says 'not found' or 'No such file', its hash table is stale: run hash -r
exit=0
$ ./why-command-not-found.sh gti
✗ gti is not on PATH
  did you mean git? (/usr/bin/git)
exit=1

It uses type -P, which searches PATH for files only and ignores aliases and functions, so it sees what any other script would see. It repeats the lookup with PATH=/usr/bin:/bin to catch the cron case, reads the first line of the file for a carriage return, checks ./NAME, looks in the usual places installers put binaries without touching PATH (~/.local/bin, ~/.cargo/bin, /usr/sbin, /opt/*/bin), and tries every adjacent-letter swap of the name, which is how gti became git. It exits 1 for every cause it finds, so ./why-command-not-found.sh rsync || exit 1 can gate a deploy script.

The order that separates them

type -a NAME for everything bash can see. hash -t NAME when the message says No such file or directory about a path. echo "$PATH" | tr : '\n' when the answer is "nothing". And env -i PATH=/usr/bin:/bin bash -c NAME before anything goes in a crontab.

The stale hash is the one cause where everything you check says the tool is fine: ls and type -a both show it right where it should be. One hash -r, or one new terminal, is the whole fix.

All six causes reproduced, with the diagnosis script and what each check does: https://bashsnippets.xyz/snippets/bash-command-not-found

If 127 is all a log gives you, the Bash Exit Code Lookup explains it next to 126, and the Bash $PATH Debugger checks a pasted PATH for missing directories and duplicates. The rest of the library is at https://bashsnippets.xyz

I Pointed rsync --delete at a Nearly Empty Directory. It Deleted 3,800 Files and Exited 0.

Anguishe — Tue, 20 Oct 2026 13:00:00 +0000

The rsync accident that ends backups is not an error. It is rsync doing exactly what it was told, successfully, to the wrong directory. I wanted to see how quiet that is, so I rebuilt it in a scratch directory on my own machine with rsync 3.5.0.

The destination held 3,800 files, standing in for a nightly mirror. The source held one file, index.nginx-debian.html — what Debian's nginx package leaves behind in /var/www/html after you move a site's web root somewhere else and forget that the backup script still points at the old path. Then the mirror command anyone would have in that script: rsync -a --delete src/ dst/.

It exited 0. Afterwards the destination held one file: index.nginx-debian.html. Three thousand eight hundred files gone, in a run that any cron wrapper, healthcheck ping or ✓ backup complete log line would have recorded as a success. Run that nightly and every green checkmark in the dashboard is a record of the deletion.

Exit 0 doesn't mean what your monitoring thinks

The bug is not in rsync. It is in what we assume its exit code promises. rsync reserves non-zero codes for failures on its own terms: an unreachable host, a protocol error, an I/O failure, source files vanishing mid-transfer (code 24), a partial transfer (code 23). Making a destination match a near-empty source is none of those. It is the requested operation, delivered without incident. A monitor that checks the exit code answers "did rsync finish?" and most of us have labelled that answer "is the data safe?" Those are different questions.

rsync's sharp edges fail by succeeding

Once you see that shape, you find it all over the command. A trailing slash on the source means "copy the contents"; no trailing slash means "copy the directory itself", which nests a second directory inside your destination and, on a --delete mirror, changes what gets compared and therefore what gets removed. Both versions run clean. Drop -a and rsync stops preserving permissions, symlinks and timestamps, and because its quick check decides "already transferred" by size and modification time, a destination full of transfer-time mtimes never matches again, so every run quietly becomes a full re-copy that still exits 0. Hand a comma-separated list to a single --exclude and it is treated as one pattern with a comma in it, so it matches nothing and node_modules rides along every night. None of these produce an error. Each completes a different operation from the one in your head.

The flag that shows the damage first

The habit that prevents the whole class is previewing every destructive sync. Same scratch directories, same command, with --dry-run and -v:

$ rsync -a --delete --dry-run -v src/ dst/
sending incremental file list
deleting photo-999.jpg
deleting photo-998.jpg
deleting photo-997.jpg
…

Thousands of deleting lines, and afterwards the destination still held all 3,800 files. A wall of deletions in a preview you are reading is impossible to miss. The same wall executed in a log nobody opens is a green checkmark.

One detail the documentation does not shout: the -v is not optional. I ran --dry-run without it against a second scratch directory and it printed nothing at all and exited 0. A dry run you cannot see is not a preview. Use -v, or --itemize-changes if you want one line per file with a change code.

So the builder makes the preview the default

The Rsync Command Builder assembles the command in a live preview while you toggle the real options: archive mode, verbose output, compression, resume for unreliable links (--partial --progress), SSH transport, a bandwidth cap for syncs that share a link with people trying to use it, and exclude patterns. You type the excludes comma-separated and it emits one individually quoted --exclude flag per pattern, so the shell never expands your glob and the comma mistake cannot happen.

The opinions built into it are the point. Turn on --delete without dry-run and a red warning appears: those deletions are permanent, preview them first. Turn dry-run on and it switches to a green note telling you to remove --dry-run only after you have read the output. The Mirror preset ships with dry-run already enabled — mirroring is the one mode where you switch the safety off on purpose instead of remembering to switch it on. The whole thing runs in your browser; nothing you type leaves the page.

The scratch run took a fraction of a second to delete 3,800 files and report success. The preview took the same fraction of a second to list every one of them. The only difference was a flag.

Build the command with the guardrails on: https://bashsnippets.xyz/tools/rsync-command-builder

For the script around it — cron scheduling, SSH keys, a hardened nightly job — the Rsync Remote Backup snippet pairs with this tool, Automated File Backup covers the local variant, and the rest of the library is at https://bashsnippets.xyz

My Machine Has No mail Command. That's Where Most Disk Alerts Die.

Anguishe — Thu, 15 Oct 2026 13:00:00 +0000

I recently found that the disk check on my own Linux machine had been printing WARNING: Disk at 88% into nothing: cron's journal said No MTA installed, discarding output beside every run. So I checked what it would take for that warning to reach me by email. mail: not installed. mailx: not installed. sendmail: not installed. msmtp: not installed.

That is normal for a desktop install and common on minimal server images. It also means every monitoring script that ends in | mail -s "…" you@example.com is a script whose last line fails. Run it by hand and you would at least see command not found. Run it from cron and the error goes the same way as the output: cron tries to mail it to you, has no mail agent to do it with, and discards it. Two layers of silence, and the check itself is working perfectly.

Detection and delivery are different jobs

The detection logic is the part everybody polishes: read the number, compare it, decide. Telling a person is a separate job, and a monitor that only appends to a log file or prints to stdout has not done it — it has relocated the problem. Somebody still has to poll that output, and that somebody is a person with better things to do at 2am. The fix is to push the news to the place you already check compulsively, which for most of us is an inbox.

The core is small enough to memorize:

#!/bin/bash
set -euo pipefail

USAGE=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -gt 80 ]; then
  printf 'Disk on %s hit %s%% at %s\n' "$(hostname)" "$USAGE" "$(date '+%F %T')" \
    | mail -s "[ALERT] disk space on $(hostname)" you@example.com
fi

Read the number, compare it, and when it is over the line, pipe a message into mail with a subject that names the machine — because once the same check runs on several hosts, an alert that does not say which server is on fire is a puzzle, not an alert.

The alert you "sent" that never arrived

Installing mail fixes the first layer and exposes the next one. mail exiting 0 does not mean your alert reached anyone. It means the message was handed to whatever mail transfer agent lives on the box; delivery is the MTA's problem, and the exit code you can observe stops at the handoff. On a server with no MTA configured for the outside world, that handoff often drops the message into a local mailbox under /var/mail — a file on the same machine that no human reads. The script reports success, and the alert sits a few directories away from the log it was supposed to replace.

Cron adds one more trap on top. Jobs run with a minimal PATH, so a mail that resolves in your login shell can be command not found under cron. The test that proves anything is not running the script by hand and receiving an email. It is letting cron run it and receiving an email, then reading the MTA's log to see what it actually did.

Real delivery means giving the box a real path out. On Debian or Ubuntu, mailutils pulls in an MTA, and the "Internet Site" option covers a machine that is allowed to send directly. Plenty are not — outbound mail from cloud IP ranges gets filtered or spam-binned constantly — so relay through a provider instead: msmtp pointed at a Gmail account with an app password is a few lines of config, and curl can speak SMTP to a relay on boxes where you would rather not install an MTA at all. Either turns "handed to the MTA" into "accepted by a server whose whole job is delivering".

The opposite failure: the alert that cries wolf

Once delivery works you meet the inverse problem. A threshold does not trip once; it stays tripped. Disk at 81 % at nine o'clock is still 81 % at ten, and an hourly cron job will mail you the same fact twenty-four times a day until you fix the disk or stop reading, which comes first. An alert channel you have learned to skim is a silent failure with extra steps.

The guard is a dedupe sentinel: on send, touch a marker file; before sending, check the marker's age with find "$MARKER" -mmin -1440. I ran that logic three times on my machine. The first call sent. The second, a moment later, was suppressed. Then I backdated the marker 25 hours with touch -d and the third call sent again. One email per incident per day; the condition is still measured every hour, only the nagging is suppressed.

Delivery is the whole difference

The check on my machine had the right threshold and the right schedule-shaped intentions. What it lacked was any route from the machine to me, and nothing in its own output could say so, because its output had nowhere to go either. Before you trust an alert, send one on purpose, from cron, and watch it arrive.

The full script — the alert body with the top disk consumers included, the msmtp config for a Gmail relay, the once-per-24-hours dedupe guard, and the cron traps that make alerts vanish — is here: https://bashsnippets.xyz/snippets/bash-send-email-alert

An email alert is the last guard an unattended job needs, not the first: the Cron Wrapper Generator wires alert-on-failure in alongside the lock, timeout and retry it generates, and Bash Scripts That Survive Cron is the full playbook for jobs nobody watches. The rest of the library is at https://bashsnippets.xyz