DEV Community: Schiff Heimlich

Who Watches Your Watchdog? (And Why It Probably Isn't You)

Schiff Heimlich — Thu, 03 Sep 2026 17:01:36 +0000

Who Watches Your Watchdog? (And Why It Probably Isn't You)

Been thinking about this after reading about someone who built a cron-job monitoring service and had it silently stop running. The watchdog stopped watching. No alerts, no notifications. Just quiet failure.

It's one of those things that sounds obvious in hindsight but catches a lot of people.

The setup

You've got some jobs running on a schedule. Maybe they're important, maybe they're not, but you want to know when they fail. So you build or deploy something that checks on them - a watchdog service that sends you an alert if a job misses its window.

This works fine. Until the watchdog itself stops working.

What went wrong

The person running PulseWatch (a Python-based cron watchdog SaaS) had it running on GitHub Actions as a scheduler. The scheduler hiccuped - not the jobs being monitored, but the thing checking the jobs. The watchdog just... stopped.

Here's the thing: if your watchdog dies, you don't get notified. That's the whole point of having a watchdog, and it's also the failure mode.

The fix

The solution was to chain the watchdog to an external probe - specifically, a Healthchecks.io endpoint. The watchdog pings Healthchecks.io on a schedule, and if Healthchecks.io doesn't hear from it, that triggers an alert.

Now you have two layers:

Your watchdog monitors your jobs
Healthchecks.io monitors your watchdog

The failure domains are separated. If your watchdog dies, an external service notices.

The grace period trap

There's another lesson here about how grace periods work. The author had a "15-minute cron" but assumed that meant a 15-minute grace period. It doesn't.

A cron expression of */15 * * * * means "run every 15 minutes" but doesn't guarantee exactly every 15 minutes. GitHub Actions might run it at :00, :15, :30, :45 - or it might be :02, :17, :32, :47. The scheduler has its own timing.

If your grace period is tied to your cron interval, you're going to get false alerts when the scheduler drifts. The fix is to make the grace period longer than the cron interval, with enough buffer for scheduling variance.

What to check

If you're running any kind of monitoring or alerting system:

How do you know it's still running? Is there an external check on the check?
What's your grace period? Is it longer than your check interval plus scheduling variance?
Where does the alert go? If your primary alerting channel is down, is there a fallback?

The boring truth is that monitoring systems fail too. The question is whether you'll notice.

Original research from Saijo. Source: https://dev.to/chriscompiles/who-watches-the-watchdog-the-boring-work-behind-a-monitoring-saas-7n8

Why Your Alpine Containers Might Be Hiding a Security Blind Spot

Schiff Heimlich — Sat, 29 Aug 2026 17:12:18 +0000

Hey,

Been looking at container base images lately and noticed something worth sharing about Alpine that might not be obvious.

Most of us reach for Alpine when we want small, efficient container images. It's become pretty standard for a lot of workloads. But there's a detail about its base that's easy to overlook: BusyBox.

BusyBox bundles a bunch of Unix utilities into a single executable. It's great for embedded systems, but it wasn't really built with cloud-native container security in mind. The issue isn't that BusyBox is bad—it's that when a vulnerability shows up in any part of it, you've got exposure across your entire userspace since it provides so many core utilities.

What's tricky is that this isn't something your usual container scanners will flag prominently. They'll tell you about CVEs in specific packages, but the BusyBox situation is a bit different—it's more about the architecture of the base image itself.

Some teams are starting to address this by looking at base images that either minimize or eliminate BusyBox from the production image build process. The idea is to move some of that security consideration earlier during the image build phase rather than relying solely on runtime scanning.

If you're using Alpine (or similar images), it might be worth checking:

What's actually in your base image beyond your application dependencies
Whether your scanning tools are giving you visibility into the userspace components
If there's a build step where you could simplify or replace parts of the base image for your specific use case

It's not about abandoning Alpine—it's about understanding what's in it and making sure that fits your security model.

Let me know what you've seen in your own setups.

--Schiff

When 'No space left on device' has nothing to do with disk space

Schiff Heimlich — Fri, 28 Aug 2026 17:10:40 +0000

You're running a log shipper or a config reloader and suddenly it starts failing with ENOSPC — "No space left on device." You check df -h, and the disk is fine. Plenty of inodes, plenty of blocks. So why is the kernel refusing to do work?

Chances are it's not disk space at all. It's the inotify watch limit.

inotify is the kernel's file change notification mechanism. Every directory you watch with tools like inotifywait, a config reloader, or a log shipper consumes a "watch." Those watches come from a finite pool, and when you exhaust it, the kernel returns ENOSPC even though the filesystem is nowhere near full. The error message is misleading, and it's easy to burn an hour chasing disk space that was never the problem.

Check the limits first

Two sysctls control this:

# Max number of watches (default is often 8192 or 65536)
sysctl fs.inotify.max_user_watches

# Max number of queued events per instance
sysctl fs.inotify.max_user_instances

The one that bites most people is max_user_watches. If you're running a bunch of containers or a monitoring agent that watches a large directory tree, 8192 goes fast.

Confirm it's actually inotify

If you're getting ENOSPC and the disk is fine, check the kernel logs:

dmesg | grep -i inotify
journalctl -k | grep -i inotify

You'll often see something like:

inotify watch limit reached

That's your smoking gun. It's not disk space, it's the watch table.

How many watches do you actually need?

Rough rule of thumb: one watch per directory you're monitoring. If you're watching a tree with 50,000 directories, you need at least 50,000 watches. You can get a rough count of what a tool is using with strace, but for planning purposes, size it against the directory count of whatever you're shipping logs from.

Fix it persistently

The one-liner that works until reboot:

sysctl -w fs.inotify.max_user_watches=524288

But if you want it to survive a reboot, put it in a sysctl drop-in file:

echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
sysctl --system

sysctl --system applies everything in /etc/sysctl.d/, so it takes effect immediately and sticks across reboots. On systemd systems this is the clean way to do it — don't edit /etc/sysctl.conf directly when a drop-in file does the job and keeps your change obvious.

The part people forget

Raising the watch limit is the easy half. The other thing to check is max_user_instances — the number of separate inotify instances a single user can create. If you're spawning many short-lived watchers (some tools do this per-connection), you can hit that limit too. Bump it in the same drop-in file if you need to.

Also remember: watches are a per-user resource, and the limit applies to your user, not the whole box. If you run monitoring as root, root has its own budget. Container runtimes often share this, so a busy host can blow through it faster than you'd expect.

The takeaway

When a monitoring tool fails with "No space left on device" and the disk is fine, don't fight the filesystem. Check fs.inotify.max_user_watches, confirm with dmesg, and raise it via a sysctl drop-in file. It's a ten-minute fix that looks like a disk emergency until you know what you're looking at.

Nginx proxy_pass: The trailing slash that changes everything

Schiff Heimlich — Thu, 27 Aug 2026 17:08:45 +0000

You know how sometimes you copy an nginx config snippet and it just... doesn't work? You stare at it, reload, check the logs, and nothing seems wrong. Then you spot it: a missing or extra slash at the end of the proxy_pass line. It’s one of those tiny details that silently breaks your setup.

Let’s look at two examples.

First, with a trailing slash:

location /app/ {
    proxy_pass http://backend/;
}

Here, nginx replaces the matched part (/app/) with the URI in proxy_pass (http://backend/). So a request to /app/index.html goes to http://backend/index.html.

Now, without the trailing slash:

location /app/ {
    proxy_pass http://backend;
}

In this case, nginx passes the entire request URI unchanged. So /app/index.html becomes http://backend/app/index.html.

The difference is subtle but important. If your backend expects requests at the root (like /index.html), the first form is what you want. If it expects the path to be preserved (like /app/index.html), the second form works.

I’ve seen this trip up people when they move from a location block that matches a prefix to one that’s more specific. The muscle memory of copying the proxy_pass line doesn’t account for the slash.

It’s not a bug. It’s documented behavior. But it’s easy to overlook when you’re in a hurry. Next time your proxied requests are going to the wrong place, take ten seconds to check that slash.

That Time Your Deploy Failed Because SSH Thought You Were an Attacker

Schiff Heimlich — Wed, 26 Aug 2026 17:04:51 +0000

That Time Your Deploy Failed Because SSH Thought You Were an Attacker

You have got 30 Ansible tasks firing at once. Or maybe you are using Capistrano, or parallel-scp to push code to 30 servers at the same time. Mid-deploy, everything starts timing out. Connection refused. You blame the network. You blame AWS. You restart the deploy.

It is your sshd_config.

The default MaxStartups setting is 10:30:60. Let me break that down.

What MaxStartups Actually Means

The syntax is start:rate:full:

10 = allow 10 unauthenticated connections before starting to drop
30 = drop 30% of new connections once you hit 10
60 = refuse all new connections once you hit 60

So with 30 parallel SSH connections hitting the same sshd at once, you are going to start getting refused. Not throttled—refused. "Connection refused" errors that make you think the service is down when really sshd just does not like your burst.

Why This Bites Deploys Specifically

Parallel deployment tools are built for speed. They open connections to multiple hosts concurrently. If those connections all tunnel through a jump host or land on the same deployment box, you have suddenly got 30 simultaneous auth attempts from one IP.

Your laptop doing 30 parallel SSH connections to prod-server-01? That is 30 connections from your IP to one port. Sshd thinks you are running a dictionary attack.

The Fix

In /etc/ssh/sshd_config:

MaxStartups 30:30:100

Or if you know your deployment patterns, just bump it:

MaxStartups 50

Reload sshd: sudo systemctl reload sshd

If You are Using a Jump Host

The jump host takes the hit. All your parallel connections pile up there. Make sure your jump host sshd_config can handle your parallelism, or use connection multiplexing to reuse authenticated connections.

The Debugging Clue

Next time you see mysterious connection refused errors only during deploys, check if it correlates with the number of parallel tasks. If your network team says "nothing wrong," check your sshd_config. Probably the sshd_config.

The default MaxStartups exists for good reason—it is a basic throttle against SSH brute force. But it was not designed for automated parallel workflows. Know your tool defaults before you spend an hour debugging the network.

That Time Client Retries Turned a Recovery Into a 7-Hour Outage

Schiff Heimlich — Tue, 25 Aug 2026 17:06:20 +0000

GitHub had an interesting incident last August. A component in Central US failed under load, and when it started recovering, the recovery took much longer than it should have. The culprit: clients hammering the recovering system with retries.

What happened

During the outage, GitHub's auth system was down. When it came back up, every client that had been patiently waiting started hammering it with retry requests simultaneously. The auth system was trying to recover, but now it was drowning in retry traffic from clients that had given up waiting.

This is the retry-loop trap. You've probably seen it in various forms:

A health check endpoint that returns 503, so clients retry every second
A connection pool that overflows, returns an error, clients reconnect immediately
A CDN origin that's down, all edge nodes retry at the same rate

The pattern

The typical flow:

System goes down or slow
Clients start retrying with some interval
System partially recovers
Retries flood the recovering system
System goes down again or stays slow
Repeat until someone intervenes

What actually helps

Exponential backoff with jitter is the standard answer. Instead of retrying every N seconds, you retry at increasingly longer intervals with some randomness. Like: 1s, 2s, 4s, 8s, 16s... with +/- 20% jitter. This spreads out the retry load.

Circuit breakers stop forwarding requests to a failing service entirely for a cooldown period. The client knows the circuit is "open" and fails fast instead of retrying.

Rate limiting on the server side helps, but during recovery you often want to gradually increase capacity, not hard-cutoff.

Client-side rate limiting is underused. Most HTTP clients will retry indefinitely by default.

The GitHub specific angle

GitHub's postmortem mentioned record traffic that day - 115M Actions runs, 2.9B monthly commits. That's a lot of automated systems hammering retry loops. The lesson isn't that GitHub's infra was bad; it's that the combination of a brief outage + automated clients with naive retry logic creates a thundering herd on recovery.

If you're running a service that clients depend on, it's worth thinking about what their retry behavior looks like. Are they using exponential backoff? Do they have a circuit breaker? Or are they just spinning on a 1-second loop?

And if you're the client, it's worth checking your default retry config. The defaults in many HTTP libraries are not kind to recovering services.

This was the GitHub Aug 17 outage lesson. Source: DevOps'ish #323 and GitHub CTO Vlad Fedorov's postmortem.

That time YAML silently broke your config because of Norway

Schiff Heimlich — Mon, 24 Aug 2026 17:05:31 +0000

A colleague pinged me last week with a config issue that had him chasing his tail for an afternoon. Turns out he was setting country: NO in a YAML file, expecting it to mean Norway. YAML thought he meant boolean false.

The Problem

YAML 1.1 treats these unquoted strings as booleans:

yes, no, on, off, true, false
Case-insensitive variants work too

So if you have:

database:
  country: NO
  enabled: true

YAML parsers will often read country: NO as country: false. Your application starts, the config loads, and nobody gets an error — it just silently does the wrong thing.

This has been documented for years. It bit DoorDash. It's been in Kubernetes manifests. It's in CI configs, Terraform variables, GitHub Actions, Helm charts.

Why It Matters in Practice

You won't see a crash. Your app just reads the wrong value. The country code field becomes a boolean, silently coerces to something unexpected, and the behavior diverges from what you intended. Debugging this is annoying because there's no error — just wrong behavior.

What Helps

Quote your strings — country: "NO" always works. It's the safest habit.
Use KYAML — The SIG CLI KEP 5295 introduces a strict YAML subset that quotes strings, braces maps, and brackets lists. No more ambiguity about whether whitespace or capitalization matters.
Lint your YAML — Tools like yamllint can catch some of these issues, though they won't catch semantic boolean confusion without schema awareness.

The Fix in Practice

If you're writing config that ships to others, get in the habit of quoting anything that could be misinterpreted:

example:
  country: "NO"   # quoted - will stay a string
  flag: false     # boolean - this is intentional
  message: "yes" # quoted - stays the string "yes", not boolean

It's a small discipline that prevents a class of silent failures that are otherwise hard to track down.

Source: DevOps'ish #322 (SIG CLI, KEP 5295)

The sudoers.d Trap That'll Waste an Hour of Your Life

Schiff Heimlich — Sun, 23 Aug 2026 17:03:51 +0000

The Sudoers.d Trap That'll Waste an Hour of Your Life

If you've ever dropped a file into /etc/sudoers.d/ and watched your sudo rule quietly do nothing, you're not alone. This one trips up almost every Linux admin at least once.

The Problem

You add a rule like:

ALL ALL=(ALL) NOPASSWD: /usr/bin/systemctl status nginx

to /etc/sudoers.d/myapp, hit save, and... nothing changes. The user still gets prompted for a password. Or worse — sudo breaks entirely and you get:

sudo: unable to open /etc/sudoers.d/myapp: Permission denied

Two Things That Break It Every Time

1. File permissions

The file must be mode 0440 and owned by root:root. Any deviation and sudo ignores it silently.

chmod 0440 /etc/sudoers.d/myapp
chown root:root /etc/sudoers.d/myapp

2. Missing trailing newline

This one is absurd but real. sudoers files must end with a newline. If your editor strips it or you echo -n the last line, the entire file gets rejected. visudo will catch this. nano or cat >> often won't.

How to Actually Debug It

visudo -c checks all sudoers files for syntax errors:

# visudo -c
/etc/sudoers.d/myapp: wrong #! solver specifier, line 3 of...

That output tells you exactly which file and which line. Without -c you get nothing — it just silently skips the bad file.

The Habit

Always edit sudoers files with visudo -f /etc/sudoers.d/myapp instead of your editor of choice. It locks the file, validates syntax on save, and tells you immediately when something is wrong. It's the difference between a 5-second fix and a 45-minute incident.

This isn't a deep trick. It's just the thing nobody tells you until it costs you an afternoon.

Why Your Kubernetes Automation Keeps Getting Blocked by Humans

Schiff Heimlich — Sat, 22 Aug 2026 17:03:56 +0000

A recent survey hit on something I see all the time in cluster work: teams know they need automation, but 71% still require human review before any resource changes get applied. The 48% who said visibility and transparency would increase trust in automation? That's the actual problem.

When you're asking someone to approve a change they can't verify, you're not going to get buy-in. This isn't about trust in the automation algorithm. It's about trust in the person asking them to approve it blindly.

The Typical Breakdown

You've got a team running Kubernetes. Someone writes a script to right-size resource requests based on actual usage. It looks at historical CPU and memory patterns, spits out new resource specs, and... gets stuck in approval limbo because no one understands why the automation thinks a pod needs 200m CPU instead of 500m.

The reviewer sees: spec.containers[0].resources.requests.cpu: 500m → 200m
The automation sees: 6 weeks of P95 usage at 180m, with headroom for burst

No visibility into the reasoning = no approval.

What Actually Helps

1. Output the analysis, not just the result

Instead of:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    resources:
      requests:
        cpu: 200m

Generate:

# Analysis for app pod (namespace: production)
# Current request: 500m CPU, 512Mi memory
# 
# Usage data (30 days):
#   CPU P50: 120m | P95: 175m | P99: 210m
#   Memory P50: 280Mi | P95: 420Mi | P99: 480Mi
#
# Recommendation: 250m CPU, 512Mi memory
# Rationale: P95 usage is 175m, recommending 250m for headroom
# Memory looks fine at current 512Mi (P99 is 480Mi)

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    resources:
      requests:
        cpu: 250m

2. Diff-friendly output

Format recommendations as patches or diffs that are easy to review in Git:

# kubectl-automation-analyze namespace production --format=diff
- apiVersion: v1
  kind: Deployment
  metadata:
    name: app
  spec:
    template:
      spec:
        containers:
        - name: app
          resources:
            requests:
-             cpu: 500m
+             cpu: 250m

3. Dry-run by default

Never apply automatically. Even if your team trusts the automation, give them the option to review. A --auto-approve flag they can enable after watching it run for a week does more for adoption than any documentation.

The Real Takeaway

The automation gap exists because we treat "more automation" as the goal. But if your team can't see what the automation is doing, they'll default to manual review every time.

The fix isn't better algorithms. It's better output. Give people the data, show your work, and let them make an informed decision. That's not less automation - that's automation people actually use.

Bash Parameter Expansion: Two Quirks That Slip Through Code Reviews

Schiff Heimlich — Fri, 21 Aug 2026 17:04:57 +0000

Bash Parameter Expansion: Two Quirks That Slip Through Code Reviews

Here's a pair of bash parameter expansion behaviors that I've seen slip through code reviews and cause intermittent failures in production scripts. Small stuff, but worth knowing.

${var:-default} vs ${var-default}

This trips people up constantly. The colon matters:

var=""  # var is set, but empty

echo "${var:-default}"   # prints: default
echo "${var-default}"     # prints: (empty)

With the colon, bash checks if the variable is unset OR empty. Without the colon, it only checks if the variable is unset. So an empty string gets treated as "use the default" with :-, but as "use the empty value" without.

In practice, this matters most when you're handling environment variables or config values that might legitimately be empty:

# Safe: use default only when truly absent
HOME_DIR="${HOME:-/root}"

# Unsafe: empty HOME would still use /root
HOME_DIR="${HOME-/root}"

The colon form is almost always what you want when setting defaults for potentially-empty values.

[[ -v var ]] vs [[ -n $var ]]

Another subtle one. -v checks if a variable is declared (and non-empty in bash 4.2+). -n checks if the value is non-empty.

var=""  # declared but empty

[[ -v var ]]    # true (var exists)
[[ -n $var ]]   # false ($var expands to empty string)
[[ -n "$var" ]] # false (same, quoted or not)

The -n "$var" version is actually checking if the quoted string is non-empty, which is always false for an empty variable. But -v correctly identifies that the variable exists, regardless of its value.

This shows up when checking if optional flags were passed:

VERBOSE=""  # passed but empty

[[ -v VERBOSE ]] && echo "verbose mode"     # correctly matches
[[ -n $VERBOSE ]] && echo "verbose mode"     # misses it

For declared/empty check: use -v. For "has non-empty value" check: use -z or -n "$var".

Practical Takeaway

When writing scripts that handle config or environment variables, prefer ${var:-default} and [[ -v var ]]. They're the more defensive choices and behave consistently across bash versions.

Nginx proxy_buffering: the one setting that quietly breaks your backend under load

Schiff Heimlich — Thu, 20 Aug 2026 17:03:21 +0000

So you have nginx proxying to a slow backend. Maybe it's a streaming endpoint, maybe it's a legacy service with occasional slow queries. Under normal load everything works fine. Then traffic spikes and suddenly you're getting 502s, timeouts, or your OOM killer fires.

The culprit is often nginx's default buffering behavior.

What happens

By default, nginx buffers the entire backend response before sending anything to the client. This is meant to improve performance when the backend is faster than the client. But when your backend is slow or streams data, nginx will hold the full response in memory (or disk, if it exceeds proxy_max_temp_file_size) before passing it along.

During this time, that nginx worker is tied up. Enough slow responses and you run out of workers, new requests queue up, and things go downhill fast.

The fix

location /slow-endpoint/ {
    proxy_pass http://backend;
    proxy_buffering off;
    proxy_buffer_size 4k;
}

proxy_buffering off tells nginx to stream the response straight through instead of buffering it. The client gets data as it arrives, workers don't get blocked.

The gotcha with proxy_max_temp_file_size

Even with buffering on, nginx can spill to disk. The default is 1024m. If you have many slow responses, this can fill your disk and cause the same problem. Either set it to 0 to disable disk spilling entirely, or monitor your error logs for buffer-related messages.

How to diagnose

If you're not sure whether buffering is your issue, check these:

strace on nginx workers — look for large writes to temp files
lsof on nginx workers — check for open temp files in the nginx tmp directory
error log during load — buffer-related errors show up here
tcpdump if it's localhost — see if responses come back in one chunk or streamed

When to leave buffering on

Buffering is fine when your backend is fast and responses are small. It reduces backend load because nginx can serve cached responses without hitting the backend again. The issue only surfaces with slow or streaming backends.

The fix is a single line. The diagnosing is the part that takes time if you don't know where to look.

Tags: nginx, devops, sysadmin, performance

When Disk Space Is Not Disk Space: The Inode Exhaustion Problem

Schiff Heimlich — Wed, 19 Aug 2026 17:04:45 +0000

When Disk Space Is Not Disk Space: The Inode Exhaustion Problem

You run df -h, everything looks fine. Plenty of space left. Then your application throws "No space left on device" and you spend an hour debugging the wrong thing.

That's inode exhaustion. Here's how it works and how to deal with it.

What Even Is an Inode

Every file on a filesystem has an inode. It's a metadata structure that holds:

File permissions and ownership
Timestamps
Pointers to the actual data blocks
A unique inode number

The inode table has a fixed size when you format the filesystem. You can have millions of bytes of free space but if you run out of inodes, you can't create new files.

# Check inode usage
df -i

# Example output:
# Filesystem      Inodes  IUsed  IFree IUse% Mounted on
# /dev/sda1      3276800 3276800      0  100% /data

Notice the IFree column is 0. That's your problem right there.

Why This Sneaks Up on You

Small files are the usual culprit. Each file consumes at least one inode regardless of size. A directory with millions of tiny files (cache files, session data, log shards) can burn through your inode table fast.

Common situations:

/tmp filling with unnamed files from crashed processes
Mail spools with tiny deferred messages
Object storage caches creating countless small blobs
Build artifacts that leave behind many tiny intermediate files

How to Find the Culprit

# Find directories with high inode counts
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

# Specifically look at /tmp
sudo find /tmp -type f | wc -l

# Check a specific directory tree
sudo find /var -type f | wc -l

Once you find the directory, you can drill down:

sudo find /tmp -type f -ls 2>/dev/null | sort -k7 -n | tail -20

The tmpfs Gotcha

A less obvious trap: tmpfs filesystems (RAM-backed storage) mounted at /tmp or /run.

tmpfs has configurable size limits:

# Check tmpfs sizes
mount | grep tmpfs

# Example output:
# tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=512M)
# tmpfs on /run type tmpfs (rw,nosuid,nodev,size=100M)

The size= parameter limits both the byte size AND the inode count. If you set size=512M but don't set nr_inodes, the kernel auto-calculates inodes based on the size. Low memory systems often end up with surprisingly few inodes.

When tmpfs fills up, you get OOM behavior even though it's supposed to be backed by RAM. The kernel starts killing processes to free memory, which is rarely what you want.

Prevention

Monitor inodes, not just space. Add df -i to your alerting.
Set explicit inode limits on tmpfs if you're using one

   tmpfs /tmp tmpfs rw,nosuid,nodev,size=1G,nr_inodes=1M 0 0

Clean up strategies for temp directories. Logrotate isn't enough for /tmp - processes should clean their own mess, but they don't always.
Choose filesystems wisely for small-file workloads. Some filesystems (ext4 with bytes-per-inode ratio) let you tune inode density at mkfs time.

The Quick Fix When You're Stuck

# If /tmp is the problem and you need breathing room NOW
# (this clears all files in /tmp - be careful)
sudo find /tmp -type f -delete

# If you need to identify files by inode number for targeted deletion
sudo find /path/to/problem -inum 12345 -delete

The real fix is figuring out why you're creating files faster than you're cleaning them up.

Bottom Line

df -h is incomplete information. When something reports "no space" but df -h looks fine, check df -i before anything else. Inode exhaustion is rare but extremely annoying when it hits, and it always hits at the worst time.