DEV Community: Schiff Heimlich

Why logrotate silently does nothing and how to actually debug it

Schiff Heimlich — Sat, 15 Aug 2026 17:04:58 +0000

A config that doesn't error is not a config that works.

logrotate misconfigurations don't fail loudly. They fail silently. You only find out when /var/log is 100% full and something crashed.

The copytruncate race condition

The copytruncate directive exists because some processes don't respond to SIGHUP. Instead of telling the process to reopen its log, logrotate copies the file and truncates it in place.

Sounds reasonable. The problem is the gap between "copy" and "truncate." On a busy nginx server writing hundreds of lines per second, you're going to drop log entries. On a quieter system you might silently duplicate them.

/var/log/nginx/access.log {
    daily
    rotate 7
    compress
    copytruncate   # this is the problem on high-throughput services
}

The fix is the postrotate approach:

/var/log/nginx/access.log {
    daily
    rotate 7
    compress
    postrotate
        nginx -s reload
    endscript
}

Your process handles the log reopen cleanly. No race, no dropped lines.

The silent-failure trap

logrotate returning 0 doesn't mean anything rotated. A missing file, wrong path, permission issue, or a config directive that doesn't apply — all of these produce a clean exit code.

/usr/sbin/logrotate -d /etc/logrotate.conf

The -d flag runs logrotate in debug mode. It prints what it would do without doing it. Run this before shipping any config change.

One thing to know: -d still reads your actual state. If a file was already rotated today, the debug run won't show you what happens on a fresh invocation. Run it right after touching the config, not after the daily run already fired.

A quick sanity check for any config

/usr/sbin/logrotate -d /etc/logrotate.d/your-app 2>&1 | head -30

Look for:

"renaming" lines — confirms it found the target files
"empty log" warnings — the source file is missing or zero-size
"error" strings — permission or path issues

If you see "log needs rotating" but no "renaming", you hit the maxsize/rotations-per-day guard and the file was skipped intentionally.

What actually fills disks in practice

The most common real-world cause isn't a clever attack or misbehaving application. It's a logrotate config that never had compress on a high-volume file, and nobody noticed for three months because the disk monitor didn't alert on /var.

The second most common: a process that held a file descriptor open after logrotate moved the file. The rotated file stays on disk until the process restarts. lsof +L1 catches this quickly.

That's it. No magic. Check your postrotate hooks, run logrotate -d before shipping configs, and add lsof +L1 to your disk-full checklist.

One-Command VPS Reinstall: Two Things Worth Knowing

Schiff Heimlich — Fri, 14 Aug 2026 17:04:30 +0000

One-Command VPS Reinstall: Two Things Worth Knowing

There's a tool called restage that reinstalls a Linux VPS over the network with one command. Saw it mentioned last week and finally tried it. Two things I found actually useful:

1. The Dry-Run Is Real

Most "idempotency" claims in shell scripts are aspirational. With restage, --dry-run actually shows you the entire action plan before touching anything:

restage linux debian 12 --ssh-key "ssh-ed25519 AAAA..." --dry-run

It prints: target disk (by PTUUID, not device name), OS image it's pulling, boot entry it's writing, SSH settings. If the plan looks wrong, you Ctrl-C and nothing happened.

The paranoia here is warranted. A mistyped device name in a reinstall script can ruin your day.

2. Disk Selection by Partition Table UUID

Device names drift between boots on some providers. /dev/sda becomes /dev/vda, or worse, /dev/sdc. Restage selects the target disk using the partition-table UUID (PTUUID), which is stable and queryable:

blkid -s PTTYPE -o value /dev/sda

The tool captures this during planning and references it in the installer, so device renaming between the dry-run and the actual reinstall doesn't break anything.

The Catch

This still erases the disk. The dry-run protects against typos, not against choosing the wrong machine. Know which node you're on before you run it.

If you manage a handful of VPSes and keep putting off that Ubuntu-to-Debian migration because the control panel workflow is annoying, this is worth a look. The MIT-licensed toolkit is at github.com/codebyjawad/restage.

Why BusyBox in Your Alpine Containers Is a Bigger Problem Than You Think

Schiff Heimlich — Thu, 13 Aug 2026 17:06:34 +0000

When you pull an Alpine image, you are probably not thinking about what is actually running inside. Most teams see minimal and move on.

Here is the thing: Alpine Linux is built on BusyBox. One binary. Everything else links to it.

docker run --rm alpine:latest ls -la /bin/ | head -10
# lrwxrwxrwx 1 root root 12 /bin/ls -> /bin/busybox
# lrwxrwxrwx 1 root root 12 /bin/cat -> /bin/busybox
# lrwxrwxrwx 1 root root 12 /bin/cp -> /bin/busybox

Every utility you see is the same 1.3MB binary. That is your entire operating system.

The CVE Problem

When BusyBox has a vulnerability, your entire userspace has it. There is no compartmentalization.

Take CVE-2022-28391. It affected BusyBox is DHCP client. Any Alpine container acting as a DHCP client was vulnerable to remote code execution via a malicious DHCP response. This covered Kubernetes pods, CI runners, and network sidecars across the industry.

The fix most teams did: docker pull alpine:latest. But that does not guarantee you are on a patched version.

# Check BusyBox version
docker run --rm alpine:3.19 busybox | head -1

# Verify against security.alpinelinux.org

What You Can Do

1. Audit what you are actually running

Find Alpine images in your cluster and check for BusyBox presence.

2. Consider distroless for application containers

For Go or Rust services that compile statically, use distroless images. No shell, no package manager, no BusyBox. The image is slightly larger but your attack surface is smaller.

3. If you must use Alpine, pin and automate rebuilds

Use specific version tags like alpine:3.19 instead of :latest. Set up Renovate or Dependabot to watch Alpine release tags and rebuild when CVEs drop.

The Honest Tradeoff

Alpine is convenient. Distroless requires more build work. But if you are running internet-facing workloads or anything with elevated privileges, the convenience is not worth the invisible attack surface.

Go check your base images. See how many are Alpine. Then decide if you want to keep playing that game.

The logrotate copytruncate race condition that silently drops logs

Schiff Heimlich — Wed, 12 Aug 2026 17:03:29 +0000

The problem with copytruncate

If you've ever configured logrotate for a high-throughput service like nginx or syslog, you might have used copytruncate. It copies the log file then truncates it in place, avoiding the need to signal the process to reopen its file descriptors.

The issue: there's a window between the copy and the truncate. Any log lines written during that window get either dropped or duplicated when rotation runs on a busy system.

What it looks like in practice

# /etc/logrotate.d/nginx
/var/log/nginx/access.log {
    daily
    rotate 7
    copytruncate
    compress
}

On a server doing high request volume, that copy-then-truncate gap can silently eat log lines per rotation. If you're doing incident response and wondering why your access logs don't match your application logs, this is one of the places to check.

The more reliable approach

Use postrotate to signal the service to reopen its logs, then let logrotate do an atomic rename:

/var/log/nginx/access.log {
    daily
    rotate 7
    compress
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

nginx (and most well-behaved services) will finish writing the current line, then reopen to the new file on SIGUSR1. No gap, no dupes.

Another gotcha: dateext collisions

If you're using dateext with dateformat, watch out for rotation runs that trigger twice in the same second (can happen with noisy cron or manual runs). The second rotation will fail to create a unique file. Set dateformat with enough precision:

dateformat -%Y%m%d-%s

The %s gives you Unix timestamp seconds, which keeps it unique even if cron fires twice.

Bottom line

copytruncate is a workaround for services that don't handle SIGUSR1 gracefully. If your service supports log reopening, use postrotate. If you're stuck with copytruncate, be aware it has a silent data loss window on busy systems.

Bare-Metal Kubernetes: What NKP Metal Gets Right

Schiff Heimlich — Wed, 12 Aug 2026 06:25:44 +0000

Bare-Metal Kubernetes: What NKP Metal Gets Right

Here's something I ran into that might be worth a second look.

Nutanix launched NKP Metal recently - basically their Kubernetes Platform extended to run directly on physical servers. I think the interesting part isn't the bare-metal capability itself, but what it means for operational consistency.

The practical bit

If you're running AI/ML workloads or dealing with edge deployments, bare-metal matters for one reason: you avoid the hypervisor layer. GPU passthrough works cleaner when there's nothing between the hardware and your workload. No hypervisor means no resource contention from the host OS stealing CPU cycles or memory.

The other thing worth noting is the unified management angle. Same control plane whether you're running on VMs or physical hardware. That means one less management interface to switch between, one less place to check when something goes wrong.

When this actually helps

This isn't a "bare-metal is always better" take. For most workloads, VMs are still the right answer - easier to snapshot, migrate, resize.

But if you're doing GPU-heavy inference at the edge, or you've got hardware constraints that make virtualization impractical, bare-metal Kubernetes becomes relevant. And if you're already in the Nutanix ecosystem, having both under the same console means less context-switching.

The takeaway

The interesting part of NKP Metal isn't the bare-metal capability in isolation. It's the operational consistency - same API, same tooling, same management plane whether the underlying host is a VM or a physical server. That's the practical value for teams already running Nutanix.

If you're evaluating Kubernetes infrastructure options and have GPU workloads that need direct hardware access, it's worth a look. If your workloads fit comfortably in VMs, this doesn't change much.

Schiff Heimlich | Sysadmin who pays attention to operational simplicity

Your Internal Developer Platform Is Probably Ignoring Your Java Services

Schiff Heimlich — Tue, 04 Aug 2026 17:04:16 +0000

Your Internal Developer Platform Is Probably Ignoring Your Java Services

Here's something I keep seeing in platform engineering work.

You build an Internal Developer Platform. You want self-service for everyone. Kubernetes, CI templates, deployment pipelines - all unified under one platform.

Then your Java teams start filing tickets. Something's off. Deploys take longer than expected. Memory usage is higher than it should be. The platform team says "it works for everyone else."

The problem

Most IDPs are built by people who primarily work with Go, Python, or Node. Those are straightforward - compile or interpret, run as a process, resource requests are basically "give me some RAM."

Java doesn't work that way. A Spring Boot microservice has JVM heap tuning, garbage collector settings, classpath management, Maven or Gradle build quirks. It needs a sidecar for metrics because the JVM exposes different telemetry than a native process. When you treat a 512MB JVM pod the same as a 50MB Go binary, you're already wrong.

What generic IDPs get wrong

The platform gives you a deployment template. You fill in your image, your CPU request, your memory limit. Done.

But for a JVM service, the memory limit isn't just "how much RAM does the process use." It's "what does -Xmx need to be relative to the container limit." If you set container memory to 1GB and don't tune the JVM, the runtime might only use 256MB heap and spend the rest in metaspace, native memory, and GC overhead.

Same with sidecars. Java services with Spring Boot typically need specific sidecar configurations - actuator endpoints for health, Micrometer for metrics, language-specific logging patterns. Generic sidecar injection assumes a process that behaves like everything else.

What helps

Java-aware IDP templates account for this:

JVM flags tied to container limits (e.g. -XX:MaxRAMPercentage instead of fixed -Xmx)
Health endpoints that actually work with Java (not just "does the process exist")
Build tool awareness - knowing whether you're using Maven wrapper or Gradle, how the artifact gets built
GC tuning profiles that match the workload type

If you're running Spring Boot on Kubernetes and your platform team doesn't know what -XX:+UseG1GC does, that's a gap worth filling.

The practical point

IDPs work best when they know what they're deploying. A platform that treats all services equally is a platform that works equally poorly for anything with specific runtime requirements. Java's got those. So does anything with native dependencies, or GPU workloads, or stateful services.

The answer isn't to build a separate platform for Java. It's to make sure whoever owns the IDP understands the actual workloads running on it.

Schiff Heimlich | Sysadmin who reads the JVM flags

SSH Key Permissions: The 5-Minute Fix That Actually Works

Schiff Heimlich — Mon, 03 Aug 2026 17:07:01 +0000

SSH Key Permissions: The 5-Minute Fix That Actually Works

Here's a fun one that bites you at 2am.

User calls: "I added a new key to authorized_keys but it still won't work." You check the file — it's there, permissions look fine. You cat ~/.ssh/authorized_keys and the key looks correct. You ssh -v and... nothing useful.

Nine times out of ten, the problem isn't the key. It's the permissions on something upstream.

The Exact Permission Matrix

OpenSSH is picky about permissions. Not politely suggestive — actively hostile. It will silently ignore your authorized_keys file if any of these are wrong:

Home directory:      755 or stricter (no 777, no 775 group-writable)
.ssh directory:     700 (only the owner can access it)
authorized_keys:    600 (only the owner can write it)

That's it. Those are the three you need to check.

How to Verify Quickly

The sshd -T command does a dry-run of your sshd config and prints out the effective settings. But more useful: you can test a specific user's key authentication with:

sshd -T -C user=username -C host=hostname -C laddr=0.0.0.0 -C lport=22

But for a quick local check on the permission issues themselves, just use namei:

namei -l ~/.ssh/authorized_keys

This shows you the permission bits on every component of the path. If you see anything like drwxrwxr-x instead of drwxr-x---, that's your culprit.

The chmod Commands (When You're Fixing It)

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 500 ~  # if needed, though 755 usually works

Don't do chmod 777 on anything. I know it's tempting when you're tired. Don't.

One Gotcha Worth Knowing

If the user's home directory itself is group-writable or world-writable, OpenSSH will still refuse to trust the .ssh directory. So even if your .ssh and authorized_keys are perfect, a sloppy home directory breaks everything.

# Check this too
stat -c "%a %n" ~

If it returns anything other than something like 755 or 700, fix it first.

The Payoff

Once permissions are right, the key works. No daemon restart needed. No config change. Just permissions.

It's one of those things that's obvious in retrospect but easy to overlook when you're staring at a correct-looking authorized_keys file wondering why SSH keeps asking for a password.

Tags: sysadmin, ssh, security, devops

When Your Script Dies with EPIPE: The Bug Is Two Functions Upstream

Schiff Heimlich — Sun, 02 Aug 2026 17:05:03 +0000

You're staring at a Python script that pipes output to grep or awk or any filter. It works fine in your shell. It fails in CI with a traceback pointing to a write call, something like:

OSError: [Errno 32] Broken pipe

The instinct is to look at line 47 where the write happens. The instinct is wrong.

What Actually Happens

The EPIPE error fires at the writer because the reader closed its end of the pipe first. The reader in your pipeline — grep, head, awk, whatever — reached its input limit and exited. Your script kept writing. The kernel said "no more readers, stop."

The bug isn't at the write. The bug is upstream in two ways:

The reader has a finite consumption rate. head -n 100 closes its end after 100 lines. grep exits on first match with -m 1. awk might finish a pattern and close. Your writer doesn't know this is coming.
Buffering lies to you. Pipes have a kernel buffer — typically 64KB on Linux. Your writer fills that buffer, then blocks. Meanwhile the reader is already gone. When the buffer drains, the writer wakes up to a closed pipe and gets EPIPE.

The Fix: Handle the Reader's Lifecycle

import subprocess
import signal
import os

# Option 1: Ignore EPIPE and let the reader drive
proc = subprocess.Popen(['awk', 'NR>1{print}', 'data.csv'], 
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# Your script's logic here
for chunk in iter(lambda: sys.stdin.read(8192), ''):
    if proc.poll() is not None:
        break  # reader exited, stop writing
    proc.stdin.write(chunk.encode())

# Option 2: in bash, handle it explicitly
set -o pipefail
grep pattern large-file.txt | awk '{print $2}' || {
    ret=$?
    if [[ $ret -eq 141 ]]; then
        # 141 = 128 + 13 (SIGPIPE)
        # Reader exited before writer finished - this is OK
        :
    fi
}

The set -o pipefail approach isn't enough by itself — you still need to catch the case where the writer gets EPIPE before the pipeline overall exits.

The Debugging Rule

When you get a broken pipe error, look at the downstream command in your pipeline first. Ask:

Does it have an input limit? (head -n, grep -m, tail -n)
Does it exit early on some condition?
What happens if input is shorter than expected?

The traceback points at where the symptom is, not where the cause is.

Practical Check

If you're building pipelines in scripts, add this before you blame the writer:

# Check if the downstream tool exits early
timeout 5 grep -m 1 pattern file.txt && echo "grep exited cleanly"

If it exits before your writer is done, you found your culprit.

This comes up often enough that it's worth building the habit: EPIPE is a reader problem, not a writer problem.

Your Alpine Base Image Has BusyBox. That's Probably Fine Until It's Not.

Schiff Heimlich — Sat, 01 Aug 2026 17:05:36 +0000

Let's talk about something that shows up in most container Dockerfiles without anyone really thinking about it: BusyBox.

If you're using Alpine as your base image, you're pulling in BusyBox. It's in BusyBox-based images, MiniCoI, and various other minimal Linux distros designed for containers. It's small, it's handy, and it's been there forever.

Here's the thing though — BusyBox was never designed for cloud production environments. It was designed for embedded systems. That's a meaningful difference.

What BusyBox actually is

BusyBox is a single binary that bundles implementations of dozens of Unix utilities: ls, cat, cp, sh, and so on. The idea is you get a working userspace in a tiny package. For Alpine, it's what makes ash (the shell) and basic utilities work.

The problem isn't that BusyBox is bad software. It's that it's a single point of exposure. When BusyBox has a vulnerability — and it does get them, like any software — your entire userspace is affected. Runtime scanners catch some of this, but the damage is already baked into your image layers.

The layered problem

When you build from Alpine, you're not just getting your app. You're getting:

The Alpine base (BusyBox)
Your package manager additions
Your application layers

If the base layer has a vulnerability, rebuilding your app layer doesn't fix it. You have to rebuild from a patched base, which means your whole image stack changes.

This is the "compatibility" problem that CleanStart and similar approaches are trying to solve at the build level rather than the runtime level. Make the base composition a policy decision, not an accident.

What you can actually do today

If you're using Alpine images:

Pin your base image versions explicitly, don't use latest
Track CVEs for BusyBox specifically, not just your app dependencies
Consider whether your production containers actually need a full shell and utilities — distroless or scratch images may be more appropriate if you're just running a single process

This isn't a crisis. Most teams are running fine with Alpine. But it's worth knowing what's actually in your base image and making that an intentional choice rather than a default you inherited.

Just a practical observation, not a scare piece. BusyBox has a good security track record — but so did a lot of things until they didn't.

set -e Will Kill Your Script For No Good Reason

Schiff Heimlich — Thu, 30 Jul 2026 17:06:15 +0000

`set -e` Will Kill Your Script For No Good Reason

You add set -e to your bash script because you want it to stop on errors. Reasonable expectation. What could go wrong?

Plenty.

The Problem

set -e makes your script exit immediately when any command returns non-zero. Sounds fine until you remember how many things legitimately return non-zero:

grep -q "pattern" file    # returns 1 when no match found
curl -s https://example.com  # returns non-zero on 4xx/5xx
(( counter > 5 ))          # returns 1 when false

None of those are errors. But set -e doesn't care.

Real Example

#!/bin/bash
set -e

# This fails silently when the string isn't found
name=$(grep "^username=" /etc/app/config | cut -d= -f2)

# This fails when count is zero  
if (( count > 0 )); then
    process_items
fi

The grep returns 1 when there's no match. set -e kills the script. Same with arithmetic that's false.

The Workarounds

Option 1: Disable locally

set -e
grep -q "pattern" file || true   # always succeeds

Option 2: Toggle around suspect code

set +e
grep -q "pattern" file
result=$?
set -e

Option 3: Check explicitly

if grep -q "pattern" file; then
    name=$(grep "^username=" /etc/app/config | cut -d= -f2)
fi

The Smarter Default

Skip set -e entirely. Check your commands explicitly:

if ! grep -q "pattern" file; then
    echo "pattern not found" >&2
    exit 1
fi

More verbose, but you control exactly what fails and why.

Why This Bites You At 3am

Because the script worked for months, then one day curl starts returning 404 for a URL that moved, and your whole deploy stops. The actual failure is trivial, but set -e treats it like the world ended.

It's not your logic that's broken. It's set -e being overprotective.

This isn't a reason to never use set -e. Just don't use it without understanding what it actually checks. Most shell script bugs I debug come down to this flag surprising someone at the worst moment.

The One-Line Audit That Might Save You a Container Incident

Schiff Heimlich — Wed, 29 Jul 2026 17:05:43 +0000

The One-Line Audit That Might Save You a Container Incident

Here's something I check whenever I'm reviewing a new container image.

docker run --rm [image] which busybox

If it returns a path, you're running BusyBox. And when BusyBox has a vulnerability, your entire userspace has that vulnerability.

Alpine Linux is the most common case. It's the default base for a lot of Helm charts and CI templates. The selling point is "minimal" and "secure." What you're actually running is BusyBox with a package manager attached.

The issue isn't that Alpine is bad. It's that BusyBox was designed for embedded systems - single-purpose devices that run one thing forever. When you put it in a multi-tenant cloud environment, you're trusting that the same binary handles sh, ls, grep, wget, and everything else in your container correctly.

One CVE, your whole userspace is exposed.

The fix isn't to stop using Alpine. It's to know what you're running and factor that into your vulnerability scanning. Most scanners catch CVEs in Alpine packages. They don't always catch CVEs in BusyBox itself.

Check your base images. Know what's inside them. Two minutes of auditing might save you an incident.

Schiff Heimlich | Sysadmin who checks the obvious things

Why Your rsyslog Config Is Quietly Breaking (And How to Fix It)

Schiff Heimlich — Tue, 28 Jul 2026 17:04:55 +0000

Why Your rsyslog Config Is Quietly Breaking (And How to Fix It)

RFC 6587 has been sitting there for years. Most of us didn't notice.

Here's what actually happened: you have rsyslog configs from 2012 that work fine on UDP port 514. Then one day someone asks "can you send these logs to our SIEM over TCP?" and you spend three hours wondering why connections keep timing out.

The answer is usually buried in how rsyslog parses messages.

The %TIMESTAMP% Problem

Old rsyslog configs often use TraditionalForward mode, which expects a BSD syslog-style timestamp at a fixed offset in the message. RFC 6587 changed how syslog over TCP works — it uses octet-counting framing (each message is prefixed with its byte count) rather than newline-delimited messages.

If you do this:

action(type="omfwd" protocol="tcp" target="siem.example.com" port="514")

...and your config still has %TIMESTAMP% at a fixed column position, the receiving SIEM (especially one expecting RFC 6587 compliant input) will read garbage.

The fix in rsyslog is straightforward — enable RFC 6587 mode:

action(type="omfwd" protocol="tcp" target="siem.example.com" port="514"
       streaming="on")

That streaming="on" switches to octet-counting. But that's only half the problem.

Structured Logging and the Legacy Template Problem

Compliance auditors are now asking about RFC 5848 (signed log streams) and structured logging (JSON-formatted messages). If your rsyslog templates still use:

$template TraditionalFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"

...you're not sending structured data. You're sending text that a human can read but a SIEM has to parse with regex.

The actual migration path most teams end up on is: keep rsyslog running for the legacy stuff, but bridge it to journald which handles structured logging natively.

The Bridge Setup

On a modern RHEL/Fedora box, syslog-ng sits between rsyslog and journald nicely:

# syslog-ng.conf — receive from legacy rsyslog
source s_net { tcp(port(514) flags(expect-octet-counting())); };

# Forward to journald in structured form
destination d_journal { pipe("/run/systemd/journal/syslog"); };

log { source(s_net); destination(d_journal); };

Then journald handles the structured part — you get JSON via journalctl --output=json, and you can sign streams with journalctl --setup-keys if you need RFC 5848 compliance.

What About Just Replacing rsyslog?

You can. systemd-journald alone is fine for local logging. But most enterprise environments have network devices and appliances that only know how to send traditional BSD syslog. You need something listening on UDP 514 for those.

syslog-ng Premium Edition handles the RFC 6587 stuff properly and has good template support. The community edition works too, just with slightly different syntax.

The Real Lesson

The configs from 2012 worked because the world was simpler — UDP, single SIEM, no compliance requirements. That world is gone. The fix isn't dramatic, but you need to understand what RFC 6587 actually changed about message framing before you can debug why your SIEM is getting half-formed messages at 2am.

Check your rsyslog version (rsyslogd -v) and look at whether your templates assume fixed-width fields. That's usually where things break.