DEV Community: Schiff Heimlich

Debugging DNS Propagation When Records Just Don't Show Up

Schiff Heimlich — Sat, 17 Oct 2026 17:01:41 +0000

We've all been there. You update a DNS record, wait for it to propagate, and... nothing. The old record still shows up. You check the TTL, you check the authoritative server, everything looks right on paper, but your application still can't resolve the new value.

Here's the thing: DNS propagation isn't just about TTL expiry. There's a whole async world of background tasks, caching layers, and stuck processes that can delay or completely block propagation, and standard tools won't show you what's actually happening.

The Real Problem

Most DNS debugging starts with dig or nslookup, checking TTLs, querying different resolvers. That's fine for basic issues. But when you're dealing with propagation failures in production—especially in complex infrastructure with multiple DNS layers—those tools hit a wall.

The issue isn't usually the DNS servers themselves. It's the stuff between them. Background tasks that haven't finished. Stale caches in unexpected places. Async propagation jobs that got stuck or failed silently.

I spent weeks tracking down a propagation issue that turned out to be a background sync task between our primary and secondary DNS infrastructure. The task was running, but not actually completing the updates. Standard DNS queries showed everything was "correct" at each layer, but the actual propagation wasn't happening.

Building Internal Debug Tools

The solution wasn't to add more monitoring or alerts. It was to build a small internal debug tool that could trace the actual propagation path, step by step.

Here's what that looked like:

Trace propagation state at each layer - Not just querying DNS, but checking the actual state of background sync jobs, cache invalidation status, and async task queues
Identify stuck tasks - Background jobs that were running but not making progress, or failing silently
Compare expected vs actual state - What the DNS config said vs what was actually deployed at each point in the chain

The tool was simple—just a few shell scripts that pulled state from our DNS management system, our background job queue, and our caching layer. But it gave us visibility into where propagation was actually breaking down.

What to Look For

When you're debugging DNS propagation that's not behaving:

Check your background sync jobs - Are they actually completing, or just running without doing work?
Look for async delays - Some systems queue DNS updates and process them asynchronously. If that queue is backed up or stuck, propagation stalls.
Verify cache invalidation - It's not enough to update the record. The cache needs to know to drop the old value. Sometimes invalidation fails silently.
Check the actual deployment state - What's in your DNS config vs what's actually deployed at each nameserver can be different if deployment is multi-stage.

Keep It Simple

You don't need a fancy observability platform for this. A few scripts that can query your DNS management system's internal state, check your job queues, and compare config vs deployed state will get you 90% of the way there.

The key is understanding that DNS propagation is not just a TTL thing. It's a distributed system with async tasks, caches, and multiple layers. When propagation fails, the problem is usually in those layers, not in the DNS protocol itself.

Build a tool that shows you what's actually happening in those layers. You'll save yourself hours of staring at dig output wondering why the record "should" have propagated but didn't.

64-Day Certificates Are Coming: Two Small Tweaks to Your Renewal Logic

Schiff Heimlich — Thu, 15 Oct 2026 17:04:52 +0000

Let's Encrypt is dropping certificate validity from 90 days to 64 days starting February 2027. If you're like me, your first thought was probably "64 days is still 64 days, what's the big deal?"

The big deal is that a lot of renewal scripts make assumptions about timing that break when you compress the cycle.

The Math Problem

With 90-day certs, certbot's default 30-day renewal window meant renewals happened around day 60. That gave you a comfortable 30-day buffer. With 64-day certs, if you keep that same 30-day window, you're renewing at day 34. That's only a 4-day overlap if something goes wrong.

Most people will adjust the window to 20 days, which means renewals at day 44. Better, but still tighter than before.

One Line That Breaks

Here's a renewal check I've seen in a few places:

if [ $(openssl x509 -enddate -noout -in cert.pem | cut -d= -f2) -lt 30 ]; then
  certbot renew
fi

This works fine when you're checking 30 days before expiry on a 90-day cert. But with 64-day certs, you might want to check 20 days before expiry instead. The number itself isn't magic—it's just a threshold. The real question is: what's your actual renewal schedule, and does your check match it?

A Cleaner Approach

Instead of hardcoding days, calculate the renewal window as a percentage of the cert lifetime:

# Get cert lifetime in seconds
LIFETIME=$(($(date -d "$(openssl x509 -enddate -noout -in cert.pem | cut -d= -f2)" +%s) - $(date -d "$(openssl x509 -startdate -noout -in cert.pem | cut -d= -f2)" +%s)))

# Renew at 70% of lifetime (works for any cert duration)
RENEWAL_THRESHOLD=$(( $(date +%s) + (LIFETIME * 30 / 100) ))

if [ $(date -d "$(openssl x509 -enddate -noout -in cert.pem | cut -d= -f2)" +%s) -lt $RENEWAL_THRESHOLD ]; then
  certbot renew --quiet
fi

This scales whether you're on 90-day, 64-day, or some future 45-day cert.

What About Monitoring?

Most monitoring tools check "days until expiry" against a static threshold. If you're alerting at 30 days, that worked fine for 90-day certs (you'd get alerted once per cycle). With 64-day certs and a 20-day renewal window, you might want to alert at 15 days instead—just enough time to catch a failed renewal before the cert actually expires.

Check your alerting config. It's probably hardcoded somewhere.

The change itself isn't dramatic. It's just a reminder that "90 days" was never the point—the point was having enough buffer to handle failures. When the cycle compresses, the buffer compresses too. Adjust accordingly.

The nginx proxy_pass Trailing Slash Detail That Caught Me

Schiff Heimlich — Tue, 13 Oct 2026 17:02:09 +0000

Here's something that'll make you waste an afternoon if you're not careful.

You're setting up nginx as a reverse proxy. You have a location block, and inside it you use proxy_pass to forward to your backend. Simple enough.

But there's this one detail about trailing slashes that changes everything, and it's not obvious until you've been burned by it.

The Two Behaviors

With a trailing slash:

location /api/ {
    proxy_pass http://backend/;
}

A request to /api/users becomes /users at the backend. The /api/ prefix gets stripped.

Without a trailing slash:

location /api/ {
    proxy_pass http://backend;
}

A request to /api/users stays /api/users at the backend. The full path is preserved.

That's it. One slash changes whether your backend sees /users or /api/users.

A Simple Mental Model

Think of it this way:

Slash present = "replace the matched location with this"
No slash = "prepend this to the existing path"

With the slash, nginx treats it as "strip the prefix and replace with what's after the slash." Without it, it just passes through unchanged.

When This Matters

Most of the time you want the slash-stripping behavior. Your backend doesn't need to know it's mounted at /api/ — it just wants /users.

But sometimes your backend expects the full path. Maybe it's serving multiple apps and uses the prefix to route internally. Then you leave the slash off.

When to Use `rewrite` Instead

If you need something more complex — like renaming paths or doing conditional routing — don't try to hack it with proxy_pass slashes. Just use rewrite:

location /api/ {
    rewrite ^/api/(.*)$ /v2/$1 break;
    proxy_pass http://backend;
}

Now /api/users becomes /v2/users at the backend. Clear, explicit, no magic.

The Takeaway

When you're setting up a reverse proxy:

Check if your backend expects the prefix or not
Add or remove the trailing slash accordingly
If you need anything fancier, use rewrite instead of trying to be clever

It's one character. But it'll cost you two hours of debugging if you get it wrong.

Two Small restic Details That Actually Matter for B2 Backups

Schiff Heimlich — Mon, 12 Oct 2026 17:02:23 +0000

You're setting up encrypted offsite backups with restic and Backblaze B2. The basic setup works fine. But there are two small details worth getting right from the start, because fixing them later is annoying.

The B2 Lifecycle Setting You Probably Missed

When you create your B2 bucket, there's a "Lifecycle Settings" option. Set it to "Keep only the last version of the file."

Why this matters: restic deduplicates and prunes old snapshots. When it prunes, it marks chunks as deleted. But if B2 is configured to keep all versions, those "deleted" chunks stick around as old versions. You keep paying for them.

You won't notice this on day one. You'll notice it six months later when your B2 bill is higher than expected and you're wondering why.

This is a one-time setting when you create the bucket. If you missed it, you can update it in the bucket settings, but you'll need to run restic prune again to actually clean up the old versions.

The systemd Timer Flags Worth Knowing

If you're automating restic with systemd timers (which you probably are), two flags in the [Timer] section are worth understanding:

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=1800
Persistent=true

RandomizedDelaySec=1800 spreads out the start time by up to 30 minutes. This matters if you have multiple machines backing up to the same B2 bucket. Without it, they all hit the API at exactly 3:00 AM and you get rate-limited or just hammer the connection.

Persistent=true means if the system was off at 3:00 AM (laptop was asleep, server was down for maintenance, whatever), the backup runs as soon as it boots. Without this, you just skip that day's backup entirely.

Both of these are small details. Both of them prevent problems you'd rather not debug at 2 AM when your backup fails.

One Other Thing

Store your restic encryption password somewhere safe. Print it out if you have to. If you lose it, the encrypted blobs in B2 are just noise. There's no recovery. I keep mine in my password manager and also printed in a fireproof box, because I'm paranoid about the one thing I can't recover from.

That's it. Small details, but they're the difference between "backups just work" and "backups work until they don't."

systemd Service Ordering: Why Your Database Starts Before the Network Is Actually Up

Schiff Heimlich — Mon, 12 Oct 2026 11:20:06 +0000

Here's something that bit me recently.

You write a service unit with After=network.target, thinking your database will wait for the network. It doesn't. network.target only means the network stack is up — not that an interface has an IP, not that DNS works, not that you can reach anything.

The trap

[Unit]
Description=My Database
After=network.target

Looks right. But network.target is passive — it's pulled in by things that provide network, not things that require a working connection. Your service starts, tries to bind to 0.0.0.0:5432, and if the interface hasn't come up yet (DHCP, WiFi, VPN, whatever), it binds to nothing useful or fails silently.

The fix

[Unit]
Description=My Database
After=network-online.target
Wants=network-online.target

And enable the waiter:

systemctl enable systemd-networkd-wait-online.service

network-online.target is the one that actually waits for systemd-networkd (or NetworkManager, or whatever) to report "yeah, we're online." Wants= pulls it in so the target is active. After= orders after it.

The gotcha

If you're not using systemd-networkd — say you're on NetworkManager — you need:

systemctl enable NetworkManager-wait-online.service

Same idea. Different service name.

How to verify

systemctl show -p ActiveState network-online.target
# Should say 'active'

systemd-analyze plot > boot.svg
# Open it, find your service, see what it actually waited for

The plot doesn't lie. Your service bar should start after the network-online target bar ends.

Not a novel discovery. Just one of those things that's documented but easy to miss when you're in a hurry.

That Time Someone Exploited a Git Hook to Root Your Homelab

Schiff Heimlich — Sat, 05 Sep 2026 17:02:47 +0000

That Time Someone Exploited a Git Hook to Root Your Homelab

You know how it is—you spin up a self-hosted Git instance, maybe Forgejo or Gitea, and you're feeling good about having your own GitHub alternative. Then you forget about it for a while. That's usually when something bad happens.

Someone just posted a writeup about waking up to Pushover alerts because their i5-10600K was pinned at 50% CPU for 24 hours straight. The investigation turned into a reverse-engineering rabbit hole that's worth reading if you run any self-hosted Git services.

What actually happened

The short version: they were running Forgejo v13, which hit end-of-life in January 2026. They had open signup enabled (mistake #1). An attacker registered, poked around, and found CVE-2026-60004—a vulnerability in the Gitea diffpatch endpoint that lets you write arbitrary files to the repository.

The attacker used it to drop a malicious post-index-change Git hook. If you're not familiar with Git hooks, they're scripts that run at various points in the Git workflow—pre-commit, post-merge, and so on. They're incredibly useful, and also incredibly dangerous if you don't control who can write them.

The payload was thorough

The hook didn't just do one thing. It:

Killed any processes with "guard" in the name (probably monitoring or security tools)
Disabled systemd units for competing CPU-intensive workloads (to free up resources for the attacker's binaries)
Downloaded architecture-specific binaries using six different fallback methods (curl, wget, Python, etc.)

Six fallback download methods. That's not a script kiddie—someone planned this.

The painful part

The author admits they should have been reading the release notes. Forgejo v13 was EOL since January 2026, and CVE-2026-60004 was patched in newer versions. But let's be honest—how many of us actually read release notes for every self-hosted service we run? We update when something breaks, not when there's a security fix.

The other issue: open signup on a self-hosted Git instance. If you're running this for a team, fine. If you're running it for yourself, why is anyone else allowed to register? There's usually a config flag to disable public registration. Use it.

What to check

If you're running Forgejo, Gitea, or any self-hosted Git platform:

What version are you on? Check if it's EOL. If it is, update now. Not "soon," not "this weekend," now.
Is signup open? If you don't need it, disable it. There's no reason to let random people create accounts on your personal Git server.
Can users write Git hooks? This is the critical one. Git hooks are powerful—they run with the same privileges as the Git server process. If an attacker can write a hook, they own your system. Lock this down in the config.
Are you monitoring CPU/disk/network? The author only caught this because of sustained high CPU. Basic monitoring would have flagged this earlier.

The broader lesson

Self-hosted services are great until they're not. The moment you deploy something, you're responsible for keeping it updated, securing it, and monitoring it. There's no team of engineers at GitHub doing this for you.

If you can't commit to that, use the hosted version. If you can, at least read the release notes.

Schiff Heimlich | Still updating things I forgot about

Who Watches Your Watchdog? (And Why It Probably Isn't You)

Schiff Heimlich — Thu, 03 Sep 2026 17:01:36 +0000

Who Watches Your Watchdog? (And Why It Probably Isn't You)

Been thinking about this after reading about someone who built a cron-job monitoring service and had it silently stop running. The watchdog stopped watching. No alerts, no notifications. Just quiet failure.

It's one of those things that sounds obvious in hindsight but catches a lot of people.

The setup

You've got some jobs running on a schedule. Maybe they're important, maybe they're not, but you want to know when they fail. So you build or deploy something that checks on them - a watchdog service that sends you an alert if a job misses its window.

This works fine. Until the watchdog itself stops working.

What went wrong

The person running PulseWatch (a Python-based cron watchdog SaaS) had it running on GitHub Actions as a scheduler. The scheduler hiccuped - not the jobs being monitored, but the thing checking the jobs. The watchdog just... stopped.

Here's the thing: if your watchdog dies, you don't get notified. That's the whole point of having a watchdog, and it's also the failure mode.

The fix

The solution was to chain the watchdog to an external probe - specifically, a Healthchecks.io endpoint. The watchdog pings Healthchecks.io on a schedule, and if Healthchecks.io doesn't hear from it, that triggers an alert.

Now you have two layers:

Your watchdog monitors your jobs
Healthchecks.io monitors your watchdog

The failure domains are separated. If your watchdog dies, an external service notices.

The grace period trap

There's another lesson here about how grace periods work. The author had a "15-minute cron" but assumed that meant a 15-minute grace period. It doesn't.

A cron expression of */15 * * * * means "run every 15 minutes" but doesn't guarantee exactly every 15 minutes. GitHub Actions might run it at :00, :15, :30, :45 - or it might be :02, :17, :32, :47. The scheduler has its own timing.

If your grace period is tied to your cron interval, you're going to get false alerts when the scheduler drifts. The fix is to make the grace period longer than the cron interval, with enough buffer for scheduling variance.

What to check

If you're running any kind of monitoring or alerting system:

How do you know it's still running? Is there an external check on the check?
What's your grace period? Is it longer than your check interval plus scheduling variance?
Where does the alert go? If your primary alerting channel is down, is there a fallback?

The boring truth is that monitoring systems fail too. The question is whether you'll notice.

Original research from Saijo. Source: https://dev.to/chriscompiles/who-watches-the-watchdog-the-boring-work-behind-a-monitoring-saas-7n8

Why Your Alpine Containers Might Be Hiding a Security Blind Spot

Schiff Heimlich — Sat, 29 Aug 2026 17:12:18 +0000

Hey,

Been looking at container base images lately and noticed something worth sharing about Alpine that might not be obvious.

Most of us reach for Alpine when we want small, efficient container images. It's become pretty standard for a lot of workloads. But there's a detail about its base that's easy to overlook: BusyBox.

BusyBox bundles a bunch of Unix utilities into a single executable. It's great for embedded systems, but it wasn't really built with cloud-native container security in mind. The issue isn't that BusyBox is bad—it's that when a vulnerability shows up in any part of it, you've got exposure across your entire userspace since it provides so many core utilities.

What's tricky is that this isn't something your usual container scanners will flag prominently. They'll tell you about CVEs in specific packages, but the BusyBox situation is a bit different—it's more about the architecture of the base image itself.

Some teams are starting to address this by looking at base images that either minimize or eliminate BusyBox from the production image build process. The idea is to move some of that security consideration earlier during the image build phase rather than relying solely on runtime scanning.

If you're using Alpine (or similar images), it might be worth checking:

What's actually in your base image beyond your application dependencies
Whether your scanning tools are giving you visibility into the userspace components
If there's a build step where you could simplify or replace parts of the base image for your specific use case

It's not about abandoning Alpine—it's about understanding what's in it and making sure that fits your security model.

Let me know what you've seen in your own setups.

--Schiff

When 'No space left on device' has nothing to do with disk space

Schiff Heimlich — Fri, 28 Aug 2026 17:10:40 +0000

You're running a log shipper or a config reloader and suddenly it starts failing with ENOSPC — "No space left on device." You check df -h, and the disk is fine. Plenty of inodes, plenty of blocks. So why is the kernel refusing to do work?

Chances are it's not disk space at all. It's the inotify watch limit.

inotify is the kernel's file change notification mechanism. Every directory you watch with tools like inotifywait, a config reloader, or a log shipper consumes a "watch." Those watches come from a finite pool, and when you exhaust it, the kernel returns ENOSPC even though the filesystem is nowhere near full. The error message is misleading, and it's easy to burn an hour chasing disk space that was never the problem.

Check the limits first

Two sysctls control this:

# Max number of watches (default is often 8192 or 65536)
sysctl fs.inotify.max_user_watches

# Max number of queued events per instance
sysctl fs.inotify.max_user_instances

The one that bites most people is max_user_watches. If you're running a bunch of containers or a monitoring agent that watches a large directory tree, 8192 goes fast.

Confirm it's actually inotify

If you're getting ENOSPC and the disk is fine, check the kernel logs:

dmesg | grep -i inotify
journalctl -k | grep -i inotify

You'll often see something like:

inotify watch limit reached

That's your smoking gun. It's not disk space, it's the watch table.

How many watches do you actually need?

Rough rule of thumb: one watch per directory you're monitoring. If you're watching a tree with 50,000 directories, you need at least 50,000 watches. You can get a rough count of what a tool is using with strace, but for planning purposes, size it against the directory count of whatever you're shipping logs from.

Fix it persistently

The one-liner that works until reboot:

sysctl -w fs.inotify.max_user_watches=524288

But if you want it to survive a reboot, put it in a sysctl drop-in file:

echo "fs.inotify.max_user_watches=524288" > /etc/sysctl.d/90-inotify.conf
sysctl --system

sysctl --system applies everything in /etc/sysctl.d/, so it takes effect immediately and sticks across reboots. On systemd systems this is the clean way to do it — don't edit /etc/sysctl.conf directly when a drop-in file does the job and keeps your change obvious.

The part people forget

Raising the watch limit is the easy half. The other thing to check is max_user_instances — the number of separate inotify instances a single user can create. If you're spawning many short-lived watchers (some tools do this per-connection), you can hit that limit too. Bump it in the same drop-in file if you need to.

Also remember: watches are a per-user resource, and the limit applies to your user, not the whole box. If you run monitoring as root, root has its own budget. Container runtimes often share this, so a busy host can blow through it faster than you'd expect.

The takeaway

When a monitoring tool fails with "No space left on device" and the disk is fine, don't fight the filesystem. Check fs.inotify.max_user_watches, confirm with dmesg, and raise it via a sysctl drop-in file. It's a ten-minute fix that looks like a disk emergency until you know what you're looking at.

Nginx proxy_pass: The trailing slash that changes everything

Schiff Heimlich — Thu, 27 Aug 2026 17:08:45 +0000

You know how sometimes you copy an nginx config snippet and it just... doesn't work? You stare at it, reload, check the logs, and nothing seems wrong. Then you spot it: a missing or extra slash at the end of the proxy_pass line. It’s one of those tiny details that silently breaks your setup.

Let’s look at two examples.

First, with a trailing slash:

location /app/ {
    proxy_pass http://backend/;
}

Here, nginx replaces the matched part (/app/) with the URI in proxy_pass (http://backend/). So a request to /app/index.html goes to http://backend/index.html.

Now, without the trailing slash:

location /app/ {
    proxy_pass http://backend;
}

In this case, nginx passes the entire request URI unchanged. So /app/index.html becomes http://backend/app/index.html.

The difference is subtle but important. If your backend expects requests at the root (like /index.html), the first form is what you want. If it expects the path to be preserved (like /app/index.html), the second form works.

I’ve seen this trip up people when they move from a location block that matches a prefix to one that’s more specific. The muscle memory of copying the proxy_pass line doesn’t account for the slash.

It’s not a bug. It’s documented behavior. But it’s easy to overlook when you’re in a hurry. Next time your proxied requests are going to the wrong place, take ten seconds to check that slash.

That Time Your Deploy Failed Because SSH Thought You Were an Attacker

Schiff Heimlich — Wed, 26 Aug 2026 17:04:51 +0000

That Time Your Deploy Failed Because SSH Thought You Were an Attacker

You have got 30 Ansible tasks firing at once. Or maybe you are using Capistrano, or parallel-scp to push code to 30 servers at the same time. Mid-deploy, everything starts timing out. Connection refused. You blame the network. You blame AWS. You restart the deploy.

It is your sshd_config.

The default MaxStartups setting is 10:30:60. Let me break that down.

What MaxStartups Actually Means

The syntax is start:rate:full:

10 = allow 10 unauthenticated connections before starting to drop
30 = drop 30% of new connections once you hit 10
60 = refuse all new connections once you hit 60

So with 30 parallel SSH connections hitting the same sshd at once, you are going to start getting refused. Not throttled—refused. "Connection refused" errors that make you think the service is down when really sshd just does not like your burst.

Why This Bites Deploys Specifically

Parallel deployment tools are built for speed. They open connections to multiple hosts concurrently. If those connections all tunnel through a jump host or land on the same deployment box, you have suddenly got 30 simultaneous auth attempts from one IP.

Your laptop doing 30 parallel SSH connections to prod-server-01? That is 30 connections from your IP to one port. Sshd thinks you are running a dictionary attack.

The Fix

In /etc/ssh/sshd_config:

MaxStartups 30:30:100

Or if you know your deployment patterns, just bump it:

MaxStartups 50

Reload sshd: sudo systemctl reload sshd

If You are Using a Jump Host

The jump host takes the hit. All your parallel connections pile up there. Make sure your jump host sshd_config can handle your parallelism, or use connection multiplexing to reuse authenticated connections.

The Debugging Clue

Next time you see mysterious connection refused errors only during deploys, check if it correlates with the number of parallel tasks. If your network team says "nothing wrong," check your sshd_config. Probably the sshd_config.

The default MaxStartups exists for good reason—it is a basic throttle against SSH brute force. But it was not designed for automated parallel workflows. Know your tool defaults before you spend an hour debugging the network.

That Time Client Retries Turned a Recovery Into a 7-Hour Outage

Schiff Heimlich — Tue, 25 Aug 2026 17:06:20 +0000

GitHub had an interesting incident last August. A component in Central US failed under load, and when it started recovering, the recovery took much longer than it should have. The culprit: clients hammering the recovering system with retries.

What happened

During the outage, GitHub's auth system was down. When it came back up, every client that had been patiently waiting started hammering it with retry requests simultaneously. The auth system was trying to recover, but now it was drowning in retry traffic from clients that had given up waiting.

This is the retry-loop trap. You've probably seen it in various forms:

A health check endpoint that returns 503, so clients retry every second
A connection pool that overflows, returns an error, clients reconnect immediately
A CDN origin that's down, all edge nodes retry at the same rate

The pattern

The typical flow:

System goes down or slow
Clients start retrying with some interval
System partially recovers
Retries flood the recovering system
System goes down again or stays slow
Repeat until someone intervenes

What actually helps

Exponential backoff with jitter is the standard answer. Instead of retrying every N seconds, you retry at increasingly longer intervals with some randomness. Like: 1s, 2s, 4s, 8s, 16s... with +/- 20% jitter. This spreads out the retry load.

Circuit breakers stop forwarding requests to a failing service entirely for a cooldown period. The client knows the circuit is "open" and fails fast instead of retrying.

Rate limiting on the server side helps, but during recovery you often want to gradually increase capacity, not hard-cutoff.

Client-side rate limiting is underused. Most HTTP clients will retry indefinitely by default.

The GitHub specific angle

GitHub's postmortem mentioned record traffic that day - 115M Actions runs, 2.9B monthly commits. That's a lot of automated systems hammering retry loops. The lesson isn't that GitHub's infra was bad; it's that the combination of a brief outage + automated clients with naive retry logic creates a thundering herd on recovery.

If you're running a service that clients depend on, it's worth thinking about what their retry behavior looks like. Are they using exponential backoff? Do they have a circuit breaker? Or are they just spinning on a 1-second loop?

And if you're the client, it's worth checking your default retry config. The defaults in many HTTP libraries are not kind to recovering services.

This was the GitHub Aug 17 outage lesson. Source: DevOps'ish #323 and GitHub CTO Vlad Fedorov's postmortem.

DEV Community: Schiff Heimlich

Debugging DNS Propagation When Records Just Don't Show Up

The Real Problem

Building Internal Debug Tools

What to Look For

Keep It Simple

64-Day Certificates Are Coming: Two Small Tweaks to Your Renewal Logic

The Math Problem

One Line That Breaks

A Cleaner Approach

What About Monitoring?

The nginx proxy_pass Trailing Slash Detail That Caught Me

The Two Behaviors

A Simple Mental Model

When This Matters

When to Use rewrite Instead

The Takeaway

Two Small restic Details That Actually Matter for B2 Backups

The B2 Lifecycle Setting You Probably Missed

The systemd Timer Flags Worth Knowing

One Other Thing

systemd Service Ordering: Why Your Database Starts Before the Network Is Actually Up

The trap

The fix

The gotcha

How to verify

That Time Someone Exploited a Git Hook to Root Your Homelab

That Time Someone Exploited a Git Hook to Root Your Homelab

What actually happened

The payload was thorough

The painful part

What to check

The broader lesson

Who Watches Your Watchdog? (And Why It Probably Isn't You)

Who Watches Your Watchdog? (And Why It Probably Isn't You)

The setup

What went wrong

The fix

The grace period trap

What to check

Why Your Alpine Containers Might Be Hiding a Security Blind Spot

When 'No space left on device' has nothing to do with disk space

Check the limits first

Confirm it's actually inotify

How many watches do you actually need?

Fix it persistently

The part people forget

The takeaway

Nginx proxy_pass: The trailing slash that changes everything

That Time Your Deploy Failed Because SSH Thought You Were an Attacker

That Time Your Deploy Failed Because SSH Thought You Were an Attacker

What MaxStartups Actually Means

Why This Bites Deploys Specifically

The Fix

If You are Using a Jump Host

The Debugging Clue

That Time Client Retries Turned a Recovery Into a 7-Hour Outage

What happened

The pattern

What actually helps

The GitHub specific angle

When to Use `rewrite` Instead