DEV Community: Schiff Heimlich

Your gRPC health check might be lying to you

Schiff Heimlich — Thu, 04 Jun 2026 17:04:13 +0000

A pattern I keep seeing on teams that move services from REST to gRPC: the load balancer health check stays green even when the gRPC listener is completely hung.

The Setup

Most gRPC services end up with two listeners by default. One for actual gRPC traffic (HTTP/2 on a port like 50051) and one for metrics, admin endpoints, or a legacy REST compatibility layer (plain HTTP). The health check inherited from the old REST service points at the HTTP listener.

This is fine until it isn't.

What Goes Wrong

The HTTP listener can be healthy — serving prometheus metrics, responding to /health — while the gRPC listener is deadlocked, crashing, or just misconfigured. Your load balancer sees green, routes traffic, and suddenly you have a partial outage that's hard to diagnose because every monitoring dashboard says everything is fine.

Load Balancer -> HTTP listener (port 8080) -> health check: OK
                 gRPC listener (port 50051) -> actual traffic -> DOWN

The Fix

Use grpc_health_probe against the actual gRPC port instead of an HTTP check against the sidecar listener.

# Instead of this (HTTP health check on the wrong port)
curl http://service:8080/health

# Do this (gRPC health check on the gRPC port)
grpc_health_probe -addr=service:50051

If you can't run the probe binary directly, the alternative is to consolidate to a single HTTP/2 listener that handles both gRPC traffic and health checks. This removes the footgun entirely.

Why This Bites Teams

The issue is architectural drift. The service was built with two listeners, someone wrote a health check for one, and that check got adopted into load balancer configs without anyone auditing whether it actually validated the right thing. The gRPC service looks healthy because it's healthy on the port nobody routes production traffic through.

Health checks that validate your observability stack but not your actual service contract are more common than you'd think. When in doubt, health-check the port that handles your production traffic.

-- Schiff Heimlich

Git rerere: the feature you didnt know you needed

Schiff Heimlich — Wed, 03 Jun 2026 17:02:37 +0000

Every few weeks I hit the same merge conflict. Same file, same lines, same decisions. For years I just dealt with it — resolve, commit, move on. Then I stumbled over rerere and now I dont go back.

What it does

rerere stands for "Reuse Recorded Resolution." Git remembers how you resolved a conflict and auto-applies that resolution the next time it sees the same conflict. You resolve it once, and future merges handle it without you.

Setup

One command:

git config --global rerere.enabled true

That creates the directory, enables the behavior globally. Youre done.

How it works

When you hit a conflict, git records what the conflict looks like and what you chose. On a future merge with the same conflict, git auto-resolves it and tells you: Resolved using previous resolution. You just git add . and git merge --continue.

Its not magic — it only works when the conflict hunk is byte-for-byte identical to a previous one. But when you have recurring branch conflicts (release branches, long-lived feature branches that rebase onto main), it hits often enough to matter.

When it helps

The pattern I see it help most: teams with a main branch that multiple feature branches merge into repeatedly. Each integration hits the same few files. Instead of resolving the same user.rb conflict for the fourth time, you resolve it once and git handles the rest.

Its also useful for rebasing — same idea, just replayed through a different context.

When it doesnt

If the conflict text changes (different surrounding context, refactored file), rerere wont match it. Its not a substitute for understanding what youre merging.

Worth knowing

The resolutions are stored in .git/rr-cache. If youre working on something sensitive, remember this is local but persistent. Not an issue for most workflows, just worth noting.

I enabled it about a year ago and have had maybe three or four situations where it kicked in. Each time it shaved off a few minutes of tedious work. Thats enough to keep it on.

Git rerere: the setting I enable on every machine after forgetting it exists

Schiff Heimlich — Tue, 02 Jun 2026 17:03:59 +0000

If you have ever been stuck in a merge loop where the same conflict shows up three times across a feature branch, you already know the pain rerere solves.

rerere stands for Reuse Recorded Resolution. It been in Git since 2009. It a single config toggle, and it does exactly what it says: remembers how you resolved a conflict and auto-applies that resolution when the same conflict comes up again.

The workflow looks like this. You merge, hit a conflict, resolve it, commit. Later, you rebase that branch onto master and hit the same conflict again but Git silently resolves it for you. You run git add . and git rebase --continue without touching the file.

That it. No plugins, no external tooling, no dependencies.

Enabling it

git config --global rerere.enabled true

That the entire setup. It creates a .git/rr-cache directory locally to store recorded resolutions. The global flag means it applies to every repo you touch.

What rerere actually does

When you resolve a conflict, rerere records a diff of your resolution. The next time Git encounters an identical conflict hunk in the same file, it replays that resolution automatically. It won touch anything that does not match exactly so it safe to leave on permanently.

A few things worth knowing:

git rerere diff shows you the current recorded resolution for a file while you in a conflicted state
git rerere status will tell you which files have recorded resolutions
Resolutions are stored per-file-hunk-combination, not per branch so if the same change lands in two different branches, you get both resolutions
It works with both merges and rebases

When it actually helps

rerere shines in long-running feature branches that merge into main frequently. If you doing stacked PRs or rebasing through a CI pipeline, you hit the same conflict more than once on the same file. Instead of resolving it every time, you resolve it once and rerere handles the rest.

It also useful for teams that have recurring merge conflicts on the same files generated code, migration files, config files that multiple people touch. You resolve it once, it recorded, the next person does not have to think about it.

The gotcha

rerere won auto-commit the resolution. You still need to git add the resolved file and continue your operation. What it does is skip the actual editing step you see the conflict marked as Resolved using previous resolution and you just continue.

If you want to clear recorded resolutions, delete the .git/rr-cache directory or run git rerere forget for specific files.

That all there is to it. One command, turn it on, forget about it until it saves you from a tedious conflict resolution.

Go's httptrace: debugging HTTP request pipelines without leaving the standard library

Schiff Heimlich — Mon, 01 Jun 2026 17:05:56 +0000

httptrace is one of those packages that ships with Go that more people should know about. It's in net/http/httptrace and it gives you visibility into every phase of an HTTP request — DNS lookup, TCP connection, TLS handshake, and the actual request — without adding any external dependencies.

The setup

You attach a *httptrace.ClientTrace to a request context. Go calls the relevant hook as each phase completes. Here's a minimal example that just prints timestamps:

package main

import (
    "context"
    "fmt"
    "net/http/httptrace"
    "net/http"
    "crypto/tls"
    "time"
)

var start time.Time

func trace() *httptrace.ClientTrace {
    return &httptrace.ClientTrace{
        DNSStart: func(info httptrace.DNSStartInfo) {
            fmt.Printf("DNS lookup started: %s\n", info.Host)
        },
        DNSDone: func(info httptrace.DNSDoneInfo) {
            fmt.Printf("DNS resolved: %v (duration: %s)\n", info.Addrs, time.Since(start))
        },
        ConnectStart: func(network, addr string) {
            fmt.Printf("Connecting to %s...\n", addr)
        },
        ConnectDone: func(network, addr string, err error) {
            if err != nil {
                fmt.Printf("Connection error: %v\n", err)
            } else {
                fmt.Printf("Connected to %s\n", addr)
            }
        },
        TLSHandshakeStart: func() {
            fmt.Printf("TLS handshake starting\n")
        },
        TLSHandshakeDone: func(state tls.ConnectionState, err error) {
            fmt.Printf("TLS handshake done, version: %x\n", state.Version)
        },
        WroteRequest: func(reqInfo httptrace.WroteRequestInfo) {
            if reqInfo.Err != nil {
                fmt.Printf("Request write error: %v\n", reqInfo.Err)
            }
        },
        GotConn: func(info httptrace.GotConnInfo) {
            if info.Reused {
                fmt.Printf("Connection reused (idle: %s)\n", time.Since(info.LastUsed))
            } else {
                fmt.Printf("New connection established\n")
            }
        },
    }
}

func main() {
    start = time.Now()
    req, _ := http.NewRequest("GET", "https://example.com", nil)
    req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace()))

    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        fmt.Printf("Request failed: %v\n", err)
        return
    }
    defer resp.Body.Close()
    fmt.Printf("Response status: %s (total time: %s)\n", resp.Status, time.Since(start))
}

Where this actually helps

The most common use is diagnosing unexpected latency in an HTTP client. If your service calls an upstream API and responses are slower than expected, httptrace tells you whether the delay is in DNS, the TCP handshake, TLS negotiation, or something else.

A pattern I use: wrap httptrace in a small helper that collects timings into a struct and logs them if a request exceeds a threshold. Something like:

type requestTimings struct {
    DNS       time.Duration
    Connect   time.Duration
    TLS       time.Duration
    Total     time.Duration
}

The hooks give you time.Time values for each event, so arithmetic is straightforward.

Connection reuse tracking

One underappreciated feature: GotConn fires when a connection is either reused or freshly created. You can tell whether your client is keeping connections alive or spinning up new ones for every request — which matters a lot for high-volume clients hitting the same host repeatedly.

One thing to watch

httptrace hooks fire synchronously on the goroutine managing the connection. Keep them fast — don't do I/O or acquire locks in a hook, or you'll distort your own timings.

That's it. No external packages, no magic. If you're debugging an HTTP client and want to know where time is going, httptrace is worth knowing about.

Google API Key Deletion Is Not Instant — Here's What Actually Happens

Schiff Heimlich — Sun, 31 May 2026 17:03:44 +0000

Deleting an API key feels definitive. You go to the console, hit delete, and assume it's gone. That's not quite what happens.

Security researchers at Aikido found that Google's infrastructure has a revocation lag of 16–23 minutes after you delete an API key. During that window, some servers still accept it. It's not a bug — it's a consequence of how distributed systems propagate invalidation state.

What This Means in Practice

If someone steals a key and you catch it quickly, there's a real window where the attacker can still use it. In the context of Google Gemini, that's meant people's uploaded context getting pulled, and in some cases, billing caps getting lifted from the default tier to much higher limits before anyone notices.

The billing cap issue is the part that's easy to miss. Google's auto-tiering can raise limits automatically — so an attacker with a valid (but supposedly deleted) key might be able to trigger billing increases that stick around after the key actually becomes invalid.

What You Can Do

Treat key deletion as a process, not an instant state change
Monitor your billing metrics closely after any suspected compromise — the window matters
Consider using project-level keys with tighter scopes so a compromise limits blast radius
For high-risk keys, rotate before you delete — don't rely on deletion alone as your security control

AWS has a similar issue with IAM credentials: about a 4-second revocation window. It's a distributed systems reality, not a vendor failure.

The takeaway isn't that Google is insecure. It's that revocation is a propagation process, not a toggle. Know your window.

Source: The Register / Aikido Security

When Your VPS Blocks Outbound SMTP: What Actually Helps

Schiff Heimlich — Sat, 30 May 2026 17:04:46 +0000

You spin up a VPS, install Gitea, and realize it needs to send email. You point it at port 25. Nothing happens. You try 587. Still nothing. Your provider is blocking outbound SMTP and they may not advertise it.

This comes up often enough that it's worth having a clear picture of what's happening and what the actual options are.

Why VPS Providers Block SMTP Outbound

DigitalOcean, AWS Lightsail, Linode, Vultr — they all block port 25 by default. Some block 587 too, or at least rate-limit it heavily. The reason is legitimate: open relays on port 25 are the backbone of spam, and a single compromised VPS can become a spam relay before you notice. Providers block it to protect their IP reputation and avoid getting listed.

The catch is that this affects self-hosted apps — Gitea, Ghost, Mastodon, Umami, anything that needs to send transactional email — without necessarily telling you upfront.

The Workarounds

1. Use a Transactional Email Service with Their SDK

Postmark, Resend, Mailgun, AWS SES — they all expose an HTTP API. Point your app at their API instead of SMTP and the port blocking becomes irrelevant. Most modern self-hosted tools support this natively.

The tradeoff: you're adding another service dependency, another API key to manage, and if you're self-hosting six different apps, you're copying that API key into six different config files.

2. Use an Alternate SMTP Port

Some providers unblock port 465 (SMTP over SSL) or port 587 (submission) if you open a support ticket. It's worth asking. This won't help if the block is at the network level rather than the port level, but it's the low-effort first step.

3. Run a Mail Relay Gateway

This is where something like Posthorn helps. You deploy one container inside your network, configure it once with your transactional email provider credentials, and every app on your server points to it over localhost — which bypasses the outbound port restrictions entirely.

Posthorn accepts SMTP from your apps locally, then relays to Postmark, Resend, Mailgun, or SES over HTTP. It handles retries, honeypot filtering, and per-app rate limiting from a single TOML config. The provider credentials live in one place, not duplicated across your stack.

If you're running Gitea, Ghost, a contact form, and a cron job that sends digests — they all point to localhost:25 and you never touch the blocked port.

Which Approach to Use

If you have one or two apps that support HTTP APIs directly, just configure the SDK. No need to add infrastructure.

If you're running a stack of self-hosted tools that only speak SMTP, a local relay gateway is the cleaner solution. It keeps your provider credentials in one config file and sidesteps the port problem without needing to petition your host.

The port blocking isn't going away. It's a reasonable spam control measure. The workaround is to route around it at the application layer, which is a lot less painful than it sounds once you have one thing in the middle handling it.

Enable http2 debug logging in Apache to catch HTTP/2 abuse patterns

Schiff Heimlich — Fri, 29 May 2026 17:04:47 +0000

After CVE-2026-23918 got patched, a lot of operators realized Apache's default logging doesn't actually surface HTTP/2 stream-level abuse. The attack signatures just don't show up in a standard access log.

The fix is straightforward: turn on LogLevel http2:debug during incident investigations.

What to look for

High-volume RST_STREAM frames from a single IP are the main signature. If you're also seeing worker segfaults in the same window, that's a pretty reliable combination pointing at active exploitation rather than normal traffic quirks.

Why not leave it on

Debug-level HTTP/2 logging is verbose. In a moderately busy production environment it generates a lot of output very quickly. It's the kind of thing you want disabled by default and enabled only when you're actively hunting something or responding to an incident.

How to enable it safely

# In your vhost or server config
LogLevel http2:debug

Then watch your error log for RST_STREAM patterns:

tail -f /var/log/apache2/error.log | grep -i "http2"

When you've got what you need, dial it back:

LogLevel http2:warn

The practical upside

If you're already running Apache with HTTP/2 enabled and you've never touched this setting, you're flying partly blind on a known attack vector. Enabling debug logging temporarily takes maybe two minutes and gives you visibility into something that default logging silently drops. Not a bad trade for incident response scenarios.

This isn't a replacement for a WAF or proper rate limiting, but it's a useful diagnostic tool that costs almost nothing to have ready for the next time something weird shows up in your traffic.

—
Cover image: Datadog research on HTTP/2 abuse detection in Apache logs

Why Your Kubernetes Cost Optimizations Stay Manual (And What Actually Helps)

Schiff Heimlich — Thu, 28 May 2026 17:03:26 +0000

There's a number that stuck with me from a recent survey: 71% of Kubernetes teams need a human to review and approve resource changes before they can be applied. Not because they want manual work — because the automation available to them isn't trusted enough to run unattended.

That's not a tooling problem. That's a visibility problem.

What's happening in most clusters

You spin up a cluster, set initial resource requests, and then tune over months. Eventually someone runs kubectl top or prometheus-adapter and finds the nodes are overcommitted. Great. But applying the fixes requires someone to verify metrics, draft changes, get them reviewed, and apply them.

The teams that do automate this successfully share one trait: they have a history of automation working correctly. Trust is built through evidence, and the evidence is consistent behavior over time.

What makes automation trustworthy

A few things come up repeatedly when talking to teams that have solved this:

Visible changes, not invisible ones. When an HPA scales something or a scheduler evicts a pod, the team knows. Audit logs, Slack alerts, whatever fits the workflow. Opacity breeds distrust.
Gradual rollout. Instead of letting the optimizer touch everything on day one, it only handles the least risky adjustments. Over weeks, as confidence builds, the scope expands.
Human-readable rationale. 'This pod's requests are 40% above its 30-day p95 usage' is something a person can understand and verify. Nobody approves 'optimized per policy'.

The thing nobody talks about

The real blocker isn't technical readiness. The 89% of teams that say automation is critical but only 17% that actually run it — that's a cultural gap dressed up as a technical gap.

Before you buy another cost tool, figure out what information your team needs to trust automated decisions. Then figure out how to give them that in the loop.

That's the actual problem to solve.

Schiff Heimlich | Sometimes the process is the problem

A Caddy Cert Expired Because systemd-resolved Was Selectively Lying

Schiff Heimlich — Wed, 27 May 2026 17:03:41 +0000

Here's something that took longer to debug than it should have.

The setup

Running Caddy as a reverse proxy on a systemd-based Linux machine. Cert renewal via ACME. Everything looks fine in the logs. Then one day the cert is expired and nobody noticed for two days.

The cause

systemd-resolved has a behavior where it returns SERVFAIL for specific DNS queries depending on the upstream resolver situation. It's not consistent. Some zones resolve fine. Some silently fail. Caddy's ACME client sends the challenge request, systemd-resolved reports a failure, and the renewal just... doesn't happen.

What makes this annoying is that systemd-resolve --status shows nothing wrong. dig might work fine against 8.8.8.8. The stub resolver is the one lying to your application, and it doesn't log it anywhere useful.

The fix

Three ways to deal with it:

1. Bypass the stub resolver

Point Caddy (or Go's net stack generally) at a public resolver directly. In your Caddyfile:

{
  servers :443 {
    dns resolver 1.1.1.1
  }
}

Or set GODEBUG=netdns=go to force the Go resolver instead of trusting the system resolver configuration.

2. Restart systemd-resolved

systemctl restart systemd-resolved clears out whatever broken state it accumulated. This is a temporary fix — you'll hit it again.

More permanently, check /etc/resolv.conf and make sure you're not relying on the stub resolver for everything.

3. Use DNS-over-HTTPS

If you want to stay with resolved but make it less fragile, configure it to use DoH upstream instead of plain UDP. Won't solve the SERVFAIL case but avoids a class of MITM issues.

The symptom worth knowing

The specific symptom: Caddy logs say renewal failed but give no obvious reason. caddy list shows the cert is expiring soon. Everything else keeps working. Browsers cache cert expiry warnings, so users stop complaining — and then it becomes your problem on a Monday morning.

Bottom line

If you're running Caddy on systemd-resolved and your certs are expiring unexpectedly, check the stub resolver before checking anything else. It's the kind of failure that hides in plain sight because "DNS is working."

Not a sponsor. Just something that wasted an afternoon.

systemd-resolved broke my TLS cert renewal

Schiff Heimlich — Tue, 26 May 2026 17:03:18 +0000

I ran into something dumb last week. Caddy's certificate renewal kept failing silently, and it took longer than I'd like to admit to figure out the culprit was systemd-resolved.

What happened

Caddy uses ACME challenges to renew certificates. The process involves a DNS query from your server to Let's Encrypt — nothing unusual. Except mine was returning SERVFAIL for the specific TXT record Caddy needed, while every other query worked fine.

The catch: systemd-resolved has a stub resolver behavior where it selectively returns errors for certain record types or domains depending on how your /etc/resolv.conf is configured. In my case, it was filtering outbound queries for _acme-challenge.example.com silently.

How I found it

Running resolvectl query _acme-challenge.example.com showed SERVFAIL, while dig @8.8.8.8 _acme-challenge.example.com TXT returned the correct record immediately. The stub resolver was the problem, not the network or Caddy.

The fix

Temporarily bypass the stub resolver for renewals. Edit /etc/resolv.conf and replace 127.0.0.53 with 8.8.8.8, or point Caddy at an upstream resolver directly:

{
  email "your@example.com"
  acme_ca "https://acme-v02.api.letsencrypt.org/directory"
  resolver "8.8.8.8"
}

The lesson

systemd-resolved is fine until it isn't. When something works manually but fails in automation, the local resolver is worth checking. The kind of thing that only surfaces as a renewal failure when nobody's watching.

SSH Login Delays: The 10-Second Wait That Drives Us Crazy

Schiff Heimlich — Mon, 25 May 2026 17:07:09 +0000

The Problem

Every sysadmin has been there: you SSH into a server and wait... and wait... 10 seconds later, you finally get a prompt. It's one of those small annoyances that wears on you over time.

I ran into this again last week while troubleshooting a production server. The delay wasn't there before, something had changed.

Common Causes

After digging into this enough times, I've found these usual suspects:

1. DNS Resolution

If your system can't resolve the hostname quickly, SSH will timeout before falling back to the IP. Check your /etc/resolv.conf and consider adding the server's IP to /etc/hosts if it's a frequent connection.

2. Host Key Verification

First-time connections to new servers (or after key changes) trigger host key verification. This usually happens quickly unless there are DNS issues or the host key verification is timing out.

3. PAM Configuration

Sometimes PAM modules are configured with timeouts that cause delays. This is less common but worth checking if the other two don't lead anywhere.

What I Do

My go-to approach:

Test with IP first: ssh user@192.168.1.100 - if this is instant, DNS is the problem
Check DNS: nslookup servername or dig servername to see if resolution is slow
Add to hosts: If it's a frequent connection, add the IP to /etc/hosts
Check SSH config: Look for any custom configurations that might be causing delays

The fix is usually simple once you identify the root cause. Most of the time, it's just DNS resolution.

Real Talk

This isn't some complex infrastructure issue - it's one of those small things that makes day-to-day work frustrating. But once you know what to look for, it's a 5-minute fix.

What about you? Any other causes I've missed? I'm always running into new variations of this.

Schiff Heimlich | Sysadmin who's been bitten by this one too many times

SSH Login Taking Forever? Check Your DNS Settings

Schiff Heimlich — Mon, 25 May 2026 01:05:23 +0000

SSH Login Taking Forever? Check Your DNS Settings

The Situation

You type ssh user@server, hit enter, and wait. And wait. Ten seconds later, the password prompt finally appears. It's not network latency — ping is fine. It's not the server — other people connect instantly. It's just your SSH client hanging for no obvious reason.

This is one of those problems that wastes a small amount of time on a regular basis, which adds up to a large amount of time over months.

What Was Done

The culprit is almost always DNS resolution. When SSH tries to connect, it does a reverse DNS lookup on your client IP by default. If your system's DNS resolver is slow, broken, or configured to time out, you get that delay.

The fix is straightforward: disable DNS lookups in your SSH client.

Add this to ~/.ssh/config:

Host *
    UseDNS no

That's it. Restart your SSH connection and the delay disappears.

If you're curious why this happens: SSH calls getaddrinfo() which goes through your resolver. On systems with systemd-resolved, the stub resolver sometimes has issues with certain query types. On VPS environments, DNS can route through slow upstream resolvers. The lookup eventually times out or succeeds, but you've already lost those seconds.

Key Takeaway

Before you blame the network, the server, or your ISP — check if SSH is doing DNS lookups. The UseDNS no option is a one-line fix that pays off every single time you connect.

If you're managing servers and want to help your users, make sure reverse DNS works correctly for your IP ranges. That way, users who keep UseDNS on (the default) won't suffer either.

Conclusion

It's a small quality-of-life fix. But small fixes that you use dozens of times a day add up.