DEV Community: speed engineer

The Mental Model That Ended My Guess-and-Check Debugging

speed engineer — Sun, 16 Aug 2026 03:41:57 +0000

The problem

A few years ago I watched a senior engineer "debug" a slow checkout API for two hours. He added an index. No change. He bumped the connection pool. No change. He wrapped the handler in a cache. Marginal change, wrong reason. He was pattern-matching against past incidents instead of measuring the current one — and most of us do this more than we'd like to admit.

The tell isn't lack of skill. It's the absence of a mental model that forces you to look at the machine before you touch the code.

Why it happens

Modern systems give you too many plausible culprits: database, network, GC pauses, lock contention, a noisy neighbor container, disk I/O. Without a framework, your brain defaults to "what fixed it last time," which is a bias, not a diagnosis. You end up changing five things and shipping a fix you can't actually explain.

What to do about it

I now start every performance investigation with the USE method (credit to Brendan Gregg): for every resource — CPU, memory, disk, network — check three things:

Utilization: is the resource busy?
Saturation: is work queued waiting for it?
Errors: is it throwing errors that force retries or fallbacks?

On that checkout API, here's what USE actually surfaced in about six minutes:

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0      0 812340  20144 933212    0    0     0    18 1200 2400  8  3 60 29  0

wa (I/O wait) at 29% with CPU idle at 60% told me immediately this was not a CPU or code-logic problem — it was disk saturation. iostat -x 1 confirmed one EBS volume at 98% utilization with a queue depth climbing past 12. The "slow" endpoint was writing synchronous audit logs to the same volume as the primary database, and a batch job had started hammering that disk ten minutes earlier.

No amount of query optimization or connection pool tuning was ever going to fix that, because the bottleneck wasn't in the code path at all — it was contention on a shared resource two layers down. The fix was moving the audit log writes to a separate volume. Twenty-minute change, once we knew where to look.

The point isn't that USE always finds the answer that fast. The point is it stops you from guessing. You walk CPU, memory, disk, network in order, and for each one you either rule it out with data or you find your suspect. You never touch code before you've ruled out the machine.

A second habit that pairs well with this: write down your hypothesis before you look at a single metric. "I think this is CPU-bound because the handler does JSON serialization in a loop." Then check. When the data disagrees with your hypothesis — and it will, more often than your ego wants — that gap is the actual lesson. I keep a running list of my wrong hypotheses next to my right ones. The wrong ones taught me more.

Key takeaways

Guess-and-check debugging is a pattern-matching bias, not a diagnostic process — it feels like progress without producing evidence.
The USE method (Utilization, Saturation, Errors) forces you to check the machine's resources in order before touching application code.
I/O wait time in vmstat and queue depth in iostat -x will tell you in minutes whether you have a disk-saturation problem — no code change will fix a hardware-contention issue.
Write your hypothesis down before you look at data. The mismatches are where you actually learn.

Your Coding Interview Stopped Measuring Anything the Day Candidates Got Copilot

speed engineer — Fri, 14 Aug 2026 04:04:07 +0000

The problem

Last year I sat in on a loop for a senior backend role. The candidate solved a graph traversal problem in eleven minutes — clean code, right complexity, no hints needed. On paper, a hire. Ninety days later, on the job, the same person couldn't work out why a service was leaking connections under load. Not a knowledge gap — a reasoning gap. They'd never had to debug something they didn't already understand end to end.

That disconnect isn't rare anymore. It's the default outcome of running a 2019-era interview loop in 2026.

Why it happens

The classic algorithmic interview measures one thing well: can you recognize a known pattern (two-pointer, DP, BFS, sliding window) and implement it correctly under time pressure. That's a recall-and-execute task. It was never a great proxy for engineering judgment, but for a decade it correlated well enough with "smart, prepared, can code" to be useful.

AI broke the correlation, not the test. Any candidate who's spent even a few months pairing with an AI assistant has effectively memorized the pattern library through repetition, whether or not they touch a model during the interview itself. The rehearsal loop got faster, so the "recall a pattern under pressure" signal compressed toward everyone scoring well. You're no longer measuring engineering ability — you're measuring how many hours someone spent grinding a fixed problem set, which is a much weaker signal and correlates poorly with what the job actually requires.

Meanwhile the skill that was always the real differentiator — debugging code you didn't write, inside a system whose invariants you don't fully know, with incomplete information and someone waiting on you — never showed up in the interview at all. It's roughly 70-80% of senior engineering work by time spent, and it was zero percent of the assessment.

What to do about it

We rebuilt the loop around three changes, and the signal quality difference was immediate and obvious to every interviewer on the panel within the first week of running it.

1. Replace "implement X" with "here's a broken 150-200 line service, find it." We handed candidates real code (sanitized) with a bug that only reproduced under concurrency — a race between a cache invalidation and a read path. No algorithm to recall. Just: read unfamiliar code, form a hypothesis, test it, narrow it down. This is the actual daily loop of debugging in production, and it can't be shortcut by pattern memorization.

2. Allow AI assistance explicitly, and interview the usage, not just the output. We stopped pretending we could detect or prevent AI use and started treating it as a tool candidates would obviously have on the job. The question shifted from "did you use AI" to "when the AI suggested that fix, why did you accept it — and here's a case where its suggestion is subtly wrong, catch it." Candidates who understood the system could catch the wrong suggestion in seconds. Candidates who were pattern-matching couldn't, even with the AI's help, because they didn't have the mental model to evaluate what they were looking at.

3. Ask "why" mid-task, repeatedly, and watch the pause before the answer. Not gotcha questions — just "why that line, why not the other approach." The hesitation pattern between someone reasoning live versus someone recalling a rehearsed justification is very distinguishable once you're listening for it. It's a soft signal, but paired with the debugging task it stopped being noisy.

None of this eliminates false positives. It doesn't need to — it just needs to raise the correlation between loop performance and 90-day performance back above where the old format had fallen to.

Key takeaways

Algorithmic interviews measure pattern recall under time pressure — AI collapsed that signal by making rehearsal cheap and universal.
The skill that predicts job performance — debugging unfamiliar code under partial information — was never directly tested by the old format, AI or not.
Allowing AI explicitly and interviewing the judgment behind its use produces more signal than trying to detect or ban it.
Watching how a candidate evaluates a wrong suggestion tells you more than watching them produce a correct one.

Your Multithreaded Code Is Correct and Still 8x Slower: False Sharing Explained

speed engineer — Mon, 10 Aug 2026 05:01:35 +0000

The problem

Eight cores. Eight threads. Each thread owns a private counter and increments it in a tight loop, millions of times a second, with zero locks and zero shared state at the logical level. No mutexes, no atomics contention, no obvious bottleneck in the code.

Throughput should scale close to linearly. Instead you measure it and get roughly the same total throughput as one thread — sometimes worse. Nobody touched the same variable. There's no race condition. And it's still catastrophically slow.

I hit this exact case profiling a stats-collection layer: a Counter counters[NUM_THREADS] array, one slot per thread, each thread only ever writing its own index. On paper, embarrassingly parallel. In practice, adding cores made things worse.

Why it happens

The CPU doesn't move data in individual bytes or even individual variables — it moves it in cache lines, almost always 64 bytes on modern x86 and ARM. When a core writes to any byte in a cache line, the cache-coherency protocol (MESI, or a variant of it) invalidates every other core's cached copy of that entire line, not just the byte that changed.

An int counters[8] array is 32 bytes. On a 64-byte cache line, all eight counters — one per thread — can live in a single cache line, or split across two. It doesn't matter that thread 3 only ever writes counters[3] and never looks at counters[5]. As far as the hardware is concerned, every write to that line by any core forces the other cores' copies to be invalidated and re-fetched.

The result is the cache line physically ping-ponging between L1 caches across cores, even though there is zero logical data dependency between the threads. Each of those cross-core transfers costs on the order of tens to over a hundred nanoseconds — dwarfing the single-digit-nanosecond cost of the increment itself. You end up serializing on cache coherency traffic that never shows up as a lock, a mutex, or anything else you'd normally look for in a profiler's call graph.

This is false sharing: a performance bug with no corresponding correctness bug. Your code is right. Your memory layout is wrong.

What to do about it

1. Detect it before you "fix" it. Don't guess — false sharing has a specific fingerprint: HITM events (cache-line hit in "Modified" state, transferred cache-to-cache). On Linux:

perf c2c record -- ./your_binary
perf c2c report

This will point you at the exact cache line and the specific cores/threads fighting over it. Intel VTune's memory-access analysis surfaces the same signal on non-Linux setups.

2. Pad or align the hot data to cache-line boundaries.

struct alignas(64) PaddedCounter {
    std::atomic<uint64_t> value;
    char pad[64 - sizeof(std::atomic<uint64_t>)];
};

PaddedCounter counters[NUM_THREADS];

Now each counter owns its own cache line. No cross-core invalidation traffic, because no two threads' hot variables share a line anymore.

3. Prefer thread-local accumulation over shared arrays when you can. Instead of N threads writing into a shared array of N slots, give each thread a genuinely private (thread-local or stack-local) counter and aggregate once at the end. This sidesteps the layout problem entirely instead of papering over it with padding.

4. Don't pad everything reflexively. Padding trades memory and cache footprint for coherency traffic. On data that's read-mostly, or accessed by a single thread anyway, padding just wastes cache capacity and can hurt performance elsewhere. Measure with perf c2c first, pad the specific structures it flags, and stop there.

Key takeaways

False sharing produces zero logic errors — your tests pass, your assertions hold, and your scaling is still terrible.
The unit of cache coherency is the cache line (typically 64 bytes), not the variable — layout decisions you never think about become hardware-level contention.
perf c2c turns "this is inexplicably slow" into "these two cores are fighting over this exact cache line," which is the difference between guessing and fixing.
Padding and thread-local accumulation are targeted fixes for a measured problem, not a default you apply to every struct.

Hyrum's Law: Why You Can Never Actually Deprecate Anything

speed engineer — Sun, 09 Aug 2026 03:39:47 +0000

The problem

We shipped a "deprecated" internal field, response.legacy_id, and left it in for backward compatibility — marked deprecated in the docs, with a removal date two releases out. Nobody consumed it directly; no client code referenced it anywhere. So we cut it.

Three unrelated services broke in production within about forty minutes. Not because they read the field. Because a shared caching proxy fingerprinted the entire response body to build cache keys, and removing a field changed every hash, which invalidated caches that three completely different systems depended on for reasons that had nothing to do with legacy_id.

Nobody used the field. Everybody depended on it anyway.

Why it happens

This has a name — Hyrum's Law, coined by Hyrum Wright at Google: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody."

The key word is observable, not documented. Your contract says "this field is deprecated, don't rely on it." Reality doesn't care what the contract says. If a behavior is observable — response ordering, field presence, error message text, timing, byte-for-byte JSON shape, even header casing — someone, somewhere, at scale, will build on it. Usually not maliciously. Usually accidentally, through a generic tool (a diffing layer, a schema validator, a caching proxy, a test snapshot) that treats the entire observable surface as the contract, because that's the only contract it can see.

The bigger your user base, the more certain this becomes. At 10 callers, a dependency on field ordering is a coincidence. At 10,000, it's a statistical certainty that it exists somewhere.

This is also why "nobody's using it, I checked the code" is a weaker guarantee than it feels. You checked the code that reads your API directly. You didn't check every downstream cache key, diff tool, monitoring rule, or test fixture built against your response shape.

What to do about it

You can't design your way out of Hyrum's Law — you can only manage the blast radius.

Version your contract explicitly, and make it narrower than what you actually return. If your spec only promises three fields, undocumented extra fields are fair game to change — but say so loudly, because "undocumented" doesn't mean "unobserved."

Make deprecation observable before it's real. Don't silently remove a field. First return it with a sentinel value for a full release cycle, log every caller that still reads a non-null value, and ship machine-readable deprecation signals (like Sunset/Deprecation headers) instead of a line in a changelog nobody reads.

Change shape deliberately, not incidentally. If you're removing a field, make it its own atomic change — not bundled with an unrelated refactor — so if something breaks, the diff that caused it is one commit, not forty.

Treat internal consumers with the same discipline as external ones. The break above wasn't a public API — it was internal service-to-service traffic, which teams often skip versioning discipline on because "we control both sides." You control both codebases. You don't control every cache and proxy sitting between them.

Assume the null hypothesis is wrong. Before removing any observable behavior, default to "something depends on this," not "probably nothing depends on this." That's cheap to disprove with real production traffic sampling, and expensive to discover the hard way.

Key takeaways

Hyrum's Law: at scale, every observable behavior of your system becomes a dependency, whether you documented it or not.
"Nobody calls this field" only accounts for direct callers — indirect consumers (caches, diff tools, snapshots, monitors) depend on the whole observable shape.
Deprecation should be logged and observable before it's enforced; silent removal is where the surprises live.
Internal APIs need the same versioning discipline as public ones — "we control both sides" isn't the same as "we control every intermediary."
Default to assuming something depends on the behavior you're about to change, and verify against real traffic before you cut it.

The UDP Buffer Nobody Tunes: How a 256KB Kernel Default Cost Us 30% of Our Packets

speed engineer — Sat, 08 Aug 2026 03:46:51 +0000

The problem

A video platform I worked on wasn't crashing — it was worse than that. Calls stayed "up" but froze every few seconds, audio turned robotic, and it only happened between 5 and 7 PM. WebRTC sessions were logging close to 30% packet loss during that window. Network ops checked switches, routers, bandwidth — utilization sat around 40%, nothing congested. Ticket closed: "not a network problem."

It was. Just not in the place anyone was looking.

Why it happens

The instinct with UDP is to treat it like "TCP without the reliability guarantees." That framing hides the part that actually matters: TCP has backpressure built in. When a TCP receive buffer fills, the receiver shrinks its window and the sender slows down. UDP has none of that. The kernel gets a datagram, tries to place it in a receive buffer, and if there's no room, it silently drops it — no retry, no signal to the sender, nothing. From the app's point of view, the packet simply never existed.

tcpdump showed packets arriving cleanly at the NIC. Application logs showed holes in the sequence numbers. Packets were vanishing somewhere between the wire and the process — which meant the kernel, not the code, was the suspect.

The confirming metric was udp_receive_buffer_errors, sitting at zero off-peak and spiking into the thousands-per-second exactly when complaints rolled in. netstat -su made it explicit: 223,401 receive buffer errors out of ~752K packets received. The Ubuntu default for net.core.rmem_default was 212,992 bytes — roughly 256 KB. At peak, the service was pushing ~50,000 packets/sec at ~1,200 bytes each: about 60 MB/sec into a 256 KB bucket. That's roughly 4 milliseconds of headroom before the buffer fills. Get descheduled for longer than that — completely normal under load — and the kernel starts discarding.

It compounds, too: WebRTC detects the quality drop from the loss and adds redundancy and retries at the application layer, which throws more traffic at the same undersized buffer. The fix for congestion becomes the thing that deepens it.

What to do about it

The fix is two layers, and both are required — most teams only do one.

Kernel layer — raise the ceiling and the default:

# /etc/sysctl.conf
net.core.rmem_max = 16777216      # 16 MB max
net.core.rmem_default = 16777216  # 16 MB default for new sockets

sysctl -p

16 MB instead of 256 KB turns 4ms of headroom into roughly 260ms.

Application layer — the socket still has to ask for it, and you have to verify what it actually got:

int sock = socket(AF_INET, SOCK_DGRAM, 0);
int buffer_size = 16 * 1024 * 1024;

if (setsockopt(sock, SOL_SOCKET, SO_RCVBUF,
               &buffer_size, sizeof(buffer_size)) < 0) {
    perror("Failed to set socket receive buffer");
}

int actual_size;
socklen_t len = sizeof(actual_size);
getsockopt(sock, SOL_SOCKET, SO_RCVBUF, &actual_size, &len);
printf("Requested: %d bytes, Got: %d bytes\n", buffer_size, actual_size);

That getsockopt check matters more than it looks — some distros silently double what you request for internal bookkeeping, so "did I get what I asked for" is not a safe assumption.

One dead end worth naming: bumping the packet-processing thread's scheduling priority. It seems logical — drain the buffer faster — but it starves other threads of CPU time, and now you've traded packet loss for database timeouts. Buffer sizing is the correct first move; thread priority is a last resort, not a first instinct.

Key takeaways

UDP has zero backpressure — a full receive buffer means silent, unrecoverable loss, not slowdown.
Kernel socket buffer defaults (net.core.rmem_default) are generic and almost never sized for sustained high-throughput UDP.
Fixing it requires both the sysctl change and an explicit setsockopt(SO_RCVBUF) call in your app — plus a getsockopt check to confirm what you actually got.
Watch udp_receive_buffer_errors from netstat -su as a standing metric; it should be zero, and any sustained non-zero value is a real signal, not noise.
Think in bursts, not averages — a tame req/sec number can still overflow a small buffer during a 200ms spike.

I wrote up the full debugging story — including the wrong turn I took first and what I now monitor because of it — on Medium.

Lost in the Middle: Why Feeding Your Agent More Context Makes It Dumber

speed engineer — Wed, 05 Aug 2026 05:06:00 +0000

The problem

You build a RAG pipeline. You test it with 3 retrieved documents and the correct answer is right there in document 2 - the model nails it every time. Feeling good, you bump top_k from 3 to 15 "to be safe," figuring more context can only help.

Accuracy drops. Not a crash, not a timeout, no error in your logs - the model just starts confidently giving wrong answers, or missing facts that are sitting in plain text inside the prompt you sent it. You re-read the context window by hand and the answer is right there. The model saw it. It just didn't use it.

If you've hit this, you didn't build a bad retriever. You ran into a well-documented, model-agnostic failure mode: LLMs don't recall information uniformly across a context window. They recall the beginning and the end. The middle is where information goes to die.

Why it happens

This isn't a rumor - it's been measured directly. Liu et al.'s "Lost in the Middle" study (2023) ran controlled retrieval tests across context lengths and found a consistent U-shaped accuracy curve: performance is highest when the relevant fact sits at the very start or very end of the context, and drops - sometimes below random-guessing-adjacent territory - when it's buried in the middle. This held across multiple model families, not one vendor's quirk.

The mechanism is architectural, not a bug you can patch. Self-attention doesn't distribute recall evenly across token positions. Models get heavy exposure to short-range dependencies during training (the next word usually depends on nearby words), and comparatively little training signal that rewards precise retrieval from the geometric middle of a long, unstructured span. Positional encodings compound this - most schemes bias attention toward strong local recency and, separately, toward the sequence start (a natural anchor point), leaving the middle structurally under-attended.

Here's the part that actually bites in production: the advertised context window and the effectively usable context window are different numbers, and vendors report the former. A model with a "1M token context" doesn't mean it retrieves reliably across 1M tokens - it means it doesn't error out before 1M tokens. Those are not the same claim, and the gap between them is exactly where "it worked in the demo, it's flaky in prod" bugs come from.

And it's silent by design. There's no exception to catch. Your retriever did its job, your prompt assembly did its job, the tokens are unambiguously present in the context - the failure is purely in what the model chooses to attend to, which is invisible from the outside unless you're specifically testing for it.

What to do about it

Stop treating top_k as a safety dial. More retrieved chunks doesn't monotonically improve recall - past a point it actively pushes your one correct chunk further into the dead zone. Tune top_k down, not up, and measure.

Reorder, don't just retrieve. After ranking, place your highest-confidence chunks at the start and end of the context, not in ranked order top-to-bottom. A chunk ranked #1 by your retriever but positioned dead-center in the prompt will underperform a chunk ranked #4 placed at the edges.

Add a reranker stage. A cheap cross-encoder rerank pass after initial retrieval, feeding only the top few into the prompt in a position-aware order, consistently outperforms "retrieve broadly, stuff it all in, let the model sort it out."

Build your own needle-in-a-haystack eval. Don't trust a vendor's long-context benchmark for your workload. Position sensitivity varies by model, by model version (it can regress on a silent upgrade), and by the structure of your specific documents. Take a real query from your logs, plant the known-correct fact at position 10%, 50%, and 90% of your typical context length, and measure accuracy at each. Rerun it every time you swap models.

Consider narrowing before you widen. A multi-step retrieval loop that progressively filters down to a small, high-confidence context often beats a single giant context stuffed with "everything that might be relevant." Fewer, better-placed tokens beat more tokens almost every time.

Key takeaways

Context window size and effective retrieval range are different numbers - don't conflate them.
The "Lost in the Middle" effect is architectural and measured across model families, not a one-off bug.
The failure mode is silent: no error, just quietly wrong answers on facts that are technically present.
Fixes: shrink top_k, rerank and position-aware reorder, build your own positional eval, and prefer narrowing retrieval over widening it.

Why Your p99 Latency Explodes Long Before Your Servers Look Busy

speed engineer — Mon, 03 Aug 2026 04:47:12 +0000

The problem

A service humming along at 40ms average latency. CPU sits at 60%. Dashboards green. Then the pager goes off: p99 is 900ms, and it's been that way for weeks — nobody noticed because the average never moved. On-call pulls up flame graphs, finds nothing pathological, and closes the ticket as "noise."

It isn't noise. It's queueing.

Why it happens

Average latency measures the common case. Tail latency measures what happens when several unlikely things line up at once — and unlikely things line up more often than intuition suggests. Two mechanisms do almost all the damage.

1. Utilization is not linear with latency. For a simple queue (the M/M/1 approximation), expected wait time scales roughly with ρ/(1-ρ), where ρ is utilization. At 50% utilization, that multiplier is 1x. At 80%, it's 4x. At 95%, it's 19x. At 99%, it's 99x. The curve stays flat for a long time and then goes vertical. A service that "looks fine" at 60% CPU isn't 60% away from saturation — it's most of the way up a curve that's about to explode.

2. Fan-out amplifies tails. If a request fans out to 20 backend calls and waits on all of them, your response time is the max of 20 samples, not the average of 20. Even if every backend has a p99 of 100ms (99% chance of finishing under 100ms), the odds that all 20 finish under 100ms is 0.99^20 ≈ 82%. Close to 1 in 5 requests gets dragged down by whichever one of the twenty had a bad moment. That's not a bug — it's arithmetic. Add naive retries on top and you can amplify load at exactly the moment a dependency is already struggling.

The two mechanisms compound: fan-out multiplies your exposure to tail events, and rising utilization on any one dependency makes tail events on that dependency more frequent.

What to do about it

Stop planning capacity around average utilization. Plan around the utilization at which queueing delay becomes unacceptable, then keep steady-state usage well under that line — most latency-sensitive paths want something like 65-75%, not 90%+.

Bound fan-out instead of letting it grow. Every extra parallel call you wait on is another chance to hit someone else's bad millisecond. If you don't need all 20 results, don't wait for all 20.

Use hedged (backup) requests instead of blind retries. Fire a second request to a different replica if the first hasn't returned by roughly your own p95, and take whichever comes back first, cancelling the loser. This trims the tail without the pile-on effect of retry-on-timeout, which just adds load to an already-struggling dependency.

Add jitter to timeouts and retries. Synchronized retries across many clients are how a small blip becomes a cascading outage — everyone times out at the same millisecond and hits the backend at once.

Isolate noisy neighbors. A single slow dependency behind a shared thread or connection pool can drag down requests that never even touch it, because everything is waiting in the same queue for a free worker. Bulkhead pools per-dependency.

Key takeaways

Average and tail latency are different problems with different causes. A flat average tells you nothing about how close you are to the queueing cliff.
Utilization near saturation makes latency blow up nonlinearly — plan capacity against that curve, not against "CPU still has headroom."
Fan-out to N parallel calls means your latency is governed by the worst of N, not the average of N. Bound N wherever you can.
Hedged requests with jittered timeouts beat blind retries for taming tails without triggering retry storms.
If your p99 is bad and your average looks fine, check utilization and fan-out width before you check code.

Why Your p99 Latency Explodes Long Before Your Servers Look Busy

speed engineer — Mon, 03 Aug 2026 04:46:39 +0000

The problem

It isn't noise. It's queueing.

Why it happens

The two mechanisms compound: fan-out multiplies your exposure to tail events, and rising utilization on any one dependency makes tail events on that dependency more frequent.

What to do about it

Bound fan-out instead of letting it grow. Every extra parallel call you wait on is another chance to hit someone else's bad millisecond. If you don't need all 20 results, don't wait for all 20.

Key takeaways

Average and tail latency are different problems with different causes. A flat average tells you nothing about how close you are to the queueing cliff.
Utilization near saturation makes latency blow up nonlinearly — plan capacity against that curve, not against "CPU still has headroom."
Fan-out to N parallel calls means your latency is governed by the worst of N, not the average of N. Bound N wherever you can.
Hedged requests with jittered timeouts beat blind retries for taming tails without triggering retry storms.
If your p99 is bad and your average looks fine, check utilization and fan-out width before you check code.

Your Backups Have Been Silently Corrupt Longer Than Your Retention Window

speed engineer — Sat, 01 Aug 2026 06:56:26 +0000

The problem

Corruption gets discovered on read, almost always. A report pulls a file, the numbers look wrong, someone finally opens the raw bytes and finds the sector is garbage. Fine — that's what checksums are for. You confirm the file is bad, you go pull it from backup, and you breathe out.

Then the restored copy has the exact same corruption.

So does the one from three days before that. So does the one from three weeks before that. You don't have a "restore from backup" problem anymore. You have a "how far back do I have to go to find a good copy, and do I still have one" problem — and the honest answer, more often than teams expect, is no.

Why it happens

Almost every system checksums on read: verify the bytes when something asks for them, so you at least know not to trust a bad file. Far fewer systems checksum on write: verify the bytes the moment they're stored, so corruption gets caught within seconds of happening instead of whenever someone next happens to need that exact object.

That gap between "corruption occurs" and "corruption is detected" is the dwell time. Nothing about a backup process shortens it. A backup job doesn't know what "correct" looks like — it just faithfully, efficiently copies whatever bytes are sitting there, intact or not. If the source was already corrupted when last night's backup ran, you now have a corrupted backup, indistinguishable from a good one, sitting in your retention window right next to a dozen other backups with the identical problem.

The failure only becomes visible once dwell time exceeds retention window. A firmware bug flips bits in a file on day one. Nobody reads that specific file again until day forty-five, for a quarterly report. If your retention is thirty days, every single backup you're holding — all thirty of them — postdates the corruption. There is no "go back further" option, because further back doesn't exist anymore. The backup succeeded every night. The system was never lying to you about that. It just was never checking the thing you actually needed it to check.

What to do about it

Checksum at write time, not just read time. Compute and store a checksum (or use a filesystem/object store that does this natively — ZFS, Btrfs, S3's built-in integrity checks) the moment data lands. Verify on every read after that, and alert immediately on a mismatch — don't wait for someone to notice the number looks off.
Run scheduled integrity scans, not just on-demand checks. A read-triggered checksum only catches corruption in data someone actually reads. Cold data — old logs, archives, rarely-touched records — can sit corrupted indefinitely with nothing ever tripping the check. A periodic full-corpus scrub (ZFS scrub, or a scheduled job that reads and re-verifies every object) forces detection on a schedule you control, instead of waiting on read patterns you don't.
Size your scrub interval against your retention window, deliberately, as a stated relationship — not two settings that happen to coexist. If backups are kept 30 days, your worst-case detection time — the longest possible gap before a scrub or a read would catch a given piece of corruption — needs to be meaningfully shorter than 30 days. If it isn't, you don't actually have 30 days of real recovery coverage; you have some smaller, undefined number that depends on when corruption happens to occur relative to your scan cycle.
Keep at least one backup generation older than your detection SLA, treated as untouchable. Even with fast detection, you want a fallback that predates your worst plausible dwell time — a monthly or quarterly snapshot held outside the normal rotation, specifically so "go back further" is still a real option the day you need it.

Key takeaways

Checksums that only run on read tell you a file is bad; they don't limit how long it's been bad, and that gap is what actually determines whether backups can save you.
A backup process has no concept of "correct" — it copies whatever bytes exist, so corruption propagates through every generation made after it occurred.
The real risk metric isn't "do we have backups," it's "is our worst-case corruption dwell time shorter than our retention window" — most teams have never measured this.
Write-time checksums plus scheduled full-corpus scrubbing are what actually bound dwell time; read-triggered checks alone leave cold data unprotected indefinitely.
Keep one backup generation deliberately older than your detection SLA as an untouchable fallback.

I wrote about the mechanics of catching corruption in the first place — checksum algorithms, where to place the check, what it costs — on Medium: https://medium.com/@speed_enginner/checksum-everything-corruption-caught-before-catastrophe-5cace12122fa

Your Hiring Freeze Has an Adverse Selection Problem

speed engineer — Fri, 31 Jul 2026 05:24:55 +0000

The problem

Every hiring freeze I've watched up close follows the same script: leadership announces a freeze, comp reviews get "paused pending Q results," and six months later someone asks why the team feels weaker than it did a year ago — even though the attrition numbers look fine on a dashboard.

The dashboard is lying. Not with fake numbers, but with the wrong number. Total attrition can sit at a perfectly normal 8-10% while the composition of who's leaving quietly inverts.

Why it happens

A freeze almost never freezes just headcount. It freezes comp too — off-cycle adjustments get shelved "until things stabilize." Meanwhile the external market for the same role doesn't freeze at all. Two years of that gap and you have real comp compression: a senior engineer hired externally today would clear a number your three-year tenured senior engineer is nowhere near.

Here's the part that doesn't show up in an attrition chart: the tenured engineer's ability to act on that gap is not evenly distributed. Your strongest performers pass external loops easily — they have a portfolio, a track record, recruiters already in their inbox. A credible outside offer is a phone call away. Your average-to-below-average performers don't have that option nearly as reliably; external interviews are a real filter, and they know it.

So when comp quietly falls behind market during a freeze, the population that's able to leave and the population that's underpaid only partially overlap — and the overlap skews toward your best people. This is adverse selection, the same mechanic Akerlof described in the market for lemons, just running on your internal talent pool instead of used cars: the participants with better private information about their own value are the ones who exit the pool, leaving a residual population that's systematically weaker than the average you started with.

It compounds. The freeze didn't shrink the workload, so the remaining team — now missing its strongest members — absorbs it. More load lands on a team with a lower average ceiling, which raises the burnout and attrition risk of exactly the people you have left. That's not a one-time dip in team quality. It's a loop.

What to do about it

Three things I've seen actually work, none of which require lifting the freeze:

Track regretted attrition separately from total attrition. A flat 9% churn number can hide a regretted-attrition rate that's climbing fast. If you're only watching the aggregate, you find out about the problem a year after it started.

Compute compa-ratio against current external market data, not last year's band. Bands set eighteen months ago are exactly the thing comp compression exploits. If you're benchmarking against a stale band, you'll conclude you're "fine" right up until your best engineer gives notice.

Spend retention budget as a scalpel, not a freeze. A blanket comp freeze and a handful of targeted off-cycle adjustments for flight-risk high performers cost very differently, and only one of them stops the adverse-selection loop. The org that can't do targeted retention during a freeze is choosing, whether it says so or not, to let the loop run.

Key takeaways

A stable total-attrition number can coexist with a rapidly worsening composition of who's leaving.
Comp compression during a freeze isn't just "everyone underpaid equally" — the people most able to act on it are disproportionately your strongest performers.
This is adverse selection: the exit is correlated with ability, not random.
Fix the measurement first (regretted attrition, live compa-ratio) — you can't manage a loop you can't see.

The CPU Graph Said 40%. We Were Being Throttled Every Second.

speed engineer — Thu, 30 Jul 2026 04:01:52 +0000

The problem

3 AM, paged for p99 latency on a checkout service that was, by every dashboard we had, fine. Average CPU utilization: 38%. Memory: comfortable. Error rate: near zero. And yet roughly one request in twenty was taking 800ms-1200ms instead of the usual 40ms, in bursts that lasted a few seconds and then vanished.

We did what everyone does first. Checked GC logs — no long pauses. Checked the database — query times were flat. Checked the network — no retries, no timeouts upstream. Checked lock contention in the app — nothing held for more than a few microseconds. Every obvious suspect had an alibi.

The graph that was supposedly clearing us — CPU usage — was the thing lying to us.

Why it happens

Kubernetes (and Docker, and anything built on cgroups) doesn't enforce a CPU limit as "don't average more than X cores." It enforces it as a hard budget inside the CFS (Completely Fair Scheduler) bandwidth controller: every 100ms accounting period, your container gets a fixed slice of CPU-time — say 200ms of combined core-time for a "2 CPU" limit — and once that slice is spent, every thread in the container is frozen until the next period starts. No exceptions, no borrowing from a quiet neighbor period.

That period is 100ms. Your dashboard is almost certainly scraping and averaging over 15s, 30s, or 60s windows. A service can burn its entire 100ms quota in the first 8-10ms of a period — because a burst of concurrent requests briefly spun up enough threads to blow through the budget — sit frozen for the remaining ~90ms, and repeat that dozens of times a second. Averaged over a minute, that's "38% CPU usage." Inside any individual 100ms window, it's 100% then 0%, over and over, and every request that happened to land in a frozen window paid for it in latency.

This is the part that trips up even experienced engineers: CPU limits cap peak usage inside a tiny accounting window, not usage averaged over the window your monitoring actually shows you. The two numbers can point in completely opposite directions.

What to do about it

The metric almost nobody has on a dashboard is throttling, not usage. On the node:

cat /sys/fs/cgroup/cpu.stat
# nr_periods 48213
# nr_throttled 9104
# throttled_usec 182004112

nr_throttled divided by nr_periods is your throttling ratio. Ours was north of 18% — meaning nearly one in five scheduling periods, the container was frozen mid-request. If you're on Kubernetes with cAdvisor/Prometheus, the same signal is container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total:

rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])

Anything consistently above ~5% is worth investigating; above 20-25%, it's very likely a real contributor to tail latency, not a coincidence.

Once you can see it, the fixes are unglamorous but effective, roughly in order of how fast they help:

Raise the CPU limit (or remove it and rely on requests + node-level capacity). Immediate relief, costs more compute headroom.
Right-size your concurrency. If your runtime spins up more worker threads than your quota can actually sustain in a 100ms burst, you're manufacturing your own throttling. Cap thread/worker pool size to something the quota can plausibly cover.
Watch the ratio, not the average. Add the throttling metric to the same dashboard as CPU usage so the two numbers can't tell different stories unchallenged.

We ended up doing both 1 and 2 — a modest limit bump plus trimming an overeager thread pool that had been sized for a machine, not a slice of one. The throttling ratio dropped from ~18% to under 1%, and the "random" p99 spikes went with it.

Key takeaways

A CPU limit is a hard budget per 100ms accounting period, not a cap on the average your dashboard shows you.
Low average CPU usage does not rule out throttling — bursty workloads can be throttled constantly while looking idle on a 1-minute chart.
Check nr_throttled in cpu.stat (or container_cpu_cfs_throttled_seconds_total in Prometheus) directly — it's a different signal than usage, and most teams never graph it.
If concurrency inside a container regularly exceeds what its quota can sustain in one period, you're throttling yourself before any external load shows up.

Why Chaining Five Fast Services Gives You One Slow One

speed engineer — Tue, 28 Jul 2026 04:48:41 +0000

The problem

Five services, each with a P99 latency of 200ms and a comfortable P50 of 20ms. Every team's dashboard is green. Every individual SLO is met. And yet the end-to-end request that touches all five has a P99 well over a second, and nobody can find the "slow service" causing it — because on any given slow request, a different service is the culprit.

This is the conversation that repeats in incident review after incident review: "It's not us, our P99 is 200ms, look at the graph." Five teams, five graphs, five clean dashboards, one furious product owner staring at a checkout flow that blows its SLA on 1 request in 20.

Why it happens

Percentiles don't add. That's the part almost nobody's intuition gets right on the first try.

If five services are called in sequence and each is independently slow (above its own P99 threshold) 1% of the time, the probability that the whole chain stays fast — every single hop landing below its own P99 — is 0.99 raised to the fifth power, which is about 95.1%. Flip it around: roughly 4.9% of end-to-end requests will hit at least one hop having a bad day. What was a 1-in-100 event for any single service becomes something close to a 1-in-20 event for the request your user actually experiences. You didn't add latency. You added opportunities to be unlucky, and tail latency is entirely about how often you get unlucky.

Fan-out makes this worse, not better. A sequential chain waits on one slow link; a scatter-gather call — hitting N shards or N replicas in parallel and waiting for all of them — waits on the slowest of N independent draws from the same distribution. Jeff Dean and Luiz André Barroso's 2013 paper "The Tail at Scale" has the sharpest framing of this I've seen: at large fan-out (hundreds of parallel calls, common in search or ad-serving backends), the probability that at least one call lands in the tail approaches 1. Almost every request ends up waiting on somebody's P99, even though no individual server is slow very often.

Two things quietly make this worse in real systems: hops usually aren't fully independent (a shared resource — a connection pool, a downlink, a noisy-neighbor node — correlates slowness across calls that "shouldn't" be related), and a single very slow hop (a GC pause, a retry) can dominate the whole chain's latency by itself.

What to do about it

Measure the critical path, not per-service dashboards. Distributed tracing (OpenTelemetry span data is enough) shows you which hop is actually on the critical path for slow requests — not which service has the worst P99 in isolation, which is usually a different, less useful ranking.

Use hedged (backup) requests for latency-critical fan-out. Dean and Barroso's mitigation, and it's simple: if a request hasn't returned within, say, the 95th-percentile expected time, fire a second, identical request to a different replica and take whichever comes back first, cancelling the loser. This trades a small amount of duplicate work for a large cut in tail latency, because now both attempts have to be slow, not just one.

Set per-hop deadlines that shrink as they propagate. A 2-second end-to-end budget split across five sequential hops shouldn't give each hop 2 seconds to retry in — propagate a shrinking deadline so a hop that's already burned 1.5 seconds doesn't get to spend another 2 trying again.

Reduce fan-out width before you optimize per-node latency. Cutting a scatter-gather from 50 shards to 10 often does more for tail latency than shaving 20ms off each shard's P99, because you're directly attacking the exponent, not the base.

Budget for P99, not P50, at every hop you add. Each additional synchronous hop taxes your tail latency measurably even if it barely moves your average — that tax is invisible until you do the math above.

Key takeaways

Percentiles don't add across hops — probabilities of badness compound, and tail latency is a probability-of-badness problem.
A chain of N services, each bad p% of the time independently, fails end-to-end at roughly 1 - (1-p)^N of requests — far more often than any single hop's own SLO suggests.
Parallel fan-out is worse than sequential chaining for tail latency: you wait on the slowest of N draws, not just one.
Hedged/backup requests, shrinking per-hop deadlines, and narrower fan-out all directly attack this multiplication; shaving per-service P50 mostly doesn't.
Trace the critical path per slow request instead of trusting per-service dashboards — they can all be green while the request isn't.