DEV Community: speed engineer

Little's Law Explains Why Your Connection Pool Fix Actually Worked

speed engineer — Sun, 26 Jul 2026 03:44:16 +0000

The problem

Earlier this week I wrote about a team that cut their Postgres connection pool by 90% and watched P99 latency drop from 4.1s to 80ms. Every time that story comes up, someone asks the same question: how do you know the right pool size instead of just guessing and hoping?

There's a formula for this. Most engineers met it once, filed it under "queueing theory, not my problem," and never touched it again. That's a mistake — it's one of the few genuinely load-bearing formulas in systems work.

Why it happens

Little's Law: L = λW

L = average number of things in the system (requests in flight, jobs queued, connections checked out)
λ = average arrival rate (requests/sec)
W = average time each thing spends in the system (latency)

What makes it unusual is that it holds for any stable system, regardless of arrival pattern, service time distribution, or scheduling policy. It's not a statistical model with assumptions you can violate — it's closer to a conservation law, like counting cars on a highway. It only breaks if the system isn't in steady state.

Here's the part that explains the connection pool story: L isn't a free variable you get to pick independently. If λ is fixed (your traffic isn't going away) and you increase L (more concurrent connections, bigger pool), the law says W has to move too — and not always in the direction you want.

Past a certain point, more concurrency doesn't buy you more throughput; it buys you contention. Locks, cache lines, a connection limit on the database side — something starts queueing invisibly. That contention inflates W. And because L = λW isn't optional, an inflated W with steady λ means L keeps climbing: more requests in flight, not because you're serving more, but because everything is stuck waiting longer. The pool that was supposed to give the database room to work was actually the thing manufacturing the queue.

What to do about it

Use the law forward, not just as an explanation after the fact:

Measure λ — your actual arrival rate, from real traffic, not capacity-planning guesses.
Decide your target W — the latency you're actually willing to tolerate.
Solve for L — that's your target concurrency: pool size, worker count, whatever "in flight" means for your system.

If your current L is far above λ × W_target, you're not under-provisioned — you're backed up, and adding more capacity to the same contended resource will make W worse, not better. That was the trap in this week's story: the fix wasn't more connections, it was fewer, because fewer meant less contention, which meant lower W, which the law forces back into a smaller L for the same throughput.

The other use is live, during an incident. Queue depth (L) and request rate (λ) are usually easy to read off a dashboard even when your latency percentiles are lying to you — sampled, delayed, or only counting requests that finished. W = L / λ gives you a real average latency number you can trust in the middle of a page, before your APM tooling catches up.

Key takeaways

Little's Law (L = λW) holds for any stable system — no distribution assumptions required.
It's a relationship between averages, not a target to hit — you don't get to move one variable without the others responding.
More concurrency past a contention point doesn't raise throughput; it raises W, which the law forces back into a bigger L — a growing backlog dressed up as "more capacity."
It only holds in steady state — apply it to a bursty, non-equilibrium window and it'll look "wrong," when really the assumption broke, not the law.
During an incident, L and λ are often more trustworthy signals than your latency dashboard — use them to back into W directly.

The 45-Hour Week That Burns You Out Faster Than the 90-Hour One

speed engineer — Fri, 24 Jul 2026 04:41:07 +0000

The problem

Every burnout retro ends the same way: cut the meetings, cap the hours, mandate PTO. Six weeks later, the same people are just as fried.

Meanwhile you've got a founder pulling 90-hour weeks who seems to be thriving, and an engineer working a perfectly reasonable 45 who is visibly done. If hours were the mechanism, this shouldn't be possible. It is, constantly, and it means we've been treating the wrong variable.

Why it happens

The model that actually explains this is older than most of the productivity advice built on top of it. Robert Karasek published the Job Demand-Control model in 1979, and it's held up remarkably well because it doesn't measure workload on one axis — it measures two: how much is being demanded of you, and how much control (decision latitude) you have over how, when, and what you do about it.

Cross those two axes and you get four quadrants, not a line:

Low demand, high control — comfortable, low strain.
Low demand, low control — understimulated, disengaged, a different failure mode entirely.
High demand, high control — Karasek calls this an "active" job. Hard, fast-paced, and — this is the counterintuitive part — protective rather than harmful, because you're the one steering it. This is your 90-hour founder, or a senior engineer who chose the scope they're carrying.
High demand, low control — "high strain." This is where burnout actually lives. Not high demand alone. High demand with no say in how you respond to it.

Control isn't a nice-to-have that makes a hard job more pleasant. In this model it's the thing that determines whether demand turns into strain at all.

On-call is close to a textbook high-strain design if you don't deliberately engineer around it. The demand is high and unpredictable — you don't choose when the page fires. And the control is often close to zero: you can't usually decide that the root cause gets fixed before the next feature ships, you can't set your own escalation policy, and half the time you can't even declare "this alert is noise, mute it" without going through someone else. Forty-five reasonable-looking hours with zero latitude over the worst parts of them will wreck someone faster than ninety hours they actually chose.

What to do about it

This is why "just work less" so often fails as burnout advice: it only pulls the demand lever. If the actual mechanism is low control, reducing hours without touching authority leaves the high-strain structure fully intact — people get a shorter version of the same trap.

The interventions I've watched actually move the needle target control instead, often without touching hours at all:

Give the on-call engineer real authority to block new feature work until a recurring root cause is fixed — not a suggestion they can raise in a retro, actual authority to stop the line.
Let the person closest to an alert decide it's noise and mute it, instead of routing that judgment call up a chain.
Let people negotiate their own swap and escalation terms instead of inheriting a rotation designed by someone who's never been paged from it.

None of that reduces the number of hours worked. All of it reduces strain, because it changes who's steering.

A useful diagnostic for any recurring stressor, on-call or otherwise: does the person experiencing this have the authority to change what's causing it? If yes, you've got a demand problem, and reducing load will probably help. If no, you've got a control problem, and cutting hours will just give someone a smaller version of the same powerlessness.

One caveat worth keeping: Karasek and Theorell later added a third axis, social support, because a high-strain job with strong peer and manager backing is more survivable than the same job in isolation. Support doesn't replace control, but it buys people time while you fix the actual structure.

Key takeaways

Burnout correlates with low control under high demand — not with hours worked in isolation.
High demand plus high control ("active" jobs) is a different, more sustainable category than high demand plus low control ("high strain").
On-call and other interrupt-driven work is structurally high-strain unless it's deliberately paired with real authority over the interrupt.
Fixing "how many hours" without fixing "who decides" rarely works. The reverse often does, even without touching hours at all.
Diagnostic question: does the person under stress have authority to change what's causing it?

Temperature=0 Doesn't Mean Deterministic. Your Batch Size Does.

speed engineer — Wed, 22 Jul 2026 04:41:04 +0000

The problem

A team I worked with had an eval suite that called their model at temperature=0 and diffed the output against a golden completion, on the theory that greedy decoding meant identical input in, identical output out — perfect for catching regressions. Most of the time it worked. A few times a week, a completion that had passed for months would suddenly diff as "failed," with no code change, no prompt change, no model version bump. They spent real engineering hours convinced their harness had a race condition.

It didn't. The model was doing exactly what it was supposed to do. Their assumption about what temperature=0 guarantees was the bug.

Why it happens

temperature=0 makes exactly one thing deterministic: the sampling step. Instead of drawing from a probability distribution, the model always picks the highest-probability token — greedy decoding. That's it. That's the whole guarantee.

It says nothing about whether the forward pass that produces those probabilities is deterministic, and on a real GPU inference server, it usually isn't. Matmuls, attention, and normalization layers all involve summing large numbers of floating-point values, and floating-point addition is not associative — (a + b) + c can produce a different result than a + (b + c) once you're dealing with rounding error, and at model scale you're summing thousands of terms per reduction. The order those sums happen in is decided by the kernel's reduction strategy, which depends on how work gets scheduled across the GPU.

Here's the part that actually causes the flakiness: inference servers batch concurrent requests together to keep GPU utilization high. The exact set of requests riding alongside yours in a batch changes from one call to the next, based on real-time load — and that changes the batch size and shape the kernel sees, which changes the reduction order, which changes the tiny rounding error in the output logits. Normally this is invisible — a difference in the 6th decimal place of a logit doesn't change which token has the highest probability. But when the top two candidate tokens are extremely close, that noise is occasionally enough to flip the argmax. One flipped token early in a completion cascades into a completely different rest-of-sequence, because everything after it is now conditioned on a different prefix.

Your request didn't change. The model didn't change. The batch of other people's requests sharing the GPU with you at that exact millisecond did — and that was enough.

What to do about it

Stop treating temperature=0 as a promise of bit-identical output. It's a promise about the sampling step only. Forward-pass determinism is a separate, much harder property.
If you genuinely need bit-identical reproducibility — for compliance, security-sensitive audits, or debugging a subtle regression — know that it requires batch-invariant kernels: reduction implementations for matmul, attention, and normalization that are written to produce the same result regardless of batch composition. This is an active area of inference research, and early implementations that guarantee it have shown real throughput costs, with newer optimized versions cutting that overhead from roughly 60% down to around 34%. Determinism is purchasable, but it isn't free.
For everyday eval and regression suites, design for tolerance instead of exact match: compare semantic equivalence, run each case N times and check for stability within a distribution, or flag "near-tie" completions (where the model's top-2 token probabilities were close) as inherently higher-variance rather than expecting them to byte-diff clean forever.
Treat a flipped early token as a signal, not just noise — if your eval is riding a razor-thin margin between two candidate tokens, that's useful information about how confidently your prompt is actually constraining the model, independent of the flakiness it causes.

Key takeaways

temperature=0 guarantees deterministic token selection (always argmax) — it does not guarantee a deterministic forward pass.
Floating-point addition isn't associative; GPU reduction kernels sum in an order that depends on batch composition.
Batch composition shifts with real-time server load from other concurrent requests, which is why identical calls can occasionally produce different completions.
Bit-identical determinism is achievable with batch-invariant kernels, but at a real, currently non-trivial throughput cost.
Build evals that tolerate token-level variance instead of assuming byte-for-byte reproducibility.

We Cut Our Connection Pool 90% and P99 Latency Dropped From 4.1s to 80ms

speed engineer — Tue, 21 Jul 2026 16:29:22 +0000

The problem

Every few weeks, the same alert fired: API p99 latency climbing past 4 seconds under peak traffic, timeouts cascading through the stack, and a Slack thread full of people staring at a Postgres primary that was, by every dashboard we had, nowhere near maxed out. CPU sat at 40-50%. Disk I/O was fine. Memory was fine.

The fix that "worked" every time was the same one: bump the connection pool. Someone would raise max_connections on Postgres, raise the app-side pool size, redeploy, and latency would recover for a day or two. Then it would come back worse. Over about three months we crept from a sane pool configuration to 20 app pods each holding up to 100 connections — 2,000 potential connections aimed at a 16-core primary. The outages got worse, not better, and we kept reaching for the same lever because the error message ("pool exhausted," "too many clients already") looked identical every time.

Why it happens

Postgres uses a process-per-connection model — every connection is a full OS process, not a lightweight thread. Coordinating shared state across those processes (the buffer pool, the lock manager, WAL insertion) happens through internal lightweight locks. Up to a point, adding connections adds throughput, same as adding lanes to a highway. Past that point — the concurrency "knee" — every new connection adds more coordination overhead than it adds useful work. You get more processes competing for the same finite cores, more context switching, more time spent acquiring and waiting on internal locks relative to time spent executing queries.

This is coherency delay, the term Neil Gunther's Universal Scalability Law uses for it, and it's why the curve doesn't plateau — it retrogrades. More concurrent connections can mean fewer completed queries per second, because most of those connections are idle-in-transaction or queued behind a lock, and the scheduler is burning cycles switching between hundreds of mostly-idle processes instead of finishing the handful doing real work. That's also why CPU looked fine: the bottleneck wasn't compute, it was coordination, and coordination overhead doesn't show up as "CPU busy" on a basic dashboard.

This matches what you'll see in any pgbench scaling test on a fixed-core box: throughput climbs, peaks around roughly 2-4x the core count, and then falls as you keep adding concurrent connections beyond that.

What to do about it

1. Compute a target pool size instead of guessing. A widely-cited starting heuristic (popularized by HikariCP's Brett Wooldridge, borrowed from Baron Schwartz): connections ≈ (core_count × 2) + effective_spindle_count. For a 16-core primary on NVMe storage (effective spindle count ~1), that's around 33 — call it 40 with headroom. Not 2,000.

2. Size the pool for the database, not for the fleet. The mistake was letting "how many connections does Postgres allow" be a function of "how many app pods happen to be running." Put a pooler in front — PgBouncer in transaction pooling mode is the standard choice — so the app fleet can open however many logical connections it wants against PgBouncer, while PgBouncer multiplexes them down to a small, fixed number of real backend connections. App-side scaling and database-side connection count become two separate knobs instead of one.

3. Validate with Little's Law. L = λ × W: the concurrency you need (L) is your throughput (λ) times your average latency (W). At 2,000 queries/sec and 5ms average query time, you need roughly 10 connections doing real work at any instant — not 2,000. Most teams are wildly overprovisioned on paper for concurrency they don't actually need, while starving on the throughput they do.

4. Look for idle-in-transaction connections specifically. They hold locks and snapshots without doing work, and they're usually the actual leak — "not enough connections" and "too many connections" throw the identical error message, but the fixes are opposite. Check pg_stat_activity for idle in transaction before you touch the pool size.

After we put PgBouncer in transaction mode between the fleet and the primary and capped real backend connections around 40, p99 latency went from 4.1s to roughly 80ms — same hardware, same queries, no code changes. DB CPU dropped too, because it was spending cycles executing instead of switching between mostly-idle processes.

Key takeaways

More connections isn't more capacity. Past the concurrency knee, added connections increase contention and can reduce throughput.
Low CPU utilization doesn't mean you have room for more connections — check lock waits and idle-in-transaction time, not just CPU%.
Size the pool for the database's hardware (cores + effective spindles), not for how many app instances happen to be running.
Put a transaction-mode pooler between your fleet and the database so app-side scaling and DB-side connection count scale independently.
Little's Law tells you the concurrency you actually need. Compute it before you touch the number.

Your P99 Latency Has a Suspicious 40ms Floor. Two 1980s Algorithms Are Arguing.

speed engineer — Mon, 20 Jul 2026 05:16:21 +0000

The problem

A service I worked on had a weird latency signature: P50 was fine, ~4ms. P99 wasn't a long tail, it was a plateau — a large cluster of requests sitting at almost exactly 40ms, then a normal tail above that. Not "sometimes slow." A specific number, showing up over and over, like the latency had a floor.

Nothing in the app logs explained it. No GC pauses, no lock contention, no slow query. The handler itself, timed in isolation, ran in under a millisecond. The 40ms was appearing somewhere in the network path, and it was suspiciously exact for something people kept shrugging off as "random."

It wasn't random. It was two TCP behaviors from the 1980s, both individually reasonable, quietly canceling out each other's assumptions.

Why it happens

The client was writing a request in two pieces: a small header write, then a separate write for the body a few milliseconds later — an artifact of how the client library serialized the message, header framing then payload. Nothing exotic; plenty of RPC and HTTP client code does this without thinking about it.

Nagle's algorithm (RFC 896, 1984) says: if you have unacknowledged data in flight, don't send another small segment — buffer it, and wait for either an ACK or a full MSS worth of data. It exists so an interactive session (originally: someone typing over Telnet) doesn't flood the network with one-byte packets. Reasonable, on its own.

Delayed ACK (from RFC 1122) says: a receiver doesn't have to acknowledge a segment immediately. It can hold off briefly to see if it'll have outgoing data to piggyback the ACK on, or to combine acknowledgment of multiple segments into one. Also reasonable, on its own — fewer bare ACK packets on the wire.

Put them together and you get a standoff. The client's second write (the body) gets held by Nagle, because the header segment is still unacknowledged. The server has nothing to say yet — it's still waiting for the body before it can respond — so it just delays the ACK, per policy, waiting to see if it'll have something to piggyback on. Both sides are behaving correctly by their own rules. The ACK only goes out when the server's delayed-ACK timer fires, which on Linux defaults to around 40ms (other stacks have historically used up to 200ms). The moment that ACK lands, Nagle releases the body segment, and the request finally completes.

Neither side logs an error. Nothing retries. Throughput graphs look completely normal, because this doesn't cost bandwidth — it costs latency, on a subset of requests, in a way that's invisible unless you're specifically looking at the tail or capturing packets.

What to do about it

The standard fix is TCP_NODELAY — disable Nagle's algorithm on the socket, so small writes go out immediately instead of waiting on an ACK. Almost every mainstream production HTTP client and server sets this by default now, which is exactly why this bug tends to hide in custom protocols, internal RPC layers, or "just open a raw socket" code that never inherited that default.

But TCP_NODELAY alone just trades the problem for a different one (a burst of tiny packets instead of one clean one). The more durable fix is to not create the standoff in the first place: coalesce the header and body into a single write (or a single writev) so there's only one segment in flight, with nothing for Nagle to hold back. That fixes it regardless of which side's delayed-ACK settings you don't control — useful when the other end of the connection isn't yours to configure.

If you suspect this is happening to you, the fastest confirmation is a packet capture on one affected request. You're looking for a small segment sent, silence, then an ACK arriving right before the next segment goes out — with the silence landing close to your OS's delayed-ACK timer. Application logs will never show you this; it lives entirely below the layer your code can see.

Key takeaways

A latency plateau at a suspiciously round number (40ms, 200ms) is a strong signal of a fixed timer somewhere below your application — delayed ACK is a common culprit.
Nagle's algorithm and delayed ACK are each sensible in isolation. The bug only exists in the interaction between a sender that splits small writes and a receiver that delays ACKs.
TCP_NODELAY treats the symptom. Coalescing writes so a request is never split into two small segments treats the cause.
This is invisible to bandwidth/throughput monitoring and invisible to application-level logging — you need a packet capture to actually see it.

The Novelty Tax: Why Doing the Same Great Work Twice Gets You a Lower Review

speed engineer — Fri, 17 Jul 2026 03:40:47 +0000

The problem

Two cycles ago you shipped the migration that cut p99 latency in half. Everyone said it was the best work of your career. This cycle you did it again — same rigor, same impact, a different system, a bigger number. And your rating went down.

Nobody can point to a mistake. Your manager says the work was "great, consistent with last cycle." That sentence is the whole problem, and almost no one clocks it as one.

Why it happens

Calibration committees don't score absolute output. They score trajectory — the story of change since last time. A committee sits in a room with fifteen minutes per person and a forced curve to fill. The fastest signal they can extract isn't "was this good," it's "was this different from what we already believe about this person."

If your story last cycle was "shipped the big win," this cycle's story needs to be a new big win, ideally a bigger one, or the room reads you as flat. Repeat excellence doesn't update anyone's prior, so it doesn't generate the kind of narrative that wins the argument in the room. Growth reads as signal. Consistency reads as noise.

There's a second, quieter mechanism: advocacy fatigue. The person who fought for you last cycle spent their political capital telling that story once. They don't have a fresh anecdote to spend capital on this time, and reusing the old one in the room sounds like padding, not evidence. So the strongest voice in the room for you is quieter the second time, even though the work was just as good — or better.

This is the same failure mode as recency-weighted metrics in any noisy system: the committee is measuring the derivative, not the value. A flat line at a high number gets treated like nothing happened, even though "nothing happened" at that altitude is the hard part.

What to do about it

You can't fix a broken measurement system from below, but you can stop feeding it the wrong inputs.

First, manufacture the delta yourself. If the work's absolute value is flat, change the framing before the room does it for you: scope, blast radius, who now depends on it, what broke without it. "I did the same kind of thing" and "I did the thing that is now load-bearing for four other teams" are the same work described at different resolutions — pick the one that shows change.

Second, get a new advocate into the room, not just a returning one. A different manager, skip-level, or partner team lead telling your story for the first time reads as fresh evidence, even if the underlying work is a continuation.

Third, name the trap out loud in your self-review. Write the sentence "this is the second consecutive cycle where I delivered at this level, and consistency at this altitude is the achievement" — you're pre-empting the "flat" read before the committee invents it themselves.

Fourth, if you're the one calibrating other people's cases: catch yourself doing this to someone else. Ask "would this same output, from someone I hadn't seen do it before, read as a big deal?" If yes, you're penalizing them for not surprising you, which is not a performance measure — it's a novelty tax.

Key takeaways

Calibration committees implicitly score trajectory (change), not absolute output — repeat excellence gets under-weighted because it doesn't update anyone's belief.
Advocacy fatigue compounds it: the same champion has a weaker story to tell the second time, even for equally strong work.
Counter it by reframing scope/impact to surface real delta, bringing in a fresh advocate, and naming the "flat" narrative before the room invents it.
If you sit on a calibration committee, watch for this bias in how you evaluate others — "not surprising" isn't the same as "not valuable."

Two Workers Both Held the 'Only One' Lock. The Clock Was Never Synced.

speed engineer — Thu, 16 Jul 2026 04:06:12 +0000

The problem

We had a nightly billing-reconciliation job that was only supposed to run once. To enforce that across three worker instances, we did the standard thing: grab a Redis lock with SET lock:reconcile <owner> NX PX 30000 before starting, release it when done, and let the TTL clean up after a crash.

One night it ran twice, at the same time, on two different workers. Same customer batch, processed twice, twice the reconciliation entries written. Nobody deployed anything that day. The lock code hadn't changed in months.

Why it happens

The lock logic was correct. The assumption underneath it wasn't.

PX 30000 means "expire in 30 seconds," and Redis measures that 30 seconds using its own server clock. Worker A acquires the lock at what its own local clock reads as 10:00:00.000 and plans to finish in 12 seconds, well inside the 30-second budget. But Worker A's local clock — the one it used to decide "I'm still safe, no need to renew" — was running about 9 seconds behind the Redis host's clock, thanks to an NTP daemon that had silently stopped syncing after a container restart weeks earlier. Redis's clock said the key expired at 30 seconds from its view of "now," which came 9 seconds sooner in wall-clock terms than Worker A expected.

The lock expired mid-job. Worker B's health check tried to acquire it, found it free, and started the same reconciliation run. Worker A, still working under its own (wrong) assumption that it had 18 seconds of runway left, kept going too. Two processes, one "exclusive" resource, both convinced they were the only one holding it.

This is the sharp edge of any lease-based lock (Redis, Zookeeper ephemeral nodes, DynamoDB conditional writes with TTL, etc.): the lock's validity is defined by the lock server's clock, but the thing deciding "am I still safe to keep working" is the client's clock. If those two clocks disagree by more than your safety margin, the lock can expire out from under you without either side doing anything wrong locally.

Clock drift like this is rarely dramatic. It's usually single-digit seconds, invisible in normal operation, and only bites when a job's runtime gets close to the lock's TTL — which is exactly the case that matters for a lock.

What to do about it

Three changes, in order of how much they actually helped:

Auto-renew the lock instead of trusting a fixed TTL. A background thread extends the lease every few seconds (PEXPIRE with a check-and-set on ownership) as long as the job is alive. If the process dies, renewal stops and the lock expires naturally. This removes the "did I estimate my own runtime correctly" problem entirely — you stop trying to predict duration up front.

def renew_lock(redis, key, owner, ttl_ms=30000, interval_s=5):
    while not job_done.is_set():
        # only renew if we still own it
        redis.eval(RENEW_IF_OWNER_SCRIPT, 1, key, owner, ttl_ms)
        time.sleep(interval_s)

Fence the actual write, not just the lock acquisition. Give every lock acquisition a monotonically increasing fencing token. When writing the reconciliation results, include the token, and have the write path reject any token lower than the highest one already seen. Even if two workers both believe they hold the lock briefly, only the one with the newer token's writes land.
Fix the actual clock drift. We found the NTP daemon silently failing on two of three workers — chronyc tracking showed one host 9s off and another 4s off, both marked as "not synchronized" for weeks with no alert on it. We added a monitoring check on NTP sync status itself, not just on the job outcomes.

Notice the order: fencing tokens fix the correctness problem even if clocks are never perfectly synced. Clock monitoring fixes the root cause but won't help you retroactively. You want both — the fence is your safety net, the clock fix is why you rarely need it.

Key takeaways

A lease-based lock's TTL is measured by the lock server's clock, not the client's — if they drift apart, "I still have time" can be wrong without any bug in your lock code.
Prefer renewing a lock in the background over estimating job duration up front and hoping it fits inside the TTL.
Fencing tokens on the actual write are what make a lock's mutual exclusion durable even when the lock itself briefly fails — don't rely on acquisition alone.
Monitor NTP sync status as its own signal. A host silently drifting for weeks produces zero symptoms until a job's timing happens to land on the wrong side of it.

Your AI Agent's Bill Tripled Overnight. The Prompt Cache Broke, Not the Model.

speed engineer — Wed, 15 Jul 2026 03:39:29 +0000

Nobody touched the model. Nobody touched the traffic. Nobody touched the prompts, as far as anyone could tell from the diff. And yet the API bill for our agent tripled between one deploy and the next, and average response latency to first token nearly doubled with it.

The model was fine. The prompt cache was dead — and it had been killed by a single line we added to be "helpful."

How prompt caching actually works

Most people treat prompt caching as a magic discount switch: turn it on, get cheaper calls. What it actually does is match a byte-for-byte identical prefix of your request against a cache from a previous call, up to a breakpoint you (or the SDK) declare. If the prefix matches exactly, the provider skips re-processing those tokens through the model's attention layers and charges you a fraction of the price for them — often around 10% of the input cost, with a large cut in time-to-first-token too, since the KV cache for that prefix is already computed and sitting in memory.

The keyword is exact. Not "semantically similar." Not "mostly the same." One different token anywhere before the breakpoint, and the entire prefix after that point stops matching — even if 99% of it is identical to the last call.

The mistake

Our agent's system prompt looked roughly like this:

Current time: 2026-07-15T09:14:02Z
User timezone: America/Chicago

[~18,000 tokens of tool schemas, retrieval docs, and few-shot examples — completely static across every single call]

[dynamic conversation turn]

Putting the current timestamp at the top felt reasonable — it's genuinely useful for the model to know what "now" means when reasoning about dates. But it meant the very first tokens of every request were unique, every time, down to the second. The cache breakpoint we'd set after the static block never mattered, because the match check fails at token one. That 18k-token block of tool definitions and docs — identical on every call — was being fully re-processed and fully billed, every single time, for months.

We only caught it because someone happened to log cache-hit-rate as a metric out of curiosity. It read 0%. Not "lower than expected." Zero.

The fix

Cache stability has one rule: everything static goes before the breakpoint, everything volatile goes after it. Full stop.

[STABLE PREFIX — cache breakpoint here]
  tool schemas
  retrieval docs
  few-shot examples
  system instructions that never change per-request

[VOLATILE SUFFIX]
  current time
  session id
  user's live message
  retrieved context for this specific turn

If the model needs to know the current time, inject it in the volatile suffix, right next to the user's turn — not at position zero. The reordering cost us nothing in capability. It cost us months of a bill that was 3-10x higher than it needed to be, depending on the day's call volume.

Do the math on your own traffic: if you're sending an 18k-token static block on 3,000 calls a day at full price versus ~10% cached price, that's the difference between paying full rate on roughly 54M tokens a day and paying full rate on effectively none of them. At any reasonable per-token rate, that gap is not a rounding error — it's most of your bill.

The part that will bite you even if you know this rule

Frameworks and agent SDKs will silently do this to you even when your own prompt template is written correctly. Two common ways:

Auto-injected metadata (timestamps, request IDs, trace IDs) that some middleware layer prepends before your carefully-ordered system prompt.
Non-deterministic ordering — a dict, a set, or a "for tool in available_tools" loop where the iteration order isn't guaranteed stable across process restarts, so your tool definitions serialize in a different order every time your service redeploys.

You won't see either of these by reading your prompt template, because your template is fine. You'll only catch them by diffing the actual serialized bytes sent to the API across two consecutive calls. If they're not identical up to your intended breakpoint, something upstream is injecting or reordering, and your cache hit rate is lying to you even though your code looks correct.

Treat cache-hit-rate as a first-class metric, not a nice-to-have. It isn't just a performance number — for any agent making non-trivial numbers of calls with a large static context, it is your bill.

What's something that quietly broke your cache without anyone noticing for weeks?

Cache Stampede: The Failure Mode Hiding Behind a 99% Hit Rate

speed engineer — Tue, 14 Jul 2026 03:48:17 +0000

The problem

A cache hit rate of 99% sounds like a solved problem. Most dashboards stop there. But I've watched a service with a 99.2% hit rate take its database down twice in one quarter — both times during completely unremarkable traffic. No spike, no bad deploy, no slow query anywhere else in the system. The pattern both times: a single popular cache key expired, and for about 400ms, every one of the ~3,000 requests/sec hitting that key sailed straight past the cache and landed on the database at once.

That's a cache stampede (also called a thundering herd). The 0.8% miss rate isn't spread evenly across time — it clusters at the exact moment a hot key's TTL runs out, because that's when every concurrent request checking that key misses simultaneously.

Why it happens

Caches expire keys independently, but request traffic doesn't arrive independently — it's driven by whatever's popular right now. When a key is hot enough that dozens or hundreds of requests check it within the same 50-100ms window, a normal TTL expiry doesn't produce one cache miss. It produces all of them, at once, because nothing coordinates "who's responsible for recomputing this."

Every one of those requests independently decides the same thing: not in cache, better go compute it. If "compute it" means a database query, you just turned one popular cache key into N simultaneous identical queries. If computing the value is itself expensive — a join, an aggregation, a call to a slower downstream service — the database doesn't see N requests it can absorb one at a time. It sees N copies of your worst query, simultaneously, on a key that was supposed to be protecting it from exactly this.

The 99% hit rate number hides this completely, because it's an average over the whole time window. It doesn't tell you the 1% is bursty, correlated, and concentrated on your hottest keys — the ones where a stampede does the most damage.

What to do about it

Four fixes, roughly in order of effort:

Jitter your TTLs. If every replica of a key expires at exactly the same computed time, you've synchronized your own stampede. Add randomness — ttl = base_ttl + random(0, base_ttl * 0.1) — so expiries spread out instead of landing in the same window.

Request coalescing (single-flight). When a key misses, the first request acquires a lock (in-process, or distributed via Redis SETNX) and recomputes. Every other request that misses on the same key while the recompute is in flight waits for that one result instead of triggering its own. Go's singleflight package and most modern cache client libraries have this built in — usually a one-line wrap around your fetch function, not a rewrite.

Probabilistic early recomputation. Instead of waiting for a hard expiry, let requests probabilistically recompute a key slightly before it expires, with the probability rising as the expiry approaches. One request refreshes the value early; everyone else keeps serving the still-valid cached copy. This avoids the "everyone queues behind one lock" latency spike that pure single-flight can still cause on the very first miss.

Never let a cache miss mean "block until we have a fresh value." For keys expensive enough to matter, serve a slightly stale value while a background refresh runs, instead of making the request wait on a synchronous recompute. Stale-while-revalidate turns your worst case (recompute latency, on the critical path, times N concurrent requests) into your best case (cached latency, every time).

Key takeaways

A high average hit rate can hide a stampede — look for miss clustering on your hottest keys, not just the overall rate.
The failure isn't the cache missing. It's every concurrent request treating that single miss as its own personal problem to solve, independently, at the same instant.
Jittered TTLs stop you from synchronizing your own failure.
Single-flight / request coalescing turns N simultaneous recomputes into 1.
For expensive keys, serve stale-while-revalidate instead of blocking on a synchronous recompute — a cache miss should never be visible to the end user as added latency.

False Sharing: Why Adding Threads Made Your Code Slower

speed engineer — Mon, 13 Jul 2026 04:43:40 +0000

The problem

A team I worked with had a per-thread request counter: one long per worker thread, stored in a flat array, no locks, no shared state by design. Four threads, four counters, each thread only ever touches its own slot.

They scaled from 4 threads to 16 to handle more load. Throughput didn't scale — it got worse. Per-increment latency went from about 1.2ns on 4 threads to over 18ns on 16. No new locks had been added. No logic had changed. On paper, there was zero sharing between threads. And yet the more cores they threw at it, the slower each one got.

Why it happens

The bug isn't in your code's logic — it's in the CPU's cache coherence protocol, and it's invisible if you only reason about your program at the level of variables.

x86 and ARM caches move data in fixed 64-byte chunks called cache lines. That's the actual unit of coherence traffic between cores — not your long, not your int, not your struct field. When counters[16] is a plain array of 8-byte longs, eight of them fit on a single 64-byte line. So even though thread 3 and thread 7 never touch each other's counters in your source code, their counters can physically live on the same cache line.

Every time thread 3 writes its counter, the MESI coherence protocol invalidates that entire line in every other core's cache — including thread 7's copy, even though thread 7's data on that line hasn't changed at all. Thread 7 then has to fetch the line again on its next access, at full cross-core latency instead of an L1 hit. Both threads end up fighting over line ownership (a Request-For-Ownership round trip) purely because of memory layout, not because their data is actually related. This is false sharing: logically independent data, physically sharing a coherence unit.

It's nasty specifically because a normal CPU profiler will not show it to you. A flame graph will point at the increment instruction and say "this line is hot" — which is true, but misleading. The instruction itself is one cycle. The 15+ extra nanoseconds are coherence traffic that doesn't show up as "your code," it shows up as ordinary-looking time spent on an add.

What to do about it

The fix is to make sure independent hot data can't share a line: pad or align each thread's slot to 64 bytes.

struct padded_counter {
    long value;
    char pad[64 - sizeof(long)];
} __attribute__((aligned(64)));

struct padded_counter counters[16];

Now each counter owns its own cache line. No thread's write can ever invalidate another thread's copy, because they're not on the same line anymore. In that same 16-thread test, padding took per-increment latency from ~18ns back down to ~1.3ns — roughly back to single-thread speed, just distributed across cores.

The cost is memory: you're deliberately wasting up to 56 bytes per counter to buy back coherence. For a handful of hot counters or ring-buffer indices, that trade is trivial. For a large array of per-item state, you'd only pad the specific hot fields, not the whole structure.

To actually find this in the wild, don't start from a flame graph. On Linux, perf c2c (cache-to-cache) is built for exactly this — it shows you which cache lines are bouncing between cores and which offsets within them are contended. If you see unexpectedly high cross-core cache misses on data your logic says is private, check the layout before you check the algorithm.

Key takeaways

Cache lines (64 bytes on most x86/ARM chips), not variables, are the real unit of contention — "independent" data can still collide if it's packed together in memory.
False sharing gets worse as you add threads, which makes it look like a scaling problem when it's actually a layout problem.
Standard CPU profilers/flame graphs won't show you this — the hot line looks innocent because the instruction really is cheap; the cost is coherence traffic.
perf c2c (or your platform's equivalent cache-line profiler) is the right tool, not a sampling profiler.
The fix — padding/aligning to 64 bytes — is cheap in memory and can recover an order of magnitude in throughput on hot per-thread state: counters, ring-buffer indices, spinlock flags, sharded stats.

Your Redundancy Math Assumes Independence. Production Doesn't.

speed engineer — Sun, 12 Jul 2026 03:42:01 +0000

The problem

Every redundancy calculation you've ever been handed — RAID fault tolerance, N+1 replica math, "three nines times three nines," retry-until-success — leans on one silent assumption: that the failures being multiplied together are independent events.

They usually aren't. And the gap between the math and reality is exactly where the 2 a.m. pages come from.

Why it happens

Take a RAID array everyone treats as "safe because it tolerates one drive failure." The textbook argument: if a single drive has a 2% annual failure rate, the odds of two specific drives failing at once are roughly that number squared — vanishingly small. Nobody loses sleep over a one-in-thousands event.

Here's the math on what actually happens once a drive fails and the rebuild starts:

Per-drive rebuild-window failure prob (independent):  0.00011  (~0.01%)
P(any second failure during rebuild), independent:     0.00077  (~0.08%)
Correlated conditional second-failure prob range:      0.15 - 0.35  (15-35%)
Ratio vs independent model:                            ~200x - 450x

The independent model says "any second failure during the rebuild window" is roughly a 1-in-1,300 event. But a rebuild isn't a quiet passive period — it's a full-surface read of every remaining drive, all at once, under sustained load. If those drives came from the same manufacturing batch (they almost always did — you bought them together), they share the same wear curve, the same firmware, the same thermal environment. The rebuild doesn't just wait for an independent second failure to happen to arrive — it actively creates the shared stress event that triggers one. Field data on large-array rebuilds puts that conditional risk in the 15-35% range, not 0.08%. That's not a rounding error; it's a 200-450x miss.

This is the same failure of imagination that produces retry storms (a shared timeout misconfig means retries aren't independent attempts, they're a synchronized herd), that makes "three nines of compute times three nines of network" wildly optimistic (both often go down together because they share a rack, a switch, or a cloud AZ), and that quietly invalidates a lot of fan-out latency modeling (slow servers cluster around shared causes — a noisy neighbor, a GC pause propagating through a connection pool — rather than failing as independent coin flips).

The pattern: multiplying probabilities is only valid when the events don't share a cause. In production, almost everything shares a cause if you look one layer up.

What to do about it

You can't out-math correlation, but you can design against it:

Ask "what do these share?" before you trust the math. Same batch of drives, same AZ, same upstream dependency, same deploy, same on-call engineer who fat-fingers the same runbook step — any shared factor is a channel for correlated failure. Redundancy without diversity is theater.
Break the shared cause deliberately. Stagger drive purchase batches. Spread replicas across AZs and, for anything critical, across providers. Jitter your retries so a shared trigger doesn't produce a synchronized herd. Canary and stagger deploys instead of pushing to 100% at once.
Test the correlated case, not just the independent one. Chaos-test by killing the shared dependency, not just a random node. If your DR plan has never survived "the AZ that holds your primary and your replica both go dark," you don't actually know your real fault tolerance — you know your spreadsheet's.
Treat rebuild/recovery windows as the highest-risk moment, not a safe one. RAID rebuilds, replica re-syncs, and cache warm-ups all impose the exact kind of correlated stress that turns a "safe" single failure into a double one. Budget extra caution (and extra monitoring) for that window specifically.

Key takeaways

Redundancy math (RAID, N+1, retries, five-nines stacking) is usually independence math wearing a trench coat.
Correlated failure modes — shared batch, shared rack, shared dependency, shared deploy — can push real risk 100-400x above the naive calculation.
Recovery windows (rebuilds, re-syncs) don't just wait for a second failure; they actively manufacture the shared stress that causes one.
The fix isn't better arithmetic. It's designing for diversity of failure cause, and testing the correlated scenario deliberately.

RAID Doesn't Protect You From the Corruption That Actually Gets You

speed engineer — Sat, 11 Jul 2026 06:45:22 +0000

The problem

A client's RAID 6 array did exactly what it was designed to do for eighteen months straight: survive drive failures without losing data. Meanwhile, bit rot quietly ate through their primary research database the entire time. By the time the application started throwing errors, 23% of a 2.3TB genomics dataset was corrupted — and RAID had faithfully mirrored every corrupted byte across every drive, because that's all RAID actually promises to do.

Recovery took 72 hours and cost six figures in emergency consulting and reconstruction. The failure wasn't a bad disk. It was a wrong mental model.

Why it happens

RAID protects against hardware failure: a drive dies, redundancy rebuilds it. It says nothing about data corruption that happens before a write ever reaches the disk — a bit flipped by a cosmic ray in RAM, an L2 cache glitch during computation, a firmware bug in the storage controller, a kernel bug during I/O. None of that trips RAID's failure detection, because from RAID's point of view nothing failed. The corrupted bytes get written, mirrored, and backed up with the same fidelity as good data.

This is exactly where traditional filesystems make it worse. EXT4, NTFS, and XFS checksum their own metadata — the bookkeeping that tracks where files live — but not file contents. So cp, rsync, and tar will silently propagate corrupted data through your entire backup chain, and every copy will look identical to a checksum that was never actually checking the payload.

What to do about it

The fix is checksumming at the layer where corruption can't hide: end-to-end, verified on every read, not just on write.

Filesystems built for this — ZFS, Btrfs — store a hash alongside every data block and verify it on access, not just during scheduled scrubs:

$ zfs set checksum=sha256 tank
$ zpool status -v tank
  ada0  ONLINE  0 0 3   # 3 checksum errors detected and corrected
  ada3  ONLINE  0 0 1   # 1 checksum error detected and corrected

If you can't move the whole stack to ZFS/Btrfs, application-level checksums are portable and let you choose the right trade-off for your access pattern. The trade-off is bigger than people assume — on a 1GB file:

Algorithm	Time	Notes
SHA-256	3.7s	Cryptographically secure, slowest
SHA-1	2.3s	Faster, weaker guarantees
CRC32 (hardware)	0.8s	Hardware-accelerated via dedicated CPU instructions
xxHash64	0.2s	~18x faster than SHA-256, non-cryptographic

That's not a rounding error at scale — it's the difference between checksumming being "too expensive to do everywhere" and "basically free." Modern CPUs expose a CRC32C instruction directly:

uint32_t hardware_crc32c(const void* data, size_t length) {
    const uint8_t* buffer = (const uint8_t*)data;
    uint32_t crc = 0xFFFFFFFF;
    while (length >= 8) {
        crc = _mm_crc32_u64(crc, *(const uint64_t*)buffer);
        buffer += 8; length -= 8;
    }
    return crc ^ 0xFFFFFFFF;
}

Pick your algorithm by what you're protecting: xxHash64 or hardware CRC32C for hot-path integrity checks where speed matters more than cryptographic guarantees; SHA-256 for anything where an attacker (not just cosmic rays) might want to forge a matching checksum — financial records, medical data, anything audited.

The part people skip: verify on read, not just on write. A checksum you only calculate once, at write time, protects against nothing — corruption happens after the write, sitting on disk or in transit. The value is in comparing stored-vs-current at every access.

Key takeaways

RAID protects against drive failure, not data corruption — they are different problems with different defenses.
Traditional filesystems (EXT4/NTFS/XFS) checksum metadata, not your actual data. Corruption in file contents propagates silently through every backup.
Checksum algorithm choice is a real trade-off, not a formality — hardware-accelerated CRC32C and xxHash64 are ~5-18x faster than SHA-256 on the same data.
A checksum only protects you if you verify it on every read, not just calculate it once on write.

A longer version of this piece, with the full financial-impact breakdown, first ran on Medium.