DEV Community: speed engineer

Two Random Choices Beat One Careful One: The Load Balancer Mental Model Nobody Teaches

speed engineer — Sun, 06 Sep 2026 03:42:02 +0000

The problem

Picture a fleet of 100 identical workers behind a load balancer. You send requests uniformly at random — each request goes to a random worker, independent of the others. It sounds fair. It isn't.

If you send 100 requests to 100 workers this way, the most loaded worker doesn't get 1 request. On average, it gets around log(100) / log(log(100)) — roughly 4 to 5 requests — while plenty of workers sit idle. Send 10,000 requests to 10,000 workers and the busiest one gets over 9, not ~1. This isn't a bug in your random number generator. It's math, and it's called the "balls into bins" problem.

Most engineers have felt the symptom — a "perfectly balanced" random or round-robin-ish LB that still produces one hot node, one node pegged at 90% CPU while its siblings idle at 20% — without ever learning the mechanism. So they reach for the wrong fix: bigger instances, more replicas, a mysterious "just restart it" ritual. None of that touches the actual cause.

Why it happens

Uniform random assignment doesn't spread load evenly — it clusters, the same way random points in a room form clumps and empty patches instead of a neat grid. The formal result: if you throw n balls into n bins independently and uniformly at random, the maximum bin load is, with high probability, Θ(log n / log log n).

That's not linear in n, but it's not constant either — it grows, slowly but unboundedly, as your fleet scales. The more workers you add expecting things to smooth out, the more that log(n) term keeps producing a stubborn outlier. This is exactly why "just add more instances" often makes the imbalance more visible in absolute terms even as it helps in relative terms — the tail keeps growing with the fleet.

What to do about it

The fix has a name — the power of two choices (Azar, Broder, Karlin, Upfal, 1994; popularized for systems by Michael Mitzenmacher) — and it's absurdly cheap for how much it buys you.

Instead of picking one random worker, pick two random workers and send the request to whichever currently has less load. That's the entire algorithm. The result: the maximum load drops from Θ(log n / log log n) to Θ(log log n / log log log n) — an exponential improvement in the exponent. At n = 10,000, that's the difference between a worst node carrying ~9x the average and one carrying ~2-3x.

You've probably already used this without naming it:

Envoy's P2C (power-of-two-choices) load balancer is a built-in policy, not something you write yourself.
HAProxy's leastconn and AWS ALB's least-outstanding-requests algorithm are cousins of the same idea — they just skip the "pick two" sampling step and check global state directly, which works at small scale but gets expensive to coordinate globally at large scale (hence P2C's popularity: it needs no central coordinator).
Consistent hashing ring hot spots — the classic complaint that one shard runs hot even with a "good" hash function — are the same balls-into-bins effect. The standard mitigation, virtual nodes (100-200 vnodes per physical node), works by turning one ball into many smaller, independently-placed balls, which flattens the same log(n) tail.

The one gotcha that bites people who implement this from scratch: power of two choices needs reasonably fresh load signal. If your "current load" metric is stale — cached for 30 seconds, propagated through a slow gossip protocol — every requester samples the same stale "least loaded" node and stampedes it. You've now built a synchronized herd instead of a load balancer. The fix is either querying live local queue depth at decision time (what Envoy does) or adding jitter/randomization to which two nodes get sampled, so staleness doesn't correlate across requesters.

Key takeaways

Uniform random load balancing is not fair load balancing — balls-into-bins guarantees a growing max-load outlier as your fleet scales, Θ(log n / log log n).
Sampling two random candidates and picking the lesser-loaded one collapses that outlier to Θ(log log n / log log log n) — two lookups instead of one, no central coordinator required.
This is already built into Envoy, and it's the theoretical justification behind leastconn-style balancers you've probably deployed without reading the paper behind them.
The failure mode of power-of-two-choices itself is stale load data causing correlated stampedes — check freshness before you trust the "least loaded" signal.

The One Number That Actually Moves Your Latency (And Why Your Team Keeps Optimizing the Wrong Thing)

speed engineer — Sun, 30 Aug 2026 09:12:27 +0000

The problem

A team I worked with once burned a full sprint shaving a service from 9ms down to 6ms. Clean win, nice PR, everyone felt good. Total request latency at p99: unchanged. Not "improved slightly." Unchanged, down to the millisecond.

That service was never the problem. It was just the easiest one to fix — small codebase, one owner, an obvious N+1 query to kill. Meanwhile a lock-heavy write in a shared Postgres table, three hops downstream, was eating 280ms on the same request path, and nobody had touched it in months because it was owned by a different team and looked scary.

This happens constantly, and it has a name that most engineers know from manufacturing and never apply to their own systems: Theory of Constraints. Eli Goldratt's version is blunt — a chain is only as strong as its weakest link, and reinforcing any other link does nothing for the chain's strength. Applied to a request path: your system has exactly one bottleneck at any given moment, and improving anything that isn't the bottleneck is not "a smaller win." It's a rounding error that shows up in your commit history and nowhere else.

Why it happens

Two things make this trap easy to fall into.

First, the bottleneck is usually the least pleasant thing to fix. It's often owned by someone else, wrapped in a lock or a queue you don't fully understand, or requires a schema change instead of a code change. The 9ms service is pleasant. The 280ms lock contention is not. Teams under sprint pressure gravitate toward pleasant.

Second, most latency dashboards show you averages or per-service breakdowns, not the serial chain a single request actually walks through. If service A is 9ms and service B is 280ms but they're graphed on separate panels with separate y-axes, they look like two roughly-equal-sized problems. They are not. One of them is 97% of your controllable latency and the other is noise.

What to do about it

Goldratt's original framework has five steps, and they map onto engineering almost without translation:

Identify the constraint. Don't guess — trace one real request end-to-end (a flame graph, distributed trace, or even manual timestamps at each hop) and rank stages by wall-clock time, not by whose code it is or how ugly it looks.
Exploit the constraint. Before you architect anything new, squeeze the bottleneck itself: can that lock be shortened, that query indexed, that call made async, without touching anything else?
Subordinate everything else to it. This is the step teams skip. If service A finishes in 9ms and immediately has to wait on service B's 280ms lock, optimizing A to 3ms buys you exactly nothing — A was never the pacing item. Stop spending story points there until the constraint moves.
Elevate the constraint. If step 2 isn't enough, this is where you actually add capacity — a read replica, a cache in front of the hot table, breaking the lock's critical section apart, splitting the write path.
Repeat. Once you fix the constraint, a new one appears somewhere else in the chain. This isn't a one-time exercise; it's a loop.

The practical version of step 1, if you don't have distributed tracing yet: pick your ten slowest requests from the last day, and for each one, log the wall-clock time spent in every downstream call. Sum by destination, not by your own service boundary. The bottleneck is almost never where the on-call rotation assumes it is — it's usually invisible precisely because nobody's dashboard is shaped like the actual request path.

Key takeaways

A system has one bottleneck at a time; everything else you optimize is a rounding error on the metric that matters.
"Easy to fix" and "worth fixing" are unrelated — the bottleneck is often the ugly, shared, poorly-owned piece nobody wants to touch.
Before adding capacity anywhere, trace a real request end-to-end and rank stages by actual wall-clock time, not by service ownership.
Subordinate step 3 is the one teams skip: stop improving non-bottleneck stages, even when it feels like progress.
Fixing the constraint doesn't end the exercise — it just reveals the next one.

The TCP Checksum Passed. The Data Was Corrupted Anyway.

speed engineer — Sat, 29 Aug 2026 13:58:05 +0000

The problem

"It's fine, it's over TCP" is one of the more expensive sentences in engineering. Teams treat TCP's checksum as a data-integrity guarantee. It isn't one, and the gap between "checksum passed" and "data is correct" is exactly where silent corruption gets through.

Why it happens

TCP's checksum is a 16-bit one's-complement sum over the segment — a design from an era when the threat model was electrical noise on a cable, not a buggy NIC driver or a router with corrupted line-card memory. A 16-bit sum has a small, fixed number of possible values. Specific corruption patterns — certain multi-bit flips, certain byte-swaps — land on the same sum as the clean data and pass straight through. This isn't theoretical: Jonathan Stone and Craig Partridge's measurement study of real production traffic ("When the CRC and TCP Checksum Disagree," SIGCOMM 2000) found that a small but persistent fraction of segments arrive with corrupted payloads and checksums that say everything is fine.

Modern hardware narrows the window further. Most NICs compute the TCP checksum in hardware, before your OS's network stack — let alone your application — ever touches the bytes. That's great for throughput and bad for the mental model of "the kernel checked this for me." Anything that corrupts memory between the NIC's checksum step and your application reading the buffer is invisible to TCP, full stop.

This is the same failure shape as the storage side of this problem (the deep-dive on that is linked below): a RAID array faithfully mirroring corrupted bytes across every disk because nobody told it to verify content, only to survive a drive failure. The layer you're trusting was never designed to certify the thing you actually care about.

What to do about it

Treat TCP as "probably intact," not "guaranteed intact," anywhere correctness actually matters — financial records, replicated state, anything you'd hate to silently corrupt. The fix is the same principle as filesystem-level checksums: push verification to the two endpoints that know what "correct" means, not the layers in between.

A minimal version of this, independent of whatever transport you're on:

func sendChecked(w io.Writer, payload []byte) error {
    sum := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    header := make([]byte, 8)
    binary.LittleEndian.PutUint32(header[0:4], uint32(len(payload)))
    binary.LittleEndian.PutUint32(header[4:8], sum)
    if _, err := w.Write(append(header, payload...)); err != nil {
        return err
    }
    return nil
}

func recvChecked(r io.Reader) ([]byte, error) {
    header := make([]byte, 8)
    if _, err := io.ReadFull(r, header); err != nil {
        return nil, err
    }
    length := binary.LittleEndian.Uint32(header[0:4])
    want := binary.LittleEndian.Uint32(header[4:8])
    payload := make([]byte, length)
    if _, err := io.ReadFull(r, payload); err != nil {
        return nil, err
    }
    got := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    if got != want {
        return nil, fmt.Errorf("payload checksum mismatch: want %x got %x", want, got)
    }
    return payload, nil
}

A few things follow from that:

This is why gRPC computes its own message-level checksums on top of HTTP/2 on top of TCP. The protocol authors didn't trust the bottom layer to be the last line of defense, because it was never designed to be one.
For anything replicated or cached, checksum the object once at rest and re-verify on read — the same discipline ZFS and Btrfs apply at the filesystem layer. Corruption introduced anywhere in the path between two verifications gets caught at the next one.
"It's on TLS, so it's covered" has the identical gap. TLS's integrity check does verify what crossed the TLS boundary — but only that boundary. Corruption in application buffers before encryption, or after decryption, or introduced by a proxy that terminates and re-encrypts, is outside what TLS ever promised to catch.
Pick your algorithm for the job: CRC32C (hardware-accelerated on most modern CPUs) if you want speed, SHA-256 or BLAKE3 if you want cryptographic strength against deliberate tampering, not just accidental corruption.

Key takeaways

TCP's checksum is a coarse, best-effort noise filter, not an integrity guarantee — and real measurement studies confirm corrupted-but-checksum-valid segments do occur in production networks.
Hardware checksum offloading shrinks the window TCP actually protects, since neither the OS nor the application ever inspects the raw wire bits.
TLS's integrity guarantee has the same shape of gap: it protects its own boundary, not your application buffers on either side of it.
The fix is end-to-end verification at the layer that actually knows what "correct" means for your data — not any transport underneath it. It's the same lesson filesystem-level checksums (ZFS, Btrfs) teach at a different layer of the stack: never let an intermediate hop stand in for verification it was never designed to do.

Full deep-dive on the storage side of this — corruption sources most teams never audit, and what ZFS/Btrfs do differently from ext4/XFS — on Medium.

Your Promo Case Wasn't Judged on Its Own Merits. It Was Judged Against Whoever Went Before You.

speed engineer — Fri, 28 Aug 2026 03:40:49 +0000

The problem

Two engineers, same level, same tenure, comparable years of scope. One gets "exceeds expectations" in calibration. The other gets "meets." Same manager, same evidence packet quality, same quarter. The only real difference: the order they were discussed in a three-hour room with forty cases on the docket.

I've sat in enough calibration meetings — as the person presenting cases and as one of the raters — to know this isn't an edge case. It's the default failure mode of how leveling and performance calibration actually happens, and almost nobody names it out loud.

Why it happens

Calibration meetings are a textbook setup for anchoring bias. The first case discussed in the room sets an implicit reference point for "what exceeds looks like" and "what meets looks like" — and every case after it gets judged relative to that anchor, not against a fixed bar.

Run the math on a typical session: eight raters, forty people to calibrate, three hours. That's under 4.5 minutes per person once you subtract the inevitable tangents. Nobody has time to re-derive a rubric from first principles for case #23. They pattern-match to case #1 or #2, because that's what's fresh and vivid in working memory.

It compounds in a specific direction, too. If the first case presented is a strong, well-documented "exceeds," the bar for everyone after is dragged up — good "meets" performers start looking merely adequate by contrast. If the first case is a middling "meets," the opposite happens: the bar sags, and genuinely strong later cases don't stand out because the room's calibration is already loose. The order of presentation isn't neutral. It's load-bearing.

There's a second-order effect that makes this worse: managers who present early in their careers at a company learn (correctly, if cynically) that going first with your strongest case is a real lever. It's not gaming the system maliciously — it's responding rationally to an unstated rule nobody wrote down.

What to do about it

The fix isn't "try to be more objective." Anchoring survives good intentions; it's a property of how comparative judgment works under time pressure, not a character flaw in the raters. You have to change the structure, not the willpower.

Three things that actually work, in order of how much they cost to implement:

Independent pre-scores before the room opens. Every rater submits a written score against the written rubric — not a gut read, an evidence-backed score — before anyone talks. The discussion becomes about resolving disagreement between pre-scores, not building consensus from a blank slate live in the room. This alone kills most of the anchoring effect, because the anchor gets set individually, forty separate times, instead of once for the whole room.

Randomize presentation order every session. If order is going to have an effect no matter what, at least make the effect random instead of systematic. Don't let managers self-select who goes first. Draw it.

Write the rubric's evidence bar down before the meeting, with real examples. Not "exceeds = significant impact." A concrete example of what shipped, what scope it touched, what the blast radius of failure would have been. Cases get compared to a written example instead of to whichever case is loudest in short-term memory.

None of these are exotic. They're the same fixes structured interviewing uses to fight interviewer anchoring — write the rubric first, score independently, discuss after. Performance calibration is just structured interviewing with worse incentives to fix it, because the "customer" of a bad calibration outcome is an employee who usually never finds out why.

Key takeaways

Calibration meetings anchor hard on whichever case is discussed first — the effect is structural, not a rater character flaw.
Time pressure (minutes per case) is the mechanism: nobody has bandwidth to re-derive a bar from scratch forty times in a row.
Fix the process, not the people: independent pre-scores, randomized order, and a written evidence-based rubric before the room opens.
If your org calibrates on live-discussion consensus with no pre-scoring, the outcome for any given person depends more on scheduling than on their work — and that's worth saying out loud to whoever owns the process.

We Failed Over to a Healthy Region. The JVM Never Noticed — It Had Cached the DNS Answer Forever.

speed engineer — Thu, 27 Aug 2026 03:44:11 +0000

The problem

3:40 AM. One region's load balancers started throwing 5xx at roughly 8% of traffic â€” enough to page, not enough to look catastrophic. We did the standard move: flipped the Route 53 weighted record to send 100% of traffic to the healthy region and watched the dashboard.

Error rate didn't move. Not "improved slowly" â€” didn't move at all, for 45 minutes, on a subset of hosts that kept hammering the dead region like nothing had happened.

dig from a bastion host showed the DNS answer had updated within seconds, exactly as expected. The record was correct. The resolvers were correct. And a chunk of our fleet was still connecting to a region that no longer existed as far as DNS was concerned.

The affected hosts had one thing in common: they were long-running JVM processes that made outbound HTTP calls through Java's built-in HttpURLConnection / InetAddress resolution path, not through a client that did its own re-resolution.

Why it happens

The JVM does not use your OS resolver's TTL. It has its own DNS cache, controlled by two properties most teams never set: networkaddress.cache.ttl and networkaddress.cache.negative.ttl.

The default behavior, baked in since the early 2000s for a reason that made sense at the time (mitigating DNS-rebinding attacks against applets running under a SecurityManager), is this: if a SecurityManager is installed, successful lookups are cached forever â€” TTL of -1, meaning "never expire, never re-resolve." If no SecurityManager is installed, the JDK falls back to a default of 30 seconds, which is more reasonable but still isn't reading the actual DNS record's TTL â€” it's a hardcoded JVM constant that has nothing to do with what your DNS provider configured.

Our long-running services had a SecurityManager set (leftover from an old compliance requirement, unrelated to this code path) and had never touched networkaddress.cache.ttl in java.security. So the first successful resolution of the load balancer's hostname, made whenever that JVM process last happened to open a connection to it, was cached in-process for the lifetime of that JVM. Some of those processes had been running for eleven days. They were never going to re-resolve on their own, no matter what Route 53 said, no matter how many times dig came back clean.

This is the part that makes it a nasty bug rather than a simple misconfiguration: it's invisible under normal operation. Everything works fine for months because your load balancer's IP rarely changes. The cache only becomes a liability at the exact moment you need DNS-based failover to work â€” during an actual regional failure â€” which is the worst possible time to discover it.

What to do about it

Set the TTL explicitly and don't rely on the JDK default either way:

# In $JAVA_HOME/lib/security/java.security, or as a JVM property:
networkaddress.cache.ttl=30
networkaddress.cache.negative.ttl=10

Or per-process, without touching the shared security file:

-Dsun.net.inetaddr.ttl=30
-Dsun.net.inetaddr.negative.ttl=10

A few things worth knowing beyond just setting the number:

sun.net.inetaddr.ttl only takes effect when no SecurityManager is present â€” if you do run one, you have to set networkaddress.cache.ttl in the security policy itself, not the system property. We'd set the wrong knob on our first attempt and spent twenty minutes confused about why nothing changed.

Don't rely on DNS TTL as your only failover mechanism for anything that matters. Pair it with an active health check at the client layer â€” a connection pool that evicts dead backends, or a client-side load balancer (Envoy, a service mesh sidecar, or even a simple periodic re-resolve-and-swap in application code) that doesn't depend on any single cache expiring correctly. DNS-based failover is a blunt instrument; treat a 30-second cache as the floor of your recovery time, not the whole plan.

Test failover on a live, long-running process, not a freshly started one. A JVM that's been up for ten minutes and one that's been up for ten days can behave completely differently here, and most staging environments get restarted far more often than production ever does.

Key takeaways

The JVM caches successful DNS lookups independently of the OS and independently of the record's real TTL â€” forever, by default, if a SecurityManager is present.
This is invisible until the one moment it matters: an actual failover event.
Set networkaddress.cache.ttl explicitly; know that -Dsun.net.inetaddr.ttl is a no-op under a SecurityManager.
DNS TTL is not a failover mechanism on its own â€” pair it with active health checking at the client.
Test failover against long-lived processes, not fresh ones. That's where caches like this one hide.

Upgrading Your Embedding Model Doesn't Break RAG Loudly — It Breaks It Quietly

speed engineer — Wed, 26 Aug 2026 04:38:34 +0000

The problem

A team I was helping upgraded their embedding model to cut cost — swapped an older general-purpose embedding model for a newer, cheaper one. No schema change, no downtime, no errors in any log. Over the next three weeks, support tickets crept up: "the assistant is confidently answering with the wrong doc." Nobody connected it to the embedding swap because nothing had crashed. Retrieval doesn't throw an exception when it's wrong. It just returns the nearest vectors — and "nearest" quietly stopped meaning anything.

Why it happens

Here's the part that trips people up: embedding spaces are not portable across models. Two different embedding models can both output 1536-dimensional vectors, both be excellent, and still be totally incompatible with each other — because "dimension 47" in model A's space and "dimension 47" in model B's space encode nothing in common. Each model learns its own geometry during training, shaped by its own objective and data. There's no shared coordinate system, no translation layer, no reason two models would ever agree on what "close" means.

So when you re-embed only new documents with the new model but leave old vectors sitting in the same index — which is what happened here, because a full reindex looked expensive and "we'll backfill later" — you end up with a vector store where some entries speak model A and some speak model B. A query embedded with model B gets compared against both. Against the model-B vectors, cosine similarity is meaningful. Against the model-A vectors, it's closer to noise — sometimes high, sometimes low, with no reliable relationship to actual semantic relevance.

I ran a quick sanity check to see how bad "noise" actually looks in practice:

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# same-model vectors for related concepts cluster tight and high
same_model_sim = 0.83   # typical for genuinely related text, same model

# cross-model comparison: query embedded with model B,
# candidate embedded (weeks ago) with model A
cross_model_sims = [0.71, 0.44, 0.79, 0.52, 0.68]  # no relationship to actual relevance

The cross-model numbers aren't uniformly bad — that's the trap. Some land high by coincidence, which is worse than all of them landing low, because a high score that means nothing still gets retrieved with confidence and handed straight to your LLM as "relevant context." The model doesn't hesitate on garbage context. It writes a fluent, confident answer built on a document that was never actually related to the question.

What to do about it

Treat an embedding model change like a schema migration, not a config tweak. A few things that actually hold up in production:

Full reindex, not incremental backfill. If the model changes, every vector in that index needs to be re-embedded with it. Partial migrations are the exact failure mode above — a two-model index that looks fine and silently isn't.
Version-tag every vector's metadata with the embedding model name and version. It costs one field and lets you query "how much of my index is stale" instead of guessing.
Shadow-evaluate before flipping. Stand up the new index in parallel, run a fixed eval set of real queries through both, and compare retrieval@k and answer quality before it's live. This is the step that gets skipped under time pressure, and it's the one that would've caught this in an afternoon instead of three weeks of tickets.
Never mix embedding models in one index, even "temporarily." Temporary is exactly when nobody's watching.
Batch the re-embedding cost down, don't skip it — queue it, rate-limit it, run it overnight. It's still cheaper than a support queue full of confidently wrong answers.

Key takeaways

Different embedding models produce vector spaces that are not comparable, even at matching dimensions.
Mixing vectors from two models in one index doesn't fail loudly — it silently corrupts a subset of your retrieval, sometimes convincingly.
Treat embedding model upgrades as full-index migrations with a version tag and a shadow evaluation, not a drop-in model swap.

Why CPU-Based Autoscaling Makes Traffic Spikes Worse Before It Makes Them Better

speed engineer — Tue, 25 Aug 2026 05:28:11 +0000

The problem

A traffic spike hits. The Horizontal Pod Autoscaler (HPA) is watching average CPU utilization, target 70%. Requests per second triples in under a minute. Instead of smoothly adding capacity, the system does the opposite of what you'd expect: latency climbs, then error rate climbs, and pods keep getting added anyway — but too late, and too many at once. By the time things stabilize, you've paged three people and burned twenty minutes at 4x normal latency.

Nobody misconfigured anything. The autoscaler is working exactly as designed. The design is the problem.

Why it happens

Break down what HPA actually measures, and when.

1. Metric collection lag. metrics-server scrapes kubelets on an interval (commonly 15-60s), and HPA itself evaluates on its own sync period (15s by default). Your "current CPU" is already tens of seconds old by the time a scaling decision gets made on it. During a spike that doubles load in 90 seconds, that lag alone means every decision is made against traffic that no longer exists.

2. Average CPU is a saturating signal, not a demand signal. Once every existing pod is pegged near 100%, average CPU plateaus near 100% whether you're 10% over capacity or 300% over capacity. The metric that's supposed to tell HPA "how much more do I need" stops carrying that information exactly when you need it most — it can tell you you're maxed, not by how much.

3. New pods aren't instant capacity. Scheduling, image pull, readiness-probe delay, and — for anything with a warm cache or JIT — real warmup time before a pod is actually absorbing its share of load. A pod can show Running and Ready for a full minute before it's doing useful work. Meanwhile HPA's stabilization window can let it pile on more pods before the first batch has ramped up, overshooting the correction.

4. Thundering herd at the next layer down. Every new pod opens its own DB connection pool on boot. Scale from 10 pods to 40 in one HPA decision and you've just asked your database for 4x the connections in seconds — often the actual cause of the outage, not the original traffic spike. The layer you scaled to protect (compute) just attacked the layer you didn't (the database's max_connections).

What to do about it

Scale on a leading indicator, not a lagging one. Request queue depth, in-flight request count, or requests-per-second-per-pod predicts saturation before CPU does, because it moves before compute exhausts. Custom metrics via Prometheus Adapter or KEDA let HPA target these instead of, or alongside, CPU.
Don't try to out-tune the lag. Shortening the metrics window helps marginally but you're fighting collection lag with more collection, which has its own noise and cost tradeoff. Treat it as a mitigation, not a fix.
Pre-provision for known patterns. If your spike is a marketing send or a cron-triggered batch job, scheduled scaling (a KEDA cron scaler, or a plain scheduled kubectl scale) beats reactive scaling every time — you're not waiting on a metric at all.
Cap max replicas at what your downstream can actually absorb, and enforce that cap explicitly instead of discovering it as an outage. Pair it with connection pooling (PgBouncer, RDS Proxy) so a burst of new pods doesn't equal a burst of raw DB connections.
Gate readiness on real warmup, not process start. If your workload is cache- or JIT-sensitive, a readiness probe that only checks "the process is up" will route production traffic to a pod that isn't actually ready to serve it well.

Key takeaways

Reactive CPU-based autoscaling has lag from three compounding sources — metric collection interval, HPA evaluation interval, and pod startup/warmup time — and they stack, not average out.
Average CPU stops being informative exactly when you need it most, because it saturates instead of scaling with demand.
Autoscaling that isn't capacity-aware of its downstream dependencies doesn't prevent outages — it relocates them, usually straight into your database's connection pool.
The fix isn't "scale faster." It's "scale on a signal that doesn't lag, and cap scaling at what downstream can survive."

We Cut API Gateway Connections 6x With HTTP/2. One Bad Packet Then Stalled Every Request Sharing It.

speed engineer — Mon, 24 Aug 2026 03:51:59 +0000

The problem

A team I was helping migrated their API gateway's upstream connections from HTTP/1.1 to HTTP/2. The pitch was straightforward: instead of maintaining 6 parallel TCP connections per backend host (the typical HTTP/1.1 client default), multiplex everything over a single connection. Fewer connections, less TCP slow-start overhead, less TLS handshake cost, lower idle memory footprint on the backend. Benchmarks under clean conditions backed it up — p50 dropped, connection count dropped, everyone was happy.

Then a routine network blip hit — the kind that happens between availability zones a few times a month, briefly pushing packet loss to somewhere around 1-2%. Historically this cost the gateway a small, proportional hit: a couple percent of requests got slow or retried. This time, the entire gateway's tail latency spiked. Not 2% of requests — nearly all in-flight requests on the affected hosts.

Why it happens

HTTP/2's multiplexing does exactly what it promises at the HTTP layer: multiple request/response "streams" get interleaved as frames over one TCP connection, so one slow response no longer blocks the next one from starting — the classic HTTP/1.1 head-of-line blocking problem. That part genuinely works.

The problem is one layer down. TCP guarantees in-order, reliable delivery of bytes on a connection. If a single segment is lost, TCP will not hand any later bytes to the application — including bytes belonging to completely unrelated HTTP/2 streams — until the lost segment is retransmitted and the gap is filled. One dropped packet freezes the entire connection's delivery, regardless of how many logically independent streams are riding on it.

With 6 separate HTTP/1.1 connections, a lost packet on one connection only stalls the requests using that one connection — roughly 1/6 of in-flight traffic to that host. Collapse those into a single HTTP/2 connection carrying, say, 40 concurrent streams, and the same lost packet now stalls all 40. You didn't just move the head-of-line blocking problem from HTTP to TCP — you concentrated its blast radius. Fewer connections means fewer independent failure domains.

This is precisely the motivation behind HTTP/3 and QUIC: QUIC runs over UDP and implements its own per-stream loss recovery, so a lost packet affecting one stream doesn't stall the others multiplexed alongside it. HTTP/2-over-TCP structurally cannot do this, no matter how it's tuned.

What to do about it

A few things actually move the needle, in rough order of effort:

Don't collapse to exactly one connection. Most HTTP/2 client and proxy configs let you cap concurrent streams per connection and open a small number of connections per host (2-4) instead of 1. This costs back some of the overhead savings but bounds the blast radius of a single loss event — it's a direct trade of connection overhead against blocking risk.
Measure loss, not just latency, on the paths that matter. Most teams monitor p50/p99 and CPU, and few monitor per-path packet loss. If you'd graphed loss on the AZ-to-AZ path already, this incident would have been a two-minute diagnosis instead of a multi-hour one.
Consider QUIC/HTTP/3 for genuinely loss-prone paths — mobile-facing edges especially, where 1-3% loss is closer to normal than exceptional. It solves this at the transport layer instead of asking you to hand-tune connection counts.
Don't assume "fewer connections is strictly better." It's a real trade-off. Optimize for it deliberately instead of taking the default multiplexing pitch at face value.

Key takeaways

HTTP/2 solves head-of-line blocking at the HTTP layer, not the TCP layer — TCP's in-order delivery guarantee reintroduces it underneath.
Multiplexing more streams onto fewer connections doesn't just save overhead — it concentrates the blast radius of any single packet loss event.
Under clean-network benchmarks this never shows up. It only bites at the loss rates real production paths hit occasionally, so test — or at least monitor — under loss, not just load.
QUIC/HTTP/3 exists specifically to fix this, with per-stream loss recovery over UDP.

Little's Law: The Formula Behind Every Thread Pool You'll Ever Size

speed engineer — Sun, 23 Aug 2026 03:52:02 +0000

The problem

Most thread pool and connection pool sizes get picked by vibes: someone sets max_connections: 100 because it's a round number, copies whatever a tutorial used, or doubles the old value after an outage. It holds up for months — until traffic grows 30%, latency creeps up, and the exact same pool suddenly can't keep up, even though nothing about the code changed.

There's a one-line formula that replaces the guessing, and most engineers go their whole career without using it on purpose.

Why it happens

Little's Law, from queueing theory, states: L = λW.

L — the average number of requests in your system at any moment (being processed or waiting)
λ (lambda) — the average arrival rate (requests per second)
W — the average time each request spends in the system, start to finish (latency)

It holds for any stable system, regardless of how arrivals are distributed or how the system processes them internally. You don't need to model your service's internals to use it — you need two numbers you probably already have in a dashboard: throughput and latency.

Here's why it matters for pool sizing: your thread or connection pool has to hold at least L requests concurrently, on average, or requests start queuing behind a full pool. Most people size pools against throughput alone (say, 2,000 req/s, done) and never bring latency into the equation — but latency is exactly what turns a given throughput into a concurrency requirement.

What to do about it

Work it with real numbers. A service handling 2,000 req/s with a P50 latency of 40ms (0.04s):

L = λ × W
L = 2000 × 0.04
L = 80

On a typical request, 80 requests are in flight at once. A pool capped at 50 is structurally undersized — it will queue during completely normal traffic, not just during spikes. No amount of code optimization fixes that, because the pool itself is the bottleneck.

Here's the part that catches people: W isn't one number. If P50 is 40ms but P99 is 400ms — a 10x tail, which is common and often invisible until it's actually plotted — then during the moments P99 dominates (a downstream dependency hiccups, a GC pause runs long), real-time L spikes to roughly 800, not 80. Size for the P50 case and you're 16x under capacity exactly when it matters most, and everything downstream queues, times out, and retries into a worse pileup.

Two practical uses once this clicks:

Sizing forward — before picking a pool size, measure λ and W from real traffic (both P50 and P99), not from a guess. Size closer to the tail, with headroom, not the average case.
Diagnosing backward — if a pool that's always been fine suddenly starts queuing, don't assume traffic (λ) grew. Check W first. A single slow downstream dependency can inflate required concurrency just as much as a real traffic spike, and it's a far more common cause of a sudden capacity problem.

Key takeaways

Little's Law (L = λW) holds for any stable queue — no assumptions about arrival patterns required — so it applies to thread pools, DB connection pools, queue consumers, and request-handling capacity generally.
Pool sizing is a measurement problem: you need real λ and W from production, not a number picked in advance.
W is not a single value. P99 latency, not P50, tells you the concurrency you actually need to survive.
A capacity problem that shows up out of nowhere is more often a latency regression than a traffic spike — check W before assuming λ moved.

Cache-Aligned Isn't Zero-Copy: The Hidden memcpy in Your "Fast" Binary Protocol

speed engineer — Sat, 22 Aug 2026 05:42:27 +0000

The problem

A former colleague once showed me a binary protocol that did everything right: fields ordered by access frequency, structs padded to land exactly on 64-byte cache-line boundaries, hot fields separated from cold ones — the kind of layout work covered in Binary Protocols: Designing Messages For Cache Lines, which took a message-processing pipeline from 14,700 to 50,100 messages/sec by fixing exactly this.

Except this second system had the identical struct layout and was still slow — not "8,000 msg/sec and the CPUs are asleep" slow, but a flat 30% overhead that profiling kept blaming on "GC pressure" and "allocation churn," with no obvious smoking gun. Cache-line alignment was right there in the struct definition. So why did the allocator show up in every flame graph?

Why it happens

Alignment tells the CPU where a field will land once it's in memory. It says nothing about whether the field is read from the buffer it arrived in, or copied somewhere else first.

The common pattern: bytes arrive off the wire into a receive buffer, and the very first thing the code does is deserialize — allocate a fresh struct on the heap, and field-by-field (or via memcpy) copy the wire bytes into it. Only then does application logic touch the "nice," cache-aligned struct.

That copy reintroduces almost everything the layout work was supposed to remove:

An allocation for every message, which means allocator bookkeeping and, in managed runtimes, eventual GC pressure — the exact thing the flame graphs were pointing at.
A second full read of the same bytes — once to copy them, once to actually use them — so you pay the cache-miss cost you just engineered around, just delayed by one step instead of eliminated.
A brand-new memory address for the copy, which may or may not land on your carefully-chosen 64-byte boundary, because the allocator — not your __attribute__((aligned(64))) — decides where the copy lives. On some allocators you get lucky. On others, alignment guarantees you set at compile time quietly stop applying at runtime.

The struct in the original design is fast to access. Nothing in that design says anything about being fast to obtain. Those are two different problems, and it's easy to solve the first while never noticing you still have the second.

What to do about it

The fix is to stop deserializing and start viewing. Instead of copying wire bytes into a new object, cast a pointer directly into the receive buffer and read through it:

// Deserializing (the tax):
CacheOptimizedMessage* parse_message(const char* wire_bytes, size_t len) {
    CacheOptimizedMessage* msg = malloc(sizeof(CacheOptimizedMessage)); // allocation
    memcpy(msg, wire_bytes, sizeof(CacheOptimizedMessage));             // full copy
    return msg; // caller must remember to free() this
}

// Viewing (zero-copy):
static inline const CacheOptimizedMessage* view_message(const char* wire_bytes, size_t len) {
    if (len < sizeof(CacheOptimizedMessage)) return NULL;                  // bounds check
    if ((uintptr_t)wire_bytes % _Alignof(CacheOptimizedMessage) != 0)      // alignment check
        return NULL;  // or fall back to a copy for this one message
    return (const CacheOptimizedMessage*)wire_bytes;                       // no copy, no allocation
}

Two things make this safe rather than just fast:

Bounds-check before you cast. A view is only as safe as the length check in front of it — you're trusting the buffer's declared length matches its actual contents, so validate len before you dereference anything, especially on untrusted input.
Check alignment before you dereference, don't assume it. Receive buffers aren't always aligned the way your struct wants — network stacks, mmap, and ring buffers each have their own alignment guarantees (or lack of them). If the buffer isn't aligned, either fall back to a copy for that one message (rare path, still correct) or use an accessor that does unaligned reads on purpose, rather than relying on the compiler to save you.

This is the same principle the original article's VariableSection uses for variable-length fields — get_variable_field() returns a pointer into the existing buffer via field_offsets[], not a freshly-copied string. Extend that same idea to the fixed-size header instead of just the variable tail, and the copy disappears from the entire message, not just the flexible part.

The trade-off: a view is only valid as long as the underlying buffer is. Copy-then-mutate code gets to outlive and modify its input freely; view-then-read code has to either finish before the buffer is reused (fine for most request/response and streaming pipelines) or explicitly copy out the one or two fields it needs to keep past that point.

Key takeaways

Cache-line alignment makes a struct fast to read; it says nothing about whether you're paying to obtain it first via a hidden allocation and copy.
If your flame graph blames the allocator on a "fast" binary protocol, check whether you're deserializing into a new object instead of viewing the wire buffer in place — that's a different bottleneck than the one alignment fixes.
A zero-copy view needs a bounds check and an alignment check in front of every cast; skip either one and you've traded a performance bug for a memory-safety one.
Apply the same in-place-pointer trick your variable-length fields probably already use to the fixed-size header too — it doesn't have to stay confined to the "flexible" part of the message.

Your 4 PM Interview Slot Is Silently Grading Candidates Harder Than Your 9 AM One

speed engineer — Fri, 21 Aug 2026 04:56:35 +0000

The problem

A few years ago I pulled the numbers on a hiring loop I'd been running for two quarters: 40-some candidates, five-stage loops, four different interviewers rotating through. Same rubric, same take-home, same debug exercise. When I sorted the composite scores by which slot a candidate landed in — first interview of the day versus fourth — the fourth slot averaged almost half a point lower on our 4-point scale. Same interviewers. Same difficulty exercise, randomly assigned. The only variable that moved was what number interview it was for the person holding the clipboard.

Nobody on the team believed me at first, because it doesn't feel like something you'd do on purpose. And you wouldn't — that's the point.

Why it happens

This isn't really about interviewing. It's about sequential evaluation under cognitive load, and it's been documented outside of tech for a while — the best-known version is a 2011 study on Israeli parole boards, where judges granted parole in roughly 65% of cases right after a food break, and that rate drifted down toward near zero by the time the next break rolled around. Same cases, same law, same judges. The only thing tracking with the outcome was how long it had been since the last rest.

Interview loops run the same machinery. Every candidate forces a fresh context switch: new resume, new background, new code, a new person's nervous energy in the room. By the third or fourth switch in a day, most interviewers aren't consciously grading worse — they're unconsciously grading faster. The structured rubric gets skimmed instead of applied line by line. Borderline calls that would've gotten benefit of the doubt at 9 AM get a reflexive "meh, no" at 3 PM because the mental energy to build a real case for "yes" is gone. It shows up as harsher scoring, but the real cause is depleted evaluation effort, not a genuinely worse candidate pool later in the day.

There's a second, quieter effect stacked on top: anchoring. Interviewer four remembers interviewer three's candidate, who was strong, and grades relative to that recent memory instead of the absolute bar. A perfectly good candidate can read as "fine, I guess" purely because they were unlucky enough to follow someone great.

What to do about it

None of the fixes require new tooling, just changing how loops get scheduled and scored.

Cap interviews per person per day. We went from "however many fit on the calendar" to a hard limit of two technical interviews per interviewer per day. The scores stopped drifting within about three weeks of changing this.

Score immediately, before the next context switch. Interviewers write their score and two supporting sentences within five minutes of the candidate leaving the room — not at the end of the day, not "when I get a chance." Delayed scoring compounds the fatigue effect because you're now recalling a faded impression instead of a fresh one.

Track score against slot position, not just against candidate. Once a quarter, pull the data: average score by interview-of-the-day number, across interviewers. If slot four is reliably scoring below slot one by more than noise, that's a calibration problem, not a talent-pool problem, and it's fixable by scheduling alone.

Rotate who goes first. If the same senior interviewer always opens the loop and the same junior interviewer always closes it, you've baked a second bias directly into the schedule on top of the fatigue effect. Rotate the order across the week.

Key takeaways

Interview scores drift downward across a single evaluator's day even when candidate quality doesn't — this is a well-documented pattern in sequential judgment tasks, not unique to hiring.
The mechanism is depleted evaluation effort and anchoring on the immediately preceding candidate, not a real signal about candidate quality.
Capping interviews per interviewer per day and scoring immediately after each one are the two highest-leverage fixes.
If you've never checked whether your loop's average score correlates with slot position, it's worth one afternoon with a spreadsheet before you trust your last two quarters of hiring data.

The Container Got OOMKilled at 61% Heap Usage. Here's Why Kubernetes Wasn't Lying.

speed engineer — Thu, 20 Aug 2026 05:11:09 +0000

The problem

A service I ran was getting OOMKilled roughly twice a week. Pod restarts, a blip in the dashboards, on-call gets paged, everyone shrugs because "it auto-heals." Except it kept happening, and every time I pulled up the JVM heap graph right before the kill, it showed the same thing: 55-65% heap utilization. Not climbing. Not a leak. A perfectly healthy-looking heap, on a process the kernel had just killed for using too much memory.

That contradiction is what actually got me to stop shrugging.

Why it happens

The mental model most people carry around is "the JVM manages its own memory, so if the heap looks fine, memory is fine." That's wrong, and it's wrong in a way that costs real debugging time, because the thing enforcing your container's memory limit has never heard of "heap."

Kubernetes sets a cgroup memory limit on your pod. The kernel tracks one number against that limit: the cgroup's total resident memory â€” every byte the process has touched and is holding onto. That includes the JVM heap, yes, but also:

Thread stacks. Default stack size is often 1MB. A connection-pool-per-request pattern or an under-tuned executor that spins up 300 threads under load just quietly reserved ~300MB that no heap dashboard will ever show you.
Metaspace (class metadata) and the JIT code cache.
Direct/native ByteBuffers â€” this is the one that actually got us. We used a Netty-based client for an internal RPC path, and Netty allocates a large chunk of its buffer pool off-heap by design, specifically so GC doesn't have to touch it. Great for GC pause times. Invisible to a dashboard that only scrapes jvm.memory.heap.used.
Whatever native libraries and mmap'd files are in play.

Heap was 60% utilized. Total process RSS was creeping past the container's limit because of everything sitting outside the heap. The kernel's OOM killer doesn't negotiate â€” it doesn't wait for a GC pause, doesn't throw an OutOfMemoryError the JVM can log and handle, doesn't unwind a stack. It sends SIGKILL the instant the cgroup crosses its limit. The process is just gone. Kubernetes reports exit code 137 and "OOMKilled," and if you're only watching heap, you have zero signal about what actually happened.

What to do about it

Three things fixed this for us:

Stop sizing the heap as a fraction of host memory. Older JVMs (pre-JDK 10, or newer ones without the right flags) can miscalculate default heap sizing when they don't correctly respect cgroup limits. Set -XX:MaxRAMPercentage explicitly, and leave real headroom â€” heap should be a portion of the container limit, not the whole thing. We settled on heap at roughly 60% of the container memory limit, leaving the rest for stacks, metaspace, and off-heap buffers.
Monitor RSS, not just heap. container_memory_working_set_bytes (cAdvisor / kube-state-metrics) tracks what the kernel actually cares about. We added it next to the heap graph so the two could be compared directly â€” the gap between them became a metric we alerted on.
Count your threads and your off-heap allocators. -XX:NativeMemoryTracking=summary plus a periodic jcmd <pid> VM.native_memory will show you where non-heap memory is actually going. In our case, a connection pool with no cap on concurrent threads was the biggest single contributor once we actually measured it â€” heap monitoring alone never would have pointed there.

Key takeaways

A container gets OOMKilled based on total RSS, not JVM heap usage â€” these are different numbers, and only one of them is what your dashboard probably shows you.
Off-heap memory (thread stacks, metaspace, direct buffers, native allocations) doesn't show up in heap metrics but counts fully against your cgroup limit.
If your service gets OOMKilled with heap graphs that look calm, stop looking at the heap. Compare heap usage to container RSS â€” the delta is where your answer is.
Set an explicit heap ceiling well under the container limit, and monitor RSS directly. The kernel isn't going to give you a stack trace before it kills you.