DEV Community: speed engineer

CAP Theorem Covers 0.01% of Your Database's Life. PACELC Covers the Other 99.99%.

speed engineer — Sun, 20 Sep 2026 03:39:51 +0000

The problem

Every distributed systems interview eventually gets to CAP theorem: during a network partition, you can have consistency or availability, not both. Engineers love quoting this. It sounds rigorous, it name-drops a real theorem, and it explains an outage story well.

Here's what it doesn't explain: why your "strongly consistent" database is slower than your "eventually consistent" one on a totally healthy Tuesday afternoon, with zero partitions in sight.

CAP only makes a claim about the P — the rare window when the network is actually split. For a well-run multi-region system, that might be minutes per year. The other 99.99% of the time, you're still making a consistency trade-off, on every single request, and CAP has nothing to say about it. The theorem that does is PACELC, and almost nobody outside distributed-systems papers has heard of it.

Why it happens

PACELC, coined by Daniel Abadi in 2010, says: if Partitioned, choose Availability or Consistency (that's CAP); Else, choose Latency or Consistency.

That "Else" is the part that actually governs your day-to-day production behavior. Strong consistency isn't free even when the network is fine — it costs coordination, and coordination costs round trips.

Take a quorum-based system with N=3 replicas, W=2, R=2 (Dynamo-style, or Cassandra with QUORUM consistency). To satisfy a strongly-consistent read after a write, you need overlapping quorums, which means a write has to get acknowledged by 2 of 3 nodes before it returns, and a read has to check 2 of 3 nodes and reconcile versions before it returns. If those replicas are same-AZ, that's maybe 1-2ms of extra round-trip. If they're cross-region — which is exactly when you'd want the durability guarantee most — each of those round trips is 40-150ms depending on the region pair.

Compare that to reading from the nearest single replica with no quorum check: sub-5ms, no coordination, no waiting on the slowest of N acks. Same data. Same hardware. The only difference is how much agreement you demanded before answering, and that demand is a tax you pay on every request, forever, not just during a partition.

This is why Spanner (optimizes for the C, pays the L, uses TrueTime and atomic clocks to shrink the tax as much as physics allows), DynamoDB in eventual-read mode (optimizes for L, accepts stale reads), and Cassandra with tunable consistency (lets you dial the trade-off per query) all make genuinely different, defensible choices — they're not disagreeing about CAP, they're landing in different spots on the E-L-C axis, which is a decision you make thousands of times a day, not once during an incident.

What to do about it

Stop treating "consistency" as a single on/off setting picked once at database-selection time. Instead:

Identify which specific reads actually need read-your-writes or linearizable guarantees (payment status, inventory decrement, leader election) versus which ones are fine stale by a few hundred milliseconds (a feed, a dashboard, a "last seen" timestamp).
For the first group, pay the latency tax deliberately and measure it — know your P99 quorum round-trip, don't discover it in an incident.
For the second group, actually use the weaker consistency mode your database offers. Most teams default every query to the strongest available mode "to be safe" and then wonder why a read-heavy service has a P99 three times higher than it needs.
If you're using a system with tunable consistency (Cassandra, ScyllaDB, CockroachDB's follower reads), treat the consistency level as a per-query parameter, not a cluster-wide constant.
When evaluating a new datastore, ask what it does on the "Else" branch specifically — vendors will happily tell you their partition behavior, but the everyday latency-consistency trade-off is usually buried three pages into the architecture docs.

Key takeaways

CAP theorem describes rare-event behavior (during a partition); it says nothing about normal operation.
PACELC's "Else" clause — latency vs. consistency with no partition present — governs your system's behavior essentially all the time.
Strong consistency costs real, measurable round-trip latency because it requires coordination (quorums, consensus), not because of anything partition-related.
Different databases making different CAP choices often aren't disagreeing about partitions at all — they're landing in different spots on the everyday latency/consistency trade-off.
Treat consistency level as a per-query decision informed by what that specific read or write actually needs, not a single global setting.

The ID Went Negative. The Sharding Router Didn't Notice For Six Days.

speed engineer — Thu, 17 Sep 2026 18:11:17 +0000

The problem

A payments-adjacent service I worked on used a plain AtomicInteger-backed auto-increment ID for a high-write event table. It had been running fine for two years. Then a backfill migration kicked write throughput from ~300/sec to ~4,200/sec for three weeks straight, and the counter did something nobody had modeled: it hit 2,147,483,647 and wrapped to -2,147,483,648.

No crash. No exception. No log line. The service kept issuing IDs — they just went negative.

Six days later, a nightly reconciliation job flagged that roughly 0.006% of events existed in the source stream but not in the sharded store. Not zero — worse than zero. A small, silent, growing hole.

Why it happens

The routing layer picked a shard with shard = id % NUM_SHARDS. That line had been correct for two years because id had always been positive. Once IDs went negative, the formula didn't stop working — it started returning different wrong answers depending on which service evaluated it, because languages don't agree on what negative modulo means.

Python uses floored division: the result always takes the sign of the divisor. -7 % 16 in Python is 9 — still a valid, in-range shard index.

Java, C, C++, C#, JavaScript, and Go (and Rust, by default) use truncated division: the result takes the sign of the dividend. -7 % 16 in Java is -7. Not in range. Not a valid shard.

Our ID generator was a Java service. Our routing layer was Go. Both computed id % 16 on the exact same ID and both were "correct" by their own language's rules — and both disagreed with the one thing that mattered, which was "which of our 16 real shards owns this record."

The Go router held shard clients in a map[int]*ShardClient. A negative key it had never seen didn't panic — Go just handed back the zero value for a missing map key, which in this codebase was a *ShardClient with a no-op Write() stubbed in for testing and never removed. Every write to a negative shard index silently succeeded from the caller's point of view and went nowhere.

That's the actual failure: not the overflow, and not even the sign mismatch — the fact that an impossible routing key was handled by quietly doing nothing instead of failing loudly.

What to do about it

Stop using 32-bit signed IDs for anything unbounded. At 4,200 inserts/sec, int32 gives you about six days of headroom before wraparound. int64 gives you roughly 68 years at the same rate. This is a five-minute schema change that eliminates the entire bug class.

Normalize modulo sign explicitly, or avoid modulo entirely. If you must support mixed languages or can't rule out negative inputs, use the floor-mod pattern: ((id % n) + n) % n. Better: if your shard count is a power of two, use id & (n - 1) instead of id % n. Bitwise AND operates on the two's-complement bit pattern directly, so it's immune to the sign question altogether — and it's faster than a division-based modulo to boot.

Make "unknown routing key" a hard failure. A map lookup that silently returns a zero-value client is a landmine. Write() on an unrecognized shard should panic, alert, or reject — never no-op. The cost of a loud failure is a page. The cost of a quiet one is a six-day-old, slowly growing gap in your data that a human has to notice on their own.

Instrument the counter, not just the outage. We added an alert at 80% of int32 max on every unbounded counter in the system, well before any of them are near the edge. Overflow should be a scheduled maintenance ticket, not an incident.

Key takeaways

Signed integer overflow doesn't crash — it wraps and keeps producing plausible-looking, wrong answers.
The identical id % n expression can be correct in one language and silently wrong in another, because languages disagree on the sign of negative modulo.
For power-of-two shard counts, id & (n - 1) is both faster and sign-safe — prefer it over modulo.
An unrecognized routing key should fail loudly. A silent no-op is how a bug becomes a six-day data gap instead of a two-minute page.

Amdahl's Law Predicts a Plateau. It Doesn't Explain Why We Went Slower.

speed engineer — Sun, 13 Sep 2026 08:37:22 +0000

The problem

We had a write-heavy service backed by a cache cluster, and throughput wasn't keeping up with load. The fix looked obvious: add more nodes. We went from 8 to 16 — throughput went up, as expected. Feeling good, we kept going, 16 to 24.

Throughput went down.

Not "leveled off." Down. Fewer completed requests per second with more hardware running than we had at 16 nodes. The first instinct on the call was "we shipped a regression." We hadn't. What we'd hit is a well-documented, 40-year-old-plus piece of math that most engineers never see until it bites them: retrograde scaling.

Why it happens

Everyone's heard of Amdahl's Law: speedup from parallelism is capped by the fraction of work that's inherently serial. The formula is:

Speedup(N) = 1 / (s + (1 - s) / N)

Where s is the serial fraction. As N grows, speedup approaches 1/s and flattens out. That's the ceiling everyone budgets for. It predicts a plateau.

It cannot predict a decline. If your only cost is "some work can't be parallelized," adding more workers never makes you slower — it just stops helping. So why did our throughput actively drop?

Because there's a second cost Amdahl's Law doesn't model: what nodes pay to stay consistent with each other. Dr. Neil Gunther formalized this as the Universal Scalability Law (USL), which extends Amdahl's Law with a coherency (or "crosstalk") term:

C(N) = N / (1 + α(N - 1) + βN(N - 1))

α is the same contention/serialization cost as Amdahl's s. β is new: the cost of keeping nodes coherent — cache invalidation broadcasts, distributed lock coordination, gossip protocols, leader-election chatter, quorum round trips. Critically, that coordination cost tends to scale closer to O(N²) than O(N), because every node potentially has to talk to every other node.

When β is exactly zero, USL collapses back into Amdahl's Law and you get a plateau. When β > 0, the function has a maximum, at:

N* = sqrt((1 - α) / β)

Past N*, each additional node adds more coordination overhead than it adds useful work, and total throughput falls. That's retrograde scaling, and it's exactly what we watched happen on that call.

In our case, the culprit was cache invalidation: every write triggered an invalidation broadcast to every other node in the cluster to keep reads consistent. At 8 nodes, that's 8×7=56 messages per write cycle. At 24 nodes, it's 24×23=552 — a 10x jump in coordination traffic for a 3x increase in nodes. The useful work per node grew linearly; the coordination tax grew quadratically. Past a certain point, the tax bill exceeded the income.

What to do about it

Load-test at more than two points. Two data points (before/after one scaling change) can only show you a line. You need at least three node counts to see curvature — and curvature is the whole signal for retrograde scaling.

Fit α and β instead of guessing. With three or more (N, throughput) pairs, you can fit USL's parameters with basic nonlinear regression (Gunther has published spreadsheet and R/Python tooling for this). You don't need a PhD-level setup — three good load-test runs and a regression call will tell you your predicted peak node count before you buy or provision anything past it.

Go looking for the O(N²) term specifically. In practice, β is almost always one of: cache invalidation fan-out, distributed lock/mutex contention, consensus protocol round trips (Raft/Paxos elections and log replication), or gossip-based membership protocols. If any of these exist in your system and scale with cluster size, you have a β term whether or not you've measured it yet.

Bound the coherence domain, not just the node count. If β is unavoidable (some form of consistency requires it), the fix usually isn't "add fewer nodes" — it's sharding, so that any given coordination domain stays small even as your total fleet grows. Twenty-four nodes split into six coherence domains of four each can dramatically outperform twenty-four nodes all coordinating with each other.

Key takeaways

Amdahl's Law bounds speedup from serialized work. It predicts a ceiling — never a decline.
The Universal Scalability Law adds a coherency term (β) for the cost of nodes staying consistent with each other, and that term is why throughput can actively fall as you scale out.
A peak node count, N* = sqrt((1 - α) / β), exists whenever β > 0 — and you can estimate it from as few as three load-test data points, before it costs you an incident.
If a system gets slower after you scale it out, look for O(N²) cross-node chatter first: cache invalidation, distributed locks, consensus, gossip. That's β made concrete.

The Ledger Was Off By One Cent. By Month-End It Was $340.

speed engineer — Thu, 10 Sep 2026 14:24:07 +0000

The problem

A nightly job reconciled every transaction against the daily balance and flagged anything that didn't tie out to the cent. For months it was silent. Then one Tuesday it flagged a $0.01 discrepancy on an account with about 40,000 line items. Nobody paged over a penny â€” the ticket sat in the backlog for a week.

By the following Monday, the same account was off by $2.14. Two weeks after that, $340.07. The discrepancy wasn't random noise that would cancel out â€” it only ever grew in one direction, and it grew faster the more transactions the account processed that month. That's the detail that turned "round to the nearest cent and move on" into an actual incident.

The balance was computed by summing a Python list of float transaction amounts â€” sum(amount for amount in transactions) â€” then rounding the total to two decimal places for display. Every individual amount, printed on its own, looked exactly right: 19.99, 4.50, 102.33. The bug wasn't in any single number. It was in adding them together, tens of thousands of times, in whatever order the query happened to return them.

Why it happens

float in Python (and in virtually every language) is IEEE-754 double precision: a fixed 53 bits of mantissa. Most decimal amounts â€” 0.10, 19.99 â€” have no exact binary representation, the same way 1/3 has no exact finite decimal representation. 0.10 in a double is actually stored as 0.1000000000000000055511151231257827021181583404541015625. That error is roughly 5.5e-18 per value â€” utterly invisible when you print one number, because printing rounds it back to something that looks clean.

The part most people miss: floating-point addition is not associative. (a + b) + c does not always equal a + (b + c) once rounding is involved, because each intermediate addition rounds to the nearest representable double, and which values you round first changes the final result. Summing 40,000 of these tiny errors doesn't average out to zero the way independent random noise would â€” the rounding direction correlates with the sign and magnitude of the running total, so on this workload the errors compounded in the same direction almost every time. Change the table scan order, add an index, reorder the query â€” any of it could push the running total onto a different, still-wrong, path.

This is exactly why the discrepancy only grew: it wasn't measurement noise, it was systematic error from a data type that was never designed to represent money exactly in the first place.

What to do about it

Never use float/double for currency amounts you intend to sum, compare, or reconcile. Two solid options:

Integer minor units. Store and sum everything in cents (or the smallest currency unit) as an integer â€” 1999 instead of 19.99. Integer addition has no rounding error, period. Convert to a display string only at the edge, right before rendering.

# Instead of this:
total = sum([19.99, 4.50, 102.33])  # accumulates binary rounding error

# Do this:
total_cents = sum([1999, 450, 10233])  # exact integer arithmetic
display = f"${total_cents / 100:.2f}"  # convert only for display

decimal.Decimal, if you need fractional-cent precision (tax calculations, currency conversion) or can't restructure storage to integers:

from decimal import Decimal
total = sum((Decimal("19.99"), Decimal("4.50"), Decimal("102.33")))

Decimal is exact for base-10 values because it stores digits, not binary fractions â€” but it's slower, and it's easy to accidentally reintroduce the bug by constructing a Decimal from a float (Decimal(19.99) inherits the float's existing error; Decimal("19.99") built from the string does not).

If you're stuck with floats for now â€” legacy schema, a third-party API you don't control â€” at minimum switch your summation to Kahan summation, which tracks and compensates for the running rounding error at each step. It doesn't fix the root representation problem, but it caps the error growth instead of letting it compound.

Whatever you pick, add a reconciliation test that sums a large, realistic transaction set and asserts the total matches a value computed an independent, exact way (integer cents, say) â€” not just a handful of clean round numbers in a unit test, which is exactly the kind of input that hides this bug.

Key takeaways

float/double cannot represent most decimal fractions exactly â€” the error per value is tiny, but real.
Floating-point addition is not associative; summing many floats compounds rounding error in ways that don't cancel out, especially under a consistent processing order.
Never sum currency as float. Use integer minor units (cents) or decimal.Decimal, and construct Decimal from strings, not from floats.
A bug invisible at the single-value level can still be a systemic, growing problem in aggregate â€” test reconciliation logic against large, realistic datasets, not clean round numbers.

Two Random Choices Beat One Careful One: The Load Balancer Mental Model Nobody Teaches

speed engineer — Sun, 06 Sep 2026 03:42:02 +0000

The problem

Picture a fleet of 100 identical workers behind a load balancer. You send requests uniformly at random — each request goes to a random worker, independent of the others. It sounds fair. It isn't.

If you send 100 requests to 100 workers this way, the most loaded worker doesn't get 1 request. On average, it gets around log(100) / log(log(100)) — roughly 4 to 5 requests — while plenty of workers sit idle. Send 10,000 requests to 10,000 workers and the busiest one gets over 9, not ~1. This isn't a bug in your random number generator. It's math, and it's called the "balls into bins" problem.

Most engineers have felt the symptom — a "perfectly balanced" random or round-robin-ish LB that still produces one hot node, one node pegged at 90% CPU while its siblings idle at 20% — without ever learning the mechanism. So they reach for the wrong fix: bigger instances, more replicas, a mysterious "just restart it" ritual. None of that touches the actual cause.

Why it happens

Uniform random assignment doesn't spread load evenly — it clusters, the same way random points in a room form clumps and empty patches instead of a neat grid. The formal result: if you throw n balls into n bins independently and uniformly at random, the maximum bin load is, with high probability, Θ(log n / log log n).

That's not linear in n, but it's not constant either — it grows, slowly but unboundedly, as your fleet scales. The more workers you add expecting things to smooth out, the more that log(n) term keeps producing a stubborn outlier. This is exactly why "just add more instances" often makes the imbalance more visible in absolute terms even as it helps in relative terms — the tail keeps growing with the fleet.

What to do about it

The fix has a name — the power of two choices (Azar, Broder, Karlin, Upfal, 1994; popularized for systems by Michael Mitzenmacher) — and it's absurdly cheap for how much it buys you.

Instead of picking one random worker, pick two random workers and send the request to whichever currently has less load. That's the entire algorithm. The result: the maximum load drops from Θ(log n / log log n) to Θ(log log n / log log log n) — an exponential improvement in the exponent. At n = 10,000, that's the difference between a worst node carrying ~9x the average and one carrying ~2-3x.

You've probably already used this without naming it:

Envoy's P2C (power-of-two-choices) load balancer is a built-in policy, not something you write yourself.
HAProxy's leastconn and AWS ALB's least-outstanding-requests algorithm are cousins of the same idea — they just skip the "pick two" sampling step and check global state directly, which works at small scale but gets expensive to coordinate globally at large scale (hence P2C's popularity: it needs no central coordinator).
Consistent hashing ring hot spots — the classic complaint that one shard runs hot even with a "good" hash function — are the same balls-into-bins effect. The standard mitigation, virtual nodes (100-200 vnodes per physical node), works by turning one ball into many smaller, independently-placed balls, which flattens the same log(n) tail.

The one gotcha that bites people who implement this from scratch: power of two choices needs reasonably fresh load signal. If your "current load" metric is stale — cached for 30 seconds, propagated through a slow gossip protocol — every requester samples the same stale "least loaded" node and stampedes it. You've now built a synchronized herd instead of a load balancer. The fix is either querying live local queue depth at decision time (what Envoy does) or adding jitter/randomization to which two nodes get sampled, so staleness doesn't correlate across requesters.

Key takeaways

Uniform random load balancing is not fair load balancing — balls-into-bins guarantees a growing max-load outlier as your fleet scales, Θ(log n / log log n).
Sampling two random candidates and picking the lesser-loaded one collapses that outlier to Θ(log log n / log log log n) — two lookups instead of one, no central coordinator required.
This is already built into Envoy, and it's the theoretical justification behind leastconn-style balancers you've probably deployed without reading the paper behind them.
The failure mode of power-of-two-choices itself is stale load data causing correlated stampedes — check freshness before you trust the "least loaded" signal.

The One Number That Actually Moves Your Latency (And Why Your Team Keeps Optimizing the Wrong Thing)

speed engineer — Sun, 30 Aug 2026 09:12:27 +0000

The problem

A team I worked with once burned a full sprint shaving a service from 9ms down to 6ms. Clean win, nice PR, everyone felt good. Total request latency at p99: unchanged. Not "improved slightly." Unchanged, down to the millisecond.

That service was never the problem. It was just the easiest one to fix — small codebase, one owner, an obvious N+1 query to kill. Meanwhile a lock-heavy write in a shared Postgres table, three hops downstream, was eating 280ms on the same request path, and nobody had touched it in months because it was owned by a different team and looked scary.

This happens constantly, and it has a name that most engineers know from manufacturing and never apply to their own systems: Theory of Constraints. Eli Goldratt's version is blunt — a chain is only as strong as its weakest link, and reinforcing any other link does nothing for the chain's strength. Applied to a request path: your system has exactly one bottleneck at any given moment, and improving anything that isn't the bottleneck is not "a smaller win." It's a rounding error that shows up in your commit history and nowhere else.

Why it happens

Two things make this trap easy to fall into.

First, the bottleneck is usually the least pleasant thing to fix. It's often owned by someone else, wrapped in a lock or a queue you don't fully understand, or requires a schema change instead of a code change. The 9ms service is pleasant. The 280ms lock contention is not. Teams under sprint pressure gravitate toward pleasant.

Second, most latency dashboards show you averages or per-service breakdowns, not the serial chain a single request actually walks through. If service A is 9ms and service B is 280ms but they're graphed on separate panels with separate y-axes, they look like two roughly-equal-sized problems. They are not. One of them is 97% of your controllable latency and the other is noise.

What to do about it

Goldratt's original framework has five steps, and they map onto engineering almost without translation:

Identify the constraint. Don't guess — trace one real request end-to-end (a flame graph, distributed trace, or even manual timestamps at each hop) and rank stages by wall-clock time, not by whose code it is or how ugly it looks.
Exploit the constraint. Before you architect anything new, squeeze the bottleneck itself: can that lock be shortened, that query indexed, that call made async, without touching anything else?
Subordinate everything else to it. This is the step teams skip. If service A finishes in 9ms and immediately has to wait on service B's 280ms lock, optimizing A to 3ms buys you exactly nothing — A was never the pacing item. Stop spending story points there until the constraint moves.
Elevate the constraint. If step 2 isn't enough, this is where you actually add capacity — a read replica, a cache in front of the hot table, breaking the lock's critical section apart, splitting the write path.
Repeat. Once you fix the constraint, a new one appears somewhere else in the chain. This isn't a one-time exercise; it's a loop.

The practical version of step 1, if you don't have distributed tracing yet: pick your ten slowest requests from the last day, and for each one, log the wall-clock time spent in every downstream call. Sum by destination, not by your own service boundary. The bottleneck is almost never where the on-call rotation assumes it is — it's usually invisible precisely because nobody's dashboard is shaped like the actual request path.

Key takeaways

A system has one bottleneck at a time; everything else you optimize is a rounding error on the metric that matters.
"Easy to fix" and "worth fixing" are unrelated — the bottleneck is often the ugly, shared, poorly-owned piece nobody wants to touch.
Before adding capacity anywhere, trace a real request end-to-end and rank stages by actual wall-clock time, not by service ownership.
Subordinate step 3 is the one teams skip: stop improving non-bottleneck stages, even when it feels like progress.
Fixing the constraint doesn't end the exercise — it just reveals the next one.

The TCP Checksum Passed. The Data Was Corrupted Anyway.

speed engineer — Sat, 29 Aug 2026 13:58:05 +0000

The problem

"It's fine, it's over TCP" is one of the more expensive sentences in engineering. Teams treat TCP's checksum as a data-integrity guarantee. It isn't one, and the gap between "checksum passed" and "data is correct" is exactly where silent corruption gets through.

Why it happens

TCP's checksum is a 16-bit one's-complement sum over the segment — a design from an era when the threat model was electrical noise on a cable, not a buggy NIC driver or a router with corrupted line-card memory. A 16-bit sum has a small, fixed number of possible values. Specific corruption patterns — certain multi-bit flips, certain byte-swaps — land on the same sum as the clean data and pass straight through. This isn't theoretical: Jonathan Stone and Craig Partridge's measurement study of real production traffic ("When the CRC and TCP Checksum Disagree," SIGCOMM 2000) found that a small but persistent fraction of segments arrive with corrupted payloads and checksums that say everything is fine.

Modern hardware narrows the window further. Most NICs compute the TCP checksum in hardware, before your OS's network stack — let alone your application — ever touches the bytes. That's great for throughput and bad for the mental model of "the kernel checked this for me." Anything that corrupts memory between the NIC's checksum step and your application reading the buffer is invisible to TCP, full stop.

This is the same failure shape as the storage side of this problem (the deep-dive on that is linked below): a RAID array faithfully mirroring corrupted bytes across every disk because nobody told it to verify content, only to survive a drive failure. The layer you're trusting was never designed to certify the thing you actually care about.

What to do about it

Treat TCP as "probably intact," not "guaranteed intact," anywhere correctness actually matters — financial records, replicated state, anything you'd hate to silently corrupt. The fix is the same principle as filesystem-level checksums: push verification to the two endpoints that know what "correct" means, not the layers in between.

A minimal version of this, independent of whatever transport you're on:

func sendChecked(w io.Writer, payload []byte) error {
    sum := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    header := make([]byte, 8)
    binary.LittleEndian.PutUint32(header[0:4], uint32(len(payload)))
    binary.LittleEndian.PutUint32(header[4:8], sum)
    if _, err := w.Write(append(header, payload...)); err != nil {
        return err
    }
    return nil
}

func recvChecked(r io.Reader) ([]byte, error) {
    header := make([]byte, 8)
    if _, err := io.ReadFull(r, header); err != nil {
        return nil, err
    }
    length := binary.LittleEndian.Uint32(header[0:4])
    want := binary.LittleEndian.Uint32(header[4:8])
    payload := make([]byte, length)
    if _, err := io.ReadFull(r, payload); err != nil {
        return nil, err
    }
    got := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    if got != want {
        return nil, fmt.Errorf("payload checksum mismatch: want %x got %x", want, got)
    }
    return payload, nil
}

A few things follow from that:

This is why gRPC computes its own message-level checksums on top of HTTP/2 on top of TCP. The protocol authors didn't trust the bottom layer to be the last line of defense, because it was never designed to be one.
For anything replicated or cached, checksum the object once at rest and re-verify on read — the same discipline ZFS and Btrfs apply at the filesystem layer. Corruption introduced anywhere in the path between two verifications gets caught at the next one.
"It's on TLS, so it's covered" has the identical gap. TLS's integrity check does verify what crossed the TLS boundary — but only that boundary. Corruption in application buffers before encryption, or after decryption, or introduced by a proxy that terminates and re-encrypts, is outside what TLS ever promised to catch.
Pick your algorithm for the job: CRC32C (hardware-accelerated on most modern CPUs) if you want speed, SHA-256 or BLAKE3 if you want cryptographic strength against deliberate tampering, not just accidental corruption.

Key takeaways

TCP's checksum is a coarse, best-effort noise filter, not an integrity guarantee — and real measurement studies confirm corrupted-but-checksum-valid segments do occur in production networks.
Hardware checksum offloading shrinks the window TCP actually protects, since neither the OS nor the application ever inspects the raw wire bits.
TLS's integrity guarantee has the same shape of gap: it protects its own boundary, not your application buffers on either side of it.
The fix is end-to-end verification at the layer that actually knows what "correct" means for your data — not any transport underneath it. It's the same lesson filesystem-level checksums (ZFS, Btrfs) teach at a different layer of the stack: never let an intermediate hop stand in for verification it was never designed to do.

Full deep-dive on the storage side of this — corruption sources most teams never audit, and what ZFS/Btrfs do differently from ext4/XFS — on Medium.

Your Promo Case Wasn't Judged on Its Own Merits. It Was Judged Against Whoever Went Before You.

speed engineer — Fri, 28 Aug 2026 03:40:49 +0000

The problem

Two engineers, same level, same tenure, comparable years of scope. One gets "exceeds expectations" in calibration. The other gets "meets." Same manager, same evidence packet quality, same quarter. The only real difference: the order they were discussed in a three-hour room with forty cases on the docket.

I've sat in enough calibration meetings — as the person presenting cases and as one of the raters — to know this isn't an edge case. It's the default failure mode of how leveling and performance calibration actually happens, and almost nobody names it out loud.

Why it happens

Calibration meetings are a textbook setup for anchoring bias. The first case discussed in the room sets an implicit reference point for "what exceeds looks like" and "what meets looks like" — and every case after it gets judged relative to that anchor, not against a fixed bar.

Run the math on a typical session: eight raters, forty people to calibrate, three hours. That's under 4.5 minutes per person once you subtract the inevitable tangents. Nobody has time to re-derive a rubric from first principles for case #23. They pattern-match to case #1 or #2, because that's what's fresh and vivid in working memory.

It compounds in a specific direction, too. If the first case presented is a strong, well-documented "exceeds," the bar for everyone after is dragged up — good "meets" performers start looking merely adequate by contrast. If the first case is a middling "meets," the opposite happens: the bar sags, and genuinely strong later cases don't stand out because the room's calibration is already loose. The order of presentation isn't neutral. It's load-bearing.

There's a second-order effect that makes this worse: managers who present early in their careers at a company learn (correctly, if cynically) that going first with your strongest case is a real lever. It's not gaming the system maliciously — it's responding rationally to an unstated rule nobody wrote down.

What to do about it

The fix isn't "try to be more objective." Anchoring survives good intentions; it's a property of how comparative judgment works under time pressure, not a character flaw in the raters. You have to change the structure, not the willpower.

Three things that actually work, in order of how much they cost to implement:

Independent pre-scores before the room opens. Every rater submits a written score against the written rubric — not a gut read, an evidence-backed score — before anyone talks. The discussion becomes about resolving disagreement between pre-scores, not building consensus from a blank slate live in the room. This alone kills most of the anchoring effect, because the anchor gets set individually, forty separate times, instead of once for the whole room.

Randomize presentation order every session. If order is going to have an effect no matter what, at least make the effect random instead of systematic. Don't let managers self-select who goes first. Draw it.

Write the rubric's evidence bar down before the meeting, with real examples. Not "exceeds = significant impact." A concrete example of what shipped, what scope it touched, what the blast radius of failure would have been. Cases get compared to a written example instead of to whichever case is loudest in short-term memory.

None of these are exotic. They're the same fixes structured interviewing uses to fight interviewer anchoring — write the rubric first, score independently, discuss after. Performance calibration is just structured interviewing with worse incentives to fix it, because the "customer" of a bad calibration outcome is an employee who usually never finds out why.

Key takeaways

Calibration meetings anchor hard on whichever case is discussed first — the effect is structural, not a rater character flaw.
Time pressure (minutes per case) is the mechanism: nobody has bandwidth to re-derive a bar from scratch forty times in a row.
Fix the process, not the people: independent pre-scores, randomized order, and a written evidence-based rubric before the room opens.
If your org calibrates on live-discussion consensus with no pre-scoring, the outcome for any given person depends more on scheduling than on their work — and that's worth saying out loud to whoever owns the process.

We Failed Over to a Healthy Region. The JVM Never Noticed — It Had Cached the DNS Answer Forever.

speed engineer — Thu, 27 Aug 2026 03:44:11 +0000

The problem

3:40 AM. One region's load balancers started throwing 5xx at roughly 8% of traffic â€” enough to page, not enough to look catastrophic. We did the standard move: flipped the Route 53 weighted record to send 100% of traffic to the healthy region and watched the dashboard.

Error rate didn't move. Not "improved slowly" â€” didn't move at all, for 45 minutes, on a subset of hosts that kept hammering the dead region like nothing had happened.

dig from a bastion host showed the DNS answer had updated within seconds, exactly as expected. The record was correct. The resolvers were correct. And a chunk of our fleet was still connecting to a region that no longer existed as far as DNS was concerned.

The affected hosts had one thing in common: they were long-running JVM processes that made outbound HTTP calls through Java's built-in HttpURLConnection / InetAddress resolution path, not through a client that did its own re-resolution.

Why it happens

The JVM does not use your OS resolver's TTL. It has its own DNS cache, controlled by two properties most teams never set: networkaddress.cache.ttl and networkaddress.cache.negative.ttl.

The default behavior, baked in since the early 2000s for a reason that made sense at the time (mitigating DNS-rebinding attacks against applets running under a SecurityManager), is this: if a SecurityManager is installed, successful lookups are cached forever â€” TTL of -1, meaning "never expire, never re-resolve." If no SecurityManager is installed, the JDK falls back to a default of 30 seconds, which is more reasonable but still isn't reading the actual DNS record's TTL â€” it's a hardcoded JVM constant that has nothing to do with what your DNS provider configured.

Our long-running services had a SecurityManager set (leftover from an old compliance requirement, unrelated to this code path) and had never touched networkaddress.cache.ttl in java.security. So the first successful resolution of the load balancer's hostname, made whenever that JVM process last happened to open a connection to it, was cached in-process for the lifetime of that JVM. Some of those processes had been running for eleven days. They were never going to re-resolve on their own, no matter what Route 53 said, no matter how many times dig came back clean.

This is the part that makes it a nasty bug rather than a simple misconfiguration: it's invisible under normal operation. Everything works fine for months because your load balancer's IP rarely changes. The cache only becomes a liability at the exact moment you need DNS-based failover to work â€” during an actual regional failure â€” which is the worst possible time to discover it.

What to do about it

Set the TTL explicitly and don't rely on the JDK default either way:

# In $JAVA_HOME/lib/security/java.security, or as a JVM property:
networkaddress.cache.ttl=30
networkaddress.cache.negative.ttl=10

Or per-process, without touching the shared security file:

-Dsun.net.inetaddr.ttl=30
-Dsun.net.inetaddr.negative.ttl=10

A few things worth knowing beyond just setting the number:

sun.net.inetaddr.ttl only takes effect when no SecurityManager is present â€” if you do run one, you have to set networkaddress.cache.ttl in the security policy itself, not the system property. We'd set the wrong knob on our first attempt and spent twenty minutes confused about why nothing changed.

Don't rely on DNS TTL as your only failover mechanism for anything that matters. Pair it with an active health check at the client layer â€” a connection pool that evicts dead backends, or a client-side load balancer (Envoy, a service mesh sidecar, or even a simple periodic re-resolve-and-swap in application code) that doesn't depend on any single cache expiring correctly. DNS-based failover is a blunt instrument; treat a 30-second cache as the floor of your recovery time, not the whole plan.

Test failover on a live, long-running process, not a freshly started one. A JVM that's been up for ten minutes and one that's been up for ten days can behave completely differently here, and most staging environments get restarted far more often than production ever does.

Key takeaways

The JVM caches successful DNS lookups independently of the OS and independently of the record's real TTL â€” forever, by default, if a SecurityManager is present.
This is invisible until the one moment it matters: an actual failover event.
Set networkaddress.cache.ttl explicitly; know that -Dsun.net.inetaddr.ttl is a no-op under a SecurityManager.
DNS TTL is not a failover mechanism on its own â€” pair it with active health checking at the client.
Test failover against long-lived processes, not fresh ones. That's where caches like this one hide.

Upgrading Your Embedding Model Doesn't Break RAG Loudly — It Breaks It Quietly

speed engineer — Wed, 26 Aug 2026 04:38:34 +0000

The problem

A team I was helping upgraded their embedding model to cut cost — swapped an older general-purpose embedding model for a newer, cheaper one. No schema change, no downtime, no errors in any log. Over the next three weeks, support tickets crept up: "the assistant is confidently answering with the wrong doc." Nobody connected it to the embedding swap because nothing had crashed. Retrieval doesn't throw an exception when it's wrong. It just returns the nearest vectors — and "nearest" quietly stopped meaning anything.

Why it happens

Here's the part that trips people up: embedding spaces are not portable across models. Two different embedding models can both output 1536-dimensional vectors, both be excellent, and still be totally incompatible with each other — because "dimension 47" in model A's space and "dimension 47" in model B's space encode nothing in common. Each model learns its own geometry during training, shaped by its own objective and data. There's no shared coordinate system, no translation layer, no reason two models would ever agree on what "close" means.

So when you re-embed only new documents with the new model but leave old vectors sitting in the same index — which is what happened here, because a full reindex looked expensive and "we'll backfill later" — you end up with a vector store where some entries speak model A and some speak model B. A query embedded with model B gets compared against both. Against the model-B vectors, cosine similarity is meaningful. Against the model-A vectors, it's closer to noise — sometimes high, sometimes low, with no reliable relationship to actual semantic relevance.

I ran a quick sanity check to see how bad "noise" actually looks in practice:

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# same-model vectors for related concepts cluster tight and high
same_model_sim = 0.83   # typical for genuinely related text, same model

# cross-model comparison: query embedded with model B,
# candidate embedded (weeks ago) with model A
cross_model_sims = [0.71, 0.44, 0.79, 0.52, 0.68]  # no relationship to actual relevance

The cross-model numbers aren't uniformly bad — that's the trap. Some land high by coincidence, which is worse than all of them landing low, because a high score that means nothing still gets retrieved with confidence and handed straight to your LLM as "relevant context." The model doesn't hesitate on garbage context. It writes a fluent, confident answer built on a document that was never actually related to the question.

What to do about it

Treat an embedding model change like a schema migration, not a config tweak. A few things that actually hold up in production:

Full reindex, not incremental backfill. If the model changes, every vector in that index needs to be re-embedded with it. Partial migrations are the exact failure mode above — a two-model index that looks fine and silently isn't.
Version-tag every vector's metadata with the embedding model name and version. It costs one field and lets you query "how much of my index is stale" instead of guessing.
Shadow-evaluate before flipping. Stand up the new index in parallel, run a fixed eval set of real queries through both, and compare retrieval@k and answer quality before it's live. This is the step that gets skipped under time pressure, and it's the one that would've caught this in an afternoon instead of three weeks of tickets.
Never mix embedding models in one index, even "temporarily." Temporary is exactly when nobody's watching.
Batch the re-embedding cost down, don't skip it — queue it, rate-limit it, run it overnight. It's still cheaper than a support queue full of confidently wrong answers.

Key takeaways

Different embedding models produce vector spaces that are not comparable, even at matching dimensions.
Mixing vectors from two models in one index doesn't fail loudly — it silently corrupts a subset of your retrieval, sometimes convincingly.
Treat embedding model upgrades as full-index migrations with a version tag and a shadow evaluation, not a drop-in model swap.

Why CPU-Based Autoscaling Makes Traffic Spikes Worse Before It Makes Them Better

speed engineer — Tue, 25 Aug 2026 05:28:11 +0000

The problem

A traffic spike hits. The Horizontal Pod Autoscaler (HPA) is watching average CPU utilization, target 70%. Requests per second triples in under a minute. Instead of smoothly adding capacity, the system does the opposite of what you'd expect: latency climbs, then error rate climbs, and pods keep getting added anyway — but too late, and too many at once. By the time things stabilize, you've paged three people and burned twenty minutes at 4x normal latency.

Nobody misconfigured anything. The autoscaler is working exactly as designed. The design is the problem.

Why it happens

Break down what HPA actually measures, and when.

1. Metric collection lag. metrics-server scrapes kubelets on an interval (commonly 15-60s), and HPA itself evaluates on its own sync period (15s by default). Your "current CPU" is already tens of seconds old by the time a scaling decision gets made on it. During a spike that doubles load in 90 seconds, that lag alone means every decision is made against traffic that no longer exists.

2. Average CPU is a saturating signal, not a demand signal. Once every existing pod is pegged near 100%, average CPU plateaus near 100% whether you're 10% over capacity or 300% over capacity. The metric that's supposed to tell HPA "how much more do I need" stops carrying that information exactly when you need it most — it can tell you you're maxed, not by how much.

3. New pods aren't instant capacity. Scheduling, image pull, readiness-probe delay, and — for anything with a warm cache or JIT — real warmup time before a pod is actually absorbing its share of load. A pod can show Running and Ready for a full minute before it's doing useful work. Meanwhile HPA's stabilization window can let it pile on more pods before the first batch has ramped up, overshooting the correction.

4. Thundering herd at the next layer down. Every new pod opens its own DB connection pool on boot. Scale from 10 pods to 40 in one HPA decision and you've just asked your database for 4x the connections in seconds — often the actual cause of the outage, not the original traffic spike. The layer you scaled to protect (compute) just attacked the layer you didn't (the database's max_connections).

What to do about it

Scale on a leading indicator, not a lagging one. Request queue depth, in-flight request count, or requests-per-second-per-pod predicts saturation before CPU does, because it moves before compute exhausts. Custom metrics via Prometheus Adapter or KEDA let HPA target these instead of, or alongside, CPU.
Don't try to out-tune the lag. Shortening the metrics window helps marginally but you're fighting collection lag with more collection, which has its own noise and cost tradeoff. Treat it as a mitigation, not a fix.
Pre-provision for known patterns. If your spike is a marketing send or a cron-triggered batch job, scheduled scaling (a KEDA cron scaler, or a plain scheduled kubectl scale) beats reactive scaling every time — you're not waiting on a metric at all.
Cap max replicas at what your downstream can actually absorb, and enforce that cap explicitly instead of discovering it as an outage. Pair it with connection pooling (PgBouncer, RDS Proxy) so a burst of new pods doesn't equal a burst of raw DB connections.
Gate readiness on real warmup, not process start. If your workload is cache- or JIT-sensitive, a readiness probe that only checks "the process is up" will route production traffic to a pod that isn't actually ready to serve it well.

Key takeaways

Reactive CPU-based autoscaling has lag from three compounding sources — metric collection interval, HPA evaluation interval, and pod startup/warmup time — and they stack, not average out.
Average CPU stops being informative exactly when you need it most, because it saturates instead of scaling with demand.
Autoscaling that isn't capacity-aware of its downstream dependencies doesn't prevent outages — it relocates them, usually straight into your database's connection pool.
The fix isn't "scale faster." It's "scale on a signal that doesn't lag, and cap scaling at what downstream can survive."

We Cut API Gateway Connections 6x With HTTP/2. One Bad Packet Then Stalled Every Request Sharing It.

speed engineer — Mon, 24 Aug 2026 03:51:59 +0000

The problem

A team I was helping migrated their API gateway's upstream connections from HTTP/1.1 to HTTP/2. The pitch was straightforward: instead of maintaining 6 parallel TCP connections per backend host (the typical HTTP/1.1 client default), multiplex everything over a single connection. Fewer connections, less TCP slow-start overhead, less TLS handshake cost, lower idle memory footprint on the backend. Benchmarks under clean conditions backed it up — p50 dropped, connection count dropped, everyone was happy.

Then a routine network blip hit — the kind that happens between availability zones a few times a month, briefly pushing packet loss to somewhere around 1-2%. Historically this cost the gateway a small, proportional hit: a couple percent of requests got slow or retried. This time, the entire gateway's tail latency spiked. Not 2% of requests — nearly all in-flight requests on the affected hosts.

Why it happens

HTTP/2's multiplexing does exactly what it promises at the HTTP layer: multiple request/response "streams" get interleaved as frames over one TCP connection, so one slow response no longer blocks the next one from starting — the classic HTTP/1.1 head-of-line blocking problem. That part genuinely works.

The problem is one layer down. TCP guarantees in-order, reliable delivery of bytes on a connection. If a single segment is lost, TCP will not hand any later bytes to the application — including bytes belonging to completely unrelated HTTP/2 streams — until the lost segment is retransmitted and the gap is filled. One dropped packet freezes the entire connection's delivery, regardless of how many logically independent streams are riding on it.

With 6 separate HTTP/1.1 connections, a lost packet on one connection only stalls the requests using that one connection — roughly 1/6 of in-flight traffic to that host. Collapse those into a single HTTP/2 connection carrying, say, 40 concurrent streams, and the same lost packet now stalls all 40. You didn't just move the head-of-line blocking problem from HTTP to TCP — you concentrated its blast radius. Fewer connections means fewer independent failure domains.

This is precisely the motivation behind HTTP/3 and QUIC: QUIC runs over UDP and implements its own per-stream loss recovery, so a lost packet affecting one stream doesn't stall the others multiplexed alongside it. HTTP/2-over-TCP structurally cannot do this, no matter how it's tuned.

What to do about it

A few things actually move the needle, in rough order of effort:

Don't collapse to exactly one connection. Most HTTP/2 client and proxy configs let you cap concurrent streams per connection and open a small number of connections per host (2-4) instead of 1. This costs back some of the overhead savings but bounds the blast radius of a single loss event — it's a direct trade of connection overhead against blocking risk.
Measure loss, not just latency, on the paths that matter. Most teams monitor p50/p99 and CPU, and few monitor per-path packet loss. If you'd graphed loss on the AZ-to-AZ path already, this incident would have been a two-minute diagnosis instead of a multi-hour one.
Consider QUIC/HTTP/3 for genuinely loss-prone paths — mobile-facing edges especially, where 1-3% loss is closer to normal than exceptional. It solves this at the transport layer instead of asking you to hand-tune connection counts.
Don't assume "fewer connections is strictly better." It's a real trade-off. Optimize for it deliberately instead of taking the default multiplexing pitch at face value.

Key takeaways

HTTP/2 solves head-of-line blocking at the HTTP layer, not the TCP layer — TCP's in-order delivery guarantee reintroduces it underneath.
Multiplexing more streams onto fewer connections doesn't just save overhead — it concentrates the blast radius of any single packet loss event.
Under clean-network benchmarks this never shows up. It only bites at the loss rates real production paths hit occasionally, so test — or at least monitor — under loss, not just load.
QUIC/HTTP/3 exists specifically to fix this, with per-stream loss recovery over UDP.