DEV Community: Darshan Turakhia

How OpenTelemetry

Darshan Turakhia — Tue, 27 Oct 2026 11:00:52 +0000

14:32 UTC. PagerDuty fires on 5xx rate for /checkout/complete. The on-call engineer opens the trace dashboard to find the slow dependency. It's green. No red spans, no latency bump, nothing flagged in the service map. The load balancer says the checkout service is failing. Tracing says everything is fine.

the scramble

First theory: a bad deploy. The deploy log shows nothing shipped in the last six hours. Second theory: the payment gateway is slow again, it's happened before. The gateway's own status page is green and our circuit breaker metrics show zero trips. Third theory, the one everyone wants to believe: it's a blip, give it two minutes and watch it clear.

It doesn't clear. 5xx count on /checkout/complete goes from a baseline of about two per minute to 1,900 over nine minutes. A marketing push notification had gone out to 400,000 users twelve minutes earlier, and checkout traffic was running at roughly 6x normal. Nobody in the incident channel is arguing it's noise anymore. The trace dashboard, the thing built specifically to answer "where is this slow," is still just sitting there green.

the hunt

With tracing giving nothing to go on, the hunt moves to raw signals. Application logs (not spans, actual log lines) show a climbing count of a specific exception on the checkout path.



checkout service logs, grepped for the error

asyncpg.exceptions.TooManyConnectionsError: too many connections for role "checkout_svc"
(occurred 340 times in the last 6 minutes)

A connection pool exhaustion. Someone runs a query directly against Postgres to see what's actually holding connections open.



pg\_stat\_activity, filtered to the checkout role

SELECT pid, state, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE usename = 'checkout_svc' AND state != 'idle'
ORDER BY duration DESC LIMIT 5;

Dozens of connections sitting idle in transaction for four to five seconds each. The checkout handler wraps a row lock on the order table in the same transaction as a call to a third-party inventory and fraud-check API, a call that normally returns in 80ms and, under the spike's downstream load, was taking four to six seconds. Every slow inventory call held a database connection open and idle for the full duration. With a pool of 20 connections per pod across 14 pods, it didn't take many of these stacking up at once to starve everything else on the checkout path.

That explains the outage. It still doesn't explain why tracing showed nothing for nine minutes of a real, 1,900-error spike. That question gets parked for the retro, because the immediate fix (move the external call outside the transaction, add a hard timeout) stops the bleeding and the team moves on to writing the postmortem.

the find

During the retro, someone pulls the raw request count from the load balancer access logs for the 14:32 to 14:55 window and compares it against the span count ingested by Tempo for the checkout service over the same window.

94,612 Requests logged by the load balancer for the checkout service during the window

11,480 Spans actually ingested by Tempo for the same service, same window

Roughly 88% of the traffic that hit checkout during the outage has no trace at all. The gap starts almost exactly ninety seconds into the spike, right around when the queue behind the OpenTelemetry BatchSpanProcessor would have filled on its default settings.



tracing.py (before)

from opentelemetry.sdk.trace.export import BatchSpanProcessor

processor = BatchSpanProcessor(
otlp_exporter,
max_queue_size=2048,
max_export_batch_size=512,
schedule_delay_millis=5000,
)

At 6x traffic, span creation outran what the exporter could drain to the collector. The collector sat behind the same OTLP endpoint that was itself catching some of the spike's traffic, so it was draining slower right when it needed to drain faster. Once the queue hit 2048, the processor started refusing new spans outright. It logs a warning the first time this happens, at the Python logging module's WARNING level. It does not log it again for every subsequent drop, because the service's log pipeline runs a dedup filter on repeated identical lines to cut noise. The one warning that would have told anyone tracing had gone blind fired once, got suppressed as a duplicate, and the drops kept happening silently for the rest of the incident.

the fix

Two separate problems needed two separate fixes. The application bug: the inventory check moved outside the database transaction, with its own timeout, so a slow third-party call can no longer hold a connection hostage.



checkout.py

async def complete_order(order_id: str) -> OrderResult:
inventory_result = await asyncio.wait_for(
check_inventory_and_fraud(order_id), timeout=1.5
)
async with db.transaction():
order = await lock_order_for_update(order_id)
apply_inventory_result(order, inventory_result)
await order.save()
return OrderResult(order)

The observability gap needed a fix that didn't depend on anyone noticing a suppressed warning line. Instead of trusting the SDK's internal counters, the team wired an alert directly off the signal that actually caught this: the gap between load balancer request count and ingested span count for the same service and window.



alert rule (Prometheus-style)

alert: TraceCoverageGap expr: | (sum(rate(alb_requests_total{service="checkout"}[5m]))
- sum(rate(otel_spans_ingested_total{service="checkout"}[5m]))) / sum(rate(alb_requests_total{service="checkout"}[5m])) > 0.15 for: 2m annotations: summary: "Checkout requests are outrunning ingested trace coverage"

The queue itself got widened too, max_queue_size raised to 8192 and schedule_delay_millis dropped to 2000 so the exporter drains more often under burst, but the team treated that as a buffer, not a fix. A bigger queue just moves the point where it fills. The log dedup filter was also changed to let repeated WARNING-level lines from the tracing SDK through after the third occurrence instead of suppressing indefinitely.

the aftermath

A trace pipeline degrades under exactly the kind of load spike it exists to help diagnose, because span volume scales with request volume, and the exporter's throughput doesn't automatically scale with either.
A monitoring tool failing silent and green is more dangerous than one failing loud, because it actively points the hunt in the wrong direction instead of just leaving a blank.
Log dedup filters tuned to cut noise can suppress the exact one-time warning that would have shortened an incident, if that warning only fires once per condition instead of once per occurrence.
The most reliable way to catch a tracing pipeline lying isn't inside the tracing pipeline. It's comparing it against a source that doesn't share its failure mode, like raw load balancer request counts.

The checkout fix was the kind of bug that's obvious once found, a slow call sitting inside a transaction it never needed to be in. The nine minutes it took to find it without any tracing to lean on is the part that changed how the team treats observability itself: not as a tool that's either working or broken, but as a system with its own capacity limits that can fail exactly when the traffic that needs it shows up.

How an Unpinned Base Image Tag Let V8 Size Its Heap Off the Host, Not the Pod

Darshan Turakhia — Fri, 23 Oct 2026 11:21:11 +0000

14:47 UTC, Tuesday. Pods in the checkout-api deployment start OOMKilling in a loop, three or four at a time, in the middle of an ordinary afternoon. Traffic is flat. No deploy went out today. PagerDuty fires on KubePodCrashLooping, not on anything tied to a release, because as far as the deploy history shows, nothing was released.

the scramble

First theory: a memory leak shipped yesterday and took a day to build up enough pressure to matter. git log --since=48h -- apps/checkout-api shows exactly one commit, a routine lockfile refresh from the nightly dependency job. No application code changed in either direction.

Second theory: a traffic burst nobody's dashboard caught. Request rate for the service is within 3% of the same Tuesday last week. Whatever's killing these pods, it isn't load.

Third theory: a slow leak finally crossing a threshold. The service exposes a /debug/memory endpoint that dumps process.memoryUsage() on demand. heapUsed sits at a boring 140MB, same as it's sat for weeks. rss, meanwhile, is climbing steadily past 480MB on a pod with a 512Mi limit and hasn't leveled off. Whatever's growing, V8's own heap accounting doesn't think it's growing.

the hunt

A gap between heapUsed and rss usually means something outside V8's managed heap: a native addon, an unreleased buffer, a leaking file descriptor. That's the next hour, chasing a native leak. clinic doctor against a staging replay shows nothing unusual. No native module in the dependency tree changed in the lockfile refresh. Dead end.

The clue that actually breaks it open is which pods die fastest. Traffic is round-robin across all pods, cgroup memory limits are identical at 512Mi, and yet pods scheduled onto the cluster's larger nodes survive noticeably longer before OOMKilling than pods on the smaller ones. That shouldn't matter. A container's enforced limit is supposed to be the same number regardless of what the underlying node has installed.



exec into two pods, same deployment, different nodes

pod on a 16GB node

$ kubectl exec checkout-api-7f9d-a1 -- node -e \
"console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"
2187.4

pod on a 64GB node

$ kubectl exec checkout-api-7f9d-b2 -- node -e \
"console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"
8341.9

Neither number is close to 512. Both are absurd for a container capped at 512Mi, and they scale with the node's physical memory, not with the pod's cgroup limit, which is the opposite of what should happen. A canary pod still running the previous image, from before last night's rebuild, gives a third data point:



same check against the pre-rebuild canary

$ kubectl exec checkout-api-canary-prev -- node -e \
"console.log(require('v8').getHeapStatistics().heap_size_limit / 1e6)"
462.8

462MB, correctly tracking the 512Mi limit regardless of which node it lands on. The canary and the failing pods are running the same application code. The only thing that differs is the image.



diffing the two images

$ docker inspect checkout-api:canary-prev --format '{{.Config.Image}}'
node:18.19.1-alpine

$ docker inspect checkout-api:latest --format '{{.Config.Image}}'
node:18.20.4-alpine

The Dockerfile's first line has always read FROM node:18-alpine, a floating tag, never a pinned version. The nightly CI job that rebuilds this image runs docker build --pull, which fetches whatever node:18-alpine currently resolves to on every run. Somewhere between two nights ago and last night, that tag moved from 18.19.1 to 18.20.4. No commit, no changelog line, no PR. The base image changed underneath the Dockerfile without the Dockerfile changing at all.

the find

Root cause: Node determines its default heap ceiling by probing the container's cgroup memory limit through libuv, and falling back to the host's total memory if that probe fails. The 18.20.4 alpine layer repackaged its cgroup compatibility shims in a way that changed which path libuv checks first on this cluster's cgroup v2 nodes. On 18.19.1 the probe found memory.max and sized the heap off 512Mi, as intended. On 18.20.4 the probe missed and silently fell back to os.totalmem(), which reports whatever the underlying node actually has, 16GB on some nodes and 64GB on others.

V8 isn't wrong about the number it was given. It's sizing its default old-space target off a number that has nothing to do with what the container is actually allowed to use. GC pressure heuristics only kick in as the heap approaches its perceived limit, so with a limit in the gigabytes, V8 kept committing more memory well past 512Mi, confident it had plenty of room left. The kernel's cgroup OOM killer doesn't consult V8's opinion. It watches RSS against the pod's actual 512Mi ceiling and sends SIGKILL the moment that's crossed, regardless of what heapUsed reports.

the fix

Pin the base image to a digest, not a tag, so a rebuild can never silently change what ships:



Dockerfile, before and after

before

FROM node:18-alpine

after

FROM node:18.19.1-alpine@sha256:4a7a4e52c40e973c3e4b02f0c9d8d6c7f8e1a0b2c3d4e5f6a7b8c9d0e1f2a3b4

Second, stop letting V8 guess at a limit it can get wrong. The pod's memory limit is known at build time via Kubernetes resource specs, so the heap ceiling gets set explicitly instead of auto-detected:



deployment.yaml

env:

name: NODE_OPTIONS value: "--max-old-space-size=384" resources: limits: memory: "512Mi" requests: memory: "512Mi"

384MB leaves roughly 25% of the 512Mi limit for stack space, native allocations, and whatever else lives outside V8's managed heap, the same gap that let rss outrun heapUsed during the incident. With the ceiling fixed explicitly, a future base image change can't move it again.

Third, the CI job that builds this image now fails loudly on an unpinned FROM line instead of silently rebuilding against whatever a tag resolves to that day:



ci check, pinned-base-image.sh

if grep -qE '^FROM [a-z0-9./-]+:[a-zA-Z0-9._-]+$' Dockerfile && \
! grep -q '@sha256:' Dockerfile; then
echo "FROM line is not pinned to a digest, failing build"
exit 1
fi

the aftermath

61 OOMKill restarts across 9 pods in the 2h10m window before mitigation

18,400 Checkout requests that failed with 502 or 503 during the incident

8.3GB V8's self-reported heap_size_limit on a pod capped at 512Mi

0 App code commits in the 48 hours before the first OOMKill

A floating base image tag is a deploy that never shows up in the deploy history. The Dockerfile didn't change, the git log didn't change, and the thing that actually shipped a behavioral difference into production left no trace anyone was watching.
heap_size_limit from v8.getHeapStatistics() should always track the container's actual memory limit. If it varies by node instead of by pod spec, Node's cgroup detection isn't seeing what you think it's seeing.
heapUsed staying flat while rss climbs doesn't always mean a native leak. It can mean V8 is committing memory toward a target that was never the real limit in the first place.
Pinning a base image by digest and setting --max-old-space-size explicitly both remove the same failure mode: a runtime deciding its own resource ceiling from a number nobody in the deploy pipeline actually chose.

Nothing in this incident involved a bug in the application. The code that ran at 14:47 UTC was identical to the code that had run cleanly for weeks. What changed was three layers down, in a base image tag nobody was watching, and it took exec-ing into pods on different physical nodes to even notice the one variable that should never have mattered was the variable that did.

How a 30-Second Pod Grace Period Lost the Race Against a 300-Second ALB Deregistration Delay

Darshan Turakhia — Tue, 20 Oct 2026 11:01:20 +0000

14:32 UTC. PagerDuty fires on the checkout service's error-rate alarm. Datadog shows the 502 rate on POST /checkout jumping from a baseline of 0.02% to 4.8%, right in the middle of a rolling deploy that had shipped a dozen times before without anyone noticing it. By 14:36 the deploy finishes, the 502s stop, and the dashboard looks normal again. Nobody asked for this one to rotate twelve pods for a routine dependency bump, and yet here it was, the third deploy that week to show the exact same spike, each time clean before and after, dirty for about four minutes in the middle.

the scramble

First theory in #incidents: the new image is bad. That's the obvious read when a deploy and an error spike land on top of each other. Someone pulls up the new revision's pod logs looking for stack traces, crash loops, anything. Nothing. Zero restarts across all twelve pods. Zero 5xx logged by the app itself. Whatever's producing these 502s, it isn't the checkout service throwing an error, because the checkout service never saw the request at all.

Second theory: a health check regression is marking targets unhealthy mid-rollout and the ALB is handling the transition badly. Someone checks the target group in the AWS console. Zero unhealthy targets logged for the entire deploy window. By every signal AWS itself can show, nothing was ever wrong with any target. That's the part that stalls the room for a few minutes, because if the load balancer never thought a target was unhealthy, where are the 502s even coming from?

the hunt

With both easy theories dead, someone pulls the ALB's own access logs from S3 and greps for the deploy window instead of trusting the target-group health view. ALB access logs record a target_status_code of - for exactly one case: it picked a target, tried to open the connection, and got nothing back, connection refused or reset. That's a different failure mode than "target failed its health check," and it doesn't show up on the target group's health dashboard at all.



alb access logs, filtered to the deploy window

aws s3 cp s3://alb-logs/checkout/2026/10/20/ . --recursive
zcat *.log.gz | awk '$8 == "-" && $6 ~ /checkout/' | cut -d' ' -f4,8,14

14:33:02.418481Z - 10.4.21.117:3000
14:33:04.901223Z - 10.4.21.117:3000
14:33:05.550112Z - 10.4.21.117:3000
14:34:11.009887Z - 10.4.21.94:3000
14:34:12.733410Z - 10.4.21.94:3000

Every failed request maps to a target IP, and those IPs belong to pods. Cross-reference that against kubectl get events history for the same window: each burst of failures lines up within a handful of seconds of that exact pod's termination. 10.4.21.117 got its SIGTERM at 14:33:00. The last 502 mapped to it is at 14:33:05.55, five seconds after the process was told to shut down.

That's the actual question now: why was the ALB still sending live traffic to a pod five seconds after Kubernetes told it to die? Checking the AWS Load Balancer Controller's own logs for the same window answers it. The controller doesn't poll for pod state, it watches EndpointSlice updates and reacts by calling DeregisterTargets on the target group. The controller's reconcile loop timestamps show a four-to-seven-second gap between the pod going NotReady and the target group actually being told to drop that IP. The checkout app had no SIGTERM handler, so Node's default behavior kicked in: no listener means the process exits almost immediately, well under a second, as soon as the signal arrives.

The pod was dying in under a second. The signal that it was dying took four to seven seconds to reach the one system still routing traffic to it. The 300-second deregistration_delay.timeout_seconds on the target group, set months earlier to drain long-lived connections gracefully once a target is actually marked for removal, was never the problem and never fired, because the controller hadn't called DeregisterTargets yet when the pod went away. That setting governs what happens after deregistration starts. It has nothing to say about the gap before it starts.

the find

Root cause: the checkout app shut down faster than Kubernetes' own control plane could tell the load balancer it was going away. A bare SIGTERM with no handler killed the process in under a second. In this cluster it took four to seven seconds for that pod's IP to drop from the EndpointSlice, for the AWS Load Balancer Controller to notice, and for DeregisterTargets to actually fire on the ALB. Every request the ALB routed to that pod during that gap hit a socket nobody was listening on anymore.

the fix

Two changes, working together. First, the app needed to survive SIGTERM long enough to stop accepting new connections on its own terms instead of just dying:



checkout service, graceful shutdown handler

const server = app.listen(3000);

process.on('SIGTERM', () => {
console.log('SIGTERM received, draining connections');
server.close(() => {
console.log('server closed, no in-flight requests remain');
process.exit(0);
});

// belt and suspenders: force exit if close() hangs on a stuck connection
setTimeout(() => process.exit(1), 25_000).unref();
});

Second, and this is the half that actually fixes the race: the pod needed to delay sending SIGTERM in the first place, long enough for the EndpointSlice update and the ALB controller's deregister call to actually land before the process started shutting down.



deployment.yaml, checkout container

spec:
terminationGracePeriodSeconds: 45
containers:
- name: checkout
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz
port: 3000
periodSeconds: 5
failureThreshold: 1

The preStop hook runs before SIGTERM is ever sent to the container, and the pod is marked NotReady the moment termination begins, well before the hook finishes sleeping. Ten seconds was comfortably above the four-to-seven-second gap measured in the controller logs. terminationGracePeriodSeconds went up to 45 so the sleep plus the app's own graceful shutdown both have room to finish before Kubernetes gives up and sends SIGKILL.

the aftermath

4.8% Peak 502 rate on checkout during the deploy, against a 0.02% baseline

4-7s Measured gap between a pod going NotReady and the ALB controller deregistering its target

1,240 Failed checkout POSTs across the 14-minute rolling deploy before the fix

0 Failed requests observed across the next 30 deploys after the preStop hook shipped

A target group's deregistration_delay only governs draining after a target has been told it's leaving. It does nothing about the time it takes the control plane to decide and propagate that a target is leaving in the first place, and that gap is where this bug lived.
A process with no SIGTERM handler doesn't die gracefully, it just dies fast. During a rolling deploy that's the wrong kind of fast, fast enough to beat the signal telling everyone else it's on the way out.
ALB access logs record connection-level failures that never touch the target group's health dashboard. A target can pass every health check the dashboard tracks and still spend a few seconds quietly refusing live connections underneath it.
A synthetic check now hits /checkout every two seconds during every deploy and pages if a single 502 shows up, specifically so this class of bug can't hide behind "the dashboards looked fine" again.

Nothing about the checkout service's code was ever broken, and nothing about the ALB's configuration was wrong on its own terms. The bug lived entirely in the few seconds between two correct systems finding out about each other.

How a Bounce-Handling Bug Silently Suppressed Password Reset Emails for 1,800 Users

Darshan Turakhia — Fri, 16 Oct 2026 11:13:27 +0000

09:14 UTC, day nine. A support ticket that's been bounced between two tiers finally lands on an engineer's desk: a customer says she's requested a password reset six times since Monday and gotten nothing, not even in spam. On-call pulls her account. Email domain: contractorhq.biz. Two more tickets from the same domain, filed within the hour, same complaint.

the scramble

First theory: SES is having a bad day. The SES console shows sending quota at 12% utilization and a bounce rate of 0.4%, nowhere near the 5% threshold that gets an account put under review. Not a sending problem.

Second theory: deliverability, SPF or DKIM drifted out of alignment with the receiving domain's filters. Pulling the raw headers from a reset email sent to a different, working address shows both passing clean, DKIM signature valid, SPF aligned. Nothing about the message itself looks like spam.

The pivot comes from checking the app's own send log instead of SES. The password reset endpoint didn't throw, didn't retry, didn't log an error for any of the three accounts. It logged email_skipped: suppressed, cleanly, like that was the intended outcome. Nothing failed. The app chose not to send.

the hunt

Before calling SES for any transactional send, the app checks an internal email_suppressions table, populated by a Lambda that consumes SES bounce and complaint notifications off an SNS topic. The three accounts are all in it:



email\_suppressions, query against the three accounts

SELECT email, reason, suppressed_at FROM email_suppressions
WHERE email LIKE '%@contractorhq.biz';

email	reason	suppressed_at
j.ruiz@contractorhq.biz	bounce	2026-10-07 03:12:41+00
t.okafor@contractorhq.biz	bounce	2026-10-07 03:12:58+00
m.lee@contractorhq.biz	bounce	2026-10-07 03:13:04+00

Three timestamps, nine days earlier, nineteen seconds apart. That's not three isolated bounces on three separate occasions, that's a batch. Something sent to all three at once and all three bounced the same way within the same window.

The only thing that went out to a list that size that night was a product newsletter, sent to 40,000 subscribers at 03:09 UTC. contractorhq.biz runs its own mail server, and the timing lines up with a documented, if rare, failure mode on their end: a burst of inbound mail from one sender hitting a mailbox quota ceiling on several accounts in the same minute. SES's raw SNS bounce payload for one of the three, pulled from the archived notification, backs it up:



SES bounce notification, archived SNS message body

{
"notificationType": "Bounce",
"bounce": {
"bounceType": "Transient",
"bounceSubType": "General",
"bouncedRecipients": [
{ "emailAddress": "j.ruiz@contractorhq.biz",
"diagnosticCode": "smtp; 452 4.2.2 mailbox full" }
],
"timestamp": "2026-10-07T03:12:41.000Z"
}
}

bounceType: "Transient". A full mailbox, a temporary condition that resolves itself once the recipient clears space, not a dead address. SES's own documentation is explicit that Transient bounces should never result in permanent suppression. And yet here they are, suppressed indefinitely, nine days later.

the find

The bounce-handling Lambda's suppression logic, as deployed:



bounce-handler/index.py, as found

def handle_bounce(bounce: dict) -> None:
sub_type = bounce["bounceSubType"]
if sub_type == "General":
for r in bounce["bouncedRecipients"]:
suppress_email(r["emailAddress"], reason="bounce")

bounceSubType values aren't unique to one bounceType. "General" is a valid subtype under both Permanent (a truly dead address) and Transient (a temporary delivery problem, mailbox full, message too large, content rejected). The handler branched on the subtype alone and never checked which top-level type it belonged to, so a wave of ordinary, self-resolving mailbox-full bounces from one slow mail server got routed through the exact same path as permanently dead addresses, and suppressed forever on the first occurrence.

The newsletter send was the trigger, not the cause. It was simply the first send in months large enough to hit enough inboxes on one domain, at once, hard enough to push several of them over quota in the same minute. Any password reset or receipt email to those addresses afterward silently skipped SES entirely, logged as a clean, intentional-looking suppression rather than a failure, with no alert tied to it because nothing had technically gone wrong from the app's point of view.

the fix

Immediate containment: identify every suppression entry whose archived bounce payload says Transient, and release them.



one-off reconciliation script

UPDATE email_suppressions s
SET released_at = now()
FROM bounce_events b
WHERE b.email = s.email
AND b.bounce_type = 'Transient'
AND s.released_at IS NULL;

The Lambda now gates on bounceType first, and treats transient bounces as a backoff signal rather than a ban:



bounce-handler/index.py, after

def handle_bounce(bounce: dict) -> None:
bounce_type = bounce["bounceType"]
for r in bounce["bouncedRecipients"]:
email = r["emailAddress"]
if bounce_type == "Permanent":
suppress_email(email, reason="hard_bounce")
elif bounce_type == "Transient":
record_transient_bounce(email)
if transient_bounce_count(email, window_days=7) >= 5:
suppress_email(email, reason="repeated_transient")

A single mailbox-full bounce no longer costs an address its ability to receive a password reset. Five in a week does, which still catches genuinely dead or abandoned inboxes without punishing a recipient for one bad night on their mail server.

The other gap was visibility: email_skipped: suppressed had been treated as a routine log line, not a signal worth watching. It now feeds a dashboard counter scoped to password-reset and receipt sends specifically, with an alert if skipped-suppressed count for either exceeds a handful per hour, since those two email types have no legitimate reason to be suppressed at any volume.

the aftermath

1,800 Addresses incorrectly suppressed from one newsletter send

9 days From the bad bounce wave to the first escalated ticket

212 Password reset and receipt sends silently skipped in that window

0 Errors logged for any of the skipped sends

bounceSubType is not a reliable signal on its own. The same subtype string can appear under more than one bounceType, and any suppression logic that skips the type check is one coincidence away from treating a temporary problem as a permanent one.
A suppression system needs its own retry and backoff policy for transient failures, not a single binary gate. One bad delivery attempt and a pattern of bad delivery attempts are different signals and deserve different responses.
A clean, no-error log line can still be a failure. skipped: suppressed looked identical whether it was protecting the system from a dead address or quietly breaking account recovery for a real user.
Transactional email categories that users can't route around, like password resets, need their own monitoring, separate from aggregate sending health. SES's dashboard was green the entire nine days.

Nothing about the send pipeline ever broke. Every password reset email that got suppressed was, from SES's perspective and the app's own logs, working exactly as designed. The design just couldn't tell the difference between an inbox that was full for an hour and one that would never exist again.

How Rotating S3 Presigned URLs Silently Defeated Next.js Image Optimization

Darshan Turakhia — Sat, 10 Oct 2026 10:59:28 +0000

09:41 UTC, Tuesday. A billing alert from Vercel lands in the platform team's Slack channel: "Image Optimization usage is approaching your plan limit." The plan limit is 50,000 source images a month. The alert says the account used 2.3 million in the last 24 hours. Nobody had touched the image pipeline. Nobody had touched the plan. The only thing that had shipped recently was a new product gallery page, live since Monday morning.

the scramble

First theory: a scraper. The gallery page is public, product photos are easy to hot-link, and a bot crawling every product ID in sequence would explain a sudden jump in image requests. Pulling the Vercel function logs for /_next/image rules that out inside ten minutes, the referer header on almost every request is the app's own domain, and the request rate tracks normal daytime traffic, not a crawler's steady drumbeat.

Second theory: the gallery page itself was the problem, maybe it rendered more image variants per view than intended. The on-call engineer checks the diff for Monday's deploy. The <Image> component's sizes prop is unchanged, still three breakpoints, the same as every other image on the site. Forty products per gallery page, three sizes each, that's 120 image requests per page view at worst, nowhere near enough to explain 2.3 million unique source images in a day with the gallery's actual traffic.

Both obvious explanations are gone within fifteen minutes, and the number on the Vercel dashboard keeps climbing. Whatever is driving this, it isn't more images and it isn't a bot. It's something about how the existing images are being counted.

the hunt

Vercel's Image Optimization API doesn't bill per request. It bills per unique source image it has to fetch and transform. A healthy app re-fetches the same source rarely, the optimizer caches the transformed output at the edge and serves repeat requests for the same source URL, width, and quality straight from that cache. The volume alone doesn't explain the bill. What needs explaining is why almost none of these requests are hitting that cache.

Pulling a sample of the url= query parameters the optimizer is being asked to fetch for a single avatar, the same user's avatar, across five page loads inside two minutes, shows five different strings:



same avatar, five page loads, two minutes apart

https://assets-private.ourapp.com.s3.amazonaws.com/avatars/8841.jpg
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA.../20261006/us-east-1/s3/aws4_request
&X-Amz-Date=20261006T094012Z
&X-Amz-Expires=900
&X-Amz-SignedHeaders=host
&X-Amz-Signature=7f2a9c... (load 1)

...same path, X-Amz-Date=20261006T094047Z, X-Amz-Signature=b81e4d... (load 2)
...same path, X-Amz-Date=20261006T094119Z, X-Amz-Signature=029cf7... (load 3)

Same bucket, same key, same bytes on disk. Different query string every time. Dead end first: the team suspects an overeager cache-purge webhook, something firing on a schedule and evicting the edge cache before it can ever get used. Checking the optimizer's response headers for a repeat request to the exact same URL rules that out, when the query string is identical, the cache hits fine and returns in under 20ms. The problem isn't eviction. The problem is that the URL is never identical twice.

That one line of logs is the whole bug. Next.js's image optimizer, and Vercel's hosted version of it, treats the full request URL, including every query parameter, as part of the cache key. A presigned S3 URL carries a fresh X-Amz-Date and X-Amz-Signature on every call, by design, that's what makes it a presigned URL instead of a permanent link. Which means every single page view of the same unchanged avatar was being treated as a request for a brand-new image the optimizer had never seen before.

the find

Two weeks earlier, a SOC 2 audit finding flagged that user avatars and product photos sat in a publicly readable S3 bucket, no auth required if you had the URL. The fix, shipped quietly and correctly from a security standpoint, moved everything to a private bucket and started generating a presigned URL server-side for every image, with a 15-minute expiry, passed straight into the src prop:



image loader, as shipped two weeks earlier

// server component, product gallery
const presignedUrl = await s3.getSignedUrl('getObject', {
Bucket: 'assets-private-ourapp',
Key: products/\${product.imageId}.jpg\,
Expires: 900,
});

return ;

At the traffic the site was doing before Monday, this was a real inefficiency, every page view was already paying for a fresh optimization instead of a cached one, but the absolute numbers stayed small enough to hide inside normal usage. Nobody had a monitor on cache hit ratio for the image optimizer specifically, only a monthly usage total that nobody checks mid-cycle. The gallery launch didn't introduce the bug. It multiplied an existing one: forty product photos rendered per page view, each one a guaranteed cache miss, across a traffic volume high enough to turn a slow leak into something that blew through a 50,000-image monthly plan limit in under a day.

the fix

The access control and the cache key needed to be separated. A presigned URL is the right tool for proving the request is authorized. It is the wrong tool for identifying the image to the optimizer, because its identity changes every time it's generated. The fix routes image requests through a stable, internal path, and does the S3 fetch and the authorization check server-side, where the signature never has to leave the backend:



app/api/media/\[assetId\]/route.ts

export async function GET(
req: NextRequest,
{ params }: { params: { assetId: string } }
) {
const authorized = await checkAccess(req, params.assetId);
if (!authorized) {
return new Response('Forbidden', { status: 403 });
}

const object = await s3.getObject({
Bucket: 'assets-private-ourapp',
Key: products/\${params.assetId}.jpg\,
});

return new Response(object.Body as ReadableStream, {
headers: {
'Content-Type': 'image/jpeg',
'Cache-Control': 'public, max-age=31536000, immutable',
},
});
}

The <Image> call changes from a signed, expiring URL to a plain, stable path:



image loader, after

return (

);

product.imageId never changes for a given photo, so the optimizer's cache key is stable now, no matter who's viewing it or when, until the underlying asset actually gets replaced. The route handler still checks authorization on every call, nothing about access control got weaker, it just stopped leaking into the one place that needed to stay constant. The immutable cache directive is safe here specifically because asset IDs are never reused for a changed image, a new upload gets a new ID.

the aftermath

2.3M → 44K Daily unique source images, before and after the fix deployed

$1,840 Overage for the 24-hour window that triggered the alert

14 days The underlying cache-miss bug ran before traffic made it visible

1 Second instance of the same mistake found and fixed in an email template's inline images

A presigned URL and a cache key are different problems wearing the same string. Anything that intentionally changes on every call, a signature, a timestamp, a nonce, will defeat any cache that treats the full URL as its identity, no matter how good that cache otherwise is.
A security fix and a performance regression can ship in the same commit without anyone noticing, because nobody reviews a SOC 2 remediation for cache behavior. The review caught the access control problem it was meant to catch and missed the one it accidentally created.
Usage caps that only alert at a monthly total, not a daily rate of change, hide a slow leak until something unrelated, in this case a product launch, amplifies it past the threshold. A day-over-day anomaly alert would have caught this two weeks earlier, while the fix was still a one-line git blame away.
Keep authorization server-side and identity stable client-side. Any time a URL that gets cached, by a CDN, an image optimizer, a browser, also needs to prove the request is allowed, put the proof behind a stable endpoint instead of inside the URL itself.

The avatars never moved. The product photos never moved. For two weeks, the only thing wrong was a query string that looked identical in a browser's network tab and was, byte for byte, never the same request twice. The optimizer did exactly what it was built to do: cache what it could prove was the same image. It just never got the chance.

How a Zero-Downtime Table Swap Silently Dropped Orders From Postgres Logical Replication for 91 Hours

Darshan Turakhia — Tue, 06 Oct 2026 10:59:16 +0000

09:14 UTC, Thursday. #finance-eng: "Why has daily revenue been flat at $118,402 for four days straight? We ran a 20%-off promo Tuesday, this should have spiked, not flatlined." No PagerDuty alert fired. No panel on the production dashboard is red. Checkout is taking orders normally. Whatever broke didn't trip a single automated check, because nothing in production is actually broken.

the scramble

First theory: the promo's attribution tracking is broken, not revenue itself. Marketing's UTM tagging has flaked before. Someone pulls the Stripe payouts dashboard directly, bypassing the internal reporting pipeline entirely. Payouts for the last four days are up, in line with the promo. Money is moving. The number everyone is staring at is wrong, not the business.

Second theory: the BI tool is serving a stale cached query. The finance dashboard sits on top of an internal analytics Postgres instance, fed by logical replication from the primary rather than queried against production directly, a deliberate choice made a year ago so heavy reporting queries can never compete with checkout for connections or I/O. Someone forces a hard refresh on the dashboard, bypassing the BI tool's own cache layer. Same flat number.

Third theory, and the one that finally moves: query the analytics replica directly instead of through the dashboard.



09:41 UTC, analytics replica

SELECT count(*), max(created_at) FROM orders;

count	max
9,214,880	2026-10-02 14:02:41+00

-- same query against production, same moment
SELECT count(*), max(created_at) FROM orders;

count	max
9,222,563	2026-10-06 09:40:12+00

Production has 7,683 more rows than the replica, and the replica's newest row is timestamped almost exactly four days in the past. That timestamp isn't a random stale point. It's 12 minutes after the rename-swap migration that shipped Tuesday afternoon. The dashboard was never broken. The data feeding it stopped arriving.

the hunt

Tuesday's deploy converted the orders table's primary key from int4 to int8 using the standard zero-downtime shadow-table pattern: build orders_new with the wider key, backfill it in batches, dual-write via trigger while the backfill catches up, then an atomic rename swap inside one transaction:



Tuesday 14:02:17 UTC, cutover

BEGIN;
ALTER TABLE orders RENAME TO orders_old;
ALTER TABLE orders_new RENAME TO orders;
COMMIT;

Clean, fast, no lock held longer than the two renames. The on-call engineer who ran it watched checkout error rates for twenty minutes afterward and saw nothing. By every metric anyone was watching, the migration worked.

First check on the replication path: is the slot even still alive?



09:52 UTC, primary

SELECT slot_name, active, confirmed_flush_lsn
FROM pg_replication_slots
WHERE slot_name = 'analytics_sub';

slot_name	active	confirmed_flush_lsn
analytics_sub	t	4A2/8F1C3D00

-- run again two minutes later
analytics_sub | t | 4A2/8F22A188

Active, and the LSN is climbing between queries. That reads as healthy replication by every normal heuristic, and it's the dead end that costs the most time: the publication backing this subscription also covers customers, invoices, and three other tables that are still being written to constantly. Those writes keep the logical decoding worker busy and the confirmed LSN advancing regardless of what's happening with orders specifically. A slot-level health check can look perfectly fine while one table inside it has gone completely silent.

The next check is narrower, not "is the slot alive" but "what tables is this publication actually sending":



10:08 UTC, primary

SELECT pubname, tablename FROM pg_publication_tables
WHERE pubname = 'orders_pub';

pubname	tablename
orders_pub	orders_old
orders_pub	customers
orders_pub	invoices

orders_old. Not orders. The publication has been faithfully, correctly, silently replicating a table that nothing has written to since 14:02:17 UTC on Tuesday.

the find

ALTER PUBLICATION ... ADD TABLE resolves the table name to its object ID at the moment the command runs and stores that OID in pg_publication_rel. It does not store the name. From that point forward, the publication tracks whatever object owns that OID, not whatever object currently answers to the name orders.

The rename-swap pattern is specifically designed to be atomic and cheap, which it achieves by never actually copying data at cutover time. It just reassigns two names to two existing relations. The relation that was orders for the three years before Tuesday did not stop existing. It got renamed to orders_old and kept its OID, and the publication kept following that OID exactly as configured. The relation that's been named orders since 14:02:17 UTC is a different object with a different OID, one the publication was never told about, because the migration added a new table under a temporary name and the publication definition was never revisited after the swap.

Nothing in this chain is a bug. The publication did precisely what ALTER PUBLICATION ADD TABLE orders asked it to do, a year ago, for the table that was orders at the time. A rename is not a schema change the replication system has any reason to treat as suspicious, since tables get renamed for all kinds of reasons that have nothing to do with swapping in a different relation underneath a stable name. There was no constraint violation, no decoding error, no WAL gap, nothing for any alert to fire on. The failure mode is a correctly configured system doing exactly what it was told, applied to an assumption, "the table named orders is the table I meant," that a swap-based migration quietly invalidated.

the fix

Point the publication at the table that's actually live, and drop the dead one once its replicated history is no longer needed on the subscriber:



10:31 UTC, primary

ALTER PUBLICATION orders_pub DROP TABLE orders_old;
ALTER PUBLICATION orders_pub ADD TABLE orders;

Adding a table to an existing publication doesn't push any of its current rows to subscribers by itself. It only starts streaming changes from that point forward. The subscriber still needs the 91 hours of orders it missed, plus confirmation that its local orders table matches the live one row for row, not just going forward from here:



10:33 UTC, analytics replica

ALTER SUBSCRIPTION analytics_sub REFRESH PUBLICATION WITH (copy_data = true);
-- 11 minutes to copy and index 9.2M rows

REFRESH PUBLICATION diffs the subscription's known relations against the publication's current membership, and for anything newly added it runs a full initial copy for that relation only, while customers and invoices keep streaming without interruption the entire time. Eleven minutes later the replica's orders table matches production exactly, and the finance dashboard's next scheduled refresh picks up four days of orders in one jump.

The longer-term fix is a check that doesn't depend on anyone remembering to update the publication by hand during a migration that was never framed as a replication change in the first place:



nightly parity check, new cron

-- runs against every table/subscriber pair, alerts on the table name, not the slot
SELECT p.tablename,
now() - max(o.created_at) AS staleness
FROM pg_publication_tables p
JOIN orders o ON true
WHERE p.pubname = 'orders_pub' AND p.tablename = 'orders'
GROUP BY p.tablename
HAVING now() - max(o.created_at) > interval '2 hours';

It also became a standing step in the shadow-table migration runbook: any rename-swap that touches a table covered by logical replication requires a publication membership check in the same change window as the cutover, not as a follow-up ticket filed whenever someone happens to notice.

the aftermath

91 hrs Between cutover and the orders table rejoining replication

7,683 Orders missing from the analytics replica at discovery

$0 Actual revenue lost — Stripe and production had every order

11 min To resync the table via a targeted copy_data refresh

Postgres publications bind to a table's object ID at ADD TABLE time, not its name. A rename-swap migration that reassigns an existing name to a new relation is invisible to that binding — the publication keeps following the OID, which is now the wrong table.
A healthy replication slot (active, LSN advancing) says the slot is working. It says nothing about whether a specific table inside a multi-table publication is still part of that stream. Check pg_publication_tables directly when one table's data looks stale; don't infer table-level health from slot-level metrics.
ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_data = true) resyncs only the relations that are new to the subscription. It's a safe, targeted recovery step, not a full subscription rebuild, and it doesn't interrupt tables that were never affected.
Any migration pattern built around renaming tables, not just this one, should carry an explicit step to re-check every publication and every foreign key, trigger, or grant that was bound to the old name by object reference rather than by name. A clean rename at the schema level can still leave dependent systems pointed at the wrong object.

The migration itself worked exactly as designed, start to finish, with zero downtime and zero checkout errors. The thing it broke wasn't checkout. It was a dependency three systems away that nobody thought to list as a stakeholder for a primary-key type change, and the only signal it ever produced was a dashboard that quietly stopped changing.

Reimplementing a Trie Reminded Me How Autocomplete Actually Works

Darshan Turakhia — Fri, 02 Oct 2026 14:06:26 +0000

I spent an embarrassing amount of time last week reimplementing something I'd "known" since a data structures class a decade ago: the trie. Typing it out from scratch again reminded me why it's one of the few data structures that actually earns the "elegant" label people throw around too easily.

Here's the pitch in one sentence: a trie stores strings by character, along shared paths from a root, so two words with the same prefix walk the exact same nodes until they stop agreeing. Insert "car," "card," and "care," and you get one shared path for "c-a-r," which then splits into three different endings. Nothing about adding "care" touches the nodes "car" and "card" already built.

That sharing is the whole point, and it's easy to undersell how much it buys you. A hash set can tell you "yes, this exact string is in the collection." That's it. It cannot tell you "give me everything that starts with these three letters" without scanning the entire set and checking each entry one by one. A trie answers that second question by walking to the end of the prefix and then just... looking at what's underneath. Lookup cost depends on how long your prefix is, not on how many million other strings happen to be sitting next to it in memory. That's the entire mechanism behind every autocomplete dropdown you've ever typed into: walk the prefix, then collect every real word hanging off wherever you land.

The part that actually bit me while rebuilding this: you need an explicit marker for "a real word ends here." It sounds obvious written down, but it's the single easiest thing to forget, and the bug it causes is sneaky. Say you only ever insert "card." The path c → a → r → d exists in your trie. All four nodes are real. But "car" was never inserted as its own entry, it's just a prefix that happens to lead somewhere. If your node doesn't carry a boolean flag for "an insertion actually terminated here," you have no way to distinguish a real stored word from a string that merely happens to be a prefix of a longer one. I've seen (and once written) autocomplete bugs where a partial prefix gets suggested as a complete match because of exactly this missing flag.

The tradeoff nobody mentions in the five-minute version of this concept: a naive implementation wastes a lot of memory. The textbook approach gives every node a fixed-size array, one slot per possible next character. Twenty-six slots if you're doing lowercase English, more if you need digits or punctuation or, god forbid, full Unicode. Most of those slots sit empty on any node that doesn't branch in every direction, and that waste adds up fast across a trie with real-world vocabulary in it. The fix is one of two things: back each node with a hash map instead of a fixed array, so you only pay for children that actually exist, or compress the whole structure with something like a radix trie, which collapses long runs of single-child nodes into one node holding a whole substring instead of one character. "Cardboard" doesn't need five separate one-character hops after "card," it needs one node labeled "board."

Autocomplete is the use case everyone reaches for first, and it's a good one, but it's not the only place this shows up. Spell-checkers use the same walk to confirm a word exists in a dictionary and to find near-miss suggestions by exploring nearby paths. IP routers use a binary version of the same idea, walking a trie over address bits instead of letters, to do longest-prefix-match routing: find the most specific rule that matches a destination address. Different alphabet, same shared-prefix-sharing-a-path trick underneath.

I ended up writing both a longer explanation and a small interactive visualizer, mostly because I wanted to actually watch the branching happen rather than just trust my own mental model of it. If you want to poke at it yourself, the guide walks through the insert/search mechanics and the end-of-word gotcha in more detail, and the visualizer lets you insert your own words and watch the prefix lookup highlight in real time, suggestions and all.

If you've never implemented one, it's a genuinely good weekend exercise. Small enough to build in an hour. Annoying enough, in the best way, that you'll start second-guessing every search box you touch afterward.

How a Rolling Deploy

Darshan Turakhia — Tue, 29 Sep 2026 10:58:00 +0000

13:52 UTC. p99 latency on pricing-svc alerts at 1.8 seconds, four times its usual ceiling. The dashboard shows six healthy pods, six pods passing their readiness probe, six pods with room on their CPU requests. One of them is pegged at 97%. The other five are idling under 12%.

the setup

checkout-api calls pricing-svc over gRPC for every cart total, roughly 4,000 RPCs a minute at midday. Both run in the same namespace, and pricing-svc is exposed through a standard ClusterIP Service, resolved once by DNS to a single virtual IP with kube-proxy handling the fan-out to whichever pods are in the endpoint list.



checkout-api/pricing-client.ts

const client = new PricingServiceClient(
'pricing-svc.default.svc.cluster.local:50051',
grpc.credentials.createInsecure()
);
// one channel per pod, reused for every call it makes

That last line is the part nobody thought hard about. gRPC runs on HTTP/2, and HTTP/2 multiplexes many concurrent RPCs over a single TCP connection. A gRPC client opens one connection to a target and keeps it open, reusing it for every subsequent call instead of opening a new one per request. For a REST client that reconnects constantly, ClusterIP round robin evens out over thousands of short-lived connections. For a gRPC client, the load balancing decision happens exactly once, at connection time, and then holds for as long as that connection stays up. checkout-api runs eight pods. Each one makes that decision independently, once, on startup.

the scramble

First theory: bad node. The hot pod, pricing-svc-7d4f-x2k9, is cordoned and drained, forcing it onto a different node entirely. Eleven minutes later the alert fires again, same symptom, different pod, pricing-svc-7d4f-m8p1, now pegged while the rest idle. Different node, different hardware, same shape of failure. That rules out anything node-specific and burns fifteen minutes doing it.

Second theory: the HPA is fighting itself, scaling on a metric that doesn't reflect real load. kubectl describe hpa pricing-svc shows six of six replicas, target CPU utilization at 60%, current average across the deployment sitting at 22%. The average looks fine because five pods are nearly idle. Averages hide a skew this sharp, and the HPA has no reason to add capacity when its own number says the fleet is underloaded.

the hunt

Instead of watching pods cycle, the next step is watching the connections themselves. Shelling into a checkout-api pod and checking its open sockets to the pricing service:



kubectl exec checkout-api-9f3a-vv2q -- ss -tnp | grep :50051

ESTAB 0 0 10.2.4.19:44822 10.2.6.31:50051

One line. One connection. Every one of the eight checkout-api pods shows the exact same thing: a single established socket to a single backend IP, held open since the pod started. Cross-referencing those destination IPs against the pod list settles it: all eight client pods are connected to the same two pricing-svc pods, and nothing is connected to the other four at all.

A Prometheus query against grpc_server_handled_total, split by pod, shows just how lopsided it got: those two pods have handled 83% of all RPCs since the timestamp on a checkout-api deploy from six hours earlier, a routine dependency bump with nothing in the diff touching networking. The deploy itself didn't cause an error. It just decided, once, where six hours of subsequent traffic would go.

the find

The rollout replaced all eight checkout-api pods inside a forty-second window, standard behavior for a deployment with no maxUnavailable tuning. At the moment those eight new pods came up and each opened its one gRPC connection, pricing-svc was mid-rollout too, from an unrelated change earlier that morning. Only two of its six pods had passed the readiness probe and made it into the Service's endpoint list. kube-proxy's iptables rules can only route to what's registered, so every new connection from every checkout-api pod landed on one of those two.

By the time the other four pricing-svc pods turned ready, thirty seconds later, it didn't matter. The gRPC client's default load balancing policy is pick_first: resolve the target once, connect to the first address, and stay connected until that connection breaks. It doesn't re-resolve, doesn't rebalance, doesn't notice four more pods just joined the pool. The connection from that forty-second window was healthy, so it just kept being used, for six hours, until midday traffic pushed those two pods past the point where 1.8-second p99 was survivable.

the fix

Immediate: restart the two hot pricing-svc pods. Their connections drop, the eight checkout-api pods reconnect, and this time all six backends are already ready, so the new connections spread out. Latency back to baseline in under two minutes. That's a mitigation, not a fix, since the exact same readiness race can happen on the next deploy.

The real fix is making the client aware there's more than one backend to pick from. That means a headless Service, so DNS returns every pod IP instead of one virtual IP, paired with a client-side load balancing policy that actually uses all of them:



k8s/pricing-svc-headless.yaml

apiVersion: v1
kind: Service
metadata:
name: pricing-svc-headless
spec:
clusterIP: None
selector:
app: pricing-svc
ports:
- port: 50051



checkout-api/pricing-client.ts, after

const client = new PricingServiceClient(
'dns:///pricing-svc-headless.default.svc.cluster.local:50051',
grpc.credentials.createInsecure(),
{ 'grpc.lb_policy': 'round_robin' }
);

With round_robin, the client resolves every backend IP and opens a connection to each one, spreading calls across all of them instead of pinning to whichever pod answered first. As a second layer, pricing-svc now caps how long any single connection can live, so even a client stuck on an old load balancing policy is forced to reconnect and re-resolve periodically instead of holding one connection indefinitely:



pricing-svc/server.ts

const server = new grpc.Server({
'grpc.max_connection_age_ms': 5 * 60 * 1000,
'grpc.max_connection_age_grace_ms': 30 * 1000,
});

the aftermath

83% Of all pricing-svc RPCs handled by 2 of 6 pods for six hours

1.8s p99 latency at the point the imbalance became visible

5 min max_connection_age_ms added as a floor on any future skew

3 Other internal gRPC clients found using ClusterIP + pick_first

A ClusterIP Service load balances TCP connections, not requests. That distinction only matters for protocols that hold a connection open and reuse it, which is exactly what HTTP/2 and gRPC are built to do.
pick_first is a reasonable default for a client with one backend. Against a Service backed by multiple pods, it turns whatever pod answered first during a narrow readiness window into the only pod that matters, indefinitely.
Average CPU across a deployment can look completely healthy while two pods carry the entire fleet. Per-pod metrics on anything that does client-side connection reuse are not optional, they're the only view that shows this failure mode at all.
A connection age limit on the server is a cheap insurance policy even after fixing the client. It bounds how long any future imbalance, caused by a bug nobody's found yet, can last before the system corrects itself on its own.

Nothing about that rollout failed. Both deployments finished, both health checks passed, and kube-proxy did exactly what a ClusterIP Service is supposed to do. The problem was a forty-second window where "correct" and "evenly distributed" briefly meant two different things, and gRPC's connection reuse turned that window into six hours.

How Kafka

Darshan Turakhia — Sat, 26 Sep 2026 10:56:53 +0000

03:14 UTC. Security's on-call channel gets a message from the fraud-scanning job: a customer API key flagged and revoked four days earlier is still authorizing requests, all of them from ap-south-1. The revoke ticket has been closed as resolved since Monday.

the setup

authz-gateway sits in front of every API route and checks incoming keys against a local cache before anything touches the database. It used to call Postgres on every request; that was a flat 40ms added to p99 regardless of what the request actually needed. Six months earlier, someone replaced it with a Kafka Streams state store hydrated from a compacted topic, api-key-status, keyed by api_key_id. p99 for the auth check dropped to 4ms. An active key is a normal record. A revoked key is a tombstone, a record with the key present and the value set to null, which is how compacted topics represent deletion.



gateway/authz.js

function isKeyValid(apiKeyId) {
const record = keyStatusStore.get(apiKeyId); // local RocksDB state store
return record !== undefined && record.status === 'active';
}

The consumer group backing that store runs with group.instance.id set, static membership, turned on months earlier after a rolling deploy triggered a full rebalance and briefly spiked auth latency across every pod, not just the ones restarting. Static membership means a pod that drops out during a deploy keeps its partition assignment reserved instead of forcing an immediate reassignment. It was a good fix for that problem. It also meant nobody was watching for what happens when a pod goes quiet for reasons that have nothing to do with a deploy.

the scramble

First move: check the source of truth. Postgres's api_keys table shows the key correctly marked revoked_at four days ago. Whatever's wrong isn't the revoke itself, it happened, it's recorded, it's just not everywhere it needs to be.

Second theory: the tombstone never made it onto the topic. A console consumer against api-key-status from the revoke's approximate timestamp finds it immediately, a null-value record for that exact key, committed and replicated. The event exists. The dead end didn't cost much, five minutes to rule out, but it mattered: it meant the problem was downstream of Kafka, not upstream of it.

Third theory, on-call's: a stale edge cache. This service doesn't have one on the authz path, ruled out by reading its own deployment topology rather than by testing anything.

the hunt

The fraud job's flagged requests all traced to three pods, all in ap-south-1, all part of the same consumer group. kafka-consumer-groups.sh --describe against that group showed those three partitions current, no lag, caught up to the log's high-water mark. Not stuck. Not behind. Just wrong.



kafka-consumer-groups.sh --describe --group authz-gateway

PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
4 88213 88213 0 authz-gateway-7f2a...
7 61940 61940 0 authz-gateway-7f2a...
9 73301 73301 0 authz-gateway-7f2a...

Caught up with a wrong answer is a different bug than stuck behind. The next thing to check wasn't the topic, it was those three pods' history. Kubernetes events showed the answer: an ap-south-1-scoped NetworkPolicy change went out five days earlier as part of a node pool migration, and a rule ordering mistake in it blocked egress from those three pods to the Kafka broker subnet for roughly 30 hours before someone caught it in an unrelated ticket and reverted it. Nothing restarted those pods. Their liveness probe was an HTTP ping against the gateway's own port, which had nothing to do with whether it could reach Kafka, so it stayed green the entire time.

With no broker access, the consumer's background heartbeat thread stopped reaching the group coordinator. Kafka correctly detected that after session.timeout.ms and reassigned those three partitions to healthy members elsewhere, who kept the cache correct for everyone else. The revoke tombstone was produced and consumed by those healthy members within seconds, exactly as designed. It's what happened to the three isolated pods, once the network policy reverted and they reconnected, that broke.

the find

Kafka Streams persists its state store to local disk and checkpoints the offset it last processed. On reconnect, if that checkpoint is still within the log, it doesn't replay from scratch, it does a delta restore: read forward from the checkpointed offset, apply whatever's new. That's the whole point of a local store, cheap recovery. The three pods' checkpoints were roughly 6 hours stale when the network came back, right around when the revoke had been written.

The topic's delete.retention.ms was left at Kafka's default, 86400000ms, 24 hours. That setting exists specifically so a lagging consumer gets a grace window to see a tombstone before the log cleaner physically removes it. The three pods were unreachable for roughly 30 hours, past that window. By the time they reconnected and asked to resume from their checkpoint, the tombstone that checkpoint needed to see had already been compacted away. The delta restore read forward past where the tombstone used to be and found nothing for that key, so it never touched the in-memory record still sitting there from before the isolation: active.

Nothing in that path throws an error. The consumer isn't lagging, the topic isn't corrupt, the restore completes successfully. It just completes with a state store that is now permanently wrong for one key, with no future event on the topic ever going to fix it, because as far as the topic is concerned that key's history now starts after the revoke.

the fix

Immediate: force the three pods to drop their local state directory and rebuild from the earliest offset rather than delta-restoring from a checkpoint that could no longer be trusted.



recovery, run once per affected pod

kubectl exec authz-gateway-7f2a-xyz -- rm -rf /data/kafka-streams/authz-gateway
kubectl delete pod authz-gateway-7f2a-xyz

Then the two changes meant to keep this from being invisible again. First, a health check that actually reflects Kafka connectivity instead of just the local HTTP server:



gateway/health.js, after

let lastPollAt = Date.now();

consumer.on('poll', () => {
lastPollAt = Date.now();
});

app.get('/healthz', (req, res) => {
const staleMs = Date.now() - lastPollAt;
if (staleMs > 60_000) {
return res.status(503).json({ error: kafka poll stale for \${staleMs}ms\ });
}
res.status(200).json({ ok: true });
});

Second, delete.retention.ms on api-key-status went from 24 hours to 7 days, trading disk for a grace window wide enough to cover a multi-day network isolation, not just a routine restart. It doesn't fix the underlying risk, a consumer down longer than the retention window can still miss a tombstone, but it moves the failure mode from "a botched network policy" to "something has been broken for a week and nobody noticed," which is a bar this team was willing to accept.

the aftermath

96 hrs Time between the revoke and the fraud job catching it in ap-south-1

3 Requests authorized with the revoked key before detection

24h → 7d delete.retention.ms on api-key-status

2 Other Kafka-backed services found with the same HTTP-only healthcheck

Static group membership solves rebalance storms during deploys, but it also means a consumer that's gone quiet for an unrelated reason keeps its assignment and its local state without raising anything from outside. It needs its own signal, not the assumption that a healthy pod implies a healthy consumer.
delete.retention.ms is a grace window, not a guarantee. It's sized against how long a consumer might lag, not against how long one might be completely unreachable, and those are different failure modes with very different durations.
A local state store that restores by delta from a checkpoint is fast exactly because it trusts the checkpoint. That trust is only as good as the log's willingness to still contain everything between the checkpoint and now, and compaction doesn't make that promise indefinitely.
A healthcheck that only proves a process is running, not that it's connected to the one dependency that determines whether its answers are correct, will report green through the exact failure it exists to catch.

Kafka never logged an error either. It deleted exactly what its retention setting told it to delete, on schedule, while the only three pods that needed to see it first were on the wrong side of a firewall rule.

How Skipping a Cache Reset for Speed Let Apollo Client Show the Wrong Tenant

Darshan Turakhia — Wed, 23 Sep 2026 10:55:43 +0000

09:41 PST. A support lead pastes a screenshot into the incident channel: her own screen, mid quick-switch into Acme Corp's account, showing three ticket subjects that mention Globex Inc, a different customer entirely. The message underneath is two words: "is this normal."

the setup

The internal tool support agents used to manage tickets across customer accounts was a multi-tenant React app on top of a GraphQL API, Apollo Client for data fetching. Tenant scoping happened entirely server-side: every request carried a JWT with a tenantId claim, and every resolver filtered its database queries by that claim. The GraphQL queries themselves never mentioned a tenant. A ticket list looked like this, the same query text for every account an agent ever loaded:



TicketsOverview.graphql

query TicketsOverview {
tickets {
id
subject
status
customer
}
}

That was a deliberate simplification. Plumbing a tenant argument through every query in the app would have meant threading it through dozens of components for no functional gain, since the server already refused to return data outside the caller's token. Nobody on the frontend team had reason to think of tenant identity as something the client needed to track for itself.

Six weeks earlier, product shipped Quick Switch: a dropdown that let an agent jump from one customer account to another without a full sign-out and page reload. The old flow called client.resetStore() on every account change, which cleared the Apollo cache and refetched every active query, a UX that took 300 to 500ms and showed a loading spinner over the whole page. Quick Switch skipped that on purpose, to make account switching feel instant:



useQuickSwitch.ts, before

async function switchTenant(tenantId: string) {
const { accessToken } = await api.post('/auth/switch-tenant', { tenantId });
setAccessToken(accessToken);
// resetStore() intentionally skipped here — it added a visible
// loading flash and product wanted the switch to feel instant.
}

Every query in the app used Apollo's default cache-first fetch policy. That was fine for a single-tenant session: read from cache if present, otherwise hit the network. Nobody had reasoned through what cache-first does the instant the same query, with the same variables, needs to mean something different because the identity behind it changed.

the scramble

First theory: a session bug on the auth service, a Redis-backed token cache serving a stale tenantId claim to the wrong pod. The on-call engineer pulled the auth service's logs for the support lead's account switch. The issued JWT was correct, Acme Corp's tenant ID, timestamped to the millisecond of the click. Ruled out within ten minutes.

Second theory: the GraphQL gateway was routing to a stale replica that hadn't caught up on a recent write, some kind of read-after-write lag. Someone checked the actual network response in the support lead's browser recording, the one piece of evidence that made this theory look promising at first. The GraphQL response body for TicketsOverview, once it arrived, contained exactly Acme Corp's tickets. Correct tenant, correct data, right there in the payload.

That was the detail that stalled the investigation for the better part of an hour: every request and every response, read individually, was correct. The bug wasn't in anything that crossed the network.

the hunt

The break came from reproducing it deliberately instead of waiting for another report. Someone throttled their local network to Slow 3G in dev tools, specifically to stretch out whatever window the bug lived in, then opened Apollo Client Devtools and watched the TicketsOverview cache entry while triggering Quick Switch.

The component re-rendered twice. The first render, milliseconds after the switch, showed the previous tenant's tickets, read straight from cache. The second render, 800ms to 1.5 seconds later depending on network conditions, showed the correct tenant's tickets, once the network response landed and overwrote the cache entry.

Apollo Client's default cache key is the query name plus its serialized variables. TicketsOverview had no variables. Every tenant, every session, every agent, all produced the identical cache key: TicketsOverview:{}. The cache had no idea two different customers' data had ever passed through that key, because from the cache's perspective, nothing about the query had changed. Tenant identity lived exclusively in an HTTP header the cache never looked at.



what cache-first actually did on switch

Agent clicks "Switch to Acme Corp"
New JWT (tenantId=acme) stored, old JWT (tenantId=globex) discarded
TicketsOverview component re-renders
Apollo checks cache for key TicketsOverview:{} -> HIT (Globex's data, still cached)
cache-first policy: return the hit immediately, refetch in background
UI paints Globex's tickets under Acme Corp's header, for 0.8-1.5s
Network response for Acme's data arrives, overwrites cache, UI re-renders correctly

the find

Root cause: the Apollo Client cache had no concept of tenant identity, because tenant scoping had only ever been designed as a server-side concern. That was a correct design for the network layer, the server never returned the wrong tenant's data. It was an incomplete design for the client, because cache-first answers queries from a cache that has no way to know the caller's identity changed. Quick Switch removed the one step, resetStore(), that had been silently compensating for that gap by wiping the cache clean on every identity change. Once that step was gone, the gap was directly exposed to whoever happened to switch accounts fastest.

the fix

The immediate fix restored a cache clear on switch, but a narrower one than the old full resetStore(), which also triggers a refetch of every active query on the page, most of which weren't tenant-scoped UI at all:



useQuickSwitch.ts, after

async function switchTenant(tenantId: string) {
const { accessToken } = await api.post('/auth/switch-tenant', { tenantId });
await client.clearStore(); // empties the cache, no refetch of inactive queries
setAccessToken(accessToken);
// active queries now re-render against an empty cache and fetch fresh,
// instead of painting stale data first
}

clearStore() empties the cache without immediately refetching everything the way resetStore() does, so active components see a brief loading state rather than a full-page spinner, then fetch clean. That closed the immediate hole: an agent now sees a loading skeleton for a beat, never another customer's data.

The structural fix addressed the actual gap, that the cache had no tenant awareness at all, so the next feature that skips a manual clear doesn't reopen the same hole. Every tenant-scoped query now carries an explicit tenantId variable sourced from a reactive variable tied to the active session, and a type policy scopes the cache key by it:



cache.ts

export const activeTenantVar = makeVar(null);

export const cache = new InMemoryCache({
typePolicies: {
Query: {
fields: {
tickets: {
keyArgs: ['tenantId'],
},
},
},
},
});



TicketsOverview.graphql, after

query TicketsOverview($tenantId: ID!) {
tickets(tenantId: $tenantId) {
id
subject
status
customer
}
}

With tenantId in both the query variables and the type policy's keyArgs, Acme Corp's tickets and Globex's tickets now live under distinct cache keys, the same way two different pages of a paginated list already did. Even if a future feature forgets to clear the store on an identity change, the cache itself can no longer confuse one tenant's entry for another's; the identity is part of the key, not an invisible side channel.

the aftermath

1,900 Quick Switch actions recorded in the 30 days before the fix

46 Switches, across 12 agents, where session telemetry showed the stale render held for over 500ms

6 weeks Time between Quick Switch shipping and the first reported sighting

0 Stale cross-tenant renders recorded since the tenantId-scoped cache shipped

Because this was an internal tool, not customer-facing, no external data was exposed. It still landed as a corrective action under the company's SOC 2 tenant-isolation control, since that commitment covers who can see whose data regardless of which side of the product a screen sits on. A runtime assertion now runs on every tenant-scoped cache read in production: an Apollo Link compares the tenant embedded in a cached response against the tenant on the current session's JWT, and logs an alert if they ever disagree again. A synthetic test performs a scripted Quick Switch on a throttled connection in CI and fails the build if any tenant-scoped field renders data tagged with a different tenant ID than the active session, even for a single frame.

A cache-first policy is only as safe as the assumption that the same query and variables always mean the same data. Tenant identity, or any other value carried outside the query itself, breaks that assumption the moment it changes without the cache knowing.
Removing a slow safety step to make something feel instant is a real product tradeoff, not automatically a mistake, but it needs to be made with full knowledge of what the slow step was actually doing. Here it was quietly acting as the only thing scoping the cache by identity.
A bug built entirely out of individually correct network requests and responses won't show up by inspecting the network tab alone. It has to be watched as a render sequence over time, which is why throttling the connection to stretch the window mattered more than any single log line.
If a value determines which data a query is allowed to return, it belongs in the cache key, not only in a header. Otherwise the client is trusting the server's authorization to also double as the client's own cache boundary, and those are two different systems that happen to usually agree.

The server had been doing its job the entire six weeks, never once returning a ticket to a tenant that didn't own it. The leak was one layer up, in a cache that had no way of knowing two different customers had ever asked it the exact same question.

How a Retry Loop

Darshan Turakhia — Sat, 19 Sep 2026 10:55:58 +0000

03:14 UTC. Site search starts returning 503s. Not slow, not degraded — down. The on-call engineer's first Kibana query for search-service error rates times out too, which is its own bad sign, because Kibana is querying the same cluster that's supposedly the problem.

the setup

The cluster in question is a six-node self-managed Elasticsearch 8.x deployment handling two very different jobs. During the day it serves product and content search for the site, low latency, high query volume, nothing exotic. Overnight it also feeds a batch export: a Node job called order-events-exporter that scrolls the order_events index, about 600 million documents, and streams every doc into Snowflake for the analytics team. It's run at 03:00 UTC for two years without incident.

The export uses the Scroll API, which works by taking a point-in-time snapshot of the index's segments the moment the first request runs, then letting you page through that snapshot with a scroll_id until you're done. The snapshot is what makes it consistent, but keeping it alive costs the cluster something: every open scroll context pins the segments it's reading in place, which blocks merges and keeps otherwise-deletable segments resident in memory until the context is closed or its keep-alive expires.

the scramble

The paged engineer's first theory was a hot shard from a relevance-ranking deploy that had shipped the previous afternoon. It touched the search query builder, so it was the obvious suspect. Rolling it back didn't help; the 503s kept coming at the same rate.

Second theory: traffic spike. The dashboard showed request volume roughly flat against the same time yesterday, which ruled that out inside a few minutes, but the CPU graph in the same panel was doing something odd, climbing in a slow, steady ramp rather than the spiky pattern a traffic surge produces. Steady ramps usually mean something is accumulating, not spiking.

Third theory, and the one that actually got investigated properly: a JVM heap problem. Old-gen usage on the two data nodes handling the order_events shards was sitting above 90%, high enough to explain GC pauses long enough to look like request timeouts. But "the heap is full" isn't a root cause, it's a symptom, and nobody could yet say full of what.

the hunt

The cluster logs had already said what was wrong, twenty minutes before the page fired. It just hadn't been paired with the right alert:



elasticsearch.log, 02:54:11 UTC

[order-events-node-2] TooManyScrollContextsException: Trying to create too many scroll
contexts. Must be less than or equal to: [500]. This limit can be set by changing the
[search.max_open_scroll_context] setting.

That exception is real and specific: Elasticsearch caps concurrent open scroll contexts at 500 by default, cluster-wide, precisely because each one is a standing memory liability. Once the cap is hit, every new scroll request gets rejected, which explained why the export job had been failing since shortly after 02:54, but not yet why site search itself was down; a rejected scroll creation shouldn't touch ordinary, non-scroll search queries at all.

GET _nodes/stats/indices/search gave the number that connected the two:



GET \_nodes/stats/indices/search

"order-events-node-2": {
"search": {
"open_contexts": 1847,
"scroll_current": 1847,
"scroll_total": 2214
}
}

1,847 open contexts, well past the 500 the cluster is supposed to allow. That number only makes sense if contexts were being created faster than they were ever being closed or expired, and closed faster than they were also being renewed past their keep-alive, which pointed straight at the export job's retry logic:



order-events-exporter.js, before

async function exportBatch() {
for (let attempt = 0; attempt < 5; attempt++) {
try {
let res = await client.search({
index: 'order_events',
scroll: '10m',
size: 5000,
body: { query: { match_all: {} } },
});
return await drainScroll(res);
} catch (err) {
log.warn(export attempt \${attempt} failed, retrying\, err);
// falls through to the next loop iteration and calls
// client.search() again from scratch
}
}
throw new Error('export failed after 5 attempts');
}

On a client-side timeout, the catch block didn't resume the existing scroll_id, because it never had one to resume, the request had already timed out client-side before the response carrying it arrived. It just looped and called client.search() again, which creates a brand-new scroll context server-side, every single time. The old context the timed-out request had actually created was never told to close.

Normally this would be forgiving. A scroll context's default keep-alive is short, and an abandoned one expires and gets cleaned up on its own. But eight months earlier, a different slowness ticket had bumped this job's keep-alive from 1m to 10m "to give the export more breathing room," and three weeks before this incident, a mapping change had added a large nested line_items field to order_events to support a new analytics report. Scroll pages that used to return in roughly 80ms were now taking 600-700ms under load, comfortably past the export client's 5-second per-page timeout during the cluster's overnight compaction window. The job was retrying constantly, each retry left a fresh 10-minute-lived context behind, and contexts were piling up roughly ten times faster than they were expiring.

the find

Root cause: the export job's retry loop restarted the scroll from scratch on every timeout instead of resuming or explicitly closing the abandoned context, and a keep-alive that had been widened months earlier for an unrelated fix let each orphaned context live ten times longer than the default. Combined with a recent mapping change that slowed scroll pages enough to trigger the retry loop constantly, contexts accumulated past the 500-context cap within about twelve minutes of the job starting. Past that point, the pinned segments those contexts held open pushed old-gen heap usage over the cluster's parent circuit breaker threshold, which trips for all request types, not just scroll, once triggered:



elasticsearch.log, 03:13 UTC — one minute before the page

[order-events-node-2] [parent] Data too large, data for [] would be
[16.2gb/95%], which is larger than the limit of [15.9gb/94%], real usage: [16.1gb],
new bytes reserved: [112kb]: circuit_breaking_exception

That's the line that took down site search. The parent circuit breaker doesn't distinguish between a batch export's scroll request and a shopper's product query, once it trips it rejects both.

the fix

Immediate mitigation was a manual context flush, which closes every open scroll on the cluster and gives the circuit breaker room to reset:



manual recovery during the incident

curl -X DELETE "localhost:9200/_search/scroll/_all"

Heap usage dropped from 95% to 61% within about ninety seconds and search traffic recovered on its own. The durable fix had two parts. First, the retry logic now resumes by scroll_id and always cleans up on final failure instead of leaking:



order-events-exporter.js, after

async function exportBatch() {
let scrollId;
try {
let res = await client.search({
index: 'order_events',
scroll: '1m',
size: 5000,
body: { query: { match_all: {} } },
});
scrollId = res._scroll_id;
return await drainScroll(res, scrollId);
} finally {
if (scrollId) {
await client.clearScroll({ scroll_id: scrollId }).catch(() => {});
}
}
}

Second, the export was migrated off the Scroll API entirely, onto search_after with a point-in-time (PIT) ID, which is Elasticsearch's own recommended replacement for scroll-based deep pagination and defaults to a much shorter, non-negotiable keep-alive per request rather than one long-lived session an app can forget to close. The keep-alive that had been widened months earlier no longer exists as a setting to accidentally leave too generous.

the aftermath

38 min Site search returning 503s cluster-wide

1,847 Peak open scroll contexts against a default cap of 500

12 min Time from job start to the context cap being exceeded

0 Incidents since migrating off Scroll to search_after + PIT

A new Datadog metric now tracks open_contexts per node with an alert at 300, 60% of the default cap, so the next accumulation gets caught while it's still just a batch job failing, not the whole cluster.

A retry loop that doesn't know how to resume isn't retrying, it's restarting, and restarting a stateful server-side operation without cleaning up the state it already created is how you turn a transient timeout into a resource leak.
A keep-alive bump made for one incident months ago can quietly change the blast radius of an unrelated bug later. Nobody revisiting the retry logic knew the keep-alive had been widened, or that it mattered.
Elasticsearch's circuit breaker protects the whole cluster by design, which is exactly why an isolated batch job's misbehavior became a site-wide outage instead of staying contained to itself.
TooManyScrollContextsException was in the logs twenty minutes before the page fired. The gap wasn't missing information, it was a missing alert on a log line nobody had thought to wire up, because nothing had ever hit that limit before.

The Scroll API had been the right tool for this export for two years. It only stopped being the right tool the moment something downstream, a slower mapping, a wider keep-alive, a retry loop that didn't know its own state, started leaning on the one property Scroll assumes you'll respect: that whoever opens a context is the one responsible for closing it.

B-Tree vs LSM-Tree: the storage-engine tradeoff behind every database you use

Darshan Turakhia — Tue, 15 Sep 2026 17:32:22 +0000

Every relational database you've ever used made a bet at the storage-engine level, before you wrote a single query: reads matter more than writes, or writes matter more than reads. That bet shows up as a choice between two data structures, a B-tree or an LSM-tree. Most engineers never think about it. It's still deciding how their database behaves under load.

The actual tradeoff

A B-tree updates data in place. It walks down to the exact leaf page a key belongs on and rewrites it right there, so the tree stays fully sorted at all times. A read never does more than follow one path from root to leaf. There's no ambiguity about where a key lives, because there's only ever one copy of it.

An LSM-tree (log-structured merge-tree) refuses to do that. It never touches old data on a write. Instead it buffers writes in memory, flushes them to new immutable files on disk, and merges those files together later in the background. A write is a sequential append, not a random rewrite, which is why LSM-trees dominate anywhere ingest volume is the bottleneck: Cassandra, RocksDB, LevelDB, HBase. B-trees stay the default for read-heavy, point-lookup workloads: Postgres, MySQL's InnoDB, basically every traditional RDBMS.

What's worth understanding is what each side gives up, not which one "wins."

B-trees pay at write time. Once your dataset outgrows memory, a page update that used to be a cheap in-memory write becomes a real disk seek. Under heavy write load the tree also spends real time rebalancing, splitting pages as they fill so every leaf stays at the same depth.

LSM-trees pay at read time instead. A single key can exist simultaneously in the in-memory memtable, the most recently flushed file, and several older files. The newest write wins, but a lookup has to check each location until it finds a match. Bloom filters make this bearable in practice (a probabilistic "this file definitely doesn't have your key" check that skips most files without opening them), but a worst-case LSM-tree lookup is still checking more places than a B-tree ever has to.

There's also a cost LSM-trees defer rather than eliminate: compaction, the background process merging small files into larger ones and dropping superseded key versions. This is the part people underestimate. It's not free maintenance sitting quietly in the background, it's competing for the same disk I/O and CPU your live traffic needs. A compaction backlog is the classic LSM-tree failure mode, and it's an ugly one: writes outpace merging, read latency climbs as lookups check more uncompacted files, and the whole thing spirals if nothing throttles incoming writes to let compaction catch up. If you've operated Cassandra under sustained write pressure, you've probably met this one at 3am.

Where this actually bites you

This decision almost never gets made consciously by application code. It gets made once, implicitly, by whichever database a team picked years ago. So the first time most engineers actually think about it is after a write-heavy workload starts struggling on a B-tree-backed system that was never built for it. Time-series data, event logging, anything ingesting far more than it reads back in the same window: that's LSM-tree territory. Building it on Postgres because "that's what we use" is how you end up fighting the storage engine instead of your actual problem.

Some engines hedge instead of picking a side outright. MySQL's MyRocks storage engine swaps InnoDB's B-tree for an LSM-tree underneath the same relational interface, specifically for workloads write-bound enough to justify the read-side cost.

I wrote up the fuller comparison, including read and write amplification and where each structure shows up in real systems, here, if you want the longer version of this post.

Seeing it instead of reading about it

The part that's genuinely hard to build intuition for from a blog post is why a B-tree stays shallow even at billions of rows. So I built an interactive B-tree visualizer. Insert keys and watch a node split once it fills, the middle key pushing up into the parent every time. Search afterward and it highlights exactly which nodes the lookup touches. Entirely client-side, no backend. Took me about four minutes of playing with it to have the mechanism actually click in a way the text version never quite managed.

Next time someone asks why adding an index didn't help their write-heavy table, or why their time-series database "just feels different" from Postgres under load, this is the mechanism underneath both answers.