DEV Community: Darshan Turakhia

How a Zero-Downtime Table Swap Silently Dropped Orders From Postgres Logical Replication for 91 Hours

Darshan Turakhia — Tue, 06 Oct 2026 10:59:16 +0000

09:14 UTC, Thursday. #finance-eng: "Why has daily revenue been flat at $118,402 for four days straight? We ran a 20%-off promo Tuesday, this should have spiked, not flatlined." No PagerDuty alert fired. No panel on the production dashboard is red. Checkout is taking orders normally. Whatever broke didn't trip a single automated check, because nothing in production is actually broken.

the scramble

First theory: the promo's attribution tracking is broken, not revenue itself. Marketing's UTM tagging has flaked before. Someone pulls the Stripe payouts dashboard directly, bypassing the internal reporting pipeline entirely. Payouts for the last four days are up, in line with the promo. Money is moving. The number everyone is staring at is wrong, not the business.

Second theory: the BI tool is serving a stale cached query. The finance dashboard sits on top of an internal analytics Postgres instance, fed by logical replication from the primary rather than queried against production directly, a deliberate choice made a year ago so heavy reporting queries can never compete with checkout for connections or I/O. Someone forces a hard refresh on the dashboard, bypassing the BI tool's own cache layer. Same flat number.

Third theory, and the one that finally moves: query the analytics replica directly instead of through the dashboard.



09:41 UTC, analytics replica

SELECT count(*), max(created_at) FROM orders;

count	max
9,214,880	2026-10-02 14:02:41+00

-- same query against production, same moment
SELECT count(*), max(created_at) FROM orders;

count	max
9,222,563	2026-10-06 09:40:12+00

Production has 7,683 more rows than the replica, and the replica's newest row is timestamped almost exactly four days in the past. That timestamp isn't a random stale point. It's 12 minutes after the rename-swap migration that shipped Tuesday afternoon. The dashboard was never broken. The data feeding it stopped arriving.

the hunt

Tuesday's deploy converted the orders table's primary key from int4 to int8 using the standard zero-downtime shadow-table pattern: build orders_new with the wider key, backfill it in batches, dual-write via trigger while the backfill catches up, then an atomic rename swap inside one transaction:



Tuesday 14:02:17 UTC, cutover

BEGIN;
ALTER TABLE orders RENAME TO orders_old;
ALTER TABLE orders_new RENAME TO orders;
COMMIT;

Clean, fast, no lock held longer than the two renames. The on-call engineer who ran it watched checkout error rates for twenty minutes afterward and saw nothing. By every metric anyone was watching, the migration worked.

First check on the replication path: is the slot even still alive?



09:52 UTC, primary

SELECT slot_name, active, confirmed_flush_lsn
FROM pg_replication_slots
WHERE slot_name = 'analytics_sub';

slot_name	active	confirmed_flush_lsn
analytics_sub	t	4A2/8F1C3D00

-- run again two minutes later
analytics_sub | t | 4A2/8F22A188

Active, and the LSN is climbing between queries. That reads as healthy replication by every normal heuristic, and it's the dead end that costs the most time: the publication backing this subscription also covers customers, invoices, and three other tables that are still being written to constantly. Those writes keep the logical decoding worker busy and the confirmed LSN advancing regardless of what's happening with orders specifically. A slot-level health check can look perfectly fine while one table inside it has gone completely silent.

The next check is narrower, not "is the slot alive" but "what tables is this publication actually sending":



10:08 UTC, primary

SELECT pubname, tablename FROM pg_publication_tables
WHERE pubname = 'orders_pub';

pubname	tablename
orders_pub	orders_old
orders_pub	customers
orders_pub	invoices

orders_old. Not orders. The publication has been faithfully, correctly, silently replicating a table that nothing has written to since 14:02:17 UTC on Tuesday.

the find

ALTER PUBLICATION ... ADD TABLE resolves the table name to its object ID at the moment the command runs and stores that OID in pg_publication_rel. It does not store the name. From that point forward, the publication tracks whatever object owns that OID, not whatever object currently answers to the name orders.

The rename-swap pattern is specifically designed to be atomic and cheap, which it achieves by never actually copying data at cutover time. It just reassigns two names to two existing relations. The relation that was orders for the three years before Tuesday did not stop existing. It got renamed to orders_old and kept its OID, and the publication kept following that OID exactly as configured. The relation that's been named orders since 14:02:17 UTC is a different object with a different OID, one the publication was never told about, because the migration added a new table under a temporary name and the publication definition was never revisited after the swap.

Nothing in this chain is a bug. The publication did precisely what ALTER PUBLICATION ADD TABLE orders asked it to do, a year ago, for the table that was orders at the time. A rename is not a schema change the replication system has any reason to treat as suspicious, since tables get renamed for all kinds of reasons that have nothing to do with swapping in a different relation underneath a stable name. There was no constraint violation, no decoding error, no WAL gap, nothing for any alert to fire on. The failure mode is a correctly configured system doing exactly what it was told, applied to an assumption, "the table named orders is the table I meant," that a swap-based migration quietly invalidated.

the fix

Point the publication at the table that's actually live, and drop the dead one once its replicated history is no longer needed on the subscriber:



10:31 UTC, primary

ALTER PUBLICATION orders_pub DROP TABLE orders_old;
ALTER PUBLICATION orders_pub ADD TABLE orders;

Adding a table to an existing publication doesn't push any of its current rows to subscribers by itself. It only starts streaming changes from that point forward. The subscriber still needs the 91 hours of orders it missed, plus confirmation that its local orders table matches the live one row for row, not just going forward from here:



10:33 UTC, analytics replica

ALTER SUBSCRIPTION analytics_sub REFRESH PUBLICATION WITH (copy_data = true);
-- 11 minutes to copy and index 9.2M rows

REFRESH PUBLICATION diffs the subscription's known relations against the publication's current membership, and for anything newly added it runs a full initial copy for that relation only, while customers and invoices keep streaming without interruption the entire time. Eleven minutes later the replica's orders table matches production exactly, and the finance dashboard's next scheduled refresh picks up four days of orders in one jump.

The longer-term fix is a check that doesn't depend on anyone remembering to update the publication by hand during a migration that was never framed as a replication change in the first place:



nightly parity check, new cron

-- runs against every table/subscriber pair, alerts on the table name, not the slot
SELECT p.tablename,
now() - max(o.created_at) AS staleness
FROM pg_publication_tables p
JOIN orders o ON true
WHERE p.pubname = 'orders_pub' AND p.tablename = 'orders'
GROUP BY p.tablename
HAVING now() - max(o.created_at) > interval '2 hours';

It also became a standing step in the shadow-table migration runbook: any rename-swap that touches a table covered by logical replication requires a publication membership check in the same change window as the cutover, not as a follow-up ticket filed whenever someone happens to notice.

the aftermath

91 hrs Between cutover and the orders table rejoining replication

7,683 Orders missing from the analytics replica at discovery

$0 Actual revenue lost — Stripe and production had every order

11 min To resync the table via a targeted copy_data refresh

Postgres publications bind to a table's object ID at ADD TABLE time, not its name. A rename-swap migration that reassigns an existing name to a new relation is invisible to that binding — the publication keeps following the OID, which is now the wrong table.
A healthy replication slot (active, LSN advancing) says the slot is working. It says nothing about whether a specific table inside a multi-table publication is still part of that stream. Check pg_publication_tables directly when one table's data looks stale; don't infer table-level health from slot-level metrics.
ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_data = true) resyncs only the relations that are new to the subscription. It's a safe, targeted recovery step, not a full subscription rebuild, and it doesn't interrupt tables that were never affected.
Any migration pattern built around renaming tables, not just this one, should carry an explicit step to re-check every publication and every foreign key, trigger, or grant that was bound to the old name by object reference rather than by name. A clean rename at the schema level can still leave dependent systems pointed at the wrong object.

The migration itself worked exactly as designed, start to finish, with zero downtime and zero checkout errors. The thing it broke wasn't checkout. It was a dependency three systems away that nobody thought to list as a stakeholder for a primary-key type change, and the only signal it ever produced was a dashboard that quietly stopped changing.

Reimplementing a Trie Reminded Me How Autocomplete Actually Works

Darshan Turakhia — Fri, 02 Oct 2026 14:06:26 +0000

I spent an embarrassing amount of time last week reimplementing something I'd "known" since a data structures class a decade ago: the trie. Typing it out from scratch again reminded me why it's one of the few data structures that actually earns the "elegant" label people throw around too easily.

Here's the pitch in one sentence: a trie stores strings by character, along shared paths from a root, so two words with the same prefix walk the exact same nodes until they stop agreeing. Insert "car," "card," and "care," and you get one shared path for "c-a-r," which then splits into three different endings. Nothing about adding "care" touches the nodes "car" and "card" already built.

That sharing is the whole point, and it's easy to undersell how much it buys you. A hash set can tell you "yes, this exact string is in the collection." That's it. It cannot tell you "give me everything that starts with these three letters" without scanning the entire set and checking each entry one by one. A trie answers that second question by walking to the end of the prefix and then just... looking at what's underneath. Lookup cost depends on how long your prefix is, not on how many million other strings happen to be sitting next to it in memory. That's the entire mechanism behind every autocomplete dropdown you've ever typed into: walk the prefix, then collect every real word hanging off wherever you land.

The part that actually bit me while rebuilding this: you need an explicit marker for "a real word ends here." It sounds obvious written down, but it's the single easiest thing to forget, and the bug it causes is sneaky. Say you only ever insert "card." The path c → a → r → d exists in your trie. All four nodes are real. But "car" was never inserted as its own entry, it's just a prefix that happens to lead somewhere. If your node doesn't carry a boolean flag for "an insertion actually terminated here," you have no way to distinguish a real stored word from a string that merely happens to be a prefix of a longer one. I've seen (and once written) autocomplete bugs where a partial prefix gets suggested as a complete match because of exactly this missing flag.

The tradeoff nobody mentions in the five-minute version of this concept: a naive implementation wastes a lot of memory. The textbook approach gives every node a fixed-size array, one slot per possible next character. Twenty-six slots if you're doing lowercase English, more if you need digits or punctuation or, god forbid, full Unicode. Most of those slots sit empty on any node that doesn't branch in every direction, and that waste adds up fast across a trie with real-world vocabulary in it. The fix is one of two things: back each node with a hash map instead of a fixed array, so you only pay for children that actually exist, or compress the whole structure with something like a radix trie, which collapses long runs of single-child nodes into one node holding a whole substring instead of one character. "Cardboard" doesn't need five separate one-character hops after "card," it needs one node labeled "board."

Autocomplete is the use case everyone reaches for first, and it's a good one, but it's not the only place this shows up. Spell-checkers use the same walk to confirm a word exists in a dictionary and to find near-miss suggestions by exploring nearby paths. IP routers use a binary version of the same idea, walking a trie over address bits instead of letters, to do longest-prefix-match routing: find the most specific rule that matches a destination address. Different alphabet, same shared-prefix-sharing-a-path trick underneath.

I ended up writing both a longer explanation and a small interactive visualizer, mostly because I wanted to actually watch the branching happen rather than just trust my own mental model of it. If you want to poke at it yourself, the guide walks through the insert/search mechanics and the end-of-word gotcha in more detail, and the visualizer lets you insert your own words and watch the prefix lookup highlight in real time, suggestions and all.

If you've never implemented one, it's a genuinely good weekend exercise. Small enough to build in an hour. Annoying enough, in the best way, that you'll start second-guessing every search box you touch afterward.

How a Rolling Deploy

Darshan Turakhia — Tue, 29 Sep 2026 10:58:00 +0000

13:52 UTC. p99 latency on pricing-svc alerts at 1.8 seconds, four times its usual ceiling. The dashboard shows six healthy pods, six pods passing their readiness probe, six pods with room on their CPU requests. One of them is pegged at 97%. The other five are idling under 12%.

the setup

checkout-api calls pricing-svc over gRPC for every cart total, roughly 4,000 RPCs a minute at midday. Both run in the same namespace, and pricing-svc is exposed through a standard ClusterIP Service, resolved once by DNS to a single virtual IP with kube-proxy handling the fan-out to whichever pods are in the endpoint list.



checkout-api/pricing-client.ts

const client = new PricingServiceClient(
'pricing-svc.default.svc.cluster.local:50051',
grpc.credentials.createInsecure()
);
// one channel per pod, reused for every call it makes

That last line is the part nobody thought hard about. gRPC runs on HTTP/2, and HTTP/2 multiplexes many concurrent RPCs over a single TCP connection. A gRPC client opens one connection to a target and keeps it open, reusing it for every subsequent call instead of opening a new one per request. For a REST client that reconnects constantly, ClusterIP round robin evens out over thousands of short-lived connections. For a gRPC client, the load balancing decision happens exactly once, at connection time, and then holds for as long as that connection stays up. checkout-api runs eight pods. Each one makes that decision independently, once, on startup.

the scramble

First theory: bad node. The hot pod, pricing-svc-7d4f-x2k9, is cordoned and drained, forcing it onto a different node entirely. Eleven minutes later the alert fires again, same symptom, different pod, pricing-svc-7d4f-m8p1, now pegged while the rest idle. Different node, different hardware, same shape of failure. That rules out anything node-specific and burns fifteen minutes doing it.

Second theory: the HPA is fighting itself, scaling on a metric that doesn't reflect real load. kubectl describe hpa pricing-svc shows six of six replicas, target CPU utilization at 60%, current average across the deployment sitting at 22%. The average looks fine because five pods are nearly idle. Averages hide a skew this sharp, and the HPA has no reason to add capacity when its own number says the fleet is underloaded.

the hunt

Instead of watching pods cycle, the next step is watching the connections themselves. Shelling into a checkout-api pod and checking its open sockets to the pricing service:



kubectl exec checkout-api-9f3a-vv2q -- ss -tnp | grep :50051

ESTAB 0 0 10.2.4.19:44822 10.2.6.31:50051

One line. One connection. Every one of the eight checkout-api pods shows the exact same thing: a single established socket to a single backend IP, held open since the pod started. Cross-referencing those destination IPs against the pod list settles it: all eight client pods are connected to the same two pricing-svc pods, and nothing is connected to the other four at all.

A Prometheus query against grpc_server_handled_total, split by pod, shows just how lopsided it got: those two pods have handled 83% of all RPCs since the timestamp on a checkout-api deploy from six hours earlier, a routine dependency bump with nothing in the diff touching networking. The deploy itself didn't cause an error. It just decided, once, where six hours of subsequent traffic would go.

the find

The rollout replaced all eight checkout-api pods inside a forty-second window, standard behavior for a deployment with no maxUnavailable tuning. At the moment those eight new pods came up and each opened its one gRPC connection, pricing-svc was mid-rollout too, from an unrelated change earlier that morning. Only two of its six pods had passed the readiness probe and made it into the Service's endpoint list. kube-proxy's iptables rules can only route to what's registered, so every new connection from every checkout-api pod landed on one of those two.

By the time the other four pricing-svc pods turned ready, thirty seconds later, it didn't matter. The gRPC client's default load balancing policy is pick_first: resolve the target once, connect to the first address, and stay connected until that connection breaks. It doesn't re-resolve, doesn't rebalance, doesn't notice four more pods just joined the pool. The connection from that forty-second window was healthy, so it just kept being used, for six hours, until midday traffic pushed those two pods past the point where 1.8-second p99 was survivable.

the fix

Immediate: restart the two hot pricing-svc pods. Their connections drop, the eight checkout-api pods reconnect, and this time all six backends are already ready, so the new connections spread out. Latency back to baseline in under two minutes. That's a mitigation, not a fix, since the exact same readiness race can happen on the next deploy.

The real fix is making the client aware there's more than one backend to pick from. That means a headless Service, so DNS returns every pod IP instead of one virtual IP, paired with a client-side load balancing policy that actually uses all of them:



k8s/pricing-svc-headless.yaml

apiVersion: v1
kind: Service
metadata:
name: pricing-svc-headless
spec:
clusterIP: None
selector:
app: pricing-svc
ports:
- port: 50051



checkout-api/pricing-client.ts, after

const client = new PricingServiceClient(
'dns:///pricing-svc-headless.default.svc.cluster.local:50051',
grpc.credentials.createInsecure(),
{ 'grpc.lb_policy': 'round_robin' }
);

With round_robin, the client resolves every backend IP and opens a connection to each one, spreading calls across all of them instead of pinning to whichever pod answered first. As a second layer, pricing-svc now caps how long any single connection can live, so even a client stuck on an old load balancing policy is forced to reconnect and re-resolve periodically instead of holding one connection indefinitely:



pricing-svc/server.ts

const server = new grpc.Server({
'grpc.max_connection_age_ms': 5 * 60 * 1000,
'grpc.max_connection_age_grace_ms': 30 * 1000,
});

the aftermath

83% Of all pricing-svc RPCs handled by 2 of 6 pods for six hours

1.8s p99 latency at the point the imbalance became visible

5 min max_connection_age_ms added as a floor on any future skew

3 Other internal gRPC clients found using ClusterIP + pick_first

A ClusterIP Service load balances TCP connections, not requests. That distinction only matters for protocols that hold a connection open and reuse it, which is exactly what HTTP/2 and gRPC are built to do.
pick_first is a reasonable default for a client with one backend. Against a Service backed by multiple pods, it turns whatever pod answered first during a narrow readiness window into the only pod that matters, indefinitely.
Average CPU across a deployment can look completely healthy while two pods carry the entire fleet. Per-pod metrics on anything that does client-side connection reuse are not optional, they're the only view that shows this failure mode at all.
A connection age limit on the server is a cheap insurance policy even after fixing the client. It bounds how long any future imbalance, caused by a bug nobody's found yet, can last before the system corrects itself on its own.

Nothing about that rollout failed. Both deployments finished, both health checks passed, and kube-proxy did exactly what a ClusterIP Service is supposed to do. The problem was a forty-second window where "correct" and "evenly distributed" briefly meant two different things, and gRPC's connection reuse turned that window into six hours.

How Kafka

Darshan Turakhia — Sat, 26 Sep 2026 10:56:53 +0000

03:14 UTC. Security's on-call channel gets a message from the fraud-scanning job: a customer API key flagged and revoked four days earlier is still authorizing requests, all of them from ap-south-1. The revoke ticket has been closed as resolved since Monday.

the setup

authz-gateway sits in front of every API route and checks incoming keys against a local cache before anything touches the database. It used to call Postgres on every request; that was a flat 40ms added to p99 regardless of what the request actually needed. Six months earlier, someone replaced it with a Kafka Streams state store hydrated from a compacted topic, api-key-status, keyed by api_key_id. p99 for the auth check dropped to 4ms. An active key is a normal record. A revoked key is a tombstone, a record with the key present and the value set to null, which is how compacted topics represent deletion.



gateway/authz.js

function isKeyValid(apiKeyId) {
const record = keyStatusStore.get(apiKeyId); // local RocksDB state store
return record !== undefined && record.status === 'active';
}

The consumer group backing that store runs with group.instance.id set, static membership, turned on months earlier after a rolling deploy triggered a full rebalance and briefly spiked auth latency across every pod, not just the ones restarting. Static membership means a pod that drops out during a deploy keeps its partition assignment reserved instead of forcing an immediate reassignment. It was a good fix for that problem. It also meant nobody was watching for what happens when a pod goes quiet for reasons that have nothing to do with a deploy.

the scramble

First move: check the source of truth. Postgres's api_keys table shows the key correctly marked revoked_at four days ago. Whatever's wrong isn't the revoke itself, it happened, it's recorded, it's just not everywhere it needs to be.

Second theory: the tombstone never made it onto the topic. A console consumer against api-key-status from the revoke's approximate timestamp finds it immediately, a null-value record for that exact key, committed and replicated. The event exists. The dead end didn't cost much, five minutes to rule out, but it mattered: it meant the problem was downstream of Kafka, not upstream of it.

Third theory, on-call's: a stale edge cache. This service doesn't have one on the authz path, ruled out by reading its own deployment topology rather than by testing anything.

the hunt

The fraud job's flagged requests all traced to three pods, all in ap-south-1, all part of the same consumer group. kafka-consumer-groups.sh --describe against that group showed those three partitions current, no lag, caught up to the log's high-water mark. Not stuck. Not behind. Just wrong.



kafka-consumer-groups.sh --describe --group authz-gateway

PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
4 88213 88213 0 authz-gateway-7f2a...
7 61940 61940 0 authz-gateway-7f2a...
9 73301 73301 0 authz-gateway-7f2a...

Caught up with a wrong answer is a different bug than stuck behind. The next thing to check wasn't the topic, it was those three pods' history. Kubernetes events showed the answer: an ap-south-1-scoped NetworkPolicy change went out five days earlier as part of a node pool migration, and a rule ordering mistake in it blocked egress from those three pods to the Kafka broker subnet for roughly 30 hours before someone caught it in an unrelated ticket and reverted it. Nothing restarted those pods. Their liveness probe was an HTTP ping against the gateway's own port, which had nothing to do with whether it could reach Kafka, so it stayed green the entire time.

With no broker access, the consumer's background heartbeat thread stopped reaching the group coordinator. Kafka correctly detected that after session.timeout.ms and reassigned those three partitions to healthy members elsewhere, who kept the cache correct for everyone else. The revoke tombstone was produced and consumed by those healthy members within seconds, exactly as designed. It's what happened to the three isolated pods, once the network policy reverted and they reconnected, that broke.

the find

Kafka Streams persists its state store to local disk and checkpoints the offset it last processed. On reconnect, if that checkpoint is still within the log, it doesn't replay from scratch, it does a delta restore: read forward from the checkpointed offset, apply whatever's new. That's the whole point of a local store, cheap recovery. The three pods' checkpoints were roughly 6 hours stale when the network came back, right around when the revoke had been written.

The topic's delete.retention.ms was left at Kafka's default, 86400000ms, 24 hours. That setting exists specifically so a lagging consumer gets a grace window to see a tombstone before the log cleaner physically removes it. The three pods were unreachable for roughly 30 hours, past that window. By the time they reconnected and asked to resume from their checkpoint, the tombstone that checkpoint needed to see had already been compacted away. The delta restore read forward past where the tombstone used to be and found nothing for that key, so it never touched the in-memory record still sitting there from before the isolation: active.

Nothing in that path throws an error. The consumer isn't lagging, the topic isn't corrupt, the restore completes successfully. It just completes with a state store that is now permanently wrong for one key, with no future event on the topic ever going to fix it, because as far as the topic is concerned that key's history now starts after the revoke.

the fix

Immediate: force the three pods to drop their local state directory and rebuild from the earliest offset rather than delta-restoring from a checkpoint that could no longer be trusted.



recovery, run once per affected pod

kubectl exec authz-gateway-7f2a-xyz -- rm -rf /data/kafka-streams/authz-gateway
kubectl delete pod authz-gateway-7f2a-xyz

Then the two changes meant to keep this from being invisible again. First, a health check that actually reflects Kafka connectivity instead of just the local HTTP server:



gateway/health.js, after

let lastPollAt = Date.now();

consumer.on('poll', () => {
lastPollAt = Date.now();
});

app.get('/healthz', (req, res) => {
const staleMs = Date.now() - lastPollAt;
if (staleMs > 60_000) {
return res.status(503).json({ error: kafka poll stale for \${staleMs}ms\ });
}
res.status(200).json({ ok: true });
});

Second, delete.retention.ms on api-key-status went from 24 hours to 7 days, trading disk for a grace window wide enough to cover a multi-day network isolation, not just a routine restart. It doesn't fix the underlying risk, a consumer down longer than the retention window can still miss a tombstone, but it moves the failure mode from "a botched network policy" to "something has been broken for a week and nobody noticed," which is a bar this team was willing to accept.

the aftermath

96 hrs Time between the revoke and the fraud job catching it in ap-south-1

3 Requests authorized with the revoked key before detection

24h → 7d delete.retention.ms on api-key-status

2 Other Kafka-backed services found with the same HTTP-only healthcheck

Static group membership solves rebalance storms during deploys, but it also means a consumer that's gone quiet for an unrelated reason keeps its assignment and its local state without raising anything from outside. It needs its own signal, not the assumption that a healthy pod implies a healthy consumer.
delete.retention.ms is a grace window, not a guarantee. It's sized against how long a consumer might lag, not against how long one might be completely unreachable, and those are different failure modes with very different durations.
A local state store that restores by delta from a checkpoint is fast exactly because it trusts the checkpoint. That trust is only as good as the log's willingness to still contain everything between the checkpoint and now, and compaction doesn't make that promise indefinitely.
A healthcheck that only proves a process is running, not that it's connected to the one dependency that determines whether its answers are correct, will report green through the exact failure it exists to catch.

Kafka never logged an error either. It deleted exactly what its retention setting told it to delete, on schedule, while the only three pods that needed to see it first were on the wrong side of a firewall rule.

How Skipping a Cache Reset for Speed Let Apollo Client Show the Wrong Tenant

Darshan Turakhia — Wed, 23 Sep 2026 10:55:43 +0000

09:41 PST. A support lead pastes a screenshot into the incident channel: her own screen, mid quick-switch into Acme Corp's account, showing three ticket subjects that mention Globex Inc, a different customer entirely. The message underneath is two words: "is this normal."

the setup

The internal tool support agents used to manage tickets across customer accounts was a multi-tenant React app on top of a GraphQL API, Apollo Client for data fetching. Tenant scoping happened entirely server-side: every request carried a JWT with a tenantId claim, and every resolver filtered its database queries by that claim. The GraphQL queries themselves never mentioned a tenant. A ticket list looked like this, the same query text for every account an agent ever loaded:



TicketsOverview.graphql

query TicketsOverview {
tickets {
id
subject
status
customer
}
}

That was a deliberate simplification. Plumbing a tenant argument through every query in the app would have meant threading it through dozens of components for no functional gain, since the server already refused to return data outside the caller's token. Nobody on the frontend team had reason to think of tenant identity as something the client needed to track for itself.

Six weeks earlier, product shipped Quick Switch: a dropdown that let an agent jump from one customer account to another without a full sign-out and page reload. The old flow called client.resetStore() on every account change, which cleared the Apollo cache and refetched every active query, a UX that took 300 to 500ms and showed a loading spinner over the whole page. Quick Switch skipped that on purpose, to make account switching feel instant:



useQuickSwitch.ts, before

async function switchTenant(tenantId: string) {
const { accessToken } = await api.post('/auth/switch-tenant', { tenantId });
setAccessToken(accessToken);
// resetStore() intentionally skipped here — it added a visible
// loading flash and product wanted the switch to feel instant.
}

Every query in the app used Apollo's default cache-first fetch policy. That was fine for a single-tenant session: read from cache if present, otherwise hit the network. Nobody had reasoned through what cache-first does the instant the same query, with the same variables, needs to mean something different because the identity behind it changed.

the scramble

First theory: a session bug on the auth service, a Redis-backed token cache serving a stale tenantId claim to the wrong pod. The on-call engineer pulled the auth service's logs for the support lead's account switch. The issued JWT was correct, Acme Corp's tenant ID, timestamped to the millisecond of the click. Ruled out within ten minutes.

Second theory: the GraphQL gateway was routing to a stale replica that hadn't caught up on a recent write, some kind of read-after-write lag. Someone checked the actual network response in the support lead's browser recording, the one piece of evidence that made this theory look promising at first. The GraphQL response body for TicketsOverview, once it arrived, contained exactly Acme Corp's tickets. Correct tenant, correct data, right there in the payload.

That was the detail that stalled the investigation for the better part of an hour: every request and every response, read individually, was correct. The bug wasn't in anything that crossed the network.

the hunt

The break came from reproducing it deliberately instead of waiting for another report. Someone throttled their local network to Slow 3G in dev tools, specifically to stretch out whatever window the bug lived in, then opened Apollo Client Devtools and watched the TicketsOverview cache entry while triggering Quick Switch.

The component re-rendered twice. The first render, milliseconds after the switch, showed the previous tenant's tickets, read straight from cache. The second render, 800ms to 1.5 seconds later depending on network conditions, showed the correct tenant's tickets, once the network response landed and overwrote the cache entry.

Apollo Client's default cache key is the query name plus its serialized variables. TicketsOverview had no variables. Every tenant, every session, every agent, all produced the identical cache key: TicketsOverview:{}. The cache had no idea two different customers' data had ever passed through that key, because from the cache's perspective, nothing about the query had changed. Tenant identity lived exclusively in an HTTP header the cache never looked at.



what cache-first actually did on switch

Agent clicks "Switch to Acme Corp"
New JWT (tenantId=acme) stored, old JWT (tenantId=globex) discarded
TicketsOverview component re-renders
Apollo checks cache for key TicketsOverview:{} -> HIT (Globex's data, still cached)
cache-first policy: return the hit immediately, refetch in background
UI paints Globex's tickets under Acme Corp's header, for 0.8-1.5s
Network response for Acme's data arrives, overwrites cache, UI re-renders correctly

the find

Root cause: the Apollo Client cache had no concept of tenant identity, because tenant scoping had only ever been designed as a server-side concern. That was a correct design for the network layer, the server never returned the wrong tenant's data. It was an incomplete design for the client, because cache-first answers queries from a cache that has no way to know the caller's identity changed. Quick Switch removed the one step, resetStore(), that had been silently compensating for that gap by wiping the cache clean on every identity change. Once that step was gone, the gap was directly exposed to whoever happened to switch accounts fastest.

the fix

The immediate fix restored a cache clear on switch, but a narrower one than the old full resetStore(), which also triggers a refetch of every active query on the page, most of which weren't tenant-scoped UI at all:



useQuickSwitch.ts, after

async function switchTenant(tenantId: string) {
const { accessToken } = await api.post('/auth/switch-tenant', { tenantId });
await client.clearStore(); // empties the cache, no refetch of inactive queries
setAccessToken(accessToken);
// active queries now re-render against an empty cache and fetch fresh,
// instead of painting stale data first
}

clearStore() empties the cache without immediately refetching everything the way resetStore() does, so active components see a brief loading state rather than a full-page spinner, then fetch clean. That closed the immediate hole: an agent now sees a loading skeleton for a beat, never another customer's data.

The structural fix addressed the actual gap, that the cache had no tenant awareness at all, so the next feature that skips a manual clear doesn't reopen the same hole. Every tenant-scoped query now carries an explicit tenantId variable sourced from a reactive variable tied to the active session, and a type policy scopes the cache key by it:



cache.ts

export const activeTenantVar = makeVar(null);

export const cache = new InMemoryCache({
typePolicies: {
Query: {
fields: {
tickets: {
keyArgs: ['tenantId'],
},
},
},
},
});



TicketsOverview.graphql, after

query TicketsOverview($tenantId: ID!) {
tickets(tenantId: $tenantId) {
id
subject
status
customer
}
}

With tenantId in both the query variables and the type policy's keyArgs, Acme Corp's tickets and Globex's tickets now live under distinct cache keys, the same way two different pages of a paginated list already did. Even if a future feature forgets to clear the store on an identity change, the cache itself can no longer confuse one tenant's entry for another's; the identity is part of the key, not an invisible side channel.

the aftermath

1,900 Quick Switch actions recorded in the 30 days before the fix

46 Switches, across 12 agents, where session telemetry showed the stale render held for over 500ms

6 weeks Time between Quick Switch shipping and the first reported sighting

0 Stale cross-tenant renders recorded since the tenantId-scoped cache shipped

Because this was an internal tool, not customer-facing, no external data was exposed. It still landed as a corrective action under the company's SOC 2 tenant-isolation control, since that commitment covers who can see whose data regardless of which side of the product a screen sits on. A runtime assertion now runs on every tenant-scoped cache read in production: an Apollo Link compares the tenant embedded in a cached response against the tenant on the current session's JWT, and logs an alert if they ever disagree again. A synthetic test performs a scripted Quick Switch on a throttled connection in CI and fails the build if any tenant-scoped field renders data tagged with a different tenant ID than the active session, even for a single frame.

A cache-first policy is only as safe as the assumption that the same query and variables always mean the same data. Tenant identity, or any other value carried outside the query itself, breaks that assumption the moment it changes without the cache knowing.
Removing a slow safety step to make something feel instant is a real product tradeoff, not automatically a mistake, but it needs to be made with full knowledge of what the slow step was actually doing. Here it was quietly acting as the only thing scoping the cache by identity.
A bug built entirely out of individually correct network requests and responses won't show up by inspecting the network tab alone. It has to be watched as a render sequence over time, which is why throttling the connection to stretch the window mattered more than any single log line.
If a value determines which data a query is allowed to return, it belongs in the cache key, not only in a header. Otherwise the client is trusting the server's authorization to also double as the client's own cache boundary, and those are two different systems that happen to usually agree.

The server had been doing its job the entire six weeks, never once returning a ticket to a tenant that didn't own it. The leak was one layer up, in a cache that had no way of knowing two different customers had ever asked it the exact same question.

How a Retry Loop

Darshan Turakhia — Sat, 19 Sep 2026 10:55:58 +0000

03:14 UTC. Site search starts returning 503s. Not slow, not degraded — down. The on-call engineer's first Kibana query for search-service error rates times out too, which is its own bad sign, because Kibana is querying the same cluster that's supposedly the problem.

the setup

The cluster in question is a six-node self-managed Elasticsearch 8.x deployment handling two very different jobs. During the day it serves product and content search for the site, low latency, high query volume, nothing exotic. Overnight it also feeds a batch export: a Node job called order-events-exporter that scrolls the order_events index, about 600 million documents, and streams every doc into Snowflake for the analytics team. It's run at 03:00 UTC for two years without incident.

The export uses the Scroll API, which works by taking a point-in-time snapshot of the index's segments the moment the first request runs, then letting you page through that snapshot with a scroll_id until you're done. The snapshot is what makes it consistent, but keeping it alive costs the cluster something: every open scroll context pins the segments it's reading in place, which blocks merges and keeps otherwise-deletable segments resident in memory until the context is closed or its keep-alive expires.

the scramble

The paged engineer's first theory was a hot shard from a relevance-ranking deploy that had shipped the previous afternoon. It touched the search query builder, so it was the obvious suspect. Rolling it back didn't help; the 503s kept coming at the same rate.

Second theory: traffic spike. The dashboard showed request volume roughly flat against the same time yesterday, which ruled that out inside a few minutes, but the CPU graph in the same panel was doing something odd, climbing in a slow, steady ramp rather than the spiky pattern a traffic surge produces. Steady ramps usually mean something is accumulating, not spiking.

Third theory, and the one that actually got investigated properly: a JVM heap problem. Old-gen usage on the two data nodes handling the order_events shards was sitting above 90%, high enough to explain GC pauses long enough to look like request timeouts. But "the heap is full" isn't a root cause, it's a symptom, and nobody could yet say full of what.

the hunt

The cluster logs had already said what was wrong, twenty minutes before the page fired. It just hadn't been paired with the right alert:



elasticsearch.log, 02:54:11 UTC

[order-events-node-2] TooManyScrollContextsException: Trying to create too many scroll
contexts. Must be less than or equal to: [500]. This limit can be set by changing the
[search.max_open_scroll_context] setting.

That exception is real and specific: Elasticsearch caps concurrent open scroll contexts at 500 by default, cluster-wide, precisely because each one is a standing memory liability. Once the cap is hit, every new scroll request gets rejected, which explained why the export job had been failing since shortly after 02:54, but not yet why site search itself was down; a rejected scroll creation shouldn't touch ordinary, non-scroll search queries at all.

GET _nodes/stats/indices/search gave the number that connected the two:



GET \_nodes/stats/indices/search

"order-events-node-2": {
"search": {
"open_contexts": 1847,
"scroll_current": 1847,
"scroll_total": 2214
}
}

1,847 open contexts, well past the 500 the cluster is supposed to allow. That number only makes sense if contexts were being created faster than they were ever being closed or expired, and closed faster than they were also being renewed past their keep-alive, which pointed straight at the export job's retry logic:



order-events-exporter.js, before

async function exportBatch() {
for (let attempt = 0; attempt < 5; attempt++) {
try {
let res = await client.search({
index: 'order_events',
scroll: '10m',
size: 5000,
body: { query: { match_all: {} } },
});
return await drainScroll(res);
} catch (err) {
log.warn(export attempt \${attempt} failed, retrying\, err);
// falls through to the next loop iteration and calls
// client.search() again from scratch
}
}
throw new Error('export failed after 5 attempts');
}

On a client-side timeout, the catch block didn't resume the existing scroll_id, because it never had one to resume, the request had already timed out client-side before the response carrying it arrived. It just looped and called client.search() again, which creates a brand-new scroll context server-side, every single time. The old context the timed-out request had actually created was never told to close.

Normally this would be forgiving. A scroll context's default keep-alive is short, and an abandoned one expires and gets cleaned up on its own. But eight months earlier, a different slowness ticket had bumped this job's keep-alive from 1m to 10m "to give the export more breathing room," and three weeks before this incident, a mapping change had added a large nested line_items field to order_events to support a new analytics report. Scroll pages that used to return in roughly 80ms were now taking 600-700ms under load, comfortably past the export client's 5-second per-page timeout during the cluster's overnight compaction window. The job was retrying constantly, each retry left a fresh 10-minute-lived context behind, and contexts were piling up roughly ten times faster than they were expiring.

the find

Root cause: the export job's retry loop restarted the scroll from scratch on every timeout instead of resuming or explicitly closing the abandoned context, and a keep-alive that had been widened months earlier for an unrelated fix let each orphaned context live ten times longer than the default. Combined with a recent mapping change that slowed scroll pages enough to trigger the retry loop constantly, contexts accumulated past the 500-context cap within about twelve minutes of the job starting. Past that point, the pinned segments those contexts held open pushed old-gen heap usage over the cluster's parent circuit breaker threshold, which trips for all request types, not just scroll, once triggered:



elasticsearch.log, 03:13 UTC — one minute before the page

[order-events-node-2] [parent] Data too large, data for [] would be
[16.2gb/95%], which is larger than the limit of [15.9gb/94%], real usage: [16.1gb],
new bytes reserved: [112kb]: circuit_breaking_exception

That's the line that took down site search. The parent circuit breaker doesn't distinguish between a batch export's scroll request and a shopper's product query, once it trips it rejects both.

the fix

Immediate mitigation was a manual context flush, which closes every open scroll on the cluster and gives the circuit breaker room to reset:



manual recovery during the incident

curl -X DELETE "localhost:9200/_search/scroll/_all"

Heap usage dropped from 95% to 61% within about ninety seconds and search traffic recovered on its own. The durable fix had two parts. First, the retry logic now resumes by scroll_id and always cleans up on final failure instead of leaking:



order-events-exporter.js, after

async function exportBatch() {
let scrollId;
try {
let res = await client.search({
index: 'order_events',
scroll: '1m',
size: 5000,
body: { query: { match_all: {} } },
});
scrollId = res._scroll_id;
return await drainScroll(res, scrollId);
} finally {
if (scrollId) {
await client.clearScroll({ scroll_id: scrollId }).catch(() => {});
}
}
}

Second, the export was migrated off the Scroll API entirely, onto search_after with a point-in-time (PIT) ID, which is Elasticsearch's own recommended replacement for scroll-based deep pagination and defaults to a much shorter, non-negotiable keep-alive per request rather than one long-lived session an app can forget to close. The keep-alive that had been widened months earlier no longer exists as a setting to accidentally leave too generous.

the aftermath

38 min Site search returning 503s cluster-wide

1,847 Peak open scroll contexts against a default cap of 500

12 min Time from job start to the context cap being exceeded

0 Incidents since migrating off Scroll to search_after + PIT

A new Datadog metric now tracks open_contexts per node with an alert at 300, 60% of the default cap, so the next accumulation gets caught while it's still just a batch job failing, not the whole cluster.

A retry loop that doesn't know how to resume isn't retrying, it's restarting, and restarting a stateful server-side operation without cleaning up the state it already created is how you turn a transient timeout into a resource leak.
A keep-alive bump made for one incident months ago can quietly change the blast radius of an unrelated bug later. Nobody revisiting the retry logic knew the keep-alive had been widened, or that it mattered.
Elasticsearch's circuit breaker protects the whole cluster by design, which is exactly why an isolated batch job's misbehavior became a site-wide outage instead of staying contained to itself.
TooManyScrollContextsException was in the logs twenty minutes before the page fired. The gap wasn't missing information, it was a missing alert on a log line nobody had thought to wire up, because nothing had ever hit that limit before.

The Scroll API had been the right tool for this export for two years. It only stopped being the right tool the moment something downstream, a slower mapping, a wider keep-alive, a retry loop that didn't know its own state, started leaning on the one property Scroll assumes you'll respect: that whoever opens a context is the one responsible for closing it.

B-Tree vs LSM-Tree: the storage-engine tradeoff behind every database you use

Darshan Turakhia — Tue, 15 Sep 2026 17:32:22 +0000

Every relational database you've ever used made a bet at the storage-engine level, before you wrote a single query: reads matter more than writes, or writes matter more than reads. That bet shows up as a choice between two data structures, a B-tree or an LSM-tree. Most engineers never think about it. It's still deciding how their database behaves under load.

The actual tradeoff

A B-tree updates data in place. It walks down to the exact leaf page a key belongs on and rewrites it right there, so the tree stays fully sorted at all times. A read never does more than follow one path from root to leaf. There's no ambiguity about where a key lives, because there's only ever one copy of it.

An LSM-tree (log-structured merge-tree) refuses to do that. It never touches old data on a write. Instead it buffers writes in memory, flushes them to new immutable files on disk, and merges those files together later in the background. A write is a sequential append, not a random rewrite, which is why LSM-trees dominate anywhere ingest volume is the bottleneck: Cassandra, RocksDB, LevelDB, HBase. B-trees stay the default for read-heavy, point-lookup workloads: Postgres, MySQL's InnoDB, basically every traditional RDBMS.

What's worth understanding is what each side gives up, not which one "wins."

B-trees pay at write time. Once your dataset outgrows memory, a page update that used to be a cheap in-memory write becomes a real disk seek. Under heavy write load the tree also spends real time rebalancing, splitting pages as they fill so every leaf stays at the same depth.

LSM-trees pay at read time instead. A single key can exist simultaneously in the in-memory memtable, the most recently flushed file, and several older files. The newest write wins, but a lookup has to check each location until it finds a match. Bloom filters make this bearable in practice (a probabilistic "this file definitely doesn't have your key" check that skips most files without opening them), but a worst-case LSM-tree lookup is still checking more places than a B-tree ever has to.

There's also a cost LSM-trees defer rather than eliminate: compaction, the background process merging small files into larger ones and dropping superseded key versions. This is the part people underestimate. It's not free maintenance sitting quietly in the background, it's competing for the same disk I/O and CPU your live traffic needs. A compaction backlog is the classic LSM-tree failure mode, and it's an ugly one: writes outpace merging, read latency climbs as lookups check more uncompacted files, and the whole thing spirals if nothing throttles incoming writes to let compaction catch up. If you've operated Cassandra under sustained write pressure, you've probably met this one at 3am.

Where this actually bites you

This decision almost never gets made consciously by application code. It gets made once, implicitly, by whichever database a team picked years ago. So the first time most engineers actually think about it is after a write-heavy workload starts struggling on a B-tree-backed system that was never built for it. Time-series data, event logging, anything ingesting far more than it reads back in the same window: that's LSM-tree territory. Building it on Postgres because "that's what we use" is how you end up fighting the storage engine instead of your actual problem.

Some engines hedge instead of picking a side outright. MySQL's MyRocks storage engine swaps InnoDB's B-tree for an LSM-tree underneath the same relational interface, specifically for workloads write-bound enough to justify the read-side cost.

I wrote up the fuller comparison, including read and write amplification and where each structure shows up in real systems, here, if you want the longer version of this post.

Seeing it instead of reading about it

The part that's genuinely hard to build intuition for from a blog post is why a B-tree stays shallow even at billions of rows. So I built an interactive B-tree visualizer. Insert keys and watch a node split once it fills, the middle key pushing up into the parent every time. Search afterward and it highlights exactly which nodes the lookup touches. Entirely client-side, no backend. Took me about four minutes of playing with it to have the mechanism actually click in a way the text version never quite managed.

Next time someone asks why adding an index didn't help their write-heavy table, or why their time-series database "just feels different" from Postgres under load, this is the mechanism underneath both answers.