How setting up a local test bench uncovered a proxy tier that had never worked, a dashboard that couldn't show it, and a secret-sync failure mode that nobody was watching.
It started with a browser that wouldn't connect
I was setting up CrawlGym — our local bench for replaying crawls against recorded HTML, so we can test hypotheses and measure regressions without hammering real sites. Everything worked
except the browser tier: every escalation to LightPanda failed to connect.
So I went to check whether production had the same problem. It didn't. It had a worse one.
Production was doing something worse
The datacenter LightPanda sidecar was logging this on every single page:
$scope=frame $level=error $msg="navigate failed" err=CouldntResolveProxy
$scope=telemetry $level=warn $msg="postEvents" err=CouldntResolveProxy events=4 dropped=133
Not some pages. Every page — including LightPanda's own telemetry, which has nothing to do with our crawler. Whatever was broken was broken for every outbound request that container made.
The numbers for proxy_browser_request_total{proxy_type="datacenter"}:
| window | success | fail |
|---|---|---|
| 7 days | 0 | 67,083 |
| 90 days | 0 | 793,810 |
Zero successes across the entire metric retention window. This wasn't a regression. Tier 2's browser path had never worked, not once, since the metric existed.
Meanwhile tier 1 was healthy — 250,557 successful renders in the same 7 days. So it wasn't the browser, the image, or the network. It was specific to the datacenter tier.
One secret, two consumers, one of them broken
CURLE_COULDNT_RESOLVE_PROXY means libcurl couldn't resolve the proxy hostname — a failure before any socket opens. But the same pod's Go process was resolving that exact host fine and getting HTTP 200s through it. Same network namespace, same /etc/resolv.conf.
The two processes weren't being handed the same string.
PROXY_DATACENTER_URL in Vault ended with a trailing \n. One byte, and it broke exactly one of the two consumers, because they read the variable by completely different routes:
| consumer | path the value travels | result |
|---|---|---|
| Go crawler | env → envsubst into config.yaml text → YAML parse |
newline absorbed as end-of-scalar → clean URL, proxy works |
| LightPanda | k8s $(VAR) → argv, no YAML anywhere |
newline preserved → host proxy.example.net:8080\n → unresolvable |
That YAML absorption is the whole reason this survived for months. After substitution the config line becomes a clean scalar followed by a harmless comment line. The Go side parses fine, logs datacenter_http:true at startup, and pushes real traffic through the proxy. No error, no warning, nothing to find.
I reproduced it against the exact production image to confirm which input shape produced which error:
--http-proxy value |
error |
|---|---|
| valid host, with or without credentials | CouldntConnect |
| scheme-less with credentials | CouldntConnect |
| trailing newline | CouldntResolveProxy |
| nonexistent host | CouldntResolveProxy |
literal unexpanded $(VAR)
|
CouldntResolveProxy |
LightPanda resolves proxy hostnames perfectly well. It just can't resolve one with a line break glued to the end.
The cost: every page needing a render that tier 1 couldn't get escalated to datacenter, failed 100% of the time, and fell through to the web unlocker at 72× the cost weight. In one week,
112,228 forced escalations to the expensive tier — all downstream of a tier that structurally could not succeed.
Why the dashboard showed nothing
We have a Proxy Observability dashboard with a panel literally titled "Tier 2 (Datacenter) requests: success vs failure". It looked fine.
It queried proxy_request_total — the HTTP fast path. Only that. The browser path, proxy_browser_request_total, wasn't on the panel at all.
So a tier where the HTTP half worked and the browser half was 100% dead rendered as a healthy green line. The panel wasn't wrong, it was incomplete — and incomplete in exactly the dimension where the failure lived. Its description even claimed the tier "has recorded no direct successes historically", which was stale and pointed attention away from the real gap.
The fix was two queries instead of one, with legends that can't blend:
A: sum by (failure_category) (rate(proxy_request_total{...proxy_type="datacenter"}[$__rate_interval]))*3600
legend: HTTP · {{failure_category}}
B: sum by (outcome) (rate(proxy_browser_request_total{...proxy_type="datacenter"}[$__rate_interval]))*3600
legend: Browser · {{outcome}}
A tier with two independent failure paths needs two series. Aggregating them into one number is how you get a dashboard that is technically accurate and operationally useless.
Then dev refused to accept the fix
I patched the newline out of Vault in dev, restarted, and nothing changed. The pods still came up with the old value.
Two things were going on.
The boring one: environment variables from secretKeyRef are resolved at container start.
Updating a Secret never changes a running pod's environment — new pods are always required.
The second was not boring. The dev Secret wasn't updating at all. The sync had been dead for seven weeks:
error processing spec.data[2] (key: .../my-app-llm-api), err: Secret does not exist
Ready=False refreshTime=2026-07-15T06:48:31Z
The External Secrets Operator resolves every key in an ExternalSecret atomically. If one key fails, it writes nothing — so one missing key also froze PROXY_DATACENTER_URL, the database credentials, and everything else in the same object. Seven services, all stuck on a copy of their secrets from 15 July.
And here's the part that should worry you: nothing looked broken. ESO doesn't delete the Secret when it fails, it just stops updating it. Pods kept booting from the last-good copy.
One of them was still making successful API calls that morning, on a key synced seven weeks earlier. The only signal anywhere was a Ready=False on an object nobody looks at.
The fix was to create the one missing Vault path — present in prod, never created in dev — populated with the value already working, so restoring the sync was a no-op for that key rather than a credential rotation. All seven went Ready=False → True within two refresh cycles.
Restoring the sync then did something nice: it pushed the corrected proxy URL into the five secrets I'd never hand-patched, which were all still carrying the newline. And it validated the
blind Vault write independently — ESO fetched from Vault and produced byte-for-byte what I'd patched by hand.
The failure mode is bigger than us
Once I understood the shape of it, I audited every ExternalSecret in the namespace.
Dev: 15 of 200 failing. Ten were ours. The rest belonged to other teams:
- Three referenced a Vault path prefix that has never existed in that cluster —
secret/data/my-company.org/...instead of the.../ns/<namespace>/<app>layout everything else uses. TheirrefreshTimeisNone. They have never synced, ever. - One had the right path but the wrong property name inside it.
- Four wildcard TLS certificates were healthy until 2026-08-17, then the Vault entry vanished.
Prod: far better — 195 secrets, only 2 failing. But those two are instructive. They last synced on 2026-03-05. Six months ago. The consuming deployment is running 1/1 right now; its
pod started 2026-08-19, five months after the sync broke, and came up fine on the retained Secret.
So a production service is quietly running on a credential that can no longer be rotated.
Write a new key to Vault and it will never reach the pod. The eventual failure will look like vendor auth errors, with no obvious connection to a secret change made months earlier.
That's the real defect. Not any individual missing path — those are ordinary mistakes. The defect is that a broken secret sync is invisible by design: retention means the app keeps
working, so the failure is deferred to whenever someone next needs the value to actually change.
Which is usually a rotation, an incident, or a migration — the worst possible moments to discover your secrets have been frozen since March.
I raised this with the cloud team. They're now looking at surfacing sync failures during ArgoCD sync, so a Ready=False ExternalSecret fails loudly at deploy time instead of lurking. I handed
over the other teams' broken secrets along with it.
What I'd take from this
A single byte can be invisible on one path and fatal on another. The same value went through YAML on one side and argv on the other. Ask what transformations sit between a config value
and each of its consumers — they're rarely the same.
Verification methods can launder the bug. Shell $( ) strips trailing newlines. tr turns them into line breaks. Both "confirmed" the URL was well-formed. Use xxd.
A panel that shows one of two failure paths is worse than no panel, because it manufactures confidence. If a tier can fail two independent ways, plot two series.
Retention makes failures polite, and therefore dangerous. ESO keeping the last-good Secret is the right default for availability and the worst possible default for observability. If a system's failure mode is "everything keeps working, just frozen", nothing short of an explicit check will catch it.
Absence of a metric is not evidence of absence. Two signals nearly derailed the diagnosis:
ESO's syncedResourceVersion didn't change after a write that definitely landed, and a brand-new pod had no Prometheus series at all — not zero-valued ones — because of scrape discovery lag
and client_golang only creating a labelled series on first increment. When in doubt, read the pod's own logs.
The part that has nothing to do with newlines
Here's what actually bothers me about this bug.
We use LLMs to generate code now. That's fine — it's a tool, and it lets us implement things faster than we could before. But it only compresses one part of the job. Every other part of
engineering still exists.
For starters: we still have to make sure the damn thing we built works.
Testing methodology and observability matter more now, not less, precisely because we pay less attention to the implementation. If you didn't write the code line by line, you don't carry a
mental model of where it's fragile — so the only thing standing between you and a silent failure is a strict test that fails loudly and an alert that fires when reality diverges from what you
assumed.
This bug is what that gap looks like in production. A tier shipped that had never worked once. 793,810 failures over 90 days. A dashboard panel that couldn't show the failing half. Seven
services frozen on seven-week-old secrets, and the only evidence was a Ready=False field on an object nobody reads. Nothing here was hard to detect. It just wasn't being checked.
Impact assessment, testing, monitoring after deployment — none of that vanished with AI. You can use AI to help with all of it: have it write the test you'd have skipped, generate the alert
rule, review the diff for what you forgot to instrument. The failure here wasn't AI. It was skipping the cadence.
And I'll be honest about the uncomfortable part: because we aren't writing the code, we care less. When you can generate a working implementation in a minute, it becomes very easy to stop
at "it runs" and never do the complete thing. That pull is real, and I feel it too.
Doing the complete thing anyway is the job. That's what being a responsible engineer means, and it's the part no tool is going to do for you unless you ask it to.
Top comments (0)