DEV Community: Jerome

Claude Code Recommended: Give Up

Jerome — Wed, 19 Aug 2026 07:11:04 +0000

Nine hours into a live networking bug on my k3s cluster, Claude Code asked me a question with three options. The first was labeled (Recommended). It was to give up.

The task behind it was ordinary by this point: a fourteen-step plan to put PR preview environments behind an Istio Ambient mesh, run task by task through superpowers' subagent-driven-development — one implementer subagent per task, one reviewer subagent to check its work, me approving the boundary. Thirteen tasks had gone this way already, each with its own small fight (an ArgoCD self-heal loop fighting istiod's webhook config, a sync retry limit exhausted, a waypoint proxy quietly asking for ten times the CPU its namespace quota allowed) and each one resolved cleanly enough that I stopped expecting the fourteenth to be different. Task 14 was verification: open a real pull request, send it a request carrying x-pr-lane: 1, confirm it landed on the preview pod instead of the baseline one.

It never did. Every request, header or no header, went straight to baseline. No error, on either side of that route.

Ruling out the obvious

What followed was the kind of debugging that doesn't look like debugging from outside — it looks like a long list of things that turned out not to be the problem. istio-cni's iptables REDIRECT rule inside the pod's network namespace: confirmed catching outbound packets, nsenter and all. ztunnel's xDS config: correct, the waypoint binding was right there in the config dump at the Service level. The waypoint itself: reachable, routes programmed, ready — and in three hours of its own logs, holding zero connections. A namespace-level versus Service-level use-waypoint label swap: no change. A GitHub issue on Istio's tracker with the same symptom: different root cause, cross-namespace enrollment, didn't apply here. A host-side tcpdump that caught traffic arriving on ztunnel's own tunnel port looked, for a while, like proof the fault lived inside ztunnel's routing logic rather than upstream of it — a read that turned out to be wrong, but not obviously wrong, not at 2 a.m.

Somewhere in the middle of that list, at 03:08, Claude Code tried a fix, watched it fail, and said so plainly: the new packet capture didn't just fail to confirm the diagnosis, it contradicted it. The situation was worse than it had looked an hour earlier. It asked whether to keep going. I didn't answer for forty-four minutes — I was asleep — and when I did, I just wrote: continue.

The honest report

Fifty minutes after that, at 04:01, it stopped asking and started reporting. The message was direct about what had happened: it had gone through, in its words, nearly every known and documented possible cause, systematically, and ruled each one out. Time to say so honestly. Then the question, three options:

1. Wrap up here, mark waypoint L7 routing as a known limitation (Recommended)
2. File an issue with Istio upstream — unlikely to get an answer quickly
3. Keep digging — success rate is already falling, may just be unsolvable

I asked what, specifically, was blocking it. It answered plainly: the feature's actual selling point — routing a live PR preview by request header through the mesh — wasn't something it could deliver. Did I want it to close this out as a known limitation, or look at the other options first?

Nothing about that exchange was performative. Every hypothesis it had checked really had checked out negative or inconclusive; the elimination had been careful, not lazy. What made the "Recommended" default worth noticing wasn't that it was reached in bad faith — it's that a careful, honest elimination of every path you can currently see still isn't the same thing as having eliminated every path there is. Nine hours of good work had converged on the wrong conclusion anyway, for the ordinary reason that whatever had anchored the investigation at hour one was still anchoring it at hour nine.

A subagent that owed the last one nothing

I didn't argue the evidence, and I didn't tell it to keep grinding on the same thread either. I told it to open an Opus subagent and have that subagent investigate independently.

The instruction I gave it mattered as much as the decision to give it: don't inherit my read of this, and don't inherit yours either — if you think a verification step upstream was done wrong, don't take it on faith, redo it. The dispatch handed the new subagent the entire trail — every hypothesis raised, tested, and marked ruled-out or inconclusive over the previous nine hours — with explicit permission to distrust every one of them, plus careful, reversible access to the live cluster to go check for itself.

Twenty-eight minutes later it came back with an answer, and the answer was uncomfortable in a specific way: one of the hypotheses from hour one had been right all along. It had just never actually been tested.

The datapath nobody restarted

The candidate was Cilium's own socket-level load balancing. With kube-proxy replacement on, Cilium's eBPF datapath resolves a Service's ClusterIP to a specific pod IP before istio-cni's netns-scoped REDIRECT rule gets a chance to preserve the original ClusterIP as the connection's SO_ORIGINAL_DST. By the time ztunnel inspects the redirected connection, it's already looking at a pod IP, not the Service VIP — and a waypoint binding is keyed to the Service, so it can never apply. Cilium's own chart documents the fix for exactly this case: socketLB.hostNamespaceOnly: true, restricting that early resolution to the host network namespace and leaving pod-namespace traffic alone.

That value had been set, hours earlier, at hypothesis two of nine. helm upgrade had gone out, the cilium-config ConfigMap had updated, and the ConfigMap was what every verification step after that checked — because a ConfigMap is where Kubernetes config normally lives, and reading it back is normally proof enough. It wasn't, here. cilium-agent reads that config exactly once, at its own startup, and compiles it directly into the cgroup BPF program it runs — not something a controller loop re-reads on a timer. And the Helm chart puts no config-hash annotation on the DaemonSet's pod template, so a values-only change leaves that template byte-for-byte identical and triggers no rollout at all. The ConfigMap said true. The compiled datapath, still running the same process it booted with, was still full — silently declining every waypoint binding it was handed, with nothing anywhere logging so much as a warning.

The subagent's proof wasn't a guess dressed up as confidence — it was three independent lines converging. cilium-dbg status --verbose read Socket LB Coverage: Full, not the Hostns-only the ConfigMap promised. The DaemonSet's own history read the wrong way round: the Helm revision that introduced the setting landed nineteen hours after the running cilium-agent pod's own start time, with a restart count still at zero and a controller-revision-hash still pointing at a template from a week earlier. And a live trace, cilium-dbg monitor -t trace-sock, caught the DNAT happening — Service VIP rewritten to pod IP, inside the calling pod's own cgroup, before ztunnel had any chance to see the original destination.

The fix, once that was visible, was one line:

kubectl -n kube-system rollout restart daemonset/cilium

Socket LB Coverage read Hostns-only afterward. The header test that had failed all night routed correctly on the first try. I wrote back one word — do it — and a few minutes later: it finally worked.

Not stuck, anchored

What the first nine hours got wrong wasn't the hypothesis. Hypothesis two was correct from the start. What they got wrong was treating a ConfigMap read as proof that a change had taken effect, and then never going back to re-examine that specific piece of evidence once it had been filed away as "applied." Every hour after that built on the same foundation, and no amount of additional hours was going to notice a crack in a foundation nobody was looking at anymore. That's a different failure than "not smart enough" or "gave up too easily" — the elimination really was careful, right up until the point where it wasn't checking anything new.

An independent subagent fixed it not by being a better engineer, but by not knowing yet which piece of evidence was supposed to be settled. Told explicitly not to trust the read, it went and checked the one thing nobody had checked twice: whether the config change had reached the process actually running the datapath, or only the object Kubernetes stores it in. The gap between those two things is exactly where this kind of bug lives, and it's invisible from inside an investigation that's already decided where the line between "verified" and "still open" sits.

The smaller fix landed the same night, in the values file, as a comment for whoever changes this setting next: after touching anything under socketLB, restart the DaemonSet and check the live datapath, not the ConfigMap. A second one landed in the repo's own CLAUDE.md a couple of hours later, general enough to outlast this specific bug — a rule against testing changes directly against a live GitOps-managed resource before they're committed, because ArgoCD's self-heal will revert an uncommitted edit with nothing pointing at the cause. Both are the same lesson in different clothes: know exactly what you're actually checking, and don't let an early confirmation retire the question that led you to check it in the first place.

Synced, Healthy, Running, Wrong

Jerome — Tue, 18 Aug 2026 08:07:12 +0000

Two days after The Twenty-Hour Fuse, I had seven new alert rules on a four-core ARM box — PSI, swap consumption, D-state counts, a forecast on the memory trend. They were insurance against a repeat of an incident I'd already fixed. On August 17th they fired for something else entirely: the same rule, twice in one day, for two root causes with nothing in common.

What connects those two incidents isn't a mechanism. It's that neither of them made anything report an error. Kubernetes said Running. ArgoCD said Synced and Healthy. Helm exited zero. An admission policy checked for vulnerabilities and passed. Every layer in the stack answered the only question it knew how to ask, answered it correctly, and the composite answer was wrong.

The reader with no files open

The first alert: IO PSI full above 15 percent for two minutes. /proc/pressure/io confirmed it — full avg10 at 24.97, and avg300 at 21.99, which matters more, because it means this wasn't a spike I'd arrived after. It was a plateau I was standing on.

vmstat gave the direction in one line: bi around 90,000 blocks a second, bo essentially zero, wa over 20 percent of CPU. Reads. Disk at 38 percent, so nothing was full. A scan of /proc/*/io for two-second deltas returned only my own shell — the real consumer belonged to another user — and the same scan under sudo returned a single process well ahead of everything else: all-in-one-linux, PID 713618, reading about 93 MB/s.

Then the hunt turned over. I went looking for what it had open, and /proc/713618/fd held nothing but sockets. /proc/713618/maps showed one file mapping, its own executable. A process reading 93 MB/s off the block device had not one file open.

rchar:       18033040      # ~18 MB requested through read()
read_bytes:  63512895488   # ~63.5 GB actually fetched from the block device
write_bytes: 0

Those two counters measure different things. rchar is what the process asked for; read_bytes is what the block layer actually fetched on its behalf. For a process reading files they track each other within an order of magnitude. A gap of three thousand times means the reads were never its requests: they were its own pages being faulted back in after the kernel evicted them. The IO is billed to the process. The process did not initiate it.

That reframes the whole question, and it's the step where stopping early would have sent me somewhere useless. The top of the IO list wasn't a cause to investigate; it was a symptom to trace upstream. /proc/vmstat confirmed the scope immediately — workingset_refault_file at 31.4 million, pgscan_file at 118 million, pgsteal_file at 91.6 million. Not one process misbehaving. The whole host thrashing its page cache.

dmesg supplied the reason: three OOM kills in sixteen hours. A JVM, a trivy scan job, and at 05:39 a process named all-in-one-linu — and my suspect's etime was 07:25:53, which put its birth exactly at that kill. The heaviest reader on the box was the restarted corpse of the third victim, still thrashing in the same conditions that had killed it.

It was Jaeger, in a demo namespace, and its container limit was 128Mi. Its anon-rss at kill time was 126,032 kB. It had been living on the ceiling. The node explained why nothing had pushed back: memory requests at 41 percent of capacity, memory limits at 110 percent. The scheduler reads requests, so as far as it was concerned this node had room to spare, and it would happily keep placing pods until real usage caught up with the overcommit — which is precisely what had happened.

Through all three kills the pod's phase was Running. The kubelet restarts the container in place; the pod object never leaves the happy path. kubectl get pods showed a restart count of 1.

Fifteen checks, and the shape of the hole

That box runs an inspector — fifteen shell checks, twice a day at 09:00 and 21:00, each one either fixing something safe automatically or raising an alert for a human. Afterwards I went through the list to find which one should have caught this, and the answer was none of them, for reasons that were individually reasonable.

k3s-evicted-pods.sh queries with --field-selector=status.phase=Failed. An OOM-killed container doesn't move its pod to Failed, so this incident was never inside the query. docker-restart-storms.sh only looks at compose containers on the host, never k3s pods, and its threshold for a storm is ten restarts; Jaeger had one. The Grafana rules could see the box stalling on IO — they're what woke me up — but aggregate pressure can't tell you which container the kernel shot.

The sixteenth check reads containerStatuses[].lastState.terminated.reason == "OOMKilled" with a 24-hour lookback, wide enough that a missed run still catches everything. It only alerts; whether a limit should be raised is a judgment call and not a cron job's business.

The detail I keep coming back to is that the gap was exactly the shape of the failure. lastState is where Kubernetes files this, faithfully, for anyone who asks. Nobody was asking.

Set in git, never set in the cluster

An hour after that fix went in, the same rule fired again.

By the time I looked, avg10 was back to zero and avg300 was down to 6.15 — the event was already over. Prometheus had the history, since node-exporter had been scraping all along: rate(node_pressure_io_stalled_seconds_total[2m])*100 over three hours showed five separate waves, the largest peaking at 77.9 percent. The opposite shape from the morning. Not a plateau, a pattern. Something recurring on a schedule I hadn't found yet.

journalctl lined every wave up with trivy scan pods starting. (A self-hosted service on the same box, one that writes to SQLite, had started logging database is locked around then too. It looked like a second lead for about a minute; it was downstream of the same IO.)

The live ConfigMap said OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT: 10. My values.yaml in git said scanJobsConcurrentLimit: 3. Checking the chart's own schema explained the difference: that key belongs nested under operator:, and mine sat at the top level, where Helm silently discarded it and used the chart default of 10.

It had never once been in effect, and nothing anywhere could have told me. The git diff read correctly. ArgoCD reported Synced, accurately — the rendered manifests did match git. The operator started clean. An unrecognized key in a Helm values file is not a warning or a non-zero exit; it's nothing at all.

Ten concurrent scan jobs share one filesystem cache directory guarded by a lock file that doesn't support concurrent access. Six hours of operator logs held 36 lock timeouts, several jobs exhausting their retries into BackoffLimitExceeded — each failure arriving after the job had pulled the image layers in full. Pure wasted IO, on repeat.

And that explains the schedule. The initial cluster-wide scan on the 16th ran from 14:01 to 18:10, four hours, stretched by exactly this contention. Report TTL is 24 hours. A four-hour smear of creation times becomes a four-hour smear of expiry times, so the rescan storm inherits its own shape from the bug that caused it, every day, at the same time. The TTL mechanism is sound; it exists to catch CVEs disclosed against images that haven't changed. The bug had turned a daily routine into a four-hour siege.

Then I made the same class of mistake a second time. skipResourceByLabels belongs under trivyOperator: — the chart splits operator-controller settings from scan-job behaviour into two top-level blocks — and I put it under operator:, having just learned that operator: was where the previous key belonged. Same silence, same result.

So the habit that came out of this is mechanical: before committing any values change, run helm template <chart> --repo <url> --version <ver> -f values.yaml | grep <key> and confirm the rendered output is non-empty and correct. Reading the YAML doesn't work. Reviewing the diff doesn't work. The failure mode has no symptoms at the layer where you're looking.

Set in the cluster, and still inert

One scan job kept failing regardless: cilium's, every 45 to 80 seconds.

Cilium's DaemonSet has six initContainers plus a main container, all on the same image digest. trivy-operator packs all seven into a single scan job pod as ordinary containers: — not Kubernetes initContainers:, which would run in sequence — so Kubernetes starts all seven at once and they race each other for the same cache lock. scanJobsConcurrentLimit bounds concurrency between jobs. It has nothing to say about seven processes inside one.

The chart has a setting for this, skipInitContainers: true. I set it, confirmed the value in the live ConfigMap, watched for about ten minutes, saw no new multi-container jobs, and called it fixed.

That was wrong, and the way it was wrong is the part worth keeping. The retry interval was 45 to 80 seconds and my observation window was ten minutes — long enough to look calm during a lull, short enough to miss that the cycle had never stopped. Twenty-five minutes later it was still going, unchanged.

Reading the operator's source for that version explains it. ScanJobBuilder.Get() does call GetContainerImagesFromPodSpec(spec, s.skipInitContainers), and the filter is applied correctly — but that filtered list is only used to build a JSON annotation on the Job object. The call that actually produces the pod spec, s.plugin.GetScanJobSpec(...), is never passed the flag, and independently recomputes the container list from the original resource. The setting is read correctly and applied to the wrong output. It changes what the operator writes down, and nothing about what it runs.

This is a third kind of silence, and the most uncomfortable one. The value was correct in git, correct in the rendered chart, and correct in the running ConfigMap — every verification I'd built after the last mistake would pass. Config verification ends at the ConfigMap. Whether the config does anything is a separate question, and only behaviour answers it.

The fix was to stop scanning that DaemonSet at all, which took two tries: the chart's podLabels renders to spec.template.metadata.labels, but the operator's skip test reads the workload object's own labels, and a DaemonSet has no ReplicaSet sitting in between — the resource is the DaemonSet. So the label went on by hand, metadata only, no rollout. The price is that cilium's main container isn't scanned either. That one's a knowing trade, not a gap.

Underneath all of it was a fourth failure that had been running since the day trivy-operator went in. Trivy's CLI assumes linux/amd64 unless told otherwise and refuses anything else, and five of the images on this cluster are built single-arch arm64 by this repo's own CI. Every scan of them had failed, every day, from the beginning. Nothing ever surfaced it, because a failed scan produces no report — and no report looks exactly like no news.

A gate that admits what it can't see

Which matters, because there's a Kyverno policy on this cluster that denies pods whose Trivy report shows CRITICAL vulnerabilities with a fix available. Its condition counts matching vulnerabilities and denies when the count is greater than zero.

Run that against a workload with no report at all. The lookup returns an empty list, the count is zero, > 0 is false, and the pod is admitted. No error, no warning, and — worse — no deny event, so there's nothing in the audit trail to notice later. The policy is strongest against workloads that scan successfully and completely absent for workloads that don't, which is the inverse of what you'd want.

At that point more than ten workloads had no report, for three unrelated reasons: the arm64 five, a handful of locally-built images that 401 against a registry they were never pushed to, and a few more I still can't account for.

So the sequence of the fix mattered more than any individual piece of it. First make the arm64 images actually scan — a trivy config file setting image.platform: linux/arm64, rendered into the scan container and passed via --config, which I verified end to end in a throwaway pod against one of the failing images before it went anywhere near the real config. Then narrow the gate to the two self-built apps whose reports I'd confirmed existed. Only then flip it from Audit to Enforce. An enforcing gate on top of a blind scanner is worse than no gate, because it reports a control you don't actually have.

Where the detectors came from

The changes that came out of two days: two memory limits raised, two Helm keys moved to the blocks they belonged in, one label applied by hand, one trivy platform setting, one DaemonSet excluded, one alert's hold widened from three minutes to ten to stop it firing on ordinary noise. None of that is interesting.

What's interesting is where the things that did the finding came from. The PSI rules that caught both incidents were written two days earlier, in the postmortem of a different incident with a different cause — they were aimed at VS Code sessions and caught a memory overcommit and a scanner instead. The Prometheus history that made the second incident legible existed because node-exporter had been quietly scraping the whole time. The sixteenth check exists because of the first of these two. The helm template habit exists because of the second. The list of workloads with no vulnerability report exists because I went looking for absences, which is not something any dashboard will ever show you — dashboards show what exists.

None of it was designed. Every piece is sediment from the incident before it.

And the reason it has to accumulate that way is that nothing here malfunctioned. Kubernetes was right that the pod was Running. ArgoCD was right that the cluster matched git. Helm was right that it had applied every key it recognized. The admission policy was right that it found no fixable CRITICAL CVEs. Four correct answers to four questions I hadn't asked. Detection is the separate thing that asks the question none of them are scoped to answer, and there's no version of a well-built system that hands it to you for free.

The Twenty-Hour Fuse

Jerome — Sat, 15 Aug 2026 14:15:54 +0000

Two nights of debugging, one incident left to write about. The night of August 14th, a Claude Code session hung for an hour inside the VS Code extension, and I traced it to a per-token streaming stall — the previous two pieces, Two Clocks, Neither Lying and Coalescing the Stream, cover that hunt and the middleware that fixed it. The morning after, the same four-core box nearly went down under a load average of 38.7, and it turned out to be the same incident, twenty hours downstream. Nothing exploded that morning. A fuse had been burning since the previous afternoon, and 11:09 was simply when it reached the powder.

Reading the machine before touching it

The load numbers spoke first: 11.08, 38.69, 23.59 — one-minute, five-minute, fifteen-minute. A crisis still climbing shows the opposite order, disaster in the one-minute figure first; mine were falling, so the peak had already passed while I was still reading. Then /proc/pressure — PSI, the kernel's stall accounting for CPU, memory, and IO, and the closest thing a Linux box has to a memory of the last five minutes: memory full at 10 percent (processes genuinely stopped, waiting on page reclaim) and IO some at 45 percent. Memory pressure had turned into an IO storm.

The connection to the headline number is D state. Processes parked in uninterruptible IO waits — runc and a node healthcheck were caught in the act — count into load average exactly as if they were running. Four cores at 38.7 wasn't thirty-eight cores of computation; it was modest real work plus a crowd of processes waiting in the kernel's lobby.

Elimination was fast because the box was eloquent. No OOM kills in the kernel log — the kernel hadn't started shooting anyone. docker stats: every container under 500 MB, single-digit CPU — not the compose stacks. Disk at 65 percent — not full. A process listing sorted by memory, then grouped: llama.cpp at 3.9 GB, seven Java services at 2.3 GB, k3s at 1.1 GB — steady state, all normal, none of it news. And fifty-nine vscode-server processes holding 7.3 GB, which is not steady-state anything.

One server, two windows, and the rule that fell out of the map

The decisive hour went into mapping the process tree, PPID by PPID, because "59 processes" is a number, not a structure. The structure: Remote-SSH runs one server per client version on the box, parented to init — detached from any SSH session, and therefore under no session's death sentence — and that one server hosts every window the client opens, one extension host per window.

A server started on August 9th was hosting two. One was the window I was typing in. The other was an old window from August 14th whose extension host had grown seventeen processes and 3.2 GB — containing the very Claude session that had hung the night before, plus four more claude processes born of retries between 2:04 and 2:11 a.m., every retry a new process while the old ones never exited. A second, separate server tree — thirteen processes, another device's window from 8:26 that morning — completed the census.

The operational rule fell straight out of the map: I could not kill a server, only windows. Kill the shared server and I'd be sawing the branch I sat on. The kill list was the old window's extension-host tree, the other device's whole server tree, and nothing else.

Twenty hours, then eleven minutes

As a timeline, the incident runs at two speeds. Accumulation: from 15:17 on the 14th, when the old window connected and the stalled session began its hour-long hang, to 8:26 the next morning. Retries stacked unexiting processes; detached servers sat immune; and six versions of the server binary piled up in ~/.vscode-server — each client upgrade downloads a new one — 3.8 GB of them. Memory crept past 85 percent slowly enough that nothing intervened: no swap to absorb the creep, and memory alerts set at 80 and 90 percent are spike detectors — for growth this gradual they arrive with hours of momentum already behind them.

Ignition: 10:59 to 11:09. A new connection, a reconnect, two fresh extension hosts spinning up their language servers — Pylance and friends, CPU- and IO-heavy — on a box whose page cache was already being squeezed to stay alive. Eleven minutes of simultaneous startup on a machine with no headroom: the kernel began reclaiming the pages everything was reading, IO seized, processes dropped into D state, and the load average registered the detonation. The five-minute peak of 38.7 covers exactly those eleven minutes.

Four layers, four failures

The useful postmortem question isn't what broke; it's what should have caught this and didn't. Four layers, each with a job, each failing for a specific reason.

The trigger layer is the stall itself, and it's the one layer that held — the middleware went in the night before. This incident arrived with its ignition source already removed; the fuse just had to finish burning.

The accumulation layer failed architecturally. Servers that outlive their sessions because nothing ever kills them; an old client version that doesn't start its servers with the auto-shutdown flag the new ones carry; and — the part that's mine — a replication engine made of my own reasonable responses. Every "reopen the window and retry" was the correct move as far as I knew, and every one cloned the problem: a fresh process next to the ones that never left.

The buffer layer failed by absence. Without swap, the kernel's relief valve under memory pressure is reclaiming the page cache — the exact thing every running process is trying to read — which is how a memory shortage converts into an IO storm. Swap is the difference between slow and seized.

The capacity layer failed by design. Steady state on this box is about 15 GB of its 23 — k3s, llama.cpp, ten compose stacks, seven JVMs — leaving thin elastic headroom for the one workload that actually bursts: interactive sessions. The incident priced that layout.

The surgery, and two humbling footnotes

The cleanup went carefully, in this order: a keep-list first — the entire process chain of the window I was typing in, the shared pty host, the agent host. Then a recursive walk collecting every descendant of each root marked for death, with each root's identity re-verified against its command line, because a PID is a label that gets reused, not a name. TERM first; KILL for the survivors four seconds later. And one rule I never skip: the kill list gets checked for intersection with my own process chain, because the checker is also a process on this box, and the one tree it must never reap is its own. Fifty-nine VS Code processes became nineteen; fourteen once the dust settled.

Then 4 GB of swap — fallocate, mkswap, swapon, fstab — after which available memory jumped from 2.8 GB to 8.2 GB, mostly because the kernel stopped strangling the page cache. Then the version pile: five of the six server directories deleted, 3.2 GB back.

Footnote one: before deleting those directories I verified nothing referenced them, by grepping ps output for the directory names. Every directory came back "in use" — because ps prints my own checking command's command line, names included. The check had matched its own author. The corrected version then carried its own bug, an invalid ps invocation that silently verified nothing, and the deletion went through unguarded; a full manual census afterward confirmed nothing innocent had died. The lesson generalizes: any check that greps a process list and then acts on the result must exclude the checker — the observer is always inside the observation.

Footnote two: at 11:28 the killed device reconnected by itself — the desktop client, still open, reaching out and re-downloading 600 MB of server binary because I'd just deleted it — and the old window re-spawned an extension host onto the shared server. Killed again; four quiet minutes after that. You cannot win this fight on the server alone. Server-side cleanup plus client windows actually closed, or the tree regrows.

	11:09	11:33
Load average (1 min)	11.1 (5-min peak 38.7)	0.68
Memory used / available	20 Gi / 2.8 G	15 Gi / 8.2 G
Swap	none	4 GB, in fstab
IO PSI (10 s / 5 min)	21% / 45%	0.01% / 2.1%
VS Code processes	59	14
Server versions on disk	6, ~3.8 GB	1, 570 MB

Checked again as I write this: swap present and quiet, one server directory, load under one.

What I actually fixed

The honest accounting is that the morning's cleanup treated the wreckage, not the disease. The actual fix had happened the night before, in a middleware file: with the per-token stream gone from the pipeline, sessions end when they end, processes exit, and the accumulation layer lost its fuel. What this incident added was the rest of the defense — swap as the shock absorber, and seven new alert rules, across four signals: PSI (io and memory pressure), the count of processes stuck in D state, swap presence and consumption, and a predict_linear forecast on the memory trend. All four read metrics node_exporter was already emitting, so none of it needed new infrastructure. The instinct I reached for first — alert on the vscode-server process count directly — would have been the more legible signal, but shipping it meant a host cron job, a textfile collector, and a node-exporter mount change for one app-specific number; it's shelved for now in favor of generic pressure signals that were already flowing. And a habit: close the window, don't reopen it.

Every alarm I had was built for the eleven minutes. The trend forecast is the one now aimed at the twenty hours before it.

Coalescing the Stream

Jerome — Sat, 15 Aug 2026 14:15:28 +0000

Claude Code, on my box, doesn't talk to a model provider directly. It goes through claude-code-router (ccr) — a self-hosted proxy that lets the same CLI point at the official Anthropic API or at third-party providers — and I watch the whole thing through the VS Code extension over Remote-SSH. The previous piece, Two Clocks, Neither Lying, worked out why that path once let an approval prompt sit for an hour: one streaming event per token, an extension that couldn't burn them down fast enough, and the CLI's own control messages queued in behind six thousand deltas. This piece is about the fix, which turned out to be two problems in sequence — writing a middleware that merges those events back into chunks, and then getting the middleware to actually run. The first took an evening. The second took five attempts.

The one place I control

The fault had three addresses. The provider's granularity — theirs, not mine to change. The extension — closed source, not mine to patch. And between them, the router: my container, my compose file, my rules. The official Anthropic API already streams in merged chunks, and the extension handles that fine — so the goal was never to make the extension faster. It was to make my stream look like the one the extension already digests.

Which sounds like a small thing: a transformer that re-chunks deltas on the way out. But the constraint that governs every other decision is nastier than "merge events." The middleware must merge without changing what the stream means. A merger that corrupts semantics is worse than the stall — the stall costs an hour, a lying stream costs the session.

Merge, without changing what the stream says

The rule, in one sentence: merge consecutive content_block_delta events only when they carry string payloads for the same block index and the same delta type — text into text, thinking into thinking, partial JSON into partial JSON — and flush the moment any other event appears.

Every clause exists because its absence corrupts something. Same index: one assistant message interleaves blocks — text, then a tool call, then more text — and block boundaries are how the client tells them apart; merge across an index change and you fuse two blocks into one that never existed. Any other event flushes: content_block_start, content_block_stop, message_delta, message_stop all carry protocol meaning, so the window empties and they pass through untouched, in order. The middleware deliberately understands nothing about the conversation — it only combines things that are provably interchangeable and defers to everything else. That ignorance is the safety property.

The window is the one knob: longer windows merge more and delay more. CCR_SSE_COALESCE_MS sets it globally, and zero is the kill switch that disables the middleware outright. It later grew per-type overrides — CCR_SSE_COALESCE_THINKING_MS, _TEXT_MS, _INPUT_JSON_MS — because one number can't serve two populations: thinking deltas can afford to wait, and text is the thing a human watches appear. A per-type value of zero or less doesn't mean "off for this type"; it means fall back to the global — a zero-width window has no timer to flush it and would strand data until the stream ended. Keep-alive pings are dropped by default (CCR_SSE_DROP_PINGS=0 restores them) — not for the bytes, but because a ping landing mid-thinking-run would flush the window exactly where the stream is densest; merging that continues across keep-alives is worth more than the pings.

Two transport truths that must not go stale

Merging bodies invalidates two things the transport layer believes. Content-length: the merged body is no longer the length anyone declared, and a client honoring the stale header will truncate or hang — SSE travels chunked, so the header is simply removed. Compression: the middleware asks for accept-encoding: identity on the way in, and if a response comes back compressed anyway, it bypasses itself — passes the stream through untouched rather than decompress, merge, and re-encode a body it can't vouch for. The rule underneath both: merge what you fully understand, decline what you don't.

Don't trade a stall for a balloon

The original bug was a slow reader backing up a pipe. A naive merger just relocates that disease one floor up: buffer eagerly, and the middleware becomes a balloon that grows until something else stalls. So the pressure passes through — when the downstream consumer signals stop, the queue pauses; when it signals go, delivery resumes. The middleware can smooth the stream's shape. It can't repeal its economics.

Loaded is not running

Design done, the remaining problem looked small: get the file into the router's process. Four configurations failed before one worked, and each failure was its own lesson.

Attempt one patched globalThis.fetch — the famous door. Nothing. The router's gateway doesn't call fetch; it requires undici and dispatches through getGlobalDispatcher(). Patch the API actually in use, not the one with the brand recognition.

Attempt two moved to the dispatcher layer — intercept undici's headers/data/completion handlers — and it worked, in exactly one process. The container runs more than one node process: a core server, and the gateway it spawns. Some provider traffic (DeepSeek's, it turned out) is fetched by the core server, which never loaded the patch — only the gateway had. Right layer, wrong process: a patch deployed into one process does nothing for its siblings, and when the target is "the container," the unit of deployment is the process tree.

Attempt three made loading universal: NODE_OPTIONS with --require, so every node process in the container loads the module at birth. The module loaded. Nothing happened. --require loads a file; it doesn't call anything in it — my module exported an install() and waited politely for a caller who never came. Loading is not running. The fix was to make the module install itself at load time, idempotently, so that being loaded is being deployed.

Attempt four targeted a file that already existed: the gateway's preload. Also a trap — the core server re-creates that file from an embedded copy via writeFileSync on every startup, so an edit to it has a lifespan of one restart. Never patch the generated artifact. Bring your own file, and get it invited.

The fifth configuration is the one running now:

NODE_OPTIONS: "--require /data/.claude-code-router/sse-coalesce.cjs"

My file, kept in the ops repo and bind-mounted read-only into the container at a path the core doesn't know about — which is precisely why nothing ever overwrites it — loaded into every node process, installing itself the moment it lands. The repo is the single authority; the container is just where it runs.

That placement carries one more trap, discovered when the middleware itself first needed changing: a bind mount's contents aren't part of what compose hashes, so editing the file leaves the running container untouched — and --require loads only at process birth anyway. Changed is not reloaded. Every edit to the middleware means docker compose up -d --force-recreate, or it isn't an edit at all.

Counting what came out

Before touching real traffic, the middleware got nine unit tests — the merge itself, content-length removal, backpressure passthrough, compression bypass, the refusal to merge across block indexes, ping dropping among them. Live verification ran through DeepSeek, because Zhipu's five-hour quota ceiling ([1308]) interrupted the plan mid-check — an accident that doubled as proof the middleware doesn't care which provider it fronts. A 200-token response arrived as 85 events and left as 11, counted by the middleware's own stats log — which logs every merge and truncates itself at 256 KB, so the observability doesn't become its own incident.

One hygiene change rode along: the router's request log had been capturing full bodies — 156 MB of them — and capture went from all to errors that night, with a vacuum bringing the file down to 12 MB. Which is why the granularity numbers in the previous piece are a record of one night, not a rerun.

Three minutes, still

Deployed, the middleware killed the hour-level hang — and left a residue. The next afternoon my phone and the extension disagreed again, only smaller: the notification arrived on time, and the extension took another three minutes to show the question Claude was asking. The transcript settled which one was honest, same as before — the question landed in it the same second the notification went out. The three minutes were all display queue: the turn's last two responses had been through the merger and still arrived as 534 events, at an estimated 250 to 400 milliseconds of rendering each. That last number deserves a pause. It's the extension's per-event cost, obtained by division, and it's the ceiling everything else in this story pushes against.

The stats log also explained why the merger hadn't merged more. Across the day, the forty-millisecond window was buying only three-to-five-fold reduction: the provider emits a delta every 25 to 50 milliseconds, and a window that narrow catches one or two before it expires. The knob had been set before anyone knew the stream's tempo.

So the knob learned the tempo, per type: two hundred milliseconds globally, five hundred for thinking — ninety-nine percent of the traffic, where nobody is watching smoothness — and a hundred and twenty for text, where latency is the thing a human feels. On live traffic: 719 events in, 31 out. Then 337 in, 14 out. Twenty-three-fold and better, against a floor I'd set at fifteen. The ceiling itself is upstream and untouched — a renderer that costs a quarter-second per event on a long session — so the whole game on my side of the pipe is how many events there are to render. Fewer is the strategy. On long sessions, so is /compact. That renderer is filed upstream now, at anthropics/claude-code#86854, where the per-event cost numbers above are recorded for whoever fixes it.

The middleware is sixteen kilobytes of code. Writing it was the short half of the job. The long half was learning where code has to sit before it counts as deployed — in the API actually in use, in every process that matters, invoked rather than merely loaded, in a file nothing regenerates, recreated by force whenever it changes. A fix doesn't exist when it's written. It exists when it's running.

The story wasn't over, though. The next morning, the same box nearly went down under a load average of 38.7 — a separate-looking incident that turned out to be this one's own aftermath, twenty hours downstream. That's The Twenty-Hour Fuse.

Two Clocks, Neither Lying

Jerome — Sat, 15 Aug 2026 14:14:21 +0000

The night of August 14th, Claude Code was planning me a trip. An eight-day itinerary — not code, just travel planning — running in the VS Code extension against the remote box where my sessions live. Just past midnight my phone buzzed: the turn had finished, and Claude was waiting for me to approve its plan. That buzz was claude-code-notify, a hook I wrote myself so that a long turn, or one waiting on my input, wouldn't pass unnoticed — before it existed, nothing notified me at all. I saw the message and switched to the window. The session tab said the turn was still executing.

Both indicators were right, and the hour between them was the bug. It just took a while to see how a session could be finished and not finished at the same time.

Late, not lost

The extension wasn't frozen, and it wasn't empty. Every so often it nudged a little more text into view — and the text was old, stale by the time it reached the screen. It wasn't showing me nothing. It was showing me the past, slowly.

That distinction did most of the diagnostic work before I touched anything. A lost handshake drops things: events go missing, the session looks truncated, the missing pieces never show up. A queue does the opposite: everything arrives, in order, late. What I was watching wasn't a failure to deliver — it was a delivery schedule. The events weren't lost; they were stacked up somewhere between the CLI and the screen, and something at the front of the stack moved slowly.

What the stalled session still knew

The stalled session's process was still alive — asleep, not spinning. No CPU worth mentioning, zero network connections, zero child processes. A turn waiting on a model has a socket open; a turn running a command has children. This one had neither. It wasn't waiting on the internet and it wasn't waiting on a shell. It was waiting on something local, the way a program waits on a pipe.

The session's own transcript said the same thing, more precisely. Claude Code journals every session as a file of JSON lines, and the last line of that night's was a complete assistant message — usage accounted for, stop reason recorded — ending in a tool call: the request for me to approve the plan it had drafted. After that line, nothing. No result, no close-out. The CLI had finished producing its question and never heard the turn end; it still believed the session was mid-flight. Even the tool-call id carried a small confession — its format was the third-party provider's, not Anthropic's, proof of which road the request had taken.

An alibi, timestamped

I run Claude Code through claude-code-router on that box — ccr, a self-hosted router that lets me point the same workflow at the official Anthropic subscription or at third-party providers. That night it pointed at a Zhipu GLM coding plan. The router logs every request it forwards, so "did the model side actually finish?" had a direct answer: the turn's request completed in 1.7 seconds — HTTP 200, no retries, no credential throttling — and in the surrounding hour, 186 requests, all 200. Everything upstream of the CLI had finished long before I started asking questions.

That pinned the stall to one segment: after the CLI, before the screen.

One variable left

Which left a short list of suspects, and a matrix that cleared it in one pass. Terminal plus ccr: fine. Extension plus the official API: fine. Extension plus ccr: stuck. The two working combinations have nothing in common except that they both avoid the failing one — so the variable isn't the router and isn't the extension. It's what the stream looks like when the extension consumes what the router forwards.

The router's log had captured response bodies that night, and each body told the story in two numbers:

Logged request	Output tokens	SSE delta events
749	6,400	6,314
745	3,822	3,803
746	2,877	2,720

Nearly one streaming event per token, roughly 135 bytes each — 855 KB of server-sent events for a single response — and with thinking mode on, about 99% of them were thinking deltas. The official Anthropic API doesn't stream like that: it merges many tokens into each delta event, which puts its event rate one to two orders of magnitude lower. Same conversation, same renderer — wildly different event counts. (Those bodies are gone now; I've since turned capture down to failures only, so this table is a record of one night, not a measurement you could rerun on my box today.)

A queue with a small mouth

Now the mechanism. For every streaming event, the extension does three kinds of work: parse the event's JSON, post a message across the Remote-SSH tunnel into the webview, and re-render the conversation — the whole conversation, which on a long session is not cheap. None of it is slow per event. It's slow per six thousand events, and the provider side was producing them faster than the extension could burn them down.

Between the CLI and the extension sits a plain stdout pipe, 64 KB by default. When the reader falls behind, the pipe fills, and then the writer blocks: the CLI's next write stalls until the extension drains some of the backlog. Everything the CLI tries to say next — including small control messages, like I need you to approve this plan — queues behind tens of thousands of tiny deltas waiting their turn at the renderer. An approval prompt that was ready at 12:28 a.m. reaches the screen an hour later, not because anything crashed, but because it joined the back of a very long, very slow line.

And that is why my phone was right. claude-code-notify hangs off the CLI's own hooks — the session lifecycle — not off the extension's rendering channel. The CLI finished its message and raised its approval request on schedule; the hook fired on time, straight to Telegram, without ever standing in the render queue. The two clocks weren't disagreeing about one quantity. One was reading the session. The other was reading a queue.

Neither clock lied

The notification reported the session: message finished, approval waiting — correct, at 12:28. The extension reported its render queue: thousands of events deep and moving slowly — also correct, about a different thing. What looked like a contradiction was two clocks measuring two different pipes, and the hour between them was just the drain time.

There's a real design question upstream of my setup, though. A renderer this slow shouldn't be able to stall the protocol channel: stream events can be batched for display without making control messages wait behind them. Per-token granularity exposed it, but the coupling is the extension's. I've filed it upstream — anthropics/claude-code#86854 — with the event-granularity data.

The fix, when it came, wasn't in the extension at all — it was a small middleware that merges those deltas back into chunks before the stream ever reaches the pipe. Getting it actually loaded took five attempts, four of them failures. That's the next piece.

That fix is the subject of the next piece — Coalescing the Stream.

One Fact, Nowhere on the Machine

Jerome — Thu, 06 Aug 2026 17:05:26 +0000

This started with a k3s install and not enough memory to do it in. I shut down another project's stack by hand, which freed up enough to get k3s running. k3s loads br_netfilter when it starts, and once the module was in place a kernel switch that applies to the whole host changed state. Then I brought the stopped project back, and its containers had lost the ability to reach each other. That's its own story. This one is about what I tried next — an attempt that did nothing for that problem and quietly broke a different one.

What I tried was a docker daemon restart: take the engine down, bring everything up clean, see whether the failure came back with it. It came back. The restart's real effect landed somewhere else entirely — every container on a different network, one I hadn't touched all day, came up holding an address it hadn't had before. Two of them matter here: the reverse proxy standing in front of every service on the box, and a proxy service standing in front of the reverse proxy. Neither had a static address pinned at the time, so each got whatever docker dealt it on the way up. That's where this story actually starts.

That second proxy is worth explaining before anything else, because the whole investigation turns on it. Every service on the box sits behind the reverse proxy's access rule, which admits exactly two sources and denies everything else. The one exception is the proxy service itself — call it the gateway from here — which exists specifically as a security control: connect to it first, and it's the only thing standing between the public internet and everything else running on the box. That's not how the wiring happened to end up. It's what the wiring was built to do — turn a set of publicly reachable services into something that behaves like a private intranet, gated behind one deliberately unrestricted entry point.

The allow-list, read right

After the restart I reported the symptom as precisely as I could see it: almost everything behind the reverse proxy had gone unreachable, while the gateway's own entry point still answered fine. Was it the access rule, the server's address, or a container's address that had moved?

The agent went straight to the reverse proxy's own configuration — the per-hostname files it writes one of for each gated service. Every one of them carried the same rule: allow an address on the shared docker network, allow the VPS's own public address, deny everything else. Read against the symptom, that rule explains all of it in one pass. Those two addresses are the only two ways in; the gateway's own front door carries no such rule at all. If either of the two allowed addresses had moved, everything gated goes dark while the entrance keeps answering exactly as it did.

That's the right read, and it came fast — one config, one pattern repeated on every gated host, matched cleanly against a symptom that had more than one plausible cause before this. What it didn't yet answer was the identity of the two addresses inside the rule. That's not a gap in the diagnosis. It's the next question.

The address that matched

One of the two addresses was easy: the VPS's own public IP, 203.0.113.10 — the address anything outside the docker network shows up as. The other was a docker-network address, 172.19.0.2, inside the same subnet the reverse proxy and the gateway both sit on. The natural next question was whose address that was, and there was a fast way to answer it: check what's actually running there right now.

It checked. The reverse proxy's own container, at that moment, was sitting at exactly 172.19.0.2. Address in the config, address on the wire, same value. That's not a weak signal — a live process answering at the address in question is about as direct a confirmation as a check like that produces.

Except the container's address wasn't fixed to anything. Nothing had pinned it there; it was wherever docker's own address allocation had put it after the restart that shuffled this whole network in the first place. The match wasn't the allow-list describing the reverse proxy. It was two independent things landing on the same value at the same moment — a rule that had been sitting there, and a container that had landed on that exact address by whatever the restart's allocation did with it. Nothing forced them to agree. They just did, right then, and there was no reason yet to ask why.

There was no step left undone here in any obvious sense. Check the config, check what's live, they match — that's exactly the kind of confirmation a live system is supposed to give you. The process had a shape that quietly assumed the docker address named something, and it never had to ask what the address meant, only what currently answered there. Something always answers there. That question doesn't have a way to come back negative.

What else got checked, and cleared

The agent made two more calls during the same pass, in the other direction — flagged, then walked back, both times before either one turned into an action.

The first was the VPS's own public egress address, spotted somewhere it hadn't been expected and read, on sight, as somebody's home or office connection — a plausible guess, since a bare public IP with no label attached could belong to anyone. Instead of accepting that, the agent checked: it asked a container on the host what its own outbound address looks like, then asked the host the same question, and got the identical value both times. Separately, the reverse proxy's own access log already had that exact address on file, showing up once a minute, every minute, as the source of a self-monitoring probe that had been running and been let through the whole time. An address that shows up once a minute forever isn't someone's laptop. It's the box talking to itself.

The second was a running process carrying the name of a well-known security tool, with nothing in any of the host's package managers recording it as installed and no binary sitting at the path a real install of that tool would use. On those two facts alone, flagging it as a possible intrusion was the right call to make out loud — an unexplained process wearing that name is exactly what a compromise can look like. It didn't survive the next check. Resolving the process's /proc/<pid>/exe link led to an ordinary, already-installed script interpreter, and its parent process was the gateway application's own binary: the tool ships bundled inside the gateway software itself, which is also exactly why no package manager had ever heard of it.

Two calls, two reversals, each one on a specific checkable reason rather than a change of heart. The habit of checking was doing exactly what it does throughout this investigation — catching a coincidence in one place, an unlabeled address in another, a suspicious name in a third, and running each one down before it became a decision. None of that checking was pointed at the one address that mattered. It didn't fail there. It was never aimed there.

Written down looks decided

The address match from two sections back didn't stay a working theory. Treating the docker-network address in the allow-list as the reverse proxy's own, the agent pinned the reverse proxy's compose file to that exact address — turning a coincidence into a static declaration — and reported the fix as applied.

That's the expensive step, and it's expensive for a specific reason. Everything before it was reasoning, held loosely, revisable the moment new evidence showed up. A pin in a compose file isn't reasoning anymore. It's a written commitment, sitting in the same file that decides how the container actually starts, and it reads to anyone who looks at it afterward — including the agent itself, on the next question — as a decided fact rather than a step that happened to follow from an assumption nobody had checked. The report went out the same way: fixed, not probably fixed.

An answer with a number in it

I read the fix and didn't accept it. Traffic through the gated services comes from the gateway — that's the reason the gateway exists in the shape it does — so the address pinned in the allow-list should be the gateway's, not the reverse proxy's. I wasn't guessing on the spot. I was remembering how I'd built the system in the first place and laying that against what the report actually said, and the mismatch was immediate: pinning the reverse proxy's own compose file to the reverse proxy's own address didn't match anything about how traffic through the gateway was supposed to arrive.

The agent didn't concede the point. It tested it.

A container already sitting on the shared docker network requested the public hostname directly, the same way any client on the internet would, and the reverse proxy's access log was read for what showed up as the source. The answer: the VPS's own public address, not the container's own docker-network address. Whatever a container on that network sends toward the public domain, the reverse proxy sees it arrive from outside — which meant pinning the gateway's specific address, on top of what was already pinned, shouldn't change anything either way.

That's a real result. The container really did hit the real hostname, the log really did show the public address, and nothing about the number was wrong. It answered a question — does traffic to the public domain from an arbitrary container on this network show up carrying that container's own address? — cleanly, and the answer was no. What it didn't answer, because nothing about the test asked it to, was whether the gateway's own traffic took that same path at all. A generic container resolving a public hostname the ordinary way, and the gateway forwarding through whatever its own proxy engine does with that hostname, are two different roads that happen to look identical from a compose file. The test measured the first one and reported on the second.

It read as the stronger answer of the two, not the weaker one. My objection was a sentence about how the system was designed to work. The agent's reply was a live measurement, run and read back on the spot. Between an argument from memory and a number from a test just run, the number wins the room every time — right up until someone checks what the number was actually a number of.

The fact that wasn't on the machine

I supplied the piece that neither the config nor the test could produce: the allow-list hadn't always carried two addresses. When I first wrote it, it carried one — the docker-network address — and that was enough. It worked.

Lay that fact next to the agent's test and the test breaks. If a request from anywhere on the docker network to the public hostname always arrives at the reverse proxy carrying the VPS's own public address, no exceptions — the test had just shown exactly that — then an allow-list with only the docker-network address on it, and nothing granting the public one, could never have let a single request through. Not once. But it had. For however long the allow-list ran with just that one line, something was reaching the reverse proxy from inside the docker network, under its own docker-network address, without ever picking up the public one on the way. The test said that path didn't exist. The system's own history said it always had.

There's no config file this comes from, no log line, nothing on the box today that records what the allow-list looked like before the second address went in. It isn't derivable from the server's current state — the state has already moved past it, twice over, once when the address was added and again when the restart reshuffled everything on top of that. It exists in exactly one place: my memory of building the thing. And it's what turns the investigation around — not a new check run against the server, but an old fact about the server that the server itself was never going to hand back.

The DNS override, and the address it always pointed at

The agent read the proxy engine's own configuration directly — not inferring from what a generic client did, but reading what the gateway's own software was actually told to do — and that settled it. The proxy engine carries a DNS override for the entire domain: every hostname under it resolves, inside the gateway, not to whatever the internet's DNS would return, but straight to a fixed address on the shared docker network.

That's the road the generic test container never took. An ordinary client resolves the public hostname through public DNS and lands on the VPS's public address, same as the reverse proxy expects from anyone outside. The gateway, for that one domain, never asks public DNS at all — the override answers first, and the request goes straight to a docker-network address without leaving the host. A connection that never leaves the host is never rewritten to look like it came from the host's own public interface. It arrives at the reverse proxy carrying the gateway's real docker-network address, not the VPS's public one — which is exactly the address the allow-list's other line was written for, and exactly what a request from an arbitrary container, going the long way through public DNS, was never going to produce.

The other half of the fault was on record from the start of this: neither the reverse proxy's container nor the gateway's had a static address pinned anywhere. The docker daemon restart that kicked this whole thing off reassigned both, along with everything else on that network, to whatever addresses happened to be free at the time — and a third container, unrelated to either of them, ended up sitting on the one address the proxy engine's override had always pointed to. That's what actually broke: not the allow-list, not the reverse proxy, not the override itself, but which container each address named on any given restart.

The fix pinned both: the reverse proxy to 172.19.0.3, the gateway to 172.19.0.2, each written into its own compose file. The third container moved off the address the override needed and onto a free one. Three containers restarted to pick up the change. Not the daemon — restarting that again is how this started.

Verifying it didn't call for trusting the compose files. It called for reproducing the actual path, so the agent did: from inside the gateway container, it issued a request following the same forwarding the proxy engine's override would take — straight to the reverse proxy's pinned docker-network address, bypassing public DNS the way the gateway itself does. The request succeeded, and the reverse proxy's own access log recorded the source as 172.19.0.2 — the gateway's own address, logged by the one component in the chain that was never asked to guess what it meant.

Nowhere in writing

None of this — the gateway as the deliberate front door, the docker-network address in the allow-list as always meant for it, the allow-list's own history before the public address got added — existed in any file on the box. Not in a compose file, not in a comment, not in the proxy engine's own configuration, which describes what it does but not why. The agent had the server's current state and nothing else to check it against, and current state doesn't distinguish a coincidence from a decision. Two things landing on the same address at the same moment reads identically to one thing being pinned there on purpose, right up until someone who was there says otherwise.

That's what made the wrong pin possible, and it's also what nearly made the wrong pin permanent. The agent's own test wasn't sloppy — it measured something real and reported it accurately. What turned the investigation was me declining an answer I had reason to doubt, on grounds the agent's evidence couldn't reach, and then handing over the one fact that reached back further than any log on the machine: what the allow-list looked like before there was a reason to add a second line to it.

The compose files carry the pin now. The design intent behind it — the reason 172.19.0.2 belongs to the gateway and not to whatever last happened to sit there — exists in exactly one place outside my own memory: a page of reference notes about this host. It wasn't there when any of this started. The record came after the repair, because the repair is what produced it.

Nothing Changed, Except One Kernel Module

Jerome — Thu, 06 Aug 2026 17:05:17 +0000

Memory was tight on the box, so I stopped another project by hand to make room for a k3s install. The install pulled in br_netfilter, which flipped a kernel switch host-wide. The project I'd stopped came back up — and its containers couldn't reach each other anymore.

My first move was restarting the docker daemon, on the theory that bringing everything back up fresh would shake the problem loose. It didn't touch this one. It did reassign every container IP on a second, unrelated network, which left me with an allow-list somewhere no longer admitting the source it had been written to admit. That's its own story. This one is about the first failure, and it turned out to have nothing to do with the daemon restart at all.

The symptom held up under testing. No container on that bridge could reach any other — I checked several pairs, and they all failed the same way. Every attempt just ran until it timed out; nothing came back refused. A refused connection means a socket answered and said no. A timeout means the packet went somewhere and nothing came back, which puts the fault in the network layer, not in whatever was supposed to be listening on the other end. A different stack on the same host, on a different bridge, kept working the whole time. Whatever this was, it wasn't host-wide.

The only change

Only one thing had actually changed on the host that day: the k3s install, and specifically what it did to the kernel underneath it. k3s's own systemd unit runs ExecStartPre=-/sbin/modprobe br_netfilter — from /etc/systemd/system/k3s.service — before the service itself starts. That's k3s loading the module as shipped, not a step I added.

Loading the module isn't what matters on its own. What matters is what comes with it: br_netfilter brings net.bridge.bridge-nf-call-iptables into play, and somewhere in that install, the switch flipped — from 0 to 1, host-wide. That sysctl decides whether traffic that never leaves a bridge gets shown to iptables at all. Two containers on the same bridge talking to each other is bridge-internal traffic in the strict sense — packets forwarded between ports of the same bridge, no routing involved. At 0, the kernel never submits that traffic to iptables; nothing evaluates it. At 1, it does — the same chains that judge traffic arriving on a real interface now get asked to judge this too.

Nothing about the containers changed, and nothing about the project I'd stopped and brought back up changed either. What changed is which packets iptables now gets a say over. k3s did that on purpose: loading the module and turning on that sysctl is baked into how it starts, on any host it's installed to. It isn't a step aimed at this box specifically, and it isn't a step k3s skipped or fumbled. The install did exactly what it was built to do.

Flipping the switch, and what it doesn't prove

I tested the sysctl directly that day. net.bridge.bridge-nf-call-iptables to 0, and the containers could reach each other again — immediately, no restart, no wait. Back to 1, and they stopped again, just as fast. I flipped it a second time to be sure. Same result both directions.

That's confirmation, not a diagnosis. All it shows is that whatever's dropping this traffic sits downstream of that switch — not what that thing is, and not why it's dropping these particular packets between these particular containers instead of every packet crossing the bridge. This identifies the mechanism, not the rule. And leaving the sysctl at 0 was never a real fix — the switch is host-wide, and k3s's unit sets it unconditionally every time it starts, so leaving it at 0 only holds until the next restart flips it back. It isn't a fix scoped to whatever's actually dropping these packets. From here the question splits in two: something inside iptables is dropping this traffic, and nothing I'd tested so far said which table, which chain, or which rule.

Ruling out everything else

Bridge port STP state first — a port stuck outside forwarding would produce exactly this symptom on its own, no iptables involved. bridge link show on both ports: forwarding, forwarding. Not that.

tc had no filters attached to either interface, so there was nothing to even check a counter on; XDP had no program loaded on either side either. nftables' bridge family — the modern interface to the same hooks ebtables covers — had no tables defined at all, and ebtables itself came back with an empty ruleset. Last, the ARP cache, because a stale entry would produce this exact kind of silent black hole without touching anything above it: the destination MAC each sending container had cached matched the target container's actual, current MAC.

Five places checked, five negatives. Whatever was eating this traffic wasn't sitting in any of the spots that usually catch a problem at the bridge layer.

What conntrack didn't have

conntrack wasn't on this box to begin with — I installed it mid-investigation, specifically to watch this flow live.

It had nothing to show me. Not a dropped entry, not a half-open one — no entry, for a flow I could reproduce on demand just by opening a connection between the two containers. A conntrack entry gets created the moment a new flow is first evaluated, win or lose. If there's no entry at all, connection tracking never got a look at this traffic.

That points at exactly one place. In the IPv4 PREROUTING hook, netfilter runs raw at priority -300 and connection tracking at -200, and hooks fire in increasing priority order — raw always goes first. A packet dropped in raw never reaches conntrack at all, which is exactly the shape of what I wasn't seeing. One table left to check.

The raw table

iptables -t raw -L PREROUTING -n -v -x --line-numbers turned up two matched sets of rules, seven lines each, keyed to the same seven destination addresses — 172.18.0.2 through 172.18.0.8. Each set named a different bridge interface. One bridge was live; the other was stale, already deleted.

The live set's shape, verbatim:

-A PREROUTING -d 172.18.0.3/32 ! -i br-b951f3fb0958 -j DROP

DROP, gated on a /32 destination match and a negated input-interface match. dockerd puts one of these on every container IP itself, and what it's enforcing is anti-spoofing: drop traffic for this address unless it arrived on the bridge that's supposed to own it. ! -i br-b951f3fb0958 reads as "drop unless it came in on br-b951f3fb0958" — a real exception, as long as br-b951f3fb0958 exists. The stale set carried the identical logic, just naming br-66885a1f7aad in place of br-b951f3fb0958 — and br-66885a1f7aad no longer existed. So there was no interface left for a packet to arrive on that could ever satisfy that stale set's exception. The condition could never be true. A rule that reads as conditional is, in practice, unconditional — every packet for those seven addresses, arriving from anywhere, got dropped by the stale set regardless of where it actually came from.

The counters said it before I'd finished reasoning through it. The rules pinned to the deleted bridge were already past 4,967 hits that day, and still climbing. The rules pinned to the live bridge — the correct rules, the ones actually describing the network as it existed — sat at zero. That's not reassuring. iptables walks a chain top to bottom and stops at the first match, and the stale rules came first in the chain. Zero hits on the correct rules didn't mean they were fine. It meant they'd never been reached at all.

Whose bug this is

k3s loaded br_netfilter and flipped the sysctl because that's what it does on every host it installs to — already established, not up for reargument. The seven rules keyed to a bridge that no longer existed are a separate question, and the answer isn't k3s.

dockerd wrote those rules. This host runs Docker Engine 29.6.0, and since Docker Engine 28.0 (2025) — a deliberate hardening change, not a bug — dockerd drops unsolicited inbound traffic to a container's internal IP by default unless the port is explicitly published. The raw-table rules from the previous section are that change's implementation. The blog post doesn't get me that far on its own — it frames the change around unpublished ports and never names a table. What names it is moby#49621, which added DOCKER_INSECURE_NO_IPTABLES_RAW=1 as an opt-out for kernels built without CONFIG_IP_NF_RAW: an escape hatch that only makes sense if the rules were going into raw in the first place.

At some point before this incident, the network these seven rules belonged to went down and came back up, and dockerd built a new bridge to replace the old one — the live br-b951f3fb0958 standing in for the dead br-66885a1f7aad. The new bridge got its own seven rules. The old bridge's seven didn't go anywhere. On this host, nothing removed those rules when the interface they named stopped existing.

I looked for whether that specific gap is a known, filed defect, and I didn't find one. The closest matches are moby/libnetwork#570 — stale FORWARD-chain rules left behind after a bridge is removed — and a docker/cli report about network rm corrupting a different network's rules. Different chain in one case, different trigger in the other; neither is this. What I can say is what happened on this host on 2026-08-06. I can't call it a tracked bug, and I'm not hanging a version range on one.

And it sat there doing nothing for the same reason established at the start: at bridge-nf-call-iptables=0, bridge-internal traffic never reaches iptables at all, live rule or stale one. The seven orphaned rules weren't evaluated, weren't skipped, weren't anything — they just weren't asked. k3s didn't write this bug. It was the first thing on this box to turn the light on in the room where it had been sitting the whole time.

Seven deletions, and the check that would have caught it earlier

The fix is narrower than the last six sections might suggest: delete the seven rules pinned to br-66885a1f7aad, and leave everything else in the raw table exactly as it is — including bridge-nf-call-iptables, which stays at 1. k3s and its CNI need that sysctl on. Turning it back off doesn't fix this; it just re-hides it, and takes bridge-internal traffic on every other network on the box off iptables' radar along with it. That's not a fix, that's the same outage wearing a different cause.

iptables -t raw -D PREROUTING <line>, seven times, one call per stale rule, in descending order of line number. Delete low-to-high and every deletion renumbers the lines below it — the seventh line you meant to hit has already moved by the time you get to it. High-to-low, each deletion only touches numbers you've already dealt with, so the number you read off --line-numbers is still the number that's true when you use it.

Checked again on 2026-08-06, the fix held: iptables -t raw -S PREROUTING on this host shows exactly seven DROP rules for 172.18.0.2 through 172.18.0.8 — all seven addresses, none missing — every one of them naming the live br-b951f3fb0958. Nothing in the table names br-66885a1f7aad anymore.

That closes the one incident. It doesn't close the bug, because the bug isn't scoped to that bridge or that day — the same orphaned rule can turn up on any other network on this host, for as long as nothing on this host cleans up a stale rule when a bridge gets recreated. The check that catches it is the fix run in reverse: take every interface name the raw table's PREROUTING chain references, and confirm each one still exists.

comm -23 \
  <(iptables -t raw -S PREROUTING | grep -oP '(?<=-i )\S+' | sort -u) \
  <(ip link show type bridge | grep -oP '^\d+: \K[^:@]+' | sort -u)

Anything that prints is a rule pointed at an interface that isn't there — the same bug, on some other bridge, waiting for whatever eventually submits its traffic to iptables. Run against this host on 2026-08-06, it prints nothing: four bridge names in the raw table, all four listed in ip link show type bridge, none orphaned.

Nineteen Subdomains, One Allowed IP

Jerome — Mon, 03 Aug 2026 23:09:11 +0000

One DNS record covers the whole box: *.example.com → 203.0.113.10. I set it once, and publishing a service has not involved DNS since. I pick a name, and the name already resolves.

What's left is a form in Nginx Proxy Manager (NPM) with four fields that matter: the domain, the container name, the container port, and a checkbox that requests a certificate. That is the whole act of publishing. Fill it in, save, and the hostname is serving HTTPS from a container that was never exposed to anything.

Once publishing is a form, it is mechanisable. I stopped filling it in by hand a while ago — I describe the service, Claude Code takes it end to end (compose file, network, proxy host, certificate), and hands me back the URL.

That is how I ended up with nineteen proxy hosts. Nineteen public hostnames, all of them off-the-shelf self-hosted software, each with its own idea of what authentication means: some have real user accounts, some a single shared password, some a setup wizard that never asked, some nothing at all.

The ease is the problem. Nothing in that form makes you decide who is allowed to reach the thing you just published.

What I actually wanted wasn't to keep any of it off the internet — that stopped being an option the moment the wildcard resolved. I wanted all nineteen of them reachable from exactly one place: a private network, built entirely out of public infrastructure, where being on the internet and being reachable are two different facts.

What the wildcard buys, and what it doesn't

The wildcard is a DNS record, not a certificate. Those two get conflated constantly, and the difference decides what the firewall looks like permanently.

The record is real work removed. *.example.com points at the VPS, so every hostname I will ever invent already resolves; there is no propagation wait between deciding on a name and using it.

The certificates are not wildcard. NPM issues them per hostname over HTTP-01, so for each of those nineteen names Let's Encrypt makes an inbound request to http://name.example.com/.well-known/acme-challenge/... and expects nginx to answer on port 80. Nineteen certificates, nineteen challenges — and since they last about ninety days, the renewal timer re-runs the same challenge against every one of them, forever.

So port 80 can never close. Not "should probably stay open" — cannot close, on either of the two firewall layers this host has. And the failure mode is silent: closing 80 breaks nothing you can observe. Sites keep serving on 443 with the certificates they already have, renewals fail into a log nobody reads, and the outage lands ninety days later as every hostname going untrusted at once.

DNS-01 would buy a genuine wildcard certificate and let 80 close, at the price of giving the proxy write credentials to my DNS zone. I didn't.

One network, and why the proxy finds services by name

The network gets created once, by hand — docker network create proxy — and every compose file after that declares it as something it did not make:

services:
  trilium:
    networks: [proxy]      # this is what attaches the container
networks:
  proxy:
    external: true         # this only says where the network comes from

Those two blocks look like one setting written twice. They aren't. Leave out the one under the service and the file is still valid, the network still resolves, and nothing is on it.

With both in place, the proxy host form takes a container name: forward hostname trilium, forward port 8080. Docker's embedded DNS resolves the name across the shared network, and 8080 is reachable inside that network and nowhere else. None of the containers nginx proxies to publishes a port to the host at all.

That is the part that is easy to half-do. Joining the shared network does not reclaim a published host port. They are two separate acts, and doing only the first adds a road without removing one.

nginx serves its own admin panel the same way, by proxying 127.0.0.1:81, so 81 is never published either — the entrance shouldn't have a second door. You will be told this can't work — that inside the container 127.0.0.1 is the container, so it loops and 502s. The premise is right; the conclusion isn't. One nginx process serves every virtual host, and handing a request from the 443 listener to the 81 listener is a hop, not recursion.

docker exec npm curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:81/
# 200

Real loops come from forwarding a host to its own hostname, or 443 to 443.

The gateway, and the coincidence it runs on

One more container sits on that network: a self-hosted proxy gateway, on the same VPS as everything else.

Only one thing about it matters here, and it's where its traffic comes out. Point a device at the gateway and that device's requests leave the VPS through the VPS's public interface and come back to :443 on the same box. To nginx there is nothing exotic about them — ordinary inbound HTTPS off the public internet — and the client address it writes to the log is 203.0.113.10. The VPS's own.

I'd seen that address in that column before, and I'd been wrong about it. I was chasing intermittent connection drops at the time, and the failing requests logged a client IP identical to the server's public address. It looked like a routing fault: something on the box talking to itself, traffic eating its own tail. I filed it as a loop to untangle later.

It wasn't a fault. It was the gateway, doing what a gateway does. The log had been telling me the truth the whole time, in a shape I didn't recognise.

The loop I tried to route around is what the whole design stands on.

What the rule actually says

Every hostname that gets gated gets the same block, written by NPM into that proxy host's config file:

location / {
  auth_basic            "Authorization required";
  auth_basic_user_file  /data/access/2;

  allow 172.19.0.2/32;      # the gateway container, on the shared docker network
  allow 203.0.113.10;       # the VPS's own public address
  deny all;

  satisfy all;
}

The two allow lines are an OR. A request passes the address check by matching either one; everything else falls to deny all and gets a 403 before nginx opens a connection to the upstream at all.

They both have to be there because the same request can arrive from either side of the box. A container on the shared network connects to nginx across that network, and the client address nginx records is that container's Docker address. Anything not on that network comes in through the host's public interface, and nginx records the VPS's public address — including the gateway's, which left the box and came back. Two paths in, two addresses in the log, one rule that has to cover both. Neither line is a spare.

satisfy all is not part of that. It's a different axis: it makes the address check and basic auth both mandatory. (satisfy any — the one people pick by accident — would make either alone sufficient.) So an allowed address buys you a password prompt, not the service.

That's the design, stated in one line: nineteen public hostnames, all of them resolving, all of them presenting a valid certificate to anyone who connects — and none of that decides who gets in. The two allow lines and the password after them do. A public DNS record and a public certificate get a stranger exactly as far as a login prompt they can't clear. That's what the wildcard actually bought me: not privacy for the DNS, not secrecy for the addresses, but a private network assembled entirely out of public infrastructure — an intranet where every door is a public IP with one line of nginx behind it.

One path escapes all of it. The rule is scoped to location /, and NPM includes a separate location that opts out on purpose:

location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    auth_request off;
    allow all;
    # ...
}

Let's Encrypt publishes no IP ranges for its validation infrastructure, so there is nothing to allow-list — the challenge path stays open to the entire internet or the renewals stop. Same fact as port 80, wearing different clothes: a deliberate, documented hole that is the correct call.

The two hosts that can't be gated

Seventeen of the nineteen proxy hosts carry that block. The two that don't are the gateway's own admin panel and the endpoint that hands out client configs.

They can't carry it. The rule assumes the gateway is already carrying you — that's what puts an allowed address in the log. Reaching the panel means getting in before that's true, and setting up a new device means fetching its config from a device that isn't configured yet. You can't put the key inside the lock.

So those two sit on their own authentication and nothing else: a login form on the public internet, with whatever password I gave it and whatever rate-limiting the software ships with. Every other hostname on the box is behind two independent checks. These two are behind one, and I'd rather name that than pretend the wildcard covers it.

What the proxy broke: services that guess their own address

Putting everything behind one nginx broke a service that had run untouched for a year. Not the gateway, and not either of the ungated two — this one carries the full block now; it didn't then, and this is why. It issues each user a connection profile with a server address baked into it.

It failed in stages: one person, then two, then everyone who asked.

Everything the box could tell me was green: container up, credentials matching, nothing banned, a clean log. Curling the service's hostname from the server returned 200.

So I formed a theory, and it was a good one: a firewall rule had been deleted. I'd spent that afternoon in the cloud console editing security-list rules. Healthy inside, dead outside is the shape of a packet filter dropping traffic before it lands.

Instead of reasoning further, I asked someone who was failing to send me what their client actually said.

connect: connection refused, against 198.51.100.20. Not my server. That killed the theory twice over — a filter that drops packets gives you a timeout, not a refusal, and the client wasn't trying to reach my server at all. A reverse lookup landed on my own ISP's egress: my home connection, that morning.

nginx was doing its job: it sets X-Forwarded-For to the real client address. The service downstream read that header as its own public address — the one it writes into every config. So the address it advertised was whoever had asked last. My laptop had pulled a config from home that morning; after that, the service told everyone to connect to my house. The field that pins the advertised address was right there in the settings, empty.

The neighbouring mistake is easy: the port it listens on inside its container and the address it advertises outside sit next to each other in one form, and the first instinct is to change the wrong one. That code path had been unreachable since install — clients connected straight to the box, no header to misread.

And it hid for days because of where I stood to test. My own client goes out through the gateway, so every config I fetched for myself came back carrying 203.0.113.10 — exactly what it was supposed to carry. It read correct every time. The address that makes the access rule work is the address that hid this bug: one fact doing both jobs, read from the wrong side twice.

Where the encryption stops

The certificate lives on nginx and nowhere else. TLS terminates there; from nginx to the container the request travels as plaintext HTTP across the Docker network. That is why the Scheme field on every proxy host reads http, and why setting it to https returns a 502 immediately instead of hanging — nothing on the backend port is listening with TLS.

Fine in itself, since that network is published to nothing. But it decides where the access check has to live. Past nginx the client is gone: the container sees a connection from a Docker address on its own network, and everything else it knows about the caller arrives as a header. nginx is the last point that still has the client's address from the socket rather than from a string. The rule goes there, or it goes nowhere.

What this is not

Two allowed addresses is not zero-trust. Anything that can source traffic from either address is inside. Everyone else who uses the gateway arrives as 203.0.113.10, same as me; so does every process on the box. The rule doesn't identify a caller — it says the request came in by an expected road.

Behind it is basic auth, which is a password. One shared string, no second factor, nothing rate-limiting guesses. Two checks beat one because they fail independently, not because either is strong.

The services still trust nginx's headers without question — the same unconditional trust that spent days advertising my house. The rule doesn't fix that. It shrinks the audience for it.

Port 80 stays open permanently, and the challenge path under it stays open to everyone.

If the gateway dies, so does my access to seventeen hostnames. SSH is the way back — which is why the rule is nginx answering 403 rather than a firewall dropping packets. A 403 lives in a config file on the host; the door I need during an outage can't be the one I just shut.

The address in both columns

Nineteen hostnames gated by one line, and what holds it up is an accident of topology: traffic that leaves this box and comes back is indistinguishable from traffic that was never here. I get to allow-list a single address because the machine vouches for me to itself.

The cost is that I debug from the position the rule rewards. Whatever I check, I check while already holding the credential, so it checks out. It took someone else's error message to show me a view I couldn't produce.

The cheapest access control on this box and the bug that hid longest on it are the same string in the same column of the same log. I'd take the trade again. I'd just stop testing from inside the allow-list.

236 Tests Passed. The Release Was Still Broken.

Jerome — Fri, 24 Jul 2026 05:31:18 +0000

I fixed a small timezone bug, ran the suite, and watched all 236 tests go green. Out of habit more than doubt, I pushed to CI before tagging the release. CI went red — and then it went redder. That one small fix had two more bugs stacked underneath it, and every one of them was invisible on the machine where I'd just watched the tests pass.

The setting is a tool I maintain called claude-code-notify: it pings me when Claude Code hits a usage limit, and again when the limit resets. You don't need to know the tool. The bug is an ordinary timezone bug, and what it cost me is a lesson about tests — about how a completely green local suite can be lying to your face.

The original bug lived in the part that computes when the limit resets.

Right only by coincidence

When Claude Code tells you a limit will reset, it writes the time into the message as plain text, with the zone in parentheses: resets 5:20am (Asia/Hong_Kong). The function that read that time, parse_reset(), pulled out the hour and minute and did all the arithmetic in the host machine's local time. It never looked at the zone in the parentheses.

The host machine, in my case, is a remote server pinned to Hong Kong time. So the zone Claude Code embedded and the zone my code silently assumed were the same zone — Asia/Hong_Kong on both sides — and the answer came out right every single time. Not because the code was correct. Because two timezones I'd never thought to compare happened to be identical. Run the same tool on a laptop in London, or catch a reset that Claude Code reports in another zone, and the "your limit resets at…" ping lands hours off, with no exception and no log line to explain it — just a confident, wrong time.

The fix was small: capture the zone name out of the text and resolve it through Python's stdlib zoneinfo, falling back to host-local time only when the zone is missing or unresolvable — which is exactly what the code used to do always, now demoted to a deliberate last resort. The tests passed. This is the point where I should have been suspicious and wasn't, because locally there was nothing to be suspicious of.

The test that couldn't fail

ubuntu-latest runs in UTC. The first CI run came back 4 failed, 232 passed, and two of the failures were in tests I hadn't touched at all:

tests/test_usagelimit.py::test_parse_reset_returns_next_local_occurrence FAILED
tests/test_usagelimit.py::test_parse_reset_rolls_to_tomorrow_when_past FAILED
E   assert (13, 0) == (21, 0)

Both checked parse_reset() against resets 9pm (Asia/Hong_Kong). Both built the "current time" they fed in from a plain datetime — read as host-local — and then verified the parser's answer by converting it back, also as host-local. The same ambient timezone sat on both sides of the assertion.

Under the old, buggy parse_reset(), which also worked in host-local time, that symmetry was fatal in a quiet way: feed in host-local, compute in host-local, read back in host-local, and the timezone cancels straight out of the equation. The two sides matched on every machine on earth. The test read like it was verifying timezone handling. It was verifying that a number equals itself. It could not fail — not on my box, not on anyone's.

Bug 1's fix broke the symmetry, which is the only reason any of this surfaced. parse_reset() now resolves 9pm Hong Kong to a real instant, the same instant no matter what the host clock says. On my Hong Kong machine, reading that instant back as host-local still shows 21:00 — still green. On the UTC runner, the identical instant reads back as 13:00, because 9pm in Hong Kong is 1pm UTC. (13, 0) == (21, 0): an eight-hour gap that appears the moment the code runs somewhere my clock doesn't.

This is the one to keep. A test that derives its expected value from the same environment the code reads is self-consistent, and on a green dashboard self-consistent is indistinguishable from correct. It passes, so you trust it, and it is checking nothing. The repair is to anchor the expectation to something external and explicit — both the input time and the assertion pinned to zoneinfo.ZoneInfo("Asia/Hong_Kong") instead of "whatever this machine calls local." I re-ran it under TZ=UTC, TZ=America/Los_Angeles, and TZ=Pacific/Kiritimati before pushing again, so the test now asserts a fact about the world instead of a fact about my machine.

The tests didn't fail the way the code would

Second push. Red again, and differently:

tests/test_usagelimit.py::test_parse_reset_uses_reported_timezone_not_host FAILED
E   ModuleNotFoundError: No module named 'zoneinfo'

zoneinfo only entered the standard library in Python 3.9. The tool supports 3.8 — pyproject.toml declares requires-python = ">=3.8" — and the CI matrix runs 3.8 on purpose. The production code already accounted for that: the import of ZoneInfo sits behind a try/except ImportError, and when it fails the code falls back to host-local time, the same fallback Bug 1's fix leans on. It bends.

The tests didn't bend. The four that exercise timezone resolution — the two I'd just anchored, plus two I'd added alongside the fix — each did a bare from zoneinfo import ZoneInfo. On 3.8 that isn't a graceful fallback; it's a hard ModuleNotFoundError before the test body even runs. The code under test degraded, and the tests meant to cover it snapped instead.

The fix is one line apiece — pytest.importorskip("zoneinfo") — which skips the test cleanly when the module isn't there. The principle behind it is duller than either earlier bug and matters more than both: a test has to degrade the same way the code it covers degrades. If production tolerates a missing optional dependency, a test that hard-requires that dependency isn't testing production. It's testing a stricter promise the code never made.

Red that meant something, and red that didn't

One more job in that run was red, and it was bait. macos-latest on Python 3.8 showed as failed, sitting right next to the genuine 3.8 failure on Ubuntu — close enough to look like the same bug. It wasn't. Its log said The operation was canceled, at the setup-python step, before a single test ran. When one leg of a matrix fails, GitHub Actions cancels the rest by default, and my workflow never turns that off — so the real Ubuntu failure dragged the entire macOS column down with it, 3.8, 3.11 and 3.12 all cancelled, not one of them an actual bug. Chase that red as a fourth problem and you lose an afternoon debugging a cancellation. Telling a real failure apart from fail-fast noise is its own step; the color alone won't do it for you.

Line the three real bugs up and they rhyme. Bug 1 needs a host whose timezone differs from the reported one. So does Bug 2. Bug 3 needs a Python without zoneinfo. My dev machine is in Hong Kong, on Python 3.12 with zoneinfo built in — the one configuration on which not a single one of the three can happen. I didn't get unlucky and overlook them. The machine I built on was, structurally, the exact machine where all three were invisible. Every green run I'd trusted was true and worthless in the same breath: the code agreed with the computer that ran it. That is the whole of what a green suite proves. Whether the code is correct is a separate claim, and the space between those two claims is precisely where a machine's timezone, its Python version, and its installed modules go to hide.

The fix that actually mattered wasn't any of the three patches. It was running the matrix before tagging, instead of reading "236 passed" on one box as if it meant "shippable." Any of the three patches was ten minutes' work; the step I'd been skipping was the cheap, redundant-feeling one — letting a machine that isn't yours try the code before you call it done.

The tool is claude-code-notify — MIT-licensed, at github.com/Jeromefromcn/claude-code-notify, correct reset timezones now included. If Claude Code's usage limits have ever cost you an afternoon, it might be worth a look. And if you'd rather read the bugs than the writeup, they're three commits in the history — one per bug.

Claude Code Hit Its Limit — and the Notification Never Came.

Jerome — Fri, 24 Jul 2026 05:30:57 +0000

A while back I wrote about claude-code-notify, a small tool I built because I kept losing time to work I wasn't watching. It pings me on Telegram when a Claude Code turn finishes, when it needs my input, or when it dies with an error — so I can kick off a long task, switch windows, and stop thinking about it. The push reaches me whichever session has focus; being reachable when I'm not looking at the terminal is the whole point.

One case slipped through: the usage limit.

Hit your account's usage limit and every running session stops at once. The tool did fire — my error hook caught the dead turn and sent its generic "stopped with error" ping. But that ping tells you something broke, not that it was a limit, and not when it'll lift. Then a few hours later the limit quietly resets and nothing fires at all. So the shape was always the same: hit the limit, get a vague error, put the phone down, forget. By the time I wandered back, the limit had been open for who-knows-how-long — the exact dead time the tool exists to kill.

So I set out to add the one notification that would close the gap: a limit-specific message — something like usage limit reached, resets 5:20am — and a second ping when it actually reset.

It should have been an afternoon. Instead, just to add this one notification, I fell into a string of silent failures: the tool breaking in the one way a notification tool can't afford to, and never telling me it had.

The version that never fired

I wired it up the way you'd expect. On the error hook, read the session transcript, find the last assistant message, check whether it's a rate-limit envelope — Claude Code tags those with isApiErrorMessage: true and error: "rate_limit" — and if so, pull the reset time out of the text and send the limit-specific ping. I tested it against a saved transcript, it worked, I shipped it.

Then I hit two real usage limits over the next week. Both times: the generic "stopped with error" ping, and nothing else. No limit-specific message, no reset ping. The new detection had run and decided, both times, that this was not a usage limit — on a turn that had died of nothing else.

The generic ping was the tell. It's the consolation prize of a detection that has silently failed: the turn errored, so the old error hook fired regardless, which meant every real miss still looked like a notification arrived. I only noticed anything was wrong because I knew a limit-specific message was supposed to be there and wasn't.

So I added debug logging around the transcript read and waited to hit the limit again. When I did, the log showed the read returning nothing — no rate-limit envelope in the transcript at the moment my hook looked. Then I compared the read timestamp in my log against the file's own modification time. In that one incident, my code had read the transcript at 00:13:14.735881; the file's mtime was 00:13:14.755575. I had read it 19.7 milliseconds before Claude Code finished writing it. One measurement, one incident — I don't know the distribution — but the direction was unambiguous: the read beat the write.

Here's the mechanism, and it's the whole article. transcript_path in the hook payload guarantees the file exists. It does not guarantee the current turn is in it. Claude Code writes the transcript asynchronously, and the StopFailure hook can fire before the rate-limit envelope has been flushed to disk — the same local disk, same machine, no network anywhere in it. My detection was reading a file Claude Code hadn't finished writing, seeing no rate-limit envelope, classifying that as "not a limit," and moving on. Correct, given what was on disk. Useless, given what was true.

The patch, and the fix I should have started with

The first fix was the obvious one: if the first read comes up empty, wait and read again. One retry at 200 milliseconds — _STOP_FAILURE_RETRY_DELAYS = (0.2,), roughly ten times the gap I'd measured — gave the write enough slack to land by the second look. It worked; the next few limits sent proper reset pings.

But a 200ms sleep to paper over a race I didn't fully understand is a patch, not an answer. Before building anything else on top of it, I went and read Claude Code's official hooks documentation — which I should have done before writing a line of transcript-parsing code, because the answer was sitting in it twice over.

The race is documented behavior, not something I'd stumbled into. The reference says the transcript file "is written asynchronously and may lag the in-memory conversation, so it may not yet include the current turn's most recent messages when a hook fires," and it says what to do instead: use last_assistant_message "instead of reading the transcript." There's even a public issue for the exact symptom — anthropics/claude-code#15813 — though a stale bot closed it for inactivity rather than any fix, so this is just how it works: documented, and unowned.

The bigger miss was the part I'd assumed away. StopFailure hands you the error in the payload. It arrives in the hook's stdin JSON, right next to transcript_path: a structured error field drawn from a fixed enum — rate_limit is one of the values — plus last_assistant_message, which on an error turn holds the API error string itself. No file, no flush, no race. I'd built the entire transcript-reading path on the belief that the hook gave me nothing structured and I'd have to reconstruct the error myself. It had been handing it to me the whole time.

So I flipped the priority. The payload is the primary source now: if error is rate_limit and last_assistant_message is there, classify and send, with no disk read at all. The transcript read, retry and all, stays only as the fallback for the edge the payload doesn't cover — the racy path runs when it has to and never when it doesn't.

Then it fired when it shouldn't have

The classifier had a second bug waiting, and this one broke in the opposite direction: one day it sent me a usage-limit notification that was a lie.

The turn hadn't hit my account's usage limit at all. It had tried to use Fable 5 — a model that bills against its own usage-credits balance instead of the subscription allowance — without credits. Claude Code's message said so plainly: Fable 5 requires usage credits. Run /usage-credits to continue or switch models with /model. My tool pinged "usage limit reached" anyway.

It fooled the classifier because Claude Code tags this error with the same fields a real limit uses. Same isApiErrorMessage: true, same error: "rate_limit", same 429 status. If your rule is "it's a limit when error is rate_limit" — which was exactly my rule, on both the payload and the transcript path — a per-model credits gate is indistinguishable from an account limit. That one field is doing double duty for two conditions that mean completely different things to me.

What set them apart sat a level deeper. On disk, the genuine limits I'd inspected carried no errorDetails at all; the credits error carried one, with a structured code inside it: error.details.error_code == "credits_required". That field, and only that field, is the ground truth.

So the fix is one extra clause, on both paths: it's a usage limit only when error is rate_limit and the error body isn't a credits_required error. It keys on the structured code, never on the words "usage credits" in the message — text-matching is how you build a classifier that breaks the moment the wording changes, and not doing it is a standing rule in this project. The most specific structured field wins.

What both failures had in common

Put the two bugs side by side. One never fired; one fired when it shouldn't. Opposite symptoms — the same mistake. Both are what happens when you read one ambiguous, half-documented signal that another process hands you and trust your first reading of it. The miss trusted an empty file. The false alarm trusted an overloaded field. Wiring the hook was never the hard part; classifying the signal on the other end was the entire job.

And here's what stings: neither bug was catchable by the test suite as it stood. My fixtures encoded the happy path — a fully-written transcript with a clean rate-limit envelope. A fixture is an already-written file, so it can't reproduce a race against a file still being written. And I had no credits-error fixture at all, because I didn't know that error existed until Claude Code sent me one. The tests were green the whole time. They were confirming that my code agreed with my assumptions, which is a different thing from confirming it was right.

Both bugs surfaced the same way: a real event I couldn't have predicted, plus enough debug logging already in place to see what actually happened when it hit. The retry, the payload switch, the credits check — those were the easy part once I could see the signal. The discipline that made them possible was instrumenting the thing before I understood it, so the next strange event would leave a trace instead of a shrug. That, more than any single fix, is the skill.

The worked-out version — retry, payload-first, the credits check — lives in claude-code-notify on GitHub, MIT-licensed: github.com/Jeromefromcn/claude-code-notify. What I'd still like to collect is other people's version of the same story: a notification that fired for the wrong reason, or stayed quiet for the right one. That's usually where the interesting bug is hiding — in the signal you trusted without reading twice.

Every Field Was Correct. It Still Wouldn't Connect.

Jerome — Thu, 23 Jul 2026 05:13:28 +0000

The VPN I share with a few other people ran out of traffic for the month. I had an idle OCI ARM instance sitting there doing nothing, so I decided to stop waiting and build my own node: one VLESS + Reality endpoint, self-hosted, good enough for personal use through Clash Verge. I drove the whole thing with Claude — chatted through the plan, had Claude Code do the Docker deploy, then went back to chat to walk the panel config step by step. The build was an afternoon. The debugging was the rest of the evening, and every minute of it was spent on failures where the configuration was, provably, correct.

That's the part worth writing down. Not the happy path — the four traps, and especially the last one, where every field matched on every screen and it still refused to connect.

Standing up the panel

The management panel is 3x-ui, run as a container:

docker run -d \
  --restart unless-stopped \
  -v /etc/x-ui:/etc/x-ui \
  -p 443:443 \
  -p 2096:2096 \
  -p 54321:2053 \
  ghcr.io/mhsanaei/3x-ui:latest

Three things here are not optional. Persist /etc/x-ui to the host, or every container recreate wipes your config. Set a restart policy (unless-stopped or always) so the node comes back on its own after a reboot — a proxy you have to SSH in and restart by hand is a proxy you'll abandon. And map only the ports you actually use with -p: the node (443), the subscription port (2096 here), and the panel. After it's running, docker ps and confirm every one shows up in the PORTS column.

The moment it's up, before anything else: log in and change the default credentials. This panel is exposed to the public internet, and default logins are exactly what scanners look for. While you're in there, move the panel off its default 2053 to a random port — that's the 54321:2053 mapping above — and consider changing its base access path. None of this is real security on its own; it just takes you out of the easy-target bucket.

Two firewalls, or nothing works

On a normal VPS this is one step. On OCI it's two, and missing either one gives you the identical symptom — a plain connection timeout — which makes it genuinely hard to tell which layer is at fault.

OCI's own network layer. In the console, add ingress rules to the instance's Security List (or NSG) for every port you need: TCP, source 0.0.0.0/0.
The instance's host firewall. Ubuntu's ufw is separate and just as real:

sudo ufw allow 443/tcp     # the node
sudo ufw allow 2096/tcp    # subscription port, etc.
sudo ufw reload

Open one and not the other and you'll spend twenty minutes curl-ing from the wrong side of the problem. Open both first.

The inbound — and the button that rewrites your config

Create a VLESS inbound on port 443 (blending in with ordinary HTTPS). Most defaults are fine. Two tabs need care.

⚠️ Trap 1 — the Generate button has a side effect. The Protocol tab has Decryption/Encryption fields; for standard VLESS both should read none. But clicking Generate to create the Reality X25519 key pair also silently overwrites those two fields with an experimental post-quantum encryption string — something like mlkem768x25519plus.native.600s.... This is a newer Xray feature, and Mihomo (the core inside Clash Verge) can't parse it; it errors out with invaild vless encryption value (yes, misspelled that way in Mihomo itself). The fix: after you generate the Reality keys, go back and manually set Decryption and Encryption to none. And don't click Generate again afterward — it re-overwrites the fields and regenerates your key pair, which just adds a fresh variable.

The decoy site that's too big

On the Security tab, set the security type to Reality and pick a camouflage target — a real TLS 1.3 site your handshake will impersonate. Set uTLS to chrome; leave short IDs and SpiderX auto-generated.

⚠️ Trap 2 — some targets are literally too large. With www.microsoft.com as the target, every parameter matched and the keys were confirmed identical, but the server log said:

REALITY: processed invalid connection ... handshake did not complete successfully

This isn't your config. Xray-core reads the target's TLS Certificate record during the Reality handshake and rejects anything over a hardcoded 8192 bytes. www.microsoft.com returns a record around 8273 bytes — just over the line — so the handshake dies. It's a known, reproduced upstream limit (Xray-core #6356, #6402, discussion #6387). The fix is to pick a target with a smaller certificate. www.cloudflare.com works, and the error vanished the instant I switched.

The client and the subscription that Clash refuses

On the Add Client page, attach the client to the port-443 inbound you just made, or it's linked to nothing. Then, on the Credentials tab, change Flow from its default None to xtls-rprx-vision to match the server's Reality setup — leave it at None and the connection behaves inconsistently or fails outright.

⚠️ Trap 3 — Clash needs YAML, not the default link. 3x-ui's default subscription link (http://ip:port/sub/<id>) returns the generic Base64 format that clients like V2RayNG expect. Paste that into Clash Verge and you get:

the remote profile data is invalid yaml: invalid type: string, expected a YAML mapping

Clash/Mihomo needs a YAML subscription, which 3x-ui serves from a different path. In panel settings, confirm the Clash/Mihomo subscription is enabled, check its URI path (usually /clash/), and use:

http://server-ip:subscription-port/clash/<subscription-id>

Import that under Profiles → New → Remote. Only this path returns parseable YAML.

The trap that ate the evening

Subscription imported, node listed, everything green. Every dial failed:

[TCP] dial ... error: server-ip:443 connect error: REALITY authentication failed

We worked it in order — I say "we" because Claude was driving the diagnosis the whole way:

nc -zv server-ip 443 from my laptop — TCP connects. Network's fine.
curl from the server to the Reality target — clean 200. Target's reachable.
docker exec ... netstat -tlnp | grep 443 — the port is listening inside the container.
Then the careful part: read the actual running config.json from inside the container and compare Public Key, Short ID, SNI, UUID, and Flow against the panel UI and against the subscription Clash was consuming. Field by field.

Everything matched. An assistant that will happily read every value in every file had just confirmed all of them correct — and the connection still failed. The thing that was wrong wasn't in any file.

The answer was the version of the running binary. 3x-ui v3.5.0 ships Xray-core v26.7.11, and that release is deliberately incompatible with Mihomo — a documented breaking change (a new default minClientVer), not a local misconfiguration. Multiple people reproduced the exact REALITY authentication failed symptom on v3.5.0, and Mihomo has stated outright it won't support Xray v26.7.11+ (3x-ui #5957, #5922; Xray-core #6477). The fix is to downgrade the core to v26.6.27. Recreating clients, regenerating keys, rebuilding the inbound — none of it matters, because none of it touches the version.

There was one more layer to it, and it's the reason the fix looked like it didn't work. After downgrading, the error was still there. What I hadn't accounted for: the panel applies changes through Xray's core API and reports config changes applied through the core API, no restart needed. That hot-reload path does not swap the running Xray process. The old v26.7.11 binary was still serving traffic. A full docker restart 3x-ui loaded the downgraded core, and it connected immediately.

So the two facts sit together. The fix was the downgrade. It only took effect on a real process restart — because the panel's "no restart needed" is true for config edits and quietly false for the one thing that actually mattered.

What I'd tell myself at the start

When every file says the config is correct and it still fails, stop re-reading the files. The mismatch you can't see is the process, not the config: what version is actually running, and did it actually reload. The panel telling you "no restart needed" is a claim about config edits, not a guarantee about the process — so after any change that matters, restart the container and check the running core yourself.

A short checklist, in the order that would have saved me the evening:

nc -zv server-ip port — reachability from the client.
Confirm both the OCI security list and the host ufw allow the port.
docker exec ... netstat -tlnp — the port is really listening inside the container.
Decryption/Encryption are none, not the mlkem768... post-quantum string.
Camouflage target has a small certificate — avoid www.microsoft.com; www.cloudflare.com is safe.
Client Flow is xtls-rprx-vision; subscription is the /clash/ YAML path, not /sub/.
Check the running Xray-core version. If it's v26.7.11 and your client is Mihomo/Clash Verge, downgrade to v26.6.27.
After any change, docker restart the container — don't trust "no restart needed" — then retest.

The config was never the hard part. The hard part was trusting, for one evening too long, that a correct file meant a correct process.

The Bug That Kept Coming Back

Jerome — Wed, 15 Jul 2026 06:41:27 +0000

The first sign something was wrong wasn't a crash. It was a pattern.

blockly-platform was the first real thing I built with Claude Code end to end — a Blockly-based platform for university programming exercises, driven entirely through Claude Code's Telegram channel. No editor open, no repo checked out on my machine, just a chat thread. I'd describe what I wanted, Claude Code would build it on a box I never looked at directly, and I'd judge the result by clicking around the deployed app.

On March 22nd, the home page came up empty. GET /api/exercises/published was returning 403. I said so in the chat; a few messages later, Claude Code said it was fixed — the endpoint hadn't been added to Spring Security's permitAll() list. I moved on, tried the category filter. Also empty, also 403, also missing from the same permitAll() list — same file, same class of fix, different line. Then the exercise detail page. Same story, third time, same day. Three days later, the like button stopped working — root cause, again: POST /api/exercises/*/like had never been whitelisted either. Four times, one file, one recurring gap.

None of these were hard bugs. Each one, in isolation, is a one-line fix a competent engineer makes without thinking twice. What bothered me, once I noticed the pattern, was that I hadn't noticed it as it happened. I had no diff to scroll through, no file to glance at and think "wait, didn't we just fix this exact class of thing twice already?" I had a chat log and a live app to poke at. The fourth fix looked, from where I sat, exactly like the first: a message telling me it was resolved.

That was the moment I started to suspect the problem wasn't the model. It was that nobody — not the model, not me — had anything to look at.

Why chat-only vibe coding breaks down

Here's what makes that pattern more interesting than "the AI made a mistake": every one of those four fixes was correct. Claude Code read the error, found the missing permitAll() entry, added it, and moved on — each time, in isolation, exactly the right diagnosis and exactly the right fix. Grade any single one of those four turns on its own and it passes.

What failed wasn't a single decision. It was the absence of anything connecting the four decisions to each other. A human engineer touching SecurityConfig.java for the third time in a week, over the same kind of hole, stops and thinks "let me just check the whole file" — not because they're smarter than Claude Code, but because they've been there before and remember it. Chat-only, Telegram-driven vibe coding doesn't give either party that continuity. Claude Code sees the current turn's error and fixes the current turn's error. I see a message saying "fixed" and move to the next thing on my list. Neither of us is holding the shape of the file over time, because neither of us is looking at the file — I'm looking at a phone screen, and the model's context is whatever's in that turn's window, not an accumulated sense that this file has a recurring problem.

That's the first layer. The second layer worried me more, because it doesn't have an obvious fix inside the chat-only model at all: some of Claude Code's decisions came back to me to approve. But I hadn't read the code structure — I'd never opened the repo, never seen SecurityConfig.java, never seen how exercises, submissions, and grading fit together. So when a decision did land in my lap, I was approving it with roughly the same information the model had: none, beyond the current message. "Human in the loop" only adds oversight if the human has something the model doesn't. Mine didn't.

To be clear about the boundary of this claim: none of it means chat-only driving is always wrong. For something small and throwaway, it's fine — there's nothing to lose track of. The failure mode shows up specifically as complexity and stakes climb, once a project has enough moving parts that "the shape of the file over time" is actually worth tracking.

The repo backs this up in ways that aren't just my memory of it. blockly-platform went from an empty repository to a merged, working platform in 80 commits over 23 days — fast, and it shows. There's no design doc anywhere in it, no architecture note, nothing that records why a decision was made — only a RUNBOOK.md, a TODO.md, and an ERROR_LOG.md that reads like a list of things discovered after the fact. A CLAUDE.md — the file that would have told Claude Code the project's own rules, things like "don't forget the security config when you add an endpoint" — didn't exist until April 2nd. Every dated entry in ERROR_LOG.md, all eighteen of them, including all four permitAll() misses, falls between March 22nd and March 28th. The guardrails arrived after the bugs they would have prevented, not before them.

The rebuild

The project that replaced it, programming-learning-platform, started the same day blockly-platform's pull request merged. That's not a coincidence I engineered — it's genuinely where the first project ended and the lesson from it started applying. It isn't a separate idea, either: its own CLAUDE.md describes it as a platform for Blockly and Python exercises, the same tool rebuilt from the ground up rather than a new one started from scratch.

This time I changed two things at once. First, visibility: I moved to IntelliJ IDEA's remote development mode, with the project checked out and running on the same kind of remote box as before, but now with an actual editor attached to it — I could see the file tree, open a class, read a diff before it landed. Second, process: I turned on the superpowers skill and wrote it into the project's CLAUDE.md as non-negotiable — "Every task: Brainstorm → Plan → Implement (TDD). No skipping brainstorm. No code without a plan. No implementation before a failing test." — with "No skipping Superpowers" listed again under the project's explicit red lines, in case that wasn't clear enough the first time.

The difference shows up in the repo's shape, not just in my memory of how it felt. programming-learning-platform is 458 commits deep and still active more than three months later. It has a real docs/ tree: a PRD, per-role user flows, an architecture doc, feature specs split by priority — and a docs/superpowers/plans/ directory holding forty-four dated implementation plans, running from within its first two weeks to earlier this month, each one a record of a brainstorm-and-plan step that happened before the corresponding code did. Where blockly-platform's only paper trail was a log of things that had already gone wrong, this one has a paper trail of decisions made on purpose, in order, before they were implemented.

It worked, in the sense that mattered most: I stopped approving things blind, because I finally had a diff to actually read and a plan to check the diff against.

It also had friction I didn't expect, and I want to be precise that this is my own experience with a specific setup at a specific time, not a verdict on the products themselves. IntelliJ IDEA's remote development client, even with the actual computation happening entirely on the server, still felt heavy on my local machine — noticeably more than I expected from something meant to be a thin window onto remote work. And the JetBrains Claude Code plugin had gaps that got in the way of the one thing visibility is supposed to buy you: clear communication. I couldn't attach an image to a message — no screenshot of a broken layout, no annotated diagram — so some things I could have shown in five seconds took several paragraphs to describe instead, with more room for the description to be wrong.

Visibility and process had fixed the real problem. The tool I'd wrapped them in was still fighting me.

The setup that stopped fighting me

A friend who'd been running a similar remote setup suggested I drop IntelliJ for VS Code partway through the project — specifically, VS Code's Remote-SSH extension to manage programming-learning-platform directly on the remote box, with the Claude Code extension running against that same connection, instead of a JetBrains-specific remote protocol and its own plugin.

Nothing about the visibility or the process changed, and that's the point: the file tree was still there, diffs still landed in front of me before I approved anything, and superpowers' brainstorm-plan-TDD loop carried over untouched, because it lives in the project's CLAUDE.md, not in whichever editor happens to be open. What changed was the tool wrapped around both of those things. The client was lighter — it stopped being noticeably heavy on my local machine the way IDEA's had been. And the Claude Code extension let me attach a screenshot directly to a message, which matters more than it sounds for a platform built around a visual block editor and a grading UI: a broken Blockly toolbox or a misaligned grading panel is a five-second screenshot and a two-word question, not three paragraphs trying to describe a layout in words.

I've written before about the mechanics of why this specific combination — VS Code's Remote-SSH plus the Claude Code extension — is worth trusting with long-running, walk-away-able work: the process survives your laptop sleeping, and the compute never touches your local machine. I won't re-run that ground here. What matters for this piece is narrower: this was the first setup where visibility, process, and the tool itself all pulled in the same direction, instead of two out of three fighting the third.

The last mile

With visibility and process both solid, the last piece was the one that let me actually step away: knowing when Claude Code needed me, and being able to answer without being at my desk.

The when is claude-code-notify, a tool I built after getting tired of either babysitting a terminal or discovering twenty minutes later that a long TDD-driven turn had been sitting blocked on my sign-off the whole time. It pings me on Telegram at exactly the moments that matter — a turn is genuinely finished, background work included, or it needs my input, or it errored out. The how is Remote Control (/remote-control), which lets me approve prompts and answer questions from the Claude app on my phone instead of reopening a laptop. Get pinged that Claude needs me, answer from my pocket — that's the workflow, and I've written separately about the one way it can quietly fail if the underlying remote session doesn't survive long enough to still be there when you answer.

It's worth being precise about what this layer is and isn't. It isn't what fixed the permitAll() problem — visibility and superpowers' plan-before-code discipline did that. What notify and Remote Control add is the ability to be away from the keyboard without losing either one: I still see the diff, Claude Code still follows the plan, I just don't have to be sitting there for either to keep happening. Bolt the same walk-away layer onto the Telegram-only setup from the first project, and it wouldn't have fixed anything — you'd just get notified faster that a decision you couldn't evaluate had already been made. The order matters: visibility and process first, walk-away convenience on top of that foundation, not instead of it.

The bug, one more time

I sometimes wonder what would have happened if the fourth permitAll() miss — the one on the like button — had happened under the later setup instead of the first one. My honest guess: it still would have gotten written, because forgetting a security rule on a new endpoint is an easy thing for anyone, human or model, to do once. But it wouldn't have taken three prior occurrences to catch, because a plan step would have said "add the endpoint and its security rule" as one unit, and a diff would have shown both halves — or shown only one, right there, before I approved it. The bug wouldn't have needed a human to notice a pattern across a week of chat messages. It would have needed a human, or a model, to notice a gap in one diff — a much smaller ask, and one that visibility actually makes possible.

That's the whole shift, underneath all the tool names. Pure chat-only vibe coding doesn't fail because the model is bad at its job — every individual permitAll() fix in blockly-platform was correct. It fails because past a certain size, nobody involved, model or human, has enough in front of them to notice the second time a mistake happens, let alone stop it before the fourth. Visibility and a process that makes you plan before you build don't make Claude Code smarter. They give whoever's watching, model or human, something to actually watch.