This started with a k3s install and not enough memory to do it in. I shut down another project's stack by hand, which freed up enough to get k3s running. k3s loads br_netfilter when it starts, and once the module was in place a kernel switch that applies to the whole host changed state. Then I brought the stopped project back, and its containers had lost the ability to reach each other. That's its own story. This one is about what I tried next — an attempt that did nothing for that problem and quietly broke a different one.
What I tried was a docker daemon restart: take the engine down, bring everything up clean, see whether the failure came back with it. It came back. The restart's real effect landed somewhere else entirely — every container on a different network, one I hadn't touched all day, came up holding an address it hadn't had before. Two of them matter here: the reverse proxy standing in front of every service on the box, and a proxy service standing in front of the reverse proxy. Neither had a static address pinned at the time, so each got whatever docker dealt it on the way up. That's where this story actually starts.
That second proxy is worth explaining before anything else, because the whole investigation turns on it. Every service on the box sits behind the reverse proxy's access rule, which admits exactly two sources and denies everything else. The one exception is the proxy service itself — call it the gateway from here — which exists specifically as a security control: connect to it first, and it's the only thing standing between the public internet and everything else running on the box. That's not how the wiring happened to end up. It's what the wiring was built to do — turn a set of publicly reachable services into something that behaves like a private intranet, gated behind one deliberately unrestricted entry point.
The allow-list, read right
After the restart I reported the symptom as precisely as I could see it: almost everything behind the reverse proxy had gone unreachable, while the gateway's own entry point still answered fine. Was it the access rule, the server's address, or a container's address that had moved?
The agent went straight to the reverse proxy's own configuration — the per-hostname files it writes one of for each gated service. Every one of them carried the same rule: allow an address on the shared docker network, allow the VPS's own public address, deny everything else. Read against the symptom, that rule explains all of it in one pass. Those two addresses are the only two ways in; the gateway's own front door carries no such rule at all. If either of the two allowed addresses had moved, everything gated goes dark while the entrance keeps answering exactly as it did.
That's the right read, and it came fast — one config, one pattern repeated on every gated host, matched cleanly against a symptom that had more than one plausible cause before this. What it didn't yet answer was the identity of the two addresses inside the rule. That's not a gap in the diagnosis. It's the next question.
The address that matched
One of the two addresses was easy: the VPS's own public IP, 203.0.113.10 — the address anything outside the docker network shows up as. The other was a docker-network address, 172.19.0.2, inside the same subnet the reverse proxy and the gateway both sit on. The natural next question was whose address that was, and there was a fast way to answer it: check what's actually running there right now.
It checked. The reverse proxy's own container, at that moment, was sitting at exactly 172.19.0.2. Address in the config, address on the wire, same value. That's not a weak signal — a live process answering at the address in question is about as direct a confirmation as a check like that produces.
Except the container's address wasn't fixed to anything. Nothing had pinned it there; it was wherever docker's own address allocation had put it after the restart that shuffled this whole network in the first place. The match wasn't the allow-list describing the reverse proxy. It was two independent things landing on the same value at the same moment — a rule that had been sitting there, and a container that had landed on that exact address by whatever the restart's allocation did with it. Nothing forced them to agree. They just did, right then, and there was no reason yet to ask why.
There was no step left undone here in any obvious sense. Check the config, check what's live, they match — that's exactly the kind of confirmation a live system is supposed to give you. The process had a shape that quietly assumed the docker address named something, and it never had to ask what the address meant, only what currently answered there. Something always answers there. That question doesn't have a way to come back negative.
What else got checked, and cleared
The agent made two more calls during the same pass, in the other direction — flagged, then walked back, both times before either one turned into an action.
The first was the VPS's own public egress address, spotted somewhere it hadn't been expected and read, on sight, as somebody's home or office connection — a plausible guess, since a bare public IP with no label attached could belong to anyone. Instead of accepting that, the agent checked: it asked a container on the host what its own outbound address looks like, then asked the host the same question, and got the identical value both times. Separately, the reverse proxy's own access log already had that exact address on file, showing up once a minute, every minute, as the source of a self-monitoring probe that had been running and been let through the whole time. An address that shows up once a minute forever isn't someone's laptop. It's the box talking to itself.
The second was a running process carrying the name of a well-known security tool, with nothing in any of the host's package managers recording it as installed and no binary sitting at the path a real install of that tool would use. On those two facts alone, flagging it as a possible intrusion was the right call to make out loud — an unexplained process wearing that name is exactly what a compromise can look like. It didn't survive the next check. Resolving the process's /proc/<pid>/exe link led to an ordinary, already-installed script interpreter, and its parent process was the gateway application's own binary: the tool ships bundled inside the gateway software itself, which is also exactly why no package manager had ever heard of it.
Two calls, two reversals, each one on a specific checkable reason rather than a change of heart. The habit of checking was doing exactly what it does throughout this investigation — catching a coincidence in one place, an unlabeled address in another, a suspicious name in a third, and running each one down before it became a decision. None of that checking was pointed at the one address that mattered. It didn't fail there. It was never aimed there.
Written down looks decided
The address match from two sections back didn't stay a working theory. Treating the docker-network address in the allow-list as the reverse proxy's own, the agent pinned the reverse proxy's compose file to that exact address — turning a coincidence into a static declaration — and reported the fix as applied.
That's the expensive step, and it's expensive for a specific reason. Everything before it was reasoning, held loosely, revisable the moment new evidence showed up. A pin in a compose file isn't reasoning anymore. It's a written commitment, sitting in the same file that decides how the container actually starts, and it reads to anyone who looks at it afterward — including the agent itself, on the next question — as a decided fact rather than a step that happened to follow from an assumption nobody had checked. The report went out the same way: fixed, not probably fixed.
An answer with a number in it
I read the fix and didn't accept it. Traffic through the gated services comes from the gateway — that's the reason the gateway exists in the shape it does — so the address pinned in the allow-list should be the gateway's, not the reverse proxy's. I wasn't guessing on the spot. I was remembering how I'd built the system in the first place and laying that against what the report actually said, and the mismatch was immediate: pinning the reverse proxy's own compose file to the reverse proxy's own address didn't match anything about how traffic through the gateway was supposed to arrive.
The agent didn't concede the point. It tested it.
A container already sitting on the shared docker network requested the public hostname directly, the same way any client on the internet would, and the reverse proxy's access log was read for what showed up as the source. The answer: the VPS's own public address, not the container's own docker-network address. Whatever a container on that network sends toward the public domain, the reverse proxy sees it arrive from outside — which meant pinning the gateway's specific address, on top of what was already pinned, shouldn't change anything either way.
That's a real result. The container really did hit the real hostname, the log really did show the public address, and nothing about the number was wrong. It answered a question — does traffic to the public domain from an arbitrary container on this network show up carrying that container's own address? — cleanly, and the answer was no. What it didn't answer, because nothing about the test asked it to, was whether the gateway's own traffic took that same path at all. A generic container resolving a public hostname the ordinary way, and the gateway forwarding through whatever its own proxy engine does with that hostname, are two different roads that happen to look identical from a compose file. The test measured the first one and reported on the second.
It read as the stronger answer of the two, not the weaker one. My objection was a sentence about how the system was designed to work. The agent's reply was a live measurement, run and read back on the spot. Between an argument from memory and a number from a test just run, the number wins the room every time — right up until someone checks what the number was actually a number of.
The fact that wasn't on the machine
I supplied the piece that neither the config nor the test could produce: the allow-list hadn't always carried two addresses. When I first wrote it, it carried one — the docker-network address — and that was enough. It worked.
Lay that fact next to the agent's test and the test breaks. If a request from anywhere on the docker network to the public hostname always arrives at the reverse proxy carrying the VPS's own public address, no exceptions — the test had just shown exactly that — then an allow-list with only the docker-network address on it, and nothing granting the public one, could never have let a single request through. Not once. But it had. For however long the allow-list ran with just that one line, something was reaching the reverse proxy from inside the docker network, under its own docker-network address, without ever picking up the public one on the way. The test said that path didn't exist. The system's own history said it always had.
There's no config file this comes from, no log line, nothing on the box today that records what the allow-list looked like before the second address went in. It isn't derivable from the server's current state — the state has already moved past it, twice over, once when the address was added and again when the restart reshuffled everything on top of that. It exists in exactly one place: my memory of building the thing. And it's what turns the investigation around — not a new check run against the server, but an old fact about the server that the server itself was never going to hand back.
The DNS override, and the address it always pointed at
The agent read the proxy engine's own configuration directly — not inferring from what a generic client did, but reading what the gateway's own software was actually told to do — and that settled it. The proxy engine carries a DNS override for the entire domain: every hostname under it resolves, inside the gateway, not to whatever the internet's DNS would return, but straight to a fixed address on the shared docker network.
That's the road the generic test container never took. An ordinary client resolves the public hostname through public DNS and lands on the VPS's public address, same as the reverse proxy expects from anyone outside. The gateway, for that one domain, never asks public DNS at all — the override answers first, and the request goes straight to a docker-network address without leaving the host. A connection that never leaves the host is never rewritten to look like it came from the host's own public interface. It arrives at the reverse proxy carrying the gateway's real docker-network address, not the VPS's public one — which is exactly the address the allow-list's other line was written for, and exactly what a request from an arbitrary container, going the long way through public DNS, was never going to produce.
The other half of the fault was on record from the start of this: neither the reverse proxy's container nor the gateway's had a static address pinned anywhere. The docker daemon restart that kicked this whole thing off reassigned both, along with everything else on that network, to whatever addresses happened to be free at the time — and a third container, unrelated to either of them, ended up sitting on the one address the proxy engine's override had always pointed to. That's what actually broke: not the allow-list, not the reverse proxy, not the override itself, but which container each address named on any given restart.
The fix pinned both: the reverse proxy to 172.19.0.3, the gateway to 172.19.0.2, each written into its own compose file. The third container moved off the address the override needed and onto a free one. Three containers restarted to pick up the change. Not the daemon — restarting that again is how this started.
Verifying it didn't call for trusting the compose files. It called for reproducing the actual path, so the agent did: from inside the gateway container, it issued a request following the same forwarding the proxy engine's override would take — straight to the reverse proxy's pinned docker-network address, bypassing public DNS the way the gateway itself does. The request succeeded, and the reverse proxy's own access log recorded the source as 172.19.0.2 — the gateway's own address, logged by the one component in the chain that was never asked to guess what it meant.
Nowhere in writing
None of this — the gateway as the deliberate front door, the docker-network address in the allow-list as always meant for it, the allow-list's own history before the public address got added — existed in any file on the box. Not in a compose file, not in a comment, not in the proxy engine's own configuration, which describes what it does but not why. The agent had the server's current state and nothing else to check it against, and current state doesn't distinguish a coincidence from a decision. Two things landing on the same address at the same moment reads identically to one thing being pinned there on purpose, right up until someone who was there says otherwise.
That's what made the wrong pin possible, and it's also what nearly made the wrong pin permanent. The agent's own test wasn't sloppy — it measured something real and reported it accurately. What turned the investigation was me declining an answer I had reason to doubt, on grounds the agent's evidence couldn't reach, and then handing over the one fact that reached back further than any log on the machine: what the allow-list looked like before there was a reason to add a second line to it.
The compose files carry the pin now. The design intent behind it — the reason 172.19.0.2 belongs to the gateway and not to whatever last happened to sit there — exists in exactly one place outside my own memory: a page of reference notes about this host. It wasn't there when any of this started. The record came after the repair, because the repair is what produced it.
Top comments (0)