The three-line config I shelved, and the three-hour outage it became

For eight months, my k3s cluster did exactly nothing wrong.

I'd installed it on a Hetzner VPS at the end of August: k3s, single node, the whole Grimoire stack on top (Postgres, Mongo, Elasticsearch, Keycloak, the backend). And it's not like I left it alone. I was in there all the time: reorganizing stacks, pulling pieces out and dropping new ones in, redeploying. The workloads churned constantly.

What I never did, in eight months, was fully power the node down. The pods came and went; the host underneath, and the k3s server process on top of it, just stayed up. That distinction is the whole story. I just didn't know it yet.

It didn't help that the box has 7.6 GB of RAM and zero swap, and was running like this:

Service	RAM (approx)
k3s (apiserver, etcd, controllers)	~1.2 GB
Elasticsearch	~1.5–2 GB
Kibana	~500 MB
Keycloak	~700 MB
PostgreSQL × 2	~400 MB
MongoDB	~300 MB
Backend (Spring Boot)	~500 MB
Traefik + cert-manager + system	~400 MB
Total	~5.5–7 GB

So the cluster wasn't stable. It was sitting at the edge of its memory the entire time, and nothing had pushed it over yet. Eight months of uptime wasn't proof it was solid. It was just proof nothing had asked it a hard question.

It also helped that I didn't have much riding on it. This was a staging cluster at the time: a sandbox I stood up to learn on, not production, not my paycheck, not something I was on call for. I never really worried about it going down, and that's exactly why I treated it the way I did: no swap, no alerts, the odd config note left for "later." Casually bolting a heavy metrics stack onto an already-full box was the same reflex, the kind of thing you do without much thought to a machine you're not afraid to break.

The morning it stopped being quiet

There was no monitoring on this cluster, which is half the joke: the thing that finally tipped it over was me adding some.

I'd just put the observability stack in: OpenTelemetry tracing on the backend, and the Grafana side in-cluster (Prometheus, Tempo, Loki). Useful, and also heavy, on a box already sitting near its ceiling. Memory went straight into the red. The only real lever I had to claw it back was Elasticsearch: by far the biggest single consumer, and the one thing I could shrink without dropping a feature. So I lowered its limits and went to restart it with the new values.

That's when I found the cluster was already on the floor. My kube context kept dropping (commands hanging, the connection to the API server falling over and over), so I SSH'd into the box to look underneath. k3s was stuck in activating (auto-restart), looping. free -m told the rest:

               total        used        free      shared  buff/cache   available
Mem:            7634        7449         185          22         312         142
Swap:              0           0           0

185 MB free, no swap to fall back on. The OOM killer had got there before I did, and one of the processes it had reached for was k3s itself. Not a pod, the server.

I worked the rest of this out with an LLM open the entire time. I'm not a sysadmin, this is a layer I still know badly, and I navigated a good chunk of it by sight: pasting errors in, getting a direction, checking it against the actual state of the box. I learned a lot in three hours. I also guessed wrong more than once.

The OOM, anyway, was the easy part. It was the trigger, not the whole outage. And a crash loop is k3s trying, over and over, to do the one thing it hadn't done in eight months: start cold. That's what was about to drag everything else into the light.

Pulling it apart, one layer at a time

The token was gone. k3s wouldn't come back, complaining it couldn't read its server token. I checked: /var/lib/rancher/k3s/server/token, zero bytes. Empty. Best guess: an OOM kill landed mid-write during the crash loop and truncated it. k3s flatly refuses to start without that file. It was recoverable from the passwd credential file sitting right next to it. But I only know that now. At the time nothing told me, and a wiped token on a single-node cluster is the kind of thing that makes you briefly wonder if you're about to rebuild everything from scratch.

Then IPv6 woke up. Token restored, k3s tried to start, and threw a new error:

dial tcp [::1]:6444: connect: connection refused

This is the one I'd been warned about. Back at install I'd half-registered that Hetzner always hands you dual-stack, IPv4 and IPv6, and that k3s wants it configured explicitly. I'd filed it under yeah, I should deal with that at some point, written nothing down, and never did. Eight months later, here it was.

What happens is: /etc/hosts maps localhost to both 127.0.0.1 and ::1, but the apiserver was only bound to 127.0.0.1:6444. So every internal component reaching for localhost:6444 ended up on [::1]:6444 and got refused.

And it had been misconfigured the entire eight months. It never surfaced because the apiserver process had never restarted: as long as k3s was already up, nothing re-resolved that address. The bug needed a cold start to exist, and I'd never given the server one.

My first fix made it worse: binding to ::1 got the apiserver to pick the public IPv6 as its external host, which then clashed with the IPv4 service CIDR 10.43.0.0/16. The one that actually held, in /etc/rancher/k3s/config.yaml:

kube-apiserver-arg:
  - "bind-address=::"                    # listen on both stacks
  - "advertise-address=<public-ipv4>"    # force IPv4 external host → matches the service CIDR

Three lines. That's the config from the title, the thing I'd shelved at install. Three lines, and the cluster would have cold-booted clean any day in those eight months.

And the zombies. Every failed restart in the loop had left an orphaned containerd-shim behind. By now there were fifteen-plus, jamming the next start. No clever fix: a full VPS reboot cleared them, and on the way back up kubectl get nodes and kubectl get pods -A finally came back clean. Roughly three hours, end to end.

The whole chain, start to finish:

When	What happened
Aug 2025	k3s installed: dual-stack, no swap. Two latent bugs ship with it, unnoticed.
Aug 2025 → May 2026	Eight months up. Pods churn constantly; the server process never restarts.
May, ~07:00	Observability stack added → memory into the red → OOM kills the k3s process → crash loop.
	Server token truncated to 0 bytes mid-write → restored from the passwd credential file.
	First real cold start exposes the dormant IPv6 dual-stack bug → fixed in config.yaml.
	15+ zombie containerd-shim processes block restart → full VPS reboot.
~10:00	kubectl get nodes returns clean. ~3 hours total.

Why eight quiet months were the actual cause

Step back and the OOM is almost a distraction. The two things that turned a process restart into a three-hour outage (the undocumented token recovery and the IPv6 binding) were both there on install day, in August. Neither was introduced by the crash. They were just invisible until something forced the k3s server to start cold.

Both lived in the same place: the gap between the k3s server going fully down and coming back up. Everything I'd done over those months happened above that line (pods cycling, stacks rearranged, Elasticsearch resized), and the server underneath never flinched, so neither bug ever showed. I never restarted it on purpose to check, because it never crossed my mind to; there was simply no question in my head to answer. One deliberate restart, or one reboot, on any calm afternoon would have surfaced both cold, with nothing on the line. Which is more or less what a staging box is there for. I just hadn't thought to use it that way.

What it left behind

As usual, it was the kind of bug-chase that's annoying the whole way through and satisfying the moment it gives.

It also left the system in better shape than it found it. The box has swap now, which covers the host and the k3s process itself. The pods still don't swap. The k3s token is backed up in Infisical. And the whole chain (OOM, token, IPv6, zombies) is written up as a post-mortem with recovery procedure, so if any of it comes back it's a checklist instead of three hours of guessing.

That same box runs production now, so it's a good thing all of this came out while it was still a sandbox.