DEV Community: Théo Verdiere

The three-line config I shelved, and the three-hour outage it became

Théo Verdiere — Tue, 20 Oct 2026 12:00:00 +0000

For eight months, my k3s cluster did exactly nothing wrong.

I'd installed it on a Hetzner VPS at the end of August: k3s, single node, the whole Grimoire stack on top (Postgres, Mongo, Elasticsearch, Keycloak, the backend). And it's not like I left it alone. I was in there all the time: reorganizing stacks, pulling pieces out and dropping new ones in, redeploying. The workloads churned constantly.

What I never did, in eight months, was fully power the node down. The pods came and went; the host underneath, and the k3s server process on top of it, just stayed up. That distinction is the whole story. I just didn't know it yet.

It didn't help that the box has 7.6 GB of RAM and zero swap, and was running like this:

Service	RAM (approx)
k3s (apiserver, etcd, controllers)	~1.2 GB
Elasticsearch	~1.5–2 GB
Kibana	~500 MB
Keycloak	~700 MB
PostgreSQL × 2	~400 MB
MongoDB	~300 MB
Backend (Spring Boot)	~500 MB
Traefik + cert-manager + system	~400 MB
Total	~5.5–7 GB

So the cluster wasn't stable. It was sitting at the edge of its memory the entire time, and nothing had pushed it over yet. Eight months of uptime wasn't proof it was solid. It was just proof nothing had asked it a hard question.

It also helped that I didn't have much riding on it. This was a staging cluster at the time: a sandbox I stood up to learn on, not production, not my paycheck, not something I was on call for. I never really worried about it going down, and that's exactly why I treated it the way I did: no swap, no alerts, the odd config note left for "later." Casually bolting a heavy metrics stack onto an already-full box was the same reflex, the kind of thing you do without much thought to a machine you're not afraid to break.

The morning it stopped being quiet

There was no monitoring on this cluster, which is half the joke: the thing that finally tipped it over was me adding some.

I'd just put the observability stack in: OpenTelemetry tracing on the backend, and the Grafana side in-cluster (Prometheus, Tempo, Loki). Useful, and also heavy, on a box already sitting near its ceiling. Memory went straight into the red. The only real lever I had to claw it back was Elasticsearch: by far the biggest single consumer, and the one thing I could shrink without dropping a feature. So I lowered its limits and went to restart it with the new values.

That's when I found the cluster was already on the floor. My kube context kept dropping (commands hanging, the connection to the API server falling over and over), so I SSH'd into the box to look underneath. k3s was stuck in activating (auto-restart), looping. free -m told the rest:

               total        used        free      shared  buff/cache   available
Mem:            7634        7449         185          22         312         142
Swap:              0           0           0

185 MB free, no swap to fall back on. The OOM killer had got there before I did, and one of the processes it had reached for was k3s itself. Not a pod, the server.

I worked the rest of this out with an LLM open the entire time. I'm not a sysadmin, this is a layer I still know badly, and I navigated a good chunk of it by sight: pasting errors in, getting a direction, checking it against the actual state of the box. I learned a lot in three hours. I also guessed wrong more than once.

The OOM, anyway, was the easy part. It was the trigger, not the whole outage. And a crash loop is k3s trying, over and over, to do the one thing it hadn't done in eight months: start cold. That's what was about to drag everything else into the light.

Pulling it apart, one layer at a time

The token was gone. k3s wouldn't come back, complaining it couldn't read its server token. I checked: /var/lib/rancher/k3s/server/token, zero bytes. Empty. Best guess: an OOM kill landed mid-write during the crash loop and truncated it. k3s flatly refuses to start without that file. It was recoverable from the passwd credential file sitting right next to it. But I only know that now. At the time nothing told me, and a wiped token on a single-node cluster is the kind of thing that makes you briefly wonder if you're about to rebuild everything from scratch.

Then IPv6 woke up. Token restored, k3s tried to start, and threw a new error:

dial tcp [::1]:6444: connect: connection refused

This is the one I'd been warned about. Back at install I'd half-registered that Hetzner always hands you dual-stack, IPv4 and IPv6, and that k3s wants it configured explicitly. I'd filed it under yeah, I should deal with that at some point, written nothing down, and never did. Eight months later, here it was.

What happens is: /etc/hosts maps localhost to both 127.0.0.1 and ::1, but the apiserver was only bound to 127.0.0.1:6444. So every internal component reaching for localhost:6444 ended up on [::1]:6444 and got refused.

And it had been misconfigured the entire eight months. It never surfaced because the apiserver process had never restarted: as long as k3s was already up, nothing re-resolved that address. The bug needed a cold start to exist, and I'd never given the server one.

My first fix made it worse: binding to ::1 got the apiserver to pick the public IPv6 as its external host, which then clashed with the IPv4 service CIDR 10.43.0.0/16. The one that actually held, in /etc/rancher/k3s/config.yaml:

kube-apiserver-arg:
  - "bind-address=::"                    # listen on both stacks
  - "advertise-address=<public-ipv4>"    # force IPv4 external host → matches the service CIDR

Three lines. That's the config from the title, the thing I'd shelved at install. Three lines, and the cluster would have cold-booted clean any day in those eight months.

And the zombies. Every failed restart in the loop had left an orphaned containerd-shim behind. By now there were fifteen-plus, jamming the next start. No clever fix: a full VPS reboot cleared them, and on the way back up kubectl get nodes and kubectl get pods -A finally came back clean. Roughly three hours, end to end.

The whole chain, start to finish:

When	What happened
Aug 2025	k3s installed: dual-stack, no swap. Two latent bugs ship with it, unnoticed.
Aug 2025 → May 2026	Eight months up. Pods churn constantly; the server process never restarts.
May, ~07:00	Observability stack added → memory into the red → OOM kills the k3s process → crash loop.
	Server token truncated to 0 bytes mid-write → restored from the `passwd` credential file.
	First real cold start exposes the dormant IPv6 dual-stack bug → fixed in `config.yaml`.
	15+ zombie `containerd-shim` processes block restart → full VPS reboot.
~10:00	`kubectl get nodes` returns clean. ~3 hours total.

Why eight quiet months were the actual cause

Step back and the OOM is almost a distraction. The two things that turned a process restart into a three-hour outage (the undocumented token recovery and the IPv6 binding) were both there on install day, in August. Neither was introduced by the crash. They were just invisible until something forced the k3s server to start cold.

Both lived in the same place: the gap between the k3s server going fully down and coming back up. Everything I'd done over those months happened above that line (pods cycling, stacks rearranged, Elasticsearch resized), and the server underneath never flinched, so neither bug ever showed. I never restarted it on purpose to check, because it never crossed my mind to; there was simply no question in my head to answer. One deliberate restart, or one reboot, on any calm afternoon would have surfaced both cold, with nothing on the line. Which is more or less what a staging box is there for. I just hadn't thought to use it that way.

What it left behind

As usual, it was the kind of bug-chase that's annoying the whole way through and satisfying the moment it gives.

It also left the system in better shape than it found it. The box has swap now, which covers the host and the k3s process itself. The pods still don't swap. The k3s token is backed up in Infisical. And the whole chain (OOM, token, IPv6, zombies) is written up as a post-mortem with recovery procedure, so if any of it comes back it's a checklist instead of three hours of guessing.

That same box runs production now, so it's a good thing all of this came out while it was still a sandbox.

Yet another personal project. I over-engineered it on purpose

Théo Verdiere — Tue, 06 Oct 2026 14:00:00 +0000

It started one evening, out of the ordinary friction of the job: the organizational and budgetary constraints that come with any system at scale, and never enough time to keep things clean and pleasant to work on.

Not a startup idea, not a side hustle with a monetization plan. Just a developer who'd spent a long time working inside systems he couldn't reshape, opening a fresh Kotlin project at 9pm because he wanted somewhere those limits didn't apply.

So yes, this is yet another personal project. The internet has no shortage of them. I knew from the start it would be oversized: a backend, an Android app, CI, infrastructure, a deployment. All for an audience of maybe six people. A bit of a champagne app on a beer budget.

The overkill wasn't the surprise; that part was deliberate. What I didn't plan was building a whole product. I started out just wanting to set up the infrastructure side myself, with something concrete running on it: a real frontend, a real backend, a real server.

By day I work on systems that serve tens of thousands of concurrent users, and I'd only ever seen most of that machinery from inside a job. I wanted to take it apart myself, in close-to-production conditions, on something that was mine.

It's been going for about 22 months now. What it became, and why I kept reaching for things it never needed, is the rest of this post.

The product, and the reason

Before going further, here's what all that infrastructure serves.

Grimoire is a place to keep your recipes and how-to procedures, which it calls Spells. Your Grimoire is private: the stuff you reach for every day. That's the part that's live today, Android-only, on the Play Store.

It's wrapped in a cozy fantasy theme: Spells, a Grimoire, an Academy. I wanted it a bit geeky so it wouldn't feel like more work at 9pm. It's mobile-first (you cook, hands dirty, the phone's already on the counter) and deliberately slow: no feed, no nudges. You don't fill a Grimoire in a day.

The rest is still being built. The Academy, a shared space where Spells get published, discovered and adapted: pull one into your own book, tweak it, share it back. A "lineage" view of how a published Spell gets forked downstream. Quests, for one-shot tasks. Guilds, smaller closed spaces to share within a group.

That's the product. Most of it could have shipped on a single rented box. The app needed some of what follows; most of it I built because I wanted each layer in my own hands rather than borrowed from a job.

The thesis

A personal project with no deadline and no boss is the easiest place in the world to over-engineer. I've come to think that's not a flaw. It's the point.

At work you ship the minimum: someone's paying, and it's not your time to gamble with. Strip that away and you can afford to try things and see what happens, with nothing lost if they don't pan out.

So I did the unreasonable things on purpose. Hexagonal architecture, which later let me swap search engines in a weekend. Zero-downtime deploys on an Elasticsearch I didn't need yet. A pinch of DDD while the app was young enough to absorb it.

You only learn what a pattern is worth once you've lived with it, even with an understanding as humble, and sometimes as shaky, as mine. What hexagonal and DDD actually looked like in this codebase is its own post.

None of this is advice: having no users and no revenue let me make trade-offs I couldn't defend at work.

Growing it one plateau at a time

I never planned this as a stack. It grew in plateaus, each opened only once the previous one felt solid, and usually because I'd hit a wall that made the next layer worth building.

Backend, alone, for three months. December 2024 to March 2025: just Kotlin and Spring Boot. I used the time to sit properly with hexagonal architecture and a contract-first OpenAPI spec. No client yet, just the shape of the system.

It was also where I really learned Kotlin and Gradle, and came to like how little Kotlin asks of you to get something clean working. I made my share of design mistakes in there too; some I only saw later, once a second layer started leaning on them.

Then it got a face. March 2025: Android with Jetpack Compose, built alongside the backend. Up to then the app had lived as a wall of requests in Insomnia; this is where it stopped being an abstraction.

A pipeline, earlier than I needed one. Around the same time, Dagger.io for CI: pipelines as real code, not YAML. Over-built in places, no question. But never wasted: it's what I run every day now, without thinking about it.

Infrastructure, in a sandbox first. Summer 2025: Pulumi, but on a local KinD cluster, not a live server. The same principle I lean on everywhere: the local environment should look as much like the real one as possible. So I got the deployment working locally before trusting it anywhere real.

Finally, a real server. August 28th, 2025: a Hetzner VPS running k3s. I did this the old way, on a deliberately bare box: cheap, and empty. Everything a managed platform hides was now mine to stand up by hand: reverse proxy, cert manager, the lot. The point wasn't hardship for its own sake. A managed control plane lets you stay theoretical about that whole sysadmin layer, and I wanted to actually meet it. (It did break, spectacularly, eight months later. The post-mortem gets its own write-up.)

From there it kept growing, driven by friction rather than a roadmap: Elasticsearch eating my VPS pushed me to Meilisearch; and since Spells are meant to be shared publicly, I built a small React admin to moderate them before opening that up.

Where it didn't go to plan

Deliberate overkill was only half of it. Plenty of the project came from the other direction: plans that met reality and bent.

I set up a full Karate E2E suite early, expecting to lean on it; later I admitted it was the wrong call for a project this size and tore it out. I ran three separate repos for a year, sure it was cleaner, until the workflow got painful enough to push me into a monorepo. Neither was a master plan. Both were course corrections, one from hindsight, the other from sheer fatigue. (The monorepo move gets a full write-up later in the series.)

Some of it was just blind spots. My backend habits had hidden a plain front-end fact from me: config gets frozen into the artifact at build time. It isn't read from the environment at startup the way it would be on a server. On mobile that's final; it ships embedded in the app. On web I could claw it back with a tiny Express server that serves the config at runtime. Either way I only hit it at deploy time, on the part of the project I'd filed under "the easy corner."

The landing page is the clearest case. It wasn't planned at all: I just needed somewhere to host the terms of service. I built a real one anyway, because I felt like it, but kept it cheap on purpose. Astro, pushed to Cloudflare Pages, built locally, done. No Pulumi, no Dagger pipeline. A small static site wasn't worth the maintenance, time and learning the backend was.

Wrapping it in the same machinery would have been absurd. (Cloudflare is exactly the managed, single-vendor thing I spent the whole infra effort pushing against. But a static site is portable because it's trivial: I can move it in an afternoon. I'll dig into where "provider-agnostic" actually holds, and where it doesn't, in a separate post.)

So the overkill dial wasn't set once and left there. It moved with what the project threw at me and what I had the energy for: studied on the backend, deliberately minimal on the landing page, and now and then just whatever I could be bothered with that month.

Where it stands

So where is all this now? The app is live, actually in production on a €10-a-month VPS, and it's been there a few weeks rather than sitting behind a "coming soon." A handful of friends and colleagues are the first real users. You can get it on the Play Store, or have a look at the landing page. The app, the CI toolkit and the landing page are public on GitLab; the infrastructure repo stays private, since it describes a cluster that's running.

Building the whole thing myself, from backend to cluster to landing page, is what finally made a system click for me, after years of only operating one slice of it. The rest is too specific for one post, so it's scattered across smaller ones: the k3s crash, Pulumi's portability promise, hexagonal and DDD in practice, the search-engine swap, where I drew the line with Keycloak, and the monorepo I backed into.

The other posts

The three-line config I shelved, and the three-hour outage it became (coming soon)
Building my infrastructure like I build software (coming soon)
Everyone has an opinion about DDD. I'd never had my own. (coming soon)
I swapped my whole search engine in a weekend. The domain never noticed. (coming soon)
What I let Keycloak handle, and what I took back (coming soon)
The API client I published and pulled, until a monorepo just built it (coming soon)