Over a few weeks I moved seven self-hosted services onto Kubernetes — ArgoCD, supply-chain policy, the whole GitOps loop — and then, on one afternoon, I moved all seven back to Docker Compose. This is the part almost nobody writes about: not the triumphant migration, but the reversal, and the reasoning that justified both halves of it. The reversal wasn't a technical failure. What broke was something more useful than any one service: the assumption I'd been carrying about what Kubernetes is for.
The migration nobody needed to read about
The upward half is the half the internet is full of, so I'll compress it. A single 4-core, 24-gig machine runs a pile of compose stacks behind one reverse proxy. I added K3s to that same box — not on new hardware, but alongside the compose services, sharing the kernel's network stack — and moved the services over in phases. ArgoCD with an app-of-apps root. A per-service migration template so each service follows the same path. Kyverno admission policy, Trivy image scanning, SealedSecrets for credentials. All the machinery that makes a cluster something you'd let near production.
The promise I'd made myself at the start was the discipline that kept the migration honest: external domains and ports stay unchanged, and the reverse proxy stays the anchor. Move a service in, get it working on k3s, then repoint the proxy rule. That way the public surface never moves while the backend swaps under it, and each service migrates as its own reversible step.
It worked. The tools did their jobs. ArgoCD reported Synced and Healthy for every Application. And then I undid it.
What came back, and why it's not a rollback
Everything went back to compose in a single day. Homepage, trilium, dify, vikunja, apprise, the LLM stack — every service I'd migrated in phases C and D. The k3s-native services that had no compose predecessor went into compose too. What stayed on k3s was a short list: lab-environment, headlamp, and the PR-lanes lab namespace, which runs a two-tier practice app named hello.
I'm careful to call it a reversal and not a rollback, because a rollback implies going back to a bad state after a failed change. Nothing failed. The things phase A, B, E, and F+G built were all kept — the cluster foundation, the GitOps loop, the supply-chain policy, the service mesh — and most of them are still running. What was reversed was one specific line: the line that said "move existing compose services into k8s."
The decision came down to one observation, and once I saw it, it explained everything.
Everything I had migrated was software I didn't write.
Kubernetes only pays you back for software you own
I don't mean "software I didn't write" in the sentiment sense. I mean it literally: open-source images pulled from upstream registries. dify, vikunja, trilium, homepage, apprise, the model runners — none of it was code I authored, iterated on, or shipped. And that one fact quietly disqualified each of them from every reason you'd run Kubernetes at all.
Take the integration cost. A third-party image doesn't want to join your cluster's network policy, your CI pipeline, your internal DNS, your admission rules. None of that machinery knows anything about how the image's internals are wired, because the person who wrote the image never intended it for your enforcement layer. So you either leave it half-integrated, or you do integration work whose entire output is making someone else's sealed box legible to your platform. The complexity is real labor either way, and it produces nothing anyone ever uses. There is no necessary reason a stock open-source image needs to integrate with your cluster at all.
Worse is the second test, because it's where the whole value proposition evaporates. Canary releases, service meshes, mTLS between services, gradual rollouts — every one of those exists to give an owner fine-grained control over their own delivery. They are tools for a producer. But a third-party image isn't something I release; it's something I consume. It has a version tag someone else cuts, and I take it as a binary on or off. There is no curve of my own to shape, no canary of my own to hold back, no service identity of my own to assert between services whose topology the upstream already sealed shut. The mesh and the canary and the grader were looking for a producer, and I showed up as a consumer.
The third test is where it stops being theory. Supply-chain policy — the phase E stuff I'd been proud of — genuinely has a notion of a clean scan. A vulnerability scanner runs against every image, and it flags the CVEs it finds. For software I write, that's a solvable problem: patch the Dockerfile, rebuild, redeploy, done. The pipeline absorbs it. For software someone else writes, "fix the CVE" means running a fork of the upstream image and rebasing my patches onto every release forever. That is not a one-time cost. It's a tax that compounds with every upstream tag.
I know because I tried it. I built wrapper images that pulled the upstream base, applied a patch, rebuilt through a signed pipeline, and only then were admitted into the cluster. It was real work, and it didn't finish — patching somebody else's image is precisely the job that never quite gets done, and each attempt eats hours nowhere near the thing you actually intended to run. Eventually I retired the wrappers and accepted the CVE they'd been blocking as low-severity. The security policy hadn't failed. It had collided with the wrong category of software: production-grade security applied to a consumer-grade artifact, and the artifacts walked away.
The bill you already paid
None of that cost is hypothetical. Repairs the cluster forced me through are sitting in my incident log, and looking back at them together is what made the pattern visible. They read as unrelated one-off outages: memory overcommitted so the kernel OOM-killed containers while Kubernetes kept reporting Running; a Trivy scan storm hammering the disk with concurrency the operator never should have allowed; a packet between a pod and a compose container disappearing into a policy-routing blackhole; a Cilium flag narrowing socket load-balancing and taking every reverse-proxied service down for ninety seconds on the way in.
Four incidents, four root causes, but one shape under all of them. Kubernetes isn't free on a single node that already runs Docker. It adds a second scheduler, a second network, a second security layer, a second source of truth — and then those layers have to agree with the ones that were already there, on the same kernel, on the same box. Compose breaks in a bounded way, and its failure is legible. Kubernetes breaks across layers, and its failures arrive as "everything is green but it's still wrong."
For a fleet of services whose code you're actively changing, that tax buys you something real. For seven third-party images on a 4/24 home box, the tax buys you nothing, and I was paying it anyway.
Who's left is the proof
The list of what stayed on k3s is the cleanest argument I didn't have to make on purpose, because it happened before I noticed it was happening. The services that remained are the two k3s-native control panels — the lab's own console and the cluster viewer, neither of which ever had a compose version — and the PR-lanes namespace, which runs a two-tier practice app named hello: a hello-frontend and a hello-backend. And here is where the "authored versus consumed" line I'd been drawing for the migrations tightens, because only one half of that pair is really mine. hello-backend is software I ship through my own CI. hello-frontend reuses a signed image that was inherited from an earlier, now-retired practice app — its build context is gone, and the digest running today can't be rebuilt from the repo at all.
That is not a counterexample to the reversal. It's the reversal, in miniature. The half of the pair I author lives on k3s and responds to the mesh and the canary and the grader; the half that is a frozen, inherited consumer artifact is exactly the kind of thing that doesn't belong there, and one day it will have to be replaced with something buildable or moved off. The machinery finally has something to bite into, and the thing it bites into is the half I own. Even the lab, scaled down to two services, reproduces the same division the migrations did.
Not a single consumer-grade production service stayed. The day of the reversal made a theorem out of my own setup without me framing it that way at the time: k3s on that box was never the successor to compose. It was the lab. It runs software I can author, so that I can practice the platform I'm trying to learn on the only software that responds to it. The rest of the services — the ones where I'm the operator and someone else is the author — belong on compose, where "run it and leave it alone" is the entire feature set, and that's the feature they need.
Three ways to acquire a capability
If you'll let me push the reasoning one more level, the useful generalization isn't about Kubernetes or Compose at all. When I want some capability, there are three ways to have it, and each maps to a kind of software.
Buy it as SaaS, when the data doesn't have to be mine and I want no operational burden. Self-host it under compose, when the data has to be mine but the software doesn't. Write it and run it on Kubernetes, when the software itself is mine and I'm going to keep changing it.
The mistake — the thing I had to undo — was a fourth option that's strictly worse than any of the three: adopting the produce platform to run consumed software. You pay the full price of running a platform for software that can never use the platform's only justification. Every benefit Kubernetes offers is an option to shape software you own. If you've already decided not to shape it — and "I'm running someone else's image as-is" is exactly that decision — then every one of those options was abandoned before you started, and you're left holding the operational cost of a platform that has nothing left to do for you.
The dividing line isn't "important software versus unimportant software." It's volatility. Software I own changes constantly, so it earns an infrastructure that absorbs change. Software I consume is frozen by upstream, so it earns the boring, cheap thing that keeps it untouched. Choose the layer by how much the thing on top is going to move — not by how much you care about it. dify matters to me a great deal. It also never moves. So it runs in compose, where nothing asks it to.
That framework is the thing I actually walked away with. The migration up taught me what the platform does. The migration back taught me who it does it for. The first half was easy to write about, and already has been, everywhere. The second half is the part worth keeping.
Top comments (0)