docker-compose vs Kubernetes: The dev/prod Parity Gap That Ate Our First Rollout

The service passed every local test in docker-compose. The first Kubernetes rollout fell over inside an hour on things Compose had never modeled: readiness, Secrets, startup order, service discovery. This is the docker-compose vs kubernetes dev/prod parity gap, and it cost us an outage before I admitted it existed.

Symptom

First prod rollout. Pod Running, but the Service sent it zero traffic.

$ kubectl get pod -l app=api
NAME                   READY   STATUS    RESTARTS   AGE
api-6c9f8b7d4-2xk9p    0/1     Running   3          2m

$ kubectl describe pod api-6c9f8b7d4-2xk9p | tail
  Warning  Unhealthy  readiness probe failed: connection refused
  ...
$ kubectl logs api-6c9f8b7d4-2xk9p --previous
  sqlalchemy.exc.OperationalError: could not connect to server: Connection refused
  Is the server running on host "postgres" (10.43.x.x) and accepting connections?

Three separate failures, none of which any green Compose test had touched:

1. No readiness probe existed, so I'd never configured one — Compose has no such concept.
2. A plaintext env var in Compose was supposed to be a Secret in k8s.
3. The app crashed reaching the DB before the DB was up. Compose's depends_on had papered over that for years.

Root cause

I'd assumed a Compose file and a set of k8s manifests were the same idea at different fidelity. They aren't. The Twelve-Factor App's tenth factor, dev/prod parity, names the trap: keep the gaps between dev and prod small — especially the tools gap, where "tiny incompatibilities crop up, causing code that worked and passed tests in development to fail in production."

Two different problems live under that one heading, and I'd conflated them:

• Backing-service parity. The canonical sin is SQLite locally, Postgres in prod. Sooner or later a behavioral difference bites at the worst moment. Same type, same version, both sides. No substitutions.
• Manifest-level bugs. A whole class of failure lives only at the Kubernetes object layer: misconfigured probes, thin RBAC, service discovery by in-cluster DNS, missing resource limits. Compose has none of these concepts, so it can surface none of these bugs.

That second bucket is the one that got me. A single Compose service is not one Kubernetes object — it's several, each carrying its own slice of production behavior:

• a Deployment — how the pods run
• a Service — how they're reached
• a ConfigMap + Secret — config and secrets
• an Ingress — external access

Every one of those is a place to misconfigure something, and a place Compose never made me think about. There's a solid breakdown of what "production-like local" actually means at this writeup on production-like local environments.

The fix that didn't work: Kompose

First instinct was to cheat and auto-convert. Kompose is a real project under the Kubernetes org, so I ran it:

# a starting point, NOT a prod manifest
kompose convert -f compose.yaml

Kompose's own authors are honest: "our conversions are not always 1-1... but will get you 99% of the way there." Production lives in that last 1%. What the conversion silently drops or mangles:

• depends_on is ignored. Kubernetes has no "start B after A." You use init containers or, better, app-level retries — reconnect to Postgres until it answers.
• build: doesn't build. Kubernetes can't build from source. The image must already be pushed to a registry.
• network_mode: host and custom networks map poorly or not at all.
• Bind mounts vanish. You're expected to use a ConfigMap/Secret.

The kicker: default Kompose output ships without requests/limits, without probes, with env vars in plaintext instead of Secrets. The automation strips out exactly the production-like properties you were trying to gain. A draft, not a deploy.

The fix that worked: a production-like k3d cluster

I switched to running the service in a real local cluster with k3d (k3s in Docker). The discipline that mattered was deciding, on purpose, what to reproduce and what to simplify.

Reproduce, no exceptions:

# pin the SAME minor as prod. Docker tags can't use '+',
# so it's 'v1.31.5-k3s1', not 'v1.30.2+k3s1'
k3d cluster create dev --image rancher/k3s:v1.31.5-k3s1

• Same backing services by type and version — Postgres, never SQLite.
• Real requests/limits, so scheduler and eviction behavior is realistic.
• The exact probe pair that would've saved the first rollout — liveness and readiness: /healthz liveness (process is up), /ready readiness (checks the Postgres connection; no traffic while the DB is unreachable, but no restart either).
• ConfigMap + Secret instead of hardcoding; Ingress + in-cluster DNS so routing matches prod.

Same real manifests, deployed the same declarative way as prod:

kubectl apply -k k8s/overlays/dev
kubectl rollout status deploy/api

Deliberately simplify, and write down why: prod scale and topology, managed cloud services (Postgres in a container instead of RDS), and load testing (doesn't belong on the box running the app it tests).

Proving the three bugs are dead, locally

The point of parity is that the failures now reproduce on the laptop. I ran each one on purpose before trusting the rollout:

# 1. startup order: kill the DB, restart the app, watch it retry not crashloop
kubectl delete pod -l app=postgres
kubectl logs -f deploy/api        # "waiting for postgres... retry 3" then ready

# 2. readiness gates traffic: app is up but DB down -> 0/1, no endpoint
kubectl get endpointslices -l kubernetes.io/service-name=api

# 3. secret is wired, not baked into the image
kubectl exec deploy/api -- printenv DATABASE_URL   # from Secret, not Dockerfile

All three passed at my desk. The next prod rollout was uneventful — the first one in that project that ever had been.

Before / after

	docker-compose only	Production-like k3d
Backing service	often SQLite/stand-in	same Postgres + version as prod
K8s version	not modeled	pinned to match prod
Probes	none	liveness + readiness, real
Secrets	plaintext env	ConfigMap + Secret
Service discovery	Compose networking	real CoreDNS / Ingress
depends_on	hides startup-order bugs	forces real retry logic
First prod rollout	collapsed on unmodeled objects	boring, matched local

Guardrail

• "Works in Compose" validates the app, not the manifests. Manifest bugs only appear after a real rollout — so test in k8s.
• Production-like is not "everything like prod." It's an environment where you know exactly what matches, what doesn't, and why. The list of documented simplifications is the most underrated line item on the whole checklist.
• You don't need the whole prod zoo. Bring up only the namespace and services you're touching this session. Parity isn't recreating everything at once.
• Don't try to copy prod exactly. Copying prod onto a laptop is impossible and pointless. The goal is narrower: shrink the gap between how the service behaves at your desk and how it behaves in prod, so packaging and deployment bugs get caught before the rollout.

What I'd do differently

Keep Compose for the earliest, fastest hacking loop — it still earns its place there. Just never again mistake a green Compose run for validation of a Kubernetes deploy. The moment a service is bound for k8s, it gets tested in a local cluster on pinned versions and the same manifests as prod. My rollouts have since gotten boring, which after that first outage is the highest praise I have.

Bottom line: a Compose file is structurally not equivalent to k8s manifests — pin the same versions and run the same manifests in a local cluster, or the manifest-layer bugs wait for prod to find them.

Sources

• The Twelve-Factor App — X. Dev/prod parity
• Kompose — convert Compose to Kubernetes (kubernetes/kompose)
• k3d — run CNCF's k3s in Docker for local clusters
• Kubernetes docs — Configure Liveness, Readiness and Startup Probes
• Kubernetes docs — Secrets
• Walkthrough on production-like local environments