Metronom

Posted on Aug 6

Run Postgres in Kubernetes for Local Development: Killing the docker-compose Parity Gap

For a year the app ran in Kubernetes and its Postgres, Redis and RabbitMQ ran in a docker compose up on the side. It worked until it didn't. If you're going to run Postgres in Kubernetes for local development, do it in the same cluster as the app — two infra descriptions drift, and the drift is where the rollout bugs come from. Here's the migration, including the Helm licensing ambush that nearly derailed it.

Symptom

myapp (FastAPI on 8080) ran in a k3d dev cluster, restarted by Tilt on every save. Postgres/Redis/RabbitMQ lived in a compose.yaml next to it. Two worlds.

The app reached its DB at localhost:5432 locally but needed a Service DNS name in the cluster. Every deploy, the same class of break:

sqlalchemy.exc.OperationalError: could not translate host name "localhost"
  to address: Name or service not known
# prod wanted: postgresql://myapp:...@postgres:5432/myapp

Everything we'd debugged in Compose — service addresses, env vars, startup order — had to be rewritten from scratch for Kubernetes, and that rewrite was exactly where the bugs lived.

Root cause

Two parallel infra descriptions. In production only the Kubernetes one exists. Anything validated in Compose proved nothing about the manifests that actually ship. This is the Twelve-Factor dev/prod parity principle: keep dev and prod the same, especially the tools. There's a solid breakdown of bringing dependencies into the cluster instead of Compose on the side at this dependencies writeup.

Bring the dependencies up in the same cluster with the same manifests and you get unified DNS (the app finds the DB by service name postgres, not localhost:5432), shared Secrets and ConfigMaps, and you catch manifest problems — RBAC, resource limits, probes, network policies — before the rollout, not on a Friday. Heuristic: if the service is going to Kubernetes, run its dependencies in the cluster.

The ambush: the 2025 Bitnami brownout

The default for stateful dependencies has always been a Helm chart, historically Bitnami (bitnami/postgresql, etc.). In 2025 that broke. Per Bitnami's own catalog-change notice, public image brownouts started 2025-08-28, and from 2025-09-29 most public Bitnami OCI charts and images moved behind a Broadcom commercial subscription. What's left public is limited and only on -latest; the rest sits in a bitnamilegacy repo, unsupported, no security patches.

Blindly running helm install bitnami/* in 2026 is no longer safe. Options: Chainguard's drop-in replacement charts (forked for compatibility), official vendor charts, or a deliberately pinned legacy tag (acceptable-ish for local, not prod).

The fix: a raw Postgres manifest

The most reliable dev Postgres turned out not to be Helm at all — a raw three-object manifest, nothing to look up in a chart repo, no subscription dependency:

# postgres.yaml -- Deployment + Service + PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: postgres-data, namespace: myapp }
spec:
  accessModes: ["ReadWriteOnce"]
  resources: { requests: { storage: 1Gi } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: postgres, namespace: myapp }
spec:
  replicas: 1
  selector: { matchLabels: { app: postgres } }
  template:
    metadata: { labels: { app: postgres } }
    spec:
      containers:
        - name: postgres
          image: postgres:17
          env:
            - { name: POSTGRES_DB, value: myapp }
            - { name: POSTGRES_USER, value: myapp }
            - { name: POSTGRES_PASSWORD, value: devpass }
            - { name: PGDATA, value: /var/lib/postgresql/data/pgdata }
          volumeMounts:
            - { name: data, mountPath: /var/lib/postgresql/data }
          readinessProbe:
            exec: { command: ["pg_isready", "-U", "myapp", "-d", "myapp"] }
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: postgres-data }
---
apiVersion: v1
kind: Service
metadata: { name: postgres, namespace: myapp }
spec:
  selector: { app: postgres }
  ports: [{ port: 5432, targetPort: 5432 }]

Now myapp reaches the DB at postgresql://myapp:devpass@postgres:5432/myapp by service name — the same shape as prod.

Wiring Tilt and getting startup order right

Tilt brings dependencies up and tears them down with the project. Ordering is the point:

k8s_yaml('postgres.yaml')
k8s_resource('postgres', port_forwards=['5432:5432'])
k8s_resource('myapp', resource_deps=['postgres'])

resource_deps makes Tilt wait until postgres passes its pg_isready readiness probe before starting myapp. Verify the ordering held with a quick check — the app should never log a connection error on a cold start:

tilt up
kubectl logs deploy/myapp -n myapp | grep -i "could not translate\|connection refused" || echo "clean start"

A trap Tilt's own Helm docs confirm: Tilt's built-in helm() is really helm template and skips Helm hooks, so for third-party charts with initialization you want the helm_resource extension (a real helm install). That distinction costs hours if you miss it.

The surprise that ate an afternoon: ephemeral data

My whole database vanished after k3d cluster delete. In k3d the default local-path provisioner k3s ships writes to /var/lib/rancher/k3s/storage inside the k3d node's container filesystem — ephemeral by default. Often desirable (clean seed every run). To persist across cluster recreation, map a host directory at creation:

k3d cluster create dev \
  --volume $HOME/k3d-storage:/var/lib/rancher/k3s/storage@all

The PVC detail that confuses everyone once: k3d's local-path StorageClass uses WaitForFirstConsumer binding and a reclaimPolicy of Delete, and its volumes are node-local — as the local-path-provisioner project notes, a local-path PVC is pinned to a node via kubernetes.io/hostname, so the pod always reschedules to the same node. None of it matters until you delete the cluster; then survival comes down entirely to whether you mapped a host dir.

Migrations: a separate Job, not the app's initContainer

An empty Postgres isn't enough — myapp expects a schema. Tempting to put alembic upgrade head in myapp's initContainer, but with multiple replicas each pod races to migrate, and a long migration risks a probe killing it mid-run. Use a standalone migrations Job, ordered in Tilt:

k8s_resource('myapp-migrate', resource_deps=['postgres'])
k8s_resource('myapp-seed',    resource_deps=['myapp-migrate'])
k8s_resource('myapp',         resource_deps=['myapp-seed'])

Seeding is another Job after migrations — trivial to disable in prod by just not wiring it up. The Job also gives you a clean audit trail: kubectl logs job/myapp-migrate shows exactly which migrations ran, and a failed migration blocks the app start instead of half-migrating under live traffic.

Before / after

	Before	After
Dependencies	docker-compose on the side	in-cluster, like prod
DB address	`localhost:5432` locally	`postgres` service name everywhere
Rollout surprises	env/order rewritten for k8s	caught locally
Chart risk	unknowingly on Bitnami	raw manifest / Chainguard
Migrations	ad hoc	separate Job, ordered by Tilt

Guardrail

Make data survival an explicit per-project decision: ephemeral for a clean seed, host-mapped for continuity across k3d cluster delete. Never a surprise.
OSS dependencies run as-is in the cluster. Managed services (S3, SQS, DynamoDB) get LocalStack for the fast loop, but verify against a real dev account for anything with subtle semantics — an emulator is never 1:1 with the cloud.
Audit every helm install for a Bitnami source before you build on it. That chart may rot underneath you.
Migrations and seed are separate Jobs, never baked into the app's startup.

What I'd do differently

Move off Compose-on-the-side the day the app went to Kubernetes, not a year later. Every week of two-worlds cost us a rollout surprise that a single cluster would have caught for free.

Bottom line: run your Postgres/Redis/RabbitMQ in the same cluster as the app so the DB address, startup order, and manifests are validated locally — and audit any Bitnami chart before it rots under you.

DEV Community