The Kubernetes Local Development Environment Checklist That Cut Onboarding From Two Months to Two Weeks

Onboarding to Kubernetes used to be a two-month slog where every engineer built a slightly different, slightly broken local setup. The fix wasn't more training. It was a single Kubernetes local development environment checklist — a written definition of "done" for a local environment. Ramp dropped to two weeks. This is the retrospective and the checklist.

What broke

Everyone onboarding to myapp (FastAPI on 8080, Postgres behind it) built their own local setup. One used docker-compose. One ran raw kubectl apply from the desktop. One had probes but no resource limits. Bugs showed up in different places for different people, and "works on my machine" was the most common sentence in standup.

The problem wasn't Kubernetes. It was inconsistency — every environment only pretended to resemble production, and each pretended differently. There's a solid recap-and-checklist for a fast, production-like local setup at this conclusion writeup.

The fix: one checklist, every item mapped to a real outage

We printed this and made it the definition of a ready local environment. Every line maps to a production failure we'd actually seen:

Same Kubernetes version and add-ons as prod. Prod runs a given version with Ingress NGINX, local does too. Version drift is the classic "worked on my machine."
Real manifests (Helm/Kustomize), not docker-compose. Same YAML locally and in prod, values swapped via overlays.
requests and limits on CPU/memory for every container. Without them a Pod starves or eats the whole node — locally it also protects the laptop.
A liveness probe — GET /healthz, confirms only that the process is alive.
A readiness probe — GET /ready, checks the Postgres connection, so the Pod leaves the Service endpoints until it can actually serve.
ConfigMap/Secret instead of values in code — config separated from code, straight from the Twelve-Factor config principle.
Ingress like prod, so routing behavior matches.
A fast inner loop — save a file, see the result in seconds. If every cycle is a manual rebuild and kubectl apply, people avoid the cluster, which defeats the point.

The underlying principle is parity: you see problems with manifests, RBAC, network policies, and service discovery locally, not from a 3 AM alert. The checklist made the principle checkable.

What two weeks looked like

Week one: each engineer stood up a k3d dev cluster, containerized myapp, wrote real manifests, and got Tilt giving them a seconds-long inner loop. Week two: Postgres in-cluster, ConfigMaps and Secrets, both probes — every checklist item ticked. Because everyone built the same thing, review got sharper (a missing readiness probe is now a review comment) and "works on my machine" basically disappeared. Parity was the productivity.

Before / after

	Before	After
Onboarding time	~2 months	~2 weeks
Local setups	one per engineer, all different	one shared, checklisted
Common bug class	found in staging	caught locally
"Works on my machine"	weekly	rare
Review signal	vague	checklist-backed

The probe distinction that stopped a recurring incident

One checklist item earns its own section, because getting it wrong caused the same production incident three times before we standardized.

Symptom

A five-second database blip turned into a cluster-wide restart storm. Otherwise-healthy Pods cycling:

$ kubectl get pods -n myapp
NAME                   READY   STATUS    RESTARTS   AGE
myapp-7d9c4b6f8-aa11   0/1     Running   6          9m
myapp-7d9c4b6f8-bb22   0/1     Running   6          9m

Root cause

We reached into a dependency — the Postgres connection — from the liveness probe. The Kubernetes probe docs are explicit: the kubelet restarts a container when liveness fails, but only pulls the Pod from Service endpoints when readiness fails. Liveness (GET /healthz) asks "is the process alive?" — fail it and Kubernetes restarts. Readiness (GET /ready) asks "can this Pod serve right now?" — fail it and the Pod leaves endpoints without a restart. So when the DB blipped, liveness failed and Kubernetes restarted healthy Pods in a cascade.

The fix

A rule: liveness stays shallow (process alive, no dependencies); readiness is where you check dependencies. Concretely:

@app.get("/healthz")   # liveness: is the process up? nothing else.
def healthz():
    return {"status": "ok"}

@app.get("/ready")     # readiness: can I serve? checks the DB pool.
def ready():
    check_db()         # fail -> leave endpoints, do NOT restart
    return {"status": "ready"}

Once every engineer built that split into their local myapp from the checklist, the restart-storm incident stopped happening. The checklist turned a lesson three separate people had to relearn into one line nobody could skip.

Next steps — staged, not all at once

The recap is explicit that these are extensions to add as the team grows, not day-one requirements. One at a time.

GitOps. Solo with kubectl apply or Tilt is fine; with a growing team the question becomes "what state is the cluster in, and who put it there?" We adopted Argo CD (CNCF Graduated since 2022) for apps — its web UI shows each Application as a tree of objects with status. Trialed in the dev cluster:

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace
kubectl port-forward svc/argocd-server -n argocd 8080:443

We left Flux (also CNCF Graduated, great for infra automation, no native UI) for later. Argo CD for apps, Flux for infra is a target, not a start.

Observability. We'd been eyeballing via kubectl logs and k9s. The next level collects the three signals systematically. Fastest path is the kube-prometheus-stack Helm chart (Prometheus, Grafana, Alertmanager, exporters, dashboards in one command) with a ServiceMonitor to scrape /metrics. Warning we heeded: the stack is heavy and can crush a weak k3d cluster — constrain requests or disable components you don't need.

Remote-cluster tools for the hard cases. For services dragging a dozen dependencies, or debugging against real staging data, Telepresence (intercepts a service's traffic to your local process; VPN model, needs root) and mirrord (runs your local binary as if in a Pod, no root; mirror copies traffic, steal intercepts). Reach for them only when a local cluster genuinely can't handle the scenario. For day-to-day myapp, k3d + Tilt is faster.

Guardrail

Don't adopt all the next steps at once. GitOps + observability + remote tools in one sprint buries a team. One at a time.
A checklist beats a wiki page. People tick boxes; they skim paragraphs.
The heavy observability stack can crush a laptop cluster — cap resources before installing.
Parity is the metric, not tool count. Same version, same manifests, real probes.
Cross-check against a separate production-readiness list at rollout — graceful shutdown, rolling updates, QoS classes. "Looks like prod locally" and "ready for prod" are related, not identical, and conflating them is its own trap.

What I'd do differently

Write the checklist before the first hire, not after the tenth. The biggest lever in Kubernetes onboarding wasn't a tool — it was agreeing, in writing, on what a good local environment means, so the team scaled instead of fragmenting into ten incompatible setups.

Bottom line: standardize the local environment with a checklist where every item maps to a real outage, and onboarding stops being tribal knowledge — then add GitOps, observability, and remote tooling one deliberate step at a time.