DEV Community

Menshikov Vasil
Menshikov Vasil

Posted on

How I Rescued My Broken Local Kubernetes Development Environment in One Afternoon (Docker + k3d + Tilt)

#kubernetes #docker #devops #infrastructure

There's a specific flavor of shame in a local Kubernetes development environment that only you can't get working. Everyone else on the team runs kubectl get pods and gets an answer. You run it and get connection refused, again, and you quietly start to wonder if the problem is you. It wasn't me, it turned out - it was that I'd never actually built the thing, I'd just accreted it. This is the story of how I tore my broken setup down and rebuilt it, deliberately, in a single afternoon.

Why this bugged me for years

I'd inherited a FastAPI service - Python 3.12, listening on 8080, backed by PostgreSQL - that the team ran on "local Kubernetes." That phrase meant something different on every laptop, and mine was easily the worst offender. kubectl installed three different ways. A Docker Desktop that half-started on a good day. A helm binary that wasn't on my PATH. And, crucially, no cluster at all. Every single kubectl command greeted me with the same connection refused, and I'd learned to just... route around it, which is the worst possible response and exactly the one I kept choosing.

I burned two full mornings guessing. Then I got tired of guessing and followed a single opinionated write-up that treats the workstation itself as the deliverable - the idea being that you're not done until kubectl get pods -A responds without errors against a real local cluster. That reframing did something to my brain. If you want the full chapter with every OS variant, someone wrote up the exact workstation setup here, and it's the map I wish I'd had two mornings earlier.

The thing that finally clicked: install bottom-up

The mindset shift that fixed everything was embarrassingly simple - install in dependency order. Docker first, because nothing runs without it, then kubectl, k3d, helm, and Tilt last. On my Mac that collapsed to almost one command:

# Docker Desktop from docker.com (daemon + GUI), then:
brew install kubectl helm k3d tilt
Enter fullscreen mode Exit fullscreen mode

k3d is the quiet hero of this whole story. It's just k3s packaged into Docker containers - k3s being a lightweight, CNCF-certified Kubernetes distribution - so a local cluster is featherweight next to "real" Kubernetes. If you're agonizing over k3d vs kind vs minikube, here's what decided it for me: k3d spins up a cluster in under five seconds, ships a built-in image registry that Tilt adores, and barely touches RAM, whereas minikube boots a whole VM and kind - Kubernetes-IN-Docker, originally built to test Kubernetes itself - sits somewhere in the middle. For a tight inner dev loop I rebuild dozens of times a day, and those saved seconds compound into genuinely real time. The one hard requirement is Docker 20.10.5 or newer; k3d's own repository lists a working Docker install as the sole prerequisite. Everything else is just host headroom - 8 GB of RAM is a comfortable floor, 16 GB means you never think about it, and you'll want 10 to 20 GB of free disk for images and layers.

On Linux the shape is identical, except you install Docker Engine with no GUI and you must add yourself to the docker group or every command demands sudo:

sudo usermod -aG docker $USER
# then log out and back in so the group takes effect
Enter fullscreen mode Exit fullscreen mode

One caution I learned the hard way: those curl ... | bash installers for k3d, helm, and Tilt are convenient, but you're piping someone else's code from the internet straight into a shell. On a work machine I now download the script, skim it, then run it - or reach for the apt/dnf packages the official docs offer for kubectl and helm, which are cleaner to maintain long-term anyway.

Naming what each tool actually does

Here's the confession at the heart of this: before that afternoon, I genuinely could not have told you which tool did what, and that fuzziness is exactly why my setup was so fragile. The mental model that finally stuck goes like this. Docker builds images and runs the containers everything else sits inside. k3d spins up the cluster itself - k3s in Docker - and tears it down just as fast. kubectl and helm manage resources and packaged charts inside that cluster. Tilt runs the fast inner dev loop, watching your files and rebuilding and redeploying on every save, and its Live Update can even sync changed files straight into a running container, collapsing edit-build-push-deploy into seconds. And k9s is the optional terminal UI that replaces a wall of kubectl commands.

Once I could name the job of each binary, the errors stopped being mysterious and started being addressable. A connection refused is Docker. A stuck rebuild is Tilt. A missing chart is helm. The abstraction layers finally lined up in my head, and honestly that was more valuable than any single command I learned.

The smoke test that ends the guessing

This is the part I wish I'd run on day one instead of day three. Rather than debugging four tools in isolation, you prove the whole chain works together at once: create a throwaway cluster, switch to it, look at the system pods, delete it.

k3d cluster create dev
k3d kubeconfig merge dev --kubeconfig-switch-context
kubectl get pods -A
k3d cluster delete dev
Enter fullscreen mode Exit fullscreen mode

When that kubectl get pods -A printed coredns, traefik, and metrics-server all sitting in Running, I actually said it out loud to my empty room: the workstation was ready. That single sequence is the entire difference between "I think it's installed" and "I proved it works," and I've never set up a machine without it since. Naming the cluster dev and reusing it later is deliberate too - k3d makes clusters so cheap to create and destroy that a named, reusable dev cluster becomes the stable target for the rest of your toolchain.

The gotchas that used to eat my mornings

Almost every failure at this stage is one of a small handful of classics, and just running the list saved me hours. The most common by far is simply that the Docker daemon isn't running - k3d and Tilt both die with connection refused, so start Docker Desktop or sudo systemctl start docker and wait for it to fully come up. After a reboot you may hit port already in use, in which case just recreate the cluster.

The trickiest one, and the one that bit me hardest, is too many open files. Pods and Tilt both watch files, and they collide with system inotify limits; the symptom shows up as too many open files in pod logs or the kubelet. On a Linux or Docker host the fix is:

sudo sysctl fs.inotify.max_user_watches=1048576
sudo sysctl fs.inotify.max_user_instances=8192
Enter fullscreen mode Exit fullscreen mode

Make it survive reboots by putting those in a file under /etc/sysctl.d/, and as a fallback for stubborn machines you can create a k3d cluster with only a server node and no agents, which cuts the total open files.

Windows has its own trap: do all your dev work inside WSL2, including the repo itself. A teammate kept the code on C:\ and ran the tools from WSL2, and the result was a crawling filesystem and Tilt file sync that silently refused to trigger uvicorn --reload. Move the repo into the WSL2 home directory and the whole problem evaporates. And two quieter ones round it out - if Docker Desktop refuses to start with a virtualization complaint, enable hardware virtualization (VT-x / AMD-V, plus SLAT for WSL2) in BIOS/UEFI, and if you try to squeeze into 4 GB, expect OOM once PostgreSQL and a few pods pile in. Give Docker 8 GB or more under Settings then Resources and the random pod deaths stop.

The little upgrade: k9s made me delete half my muscle memory

Instead of hammering kubectl get pods, kubectl describe, and kubectl logs all day, I now open k9s - a terminal UI that continually watches the cluster - and navigate with arrow keys. Logs, restarts, deletes, all from one screen.

brew install derailed/k9s/k9s
Enter fullscreen mode Exit fullscreen mode

For anyone still finding their feet, it's a genuinely lovely visual replacement for dozens of repetitive commands, and I felt a small pang deleting keystrokes I'd spent years memorizing.

How it feels now

The change is night and day, and it's less about any single number than about the shift from flailing to knowing. What used to be two mornings of guessing is now a checklist-driven afternoon. kubectl get pods -A returns system pods in Running instead of connection refused. Tilt hot reload, which used to be silently broken behind that WSL2 path issue, now fires on every save. Debugging a pod is arrow keys in k9s instead of three kubectl invocations each time. And the biggest one, the one that's hard to put in a table: I'm actually confident it works now, because I watched the smoke test pass with my own eyes.

With the toolchain proven, the natural next step is spinning up a real dev cluster for the service - built-in registry, port forwarding, the works - rather than a throwaway. But if you're setting this up for the first time, please don't improvise the way I did for two mornings. Install bottom-up, run the smoke test before you touch a line of application code, and keep the gotcha list within reach.

The thing that genuinely stuck with me is this: a local Kubernetes environment is not "done" when the binaries are installed. It's done when you've watched the smoke test pass. I'd spent years treating my dev environment as something that accreted rather than something I built, and the fix was simply to build it on purpose, prove it, and only then start stacking real work on top. Prove it first. Everything after that is easier.

Sources & further reading

Top comments (0)