DEV Community

Menshikov Vasil
Menshikov Vasil

Posted on

ImagePullBackOff in k3d even though `docker build` succeeded? Here's why (a k8s error field guide)

The most disorienting beginner moment in k3d: you build an image with docker build, the cluster says ImagePullBackOff, and you're sure the image exists. It does — in Docker. k3d nodes run on containerd, which is isolated from your Docker daemon. Docker has the image; the cluster can't see it.

Takeaways

Almost every status is diagnosed with the same three commands: kubectl describe pod (State/Reason/Events), kubectl logs --previous, kubectl get events --sort-by=.lastTimestamp.

  • ImagePullBackOff: typo in image/tag, private registry without imagePullSecrets, Docker Hub rate limit (toomanyrequests), or the k3d containerd isolation. Fix the last with k3d image import myapp:dev -c dev, or (better) a local registry — and reference the full name k3d-registry.localhost:5000/myapp:dev in the manifest. An incomplete image name is a top cause of a repeat BackOff.
  • CrashLoopBackOff: kubectl logs --previous for the dead instance's real error; check Last State: Terminated Exit Code. Common: app bug at startup, missing env/config, unavailable dependency, OOM, too-strict liveness, or exit code 0 (a long-running service that "exited successfully" — usually a wrong entrypoint).
  • Pending: FailedScheduling names the cause — usually insufficient CPU/memory for requests on a small local cluster. Also nodeSelector/affinity, taints, unbound PVC, hostPort clash.
  • OOMKilled (exit 137 = 128+9): container-level (exceeded limits.memory) or the sneaky node-level OOM — k3d nodes live in a Docker VM, so combined limits over the VM's memory kill Pods even when each app is within its own limit. Don't overcommit.
  • Service silent, Pods Running: walk Service -> endpoints -> Pod. Empty endpoints = selector mismatch (labels are case-sensitive) or port mismatch (targetPort vs containerPort). Running 0/1 = readiness failing (removed from endpoints, not restarted).
  • Tilt not updating: synced path outside build context, file not covered by sync(), fall_back_on file changed (forces rebuild), or run() before sync(). FastAPI trap: without a process restart the synced code lands but uvicorn keeps the old code in memory — use --reload or docker_build_with_restart.
  • Disk eaten: kubelet image GC is lazy (only above 85% disk). docker system df, then prune incrementally — but -a can wipe images the cluster needs and --volumes can delete your local PostgreSQL data.

Full article: https://dorokhovich.com/blog/local-k8s-troubleshooting?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-troubleshooting

Top comments (0)