DEV Community

Metronom
Metronom

Posted on

k3d Ingress Localhost Setup: Fixing "curl: could not resolve host" and the Dead Ingress

k3d Ingress Localhost Setup: Fixing "curl: could not resolve host" and the Dead Ingress

Worked in Chrome. Died in curl. Safari showed a blank page. The Ingress existed and routed to a void. I burned two days on a k3d ingress localhost setup, and not one of the three failures was a bug. Every one was a documented layer I didn't understand.

What broke

Moving a checkout service off docker-compose into local k3d, because staging kept surfacing bugs the laptops never saw. FastAPI on 8080, Postgres behind it. Pods came up green.

$ curl http://localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused
Enter fullscreen mode Exit fullscreen mode

Expected — the cluster has its own network. What I didn't grasp was the three layers between my laptop and a pod. I faceplanted on all three. There's a solid breakdown of the whole path in this local networking and Ingress writeup. Here's the compressed field version.

Symptom 1: the Ingress existed and did nothing

Reasonable manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: checkout
  namespace: checkout
spec:
  rules:
  - host: checkout.localhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: checkout
            port:
              number: 80
Enter fullscreen mode Exit fullscreen mode

kubectl apply, no errors. kubectl get ingress showed it. No route, no response, no error.

Root cause

The line from the Ingress docs:

You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.

An Ingress manifest is a declaration of rules. A controller has to be running to read them and accept traffic. I assumed "Ingress" was something Kubernetes did. It's something a controller does.

k3d is built on k3s, and k3s ships Traefik out of the box plus a ServiceLB so LoadBalancer services don't hang pending — the k3d bundled-k3s docs cover both. So the controller was there. My real problem was one layer down.

Symptom 2: no route from the host

Traefik ran inside the cluster on 80/443. Nothing on my host forwarded to it.

The fix

k3d needs a port mapping to its loadbalancer node, set at creation:

k3d cluster create dev --api-port 6550 -p "8081:80@loadbalancer" --agents 2
Enter fullscreen mode Exit fullscreen mode

-p "8081:80@loadbalancer" maps host 8081 to Traefik's port 80 — the k3d exposing-services guide documents the HOST:CONTAINER@loadbalancer syntax.

The gotcha that cost the most: you can only set this at cluster creation. There is no k3d cluster edit that adds a port to a running cluster. I hunted for one for an hour. Delete and recreate:

k3d cluster delete dev
k3d cluster create dev -p "8081:80@loadbalancer" --agents 2
Enter fullscreen mode Exit fullscreen mode

With that in place the chain is the real prod routing shape, just *.localhost instead of a domain and k3d instead of a cloud LB:

curl -> host :8081 -> Traefik :80 -> rule host: checkout.localhost -> Service -> Pod
Enter fullscreen mode Exit fullscreen mode

How I confirmed each hop rather than guessing. Traefik has the mapping the moment the LB container exposes the port:

$ docker ps --format '{{.Names}}\t{{.Ports}}' | grep serverlb
k3d-dev-serverlb   0.0.0.0:8081->80/tcp, ...
Enter fullscreen mode Exit fullscreen mode

Then the Service actually has endpoints — an empty endpoints list here means the selector doesn't match any pod, which is its own silent failure mode:

$ kubectl get endpoints checkout -n checkout
NAME       ENDPOINTS           AGE
checkout   10.42.0.14:8080     3m
Enter fullscreen mode Exit fullscreen mode

Empty ENDPOINTS? The Ingress and port mapping are irrelevant — fix the Service selector first.

Symptom 3: Chrome worked, curl and Safari didn't

After the recreate, http://checkout.localhost:8081 opened in Chrome. Four minutes of feeling clever, then QA on Safari said blank, and the integration test failed:

curl: (6) Could not resolve host: checkout.localhost
Enter fullscreen mode Exit fullscreen mode

Same URL, same cluster, opposite results by client. I blamed the cluster.

Root cause

The .localhost TLD is reserved by RFC 6761 for loopback, and resolvers are supposed to return 127.0.0.1 without hitting DNS. "Supposed to" does the heavy lifting:

  • Chrome and Firefox resolve any *.localhost to loopback themselves. Zero config. That's why it worked.
  • Safari on macOS does not — it defers to the OS resolver and finds nothing.
  • curl and every non-browser HTTP client treat *.localhost as an ordinary domain. No entry, could not resolve host.

The fix

One line, now in our onboarding README:

echo '127.0.0.1  checkout.localhost' | sudo tee -a /etc/hosts
Enter fullscreen mode Exit fullscreen mode

(Windows: C:\Windows\System32\drivers\etc\hosts, as admin.) After that, curl, Safari, and CI all behaved. Rule: Chrome/Firefox get *.localhost free; curl, Safari, and any custom domain need an /etc/hosts entry.

The DNS trap that bit a week later

Saving you the trip. Inside the cluster you reach services by name via CoreDNS. Within one namespace, checkout hit Postgres as postgres. I moved Postgres to a shared data namespace and the short name broke:

$ nslookup postgres
** server can't find postgres: NXDOMAIN
Enter fullscreen mode Exit fullscreen mode

From another namespace you need postgres.data or the full FQDN postgres.data.svc.cluster.local — the <service>.<namespace>.svc.cluster.local form in the DNS for Services and Pods spec. Test resolution from a throwaway pod:

kubectl run -it --rm dnstest --image=busybox --restart=Never -- \
  nslookup checkout.checkout.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Rule of thumb: hard-code the short name in your config and you've hard-coded "same namespace forever." I now use the two-part <service>.<namespace> form everywhere, so moving a dependency across namespaces is a one-line env change, not a debugging session.

Runbook: prod-like local routing from scratch

What I do now, every new cluster, in order:

  1. Create with the mapping baked in: k3d cluster create dev -p "8081:80@loadbalancer" --agents 2.
  2. Confirm the controller exists: kubectl -n kube-system get pods | grep traefik.
  3. Apply Deployment + Service, confirm endpoints are non-empty: kubectl get endpoints <svc> -n <ns>.
  4. Apply the Ingress with host: <app>.localhost.
  5. Add the hosts entry so curl and CI match Chrome: echo '127.0.0.1 <app>.localhost' | sudo tee -a /etc/hosts.
  6. Smoke test with curl, not the browser: curl -sS http://<app>.localhost:8081/healthz.

Step 6 with curl and not Chrome is deliberate — curl fails loudly on the DNS gap, Chrome hides it. Test with the strict client first.

When I reach for what

Port-forward, NodePort, and Ingress aren't competitors. Different tools:

  • Ingress (Traefik + *.localhost) — default. Reproduces the prod path: host/path/TLS through a controller, human-friendly domains, load-balances across replicas. Configure once, forget it.
  • kubectl port-forward — a debug tunnel to a single pod. Dies when the pod is recreated. It's a probe, not access.
  • NodePort — the lazy fallback on 30000-32767. Least prod-like. I basically never use it; k3d docs warn wide NodePort ranges can hang the system on iptables churn. 
# everyday debug reflex — straight into a pod, bypassing the Service
kubectl port-forward -n checkout deploy/checkout 8080:8080
curl http://localhost:8080/healthz
Enter fullscreen mode Exit fullscreen mode

Guardrail

When local networking "flakes," check three things in order before touching the cluster:

  1. Is a controller actually running? A bare Ingress does nothing on its own.
  2. Was the -p "HOST:80@loadbalancer" mapping set at creation? It's creation-time only — recreate if not.
  3. What client are you testing with? Chrome hides the DNS gap; curl and Safari expose it. Add the /etc/hosts line.

What I'd do differently

Put the port mapping and the /etc/hosts line in the cluster bootstrap script from day one, so nobody recreates a cluster five times to learn what a controller is. And test with curl first, not Chrome — Chrome's free *.localhost resolution papers over a gap CI will hit anyway.

Bottom line: none of these were bugs — controller, creation-time port mapping, and client-side .localhost resolution are three separate layers, and knowing which one you're on turns two lost days into two minutes.

Sources

Top comments (0)