DEV Community

Cover image for Debugging a Broken Ingress Setup: A Step-by-Step Walkthrough
Adeoye Malumi
Adeoye Malumi

Posted on

Debugging a Broken Ingress Setup: A Step-by-Step Walkthrough

#kubernetes #cloud #devops #learning

This post documents a real debugging session that started with a simple symptom — example.com wouldn't load — and ended up uncovering several separate, unrelated problems layered on top of each other. It's written to walk through the actual diagnostic process, not just the final fix, since the process is usually the more useful part to learn from.

Starting point: an Ingress setup (from a previous post) was working, then stopped working after some changes were made mid-session, including installing a different CNI (Calico) on top of the cluster's existing one. This is the story of tracking that down.

The Symptom 

❯ curl example.com
curl: (7) Failed to connect to example.com port 80 after 21007 ms: Could not connect to server
Enter fullscreen mode Exit fullscreen mode

A connection timeout like this can mean a lot of different things: DNS pointing to the wrong place, a firewall silently dropping packets, a service that's down, or a machine that simply can't reach the target network at all. The rest of this post is the process of ruling each of those out, one at a time.

Step 1: Confirm What the Hostname Actually Resolves To

Before assuming anything about networking, the first thing to check is whether example.com is even resolving to the IP it's supposed to:

❯ getent hosts example.com
10.96.101.127   example.com
Enter fullscreen mode Exit fullscreen mode

This confirmed the local /etc/hosts entry was working correctly — example.com resolved to 10.96.101.127, matching the Ingress controller's IP from earlier setup. So the problem wasn't DNS/hosts resolution. The IP itself was the next thing to check.

Step 2: Test the IP Directly, Bypassing the Hostname

If the hostname resolves correctly but the connection still fails, the next step is to test the raw IP directly — this removes DNS/hosts from the equation entirely and asks: "is this address reachable at all?"

❯ curl -v http://10.96.101.127
*   Trying 10.96.101.127:80...
* connect to 10.96.101.127 port 80 from 10.0.2.15 port 57576 failed: Connection refused
* Failed to connect to 10.96.101.127 port 80 after 21003 ms: Could not connect to server
Enter fullscreen mode Exit fullscreen mode

Two useful details came out of this:

  1. Connection refused, not a timeout. This is a meaningful difference. A timeout usually means packets are going out and nothing is answering. "Refused" happens quickly and means something did respond — but actively said "no."
  2. The source IP was 10.0.2.15. This is VirtualBox's default address for a guest machine in NAT networking mode, which confirmed this was being tested from inside a VM.

The actual explanation: 10.96.101.127 is a Kubernetes ClusterIP. A ClusterIP isn't a real address that exists anywhere on the network — it's a virtual IP that only works because of routing rules (iptables/IPVS) that Kubernetes' kube-proxy installs inside the specific machine running the cluster. No other machine — not a VM, not a laptop on the same network — can ever reach a ClusterIP directly, because that address doesn't exist anywhere outside that one machine's own routing table.

This meant the original approach (pointing /etc/hosts at the ClusterIP) was never going to work from a separate machine, regardless of any other fix.

Step 3: Find a Real, Reachable Address Instead

Since ClusterIPs don't work across machines, the next step was finding an address that does — specifically, the IP of one of the actual Docker containers backing the kind cluster's nodes, combined with the Ingress controller's NodePort (a real port opened on the node itself, unlike a ClusterIP).

❯ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED      STATUS          PORTS                       NAMES
691fe58f2715   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes   127.0.0.1:38079->6443/tcp   cka-new-control-plane
aae68ca53b7b   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker2
b51bce23b84f   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker
75634b82514e   kindest/node:v1.35.0   "/usr/local/bin/entr…"   2 days ago   Up 18 minutes                               cka-new-worker3
Enter fullscreen mode Exit fullscreen mode

None of the worker nodes had any ports published to the host — only the control-plane's Kubernetes API port was mapped. So the node's actual IP address on Docker's internal network was needed instead:

❯ docker inspect cka-new-worker | grep -A 15 '"kind"'
...
"IPAddress": "172.18.0.5",
...
Enter fullscreen mode Exit fullscreen mode

A discrepancy showed up here worth noting: this IP (172.18.0.5) didn't match what kubectl get nodes -o wide had shown earlier in the session for this same node (172.18.0.3). Docker assigns container IPs dynamically, and since these containers showed Up 18 minutes (meaning they had recently restarted), the IP had simply changed. This is a good general reminder that hardcoding container IPs is fragile — they aren't guaranteed to stay the same across restarts.

Testing the NodePort against this IP, however, still failed:

❯ curl -v http://172.18.0.5:31003 -H "Host: example.com"
*   Trying 172.18.0.5:31003...
* connect to 172.18.0.5 port 31003 from 172.18.0.1 port 53748 failed: Connection timed out
Enter fullscreen mode Exit fullscreen mode

So even with a real, reachable-in-theory IP and the correct port, nothing answered. This meant the problem was something deeper than just "wrong IP" — worth investigating the cluster's internal health next.

Step 4: Ruling Out the Obvious Culprits

At this point, several standard checks were run to narrow down where the actual fault was.

Was the Ingress controller pod even healthy?

❯ kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS      AGE   IP           NODE
ingress-nginx-controller-7d65c586d6-kvn9s   1/1     Running   3 (20m ago)   16h   10.244.1.2   cka-new-worker2
Enter fullscreen mode Exit fullscreen mode

It was Running, but had restarted 3 times recently — worth remembering, but not conclusive on its own.

Was kube-proxy (the component responsible for NodePort/ClusterIP routing) healthy on every node?

❯ kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
NAME               READY   STATUS    RESTARTS      AGE     NODE
kube-proxy-2mlbd   1/1     Running   4 (25m ago)   2d18h   cka-new-worker
kube-proxy-jb2rt   1/1     Running   4             2d18h   cka-new-worker2
kube-proxy-m2vq9   1/1     Running   4             2d18h   cka-new-control-plane
kube-proxy-rnlwc   1/1     Running   4             2d18h   cka-new-worker3
Enter fullscreen mode Exit fullscreen mode

All Running, though cka-new-worker had also restarted recently.

Was there a conflict between two CNI plugins (since Calico had been installed on top of the cluster's default kindnet networking earlier)?

❯ docker exec cka-new-worker2 ls /etc/cni/net.d/
10-kindnet.conflist
Enter fullscreen mode Exit fullscreen mode

Only one CNI config was present — no conflict there, at least on this node.

Did basic pod-to-pod networking inside the cluster even work at all?

❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world
Enter fullscreen mode Exit fullscreen mode

This was the first real breakthrough. This same command had worked earlier in the setup process — now it couldn't even resolve the Service's name. This shifted the investigation toward DNS specifically.

Step 5: Investigating the DNS Failure

First, check whether restarting kube-proxy (to clear out any stale routing rules) fixes it:

❯ kubectl delete pods -n kube-system -l k8s-app=kube-proxy
pod "kube-proxy-2mlbd" deleted from kube-system namespace
pod "kube-proxy-jb2rt" deleted from kube-system namespace
pod "kube-proxy-m2vq9" deleted from kube-system namespace
pod "kube-proxy-rnlwc" deleted from kube-system namespace
Enter fullscreen mode Exit fullscreen mode

After waiting for the new pods to come back Running:

❯ kubectl run curl --rm -it --image=curlimages/curl -- curl http://hello-world
curl: (6) Could not resolve host: hello-world
Enter fullscreen mode Exit fullscreen mode

No change. This ruled out stale kube-proxy rules as the cause.

Check whether the cluster's internal DNS service (CoreDNS) has healthy endpoints behind it:

❯ kubectl get endpoints -n kube-system kube-dns
NAME       ENDPOINTS                                                   AGE
kube-dns   10.244.0.2:9153,10.244.0.3:9153,10.244.0.2:53 + 3 more...   2d18h
Enter fullscreen mode Exit fullscreen mode

This looked healthy — CoreDNS pods were properly registered.

Check whether a real pod is even configured to use the right DNS server:

❯ kubectl exec -it hello-world-c8b6f8dbd-9t9n6 -- sh
# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
Enter fullscreen mode Exit fullscreen mode

This was also correct — 10.96.0.10 is the standard internal DNS Service IP.

So far: correct DNS configuration, healthy-looking CoreDNS, no stale kube-proxy rules — yet DNS lookups still failed. The next step was testing whether this was actually a DNS-specific issue, or something more fundamental with networking between nodes.

Step 6: Isolating DNS vs. General Networking

Test a plain HTTP connection to a Service, bypassing DNS entirely (using the raw ClusterIP):

# python3 -c "import urllib.request; print(urllib.request.urlopen('http://10.96.74.127').read())"
b'Hello, World!'
Enter fullscreen mode Exit fullscreen mode

This worked. So general Service routing (HTTP, TCP) was fine — the problem was specific to DNS.

Test raw connectivity to the DNS port itself (port 53), both against the Service IP and a CoreDNS pod's IP directly:

# python3 -c "
import socket
try:
    s = socket.create_connection(('10.96.0.10', 53), timeout=3)
    print('connected')
    s.close()
except Exception as e:
    print('failed:', e)
"
failed: timed out
Enter fullscreen mode Exit fullscreen mode 
# python3 -c "
import socket
try:
    s = socket.create_connection(('10.244.0.2', 53), timeout=3)
    print('connected')
    s.close()
except Exception as e:
    print('failed:', e)
"
failed: timed out
Enter fullscreen mode Exit fullscreen mode

Both failed — even a direct connection attempt to a CoreDNS pod's own IP, with no DNS resolution or Service routing involved at all, still timed out.

The key detail that explained this: checking which node each pod involved was running on showed that the working test (10.96.74.127, resolving to the hello-world pod) happened to be a pod on the same node the test was run from. The failing tests (both CoreDNS IPs) were both on the control-plane node — a different node entirely. This pointed toward a cross-node networking problem, not a DNS-specific one.

Step 7: Confirming Cross-Node Networking Was Broken

Checked whether the Ingress controller's NodePort Service had a networking policy that might explain uneven behavior across nodes:

❯ kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.externalTrafficPolicy}'
Local
Enter fullscreen mode Exit fullscreen mode

This turned out to be a real, separate finding: externalTrafficPolicy: Local means kube-proxy will only forward NodePort traffic on a node if that node happens to be running a copy of the target pod locally — every other node will actively drop the traffic rather than forward it elsewhere. This fully explained the earlier NodePort timeout, since the Ingress controller pod only ran on cka-new-worker2, and the test had hit a different node.

Confirmed by checking iptables on the failing node:

❯ docker exec cka-new-worker3 iptables-save | grep -i "53\|drop" | head -30
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "ingress-nginx/ingress-nginx-controller:http has no local endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 31003 -j DROP
Enter fullscreen mode Exit fullscreen mode

This line confirms it directly: any NodePort traffic to port 31003 on this node is explicitly dropped, because this node has no local copy of the Ingress controller pod.

Fixing the NodePort routing issue was straightforward — hit the correct node directly:

❯ curl -v http://172.18.0.4:31003 -H "Host: example.com"
* Connected to 172.18.0.4 (172.18.0.4) port 31003
< HTTP/1.1 504 Gateway Time-out
Enter fullscreen mode Exit fullscreen mode

The connection succeeded this time — but nginx itself returned a 504 Gateway Time-out. This meant the request reached the Ingress controller fine, but the Ingress controller couldn't reach its own backend (the hello-world Service). And, notably, the Ingress controller pod and the hello-world pod were running on two different nodes — another cross-node hop, timing out in exactly the same pattern as the DNS failures.

This was the real conclusion: same-node pod-to-pod traffic worked consistently throughout this session. Cross-node pod-to-pod traffic (DNS lookups to CoreDNS, and now the Ingress controller reaching its backend) consistently failed. Along the way, other explanations were ruled out — no competing CNI configs, no blocking NetworkPolicy, correctly configured DNS, correct iptables NAT rules, and Calico's own control-plane pods weren't even running:

❯ kubectl get pods -n calico-system -o wide
No resources found in calico-system namespace.
Enter fullscreen mode Exit fullscreen mode

The most likely explanation: installing Calico mid-way through the cluster's life (on top of the existing kindnet CNI, without removing it) had left the routes between nodes in a broken, inconsistent state — even after Calico itself was no longer running. Untangling that kind of mixed-CNI networking state by hand is generally not worth the time; the standard, reliable fix is to rebuild the cluster from scratch.

Step 8: Rebuilding the Cluster 

❯ kind delete cluster --name cka-new
Enter fullscreen mode Exit fullscreen mode

To make future testing easier, a kind config file was created to forward the Ingress controller's expected ports straight from the container to localhost on the host machine — removing the need to look up container IPs at all going forward:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: cka-new
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 31003
        hostPort: 31003
        protocol: TCP
      - containerPort: 30437
        hostPort: 30437
        protocol: TCP
  - role: worker
  - role: worker
  - role: worker
Enter fullscreen mode Exit fullscreen mode 
❯ kind create cluster --name cka-new --config kind-config.yaml
Enter fullscreen mode Exit fullscreen mode

This time, Calico was intentionally left out — the default kindnet CNI had been working fine before Calico was introduced mid-session, and nothing about the setup actually required Calico's features.

Reapplying the application and Ingress manifests:

❯ kubectl apply -f deployment.yaml -f service.yaml -f ingress.yaml
❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/aws/deploy.yaml
Enter fullscreen mode Exit fullscreen mode

Step 9: A Slow Pod Isn't Always a Stuck Pod

While waiting for the Ingress controller to come up, it sat in ContainerCreating for several minutes, which raised the question of whether something was actually wrong:

❯ kubectl get pods -n ingress-nginx -w
ingress-nginx-controller-7d65c586d6-fz54r   0/1   ContainerCreating   0   2m59s
Enter fullscreen mode Exit fullscreen mode

Checking the pod's events showed the actual cause:

❯ kubectl describe pod -n ingress-nginx ingress-nginx-controller-7d65c586d6-fz54r
...
Events:
  Warning  FailedMount  4m41s (x8 over 5m45s)  kubelet  MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      3m30s                  kubelet  Pulling image "registry.k8s.io/ingress-nginx/controller:v1.15.1@sha256:..."
Enter fullscreen mode Exit fullscreen mode

This looked alarming at first, but both issues were self-resolving, expected steps of the install process rather than actual problems:

  • The FailedMount warnings happened because two setup Jobs (ingress-nginx-admission-create and ingress-nginx-admission-patch) needed to finish first — they're responsible for generating the certificate secret the controller pod was waiting to mount. Until those Jobs completed, that secret simply didn't exist yet.
  • The image pull itself was just slow — checking kubectl get events showed the two setup Jobs' images had each taken over a minute to pull, which suggested a slow registry pull rather than anything broken locally.

Waiting it out confirmed this:

❯ kubectl get pods -n ingress-nginx
ingress-nginx-controller-7d65c586d6-fz54r   1/1   Running   0   7m34s
Enter fullscreen mode Exit fullscreen mode

Lesson here: not every ContainerCreating or warning event means something is broken — checking the actual Events section (via kubectl describe pod) before assuming failure saves a lot of unnecessary troubleshooting.

Step 10: Fixing the Service Configuration Correctly

With the controller running, the externalTrafficPolicy and Service type still needed to be set correctly, to avoid re-hitting the earlier Local-policy and LoadBalancer-pending issues.

The first attempt patched only externalTrafficPolicy:

❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller   LoadBalancer   10.96.61.252   <pending>     80:32087/TCP,443:32290/TCP
Enter fullscreen mode Exit fullscreen mode

This revealed two remaining issues: the Service was still type: LoadBalancer (stuck <pending>, same problem as before), and the NodePorts had landed on different numbers (32087/32290) than what kind-config.yaml was forwarding (31003/30437).

Fixing both at once, by explicitly setting the type to NodePort and pinning the exact port numbers:

❯ kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","protocol":"TCP","nodePort":31003},{"name":"https","port":443,"targetPort":"https","protocol":"TCP","nodePort":30437}]}}'
Enter fullscreen mode Exit fullscreen mode 
❯ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME                       TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller   NodePort   10.96.61.252   <none>        80:31003/TCP,443:30437/TCP
Enter fullscreen mode Exit fullscreen mode

This matched the port forwarding configured in kind-config.yaml.

Step 11: Confirming Everything Works 

❯ curl -v http://localhost:31003 -H "Host: example.com"
*   Trying [::1]:31003...
* connect to ::1 port 31003 from ::1 port 49216 failed: Connection refused
*   Trying 127.0.0.1:31003...
* Connected to localhost (127.0.0.1) port 31003
< HTTP/1.1 200 OK
Hello, World!
Enter fullscreen mode Exit fullscreen mode

This confirmed everything was working correctly: the port forwarding in kind-config.yaml, the Cluster traffic policy correctly routing regardless of which node the request landed on, and the Ingress controller successfully reaching the hello-world backend (no more 504 Gateway Time-out).

The ::1 (IPv6) connection attempt failing before falling back to IPv4 is expected — kind-config.yaml only forwards the port over IPv4, so curl's dual-stack attempt simply skips it and retries over IPv4 instead. Not an error to fix.

Final step — update /etc/hosts, this time pointing at 127.0.0.1 instead of any container IP, since port forwarding now handles the routing:

127.0.0.1   example.com
Enter fullscreen mode Exit fullscreen mode 
❯ curl http://example.com:31003
Hello, World!
Enter fullscreen mode Exit fullscreen mode

Why the Port Still Has to Be Specified

Running curl http://example.com without a port defaults to port 80 — but nothing is mapped to host port 80 in this setup, only 31003 and 30437. Without a port, the connection fails immediately:

curl: (7) Failed to connect to example.com port 80 after ...: Connection refused
Enter fullscreen mode Exit fullscreen mode

This is expected: the Ingress controller is exposed via a NodePort Service, and NodePorts are, by design, high-numbered ports (typically in the 30000–32767 range) — they were never going to answer on plain port 80 without an explicit mapping.

To make curl http://example.com work without a port number, kind-config.yaml could instead map host port 80 directly to the NodePort:

extraPortMappings:
  - containerPort: 31003
    hostPort: 80
    protocol: TCP
Enter fullscreen mode Exit fullscreen mode

This requires a cluster rebuild to take effect, and on Linux, binding to port 80 may require elevated (sudo) privileges since ports below 1024 are privileged. For local testing and practice environments, simply including the NodePort in the URL is generally the more common and simpler approach — production clusters solve the "no port number" problem differently anyway, using a real external load balancer or a bare-metal solution like MetalLB in front of the cluster.

Top comments (0)