DEV Community

Cover image for From “Why Is This Pod Pending?” to `HTTP/1.1 200 OK`: My NGINX Gateway Fabric Lab
Adeoye Malumi
Adeoye Malumi

Posted on

From “Why Is This Pod Pending?” to `HTTP/1.1 200 OK`: My NGINX Gateway Fabric Lab

#devops #kubernetes #learning

A beginner-friendly Kubernetes networking story about Gateway API, Services, HTTPRoutes, DNS, a few misleading errors, and the joy of finally seeing a successful response.

The mission

The goal of this exercise was simple to say and slightly less simple to do: deploy NGINX Gateway Fabric in my Kubernetes cluster, create a small NGINX web application, and make the application accessible through a Kubernetes Gateway and HTTPRoute.

The intended request path was:

My Kali terminal
      |
      | HTTP request to NodePort :30080
      v
NGINX Gateway
      |
      | HTTPRoute matches path /
      v
Kubernetes Service: frontend-svc:80
      |
      v
Pod: frontend-app (NGINX listens on port 80)
      |
      v
HTTP/1.1 200 OK
Enter fullscreen mode Exit fullscreen mode

Think of it like sending a parcel. The Gateway is the front desk, the HTTPRoute is the instruction that says where the parcel should go, the Service is the stable address for the destination, and the Pod is the actual person receiving it.

This post walks through what I did, the problems I hit, and how I worked through them. The useful lesson was not just getting the page to load. It was learning to test one layer at a time instead of randomly changing things until Kubernetes takes pity on me.

1. Installing the Gateway API building blocks

I started in my nginx-gateway-fabric working directory and applied the standard Gateway API CRDs:

kubectl apply -k config/crd/gateway-api/standard
Enter fullscreen mode Exit fullscreen mode

CRD means CustomResourceDefinition. Kubernetes comes with built-in resources such as Pods, Deployments, and Services. CRDs teach Kubernetes about additional resource types. The Gateway API CRDs add resources such as GatewayClass, Gateway, and HTTPRoute.

I then installed the NGINX Gateway Fabric-specific CRDs and the NodePort deployment for version 1.6.1:

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/crds.yaml

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/nodeport/deploy.yaml
Enter fullscreen mode Exit fullscreen mode

The NodePort deployment gave me a way to reach the Gateway using a port on a Kubernetes node. In this lab, HTTP was exposed on port 30080.

Problem 1: Kustomize timed out while fetching the CRDs

My first attempt to fetch the Gateway API CRDs directly from GitHub using kubectl kustomize failed with a Git fetch timeout. Because the command failed to produce the manifests, the following kubectl apply command reported that no objects had been passed to it.

That second message was a consequence of the first failure, not a separate mystery.

What I did: I worked with the repository locally and applied the Kustomize directory from the checked-out source:

kubectl apply -k config/crd/gateway-api/standard
Enter fullscreen mode Exit fullscreen mode

Lesson: When a command downloads configuration and then applies it, check the download/rendering step first. If it produces nothing, kubectl apply cannot magically apply it.

2. The Gateway Pod was Pending

After applying the NGINX Gateway Fabric deployment, I checked the Pod:

kubectl get pods -n nginx-gateway
Enter fullscreen mode Exit fullscreen mode

At first, it showed Pending. My first attempt to describe it failed too:

kubectl describe pod nginx-gateway-55f96886f7-j4kzq
Enter fullscreen mode Exit fullscreen mode

The Pod was not found because I had left out its namespace. It lived in nginx-gateway, not the default namespace.

The corrected command was:

kubectl describe pod nginx-gateway-55f96886f7-j4kzq -n nginx-gateway
Enter fullscreen mode Exit fullscreen mode

The important event was:

0/2 nodes are available: 2 node(s) had untolerated taint(s).
Enter fullscreen mode Exit fullscreen mode

What does that mean?

A taint is a rule on a Kubernetes node that says, in effect, “Don't schedule Pods here unless they tolerate this condition.” A toleration is a matching permission in a Pod's specification.

The Gateway Pod was not necessarily broken. Kubernetes was saying it could not find a node where the Pod was allowed to run. I also saw that k8s-worker1 briefly became NotReady before returning to Ready.

What I learned: Check the node status and the Pod's scheduling events before changing the deployment. Taints may be protecting the cluster for a reason, so don't remove them blindly just to make a Pod start.

Useful commands:

kubectl get nodes -o wide
kubectl describe pod nginx-gateway-55f96886f7-j4kzq -n nginx-gateway
kubectl describe node k8s-worker1
Enter fullscreen mode Exit fullscreen mode

Eventually, the Gateway Pod was scheduled to k8s-worker1 and progressed toward running.

3. Creating the frontend application

Next, I created a simple NGINX Pod:

kubectl run frontend-app --image=nginx --port=8080
Enter fullscreen mode Exit fullscreen mode

One subtle detail caused a later problem: --port=8080 does not reconfigure NGINX to listen on port 8080. It records container-port metadata in the Pod specification. The NGINX image still listens on port 80 by default.

I exposed the Pod as a Service:

kubectl expose pod frontend-app \
  --name=frontend-svc \
  --port=80 \
  --target-port=8080
Enter fullscreen mode Exit fullscreen mode

A Service gives a stable network address to a group of Pods. This is useful because Pods can be replaced, and their IP addresses can change. Clients can keep talking to the Service instead of tracking individual Pod IPs.

I checked the Service and its endpoints:

kubectl describe svc frontend-svc -n default
kubectl get endpoints frontend-svc -n default
Enter fullscreen mode Exit fullscreen mode

Initially, the endpoint showed port 8080. That looked consistent with the Service configuration, but it was not consistent with the port NGINX actually used.

Problem 2: The Service pointed to the wrong port

I tested NGINX from inside its own Pod:

kubectl exec frontend-app -n default -- \
  curl -I --max-time 5 http://127.0.0.1:8080/
Enter fullscreen mode Exit fullscreen mode

It failed:

curl: (7) Failed to connect to 127.0.0.1 port 8080
Enter fullscreen mode Exit fullscreen mode

Instead of guessing, I tested the default NGINX port:

kubectl exec frontend-app -n default -- \
  curl -I --max-time 5 http://127.0.0.1:80/
Enter fullscreen mode Exit fullscreen mode

That returned:

HTTP/1.1 200 OK
Server: nginx/1.31.7
Enter fullscreen mode Exit fullscreen mode

There was the clue. NGINX was alive; I had simply sent it to the wrong port.

Fix: I changed the Service's targetPort to 80:

kubectl patch svc frontend-svc -n default \
  -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'
Enter fullscreen mode Exit fullscreen mode

Then I checked it again:

kubectl describe svc frontend-svc -n default
Enter fullscreen mode Exit fullscreen mode

The endpoint now pointed to the Pod on port 80.

Lesson: The Service's port is the port clients use to reach the Service. The targetPort is the port on the backend Pod that receives the traffic. Those values can be different, but targetPort must match the application that is actually listening.

4. Building the HTTPRoute

The next step was to tell the Gateway where to send incoming HTTP requests. I created an HTTPRoute called frontend-route.

The important parts of the route were:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend-route
  namespace: default
spec:
  parentRefs:
    - name: nginx-gateway
      namespace: nginx-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend-svc
          port: 80
Enter fullscreen mode Exit fullscreen mode

In plain English:

  • Attach this route to the nginx-gateway Gateway in the nginx-gateway namespace.
  • Match requests whose path starts with /.
  • Forward matching requests to the Service named frontend-svc on port 80.

I checked the result with:

kubectl describe httproute frontend-route -n default
Enter fullscreen mode Exit fullscreen mode

The good news was:

Type:    Accepted
Status:  True

Type:    ResolvedRefs
Status:  True
Message: All references are resolved
Enter fullscreen mode Exit fullscreen mode

These conditions mean the Gateway accepted the route and Kubernetes could resolve its references. They are useful signals, but I still needed to test actual traffic.

Problem 3: The HTTPRoute could not find the backend Service

My first route was in the nginx-gateway namespace and referred to frontend-svc. Kubernetes returned:

BackendNotFound
spec.rules[0].backendRefs[0].name: Not found: "frontend-svc"
Enter fullscreen mode Exit fullscreen mode

I had created frontend-svc in the default namespace. A Service reference in an HTTPRoute is normally resolved in the route's own namespace. Giving the Service the correct name was not enough; the namespace mattered too.

I edited the route YAML and placed the working frontend-route in default, while keeping its parentRefs pointing to the Gateway in nginx-gateway.

That fixed the reference. The route then reported both Accepted=True and ResolvedRefs=True.

Lesson: In Kubernetes, always check both the resource's name and namespace. Two resources can share a name in different namespaces, and a route can be accepted while still having an unresolved backend reference.

5. Testing the Service before testing the Gateway

Before troubleshooting the Gateway, I tested the Service directly. First, I checked that CoreDNS was running:

kubectl get pods -n kube-system
kubectl get svc -n kube-system kube-dns
Enter fullscreen mode Exit fullscreen mode

Then I created a short-lived diagnostic Pod:

kubectl run dns-test --image=busybox:1.36 \
  --restart=Never --command -- sleep 300

kubectl wait --for=condition=Ready pod/dns-test \
  -n default --timeout=60s
Enter fullscreen mode Exit fullscreen mode

I inspected its DNS settings:

kubectl exec dns-test -- cat /etc/resolv.conf
Enter fullscreen mode Exit fullscreen mode

The file showed the cluster DNS Service IP, 10.96.0.10. I then tested DNS resolution:

kubectl exec dns-test -- \
  nslookup kubernetes.default.svc.cluster.local

kubectl exec dns-test -- \
  nslookup frontend-svc.default.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Both names resolved. The frontend Service resolved to 10.104.158.51.

Problem 4: One curl test could not resolve the Service name

An earlier temporary curl Pod returned:

curl: (6) Could not resolve host: frontend-svc.default.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

That was confusing because the DNS Service and CoreDNS Pods looked healthy.

Rather than restarting DNS immediately, I used a separate diagnostic Pod to test the basics. Its nslookup commands successfully resolved both the Kubernetes API Service and frontend-svc. Then I tested the Service itself:

kubectl exec dns-test -- wget -S -O- \
  http://frontend-svc.default.svc.cluster.local/
Enter fullscreen mode Exit fullscreen mode

This returned the default NGINX welcome page and an HTTP/1.1 200 OK response.

What I can conclude: DNS resolution and Service connectivity worked from the diagnostic Pod. The earlier curl test failed, but the evidence did not establish its exact cause. The right move was to reproduce the test and inspect the layers separately rather than assume CoreDNS was broken.

6. The Gateway was programmed, but my first request returned a 500

I checked the Gateway and its Service:

kubectl get svc -n nginx-gateway
kubectl describe gateway nginx-gateway -n nginx-gateway
Enter fullscreen mode Exit fullscreen mode

The Service showed:

TYPE       NodePort
PORT(S)    80:30080/TCP,443:30081/TCP
Enter fullscreen mode Exit fullscreen mode

The Gateway status showed Accepted=True and Programmed=True. That meant the Gateway configuration had been accepted and programmed, but an end-to-end test was still needed.

I checked the route in nginx-gateway and discovered that a second route with the same name existed there. Unlike the working route in default, this duplicate route still referenced a backend Service that was not in its namespace. Its status included BackendNotFound.

I deleted the broken duplicate route:

kubectl delete httproute frontend-route -n nginx-gateway
Enter fullscreen mode Exit fullscreen mode

I kept the working route in default.

My first request to the NodePort returned 500 Internal Server Error. After removing the broken duplicate route, I tried again. The request succeeded.

The logs show that the bad duplicate route existed and that the response changed after I removed it. That makes the duplicate route a strong suspect in the earlier failure, although the response alone does not prove it was the only cause.

Problem 5: Two routes, one name, different namespaces

Kubernetes permits resources with the same name in different namespaces. That is not automatically an error, but it made this troubleshooting session confusing: one frontend-route was correctly configured, while the other had an unresolved backend.

Lesson: When a route behaves unexpectedly, check all routes across namespaces:

kubectl get httproute -A
Enter fullscreen mode Exit fullscreen mode

Don't check only the namespace where you expect the route to live.

7. The final test: success through the NodePort

I checked the node addresses:

kubectl get nodes -o wide
Enter fullscreen mode Exit fullscreen mode

The worker node's internal IP was 10.0.2.4. The Gateway's HTTP NodePort was 30080, so I tested:

curl -i --max-time 10 http://10.0.2.4:30080/
Enter fullscreen mode Exit fullscreen mode

At first, I mistakenly typed the placeholder literally:

curl -i --max-time 10 http://WORKER_IP:30080/
Enter fullscreen mode Exit fullscreen mode

The terminal returned Could not resolve host: WORKER_IP. The shell did exactly what I asked: it treated WORKER_IP as a hostname. Placeholders must be replaced with actual values before running a command.

The final request returned:

HTTP/1.1 200 OK
Server: nginx
Content-Length: 896
Enter fullscreen mode Exit fullscreen mode

And, below the headers, the familiar “Welcome to nginx!” HTML page appeared.

That was the end-to-end confirmation I wanted: a request from my Kali terminal entered through the node's NodePort, reached the NGINX Gateway, matched the HTTPRoute, and was forwarded to the frontend Service and Pod.

Top comments (0)