DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

JumpToTech DevOps School ## HOMEWORK — Production Incident Response: Kubernetes HTTP 503

#kubernetes #devops #sre #networking

Course: DevOps + AI Engineering

Topic: Kubernetes Troubleshooting, Service Networking, EndpointSlices, Incident Response

Difficulty: Beginner to Intermediate

Estimated time: 60–90 minutes

Role: DevOps Engineer / Site Reliability Engineer

1. Production Incident Ticket

Ticket ID: INC-503-001

Priority: P2 — High

Status: OPEN

Environment: Production Simulation

Application: Restaurant Company

Namespace: production-lab

Service: restaurant-service

Reported by: Customer Support

Assigned to: DevOps / SRE Team

Customer Complaint

Our customers are unable to access the Restaurant Company application. The website is not responding correctly. Please investigate and restore service as soon as possible.

Business Impact:

Customers may be unable to access the restaurant menu, browse products, or place orders.

Your Responsibility:

You are the on-call DevOps Engineer. Your job is to investigate the incident, identify the root cause, restore application connectivity, verify the recovery, and document your findings.

Important: This lab simulates an application outage. An actual HTTP 503 response is not guaranteed by the Kubernetes Service failure alone. Different network components may return a connection error, timeout, or HTTP 503.

2. Learning Objectives

By the end of this homework, you should be able to:

  1. Understand how Kubernetes Pods, Deployments, Services, labels, selectors, and EndpointSlices work together.
  2. Explain the difference between a healthy Pod and an accessible application.
  3. Investigate a production incident using kubectl.
  4. Identify a Service selector mismatch.
  5. Understand why a Service can have zero endpoints while Pods are running.
  6. Restore connectivity without unnecessarily restarting healthy Pods.
  7. Verify application recovery.
  8. Prepare a professional incident report.

3. Architecture Overview

Our application uses the following architecture:

              Customer / Client
                      |
                      v
              HTTP Request
                      |
                      v
           restaurant-service
              ClusterIP :80
                      |
              Service Routing
                      |
             +--------+--------+
             |                 |
             v                 v
        Restaurant Pod 1  Restaurant Pod 2
           NGINX :80        NGINX :80
Enter fullscreen mode Exit fullscreen mode

Kubernetes components involved:

Deployment: Maintains the desired number of application replicas.

Pods: Run the NGINX application containers.

Service: Provides a stable virtual IP and port for accessing selected backend Pods.

Labels: Identify Kubernetes resources.

Service Selector: Determines which Pods belong to a Service.

EndpointSlice Controller: Maintains EndpointSlice resources that describe matching backend endpoints and their readiness.

kube-proxy: In clusters using kube-proxy, watches Services and EndpointSlices and configures the node's network forwarding rules.

CoreDNS: Resolves Kubernetes Service names into IP addresses. It does not select the backend Pod or forward HTTP requests.

4. Phase 1 — Prepare Your Environment

Before beginning, verify that kubectl is connected to your authorized Kubernetes cluster.

kubectl cluster-info

kubectl get nodes

kubectl get namespaces
Enter fullscreen mode Exit fullscreen mode

Create an isolated namespace for the homework:

kubectl create namespace production-lab
Enter fullscreen mode Exit fullscreen mode

If it already exists, reuse it only if it is your own dedicated training namespace. Do not modify another student's environment.

Verify:

kubectl get namespace production-lab
Enter fullscreen mode Exit fullscreen mode

Question 1: What is a Kubernetes Namespace? Why do companies separate development, staging, and production resources?

Question 2: When a real production incident occurs, do DevOps engineers normally create a new production namespace, or investigate the existing one?

5. Phase 2 — Deploy a Healthy Application

Create a file named:

restaurant-lab.yaml

Add the following YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: restaurant
  namespace: production-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: restaurant
  template:
    metadata:
      labels:
        app: restaurant
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 2
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: restaurant-service
  namespace: production-lab
spec:
  selector:
    app: restaurant
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP
Enter fullscreen mode Exit fullscreen mode

Deploy the application:

kubectl apply -f restaurant-lab.yaml
Enter fullscreen mode Exit fullscreen mode

Wait for the Deployment:

kubectl rollout status deployment/restaurant \
  -n production-lab --timeout=180s
Enter fullscreen mode Exit fullscreen mode

Check the Pods:

kubectl get pods -n production-lab
Enter fullscreen mode Exit fullscreen mode

Check the Service:

kubectl get svc -n production-lab
Enter fullscreen mode Exit fullscreen mode

Check the EndpointSlices:

kubectl get endpointslices -n production-lab \
  -l kubernetes.io/service-name=restaurant-service \
  -o wide
Enter fullscreen mode Exit fullscreen mode

Expected Healthy State

  • Two NGINX Pods are running and ready.
  • The Service has a ClusterIP.
  • The Service has matching backend endpoints.
  • Requests to the application succeed.

Verify HTTP Access

Use an internal Kubernetes client:

kubectl run restaurant-http-test \
  -n production-lab \
  --rm -i \
  --restart=Never \
  --image=curlimages/curl \
  -- curl -sS -o /dev/null \
  -w 'HTTP Status: %{http_code}\n' \
  http://restaurant-service
Enter fullscreen mode Exit fullscreen mode

Expected:

HTTP Status: 200
Enter fullscreen mode Exit fullscreen mode

Optional Browser Access

If your instructor has configured an EC2 browser demonstration, you may use the instructor-provided EC2 address and port.

Remember that kubectl port-forward is a debugging tunnel. It is not the same as normal Service ClusterIP routing and should not be used as proof that Service endpoints are healthy.

Screenshot Required: Healthy Pods, Service, EndpointSlices, and HTTP 200 verification.

6. Phase 3 — Production Incident Begins

Your instructor or lab partner will introduce a controlled failure.

Students must not read the instructor-only fault-injection command until they have completed their investigation.

Incident Notification

Slack — #production-incidents

ALERT: Restaurant Company application connectivity failure.

Customer Support has reported that customers cannot access the restaurant application.

The Kubernetes cluster is reachable, but application connectivity is failing.

Investigate immediately and provide a root-cause analysis.

Your Mission

You have been assigned this ticket.

Do not immediately delete Pods, restart the Deployment, or recreate the Namespace.

Begin by collecting evidence.

7. Phase 4 — Troubleshooting Investigation

Task A — Check the Pods 

kubectl get pods -n production-lab
Enter fullscreen mode Exit fullscreen mode 
kubectl get pods -n production-lab -o wide
Enter fullscreen mode Exit fullscreen mode 

bash
kub
Enter fullscreen mode Exit fullscreen mode

Top comments (0)