DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

JumpToTech DevOps/SRE Bootcamp Production Incident Response Lab — Banking Application on AWS EKS

#devops #kubernetes #aws #sre

Kubernetes Incident Response — Banking Application on AWS EKS

One of the most critical incidents I handled involved a customer-facing banking application running on AWS EKS.

Our production environment had approximately 60 frontend replicas distributed across three Availability Zones, with an AWS Application Load Balancer handling incoming traffic. We used Prometheus and Grafana for monitoring and alerting.

During a production release, we started receiving alerts that the banking application's HTTP 5xx error rate had increased significantly. The error rate reached approximately 18%, and customers were experiencing intermittent failures when accessing their accounts.

I was the primary SRE responsible for investigating the incident and coordinating the technical recovery.

My first step was to assess the impact. I checked Grafana dashboards for error rates, request latency, and available replicas. I noticed that available frontend replicas had dropped from 60 to 38, which explained why the remaining Pods were struggling to handle traffic.

I then checked the EKS cluster using kubectl get nodes, kubectl get pods, and kubectl get deployments. All worker nodes were healthy, so I ruled out a cluster-wide infrastructure failure.

When I inspected the affected Pods using kubectl describe pod, I found repeated readiness-probe failures. The new containers were Running, but they were not becoming Ready.

I checked the Kubernetes events and application logs and found that the readiness probes were returning HTTP 404.

After comparing the new deployment configuration with the previous release, I identified the root cause: the readiness probe had been changed from / to /healthz, but the application did not expose the new endpoint.

I also found that the deployment configuration allowed too many healthy Pods to become unavailable during the rollout, which contributed to the loss of serving capacity.

At that point, my priority was restoring service rather than troubleshooting the defective release further.

I rolled back the deployment using kubectl rollout undo deployment/banking-frontend and monitored the recovery using kubectl rollout status.

As the previous application version came back online, available replicas increased from 38 to 60, and the HTTP 5xx error rate returned to its normal baseline.

I verified recovery by checking Kubernetes Service endpoints, application health checks, load balancer metrics, and Grafana dashboards.

The application was restored within approximately 12 minutes of incident detection.

After the incident, I completed a root cause analysis and recommended three improvements: automated readiness-probe validation in CI/CD, safer RollingUpdate settings with maxUnavailable: 0, and canary deployments to detect faulty releases before they affect the entire application.

The outcome was that I restored full application availability, reduced the immediate customer impact, and helped establish deployment safeguards to prevent the same failure from recurring.

This incident reinforced the importance of understanding Kubernetes readiness probes, deployment strategies, monitoring, and rapid rollback procedures in a high-availability banking environment.

Numbers in this example
Metric Value
Frontend replicas 60
Availability Zones 3
Available replicas during incident 38
Unavailable replicas 22
Peak HTTP 5xx error rate 18%
Readiness-probe error HTTP 404
Recovery time 12 minutes
Replicas after recovery 60
Recovery method Kubernetes rollback

These numbers form a consistent hypothetical production incident, not a historical incident verified at JPMorgan Chase. In a real interview, students should use the metrics from their own incident evidence rather than present these figures as their personal employment history.

What makes this a senior-level answer: It demonstrates that the engineer assessed customer impact, distinguished application failure from infrastructure failure, investigated Kubernetes events and logs, identified the configuration defect, performed a controlled rollback, verified recovery, and implemented preventive measures.
Student Lab Guide · Day 2

Scenario: Banking Application Outage After a Failed Production Deployment\
Environment: AWS EKS, Kubernetes, Prometheus, Grafana, NGINX\
Role: DevOps / Site Reliability Engineer\
Difficulty: Intermediate\
Estimated duration: 3 hours\
Incident severity: Simulated SEV-2 / P2

Important note for students

This lab uses a fictional banking incident inspired by the types of production problems that can occur in large financial institutions such as JPMorgan Chase. It is not a documented incident at Chase, and you must not claim that you worked at Chase or resolved this incident in its actual production environment.

You will, however, perform a real hands-on investigation in your own Kubernetes cluster. You can discuss that experience confidently in interviews as an independently resolved production-style lab incident.

1. Your role and responsibilities

You are a DevOps/SRE Engineer supporting a banking application running on Amazon EKS.

The application allows customers to access banking services. Your team is responsible for keeping the infrastructure reliable, deploying application releases, monitoring service health, and restoring service when an incident occurs.

During this lab, you are responsible for:

  1. Monitoring: Detect when application availability decreases.
  2. Incident response: Assess the impact and prioritize recovery.
  3. Troubleshooting: Investigate Deployments, ReplicaSets, Pods, Services, probes, events, and logs.
  4. Mitigation: Restore the last known healthy application version.
  5. Verification: Confirm application availability and healthy Kubernetes status.
  6. Communication: Provide clear updates to the incident commander and application team.
  7. Root cause analysis: Explain why the failure occurred and how to prevent recurrence.

Your objective is not simply to restart a Pod. Your objective is to restore customer-facing service safely, understand the failure, and reduce the chance of recurrence.

2. The incident you will investigate

INC-2026-001

Simulated SEV-2

Banking frontend unavailable after a Kubernetes deployment

Business impact: Customers may be unable to access the banking frontend during the failed rollout.

Technical symptoms: Deployment unavailable, Pods not Ready, application health checks failing.

Expected root cause: A newly released application version fails its readiness checks.

Recovery strategy: Roll back to the previous known-good Deployment revision.

What happened?

The development team released a new version of the banking frontend.

The new container starts, but a configuration defect causes its HTTP readiness endpoint to fail. Kubernetes therefore cannot mark the new Pods as Ready.

As the rollout progresses, the old working Pods are replaced. Eventually, the application has no Ready endpoints, and customers cannot access the service.

Prometheus detects the decrease in available replicas. Grafana displays the outage and can trigger an alert.

You are the engineer responsible for diagnosing the failure and restoring the previous working release.

What success looks like

At the end of the incident:

  • The Deployment is restored to its previous healthy revision.
  • Two frontend replicas are Ready.
  • The Kubernetes Service has healthy endpoints.
  • The application returns HTTP 200.
  • Monitoring shows recovery.
  • You have recorded the root cause, commands, evidence, and preventive actions.

Do not invent recovery times, financial losses, customer counts, or performance improvements. Measure your actual lab results and use those numbers in your interview explanation.

3. Understand the production architecture

Banking customers

Customer requests

Application access layer

Production would typically use HTTPS and an ALB or Ingress

Kubernetes Service

banking-service · ClusterIP · Port 80

Frontend Pod 1

NGINX · Port 80

Frontend Pod 2

NGINX · Port 80

Prometheus collects Kubernetes health metrics. Grafana visualizes them and evaluates alert rules.

In this lab, the frontend is represented by an NGINX container. It is not a complete banking system with authentication, payment processing, or a database. The incident concerns the frontend availability layer.

4. Lab prerequisites

From the previous lesson, you should already have:

Resource Expected state
EKS cluster Active
Worker nodes Ready
Namespace banking-incident-lab
Deployment banking-frontend
Replicas 2/2 Ready
Service banking-service
Prometheus Running
Grafana Running

Use the same cluster and namespace as yesterday. Do not delete or reinstall the monitoring stack.

Part A — Establish the healthy production baseline

Step 1. Verify your AWS and Kubernetes environment

Run these commands on the machine where your kubectl is configured.

kubectl config current-context

kubectl get nodes -o wide

kubectl get pods -n monitoring

kubectl get deployment -n banking-incident-lab

kubectl get pods -n banking-incident-lab -o wide

kubectl get svc -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

Expected Deployment:

NAME               READY   UP-TO-DATE   AVAILABLE
banking-frontend   2/2     2            2
Enter fullscreen mode Exit fullscreen mode

Student task: Explain why READY 2/2 matters and how Kubernetes determines whether a Pod is ready.

Step 2. Inspect the readiness probe 

kubectl describe deployment banking-frontend \
  -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

Look for:

Readiness: http-get http://:80/
Enter fullscreen mode Exit fullscreen mode

The readiness probe is configured to request / on port 80.

The kubelet runs the probe. Successful HTTP responses (status codes 200–399) allow the container to become Ready.

A failed readiness probe does not necessarily restart the container. Instead, the Pod becomes Not Ready and is excluded from normal Service traffic.

Step 3. Verify the Service endpoints 

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o wide
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get endpoints banking-service \
  -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

You should see two Pod addresses associated with the Service.

The EndpointSlice output is the preferred way to inspect endpoint readiness in modern Kubernetes. The legacy Endpoints command is included because it is still common in interviews and older runbooks.

Step 4. Verify the application responds 

kubectl run baseline-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -sS -o /dev/null \
  -w "HTTP %{http_code}\n" \
  --max-time 10 \
  http://banking-service
Enter fullscreen mode Exit fullscreen mode

Expected:

HTTP 200
Enter fullscreen mode Exit fullscreen mode

A temporary warning about attaching to the diagnostic Pod is acceptable if the curl command still returns HTTP 200.

Evidence to save: Terminal output showing two Ready replicas and a successful HTTP response.

Step 5. Save the healthy Deployment configuration

This is an essential production habit: preserve the current state before making changes.

mkdir -p ~/banking-incident-lab/evidence

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o yaml \
  > ~/banking-incident-lab/evidence/deployment-before.yaml

kubectl rollout history deployment/banking-frontend \
  -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

Do not manually edit the saved baseline file during the incident.

Part B — Configure Prometheus and Grafana

Step 6. Open Grafana

Use the Grafana instance you configured in the previous lesson.

If Grafana is not accessible, start the port-forward on the management machine:

kubectl port-forward \
  -n monitoring \
  svc/monitoring-grafana \
  3000:80 \
  --address 127.0.0.1
Enter fullscreen mode Exit fullscreen mode

If you access Grafana through an EC2 management machine, use your existing SSH tunnel to reach its localhost port securely.

Step 7. Create a production monitoring dashboard

In Grafana, go to Dashboards → New → New dashboard → Add visualization and select your Prometheus data source.

Create these four panels.

Panel Visualization Purpose
Available replicas Stat Number of available frontend replicas
Desired replicas Stat Number Kubernetes should maintain
Unavailable replicas Stat Detect loss of Deployment availability
Container restarts Time series Track application restarts

Panel 1 — Available replicas

kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
Enter fullscreen mode Exit fullscreen mode

Healthy value: 2

Panel 2 — Desired replicas

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
Enter fullscreen mode Exit fullscreen mode

Healthy value: 2

Panel 3 — Unavailable replicas

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
-
kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
Enter fullscreen mode Exit fullscreen mode

Healthy value: 0

Panel 4 — Container restarts

sum(
  kube_pod_container_status_restarts_total{
    namespace="banking-incident-lab",
    pod=~"banking-frontend-.*"
  }
) or vector(0)
Enter fullscreen mode Exit fullscreen mode

Healthy initial value: normally 0.

Save the dashboard as:

JumpToTech — Banking Production Incident Monitoring

If a query returns No Data, verify the Prometheus data source, kube-state-metrics, namespace, and Deployment labels before proceeding. Do not treat missing metrics as a healthy result.

Step 8. Create an incident alert

Navigate to Alerting → Alert rules → New alert rule.

Configure:

Field Value
Alert name BankingFrontendUnavailable
Data source Prometheus
Evaluation interval 1 minute
Pending period 1 minute
Condition Last value greater than 0
Severity label warning
Folder Banking Production Lab
Notification Your configured email contact point

Use this PromQL expression:

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
-
kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
Enter fullscreen mode Exit fullscreen mode

Set the alert condition to Last IS ABOVE 0. Depending on your Grafana version, this may require a Reduce expression and a Threshold expression.

Use these annotations:

Summary: Banking frontend has unavailable replicas.

Description: The banking-frontend Deployment has fewer available replicas than desired. Investigate Pod readiness, Deployment rollout status, Kubernetes events, and Service endpoints.

For the lab, use No Data = No Data and Error = Error so that missing metrics or evaluation failures are not silently treated as healthy. These states should be monitored separately in a real production alerting design.

Before proceeding, verify the alert is saved and that the email contact point is selected by the notification policy.

Part C — Simulate the production incident

Controlled incident simulation

The next commands deliberately make the banking frontend unavailable. Execute them only in your dedicated training namespace, never in a real production banking environment.

Verify that kubectl config current-context points to your training cluster before continuing.

Step 9. Understand the failure before creating it

Our healthy NGINX container responds to:

GET /
HTTP 200 OK
Enter fullscreen mode Exit fullscreen mode

We will introduce a bad readiness probe that requests a nonexistent endpoint:

GET /healthz
HTTP 404 Not Found
Enter fullscreen mode Exit fullscreen mode

NGINX will still be running, but the kubelet will mark the container Not Ready.

To make the full outage reproducible, we will temporarily use the Deployment's Recreate strategy. This intentionally removes the old Pods before starting replacements. This is not a recommended production rollout strategy for a high-availability banking frontend.

This combination demonstrates two failures: a bad health-check configuration and a rollout strategy that allows total service interruption.

Step 10. Record the incident start time 

date -u +"Incident started: %Y-%m-%dT%H:%M:%SZ"
Enter fullscreen mode Exit fullscreen mode

Record the output in your incident worksheet.

Step 11. Configure the controlled rollout strategy 

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=merge \
  -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
Enter fullscreen mode Exit fullscreen mode

This changes the rollout strategy without changing the application template.

Confirm:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.strategy.type}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Expected:

Recreate
Enter fullscreen mode Exit fullscreen mode

Step 12. Introduce the faulty readiness probe 

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"frontend","readinessProbe":{"httpGet":{"path":"/healthz","port":80},"initialDelaySeconds":5,"periodSeconds":5}}]}}}}'
Enter fullscreen mode Exit fullscreen mode

This creates a new Deployment revision with a bad readiness endpoint.

Wait approximately 30–60 seconds.

Step 13. Observe the production failure

Run:

kubectl get pods -n banking-incident-lab

kubectl get deployment banking-frontend \
  -n banking-incident-lab

kubectl rollout status deployment/banking-frontend \
  -n banking-incident-lab \
  --timeout=30s
Enter fullscreen mode Exit fullscreen mode

Expected symptoms may resemble:

NAME                                READY   STATUS
banking-frontend-xxxxxxxxxx-aaaaa   0/1     Running
banking-frontend-xxxxxxxxxx-bbbbb   0/1     Running
Enter fullscreen mode Exit fullscreen mode 
NAME               READY   UP-TO-DATE   AVAILABLE
banking-frontend   0/2     2            0
Enter fullscreen mode Exit fullscreen mode

The rollout status command should time out because the new Pods never become available.

Important: A Pod can have STATUS Running while showing READY 0/1. These fields describe different aspects of Pod health.

Step 14. Confirm customer-facing impact

Inspect the endpoints:

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o yaml
Enter fullscreen mode Exit fullscreen mode

Look for:

conditions:
  ready: false
Enter fullscreen mode Exit fullscreen mode

Now test the application:

kubectl run incident-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -v --max-time 10 \
  http://banking-service
Enter fullscreen mode Exit fullscreen mode

Expected: the request fails or times out because the Service has no Ready backend endpoints.

The exact network error can vary with the cluster networking implementation.

Record the actual error. Do not assume every failed Service request returns HTTP 503.

Part D — Investigate like a production SRE

Step 15. Establish the scope

First determine whether the problem affects only the banking application or the whole Kubernetes cluster.

kubectl get nodes

kubectl get pods -n banking-incident-lab -o wide

kubectl get pods -n monitoring

kubectl get deployment banking-frontend \
  -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

Ask yourself:

  • Are Worker Nodes Ready?
  • Are Prometheus and Grafana healthy?
  • Are the banking Pods Running?
  • Are the banking Pods Ready?
  • Is the issue isolated to the frontend?

Do not immediately restart every Pod. That can destroy useful evidence without fixing the underlying configuration problem.

Step 16. Examine Kubernetes events 

kubectl get events \
  -n banking-incident-lab \
  --sort-by=.lastTimestamp
Enter fullscreen mode Exit fullscreen mode

Look for an event resembling:

Warning  Unhealthy  Readiness probe failed:
HTTP probe failed with statuscode: 404
Enter fullscreen mode Exit fullscreen mode

This is a strong clue that the application is running but failing its health check.

Step 17. Describe the affected Pods 

kubectl describe pods \
  -n banking-incident-lab \
  -l app=banking-frontend
Enter fullscreen mode Exit fullscreen mode

Find the Readiness field and the Events section.

You should observe that the probe is requesting /healthz instead of /.

Step 18. Examine application logs 

kubectl logs \
  -n banking-incident-lab \
  -l app=banking-frontend \
  -c frontend \
  --tail=100 \
  --prefix=true
Enter fullscreen mode Exit fullscreen mode

Look for NGINX requests to /healthz returning HTTP 404.

This is the evidence that connects the Kubernetes probe failures to the application endpoint.

Step 19. Compare the deployment revisions 

kubectl rollout history deployment/banking-frontend \
  -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

Inspect the live readiness configuration:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Expected:

/healthz
Enter fullscreen mode Exit fullscreen mode

Compare it with the baseline configuration saved before the incident, where the path was /.

Root cause analysis

Primary configuration defect

The new Deployment revision used /healthz as its readiness endpoint, but the NGINX application did not serve that path.

Failure mechanism

Kubelet received HTTP 404, marked the new Pods Not Ready, and the Service had no Ready endpoints.

Outage-enabling rollout configuration

The intentionally selected Recreate strategy removed the healthy old Pods before the replacement Pods passed readiness checks.

Business impact in the simulation

The banking frontend became unavailable to clients using its Kubernetes Service.

Part E — Recover production service

Step 20. Perform a Kubernetes rollback

We have identified the defective Deployment revision. Rather than manually editing the new release during the outage, restore the previous working version.

kubectl rollout undo deployment/banking-frontend \
  -n banking-incident-lab
Enter fullscreen mode Exit fullscreen mode

Check the recovery:

kubectl rollout status deployment/banking-frontend \
  -n banking-incident-lab \
  --timeout=180s
Enter fullscreen mode Exit fullscreen mode

Expected:

deployment "banking-frontend" successfully rolled out
Enter fullscreen mode Exit fullscreen mode

A Deployment rollback restores the previous Pod template. It does not automatically restore the Deployment's rollout strategy, so we must correct that separately.

Step 21. Restore the safer RollingUpdate strategy 

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=merge \
  -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"}}}}'
Enter fullscreen mode Exit fullscreen mode

With two replicas, this configuration allows a surge Pod while rounding the permitted unavailable count down to zero during normal rolling updates.

Verify:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.strategy.type}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Expected:

RollingUpdate
Enter fullscreen mode Exit fullscreen mode

Step 22. Verify recovery 

kubectl get pods -n banking-incident-lab

kubectl get deployment banking-frontend \
  -n banking-incident-lab

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o wide
Enter fullscreen mode Exit fullscreen mode

The Deployment should return to 2/2 available.

Verify the readiness endpoint was restored:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Expected:

/
Enter fullscreen mode Exit fullscreen mode

Finally, test the Service again:

kubectl run recovery-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -sS -o /dev/null \
  -w "HTTP %{http_code}\n" \
  --max-time 10 \
  http://banking-service
Enter fullscreen mode Exit fullscreen mode

Expected:

HTTP 200
Enter fullscreen mode Exit fullscreen mode

Step 23. Record the recovery time 

date -u +"Service restored: %Y-%m-%dT%H:%M:%SZ"
Enter fullscreen mode Exit fullscreen mode

Calculate the elapsed time between the incident start and recovery.

This is your measured lab recovery duration. It is not automatically the same as production MTTR, which is typically calculated across multiple incidents.

Part F — Verify monitoring and notification

Return to the JumpToTech — Banking Production Incident Monitoring dashboard.

The expected pattern is:

Illustrative availability timeline

Example only. Replace these values with observations from your Grafana dashboard.

Desired replicasAvailable replicas

00.511.52HealthyFailureInvestigationRollbackRecovered

Check whether the alert transitioned through its expected states:

Normal → Pending → Firing → Resolved

Because alert evaluations run periodically, a short outage may not last long enough to enter Firing. If no alert fired, record that observation rather than claiming that an email was sent.

If an email was delivered, preserve the timestamp and alert name as evidence.

Part G — Your incident report

Every student must complete this worksheet using their own terminal outputs and observations.

Incident report — INC-2026-001

Complete after the hands-on exercise. Copy the report into your lab submission.

Engineer name

Incident start time (UTC)

How was the incident detected?

Observed application impact

Key commands and evidence

Root cause

Mitigation and recovery actions

Service restoration time (UTC)

Measured recovery duration

Preventive actions

Copy completed incident report

Prevention recommendations

A strong SRE does more than restore service. They identify engineering improvements.

For this incident, recommend:

  1. Add automated readiness-endpoint checks to the CI/CD pipeline before deployment.
  2. Use RollingUpdate with an appropriate availability budget for customer-facing applications.
  3. Test application health checks in staging before promoting to production.
  4. Monitor desired versus available replicas and alert on sustained unavailability.
  5. Introduce canary deployments or progressive delivery for high-risk releases.
  6. Maintain a documented rollback runbook and test it regularly.

Part H — Interview preparation

The question interviewers may ask

Tell me about a production incident you independently investigated and resolved. What was the impact, root cause, and outcome?

Sample answer in English

Production-Style Incident Response — AWS EKS Banking Application

During a hands-on production incident simulation, I was responsible for investigating and restoring a banking frontend application deployed on AWS EKS.

The application was running with two replicas behind a Kubernetes ClusterIP Service. We used Prometheus and Grafana to monitor Deployment availability, Pod readiness, and container restarts.

During a deployment, the frontend became unavailable. My initial investigation showed that the desired replica count was two, but zero replicas were available.

I checked the Deployment rollout status, inspected the Pods, reviewed Kubernetes events, and examined the application logs.

The containers were running, but their readiness probes were failing with HTTP 404 errors.

I traced the issue to an incorrect readiness probe path introduced in the new Deployment revision. The application was configured to check /healthz, but NGINX was only serving the root endpoint. The rollout also used a Recreate strategy, which removed the healthy Pods before the replacement Pods were ready.

After identifying the root cause, I rolled back the Deployment to the previous working revision and restored the RollingUpdate strategy.

I verified that both replicas returned to Ready, the Service had healthy endpoints, and the application responded with HTTP 200.

The key lesson was that a running container does not necessarily mean the application is ready to serve traffic.

To prevent recurrence, I recommended automated health-check validation in CI/CD, safer rollout strategies, and Prometheus/Grafana alerting for unavailable replicas.

This exercise gave me practical experience in Kubernetes incident investigation, root cause analysis, rollback, service recovery, and production reliability practices.

This is an interview practice script for a lab incident. Personalize it with your actual evidence and recovery time. Do not present the simulation as an incident you handled while employed by a bank.

Follow-up interview questions you must master

12 technical interview questions

Tap to reveal answers

1.Why were the Pods Running but not Ready?

2.What is the difference between readiness and liveness probes?

3.Why didn't Kubernetes automatically fix the incident?

4.What did the Service do when both Pods were Not Ready?

5.Why did the deployment cause a full outage?

6.Would the default RollingUpdate strategy behave the same way?

7.Why use kubectl rollout undo?

8.Did the rollback restore every Deployment setting?

9.How did Prometheus detect the problem?

10.What is the difference between mitigation and root cause analysis?

11.What would you communicate to stakeholders?

12.What is the most important prevention?

Part I — Student submission checklist

Evidence and deliverables

0/12 completed

Deployment healthy before incident: 2/2

Baseline application test: HTTP 200

Grafana dashboard with banking metrics

Screenshot showing failed readiness probes

Kubernetes events showing HTTP 404

Application logs showing failed /healthz requests

Deployment rollout history

Successful rollback output

Deployment recovered: 2/2

Application recovered: HTTP 200

Incident report with measured recovery time

Interview explanation practiced aloud

Final learning outcome

By completing this lab, you should be able to explain the entire incident lifecycle:

1

Detect

2

Investigate

3

Mitigate

4

Verify

5

Prevent

Your goal is not to memorize a fictional story. Your goal is to reproduce the failure, understand every command, independently restore the application, and explain your own results clearly in an interview.

This is the difference between knowing Kubernetes commands and demonstrating practical DevOps/SRE incident-response skills.

JumpToTech DevOps/SRE Bootcamp — Production Incident Response Lab, Day 2

Top comments (0)