2-2 The Midnight Payment Processing Incident — Kubernetes, PostgreSQL, Terraform, CI/CD, Grafana and PagerDuty

Student hands-on guide · 3 classes × 3 hours · AWS EKS training environment

Your mission

You are a DevOps/SRE engineer supporting a banking payment-processing platform. At midnight, PagerDuty notifies the primary on-call engineer that payment transactions are failing.

You open Grafana and see increasing HTTP 503 errors. But when you inspect Kubernetes, the application Pods are all Running and Ready.

How can the application be healthy according to Kubernetes while customers cannot complete payments?

You will investigate the application, its PostgreSQL database, Kubernetes networking, deployment history, monitoring, and the infrastructure changes that could have caused the incident.

This is a realistic fictional production case study, supported by a real, runnable training lab. You must not claim that it is an actual incident at a named bank.

1. Incident ticket

INC-2026-1047

SEV-1 · Scenario

Payment Processing API — Elevated 5xx Error Rate

PagerDuty · Payments Production · 12:05 AM

Reported impact: Payment submissions failing across the customer-facing banking application.

Monitoring: HTTP 503 responses increasing, payment success rate declining, API Pods still Ready.

Initial hypothesis: Application dependency failure, database connectivity, or recent infrastructure change.

Assigned: Primary On-Call SRE

Escalation: Incident Commander, Backend Engineering, DBA, Cloud Platform, Release Engineering, Security.

In the hypothetical production architecture, the application runs on AWS EKS across three Availability Zones and connects to Amazon RDS PostgreSQL. Terraform manages security groups and networking; GitHub Actions handles application and infrastructure pipelines.

For the actual student lab, we use two Payment API replicas and PostgreSQL inside Kubernetes to keep costs and resource requirements low.

2. Production architecture and team responsibilities

Reference production architecture. The classroom lab replaces RDS and AWS security-group failure with an in-cluster database and NetworkPolicy.

Team	Responsibility during incident
Primary On-Call SRE	Acknowledge alert, assess impact, investigate, mitigate and communicate
Incident Commander	Coordinate teams, severity, updates and escalation
Backend/Application Team	Investigate application errors, connection pooling, configuration and code
Database Administrator	Check database availability, active connections, locks, CPU, I/O and query performance
Cloud/Platform Team	Investigate EKS, VPC, security groups, routing and network policies
Release/CI/CD Team	Check recent releases, GitHub Actions workflows and Terraform changes
Security Team	Review and approve necessary access-rule changes

A primary on-call engineer may initially perform several of these functions, especially on a small team. Do not wait for a meeting before checking the impact and following your incident-response runbook.

3. What you will build

The lab creates:

• Namespace payments-incident-lab
• PostgreSQL database with a simulated_payments table
• Python Payment API with two replicas
• Kubernetes Services and health probes
• Prometheus metrics for requests, errors and latency
• A controlled database connectivity incident
• Recovery and verification procedures

The lab uses synthetic payments only. It does not charge cards or connect to real payment processors.

4. Requirements

You need a training EKS cluster, working kubectl, Docker with permission to push images to a registry, and enough capacity for three small Pods in addition to monitoring.

For the NetworkPolicy incident, the cluster must have NetworkPolicy enforcement enabled. Installing a NetworkPolicy resource without an enforcing CNI will not necessarily block traffic.

First, verify your environment

kubectl config current-context
kubectl get nodes -o wide
kubectl get namespaces
kubectl get pods -n monitoring
kubectl get storageclass

Do not perform this exercise in a real production namespace.

Part A — Create all application and Kubernetes files

Create the project folders:

mkdir -p payment-incident-lab/{app,k8s,scripts,grafana,terraform}
cd payment-incident-lab

File 1 — app/requirements.txt

flask==3.1.1
psycopg[binary]==3.2.9
prometheus-client==0.22.1

File 2 — app/app.py

This is the application students will troubleshoot. Notice that /readyz checks the application process, but deliberately does not check database connectivity. This allows Pods to remain Ready while payment requests fail.

import os        try:            with connect() as conn:                with conn.cursor() as cur:                    cur.execute("SELECT 1")                    cur.fetchone()            REQ.labels("/api/db-check", "200").inc()            return jsonify(database="reachable"), 200        except Exception as exc:            app.logger.error(                "database connectivity failed: %s",                type(exc).__name__,            )            DB_ERRORS.inc()            REQ.labels("/api/db-check", "503").inc()            return jsonify(error="database_unavailable"), [e-mail]("/api/payments")def payment():    with DURATION.labels("/api/payments").time():        data = request.get_json(silent=True) or {}        key = request.headers.get("Idempotency-Key", "").strip()        if (            not key            or len(key) > 128            or not isinstance(data.get("amount_cents"), int)            or isinstance(data.get("amount_cents"), bool)            or not 1 <= data["amount_cents"] <= 100000        ):            REQ.labels("/api/payments", "400").inc()            return jsonify(                error="Idempotency-Key and integer "                      "amount_cents 1..100000 required"            ), 400        try:            with connect() as conn:                with conn.cursor() as cur:                    cur.execute(                        """                        INSERT INTO simulated_payments                        (idempotency_key, amount_cents)                        VALUES (%s, %s)                        ON CONFLICT (idempotency_key)                        DO NOTHING                        """,                        (key, data["amount_cents"]),                    )                    cur.execute(                        """                        SELECT id, amount_cents, status                        FROM simulated_payments                        WHERE idempotency_key = %s                        """,                        (key,),                    )                    row = cur.fetchone()

File 3 — app/Dockerfile

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

RUN useradd --uid 10001 --create-home appuser

USER 10001

EXPOSE 8080

CMD ["python", "app.py"]

File 4 — k8s/00-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: payments-incident-lab

File 5 — k8s/01-postgres.yaml

This database is disposable. Its storage uses emptyDir, so deleting or replacing the database Pod can destroy its data.

apiVersion: v1
kind: Secret
metadata:
  name: postgres-auth
  namespace: payments-incident-lab
type: Opaque
stringData:
  POSTGRES_PASSWORD: classroom-only-not-production
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-init
  namespace: payments-incident-lab
data:
  001-init.sql: |
    CREATE TABLE IF NOT EXISTS simulated_payments (
      id BIGSERIAL PRIMARY KEY,
      idempotency_key TEXT NOT NULL UNIQUE,
      amount_cents INTEGER NOT NULL
        CHECK (amount_cents > 0),
      status TEXT NOT NULL
        DEFAULT 'simulated_approved',
      created_at TIMESTAMPTZ NOT NULL DEFAULT now()
    );
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: lab-postgres
  template:
    metadata:
      labels:
        app: lab-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: payments
            - name: POSTGRES_USER
              value: labuser
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-auth
                  key: POSTGRES_PASSWORD
          volumeMounts:
            - name: init
              mountPath: /docker-entrypoint-initdb.d
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U labuser -d payments
            periodSeconds: 5
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              memory: 512Mi
      volumes:
        - name: init
          configMap:
            name: postgres-init
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  selector:
    app: lab-postgres
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

File 6 — k8s/02-payment-api.yaml

Replace REPLACE_WITH_YOUR_IMAGE with the container image you build.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: payments-incident-lab
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      containers:
        - name: payment-api
          image: REPLACE_WITH_YOUR_IMAGE
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: DB_HOST
              value: lab-postgres
            - name: DB_NAME
              value: payments
            - name: DB_USER
              value: labuser
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-auth
                  key: POSTGRES_PASSWORD
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments-incident-lab
  labels:
    app: payment-api
spec:
  selector:
    app: payment-api
  ports:
    - name: http
      port: 8080
      targetPort: http

File 7 — k8s/03-networkpolicy-outage.yaml

This is the controlled failure. It blocks new incoming connections to the PostgreSQL Pod from the API.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-postgres-ingress
  namespace: payments-incident-lab
spec:
  podSelector:
    matchLabels:
      app: lab-postgres
  policyTypes:
    - Ingress
  ingress: []

Important: This only works when your cluster enforces Kubernetes NetworkPolicy. On EKS, check the Amazon VPC CNI NetworkPolicy configuration before running this exercise.

File 8 — k8s/04-servicemonitor-optional.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-api
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - payments-incident-lab
  selector:
    matchLabels:
      app: payment-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

The release: monitoring label must match the selector of your Prometheus installation. This file requires the Prometheus Operator ServiceMonitor CRD.

Part B — Build, deploy and verify

Step 1. Build and push the Docker image

Run from inside payment-incident-lab.

Replace the Docker Hub username with yours.

export IMAGE='YOUR_DOCKERHUB_USERNAME/jumptotech-payment-api:v1'

docker buildx build \
  --platform linux/amd64 \
  -t "$IMAGE" \
  --push ./app

If your EKS worker nodes are ARM64, use linux/arm64 instead.

Step 2. Deploy PostgreSQL

kubectl apply -f k8s/00-namespace.yaml

kubectl apply -f k8s/01-postgres.yaml

kubectl rollout status deployment/lab-postgres \
  -n payments-incident-lab \
  --timeout=180s

Check:

kubectl get pods -n payments-incident-lab
kubectl get svc -n payments-incident-lab

The PostgreSQL Pod should show 1/1 Running.

Step 3. Deploy Payment API

sed "s|REPLACE_WITH_YOUR_IMAGE|$IMAGE|" \
  k8s/02-payment-api.yaml | kubectl apply -f -

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=180s

kubectl get deployments -n payments-incident-lab
kubectl get pods -n payments-incident-lab -o wide

Expected baseline:

NAME           READY   UP-TO-DATE   AVAILABLE
lab-postgres   1/1     1            1
payment-api    2/2     2            2

Step 4. Verify the database table

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c '\dt'

You should see the simulated_payments table.

Check the number of synthetic payments:

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c 'SELECT COUNT(*) FROM simulated_payments;'

Step 5. Test the application

Create scripts/02-smoke-test.sh:

#!/usr/bin/env bash
set -euo pipefail

NS=payments-incident-lab

kubectl run "smoke-$(date +%s)" \
  -n "$NS" \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- sh -c '
    echo "READINESS:"
    curl -sS -i --max-time 10 \
      http://payment-api:8080/readyz

    echo
    echo "DB CHECK:"
    curl -sS -i --max-time 10 \
      http://payment-api:8080/api/db-check

    echo
    echo "SYNTHETIC PAYMENT:"
    curl -sS -i --max-time 10 \
      -X POST \
      http://payment-api:8080/api/payments \
      -H "Content-Type: application/json" \
      -H "Idempotency-Key: classroom-$(date +%s)" \
      -d "{\"amount_cents\":1250}"
'

Run:

bash scripts/02-smoke-test.sh

Expected results:

Endpoint	Expected HTTP status
/readyz	200
/api/db-check	200
/api/payments	200

The temporary curl Pod may display an attach warning after completing its request. Focus on the actual HTTP response.

Part C — Configure Grafana and Prometheus

First verify the monitoring components:

kubectl get pods -n monitoring

kubectl get crd servicemonitors.monitoring.coreos.com

kubectl get prometheus -n monitoring

Inspect the ServiceMonitor selectors:

kubectl get prometheus -n monitoring -o yaml \
  | grep -A12 serviceMonitorSelector

If the ServiceMonitor label matches your Prometheus configuration:

kubectl apply -f k8s/04-servicemonitor-optional.yaml

kubectl get servicemonitor -n monitoring

In Grafana, select your Prometheus data source and create the following panels.

Panel 1 — Payment API request rate by HTTP status

sum by (status) (
  rate(payment_http_requests_total[5m])
)

This distinguishes successful requests from HTTP 400, 409 and 503 responses.

Panel 2 — HTTP 5xx error rate

sum(
  rate(payment_http_requests_total{status=~"5.."}[5m])
) or vector(0)

Panel 3 — Database errors over five minutes

sum(
  increase(payment_db_errors_total[5m])
) or vector(0)

Panel 4 — Available replicas

kube_deployment_status_replicas_available{
  namespace="payments-incident-lab",
  deployment="payment-api"
}

Panel 5 — Desired replicas

kube_deployment_spec_replicas{
  namespace="payments-incident-lab",
  deployment="payment-api"
}

Panel 6 — Pod readiness

sum by (condition) (
  kube_pod_status_ready{
    namespace="payments-incident-lab",
    pod=~"payment-api-.*"
  }
)

Panel 7 — Payment request p95 latency

histogram_quantile(
  0.95,
  sum by (le) (
    rate(
      payment_http_request_duration_seconds_bucket{
        endpoint="/api/payments"
      }[5m]
    )
  )
)

Generate test traffic before expecting request metrics to appear. Check that Prometheus is actually scraping the Payment API target. The metrics may remain absent if the ServiceMonitor selector is incorrect.

Key learning point

A dashboard can show:

Available Pods

2 / 2

Kubernetes sees healthy processes

Payment API

503

Database-dependent requests fail

This is not contradictory. Kubernetes readiness and application business functionality are different signals.

Part D — Create the incident

Class 2: Midnight outage simulation

Your instructor announces:

It is 12:05 AM. PagerDuty has triggered a critical alert. Customers cannot complete payment transactions. Grafana shows elevated HTTP 503 errors, but all Payment API Pods appear healthy. You are the primary on-call engineer. Investigate the incident and identify the failing dependency.

Step 1. Confirm the healthy baseline

kubectl get pods -n payments-incident-lab

bash scripts/02-smoke-test.sh

Do not continue until the database check and payment request return HTTP 200.

Step 2. Introduce the controlled failure

Create scripts/03-inject-outage.sh:

#!/usr/bin/env bash
set -euo pipefail

kubectl apply -f k8s/03-networkpolicy-outage.yaml

kubectl get networkpolicy \
  -n payments-incident-lab

printf '\nRun the smoke test and investigate.\n'

Run:

bash scripts/03-inject-outage.sh

Then:

bash scripts/02-smoke-test.sh

When NetworkPolicy enforcement is active, expected behavior is:

Component	State
Payment API Pods	Running, Ready
PostgreSQL Pod	Running, Ready
API readiness endpoint	HTTP 200
Database connectivity endpoint	HTTP 503
Synthetic payment request	HTTP 503
Application logs	Database connectivity errors

If requests still succeed, verify that NetworkPolicy enforcement is enabled. Do not assume the incident is active simply because kubectl apply succeeded.

Step 3. Investigate the incident

Start with the Kubernetes resources:

kubectl get pods -n payments-incident-lab -o wide

kubectl get deployments -n payments-incident-lab

kubectl get svc -n payments-incident-lab

kubectl get endpointslices -n payments-incident-lab

Inspect the API Deployment:

kubectl describe deployment payment-api \
  -n payments-incident-lab

Inspect the API Pods:

kubectl describe pods \
  -l app=payment-api \
  -n payments-incident-lab

Inspect application logs:

kubectl logs deployment/payment-api \
  -n payments-incident-lab \
  --tail=100 \
  --max-log-requests=10

Look for errors indicating the application cannot communicate with PostgreSQL.

Inspect the database:

kubectl get pods \
  -l app=lab-postgres \
  -n payments-incident-lab

kubectl logs deployment/lab-postgres \
  -n payments-incident-lab \
  --tail=100

Check the NetworkPolicies:

kubectl get networkpolicy \
  -n payments-incident-lab

kubectl describe networkpolicy block-postgres-ingress \
  -n payments-incident-lab

kubectl get networkpolicy \
  -n payments-incident-lab \
  -o yaml

Check rollout history:

kubectl rollout history deployment/payment-api \
  -n payments-incident-lab

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab

Important: In this particular incident, the API Deployment may have had no recent rollout at all. The problem is a networking configuration change, not necessarily a new container version.

Student investigation script

Create scripts/04-investigate.sh:

#!/usr/bin/env bash
set -euo pipefail

NS=payments-incident-lab

kubectl get pods,deploy,svc,endpointslices -n "$NS"

kubectl describe deployment payment-api -n "$NS"

kubectl logs deployment/payment-api \
  -n "$NS" \
  --tail=60 \
  --max-log-requests=10

kubectl get networkpolicy -n "$NS" -o yaml

kubectl rollout history deployment/payment-api -n "$NS"

Run:

bash scripts/04-investigate.sh

Root-cause analysis

In this lab, the NetworkPolicy selects the PostgreSQL Pod and allows no incoming traffic. The application can still start and answer /readyz, but new PostgreSQL connections fail.

The root cause is therefore database connectivity blocked by Kubernetes NetworkPolicy, not a crashed application Pod.

A blind kubectl rollout undo of the Payment API is not an appropriate fix because the network policy would remain in effect.

Part E — Restore service

After identifying the policy and following the classroom change procedure, remove the offending rule.

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Check that the application is still available:

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=120s

Run the payment tests again:

bash scripts/02-smoke-test.sh

Verify the database contains the successful synthetic payments:

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c 'SELECT id, amount_cents, status, created_at
      FROM simulated_payments
      ORDER BY id DESC
      LIMIT 10;'

Create scripts/05-recover.sh:

#!/usr/bin/env bash
set -euo pipefail

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab \
  --ignore-not-found

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=120s

bash scripts/02-smoke-test.sh

Run:

bash scripts/05-recover.sh

Recovery is verified when new database checks and synthetic payment requests succeed. In a real payment system, the application and database teams would also need to verify transaction integrity, retries and reconciliation.

Part F — Terraform and AWS RDS investigation

Why Terraform matters

In a production environment, PostgreSQL may run on Amazon RDS rather than inside Kubernetes.

Suppose a GitHub Actions workflow applies a Terraform change that accidentally removes the security-group rule allowing the EKS application to reach RDS on TCP port 5432.

The symptoms could resemble our lab: Kubernetes Pods are Ready, but application requests fail because database connections cannot be established.

The two failure mechanisms are different:

Classroom lab	Production reference scenario
Kubernetes NetworkPolicy blocks PostgreSQL ingress	AWS security group blocks EKS-to-RDS traffic
PostgreSQL runs inside EKS	PostgreSQL runs on Amazon RDS
Fix the Kubernetes NetworkPolicy	Restore the correct AWS security-group rule
Investigate kubectl get networkpolicy	Investigate AWS VPC, security groups, CloudTrail and Terraform changes

File 9 — terraform/EXAMPLE_ONLY.tf

This is a teaching example. Do not run terraform apply on this file against an existing AWS environment. The correct security group and identity must first be established.

variable "rds_security_group_id" {
  type = string
}

variable "approved_eks_application_security_group_id" {
  type = string
}

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id = var.rds_security_group_id

  referenced_security_group_id = (
    var.approved_eks_application_security_group_id
  )

  ip_protocol = "tcp"

  from_port = 5432
  to_port   = 5432

  description = "Approved EKS application traffic to PostgreSQL"
}

Terraform investigation commands

For an actual managed Terraform project, first identify the correct repository, workspace, state and pipeline.

terraform version

terraform workspace show

terraform state list

terraform plan

Review the plan and the last GitHub Actions run. Determine whether a rule was removed, replaced, or changed.

Do not assume that the latest Terraform deployment caused the incident merely because it happened recently. Correlate the change timestamp with application errors, RDS connection failures, AWS configuration history and network evidence.

CI/CD investigation

Students should be able to explain this workflow:

Developer creates pull request

GitHub Actions: test, build, security checks

Terraform fmt, validate, plan and review

Approved infrastructure change

Application deployment and post-deploy tests

Prometheus / Grafana validation and alerting

An important production lesson: a pipeline can finish successfully while the system is broken. Successful execution of terraform apply only proves Terraform applied the requested changes; it does not prove payment processing still works.

Part G — Three-class teaching schedule

Class	Duration	Activities
Class 1	3 hours	Architecture, team roles, Docker, PostgreSQL, EKS Deployment, Services, API testing, Grafana
Class 2	3 hours	PagerDuty ticket, incident injection, logs, database investigation, NetworkPolicy, Terraform and CI/CD analysis
Class 3	3 hours	Recovery, verification, incident report, prevention, mock interview and follow-up questions

Class 1 — Build the platform

Students must finish with a working Payment API, two healthy replicas, PostgreSQL connectivity, successful synthetic payments and visible metrics.

Instructor questions: What is the difference between a Deployment and a Service? Why do we use a Secret? What does a readiness probe check? How does the API connect to PostgreSQL? Why might database availability differ from Pod readiness?

Class 2 — Investigate the outage

Do not immediately tell students the root cause. Present the incident ticket and ask them to gather evidence.

Students should identify that the Pods are Ready, the database Pod is running, the API reports database connectivity errors, and a NetworkPolicy is blocking the connection.

Instructor questions: Is this an application crash? Why are there no container restarts? Why does the HTTP 503 error appear? Why would restarting the Pods not fix the problem? What evidence would connect this to a Terraform change in AWS?

Class 3 — Recover and explain

Students must explain the root cause, restore connectivity, validate synthetic payments, review Grafana, and present an incident report.

They should distinguish between mitigation (restoring service), root cause (identifying the failing rule), and prevention (adding change controls and connectivity tests).

Part H — Interview preparation

These are the questions students should practice aloud, with technically accurate answers.

1. Your Pods are Running and Ready, but customers receive HTTP 503. What do you investigate?

Sample answer:

I would first assess the customer impact using Grafana or another monitoring tool. I would check the HTTP 5xx error rate, latency, and affected endpoints. Then I would inspect the Kubernetes Deployments, Pods, Services and application logs. Since the Pods are Ready, I would not assume Kubernetes itself is healthy end to end. I would investigate downstream dependencies such as the database, networking, authentication and external APIs.

2. How can all Pods be Ready while payment processing is failing?

Kubernetes readiness only checks what the configured readiness probe verifies. If the probe checks only the application process, the Pod can remain Ready even when the database is unreachable. That's why I would also monitor transaction success rates and perform synthetic end-to-end checks.

3. What would you check first to determine whether the database is the problem?

I would examine the application logs for connection timeouts, authentication failures, connection-pool exhaustion and query errors. Then I would check database health and connectivity. I would also review database metrics such as CPU, memory, connections, I/O and wait events.

4. How would you prove that a Terraform change caused the outage?

I would correlate the incident start time with the Terraform pipeline execution. Then I would review the Terraform plan, the applied changes and AWS configuration history. If the change removed a required RDS security-group rule, I would verify that the affected application could no longer reach the database. I would use that evidence to establish causation rather than relying only on timing.

5. Would you immediately run kubectl rollout undo?

Not automatically. I would first determine whether the application deployment caused the failure. If the root cause is a network rule or database configuration, rolling back the application might not help. I would follow the incident runbook and select the safest mitigation supported by the evidence.

6. What is the difference between HTTP 502, 503 and 504?

Code	Meaning	Common example
502 Bad Gateway	Gateway received an invalid response from an upstream server	Reverse proxy cannot obtain a valid upstream response
503 Service Unavailable	Service cannot currently handle the request	Payment API deliberately returns 503 when database is unavailable
504 Gateway Timeout	Gateway did not receive an upstream response in time	ALB or proxy times out waiting for an application response

A database timeout does not automatically mean the customer receives 504. In this lab, the application catches the error and explicitly returns 503.

7. Who would you involve in a SEV-1 incident?

I would follow the company's incident-response procedure. Depending on the impact, I would involve the incident commander, application engineers, database team, platform engineers and release team. I would communicate the symptoms, current impact, evidence collected, and mitigation plan so the teams could work in parallel.

8. How would you investigate a delayed failure after a successful release?

I would compare the release timestamp with the first customer errors and infrastructure changes. I would investigate whether the failure was triggered by connection-pool exhaustion, database load, expired credentials, a scheduled job, network changes or traffic growth. I would also check whether the application had existing connections that continued working temporarily before new connections started failing.

9. What database metrics would you inspect?

I would review CPU utilization, available memory, connection counts, disk I/O, read and write latency, active sessions, slow queries, locks and wait events. I would use those metrics to distinguish between database overload, slow queries, connection exhaustion and network connectivity failures.

10. What is a database connection pool?

A connection pool maintains reusable database connections so an application doesn't need to establish a new connection for every request. If the pool is exhausted or connections become invalid, requests may wait, time out or fail. I would check active connections, pool utilization, wait time and timeout settings.

11. How would you restore service without creating additional risk?

I would first identify the failing dependency and the last known good configuration. I would coordinate the mitigation through the incident process, restore only the necessary access, and verify recovery using application requests, database checks, monitoring and synthetic transactions. I would avoid broad security-group changes or unnecessary restarts.

12. What would you do after recovery?

I would confirm that error rates and payment success metrics returned to baseline. Then I would document the incident timeline, impact, root cause, mitigation and prevention actions. I would recommend post-deployment database connectivity checks, better dependency monitoring, Terraform plan review and appropriate change approvals.

Part I — Student incident report

Every student must complete this report after Class 3.

Incident report template

Incident ID: INC-2026-1047

Severity: SEV-1 (training scenario)

Affected service: Payment Processing API

Detection: PagerDuty / Grafana

Start time: Record actual lab injection time

Recovery time: Record actual successful verification time

Customer impact: Synthetic payment requests failed in the lab; production customer impact is hypothetical

Observed symptoms: API Pods Ready, database-dependent requests returned HTTP 503

Root cause: PostgreSQL ingress blocked by Kubernetes NetworkPolicy

Mitigation: Removed the offending policy in the isolated lab

Verification: Database checks and synthetic payments returned HTTP 200

Prevention: Network change reviews, dependency checks, monitoring, and automated connectivity tests

Evidence: Attach command outputs, relevant logs, Grafana screenshots and the exact configuration change.

File 10 — scripts/01-check.sh

#!/usr/bin/env bash
set -euo pipefail

kubectl get nodes
kubectl get ns
kubectl get pods -n monitoring

kubectl get pods,deploy,svc \
  -n payments-incident-lab

kubectl get networkpolicy \
  -n payments-incident-lab

File 11 — scripts/06-cleanup.sh

Use only after the classes are finished. This deletes the entire training namespace and its synthetic database.

#!/usr/bin/env bash
set -euo pipefail

read -r -p \
  'Delete ENTIRE payments-incident-lab namespace? Type DELETE: ' \
  answer

if [[ "$answer" == "DELETE" ]]; then
  kubectl delete namespace payments-incident-lab
else
  echo "Canceled."
fi

To clean up the optional ServiceMonitor separately:

kubectl delete servicemonitor payment-api \
  -n monitoring \
  --ignore-not-found

Final student challenge

Your instructor will ask:

"It is 12:05 AM. Your Payment API has two healthy Pods, PostgreSQL is Running, and your application deployment has not changed. However, payment requests are returning HTTP 503. What are your first actions, how do you identify the root cause, who do you involve, and how do you restore service safely?"

You must be able to explain the investigation in your own words, demonstrate the relevant commands, identify the exact failed dependency, and verify the recovery.

Your goal is not to memorize commands. Your goal is to think and communicate like a DevOps/SRE engineer responding to a production incident.

JumpToTech DevOps Engineering · Practical Incident Response Training