Student training handbook | 3 classes × 3 hours | AWS EKS · Terraform · GitHub Actions · PostgreSQL RDS · Prometheus · Grafana · PagerDuty
Students, welcome to your next production incident investigation.
In our previous incident, you investigated Kubernetes Pods that became unavailable after a deployment. This time, the challenge is different.
All Kubernetes Pods are healthy. The deployment succeeded. The application is reachable. But customers cannot complete payments.
This is the type of scenario that tests whether you understand the entire production system—not just Kubernetes commands.
You will work as an on-call DevOps/SRE engineer at a fictional enterprise payment platform. The scenario is designed to resemble incidents that could occur in large financial technology companies, banks, and e-commerce organizations. All company details, ticket numbers, figures, and incident events below are fictional training data.
By the end of three classes, you should be able to investigate a multi-system incident, identify an infrastructure change, coordinate with other engineering teams, restore service safely, explain database connectivity, and defend your decisions during a technical interview.
INC-2026-1047 — Payment Authorization Failures
SEV-1
Payment success rate
41%
Baseline: 99.95%
HTTP 5xx rate
37%
On payment API
Kubernetes Pods
12/12
Ready and Running
Database
Available
RDS instance is running
Incident clue: A Terraform deployment completed approximately 20 minutes before the payment failures began.
The critical interview question is:
“If all Kubernetes Pods are healthy and the RDS database is available, why are customers receiving payment failures?”
You are not allowed to answer, “I would restart the Pods,” without first explaining what evidence justifies that action.
Your three-class mission
| Class | Duration | Your objective |
|---|---|---|
| Class 1 — Detection and triage | 3 hours | Understand the ticket, production architecture, incident roles, and evidence collection |
| Class 2 — Deep troubleshooting | 3 hours | Investigate Kubernetes, CI/CD, Terraform, networking, and PostgreSQL; identify the root cause |
| Class 3 — Recovery and interviews | 3 hours | Restore service, verify transactions, complete the postmortem, and pass a mock technical interview |
You will not be given the root cause at the beginning. Your instructor will reveal additional evidence as you progress.
1. The company and production environment
You work for NorthStar Payments, a fictional payment technology company processing transactions for online retailers.
NorthStar provides payment authorization, order management, transaction history, refunds, and customer account services.
The production system runs on AWS. The platform team manages infrastructure through Terraform, developers deploy applications through GitHub Actions, and SRE engineers monitor the platform using Prometheus, Grafana, CloudWatch, and PagerDuty.
Production architecture
Conceptual production architecture. AWS networking and service placement are simplified for training.
The system has these components:
| Component | Purpose | Responsible team |
|---|---|---|
| Amazon EKS | Runs containerized services | Platform / SRE |
| Payment API | Receives and processes payment requests | Backend application team |
| Payment workers | Process asynchronous events | Backend application team |
| Amazon RDS PostgreSQL | Stores transaction and payment records | Database / DBA team |
| AWS Security Groups | Control permitted network connections | Cloud platform / security |
| Terraform | Provisions and changes AWS infrastructure | Cloud platform |
| GitHub Actions | Tests and deploys infrastructure and applications | DevOps / release engineering |
| Prometheus and Grafana | Collect and visualize service metrics | SRE / observability |
| CloudWatch | AWS infrastructure logs and metrics | Cloud platform / SRE |
| PagerDuty | Routes urgent alerts to on-call responders | SRE / incident management |
| Jira Service Management | Tracks incident tickets and investigation | Service management |
Example workload sizes
For this case study, the production environment has 12 Payment API Pods, 6 background workers, 3 Availability Zones, and one Multi-AZ PostgreSQL RDS deployment. These are illustrative values, not actual figures from any bank.
The Payment API has a readiness endpoint at /ready, which checks whether the HTTP process is responsive. It does not check whether the application can reach PostgreSQL. That design choice will become important.
2. Your production engineering team
An enterprise incident is not normally handled by one engineer working alone. You may be the first person paged, but different teams own different systems.
Incident response team structure
Incident Commander
Coordinates response and decisions
Primary SRE
Triage, metrics, Kubernetes
Platform Engineer
Terraform, VPC, security groups
Backend Engineer
Payment API, application logs
Database Engineer
RDS, SQL, connection health
Release Engineer
CI/CD, change history
Security Engineer
Network access, policy review
Customer support and business stakeholders receive coordinated status updates through the incident communication process.
Who is responsible for what?
Primary on-call DevOps/SRE engineer — your role
You acknowledge the alert, establish the initial technical picture, identify impacted components, collect evidence, coordinate with the incident team, and execute approved operational actions within your access and authority.
Incident Commander
Owns coordination, severity management, escalation, communications, and ensuring someone is assigned to each workstream. The incident commander does not necessarily perform the technical fix.
Application engineer
Investigates payment request failures, database client errors, connection pool behavior, retry logic, and application code.
Cloud platform engineer
Investigates AWS networking, Terraform state and plans, security groups, IAM, EKS networking, and infrastructure changes. This engineer normally owns the corrective Terraform pull request.
Database engineer
Checks database availability, active connections, PostgreSQL errors, query performance, locks, replication or failover state, and database-side access settings.
Release engineer
Finds the GitHub Actions deployment run, commit SHA, pull request, approvers, change window, and any simultaneous releases.
Security engineer
Reviews whether a proposed security group change is permitted and ensures recovery does not introduce unnecessary network exposure.
Customer support and communications
Collect customer reports and distribute approved incident updates. They should not receive raw database credentials or sensitive customer records.
Interview question
“If you are the primary on-call engineer, are you personally responsible for fixing every component?”
A strong answer:
“I am responsible for responding to the incident, understanding the impact, and driving technical triage within my role. I would engage the correct service owners for specialized components such as RDS, networking, and application code. We work together under the incident response process to restore service safely.”
3. Where the production ticket comes from
In this scenario, Prometheus detects the elevated error rate and sends the alert to Alertmanager. Alertmanager routes it to PagerDuty. PagerDuty pages the primary on-call engineer. An incident ticket is then created in Jira Service Management through the company's configured integration or incident workflow.
PagerDuty — Production Alert
12:15 AM · Payment Platform · Primary On-Call
High Payment Failure Rate
Service: payment-api-prod
Environment: production
Cluster: northstar-prod-eks
Namespace: payments-prod
Trigger: HTTP 5xx rate exceeds the configured threshold for 5 minutes.
Customer payment authorization is degraded.
Example Jira incident ticket
Jira Service Management · INC-2026-1047
P1
Payment authorization failures — customer impact
| Status | Investigating |
|---|---|
| Assigned group | Production SRE |
| First detected | 12:15 AM |
| Customer symptoms | Payment errors and timeouts |
| Related change | CHG-2026-318 — Terraform network update |
| Impacted service | Payment API |
Description
Payment success rate has dropped from approximately 99.95% to 41%. Payment API 5xx errors have increased. Kubernetes reports all 12 API Pods as Ready. RDS instance status is Available. Investigate application dependencies, infrastructure changes, and customer impact.
Initial assignment: Primary on-call SRE.
Where else might you learn about the incident?
You may see the same incident in the PagerDuty mobile app, Slack incident channel, Jira queue, Grafana dashboard, CloudWatch alarms, or customer support escalation. Different companies use different integrations.
Student question: “What is the difference between an alert, an incident ticket, and a change ticket?”
- An alert is a monitoring signal that requires attention.
- An incident ticket tracks the unexpected service disruption and response.
- A change ticket documents an intended production change, its approvals, timing, and rollback plan.
4. The incident timeline
| Time | Event |
|---|---|
| 11:20 PM | GitHub Actions infrastructure pipeline starts |
| 11:28 PM | Terraform plan is reviewed and approved |
| 11:40 PM | Terraform apply completes successfully |
| 11:45 PM | Infrastructure pipeline reports success |
| 12:00 AM | Payment API error rate begins increasing |
| 12:10 AM | Payment success rate drops sharply |
| 12:15 AM | PagerDuty pages the primary on-call engineer |
| 12:17 AM | SRE acknowledges and begins triage |
| 12:22 AM | Incident Commander engaged |
| 12:30 AM | Database connectivity errors identified |
| 12:40 AM | Recent infrastructure change becomes the leading suspect |
| 12:50 AM | Recovery change is prepared and reviewed |
| 1:05 AM | Connectivity restored; application recovery monitored |
| 1:20 AM | Payment success and error metrics return to baseline |
| 1:35 AM | Incident moves into observation and post-incident follow-up |
These timestamps are the planned case-study narrative, not measured outcomes from the student lab.
5. The mystery: why healthy Pods cannot process payments
At first, you check Kubernetes:
kubectl get pods -n payments-prod
The output shows:
NAME READY STATUS RESTARTS
payment-api-7d8f5b6c9-a1b2c 1/1 Running 0
payment-api-7d8f5b6c9-d3e4f 1/1 Running 0
payment-api-7d8f5b6c9-g5h6i 1/1 Running 0
...
12 payment-api Pods Ready
Then you inspect the Deployment:
kubectl get deployment payment-api -n payments-prod
NAME READY UP-TO-DATE AVAILABLE
payment-api 12/12 12 12
You inspect RDS in AWS and see:
DB instance: northstar-payments-prod
Status: Available
Engine: PostgreSQL
Port: 5432
Yet the payment API returns:
HTTP/1.1 503 Service Unavailable
Application logs contain:
ERROR payment authorization failed
ERROR could not establish database connection
ERROR connection timeout to postgres endpoint:5432
This is the central lesson of the incident:
Kubernetes Pod readiness, application dependency health, and successful business transactions are three different things.
A database can be operational while a particular application cannot connect to it.
Class 1 — Detect, assess, and investigate
3 hours · Students act as the primary on-call SRE
Class 1 schedule
| Time | Activity |
|---|---|
| 0:00–0:25 | Read incident ticket, identify business impact |
| 0:25–0:50 | Understand team structure and incident communication |
| 0:50–1:20 | Investigate Kubernetes health |
| 1:20–1:50 | Investigate Grafana and HTTP errors |
| 1:50–2:20 | Check application logs and database connectivity |
| 2:20–2:45 | Identify next investigation steps |
| 2:45–3:00 | Mock interview and handover |
Task 1 — Your first five minutes
You receive the PagerDuty alert. You are the primary on-call engineer.
Your interviewer asks:
“What would you do immediately?”
A model response:
“First, I would acknowledge the PagerDuty alert. Then I would open our monitoring dashboards to validate the impact, including payment success rate, HTTP 5xx errors, latency, and Kubernetes availability. I would determine whether the issue affects all customers or a subset, identify the impacted services, and follow our incident escalation procedure. I would not immediately restart Pods because the evidence does not yet show that restarting them would help.”
What you should ask the incident team
- Which payment operations are failing: authorization, capture, refund, or all of them?
- When did the errors begin?
- Are all customers affected or only certain regions or merchants?
- Did an application release, infrastructure change, or database maintenance occur?
- Are failures isolated to one service or shared by multiple services?
- Are transactions being rejected, timing out, or potentially duplicated?
- Is there a known-good recovery procedure?
Important: During a payment incident, do not blindly retry failed transactions. A timeout does not necessarily mean a payment was never processed. The application and payment teams must verify transaction status and idempotency before retrying.
Task 2 — Kubernetes investigation
Run the following commands in the designated training namespace. In the production case study, the namespace is payments-prod.
# Confirm cluster context before running commands
kubectl config current-context
# Inspect deployments
kubectl get deployments -n payments-prod
# Inspect Pods and their nodes
kubectl get pods -n payments-prod -o wide
# Inspect Services
kubectl get svc -n payments-prod
# Inspect Service endpoints
kubectl get endpointslices -n payments-prod
# Inspect nodes
kubectl get nodes -o wide
# Inspect the payment Deployment
kubectl describe deployment payment-api -n payments-prod
# Inspect one payment Pod
kubectl describe pod POD_NAME -n payments-prod
# Review application logs
kubectl logs POD_NAME -n payments-prod --tail=100
What should you conclude?
If all Pods are Ready, you have evidence that Kubernetes considers them eligible to serve traffic. You do not yet have evidence that the payment workflow is functioning correctly.
Check the readiness probe:
kubectl get deployment payment-api \
-n payments-prod -o yaml
Suppose you discover:
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
The endpoint checks whether the application process can respond, but it does not test PostgreSQL connectivity.
Interview follow-up: “Should a readiness probe always connect to the database?”
No. Dependency-aware readiness checks can be useful, but they require careful design. If every Pod fails readiness during a shared database outage, the Service may lose all endpoints, potentially worsening recovery. Teams often separate lightweight process readiness, dependency health, and end-to-end business transaction monitoring.
Task 3 — Grafana investigation
Your instructor should show four monitoring panels.
Example incident dashboard
Available API replicas
12 / 12
Stable
Payment success rate
41%
Severely degraded
HTTP 5xx error rate
37%
Elevated
Database connection failures
Increasing
Application-side metric
PromQL examples
These queries assume that the required Kubernetes and application metrics are actually collected. Metric names such as http_requests_total and payment_transactions_total are example application instrumentation; they do not appear automatically from kube-state-metrics.
Available replicas
kube_deployment_status_replicas_available{
namespace="payments-prod",
deployment="payment-api"
}
Desired replicas
kube_deployment_spec_replicas{
namespace="payments-prod",
deployment="payment-api"
}
HTTP 5xx percentage
100 *
sum(rate(http_requests_total{
service="payment-api",
status=~"5.."
}[5m]))
/
clamp_min(
sum(rate(http_requests_total{
service="payment-api"
}[5m])),
0.001
)
Payment success percentage
100 *
sum(rate(payment_transactions_total{
result="success"
}[5m]))
/
clamp_min(
sum(rate(payment_transactions_total[5m])),
0.001
)
95th percentile HTTP latency
histogram_quantile(
0.95,
sum by (le) (
rate(http_request_duration_seconds_bucket{
service="payment-api"
}[5m])
)
)
For HTTP status-code metrics, instrument the application or collect appropriate ingress/load-balancer metrics. Prometheus cannot derive HTTP 503 counts from Pod readiness alone.
Interviewer asks
“Grafana shows all 12 Pods healthy but the payment success rate is 41%. Where do you investigate next?”
A strong response:
“I would investigate the application request path and its dependencies. I would review payment API logs, trace failed requests, check database connection metrics, and test connectivity from the application network. I would also review recent infrastructure and application changes.”
Task 4 — Application logs
Example logs:
00:11:42 INFO POST /api/payments/authorize
00:11:42 ERROR database connection timeout
00:11:42 ERROR SQLSTATE 08001
00:11:42 WARN payment authorization unavailable
00:11:42 INFO response_status=503 duration_ms=5002
Explain to the instructor:
- What does
SQLSTATE 08001suggest? - Does this prove the database server has crashed?
- What is the difference between a connection timeout and an authentication failure?
- Why might the application return 503 while its Pod remains Ready?
- Which team should investigate the database, and which team should investigate the application?
A useful answer to question 2:
“No. A database connection failure can result from networking, security groups, routing, DNS, authentication, database capacity, or a database outage. I would test each layer before deciding which component is responsible.”
Class 1 communication exercise
Students should practice this update to the incident commander:
Incident update — Payment API
We have confirmed significant customer-facing payment failures. Payment success rate has dropped to approximately 41%, and HTTP 5xx errors are elevated.
All 12 Payment API Pods remain Ready, and the RDS instance reports Available. However, application logs show PostgreSQL connection timeouts.
We are investigating the application-to-database network path and reviewing recent infrastructure changes with the platform and database teams.
The root cause is not yet confirmed. Our priority is restoring payment authorization safely and preventing duplicate transactions.
Class 1 homework
Students should submit a short incident update, the commands they would use, their initial hypotheses, and an explanation of why healthy Pods do not guarantee healthy payment processing.
Class 2 — Terraform, CI/CD, networking, and database
3 hours · Deep technical investigation
Class 2 schedule
| Time | Activity |
|---|---|
| 0:00–0:25 | Review Class 1 evidence |
| 0:25–0:55 | Trace the payment request to PostgreSQL |
| 0:55–1:25 | Investigate database connectivity and metrics |
| 1:25–1:55 | Review GitHub Actions and Terraform change history |
| 1:55–2:25 | Compare security group rules |
| 2:25–2:45 | Confirm the cause and propose recovery |
| 2:45–3:00 | Technical interview challenge |
Task 1 — Understand the database connection
The payment application uses PostgreSQL to store payment transactions.
Example database tables:
CREATE TABLE payments (
payment_id UUID PRIMARY KEY,
merchant_id VARCHAR(50) NOT NULL,
amount_cents BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
status VARCHAR(20) NOT NULL,
idempotency_key VARCHAR(100) UNIQUE NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE payment_events (
event_id BIGSERIAL PRIMARY KEY,
payment_id UUID NOT NULL REFERENCES payments(payment_id),
event_type VARCHAR(50) NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
Example payment statuses are PENDING, AUTHORIZED, DECLINED, and FAILED.
In a real payment platform, financial transaction data needs additional controls, auditing, access restrictions, and carefully designed state transitions. This simplified schema is for learning connectivity and SQL troubleshooting, not a complete payment ledger.
Normal payment request flow
- Customer sends
POST /api/payments/authorize. - ALB routes the request to the Payment API in EKS.
- Payment API validates the request and checks its idempotency key.
- Application opens or reuses a PostgreSQL connection.
- Application reads or writes payment state and coordinates the external authorization workflow.
- Application returns the appropriate response and emits metrics.
What happens if step 4 fails?
The application may be unable to read or persist the payment state. It must not blindly repeat external authorization requests or assume that a timed-out transaction did not succeed.
Task 2 — Test database connectivity
The database engineer confirms that RDS is Available. Your next question is whether the Payment API network can reach the database endpoint on TCP port 5432.
For the production case study, you would use approved access through the existing application Pod or an authorized diagnostic Pod in the same network context.
# Check the ServiceAccount and Pod configuration
kubectl get pod POD_NAME \
-n payments-prod -o yaml
# Inspect relevant application logs
kubectl logs POD_NAME \
-n payments-prod --tail=100
# Check database DNS resolution
getent hosts DB_ENDPOINT
# Check TCP connectivity
nc -vz -w 5 DB_ENDPOINT 5432
The last two commands must run in an authorized environment with equivalent network access to the affected application, not simply on an unrelated EC2 management host. A successful connection from a bastion does not prove that an EKS Pod can connect.
Expected investigation evidence:
DNS resolution: successful
TCP connection:
connect to DB_ENDPOINT port 5432:
Connection timed out
This suggests a network-path problem but does not yet identify the exact AWS rule responsible.
Interviewer follow-ups
“What if DNS fails?”
Investigate CoreDNS, VPC DNS configuration, DNS records, resolver access, and the endpoint being used.
“What if DNS works but TCP times out?”
Investigate security groups, Network ACLs, routing, NetworkPolicies, database endpoint reachability, and possible network packet drops.
“What if TCP succeeds but PostgreSQL authentication fails?”
Investigate database credentials, IAM database authentication if configured, Secrets Manager, PostgreSQL roles, SSL requirements, and authentication settings.
“What if TCP succeeds but SQL queries are slow?”
Investigate query plans, locks, active sessions, connection pools, CPU, I/O, and database wait events.
Task 3 — Inspect PostgreSQL health
The database team runs approved diagnostic queries:
-- Verify database responds
SELECT 1;
-- Count current connections
SELECT count(*) AS active_connections
FROM pg_stat_activity;
-- Inspect connection states
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;
-- Check configured connection limit
SHOW max_connections;
-- Review sessions waiting on resources
SELECT
pid,
state,
wait_event_type,
wait_event
FROM pg_stat_activity
WHERE wait_event IS NOT NULL;
If the database is responsive and the connection count is low, but Payment API Pods cannot connect, the team should investigate the network path.
A low connection count alone does not prove a security group problem. It becomes useful evidence when combined with timeouts, recent changes, and network configuration.
AWS RDS checks
aws rds describe-db-instances \
--db-instance-identifier northstar-payments-prod \
--query 'DBInstances[0].{Status:DBInstanceStatus,Endpoint:Endpoint.Address,Port:Endpoint.Port,VpcSecurityGroups:VpcSecurityGroups}'
Check CloudWatch or RDS monitoring for database connections, CPU utilization, free memory, storage, and relevant logs. Enhanced Monitoring and Performance Insights/Database Insights may provide deeper diagnostics if configured.
Task 4 — Investigate the CI/CD pipeline
The release engineer discovers a GitHub Actions workflow:
GitHub Actions — Production Infrastructure
Succeeded
Workflow: terraform-production.yml · Completed 11:45 PM
- Checkout repository
- Terraform format and validate
- Terraform plan
- Production approval
- Terraform apply
Interviewer: “If the Terraform pipeline succeeded, how could it have caused the incident?”
Answer:
“A successful Terraform apply means the requested infrastructure changes were applied. It does not prove that application connectivity or business functionality remained healthy. The pipeline may lack post-deployment integration tests or application-level validation.”
Production workflow example
Students should understand this pipeline structure:
name: Production Terraform
on:
workflow_dispatch:
permissions:
contents: read
id-token: write
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Configure approved AWS OIDC credentials
# Install a pinned, approved Terraform version
- run: terraform fmt -check
- run: terraform init -input=false
- run: terraform validate
- run: terraform plan -out=tfplan
# Production apply should be a separate approved job.
# Use a protected environment, artifact integrity
# checks, and the reviewed saved plan.
This is a teaching skeleton, not a complete production-ready workflow. In a real pipeline, the saved plan must be securely transferred between jobs, access controlled, and applied only after approval.
Questions students must answer
- What is the difference between
terraform planandterraform apply? - What does Terraform state store?
- Why use a remote backend and state locking?
- Why should GitHub Actions authenticate to AWS using OIDC rather than long-lived access keys?
- What does a protected GitHub environment accomplish?
- Why is manual approval useful but not sufficient?
- What post-apply checks would have detected the payment outage?
- Why can an infrastructure change fail without any new application image?
Task 5 — The Terraform change
The platform engineer compares the previous and current security group configuration.
Before the change
The RDS security group allows PostgreSQL access from the EKS application security group.
resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
security_group_id = aws_security_group.rds.id
referenced_security_group_id = aws_security_group.eks_application.id
ip_protocol = "tcp"
from_port = 5432
to_port = 5432
}
After the change
A Terraform refactor accidentally replaces the application security group reference with an unrelated security group.
resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
security_group_id = aws_security_group.rds.id
referenced_security_group_id = aws_security_group.management.id
ip_protocol = "tcp"
from_port = 5432
to_port = 5432
}
Important: The code is illustrative. Actual EKS Pod network traffic may use node security groups or Pod security groups depending on the VPC CNI and security-groups-for-Pods configuration. The effective source security group must be verified in the actual architecture.
What happened?
The authorized source for PostgreSQL traffic was changed.
The database remained healthy, and the EKS Pods remained Ready, but new database connections from the application network were no longer permitted by the intended ingress rule.
Existing established connections may continue temporarily because AWS security groups are stateful and connection tracking can preserve some flows. As connections close, recycle, or time out, application failures may increase.
This explains why the outage can appear after the Terraform apply rather than exactly when the change is made.
Compare the infrastructure change
git log --oneline -10
git diff PREVIOUS_COMMIT CURRENT_COMMIT -- terraform/
terraform plan
For AWS-side verification:
aws ec2 describe-security-groups \
--group-ids sg-EXAMPLE \
--query 'SecurityGroups[*].IpPermissions'
Do not assume a security group ID from a screenshot or example. Identify the actual RDS security group and the effective source security group first.
Root cause evidence chain
- Terraform changed the RDS ingress source security group.
- The infrastructure change occurred before the payment errors.
- Payment API logs show database connection timeouts.
- RDS remains operational and responsive from an authorized network.
- The application network cannot establish new TCP connections to port 5432.
- Restoring the approved application security group rule restores connectivity and payment processing.
This combination of change history, network testing, application logs, and successful recovery provides much stronger causal evidence than timing alone.
Class 2 interview challenge
“How would you prove the Terraform change caused the payment failures rather than a PostgreSQL outage?”
Expected answer:
“I would compare the Terraform plan and AWS security group configuration with the last known-good state, verify that the database itself is responsive, and test connectivity from the affected application network. I would correlate the network failures with the infrastructure change and confirm that restoring the correct rule resolves the connectivity issue.”
Class 3 — Recovery, prevention, and interview mastery
3 hours · Students act as incident responders and interview candidates
Class 3 schedule
| Time | Activity |
|---|---|
| 0:00–0:30 | Decide on mitigation and evaluate rollback risk |
| 0:30–1:00 | Restore database connectivity |
| 1:00–1:30 | Verify Kubernetes, database, and customer transactions |
| 1:30–2:00 | Write the incident report and preventive actions |
| 2:00–2:40 | Advanced technical interview |
| 2:40–3:00 | Final individual interview assessment |
Task 1 — How do you restore production?
The platform engineer has identified the incorrect RDS security group ingress rule.
The incident commander asks:
“What is the safest and fastest way to restore service?”
The team should evaluate two possible recovery approaches.
Option A — Revert the incorrect Terraform change. Restore the previously approved application security group rule in code, review the plan, obtain the necessary approval, and apply the corrective configuration.
Option B — Emergency network change. If the approved emergency procedure allows an immediate AWS-side correction, an authorized engineer can restore the exact known-good rule and subsequently reconcile Terraform code and state. This must be documented and reviewed to avoid configuration drift.
Do not open PostgreSQL to 0.0.0.0/0, disable all security groups, or expose RDS publicly as a troubleshooting shortcut.
Example corrected Terraform configuration
resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
security_group_id = aws_security_group.rds.id
referenced_security_group_id = aws_security_group.eks_application.id
ip_protocol = "tcp"
from_port = 5432
to_port = 5432
description = "PostgreSQL access from approved EKS application SG"
}
Interviewer asks
“Would you simply run terraform apply again?”
No. First inspect the planned changes. Applying the same incorrect configuration again would not solve the incident.
“Would you run terraform destroy?”
No. Destroying production infrastructure would be highly dangerous and unrelated to this targeted connectivity issue.
“Would you roll back the Kubernetes Deployment?”
Not automatically. If the root cause is an RDS security group rule, reverting the application image may have no effect. The corrective action should target the confirmed cause.
Task 2 — Verify recovery at every layer
After the approved rule is restored, verify the application network can connect to PostgreSQL.
# Verify application Pod health
kubectl get pods -n payments-prod
# Verify Deployment availability
kubectl get deployment payment-api -n payments-prod
# Verify Service endpoints
kubectl get endpointslices -n payments-prod
# Check recent application errors
kubectl logs deployment/payment-api \
-n payments-prod --tail=100
# Confirm application rollout state
kubectl rollout status \
deployment/payment-api \
-n payments-prod
Then verify database connectivity using an approved diagnostic path, confirm the application can perform its expected queries, and run a controlled synthetic payment test using a dedicated nonfinancial test transaction or payment-provider sandbox.
Recovery criteria
Students must demonstrate all five checks
Payment API can establish PostgreSQL connections
Database queries succeed without exposing sensitive records
Customer-facing 5xx rate returns to normal
Payment success rate returns to expected baseline
No unintended duplicate payments or unresolved transaction states
0 of 5 recovery checks completed
Remember: A green Grafana dashboard alone does not establish that customer transactions are correct.
Task 3 — Write the post-incident report
Postmortem — INC-2026-1047
Incident summary
Payment authorization was degraded following a production Terraform networking change.
Customer impact
The illustrative payment success rate declined from 99.95% to 41%, with elevated HTTP 5xx errors and latency.
Technical root cause
An infrastructure refactor changed the source security group allowed to reach RDS PostgreSQL on TCP port 5432. Application Pods remained Ready because their readiness checks did not verify database-dependent business functionality.
Contributing factors
- Terraform validation confirmed configuration syntax but did not test application-to-database connectivity.
- The CI/CD pipeline lacked a post-apply payment workflow smoke test.
- The alerting design detected payment failures but did not immediately identify the affected network dependency.
- The readiness probe was insufficient as an indicator of end-to-end payment health.
Mitigation
The team restored the approved database ingress rule and verified application connectivity, transaction processing, and recovery metrics.
Preventive actions
Introduce security group policy checks, infrastructure change review, post-deployment connectivity testing, synthetic payment monitoring, and clearer incident runbooks.
Who owns the preventive actions?
| Action | Owner | Verification |
|---|---|---|
| Restore and review security group configuration | Platform engineer | Terraform plan and AWS configuration |
| Add network policy tests to CI/CD | Platform / security | Automated policy checks |
| Add post-apply connectivity smoke test | DevOps / SRE | Successful synthetic check |
| Improve payment error metrics | Backend team | Grafana dashboard and alerts |
| Improve transaction reconciliation | Backend / database | Controlled recovery tests |
| Update incident runbook | SRE | Reviewed incident exercise |
| Review production change approvals | Release engineering | Change audit |
Practical hands-on lab — Build the failure safely
Students, you will reproduce the application-to-database connectivity failure in your own training EKS cluster.
You will not modify an actual bank, customer database, or production security group.
The safest classroom version uses a PostgreSQL Pod inside a dedicated namespace and a Kubernetes NetworkPolicy to block connectivity. This reproduces the symptom and troubleshooting workflow without changing live AWS infrastructure.
For the Terraform portion, you will separately review an example security group change and its plan. An optional isolated AWS RDS exercise can be added later when your instructor has prepared a dedicated account, budget, and recovery procedure.
Lab architecture
Step 1 — Create an isolated namespace
kubectl create namespace payments-incident-lab
Step 2 — Deploy PostgreSQL
This database is temporary and contains no real customer data.
apiVersion: v1
kind: Secret
metadata:
name: lab-postgres-secret
namespace: payments-incident-lab
type: Opaque
stringData:
POSTGRES_PASSWORD: "lab-only-change-me"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: lab-postgres
namespace: payments-incident-lab
spec:
replicas: 1
selector:
matchLabels:
app: lab-postgres
template:
metadata:
labels:
app: lab-postgres
spec:
containers:
- name: postgres
image: postgres:16
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: payments
- name: POSTGRES_USER
value: labuser
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: lab-postgres-secret
key: POSTGRES_PASSWORD
readinessProbe:
exec:
command:
- sh
- -c
- pg_isready -U labuser -d payments
periodSeconds: 5
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
name: lab-postgres
namespace: payments-incident-lab
spec:
selector:
app: lab-postgres
ports:
- port: 5432
targetPort: 5432
Save this as postgres-lab.yaml and apply:
kubectl apply -f postgres-lab.yaml
kubectl rollout status deployment/lab-postgres \
-n payments-incident-lab
This lab uses ephemeral PostgreSQL storage. Data is lost when the database Pod is replaced. That is acceptable for a disposable connectivity exercise but not for a production database.
Step 3 — Verify normal connectivity
Use a temporary PostgreSQL client Pod.
kubectl run payment-db-client \
-n payments-incident-lab \
--image=postgres:16 \
--restart=Never \
--env="PGPASSWORD=lab-only-change-me" \
--command -- sleep 3600
Wait until it is Running:
kubectl wait --for=condition=Ready \
pod/payment-db-client \
-n payments-incident-lab \
--timeout=120s
Run a database query:
kubectl exec -n payments-incident-lab \
payment-db-client -- \
psql -h lab-postgres \
-U labuser -d payments \
-c "SELECT 1;"
Expected:
?column?
----------
1
(1 row)
This proves that the client can reach PostgreSQL before the incident is introduced.
The example password is deliberately disposable. Never use real database credentials in shared class terminals or recorded commands.
Step 4 — Inject the incident
Apply a NetworkPolicy that selects the PostgreSQL Pod and allows no incoming connections.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: block-postgres-ingress
namespace: payments-incident-lab
spec:
podSelector:
matchLabels:
app: lab-postgres
policyTypes:
- Ingress
ingress: []
Save as block-postgres.yaml:
kubectl apply -f block-postgres.yaml
Prerequisite: The cluster must have a network policy implementation that enforces Kubernetes NetworkPolicy, such as a properly configured Amazon VPC CNI network policy agent or another compatible CNI. Otherwise, the policy may exist without actually blocking traffic.
Test again with a timeout:
kubectl exec -n payments-incident-lab \
payment-db-client -- \
sh -c 'PGCONNECT_TIMEOUT=5 psql -h lab-postgres -U labuser -d payments -c "SELECT 1;"'
Expected on a correctly enforcing cluster:
connection to server at "lab-postgres" failed:
Connection timed out
Students should now investigate why PostgreSQL is Running and Ready but the client cannot connect.
kubectl get pods -n payments-incident-lab
kubectl get svc -n payments-incident-lab
kubectl get networkpolicy -n payments-incident-lab
kubectl describe networkpolicy block-postgres-ingress \
-n payments-incident-lab
Student challenge
You are not told that the instructor introduced a NetworkPolicy.
Your job is to identify the exact resource that changed and explain how it prevents database access.
Step 5 — Restore service
Once you have identified and documented the cause, remove the injected policy:
kubectl delete networkpolicy block-postgres-ingress \
-n payments-incident-lab
Test again:
kubectl exec -n payments-incident-lab \
payment-db-client -- \
psql -h lab-postgres \
-U labuser -d payments \
-c "SELECT 1;"
Expected: SELECT 1 succeeds.
Clean up the client:
kubectl delete pod payment-db-client \
-n payments-incident-lab
When the entire lab is complete, the instructor can delete the dedicated namespace.
Important difference between the lab and production
The production case study involves an AWS security group changed by Terraform. The hands-on exercise uses Kubernetes NetworkPolicy to reproduce blocked database traffic safely.
Both teach network connectivity troubleshooting, but they are different control planes. Students must be able to explain that distinction during an interview.
Also, this initial lab tests connectivity directly; it does not yet deploy a complete payment API, transaction simulator, or the custom payment metrics used in the fictional dashboard. Those would be additional components if the class needs a fully instrumented end-to-end payment workflow.
Advanced interview question bank
Practice the following questions without looking at the answers first.
Incident response
1 / 17
You receive a payment failure alert. What are your first five actions?
Show model answerFollow-up
PreviousNext
Final mock interview: Tell me about a critical production incident
Students, your final task is to explain the entire incident in approximately three minutes. This is a model answer for discussing the case study. Adapt it to your actual experience rather than claiming to have personally resolved an incident you only practiced.
Production Incident — Payment API Database Connectivity Failure
One complex production incident scenario I worked through involved a customer-facing payment processing application running on AWS EKS with PostgreSQL on Amazon RDS.
The environment used Terraform for infrastructure management, GitHub Actions for CI/CD, and Prometheus, Grafana, CloudWatch, and PagerDuty for monitoring and alerting.
The incident began when the payment success rate dropped significantly, from approximately 99.95% to 41%. Customers were experiencing payment failures and HTTP 5xx errors.
My first step would be to acknowledge the PagerDuty alert and investigate the customer impact through Grafana. I would check the error rate, latency, request volume, and Kubernetes availability.
Interestingly, all 12 Payment API Pods were Running and Ready, and the RDS database itself reported Available.
This indicated that the problem might be outside basic Kubernetes Pod health.
I would inspect the application logs, where I would find PostgreSQL connection timeouts. I would then work with the database and cloud platform teams to investigate database health and the network path from EKS to RDS.
I would check DNS resolution, TCP connectivity on port 5432, security group rules, and any relevant Kubernetes NetworkPolicies.
At the same time, I would review recent production changes. The release engineering team would identify a successful Terraform infrastructure deployment shortly before the errors started.
By comparing the Terraform change with the previous known-good configuration, we would identify that the RDS security group ingress rule had been changed to reference the wrong source security group.
The database itself was operational, but the application could no longer reliably establish new database connections.
I would coordinate with the incident commander and platform team to restore the previously approved security group rule using the authorized emergency change procedure.
After the correction, I would verify connectivity from the application network, confirm successful database operations, and monitor payment success rate, HTTP errors, latency, and transaction reconciliation.
I would not declare recovery based only on Kubernetes Pod status. I would verify that the customer-facing payment workflow was functioning correctly.
Following recovery, I would participate in the postmortem and recommend stronger Terraform policy checks, post-deployment connectivity testing, synthetic payment monitoring, and improved incident runbooks.
The main lesson from this incident is that healthy Kubernetes Pods and a healthy database do not necessarily mean the complete application is healthy. As a DevOps engineer, I need to investigate the full request path and work across infrastructure, application, database, and security teams to restore service safely.
Final evaluation rubric
Use this scorecard to assess each student during Class 3.
Mock interview score
Adjust scores as the student answers
0 / 100
Needs more practice
Incident impact and first response
0 / 15
Kubernetes investigation
0 / 15
Database and networking diagnosis
0 / 20
Terraform and CI/CD investigation
0 / 20
Mitigation and recovery verification
0 / 15
Communication and postmortem
0 / 15
Reset for next student
What interviewers are really evaluating
This incident is especially useful for DevOps Engineer, Site Reliability Engineer, Platform Engineer, Cloud Infrastructure Engineer, and DevSecOps interviews because it tests several competencies together:
- Kubernetes: Understanding that Pod readiness is not the same as business functionality.
- AWS: Knowing how VPC networking, security groups, EKS, and RDS interact.
- Terraform: Understanding change management, infrastructure drift, and safe recovery.
- CI/CD: Recognizing that a successful pipeline does not guarantee a successful production outcome.
- Database: Knowing how to distinguish connectivity, authentication, performance, and availability failures.
- Monitoring: Using technical and business metrics together.
- Incident management: Communicating, escalating, mitigating, verifying, and documenting.
Students should be able to explain not just which command they would run, but what question that command answers.
Your final assignment: Explain why the Payment API could have 12 healthy Kubernetes Pods, an Available PostgreSQL RDS instance, and a successful Terraform deployment while customer payments were failing—and describe the evidence needed to identify and fix the real cause.
That is the central engineering challenge for these three classes.
Top comments (0)