DEV Community: Aisalkyn Aidarova

Tell me about yourself.

Aisalkyn Aidarova — Fri, 06 Nov 2026 20:11:57 +0000

Tell me about yourself.
I am DevOps Engineer with more than 5 years experience. I started at Southwest Airlines as Cloud Engineer, and after one year I got promoted to DevOps Engineer. Since September 2023 I work at Capital One. Mostly I work with AWS, Kubernetes, Terraform and GitLab CI/CD. I also used Azure, and monitoring tools like Prometheus, Grafana and Datadog. I support production, do on-call, and work on reliability, security and cost. I like to automate manual work and help developers ship apps safely.
Why BigTime?
You are building AI-powered platform, and I think AI brings many good opportunities for DevOps. Your stack is AWS, Kubernetes, Terraform and GitLab, and this is what I work with every day. I also like that engineers get ownership and solve problems independently.
What is your day to day?
In the morning I check alerts and dashboards in Datadog and Grafana. Then standup. Most of the day I work on Terraform, GitLab pipelines and deployments to EKS with Helm. I help developers when deployments fail. I do patching, upgrades, cost and security work. When I am on-call, I handle incidents. I update documentation in Confluence.
Tell me about a hard production incident.
After a release we had 60 replicas in 3 Availability Zones, and HTTP 500 errors went up to 18 percent. I was the on-call engineer. I checked dashboards, logs, Kubernetes deployment status and the deployment history. The errors started with the release, so we rolled back to the last stable version. I checked that pods were healthy and the error rate went down. Root cause was wrong database connection setting in the new version. After this I added canary deployments, smoke tests in the pipeline, automatic rollback on high error rate, and wrote a postmortem and runbook.
How do you replace long-lived credentials?
I move secrets to Vault or AWS Secrets Manager with automatic rotation. Apps on EKS use IAM roles for service accounts, so pods get short-lived credentials. GitLab pipelines use OIDC to assume IAM roles, no static AWS keys. EC2 and Windows servers use instance roles. For RDS, passwords are in Secrets Manager with rotation. This helps with SOC 2 audits too.
Windows EC2 in production, you cannot RDP, and you cannot restart. What do you do?
First I check if the app still works for users, maybe only RDP is broken. I check status checks, security group port 3389, NACL, route table, and the Route 53 record. Then I use SSM Session Manager or Run Command with PowerShell. I check if the RDP service is running and restart only this service, check Windows firewall, CPU, memory and disk space. I look at the console screenshot, system log and CloudWatch metrics. If restart is really needed, I take a snapshot, move traffic to other instances behind the load balancer, talk to the team, and do it in a maintenance window.
How did you save $160K?
I added cost allocation tags so we see who spends what. I rightsized big EC2 instances, turned off dev and test at night and weekends, cleaned unused EBS volumes, snapshots and load balancers. S3 lifecycle policies to cheaper storage. Better EKS node sizing and autoscaling. Savings Plans and Reserved Instances for stable workloads. Budgets and alerts in Cost Explorer and CloudWatch.
How did you reduce MTTD by 25 percent?
I made better dashboards in Grafana and Datadog for each service. Alerts based on real symptoms like error rate, latency and pod restarts. I removed noisy alerts so the team trusts them. I added health checks and synthetic monitoring. Alerts go to PagerDuty and Slack with runbook links.
Biggest issue you fixed with Grafana?
Booking service on EKS was slow every few days. I made a dashboard with memory, restarts and latency together. Memory went up slowly, then pods got OOMKilled and latency jumped. It was a memory leak and low memory limits. Developers fixed the leak, I fixed requests and limits and added alerts for memory growth and restarts.
What are the RDS replication types?
Multi-AZ is synchronous replication to a standby in another Availability Zone, for high availability and automatic failover, you cannot read from it. Read Replicas are asynchronous, for scaling reads and reporting, can be cross-region, and can be promoted for disaster recovery. SQL Server Multi-AZ uses Always On or mirroring.
RDS vs Aurora?
Standard RDS is managed database with its own EBS storage per instance. Aurora is AWS cloud database for MySQL and PostgreSQL. Storage is shared and separate from compute, data copied 6 times across 3 Availability Zones, storage grows automatically. Up to 15 replicas with very low lag, failover under 30 seconds. Aurora is faster but more expensive. Aurora does not support SQL Server.
GitLab pipeline with two different AWS roles, how do you configure?
I use GitLab OIDC. The job gets an ID token and assumes IAM role. Role one is read-only for terraform plan, runs on merge requests. Role two has write permissions for apply and deploy, and the trust policy allows only main branch, protected branch and production environment. Stages are validate, plan, apply, deploy. Apply is manual approval with the saved plan. Deploy updates kubeconfig and runs helm upgrade. In EKS I map the role to RBAC with access entries, so it deploys only to its namespaces.
What happens when you type bigtime.net in browser?
DNS finds the IP with UDP port 53, through the resolver to Route 53. Then TCP three-way handshake, SYN, SYN-ACK, ACK. Then TLS handshake on port 443, server sends certificate and they agree on keys. Then HTTP GET request. Traffic goes through CloudFront or load balancer, then to EC2 or EKS, and the app talks to RDS. Response comes back and browser shows the page.
How do you make sure the app uses S3 securely, encryption in transit?
Bucket policy denies requests when aws:SecureTransport is false, so only HTTPS. I can require TLS 1.2 with s3:TlsVersion. App uses IAM role, least privilege, no keys. VPC gateway endpoint so traffic stays private, and bucket policy limited to this endpoint. Block Public Access on. Check with CloudTrail and AWS Config rule s3-bucket-ssl-requests-only.
Some S3 objects are not encrypted with your KMS key, why and how to fix?
Default encryption works only for new objects, so old objects keep old encryption. Also if the client sends its own encryption header, it overrides the default. I find them with S3 Inventory. I re-encrypt with S3 Batch Operations copy in place. To prevent, bucket policy denies PutObject if encryption is not aws:kms or the key ID is not my key ARN. KMS key policy allows only the right roles. AWS Config for monitoring.
Same Terraform module in dev and prod, you add an IP range, dev plan is clean, prod plan wants to destroy route table associations. Why?
The module uses count with a list. Terraform tracks resources by index. In prod the list is longer or in different order, so new item shifts indexes and Terraform wants to destroy and recreate. Fix is for_each with stable keys like the IP range. Use moved blocks or terraform state mv to move existing resources without destroy. Add prevent_destroy on critical resources, and require approval in the pipeline if prod plan shows destroy. Also check drift with plan -refresh-only and provider versions.
Kubernetes app gets 502 for about 10 seconds during every deployment. Readiness probe is present. What do you check?
Most likely old pods shutting down. Kubernetes sends SIGTERM and removes the pod at the same time, but the load balancer still sends traffic for few seconds. Fix is preStop hook with sleep 10 to 15 seconds, app handles SIGTERM gracefully, terminationGracePeriodSeconds longer than the sleep. Rolling update with maxUnavailable 0 and maxSurge 1. With ALB, enable pod readiness gates and check deregistration delay and keep-alive timeout. Add PodDisruptionBudget. Check ALB logs and pod events.
How do you use AI in your work?
I use chat for quick questions and errors. For bigger tasks I use AI agents like Claude Code. It reads the repo, makes a plan, changes Terraform or pipelines, and runs terraform plan or tests. I review the plan and the changes before merge. Agents use tools through MCP and need limited permissions, like read-only in prod and no apply without human approval. I never put secrets or customer data into AI tools.

Next round they may also ask these, because they are in the job description:

How do you deploy and monitor .NET apps?
I build the app in GitLab CI with dotnet build and dotnet test, then package it. For Windows on EC2, I deploy with PowerShell scripts or SSM Run Command to IIS, and take the server out of the load balancer during deploy. For Linux, I put .NET Core in Docker and deploy to EKS or ECS with Helm. For monitoring I use CloudWatch agent for Windows logs and IIS logs, and Datadog APM or OpenTelemetry for traces, errors and response time.
How do you patch Windows and Linux servers?
I use AWS Systems Manager Patch Manager with patch baselines and maintenance windows. First patch dev and test, then prod. For prod, I patch in groups, so not all servers behind the load balancer go down at once. Before patching I take snapshots. After patching I check the app health and the compliance report in SSM. For EKS nodes I update the AMI and do rolling node replacement.
EKS vs ECS vs App Runner?
EKS is Kubernetes, more control and flexible, good for complex apps, but more work to manage. ECS is AWS own container service, simpler, works well with Fargate and other AWS services. App Runner is the simplest, you give it image or code, and AWS handles everything, good for small web apps and APIs.
What is Helmfile?
Helmfile manages many Helm releases in one file. You define all charts, versions and values for each environment in one place, and run helmfile apply. It is good for keeping environments consistent and works well in CI/CD.
What is Terragrunt and why use it?
Terragrunt is a wrapper for Terraform. It keeps the code DRY, so you don’t copy the same backend and provider config in every environment. It manages remote state config, dependencies between modules, and lets you run plan or apply across many modules and accounts.
How do you help with SOC 2?
I make sure access is least privilege with IAM roles and SSO, encryption at rest with KMS and in transit with TLS, logs in CloudTrail and CloudWatch, vulnerability scanning with Inspector or image scanning in the pipeline, and patching. I keep evidence and documentation ready for auditors, like access reviews and change history from GitLab merge requests.
What is your disaster recovery approach?
First we agree on RTO and RPO with the business. Backups for RDS with automated snapshots and cross-region copies, S3 versioning and replication, all infrastructure in Terraform so we can rebuild fast. We write a DR runbook and test it, because a plan that is not tested does not work.
Have you worked with Lambda?
Yes, I used Lambda for automation, like stopping dev instances at night, cleaning old snapshots, and processing S3 events. I deploy it with Terraform, give it a small IAM role, and monitor with CloudWatch logs and alarms for errors and duration.
What is Ansible used for?
Configuration management. It connects to servers over SSH for Linux or WinRM for Windows, and runs playbooks to install software, change config and patch. It is agentless and idempotent, so you can run it many times and it gives the same result.
Biggest weakness?
Sometimes I go too deep into a problem and want to fix it myself. Now I set a time limit, and if I don’t solve it, I ask the team early.
Where do you see yourself in 3 to 5 years?
I want to grow into Senior or Lead DevOps Engineer, own bigger parts of the platform, and help build automation and AI tools for the team.

HERC RENTALS INTERVIEW PREP Cloud and Platform Engineering Manager WHAT WE LEARNED IN ROUND ONE

Aisalkyn Aidarova — Fri, 06 Nov 2026 20:09:53 +0000

HERC RENTALS INTERVIEW PREP
Cloud and Platform Engineering Manager

WHAT WE LEARNED IN ROUND ONE

Team is 5 engineers, all senior and technically strong.
Main problem is no clear structure or daily plan, people work in different directions.
App teams do not fully understand the pipelines and need a simple standard path.
Architecture and infra decisions are made together as a team, but there is no formal process.
They use EKS for many microservices and ECS Fargate for small apps.
Identity is Okta.
AI is still at concept stage, integrating into backend apps, user access is limited, and they are careful because people misuse it.
The manager wants someone who observes the current process first, finds gaps day to day, works close with another manager, and removes leadership blockers.
Job post clouds are AWS, Azure, OCI with Terraform, CloudFormation and Bicep.
Job is on-site in Bonita Springs, Florida.

YOUR PITCH, TELL ME ABOUT YOURSELF

Thank you for having me again. I am a DevOps engineer with more than five years of experience on AWS. Four years at Yelp and now at Zulily. My daily work is Terraform, Kubernetes on EKS, GitHub Actions pipelines and CloudWatch monitoring. At Yelp I improved deployment speed by 35 percent with blue-green deployments and automatic rollback. I also mentored junior engineers, wrote runbooks and helped onboard new people, and that is the part I enjoy most. In our first conversation I learned your team is very senior but needs structure and a clear path for app teams. That is exactly what I like to build. I want to stay hands-on and also help the team move in one direction.

Numbers to remember: 5 plus years, 35 percent faster deployments, CKAD January 2025, AWS Solutions Architect Associate August 2026.

LEADERSHIP ANSWERS

What is your leadership experience?
I have not been a formal manager with direct reports yet. But at Yelp I mentored junior engineers, onboarded new people, wrote runbooks and led knowledge sharing. A team of five is a good size for me, I can coach and still be hands-on.

How would you lead senior engineers?
I would not micromanage. I give clear goals and priorities, and each senior person owns one area, like Terraform, CI/CD or EKS. I stay hands-on with them on hard problems. Senior people want trust and direction, not control.

How do you resolve conflict between two engineers?
I bring both together and let each explain without interrupting. We go back to the goal, cost, security, reliability or time. I ask for facts, data or a small proof of concept. If they still disagree, I decide, explain why, and write it down. Then we disagree and commit. At Yelp two engineers disagreed on speed versus safety, and we chose blue-green with automatic rollback, which gave us 35 percent faster deployments.

One engineer is not meeting expectations, what do you do?
Private one on one first, I listen and ask what is going on. Then I make expectations clear with examples and small goals with dates. I give support, pair them with a strong engineer and remove blockers. Weekly check-ins. If no progress, I work with HR on a fair, documented plan.

How do you prioritize when everything is urgent?
Business impact and risk. Production and security issues first, then work that unblocks app teams, then improvements. I agree on top priorities with my manager every week so the team is not pulled in many directions.

How do you report to leadership?
Short and simple. A monthly update with roadmap, risks, cloud cost trend, and a few metrics like deployment frequency, failed changes, incidents and time to recover.

How do you handle an outage as the escalation point?
First stop the impact with rollback or failover. Update people every 30 minutes. One person leads, others fix. After, a blameless postmortem with clear action items and owners.

TECHNICAL ANSWERS FROM ROUND ONE

Recent architecture you deployed?
At Zulily, an app on EKS with DocumentDB. VPC across multiple availability zones. Public subnets only for the load balancer, private subnets for EKS nodes and the database. Route 53 to the load balancer with TLS, then to pods. DocumentDB is private, only the app security group can reach it, IAM least privilege. HPA adds pods, Cluster Autoscaler adds nodes, so it grows during sales and shrinks after. Everything in Terraform modules, same code for dev, staging and prod. GitHub Actions with tests, terraform plan review and approval before prod. CloudWatch for alerts.

Kubernetes experience?
Four years on EKS at Yelp and now Zulily, CKAD certified. Helm charts so every team deploys the same way. Liveness and readiness probes for self-healing. HPA and Cluster Autoscaler. I troubleshoot CrashLoopBackOff, image pull errors and resource limits with kubectl describe, logs and events.

EKS versus ECS Fargate?
EKS fits many microservices, more control, Helm, one deployment pattern. Fargate fits small apps, no nodes to manage, less work. My hands-on is EKS, ECS is new for me but the ideas are very close.

Event based architecture and Lambda?
Services do not call each other directly, one sends an event and others react. EventBridge routes events, SNS sends to many, SQS queues so nothing is lost, Lambda runs code when an event comes. Watch the 15 minute timeout, memory, cold starts, IAM role, dead letter queues, and make processing safe to run twice. I am familiar, but have not built big systems myself.

Identity?
AWS IAM roles and least privilege. GitHub Actions deploys with IAM roles, no long term keys. IAM roles for Kubernetes service accounts on EKS. Okta and Active Directory I know the ideas, single sign on, MFA, groups mapped to AWS roles through IAM Identity Center.

How do you use AI?
Claude AI daily as a helper. Draft pipelines, review Terraform, explain errors, write documentation. Always review and test, never share secrets or customer data. I agree with your careful approach, start small, approved tools, human review, and guardrails like logging and access control.

NEW TOPICS LIKELY IN ROUND TWO

Landing zones
Key words: AWS Organizations, Control Tower, separate accounts per environment, SCPs, central logging.
Answer: A landing zone is the secure starting setup for all cloud accounts. Separate accounts for prod, non-prod, security and logging, with guardrails so teams cannot break the rules.

Azure basics
Key words: subscriptions, resource groups, VNet, AKS, Entra ID, Bicep.
Answer: Azure is very similar to AWS. VNet is like VPC, AKS is like EKS, Entra ID is like IAM, Bicep is like CloudFormation. Terraform works on both.

OCI basics
Key words: tenancy, compartments, VCN, OKE.
Answer: OCI uses compartments to organize resources, VCN is like VPC, OKE is their Kubernetes. Often used for Oracle databases and ERP.

Terraform at scale
Key words: modules, remote state, state locking, separate folders per environment, plan review in pull request.
Answer: I already do this at Zulily with S3 state and DynamoDB locking. The same pattern works with Azure and OCI providers.

Hybrid networking
Key words: site to site VPN, Direct Connect, ExpressRoute, Transit Gateway, private DNS.
Answer: It connects on-prem to cloud. VPN is quick and cheap, Direct Connect or ExpressRoute is private and faster for production.

Disaster recovery
Key words: RTO, RPO, backup and restore, pilot light, warm standby, multi-region.
Answer: RTO is how fast we recover, RPO is how much data we can lose. We choose the strategy by business need and cost, and we test restores regularly.

Cloud cost, FinOps
Key words: tagging, right-sizing, Savings Plans, Reserved Instances, spot, Cost Explorer, budgets and alerts.
Answer: First tag everything so we know who spends what. Then right-size, turn off unused dev at night, Savings Plans for steady work, spot for flexible work. Monthly cost report to leadership.

Security
Key words: Secrets Manager, KMS encryption, GuardDuty, Security Hub, image scanning, CIS benchmarks.
Answer: Least privilege, encrypt everything, no secrets in code, scan images in the pipeline, and work with the security team to fix findings by priority.

Observability
Key words: metrics, logs, traces, CloudWatch, Datadog, Prometheus and Grafana, alert fatigue.
Answer: Alert only on what users feel, every alert has a runbook, same dashboards across clouds.

Self-service platform
Key words: golden path, pipeline templates, Helm and Terraform modules, developer portal like Backstage.
Answer: App teams get a standard template and can deploy alone, the platform team keeps guardrails. This fixes the pipeline confusion they told us about.

Vendors and managed services
Key words: SLAs, quarterly reviews, cost, escalation path.
Answer: Clear SLAs, regular review meetings, and our team still understands what the vendor runs.

HONEST GAP ANSWERS

Pattern for every gap: say honestly you have not done it in production, show you know the idea, connect it to something you did, end with how fast you learn.

Formal manager: Not formal yet, but mentoring, onboarding, runbooks and knowledge sharing at Yelp. Ready for the next step.
Azure and OCI: Not in production, my work is AWS. Concepts map one to one and Terraform works on all three, so I can be productive fast.
Bicep and CloudFormation: Terraform is my main tool. Same idea, declare resources, review the plan, deploy.
ECS: Hands-on is EKS. ECS task definitions and services are close to Kubernetes deployments.
Active Directory and Okta: Not managed them, but I know single sign on, MFA, groups mapped to cloud roles. Strong IAM background.
Event driven and Lambda at scale: Familiar, not built big systems. I can support it with Terraform, IAM and monitoring.
FinOps and DR ownership: Have not owned it fully, familiar with right-sizing, Savings Plans, backup and restore testing, RTO and RPO.
Vendor management: Not owned vendor contracts yet, but I know the basics, clear SLAs and regular reviews.

30 60 90 DAY PLAN

First 30 days, listen and learn. One on one with each of the 5 engineers and the other manager. Sit with app teams and watch where they get stuck with pipelines. Review the AWS, Azure and OCI setup, Terraform code, incidents and cloud cost. Write a gap list and agree on top priorities with my manager.

Days 31 to 60, quick wins and structure. Weekly planning and a short daily sync so work is visible. Each senior engineer owns one area. Simple design doc and weekly design review for bigger decisions. First standard pipeline template for app teams.

Days 61 to 90, scale it. Golden path documentation and runbooks so app teams onboard alone. Monthly report to leadership with roadmap, risks, cost and delivery metrics. Plan for cost tagging, DR testing and safe AI tools.

QUESTIONS TO ASK

Do not repeat what they already answered: team size, seniority, Okta, EKS and ECS, AI stage.

For the hiring manager or director:
What would make you say after 6 months, this was the right hire?
How is work split between AWS, Azure and OCI, and is there a plan to consolidate?
How does this role divide work with the other manager?
What is the biggest risk on the platform today?

For engineers on the team:
What slows you down most in a normal week?
How do app teams ask you for help today, tickets, Slack or meetings?
What would you want from a new manager?

For HR or recruiter:
Is there relocation support for the move to Bonita Springs?
Is there a leadership program for new managers?
What are the next steps and timeline?

LOGISTICS AND FINAL TIPS

Relocation: have a clear yes and a realistic start date ready, because you live in Vancouver, WA and the job is on-site in Florida.
Salary: research the range for cloud engineering managers in Southwest Florida and give a range, not one number.
About Herc: they rent equipment to construction, industrial and event customers. Read their latest news the day before and mention one recent fact.
Every answer: short, one real example, end with a question.
Use their words back: structure, observe first, golden path for app teams, careful with AI. It shows you listened.
Send a thank you email the same day and mention one thing you discussed.
Practice the pitch out loud three times.
Practice the Azure and OCI mapping to AWS.
Prepare one outage story and one cost saving idea.

JumpToTech DevOps/SRE Capstone Project EstateFlow — Enterprise AWS EKS Production Environment

Aisalkyn Aidarova — Fri, 06 Nov 2026 04:02:22 +0000

1. The complete architecture

EstateFlow — target deployment architecture

Conceptual architecture. The database and Redis shown here are lab workloads; production environments would normally require stronger availability, backups, and disaster-recovery design.

What students will actually implement

Area	Implementation	What students must explain
Application	React + Node.js + PostgreSQL + Redis	Request flow, API, database dependencies
Infrastructure	Terraform, VPC, EKS, ECR, IAM	Infrastructure as Code
CI	GitHub Actions, tests, SonarQube, Trivy	Quality and security gates
CD	Helm + Argo CD	GitOps, drift, rollback
Networking	Services, ALB, Ingress, DNS	External and internal traffic
Security	RBAC, IAM, Pod Identity, NetworkPolicy	Authorization and isolation
Storage	StatefulSet, PVC, EBS CSI	Persistence and recovery
Scheduling	Affinity, anti-affinity, taints, tolerations	Pod placement
Reliability	HPA, PDB, probes, rolling updates	Availability and scaling
Operations	Prometheus, Grafana, alerting	Incident detection and diagnosis
Incidents	503, Pending, CrashLoopBackOff, PVC failures	Root cause, mitigation, postmortem

Important architecture decisions

For this lab, PostgreSQL and Redis run inside Kubernetes so students can learn StatefulSets, persistent volumes, and storage troubleshooting. For a higher-availability production design, I would normally evaluate Amazon RDS/Aurora and ElastiCache instead.

We will demonstrate the built-in workload controllers and selected AWS controllers, not indiscriminately install every Kubernetes controller. Each controller must have a reason to exist.

The application will have two deployment environments, estateflow-staging and estateflow-prod, plus dedicated infrastructure namespaces. We will only call the second one “prod” as a classroom environment.

2. Repository structure

Students will use one application repository and one GitOps repository.

jumptotech-estateflow/
├── frontend/
│   ├── src/
│   ├── Dockerfile
│   ├── package.json
│   └── nginx.conf
├── backend/
│   ├── src/
│   │   ├── server.js
│   │   └── db.js
│   ├── tests/
│   ├── Dockerfile
│   └── package.json
├── database/
│   └── init.sql
├── terraform/
│   ├── versions.tf
│   ├── providers.tf
│   ├── main.tf
│   ├── variables.tf
│   ├── outputs.tf
│   └── terraform.tfvars.example
├── helm/
│   └── estateflow/
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
├── k8s/
│   ├── rbac/
│   ├── security/
│   ├── scheduling/
│   ├── autoscaling/
│   └── storage/
├── argocd/
├── monitoring/
├── incidents/
├── scripts/
├── docs/
├── .github/
│   └── workflows/
├── docker-compose.yml
└── README.md

jumptotech-estateflow-gitops/
├── applications/
├── environments/
│   ├── staging/
│   └── prod/
└── charts/

We will use the following naming conventions consistently:

AWS_REGION=us-east-1
CLUSTER_NAME=estateflow-lab
APP_NAME=estateflow
STAGING_NAMESPACE=estateflow-staging
PROD_NAMESPACE=estateflow-prod

The original payments-incident-lab must not be included in any cleanup script.

3. Phase 0 — Environment and safety checks

Every student runs these commands before provisioning anything.

git --version
docker --version
aws --version
kubectl version --client
terraform version
helm version --short

aws sts get-caller-identity
aws configure get region

kubectl config get-contexts
kubectl get namespaces
kubectl get nodes -o wide

Students must confirm they are using their intended AWS account.

Cost warning: EKS, EC2 worker nodes, load balancers, EBS volumes, NAT gateways, public IPv4 addresses, and monitoring resources can incur charges. The cluster must be deliberately sized and destroyed after the project. A multi-AZ production-like environment is not a free-tier lab.

4. Phase 1 — Create the application

We will first build a working backend and frontend locally. Kubernetes must not be used to hide application defects.

4.1 Backend — Node.js API

mkdir -p jumptotech-estateflow/backend/src
cd jumptotech-estateflow/backend

cat > package.json <<'EOF'
{
  "name": "estateflow-backend",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node src/server.js",
    "test": "node --test"
  },
  "dependencies": {
    "express": "^4.21.2",
    "pg": "^8.13.1",
    "redis": "^4.7.0",
    "prom-client": "^15.1.3"
  }
}
EOF

npm install

Create backend/src/server.js:

import express from "express";
import pg from "pg";
import { createClient } from "redis";
import client from "prom-client";

const app = express();
app.use(express.json());

const pool = new pg.Pool({
  host: process.env.DB_HOST || "localhost",
  port: Number(process.env.DB_PORT || 5432),
  database: process.env.DB_NAME || "estateflow",
  user: process.env.DB_USER || "estateflow",
  password: process.env.DB_PASSWORD,
  connectionTimeoutMillis: 3000
});

const redis = createClient({
  url: process.env.REDIS_URL || "redis://localhost:6379",
  socket: { reconnectStrategy: false }
});

redis.on("error", err =>
  console.error("Redis error:", err.message)
);

client.collectDefaultMetrics();

const httpRequests = new client.Counter({
  name: "estateflow_http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "route", "status"]
});

app.use((req, res, next) => {
  res.on("finish", () => {
    httpRequests.inc({
      method: req.method,
      route: req.route?.path || req.path,
      status: String(res.statusCode)
    });
  });
  next();
});

app.get("/healthz", (_req, res) => {
  res.json({ status: "alive" });
});

app.get("/readyz", (_req, res) => {
  res.json({ status: "ready" });
});

app.get("/api/db-check", async (_req, res) => {
  try {
    await pool.query("SELECT 1");
    res.json({ database: "reachable" });
  } catch (err) {
    console.error("Database check failed:", err.message);
    res.status(503).json({ error: "database unavailable" });
  }
});

app.get("/api/properties", async (_req, res) => {
  try {
    const result = await pool.query(
      "SELECT id, title, city, price FROM properties ORDER BY id"
    );
    res.json(result.rows);
  } catch (err) {
    console.error("Property query failed:", err.message);
    res.status(503).json({ error: "database unavailable" });
  }
});

app.post("/api/bookings", async (req, res) => {
  const propertyId = Number(req.body.property_id);
  const customer = req.body.customer_name;

  if (!Number.isInteger(propertyId) || propertyId < 1 ||
      typeof customer !== "string" || !customer.trim()) {
    return res.status(400).json({ error: "invalid booking request" });
  }

  try {
    const result = await pool.query(
      `INSERT INTO bookings(property_id, customer_name)
       VALUES ($1, $2)
       RETURNING id, property_id, customer_name, status`,
      [propertyId, customer.trim()]
    );
    res.status(201).json(result.rows[0]);
  } catch (err) {
    console.error("Booking failed:", err.message);
    res.status(503).json({ error: "booking unavailable" });
  }
});

app.get("/api/cache-check", async (_req, res) => {
  try {
    if (!redis.isOpen) await redis.connect();
    await redis.ping();
    res.json({ redis: "reachable" });
  } catch (err) {
    console.error("Redis check failed:", err.message);
    res.status(503).json({ error: "cache unavailable" });
  }
});

app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
});

app.listen(8080, "0.0.0.0", () => {
  console.log("EstateFlow API listening on port 8080");
});

Teaching point: /healthz and /readyz deliberately do not require database access. A database failure can therefore produce HTTP 503 on business endpoints while Pods remain Running and Ready. This is central to our incident scenario.

For an actual production application, readiness behavior must be designed around whether a Pod can serve its intended traffic; this simplified probe is chosen to demonstrate dependency failures.

4.2 Database schema

Create database/init.sql:

CREATE TABLE IF NOT EXISTS properties (
    id SERIAL PRIMARY KEY,
    title VARCHAR(200) NOT NULL,
    city VARCHAR(100) NOT NULL,
    price NUMERIC(12,2) NOT NULL
);

CREATE TABLE IF NOT EXISTS bookings (
    id SERIAL PRIMARY KEY,
    property_id INTEGER NOT NULL
      REFERENCES properties(id),
    customer_name VARCHAR(200) NOT NULL,
    status VARCHAR(30) NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

INSERT INTO properties(title, city, price)
VALUES
('Chicago Downtown Apartment', 'Chicago', 2500),
('Schaumburg Family House', 'Schaumburg', 3200),
('Naperville Townhome', 'Naperville', 2800);

For repeated initialization, students should ensure seed data is not inserted twice; a later database migration will replace this development seed.

4.3 Backend Dockerfile

Create backend/Dockerfile:

FROM node:22-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev

COPY src ./src

ENV NODE_ENV=production
EXPOSE 8080

USER node

CMD ["node", "src/server.js"]

4.4 Frontend

Create the React application:

cd ../

npm create vite@latest frontend -- \
  --template react-ts

cd frontend
npm install

Replace frontend/src/App.tsx:

import { useEffect, useState } from "react";

type Property = {
  id: number;
  title: string;
  city: string;
  price: string;
};

export default function App() {
  const [properties, setProperties] = useState<Property[]>([]);
  const [message, setMessage] = useState("");

  useEffect(() => {
    fetch("/api/properties")
      .then(r => {
        if (!r.ok) throw new Error("API unavailable");
        return r.json();
      })
      .then(setProperties)
      .catch(() => setMessage("Unable to load properties"));
  }, []);

  async function book(id: number) {
    const response = await fetch("/api/bookings", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        property_id: id,
        customer_name: "Lab Student"
      })
    });

    setMessage(response.ok
      ? "Booking submitted successfully"
      : "Booking service unavailable");
  }

  return (
    <main style={{ maxWidth: 850, margin: "auto", padding: 30 }}>
      <h1>EstateFlow</h1>
      <p>Find your next home.</p>
      <p role="status">{message}</p>

      {properties.map(property => (
        <article key={property.id}
          style={{ border: "1px solid #ccc", padding: 20, margin: 12 }}>
          <h2>{property.title}</h2>
          <p>{property.city} — ${property.price}/month</p>
          <button onClick={() => book(property.id)}>
            Request booking
          </button>
        </article>
      ))}
    </main>
  );
}

Create frontend/nginx.conf:

server {
    listen 8080;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /healthz {
        access_log off;
        return 200 "ok";
    }

    location /api/ {
        proxy_pass http://backend:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For
            $proxy_add_x_forwarded_for;
    }
}

The backend hostname above is for local Docker Compose. In Kubernetes, the frontend configuration will use the estateflow-api Service name.

Create frontend/Dockerfile:

FROM node:22-alpine AS build

WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginxinc/nginx-unprivileged:stable-alpine

COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/dist /usr/share/nginx/html

EXPOSE 8080

4.5 Local Docker Compose

Create docker-compose.yml in the repository root:

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: estateflow
      POSTGRES_USER: estateflow
      POSTGRES_PASSWORD: local-dev-only
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./database/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U estateflow"]
      interval: 5s
      retries: 10

  redis:
    image: redis:7-alpine

  backend:
    build: ./backend
    environment:
      DB_HOST: postgres
      DB_NAME: estateflow
      DB_USER: estateflow
      DB_PASSWORD: local-dev-only
      REDIS_URL: redis://redis:6379
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started

  frontend:
    build: ./frontend
    ports:
      - "8088:8080"
    depends_on:
      - backend

volumes:
  postgres_data:

Run:

docker compose up --build -d

curl -i http://localhost:8088/healthz

curl -i http://localhost:8088/api/properties

curl -i http://localhost:8088/api/db-check

curl -i -X POST \
  http://localhost:8088/api/bookings \
  -H "Content-Type: application/json" \
  -d '{"property_id":1,"customer_name":"JumpToTech Student"}'

docker compose ps

Expected healthy baseline: frontend HTTP 200, property list HTTP 200, database check HTTP 200, booking HTTP 201.

The students must save the actual responses.

5. Phase 2 — Terraform and AWS infrastructure

We will create a separate EKS cluster named estateflow-lab, rather than risking the existing training cluster.

For classroom work, use a dedicated AWS account if possible. The reference infrastructure uses two Availability Zones, managed node groups, ECR, and EKS Pod Identity.

Terraform provider

terraform/versions.tf

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

terraform/providers.tf

provider "aws" {
  region = var.aws_region
}

terraform/variables.tf

variable "aws_region" {
  type    = string
  default = "us-east-1"
}

variable "cluster_name" {
  type    = string
  default = "estateflow-lab"
}

variable "student_id" {
  type = string
}

VPC, EKS and ECR

terraform/main.tf

data "aws_availability_zones" "available" {
  state = "available"
}

locals {
  name = "${var.student_id}-${var.cluster_name}"

  azs = slice(
    data.aws_availability_zones.available.names,
    0,
    2
  )

  tags = {
    Project     = "EstateFlow"
    Environment = "training"
    ManagedBy   = "Terraform"
    Owner       = var.student_id
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = "${local.name}-vpc"
  cidr = "10.60.0.0/16"

  azs = local.azs

  private_subnets = [
    "10.60.1.0/24",
    "10.60.2.0/24"
  ]

  public_subnets = [
    "10.60.101.0/24",
    "10.60.102.0/24"
  ]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  tags = local.tags
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = local.name
  kubernetes_version = "1.34"

  endpoint_public_access = true

  enable_cluster_creator_admin_permissions = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  addons = {
    coredns = {}
    kube-proxy = {}

    vpc-cni = {
      before_compute = true
      configuration_values = jsonencode({
        enableNetworkPolicy = "true"
      })
    }

    eks-pod-identity-agent = {}

    aws-ebs-csi-driver = {
      most_recent = true
    }
  }

  eks_managed_node_groups = {
    general = {
      instance_types = ["t3.large"]

      min_size     = 2
      max_size     = 4
      desired_size = 2

      labels = {
        workload = "general"
      }
    }
  }

  tags = local.tags
}

resource "aws_ecr_repository" "frontend" {
  name                 = "${local.name}-frontend"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  tags = local.tags
}

resource "aws_ecr_repository" "backend" {
  name                 = "${local.name}-backend"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  tags = local.tags
}

terraform/outputs.tf

output "cluster_name" {
  value = module.eks.cluster_name
}

output "frontend_repository" {
  value = aws_ecr_repository.frontend.repository_url
}

output "backend_repository" {
  value = aws_ecr_repository.backend.repository_url
}

terraform/terraform.tfvars.example

aws_region  = "us-east-1"
cluster_name = "estateflow-lab"
student_id   = "student01"

Before applying: Confirm the requested Kubernetes version is currently supported by EKS and compatible with the selected add-ons, AWS provider, and Terraform module. The example version is a starting target, not a substitute for compatibility validation.

The initial lab uses a single NAT gateway to limit cost. This is not a fully resilient multi-AZ NAT design. For a higher-availability deployment, use appropriate per-AZ egress architecture or VPC endpoints.

Terraform commands

cd terraform

cp terraform.tfvars.example terraform.tfvars

terraform init

terraform fmt -recursive

terraform validate

terraform plan -out=estateflow.tfplan

Instructor approval checkpoint: Review cost, AWS quotas, subnet allocation, cluster version, add-on permissions, node capacity, and Terraform plan before applying.

terraform apply estateflow.tfplan

Then:

aws eks update-kubeconfig \
  --region us-east-1 \
  --name STUDENT01-estateflow-lab

kubectl get nodes -o wide

kubectl get pods -n kube-system

kubectl get daemonsets -n kube-system

Replace the cluster name with the exact Terraform output; it will include the student's identifier.

Infrastructure items that must be completed before application deployment

The Terraform baseline above is not yet the entire AWS platform. Students must also configure:

EBS CSI driver IAM permissions through EKS Pod Identity or a supported IAM integration.
AWS Load Balancer Controller IAM role, Pod Identity association, Helm installation, and webhook readiness.
GitHub Actions OIDC identity with narrowly scoped ECR push permissions.
ECR lifecycle policies and image retention.
Terraform remote state in an S3 backend with locking and encryption.
Restricted EKS API access, rather than leaving unrestricted public endpoint access for production.
DNS, TLS certificates, and ALB listener rules.
Worker-node sizing for monitoring, Argo CD, database, and application workloads.

Do not assume that creating the EKS cluster automatically installs the AWS Load Balancer Controller or grants EBS CSI storage permissions.

6. Phase 3 — Kubernetes workloads

First create the application namespaces:

kubectl create namespace estateflow-staging
kubectl create namespace estateflow-prod

Do not deploy the same StatefulSet database to both environments using shared volumes. Each environment must have its own data and storage.

6.1 PostgreSQL StatefulSet

Save as k8s/storage/postgres.yaml:

apiVersion: v1
kind: Service
metadata:
  name: estateflow-postgres
  namespace: estateflow-staging
spec:
  clusterIP: None
  selector:
    app: estateflow-postgres
  ports:
    - port: 5432
      name: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: estateflow-postgres
  namespace: estateflow-staging
spec:
  serviceName: estateflow-postgres
  replicas: 1
  selector:
    matchLabels:
      app: estateflow-postgres
  template:
    metadata:
      labels:
        app: estateflow-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: estateflow
            - name: POSTGRES_USER
              value: estateflow
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: estateflow-db
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U estateflow -d estateflow
            periodSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 512Mi
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: gp3
        resources:
          requests:
            storage: 10Gi

Create a database password without storing it in Git:

read -s "DB_PASSWORD?Database password: "
echo

kubectl create secret generic estateflow-db \
  -n estateflow-staging \
  --from-literal=password="$DB_PASSWORD"

unset DB_PASSWORD

For a classroom deployment, this is acceptable as an introductory Secret exercise. In a production design, use an external secret-management solution, encryption controls, and rotation. Do not commit Secret manifests containing plaintext passwords.

Before applying the StatefulSet, verify that the gp3 StorageClass exists and that the EBS CSI driver is authorized:

kubectl get storageclass

kubectl get csidrivers

kubectl get pods -n kube-system | grep ebs

If gp3 is absent, create a CSI-backed StorageClass with ebs.csi.aws.com, appropriate binding mode, and encryption. Do not proceed until the storage controller is operational.

kubectl apply -f k8s/storage/postgres.yaml

kubectl get statefulsets -n estateflow-staging

kubectl get pvc -n estateflow-staging

kubectl get pods -n estateflow-staging -o wide

Instructor explanation: StatefulSet gives PostgreSQL a stable identity and persistent volume association. It does not automatically provide database replication, backups, or high availability. Our single-replica PostgreSQL is deliberately a learning environment.

6.2 Redis StatefulSet

Create a separate Redis StatefulSet with one replica, its own headless Service, persistent volume, and resource limits. Redis must not share the PostgreSQL PVC.

Students must explain why Redis is not the source of truth for booking records.

6.3 Backend Deployment and Service

Save as k8s/backend.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: estateflow-api
  namespace: estateflow-staging
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: estateflow-api
  template:
    metadata:
      labels:
        app: estateflow-api
    spec:
      containers:
        - name: api
          image: REPLACE_WITH_BACKEND_ECR_IMAGE
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: DB_HOST
              value: estateflow-postgres
            - name: DB_NAME
              value: estateflow
            - name: DB_USER
              value: estateflow
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: estateflow-db
                  key: password
            - name: REDIS_URL
              value: redis://estateflow-redis:6379
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
          startupProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 5
            failureThreshold: 12
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
---
apiVersion: v1
kind: Service
metadata:
  name: estateflow-api
  namespace: estateflow-staging
spec:
  type: ClusterIP
  selector:
    app: estateflow-api
  ports:
    - name: http
      port: 8080
      targetPort: http

Replace the image placeholder with the actual immutable ECR image URI before applying. The final GitOps deployment will use Helm rather than manually editing this file.

6.4 Frontend Deployment

The frontend needs a separate Deployment and ClusterIP Service.

Use three initial replicas for the classroom deployment only if node capacity allows. Configure:

Port 8080
/healthz readiness and liveness checks
Resource requests and limits
RollingUpdate
Pod anti-affinity or topology spread
A Service named estateflow-frontend

The Kubernetes NGINX configuration must proxy /api/ to http://estateflow-api:8080, not to the Docker Compose hostname backend.

6.5 HPA

Save as k8s/autoscaling/api-hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: estateflow-api-hpa
  namespace: estateflow-staging
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: estateflow-api
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Prerequisites:

kubectl top nodes
kubectl top pods -n estateflow-staging

If metrics are unavailable, fix Metrics Server before expecting CPU-based HPA scaling.

6.6 PodDisruptionBudget

k8s/autoscaling/api-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: estateflow-api-pdb
  namespace: estateflow-staging
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: estateflow-api

Explain that a PDB limits certain voluntary disruptions, such as node drains. It does not prevent all crashes, node failures, or application bugs.

6.7 Scheduling — affinity, taints and tolerations

Add this to a workload Pod specification only when the cluster has eligible nodes:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: estateflow-api
          topologyKey: kubernetes.io/hostname

For AZ distribution:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: estateflow-api

For the taints lab, create a separate dedicated node group before applying any taint.

Example:

kubectl taint node NODE_NAME \
  dedicated=estateflow:NoSchedule

Matching Pod toleration:

tolerations:
  - key: dedicated
    operator: Equal
    value: estateflow
    effect: NoSchedule

A toleration allows scheduling onto a tainted node; it does not force scheduling there. To target that node group, use appropriate node affinity or a node selector as well.

Never taint the only available worker node in a shared classroom cluster. That could disrupt CoreDNS, monitoring, and other workloads.

6.8 NetworkPolicy

The application requires a deliberate communication model:

Frontend

Backend API

PostgreSQL

Frontend

Direct PostgreSQL access

NetworkPolicy controls permitted network traffic. It does not delete Pod IP addresses or remove Pods.

k8s/security/postgres-networkpolicy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-api-only
  namespace: estateflow-staging
spec:
  podSelector:
    matchLabels:
      app: estateflow-postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: estateflow-api
      ports:
        - protocol: TCP
          port: 5432

This isolates PostgreSQL ingress and permits connections from matching API Pods in the same namespace. Other applicable policies can add permissions, so inspect all NetworkPolicies before claiming isolation.

On AWS EKS, NetworkPolicy enforcement requires a compatible network-policy-capable implementation. With AWS VPC CNI, verify that NetworkPolicy support is enabled and the policy agent is running.

kubectl get daemonset aws-node -n kube-system

kubectl get networkpolicy -n estateflow-staging

kubectl describe networkpolicy \
  postgres-allow-api-only \
  -n estateflow-staging

6.9 RBAC

k8s/rbac/read-only.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: estateflow-observer
  namespace: estateflow-staging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: estateflow-read-only
  namespace: estateflow-staging
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: estateflow-observer-binding
  namespace: estateflow-staging
subjects:
  - kind: ServiceAccount
    name: estateflow-observer
    namespace: estateflow-staging
roleRef:
  kind: Role
  name: estateflow-read-only
  apiGroup: rbac.authorization.k8s.io

Verify:

kubectl auth can-i list pods \
  --as=system:serviceaccount:estateflow-staging:estateflow-observer \
  -n estateflow-staging

kubectl auth can-i delete pods \
  --as=system:serviceaccount:estateflow-staging:estateflow-observer \
  -n estateflow-staging

Expected: listing allowed, deletion denied.

6.10 IAM and Pod Identity

Students must distinguish:

Kubernetes RBAC controls what an identity can do through the Kubernetes API.

AWS IAM controls access to AWS APIs and resources.

EKS Pod Identity allows an authorized Kubernetes ServiceAccount to obtain AWS credentials through an IAM role association.

For the lab, create a dedicated ServiceAccount for a workload that reads a narrowly scoped S3 object or CloudWatch resource. Associate it with a least-privilege IAM role using Terraform and EKS Pod Identity. Do not grant application Pods administrator permissions.

6.11 Ingress and AWS ALB

After the AWS Load Balancer Controller is installed and healthy, create:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: estateflow-ingress
  namespace: estateflow-staging
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: estateflow-api
                port:
                  number: 8080
          - path: /
            pathType: Prefix
            backend:
              service:
                name: estateflow-frontend
                port:
                  number: 8080

The ALB controller reconciles this Ingress into AWS load-balancer resources. The ALB routes requests to the selected backend target group. The Kubernetes Service defines the backend Pods; the exact packet forwarding depends on the chosen target mode and networking implementation.

Before calling this production-ready, add HTTPS, an ACM certificate, Route 53 DNS, access logs, health-check configuration, and appropriate security groups.

7. Phase 4 — GitHub Actions CI

The delivery sequence will be:

Developer opens Pull Request
          |
          v
    GitHub Actions
          |
          +-- npm install / test / build
          |
          +-- SonarQube quality analysis
          |
          +-- Trivy vulnerability scan
          |
          v
     Merge approved
          |
          v
    Build Docker images
          |
          v
    Push to Amazon ECR
          |
          v
   Update GitOps image tags
          |
          v
      Argo CD sync
          |
          v
     Kubernetes rollout

GitHub permissions

Create an AWS IAM role that trusts GitHub's OIDC provider. Restrict its trust policy to the intended repository, branch or GitHub environment, and audience sts.amazonaws.com.

Grant only the ECR permissions required to authenticate and push to the two EstateFlow repositories.

Do not put AWS access keys in GitHub Secrets. Use OIDC federation.

CI workflow

Save as .github/workflows/ci.yml:

name: EstateFlow CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  backend:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: backend
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
          cache-dependency-path: backend/package-lock.json

      - run: npm ci
      - run: npm test

  frontend:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
          cache-dependency-path: frontend/package-lock.json

      - run: npm ci
      - run: npm run build

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Trivy filesystem scan
        uses: aquasecurity/trivy-action@0.33.1
        with:
          scan-type: fs
          scan-ref: .
          severity: HIGH,CRITICAL
          exit-code: "1"

The backend currently has no committed test suite. Before making npm test a required branch-protection check, students must add tests for /healthz, input validation, and booking/database error handling.

For a production pipeline, pin GitHub Actions to reviewed commit SHAs and establish dependency-update procedures.

SonarQube

Create sonar-project.properties:

sonar.projectKey=estateflow
sonar.projectName=EstateFlow
sonar.sources=frontend/src,backend/src
sonar.sourceEncoding=UTF-8

Configure a SonarQube server, token, and quality gate. Add a dedicated CI job that runs SonarScanner and waits for the quality-gate result.

Students must understand that SonarQube performs static code analysis; it does not replace unit tests, dependency scanning, image scanning, or runtime monitoring.

CD — ECR push workflow

After CI passes on main, a protected publishing workflow must:

Assume the AWS IAM role through GitHub OIDC.
Authenticate to ECR.
Build both Docker images.
Scan the built images.
Push immutable image tags derived from the Git commit SHA.
Record the image digests.
Open a pull request updating the GitOps repository.

Example build commands:

docker build \
  -t "$FRONTEND_REPO:$GITHUB_SHA" \
  ./frontend

docker build \
  -t "$BACKEND_REPO:$GITHUB_SHA" \
  ./backend

docker push "$FRONTEND_REPO:$GITHUB_SHA"

docker push "$BACKEND_REPO:$GITHUB_SHA"

These variables must be populated by the authenticated CI job. The image repositories must already exist.

Important: Argo CD should not be given AWS credentials to build or push images. Its responsibility is reconciling the approved Kubernetes configuration from Git.

8. Phase 5 — Helm

Helm packages the application's Kubernetes resources into reusable, parameterized charts.

Create:

helm create helm/estateflow

Remove or replace the generated example templates that do not match our application.

helm/estateflow/Chart.yaml:

apiVersion: v2
name: estateflow
description: EstateFlow Kubernetes application
type: application
version: 0.1.0
appVersion: "1.0.0"

Example helm/estateflow/values.yaml:

frontend:
  replicas: 2
  image:
    repository: REPLACE_FRONTEND_ECR_URI
    tag: REPLACE_WITH_GIT_SHA

backend:
  replicas: 2
  image:
    repository: REPLACE_BACKEND_ECR_URI
    tag: REPLACE_WITH_GIT_SHA

database:
  host: estateflow-postgres
  name: estateflow
  user: estateflow

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 512Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 6
  targetCPUUtilizationPercentage: 60

ingress:
  enabled: true
  className: alb

Students will convert the backend Deployment, frontend Deployment, Services, Ingress, HPA, PDB, and application configuration into Helm templates.

Validate before deployment:

helm lint helm/estateflow

helm template estateflow \
  helm/estateflow \
  --namespace estateflow-staging \
  > /tmp/estateflow-rendered.yaml

kubectl apply \
  --dry-run=client \
  -f /tmp/estateflow-rendered.yaml

Do not use a Helm chart containing unreviewed duplicate resources already managed by Terraform or another controller.

9. Phase 6 — Argo CD GitOps

Install Argo CD into its own namespace after checking cluster capacity and the supported installation version.

Create an Argo CD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: estateflow-staging
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_ORG/jumptotech-estateflow-gitops.git
    targetRevision: main
    path: charts/estateflow
    helm:
      valueFiles:
        - ../../environments/staging/values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: estateflow-staging
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The chart path and values-file path must be verified against the actual GitOps repository layout. A common simplification is to keep environment values inside the chart directory, avoiding cross-directory Helm value-file restrictions.

For production-style operation, create an Argo CD AppProject with explicit source-repository, namespace, and resource restrictions instead of relying on the unrestricted default project.

Students must verify:

kubectl get applications -n argocd

kubectl get deployments -n estateflow-staging

kubectl get pods -n estateflow-staging

kubectl rollout status deployment/estateflow-api \
  -n estateflow-staging

Argo CD incident demonstration

Change the approved image tag in the GitOps repository to a nonexistent tag.

Expected possible result: new Pods enter ImagePullBackOff, the rollout stalls, and Argo CD reports an unhealthy application.

Restore the previous known-good image tag in Git, synchronize Argo CD, and verify the rollout.

For GitOps-managed applications, the durable correction must be committed to Git. A manual kubectl rollout undo may be overwritten by Argo CD reconciliation.

10. Phase 7 — Monitoring and alerting

Install a compatible Prometheus Operator stack, including Prometheus, Grafana, and Alertmanager. If the existing classroom cluster already has monitoring, do not install a duplicate stack without checking capacity and ownership.

Application metrics

Our backend exposes:

/metrics

Prometheus can scrape the backend using a ServiceMonitor when the operator and ServiceMonitor CRD are installed.

Example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: estateflow-api
  namespace: estateflow-staging
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - estateflow-staging
  selector:
    matchLabels:
      app: estateflow-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

The Service must expose a port named http and have the matching label. The Prometheus instance must also be configured to select this ServiceMonitor.

Grafana dashboards

Students will create dashboards showing:

Metric	PromQL example
Available API replicas	`kube_deployment_status_replicas_available{namespace="estateflow-staging",deployment="estateflow-api"}`
Desired replicas	`kube_deployment_spec_replicas{namespace="estateflow-staging",deployment="estateflow-api"}`
Pod restarts	`sum(increase(kube_pod_container_status_restarts_total{namespace="estateflow-staging"}[5m]))`
HTTP 503 rate	`sum(rate(estateflow_http_requests_total{status="503"}[5m]))`
API request rate	`sum(rate(estateflow_http_requests_total[5m]))`

For the application HTTP metrics, add the Prometheus scrape labels actually present in your deployment to avoid mixing staging and production.

Alerts

Create alerts for:

API HTTP 503 errors above a defined threshold.
Available replicas below desired replicas.
Persistent CrashLoopBackOff.
Excessive Pod restarts.
High CPU utilization.
PostgreSQL connectivity failures.
PVC capacity or availability issues.

A real SRE team should tune thresholds using observed traffic, error budgets, and service-level objectives. A classroom alert can be deliberately sensitive to make the incident visible.

PagerDuty

Connect a dedicated training PagerDuty service through Alertmanager or Grafana.

Students should experience the full sequence:

Prometheus detects condition
        ↓
Alertmanager / Grafana Alerting
        ↓
PagerDuty incident
        ↓
Primary on-call acknowledges
        ↓
Incident investigation
        ↓
Recovery verification
        ↓
Incident resolution
        ↓
Postmortem

Do not send classroom alerts to a real production on-call service.

11. Phase 8 — Main production incident: HTTP 503

This is the incident every student will investigate and present in their capstone.

SEV-1 training incident

Service degraded

EstateFlow booking requests are failing

Incident start

2:13 AM — scenario

Customer-facing symptom

HTTP 503

Kubernetes

Pods Running / Ready

Suspected dependency

PostgreSQL

The website is reachable, but booking requests cannot complete. The on-call engineer must identify the actual failure before attempting a rollback.

11.1 Incident team

Role	Responsibility
Primary SRE	Acknowledge PagerDuty, investigate, coordinate recovery
Incident Commander	Set priorities, coordinate responders, maintain timeline
Backend Engineer	Review API behavior, exceptions, connection pools
DBA	Verify PostgreSQL health, connectivity, and database state
Cloud/Platform Engineer	Investigate EKS, CNI, nodes, networking, IAM
Security Engineer	Review NetworkPolicies and security implications
Release Engineer	Review deployments, Argo CD history, recent changes
Communications Lead	Provide stakeholder updates

For a small organization, one person may hold multiple roles. Students should learn the responsibilities, not assume every incident has eight people online.

11.2 Confirm healthy baseline

Before injecting the failure:

kubectl get pods -n estateflow-staging

kubectl get deployments -n estateflow-staging

kubectl get svc -n estateflow-staging

kubectl get endpointslices -n estateflow-staging

Start a local port-forward:

kubectl port-forward \
  -n estateflow-staging \
  svc/estateflow-api \
  18080:8080

In another Terminal:

curl -i http://127.0.0.1:18080/healthz

curl -i http://127.0.0.1:18080/readyz

curl -i http://127.0.0.1:18080/api/db-check

curl -i http://127.0.0.1:18080/api/properties

Expected baseline: HTTP 200 on all four endpoints.

Record the actual timestamp, screenshots, Pod names, Deployment revisions, and API responses.

11.3 Introduce the controlled outage

Only in estateflow-staging, apply a NetworkPolicy that selects PostgreSQL and denies new ingress connections:

incidents/01-block-postgres.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: incident-block-postgres
  namespace: estateflow-staging
spec:
  podSelector:
    matchLabels:
      app: estateflow-postgres
  policyTypes:
    - Ingress
  ingress: []

Important: If postgres-allow-api-only is already present, this deny policy will not override it. Kubernetes NetworkPolicy allow rules are additive. For this controlled test, first inspect the existing policies and temporarily remove the allow policy through the approved staging change process.

kubectl get networkpolicy \
  -n estateflow-staging

kubectl describe networkpolicy \
  postgres-allow-api-only \
  -n estateflow-staging

Then, after recording the original state:

kubectl delete networkpolicy \
  postgres-allow-api-only \
  -n estateflow-staging

kubectl apply \
  -f incidents/01-block-postgres.yaml

If Argo CD manages the policy, pause or coordinate reconciliation for this controlled experiment and restore the intended policy through GitOps afterward.

Existing established database connections may survive policy changes temporarily. To test the failure reliably, force a new database connection from the application in the staging environment. Do not restart a production database or delete unrelated Pods.

11.4 Investigate the outage

First check the API:

curl -i http://127.0.0.1:18080/readyz

curl -i http://127.0.0.1:18080/api/db-check

Expected result after enforcement and new connection attempts:

GET /readyz
HTTP/1.1 200 OK

GET /api/db-check
HTTP/1.1 503 SERVICE UNAVAILABLE

{"error":"database unavailable"}

Now investigate Kubernetes:

kubectl get pods \
  -n estateflow-staging \
  -o wide

kubectl describe deployment estateflow-api \
  -n estateflow-staging

kubectl logs \
  -n estateflow-staging \
  deployment/estateflow-api \
  --all-pods=true \
  --tail=100

kubectl get networkpolicy \
  -n estateflow-staging

kubectl describe networkpolicy \
  incident-block-postgres \
  -n estateflow-staging

kubectl get events \
  -n estateflow-staging \
  --sort-by=.lastTimestamp

Students must identify the difference between a healthy Pod and a healthy business transaction.

A Pod can be Running and Ready while its application cannot connect to PostgreSQL.

11.5 The incident investigation decision tree

If the database is running and healthy, investigate connectivity and policy enforcement. If PostgreSQL itself is unhealthy, investigate storage, resource pressure, process logs, and database recovery instead.

11.6 Communicate during the incident

Students should practice realistic, concise communication.

Initial PagerDuty acknowledgement:

Acknowledged the EstateFlow booking failure alert. Investigating the HTTP 503 increase. Initial Kubernetes checks show the API Pods are Running and Ready. I am checking the database dependency and recent changes.

Message to the backend engineer:

Can you confirm whether the booking API is failing on database connection attempts? Please review recent application errors and connection-pool behavior.

Message to the platform/security engineer:

PostgreSQL appears healthy, but new API-to-database connections are failing. Please help verify NetworkPolicy enforcement, AWS VPC CNI policy-agent health, and any recent networking changes.

Incident Commander update:

We have isolated the failure to application connectivity with PostgreSQL. We are reviewing the active ingress policy and preparing a controlled restoration. Customer booking requests remain affected.

Only communicate findings that the team has actually verified.

11.7 Restore service

Remove the incident policy:

kubectl delete networkpolicy \
  incident-block-postgres \
  -n estateflow-staging

Restore the intended least-privilege allow policy:

kubectl apply \
  -f k8s/security/postgres-networkpolicy.yaml

If Argo CD owns the policy, restore the approved GitOps manifest and verify synchronization.

Test:

curl -i http://127.0.0.1:18080/api/db-check

curl -i -X POST \
  http://127.0.0.1:18080/api/bookings \
  -H "Content-Type: application/json" \
  -d '{"property_id":1,"customer_name":"Recovery Test"}'

Expected after recovery:

Database check: HTTP 200
Booking creation: HTTP 201

Also verify the newly created booking exists in PostgreSQL, application error rates decline, and alerts return to normal.

Do not close the incident based only on Pods showing 1/1 Running.

11.8 Postmortem template

EstateFlow Incident Postmortem

Incident ID: EF-INC-001\
Severity: SEV-1 (training scenario)\
Environment: estateflow-staging\
Incident Commander: [Student / instructor]\
Primary SRE: [Student]

Summary

The EstateFlow booking API experienced HTTP 503 errors because application-to-PostgreSQL connectivity was blocked by a Kubernetes NetworkPolicy.

Customer impact

[Record which synthetic transactions failed, the observed error rate, and the duration measured during the lab.]

Detection

The incident was detected through [Grafana alert / PagerDuty / synthetic monitoring]. Record the actual alert timestamp.

Timeline

Time	Event
[Time]	Healthy baseline verified
[Time]	NetworkPolicy change applied
[Time]	First HTTP 503 observed
[Time]	Alert acknowledged
[Time]	Application logs investigated
[Time]	Root cause confirmed
[Time]	Policy corrected
[Time]	HTTP 200 and booking HTTP 201 verified
[Time]	Incident resolved

Root cause

A NetworkPolicy selected the PostgreSQL Pod and blocked new ingress connections from the EstateFlow API.

The API's readiness endpoint did not depend on PostgreSQL, so Kubernetes continued reporting the application Pods as Ready even though database-dependent business requests failed.

Mitigation

The incident policy was removed and the intended least-privilege PostgreSQL access rule was restored.

Verification

Database check returned HTTP 200.
Synthetic booking returned HTTP 201.
Booking record was verified in PostgreSQL.
Error-rate metrics recovered.
Monitoring alerts resolved.

Preventive actions

Add NetworkPolicy connectivity tests before deployment.
Require review of security-policy changes.
Add synthetic booking monitoring.
Improve runbooks for HTTP 503 investigation.
Add GitOps validation and staged rollout controls.

Evidence

Attach Terminal outputs, Grafana screenshots, PagerDuty event details, Git commit history, and recovery timestamps.

12. Phase 9 — Additional production-style incident labs

After students complete the main outage, they must investigate the following scenarios. Every scenario must have its own baseline, failure injection, diagnosis, recovery, and evidence.

Incident	Injected failure	Primary investigation	Expected recovery
01 — HTTP 503	PostgreSQL NetworkPolicy block	Logs, connectivity, policies	Restore permitted DB traffic
02 — ImagePullBackOff	Invalid ECR image tag	`describe pod`, image URI, ECR	Correct GitOps image reference
03 — CrashLoopBackOff	Application exits on startup	Pod logs, env, events	Correct configuration/code
04 — Readiness failure	Invalid `/readyz` probe path	`describe pod`, probe output	Correct probe
05 — HPA overload	CPU pressure and insufficient capacity	`kubectl top`, HPA events	Tune capacity and scaling
06 — PVC Pending	Unsupported StorageClass	PVC events, CSI controller	Restore correct storage configuration
07 — Node NotReady	Dedicated training node failure	Node conditions, kubelet, events	Recover/replace node
08 — RBAC Forbidden	Observer attempts unauthorized action	`kubectl auth can-i`	Grant only justified permissions
09 — IAM AccessDenied	Missing AWS role permission	AWS API error, Pod Identity	Correct least-privilege IAM policy
10 — Scheduling Pending	Taint without toleration	Scheduler events	Correct scheduling constraints
11 — Affinity failure	Required placement unavailable	Node labels, affinity, events	Restore eligible placement
12 — Argo CD OutOfSync	Manual resource change	Argo CD diff and history	Reconcile from Git
13 — ALB 503	No healthy application targets	Target health, Ingress, endpoints	Restore healthy targets
14 — DNS failure	DNS resolution problem	CoreDNS, Service DNS	Correct DNS issue
15 — Database recovery	Controlled database restart	StatefulSet, PVC, logs	Restore database and validate data
16 — PDB and node drain	Drain constrained by PDB	Eviction events, PDB	Safely adjust capacity
17 — RollingUpdate	New release fails readiness	ReplicaSets, rollout status	Revert GitOps release
18 — Secret failure	Missing Secret key	Pod events, environment refs	Restore Secret
19 — Resource limits	Memory exhaustion	OOMKilled, memory metrics	Correct memory sizing/leak
20 — CronJob failure	Backup job fails	Job logs, events, permissions	Repair and rerun backup

For the CronJob exercise, students must verify that the backup is recoverable by restoring into a separate database. A successful backup command alone does not prove recoverability.

13. Phase 10 — Kubernetes controllers: what to demonstrate

Students should understand the controller's responsibility and why the workload needs it.

Controller / resource	EstateFlow use
Deployment	Frontend and API stateless workloads
ReplicaSet	Maintains desired Pods through Deployment
StatefulSet	PostgreSQL and Redis lab workloads
DaemonSet	VPC CNI, node monitoring, log agents
Job	One-time database migration or verification
CronJob	Scheduled backup and maintenance
HPA	Scale API replicas using resource metrics
Ingress controller	Reconcile ALB routing
EBS CSI controller	Provision and attach persistent volumes
Argo CD controller	Reconcile desired state from Git
Node controller	Detect node health changes
EndpointSlice controller	Maintain Service endpoint information
Scheduler	Select eligible nodes for unscheduled Pods

Also demonstrate ServiceAccounts, Roles, RoleBindings, ClusterRoles, ClusterRoleBindings, ResourceQuotas, LimitRanges, PriorityClasses, Pod Security Admission, PodDisruptionBudgets, topology spread, and storage reclamation policies.

Not every resource should be used in the same application workload. For example, a ClusterRole should not replace a namespaced Role when only namespace-scoped permissions are required.

14. Phase 11 — Production-readiness requirements

Before students present EstateFlow as a production-style platform, complete the following checklist.

Project readiness

0 / 10 completed

Application

Automated backend tests, frontend build, input validation, dependency checks

Container security

Non-root runtime, image scanning, immutable tags, reviewed base images

AWS infrastructure

Terraform remote state, IAM least privilege, restricted cluster access

Networking

ALB controller, HTTPS, DNS, security groups, verified NetworkPolicy

Kubernetes

Requests/limits, probes, HPA, PDB, placement, safe rolling updates

Storage

EBS CSI IAM, encryption, backup, restore test, storage retention

GitOps

Separate GitOps repository, environment values, Argo CD project restrictions

Monitoring

Metrics, dashboards, alert routing, synthetic business transactions

Incident response

On-call roles, escalation, rollback, recovery validation, postmortem

Cost and cleanup

Budget alerts, resource inventory, cleanup approval and verification

This checklist is a teaching gate, not a certification that the system meets a particular organization's production standards.

15. Phase 12 — Interview preparation

Students should learn to describe the incident using the STAR method: Situation, Task, Action, Result.

DevOps/SRE Interview — EstateFlow Incident

Question: Tell me about a Kubernetes production incident you investigated.

During my DevOps/SRE capstone, I worked on EstateFlow, a production-style real-estate application deployed on AWS EKS.

The platform used React for the frontend, Node.js for the backend, PostgreSQL for persistent data, and Redis for caching. We used Terraform for AWS infrastructure, GitHub Actions for CI, Amazon ECR for container images, and Helm with Argo CD for GitOps deployments.

One of the incidents I investigated involved HTTP 503 errors on the booking API.

The interesting part was that Kubernetes reported the application Pods as Running and Ready, but customers in our synthetic test could not complete booking requests.

I first checked the Deployment, Pods, Services, and EndpointSlices using kubectl. Then I reviewed the application logs and tested the health and database endpoints separately.

The readiness endpoint returned HTTP 200, but the database check returned HTTP 503.

That told me the application process was running, but a downstream dependency was unavailable.

I checked PostgreSQL health and the active Kubernetes NetworkPolicies. We found that a policy was blocking new ingress connections to the PostgreSQL Pod.

I coordinated the investigation using the roles of on-call SRE, backend engineer, platform engineer, and security engineer.

We restored the intended least-privilege NetworkPolicy and tested the application again.

After the fix, the database endpoint returned HTTP 200 and the booking endpoint successfully created a booking with HTTP 201.

We also checked the monitoring dashboards and documented the incident timeline, root cause, and preventive actions.

The main lesson was that healthy Kubernetes Pods do not necessarily mean healthy business transactions. Application-level monitoring and dependency checks are essential.

If asked whether this occurred at a real employer: This was a hands-on production-style lab incident that I reproduced, investigated, and recovered in AWS EKS. It was not a customer outage at a bank.

Follow-up questions students must answer

Why were the Pods Running while the API returned HTTP 503?
What is the difference between a readiness probe and a liveness probe?
How does a NetworkPolicy select Pods?
How does AWS VPC CNI enforce network isolation?
Why would rolling back the API Deployment not necessarily fix a NetworkPolicy issue?
How does an ALB find healthy Kubernetes targets?
What is the difference between IAM and Kubernetes RBAC?
Why use a StatefulSet for PostgreSQL?
What happens if a PVC remains Pending?
How does Argo CD detect configuration drift?
What is the difference between maxSurge and maxUnavailable?
Why does an HPA need Metrics Server for CPU-based scaling?
How do taints and tolerations differ from node affinity?
What does a PodDisruptionBudget protect against?
How do you prove a business service has recovered?

Every student should answer with a command they used, an output they observed, and an explanation of what that output meant.

16. Phase 13 — Final project demonstration

Each student must demonstrate:

Application: Open EstateFlow, list properties, and create a booking.
AWS: Show Terraform configuration, EKS nodes, and ECR repositories.
CI/CD: Show a GitHub Actions run, image publishing, and Argo CD deployment.
Kubernetes: Explain Deployments, StatefulSets, Services, Ingress, HPA, PDB, and scheduling.
Security: Demonstrate RBAC authorization and PostgreSQL NetworkPolicy isolation.
Monitoring: Show Grafana metrics and an alert.
Incident: Reproduce HTTP 503, identify the cause, restore connectivity, and verify a booking.
Documentation: Submit a postmortem with actual timestamps and screenshots.

17. Phase 14 — Safe cleanup

Cleanup is a separate, instructor-approved activity. It must never be an unrestricted kubectl delete --all or terraform destroy against an unverified environment.

Before cleanup, students must:

Export their incident evidence and final reports.
Verify the active AWS account and Kubernetes context.
Confirm that all resources belong to their dedicated EstateFlow project.
Back up any data they need to retain.
Identify ALBs, EBS volumes, ECR images, and other resources that might remain billable.

Then remove the EstateFlow GitOps Applications through their approved process, allow AWS controllers to clean up managed load balancers, and verify no resources are stranded.

Only after confirming Terraform manages the intended dedicated infrastructure should the student run:

cd terraform

terraform plan -destroy

Review the entire destroy plan with the instructor.

For a dedicated, disposable EstateFlow cluster only:

terraform destroy

Never run this against infrastructure that hosts payments-incident-lab or another student's resources.

18. Instructor implementation notes — important gaps to finish

The material above is a complete curriculum and implementation blueprint, with runnable examples for the core application and main incident. It is not yet a fully assembled, tested, one-command deployment repository. I want to distinguish that clearly before students copy it into their environments.

These remaining implementation tasks require actual files and validation:

Component	What still needs to be completed
Frontend	Kubernetes-specific NGINX configuration and final Helm template
Redis	StatefulSet, Service, PVC, authentication and configuration
Database	Schema migration Job, backup CronJob, restore procedure
Terraform	EBS CSI IAM, ALB Controller IAM, Pod Identity, GitHub OIDC, remote state
AWS ALB	Controller installation, HTTPS, DNS, security groups
GitHub Actions	Complete ECR publishing workflow, SonarQube job, GitOps PR automation
Helm	Full rendered templates for every application resource
Argo CD	Tested repository paths, AppProject, environment promotion
Security	Pod Security Admission, external secrets, complete policy tests
Monitoring	Tested ServiceMonitor selection, alert rules, PagerDuty routing
Incidents	Executable injection and recovery scripts for all 20 scenarios
Validation	Integration tests, failure recovery, capacity and cost checks

Those gaps matter. Students should not be told that copying this document alone creates a complete production environment.

Recommended classroom sequence

Week 1

Application, PostgreSQL, Redis, Docker Compose

Week 2

Terraform, VPC, EKS, IAM, ECR

Week 3

Kubernetes Deployments, StatefulSets, storage, Services

Week 4

ALB, Ingress, DNS, RBAC, NetworkPolicy, scheduling

Week 5

GitHub Actions, SonarQube, Trivy, Helm, Argo CD

Week 6

Prometheus, Grafana, PagerDuty, incident labs

Week 7

Incident simulation, postmortems, interview presentations

My recommendation for JumpToTech: Use this as the master student syllabus and architecture guide, but keep implementation in a real GitHub repository with tested source files. Students should build and verify each phase before moving on, rather than paste a large unvalidated script into AWS.

The final standard is straightforward: every student must be able to show the working application, demonstrate the failure, explain why it happened, restore service, and prove recovery using real evidence.

interview real time questions and answers

Aisalkyn Aidarova — Thu, 05 Nov 2026 19:04:23 +0000

Tell me about yourself.
I am DevOps Engineer with more than 5 years experience. I started at Southwest Airlines as Cloud Engineer, and after one year I got promoted to DevOps Engineer. Since September 2023 I work at Capital One. Mostly I work with AWS, Kubernetes, Terraform and GitLab CI/CD. I also used Azure, and monitoring tools like Prometheus, Grafana and Datadog. I support production, do on-call, and work on reliability, security and cost. I like to automate manual work and help developers ship apps safely.
Why BigTime?
You are building AI-powered platform, and I think AI brings many good opportunities for DevOps. Your stack is AWS, Kubernetes, Terraform and GitLab, and this is what I work with every day. I also like that engineers get ownership and solve problems independently.
What is your day to day?
In the morning I check alerts and dashboards in Datadog and Grafana. Then standup. Most of the day I work on Terraform, GitLab pipelines and deployments to EKS with Helm. I help developers when deployments fail. I do patching, upgrades, cost and security work. When I am on-call, I handle incidents. I update documentation in Confluence.
Tell me about a hard production incident.
After a release we had 60 replicas in 3 Availability Zones, and HTTP 500 errors went up to 18 percent. I was the on-call engineer. I checked dashboards, logs, Kubernetes deployment status and the deployment history. The errors started with the release, so we rolled back to the last stable version. I checked that pods were healthy and the error rate went down. Root cause was wrong database connection setting in the new version. After this I added canary deployments, smoke tests in the pipeline, automatic rollback on high error rate, and wrote a postmortem and runbook.
How do you replace long-lived credentials?
I move secrets to Vault or AWS Secrets Manager with automatic rotation. Apps on EKS use IAM roles for service accounts, so pods get short-lived credentials. GitLab pipelines use OIDC to assume IAM roles, no static AWS keys. EC2 and Windows servers use instance roles. For RDS, passwords are in Secrets Manager with rotation. This helps with SOC 2 audits too.
Windows EC2 in production, you cannot RDP, and you cannot restart. What do you do?
First I check if the app still works for users, maybe only RDP is broken. I check status checks, security group port 3389, NACL, route table, and the Route 53 record. Then I use SSM Session Manager or Run Command with PowerShell. I check if the RDP service is running and restart only this service, check Windows firewall, CPU, memory and disk space. I look at the console screenshot, system log and CloudWatch metrics. If restart is really needed, I take a snapshot, move traffic to other instances behind the load balancer, talk to the team, and do it in a maintenance window.
How did you save $160K?
I added cost allocation tags so we see who spends what. I rightsized big EC2 instances, turned off dev and test at night and weekends, cleaned unused EBS volumes, snapshots and load balancers. S3 lifecycle policies to cheaper storage. Better EKS node sizing and autoscaling. Savings Plans and Reserved Instances for stable workloads. Budgets and alerts in Cost Explorer and CloudWatch.
How did you reduce MTTD by 25 percent?
I made better dashboards in Grafana and Datadog for each service. Alerts based on real symptoms like error rate, latency and pod restarts. I removed noisy alerts so the team trusts them. I added health checks and synthetic monitoring. Alerts go to PagerDuty and Slack with runbook links.
Biggest issue you fixed with Grafana?
Booking service on EKS was slow every few days. I made a dashboard with memory, restarts and latency together. Memory went up slowly, then pods got OOMKilled and latency jumped. It was a memory leak and low memory limits. Developers fixed the leak, I fixed requests and limits and added alerts for memory growth and restarts.
What are the RDS replication types?
Multi-AZ is synchronous replication to a standby in another Availability Zone, for high availability and automatic failover, you cannot read from it. Read Replicas are asynchronous, for scaling reads and reporting, can be cross-region, and can be promoted for disaster recovery. SQL Server Multi-AZ uses Always On or mirroring.
RDS vs Aurora?
Standard RDS is managed database with its own EBS storage per instance. Aurora is AWS cloud database for MySQL and PostgreSQL. Storage is shared and separate from compute, data copied 6 times across 3 Availability Zones, storage grows automatically. Up to 15 replicas with very low lag, failover under 30 seconds. Aurora is faster but more expensive. Aurora does not support SQL Server.
GitLab pipeline with two different AWS roles, how do you configure?
I use GitLab OIDC. The job gets an ID token and assumes IAM role. Role one is read-only for terraform plan, runs on merge requests. Role two has write permissions for apply and deploy, and the trust policy allows only main branch, protected branch and production environment. Stages are validate, plan, apply, deploy. Apply is manual approval with the saved plan. Deploy updates kubeconfig and runs helm upgrade. In EKS I map the role to RBAC with access entries, so it deploys only to its namespaces.
What happens when you type bigtime.net in browser?
DNS finds the IP with UDP port 53, through the resolver to Route 53. Then TCP three-way handshake, SYN, SYN-ACK, ACK. Then TLS handshake on port 443, server sends certificate and they agree on keys. Then HTTP GET request. Traffic goes through CloudFront or load balancer, then to EC2 or EKS, and the app talks to RDS. Response comes back and browser shows the page.
How do you make sure the app uses S3 securely, encryption in transit?
Bucket policy denies requests when aws:SecureTransport is false, so only HTTPS. I can require TLS 1.2 with s3:TlsVersion. App uses IAM role, least privilege, no keys. VPC gateway endpoint so traffic stays private, and bucket policy limited to this endpoint. Block Public Access on. Check with CloudTrail and AWS Config rule s3-bucket-ssl-requests-only.
Some S3 objects are not encrypted with your KMS key, why and how to fix?
Default encryption works only for new objects, so old objects keep old encryption. Also if the client sends its own encryption header, it overrides the default. I find them with S3 Inventory. I re-encrypt with S3 Batch Operations copy in place. To prevent, bucket policy denies PutObject if encryption is not aws:kms or the key ID is not my key ARN. KMS key policy allows only the right roles. AWS Config for monitoring.
Same Terraform module in dev and prod, you add an IP range, dev plan is clean, prod plan wants to destroy route table associations. Why?
The module uses count with a list. Terraform tracks resources by index. In prod the list is longer or in different order, so new item shifts indexes and Terraform wants to destroy and recreate. Fix is for_each with stable keys like the IP range. Use moved blocks or terraform state mv to move existing resources without destroy. Add prevent_destroy on critical resources, and require approval in the pipeline if prod plan shows destroy. Also check drift with plan -refresh-only and provider versions.
Kubernetes app gets 502 for about 10 seconds during every deployment. Readiness probe is present. What do you check?
Most likely old pods shutting down. Kubernetes sends SIGTERM and removes the pod at the same time, but the load balancer still sends traffic for few seconds. Fix is preStop hook with sleep 10 to 15 seconds, app handles SIGTERM gracefully, terminationGracePeriodSeconds longer than the sleep. Rolling update with maxUnavailable 0 and maxSurge 1. With ALB, enable pod readiness gates and check deregistration delay and keep-alive timeout. Add PodDisruptionBudget. Check ALB logs and pod events.
How do you use AI in your work?
I use chat for quick questions and errors. For bigger tasks I use AI agents like Claude Code. It reads the repo, makes a plan, changes Terraform or pipelines, and runs terraform plan or tests. I review the plan and the changes before merge. Agents use tools through MCP and need limited permissions, like read-only in prod and no apply without human approval. I never put secrets or customer data into AI tools.

Next round they may also ask these, because they are in the job description:

How do you deploy and monitor .NET apps?
I build the app in GitLab CI with dotnet build and dotnet test, then package it. For Windows on EC2, I deploy with PowerShell scripts or SSM Run Command to IIS, and take the server out of the load balancer during deploy. For Linux, I put .NET Core in Docker and deploy to EKS or ECS with Helm. For monitoring I use CloudWatch agent for Windows logs and IIS logs, and Datadog APM or OpenTelemetry for traces, errors and response time.
How do you patch Windows and Linux servers?
I use AWS Systems Manager Patch Manager with patch baselines and maintenance windows. First patch dev and test, then prod. For prod, I patch in groups, so not all servers behind the load balancer go down at once. Before patching I take snapshots. After patching I check the app health and the compliance report in SSM. For EKS nodes I update the AMI and do rolling node replacement.
EKS vs ECS vs App Runner?
EKS is Kubernetes, more control and flexible, good for complex apps, but more work to manage. ECS is AWS own container service, simpler, works well with Fargate and other AWS services. App Runner is the simplest, you give it image or code, and AWS handles everything, good for small web apps and APIs.
What is Helmfile?
Helmfile manages many Helm releases in one file. You define all charts, versions and values for each environment in one place, and run helmfile apply. It is good for keeping environments consistent and works well in CI/CD.
What is Terragrunt and why use it?
Terragrunt is a wrapper for Terraform. It keeps the code DRY, so you don't copy the same backend and provider config in every environment. It manages remote state config, dependencies between modules, and lets you run plan or apply across many modules and accounts.
How do you help with SOC 2?
I make sure access is least privilege with IAM roles and SSO, encryption at rest with KMS and in transit with TLS, logs in CloudTrail and CloudWatch, vulnerability scanning with Inspector or image scanning in the pipeline, and patching. I keep evidence and documentation ready for auditors, like access reviews and change history from GitLab merge requests.
What is your disaster recovery approach?
First we agree on RTO and RPO with the business. Backups for RDS with automated snapshots and cross-region copies, S3 versioning and replication, all infrastructure in Terraform so we can rebuild fast. We write a DR runbook and test it, because a plan that is not tested does not work.
Have you worked with Lambda?
Yes, I used Lambda for automation, like stopping dev instances at night, cleaning old snapshots, and processing S3 events. I deploy it with Terraform, give it a small IAM role, and monitor with CloudWatch logs and alarms for errors and duration.
What is Ansible used for?
Configuration management. It connects to servers over SSH for Linux or WinRM for Windows, and runs playbooks to install software, change config and patch. It is agentless and idempotent, so you can run it many times and it gives the same result.
Biggest weakness?
Sometimes I go too deep into a problem and want to fix it myself. Now I set a time limit, and if I don't solve it, I ask the team early.
Where do you see yourself in 3 to 5 years?
I want to grow into Senior or Lead DevOps Engineer, own bigger parts of the platform, and help build automation and AI tools for the team.

JumpToTech DevOps Bootcamp Real Estate Company — Complete AWS EKS Integration Lab

Aisalkyn Aidarova — Thu, 05 Nov 2026 17:45:05 +0000

1. What you will build

Customer / Browser

View properties, submit inquiries

AWS Application Load Balancer

Kubernetes Ingress managed by AWS Load Balancer Controller

Frontend Deployment

Nginx + React

frontend-service

Backend Deployment

FastAPI + SQL

backend-service

Amazon RDS PostgreSQL

Main application database — outside Kubernetes

Separate PostgreSQL StatefulSet

PVC → PV → AWS EBS volume

Persistence experiment only

Fluent Bit DaemonSet

Collects node container logs → CloudWatch Logs

What each component is responsible for

Component	Responsibility
Frontend Deployment	Runs website Pods and maintains desired replicas
Backend Deployment	Runs API Pods that process property and inquiry requests
ReplicaSet	Maintains Pods created by a Deployment
ClusterIP Service	Provides a stable internal network endpoint
Ingress / ALB	Routes incoming HTTP traffic to Services
Amazon RDS	Manages PostgreSQL database infrastructure
StatefulSet	Maintains stable Pod identities for our separate database experiment
PVC / PV	Connects the StatefulSet to persistent storage
StorageClass	Defines how storage is dynamically provisioned
EBS CSI Driver	Integrates Kubernetes storage operations with AWS EBS
DaemonSet	Ensures a log collection agent runs on eligible nodes
kubelet	Starts and monitors Pod containers on a worker node
IAM	Controls AWS permissions for controllers and workloads
Kubernetes Secret	Provides application credentials to Pods

Important: We will not place RDS inside a StatefulSet. These are two separate database deployment models.

2. Lab phases

Phase	Activity
1	Inspect existing repository and EKS cluster
2	Verify frontend/backend application and Docker images
3	Deploy frontend/backend using Deployments
4	Create internal ClusterIP Services
5	Configure ALB Ingress
6	Provision and connect Amazon RDS PostgreSQL
7	Deploy separate PostgreSQL StatefulSet
8	Configure EBS CSI, StorageClass and PVC
9	Deploy Fluent Bit DaemonSet with CloudWatch
10	Test, troubleshoot, document and clean up

This is a multi-session project, not a 30-minute exercise. Complete one phase successfully before proceeding.

3. Phase 1 — Inspect the existing project

Why: A DevOps Engineer should understand the existing infrastructure and application before modifying it.

On the EC2 instance:

cd ~/real-estate-company

pwd
git status --short
git remote -v

ls -la
ls -la backend
ls -la frontend
ls -la k8s

find backend frontend k8s \
  -maxdepth 2 -type f | sort | head -100

Inspect Docker and Kubernetes files:

find . -name Dockerfile -o -name '*.yaml' -o -name '*.yml'

Check the AWS identity and cluster:

aws sts get-caller-identity
aws configure get region
kubectl config current-context
kubectl get nodes -o wide
kubectl get namespaces
kubectl get deployments,services,ingresses -A

Student checkpoint: Identify the frontend technology, backend technology, existing container ports, current Kubernetes namespace, ECR repositories and EKS cluster name.

Do not print .env files, passwords, tokens or database connection strings into shared classroom screenshots.

4. Phase 2 — Prepare the application

What the application must do

Your Real Estate Company will have two parts.

Frontend: Display available properties, prices, locations and an inquiry form.

Backend: Return property listings, receive customer inquiries, validate requests and save them in PostgreSQL.

The backend is responsible for SQL operations. Kubernetes is responsible for running and connecting the application, not for implementing its business logic.

4.1 Reference backend — FastAPI

If your existing backend already implements these features, retain it and adapt the Kubernetes configuration. Otherwise, create this reference backend in a separate directory so that you do not overwrite your classmates' work.

cd ~/real-estate-company
mkdir -p lab-reference/backend
cd lab-reference/backend

Create requirements.txt:

fastapi==0.115.12
uvicorn==0.34.2
psycopg[binary]==3.2.6

Create main.py:

import osfrom contextlib import asynccontextmanagerimport psycopgfrom fastapi import FastAPI, HTTPExceptionfrom pydantic import BaseModel, FieldDATABASE_URL = os.environ["DATABASE_URL"]def connect():    return psycopg.connect(DATABASE_URL, connect_timeout=5)@asynccontextmanagerasync def lifespan(app: FastAPI):    with connect() as conn:        conn.execute("""            CREATE TABLE IF NOT EXISTS properties (                id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,                title TEXT NOT NULL,                city TEXT NOT NULL,                price INTEGER NOT NULL            )        """)        conn.execute("""            CREATE TABLE IF NOT EXISTS inquiries (                id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,                property_id INTEGER NOT NULL REFERENCES properties(id),                customer_name TEXT NOT NULL,                email TEXT NOT NULL,                message TEXT NOT NULL,                created_at TIMESTAMPTZ NOT NULL DEFAULT now()            )        """)        count = conn.execute(

This is teaching code, not a production-ready customer-data platform. It omits authentication, authorization, rate limiting, full email validation and a proper migration workflow. Use fictitious customer information only.

Create Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

RUN useradd --uid 10001 --create-home appuser
USER 10001

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Why Docker? Kubernetes deploys container images. It does not directly deploy your Python source folder.

4.2 Reference frontend — React

For students who already have a working React frontend, retain the existing code and adapt its API calls to /api/properties and /api/inquiries.

For a new reference frontend:

cd ~/real-estate-company/lab-reference
npm create vite@latest frontend -- --template react
cd frontend
npm install

Replace src/App.jsx:

import { useEffect, useState } from "react";

export default function App() {
  const [properties, setProperties] = useState([]);
  const [selected, setSelected] = useState("");
  const [name, setName] = useState("");
  const [email, setEmail] = useState("");
  const [message, setMessage] = useState("");
  const [result, setResult] = useState("");

  useEffect(() => {
    fetch("/api/properties")
      .then(async r => {
        if (!r.ok) throw new Error("API unavailable");
        return r.json();
      })
      .then(data => {
        setProperties(data);
        if (data.length) setSelected(String(data[0].id));
      })
      .catch(() => setResult("Unable to load properties"));
  }, []);

  async function submit(e) {
    e.preventDefault();
    try {
      const response = await fetch("/api/inquiries", {
        method: "POST",
        headers: {"Content-Type": "application/json"},
        body: JSON.stringify({
          property_id: Number(selected),
          customer_name: name,
          email,
          message
        })
      });
      if (!response.ok) throw new Error("Request failed");
      const data = await response.json();
      setResult(`Inquiry saved! Reference #${data.id}`);
      setMessage("");
    } catch {
      setResult("Unable to save inquiry");
    }
  }

  return (
    <main style={{maxWidth: 900, margin: "40px auto", fontFamily: "Arial"}}>
      <h1>JumpToTech Real Estate</h1>
      <p>Explore properties and request more information.</p>

      <section style={{display: "grid", gap: 16}}>
        {properties.map(p => (
          <article key={p.id} style={{border: "1px solid #ddd", padding: 16}}>
            <h2>{p.title}</h2>
            <p>{p.city}</p>
            <strong>${p.price.toLocaleString()}</strong>
          </article>
        ))}
      </section>

      <h2>Contact an agent</h2>
      <form onSubmit={submit} style={{display: "grid", gap: 12}}>
        <select value={selected} onChange={e => setSelected(e.target.value)}>
          {properties.map(p =>
            <option key={p.id} value={p.id}>{p.title}</option>
          )}
        </select>
        <input placeholder="Your name" value={name}
          onChange={e => setName(e.target.value)} required />
        <input type="email" placeholder="Email" value={email}
          onChange={e => setEmail(e.target.value)} required />
        <textarea placeholder="Your question" value={message}
          onChange={e => setMessage(e.target.value)} required />
        <button type="submit">Send inquiry</button>
      </form>
      <p role="status">{result}</p>
    </main>
  );
}

Create nginx.conf:

server {
    listen 8080;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }
}

Create Dockerfile:

FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginxinc/nginx-unprivileged:stable-alpine
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/dist /usr/share/nginx/html
EXPOSE 8080

Why use /api? The browser can access the frontend and backend through the same ALB hostname. The Ingress sends /api/* requests to the backend, avoiding the need for browser-side access to a private ClusterIP.

5. Phase 3 — Build images and push to Amazon ECR

First define your AWS environment.

export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity \
  --query Account --output text)

export EKS_CLUSTER="YOUR_EKS_CLUSTER"
export NAMESPACE="real-estate-lab"

export FRONTEND_REPO="real-estate-frontend"
export BACKEND_REPO="real-estate-backend"

export IMAGE_TAG="lab-v1"

Replace YOUR_EKS_CLUSTER with the actual cluster name.

Create ECR repositories

aws ecr describe-repositories \
  --repository-names "$FRONTEND_REPO" \
  --region "$AWS_REGION" >/dev/null 2>&1 ||
aws ecr create-repository \
  --repository-name "$FRONTEND_REPO" \
  --region "$AWS_REGION"

Repeat for the backend:

aws ecr describe-repositories \
  --repository-names "$BACKEND_REPO" \
  --region "$AWS_REGION" >/dev/null 2>&1 ||
aws ecr create-repository \
  --repository-name "$BACKEND_REPO" \
  --region "$AWS_REGION"

Authenticate Docker:

aws ecr get-login-password --region "$AWS_REGION" |
docker login \
  --username AWS \
  --password-stdin \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"

Build and push from the reference implementation:

cd ~/real-estate-company/lab-reference

docker build --platform linux/amd64 \
  -t "$BACKEND_REPO:$IMAGE_TAG" ./backend

docker tag "$BACKEND_REPO:$IMAGE_TAG" \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$BACKEND_REPO:$IMAGE_TAG"

docker push \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$BACKEND_REPO:$IMAGE_TAG"

Frontend:

docker build --platform linux/amd64 \
  -t "$FRONTEND_REPO:$IMAGE_TAG" ./frontend

docker tag "$FRONTEND_REPO:$IMAGE_TAG" \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$FRONTEND_REPO:$IMAGE_TAG"

docker push \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$FRONTEND_REPO:$IMAGE_TAG"

These commands assume x86-64 EKS nodes. If your EKS nodes are ARM64, use linux/arm64 instead.

Verify

aws ecr list-images \
  --repository-name "$BACKEND_REPO" \
  --region "$AWS_REGION"

aws ecr list-images \
  --repository-name "$FRONTEND_REPO" \
  --region "$AWS_REGION"

Student question: Why does Kubernetes need the ECR image URI rather than the source-code directory?

Answer: The container runtime pulls a packaged image containing the application and its dependencies.

6. Phase 4 — Amazon RDS PostgreSQL

Why RDS?

In this architecture, RDS manages the database service outside the Kubernetes cluster.

Kubernetes does not schedule or restart the RDS database process. AWS manages the database instance infrastructure, while your team remains responsible for database configuration, access control, schema, queries and appropriate backup settings.

6.1 Inspect networking before provisioning

aws eks describe-cluster \
  --name "$EKS_CLUSTER" \
  --region "$AWS_REGION" \
  --query 'cluster.resourcesVpcConfig'

Find the VPC ID and subnet IDs.

For this lab, RDS should be in private subnets reachable from the EKS worker nodes. Do not make PostgreSQL publicly accessible.

You will need:

A DB subnet group covering suitable subnets in at least two Availability Zones.
An RDS security group.
A rule allowing TCP 5432 from the EKS node security group, or another narrowly scoped security group used by the workloads.
A PostgreSQL database and database user.
A secure way to provide the credentials to the application.

6.2 Create RDS

For a shared class, the instructor should provision RDS once. Students should not each create expensive AWS infrastructure without checking costs and quotas.

In the AWS Console:

Open RDS → Databases → Create database.
Select PostgreSQL.
Choose an appropriately small lab instance, such as db.t4g.micro where supported and available.
Set database name to realestate.
Use RDS-managed master credentials in AWS Secrets Manager where available.
Choose the same VPC as EKS.
Select a private DB subnet group.
Set Public access to No.
Attach the restricted RDS security group.
Configure backups and deletion protection according to the lab's cleanup plan.

Do not store the master password in Git.

After RDS becomes available, record the endpoint:

YOUR_RDS_ENDPOINT.rds.amazonaws.com

6.3 Create an application database user

Use a database administrator connection from an approved private network location.

Create a dedicated application user with the permissions required for the lab rather than using the RDS master account in the backend.

For a simplified classroom environment, an instructor can provision the database and grant the application user ownership of its schema. Production systems should use separate migration and runtime roles with narrower privileges.

6.4 Create the Kubernetes Secret

The simplest teaching method is a Kubernetes Secret. In production, prefer AWS Secrets Manager with a supported integration such as the Secrets Store CSI Driver or External Secrets Operator.

Create the namespace:

kubectl create namespace "$NAMESPACE"

On your private terminal, read the application password without echoing it:

read -rs -p "RDS application password: " DB_PASSWORD
echo

Construct the connection string using the actual endpoint, database username and URL-encoded password. Special characters in passwords must be percent-encoded in a URI.

For this lab, a simpler and safer option is to create individual Secret keys:

kubectl create secret generic realestate-db \
  -n "$NAMESPACE" \
  --from-literal=DB_HOST="YOUR_RDS_ENDPOINT" \
  --from-literal=DB_PORT="5432" \
  --from-literal=DB_NAME="realestate" \
  --from-literal=DB_USER="realestate_app" \
  --from-literal=DB_PASSWORD="$DB_PASSWORD"

Then remove the shell variable:

unset DB_PASSWORD

Do not paste passwords into a shared Zoom chat or commit them to Git.

For the reference FastAPI application, we will use a small code change to construct the PostgreSQL connection from these environment variables instead of a single URL.

Replace:

DATABASE_URL = os.environ["DATABASE_URL"]def connect():    return psycopg.connect(DATABASE_URL, connect_timeout=5)

with:

def connect():    return psycopg.connect(        host=os.environ["DB_HOST"],        port=int(os.environ.get("DB_PORT", "5432")),        dbname=os.environ["DB_NAME"],        user=os.environ["DB_USER"],        password=os.environ["DB_PASSWORD"],        connect_timeout=5,        sslmode="require"    )

Rebuild and push the backend image with a new tag, for example lab-v2, and use that tag in the Deployment below.

Production note: sslmode=require encrypts the connection but does not provide full server identity verification. Production should use verify-full with an appropriate RDS CA bundle.

7. Phase 5 — Kubernetes Deployments and Services

Create a directory for the new lab manifests:

cd ~/real-estate-company
mkdir -p lab-reference/k8s
cd lab-reference/k8s

7.1 Backend Deployment

Create backend.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: realestate-backend
  namespace: real-estate-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: realestate-backend
  template:
    metadata:
      labels:
        app: realestate-backend
    spec:
      containers:
        - name: backend
          image: BACKEND_IMAGE_URI
          ports:
            - containerPort: 8000
          envFrom:
            - secretRef:
                name: realestate-db
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /healthz
              port: 8000
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8000
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8000
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: backend-service
  namespace: real-estate-lab
spec:
  type: ClusterIP
  selector:
    app: realestate-backend
  ports:
    - port: 80
      targetPort: 8000

Replace BACKEND_IMAGE_URI with the full ECR image URI, including its tag.

Why two replicas? We want to demonstrate how a Deployment maintains multiple backend Pods and how a Service routes requests to healthy endpoints.

Why separate readiness and liveness? If RDS is unavailable, readiness should fail so the Pod stops receiving traffic. Liveness should not depend on RDS, or a temporary database outage could cause unnecessary container restarts.

7.2 Frontend Deployment

Create frontend.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: realestate-frontend
  namespace: real-estate-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: realestate-frontend
  template:
    metadata:
      labels:
        app: realestate-frontend
    spec:
      containers:
        - name: frontend
          image: FRONTEND_IMAGE_URI
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /
              port: 8080
          livenessProbe:
            httpGet:
              path: /
              port: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  namespace: real-estate-lab
spec:
  type: ClusterIP
  selector:
    app: realestate-frontend
  ports:
    - port: 80
      targetPort: 8080

Replace FRONTEND_IMAGE_URI with the full ECR URI.

7.3 Apply and verify

kubectl apply -f backend.yaml
kubectl apply -f frontend.yaml

kubectl rollout status deployment/realestate-backend \
  -n real-estate-lab --timeout=180s

kubectl rollout status deployment/realestate-frontend \
  -n real-estate-lab --timeout=180s

Check:

kubectl get deployments -n real-estate-lab
kubectl get replicasets -n real-estate-lab
kubectl get pods -n real-estate-lab -o wide
kubectl get services -n real-estate-lab
kubectl get endpointslices -n real-estate-lab

Expected result: Two frontend Pods and two backend Pods should be Ready, provided the images can be pulled and the backend can reach RDS.

If backend Pods are not Ready, investigate before moving on:

kubectl describe pod -n real-estate-lab \
  -l app=realestate-backend

kubectl logs deployment/realestate-backend \
  -n real-estate-lab --tail=50

7.4 Test the backend without ALB

Use port forwarding from a machine with kubectl access:

kubectl port-forward \
  -n real-estate-lab svc/backend-service 8081:80

In a second terminal:

curl -i http://localhost:8081/healthz
curl -i http://localhost:8081/readyz
curl -i http://localhost:8081/api/properties

Create a sample inquiry:

curl -i -X POST http://localhost:8081/api/inquiries \
  -H 'Content-Type: application/json' \
  -d '{
    "property_id": 1,
    "customer_name": "Test Student",
    "email": "student@example.com",
    "message": "I would like a property tour."
  }'

Verify:

curl http://localhost:8081/api/inquiries/count

Student checkpoint: Explain the complete path from the HTTP request through Service, Backend Pod and RDS PostgreSQL.

8. Phase 6 — AWS ALB Ingress

Why Ingress?

Our ClusterIP Services are reachable only through Kubernetes networking. Customers need an external HTTP entry point.

AWS Load Balancer Controller can create and manage an Application Load Balancer from a Kubernetes Ingress.

Before proceeding, verify that the AWS Load Balancer Controller is installed, has appropriate IAM permissions and that the VPC subnets are correctly tagged for load balancer discovery.

kubectl get deployment aws-load-balancer-controller \
  -n kube-system

kubectl get ingressclass

If the controller is missing, do not assume that creating an Ingress will automatically provision an ALB. The instructor must first install the controller using the supported AWS installation procedure and IAM configuration.

8.1 Create the Ingress

Create ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: realestate-ingress
  namespace: real-estate-lab
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: backend-service
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80

Apply:

kubectl apply -f ingress.yaml
kubectl get ingress -n real-estate-lab
kubectl describe ingress realestate-ingress \
  -n real-estate-lab

The ADDRESS field should eventually contain an ALB hostname if provisioning succeeds.

Important production distinction

This is an HTTP-only classroom configuration. Do not send real customer information through it. For a real application, configure HTTPS using ACM, an HTTPS listener, appropriate redirect rules and an approved DNS name.

The single ALB routes /api to Backend and / to Frontend. The frontend uses relative /api requests, so browser requests reach the same ALB hostname.

8.2 Verify

Once the ALB hostname is assigned:

export ALB_DNS=$(kubectl get ingress realestate-ingress \
  -n real-estate-lab \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "$ALB_DNS"

curl -i "http://$ALB_DNS/"
curl -i "http://$ALB_DNS/api/properties"

If you see 502, 503 or 504, do not immediately restart the Pods. Inspect the ALB target groups, backend health, Service endpoints, application logs and database connectivity.

9. Phase 7 — PostgreSQL StatefulSet with EBS

This phase is a separate experiment from the RDS database.

Our main Real Estate Company uses RDS. Now we will deploy another PostgreSQL instance inside Kubernetes to understand StatefulSet, PVC, PV, StorageClass and the EBS CSI driver.

9.1 Understand the storage path

StatefulSet

Creates and manages postgres-0

Pod

Runs the PostgreSQL database software

PVC

Requests 5 GiB of persistent storage

StorageClass

Selects the EBS storage provisioner

EBS CSI Driver

Coordinates EBS volume provisioning and attachment

PV / EBS Volume

Stores PostgreSQL data files

9.2 Check EBS CSI and StorageClasses

kubectl get csidrivers
kubectl get storageclass -o wide

For a standard EKS installation with the EBS CSI driver, the CSI provisioner is typically:

ebs.csi.aws.com

For EKS Auto Mode, the provisioner is:

ebs.csi.eks.amazonaws.com

Use the driver and StorageClass that actually match your cluster.

If EBS CSI is missing, the instructor must install the appropriate EKS add-on and configure its AWS IAM permissions. The CSI controller needs permissions to provision and manage EBS volumes. Merely creating a PVC will not grant those permissions.

For standard EKS with the standard EBS CSI driver, create storageclass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: realestate-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

For EKS Auto Mode, use its supported provisioner instead and review the applicable configuration before creating the class.

Apply:

kubectl apply -f storageclass.yaml

Why WaitForFirstConsumer? EBS volumes are Availability Zone-specific. Kubernetes should consider the Pod's placement before provisioning the volume.

Why reclaimPolicy: Delete? This is a disposable classroom lab. Deleting the PVC may lead to deletion of its dynamically provisioned EBS volume. Do not use this configuration for valuable data without understanding its consequences.

9.3 Create the PostgreSQL credentials

read -rs -p "Lab PostgreSQL password: " PG_PASSWORD
echo

kubectl create secret generic postgres-lab-secret \
  -n real-estate-lab \
  --from-literal=POSTGRES_USER=labuser \
  --from-literal=POSTGRES_PASSWORD="$PG_PASSWORD" \
  --from-literal=POSTGRES_DB=labdb

unset PG_PASSWORD

9.4 Create PostgreSQL StatefulSet

Create postgres-statefulset.yaml:

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  namespace: real-estate-lab
spec:
  clusterIP: None
  selector:
    app: postgres-lab
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: real-estate-lab
spec:
  serviceName: postgres-headless
  replicas: 1
  selector:
    matchLabels:
      app: postgres-lab
  template:
    metadata:
      labels:
        app: postgres-lab
    spec:
      securityContext:
        fsGroup: 999
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
              name: postgres
          envFrom:
            - secretRef:
                name: postgres-lab-secret
          env:
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 10
            periodSeconds: 10
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: realestate-gp3
        resources:
          requests:
            storage: 5Gi

Why one replica? Creating three PostgreSQL Pods does not automatically configure PostgreSQL replication. We are demonstrating persistence and stable identity, not building a highly available database cluster.

Apply:

kubectl apply -f postgres-statefulset.yaml

kubectl get statefulset -n real-estate-lab
kubectl get pods -n real-estate-lab -l app=postgres-lab
kubectl get pvc -n real-estate-lab
kubectl get pv

Wait until postgres-0 is Running/Ready and the PVC is Bound.

9.5 Create test data

Open PostgreSQL:

kubectl exec -it postgres-0 \
  -n real-estate-lab -- \
  psql -U labuser -d labdb

Inside psql:

CREATE TABLE property_notes (
    id SERIAL PRIMARY KEY,
    property_name TEXT NOT NULL,
    note TEXT NOT NULL
);

INSERT INTO property_notes (property_name, note)
VALUES ('Chicago Condo', 'Customer requested a tour');

SELECT * FROM property_notes;

Exit:

\q

9.6 Delete the PostgreSQL Pod

kubectl delete pod postgres-0 -n real-estate-lab

Wait:

kubectl rollout status statefulset/postgres \
  -n real-estate-lab --timeout=300s

Check the data again:

kubectl exec postgres-0 -n real-estate-lab -- \
  psql -U labuser -d labdb \
  -c "SELECT * FROM property_notes;"

Expected: The saved record remains because PostgreSQL is using its persistent volume.

Student explanation

The StatefulSet recreated postgres-0 with its stable identity. The PVC remained associated with the StatefulSet ordinal, and Kubernetes remounted the existing persistent storage.

If the underlying EBS volume is deleted, corrupted or unavailable, StatefulSet alone cannot recover the data.

10. Phase 8 — DaemonSet and centralized logging

What is DaemonSet responsible for?

A DaemonSet controller ensures that an eligible worker node runs a Pod for a particular node-level workload.

Typical examples include log collectors, monitoring agents and some networking components.

A DaemonSet does not replace kubelet, and it does not automatically centralize logs. We need a log collector and a destination.

For this project:

Fluent Bit DaemonSet → Amazon CloudWatch Logs

10.1 Understand the architecture

Worker Node 1

Application Pods

Fluent Bit Pod

Worker Node 2

Application Pods

Fluent Bit Pod

Amazon CloudWatch Logs

Centralized application and container logs

10.2 Verify IAM permissions

The Fluent Bit workload needs permission to create log streams and publish log events.

For standard EKS, prefer a dedicated IAM role for the collector using EKS Pod Identity or IRSA, according to your cluster configuration. Do not give every application Pod broad CloudWatch permissions.

A minimum illustrative policy for an instructor-created CloudWatch log group is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:ACCOUNT_ID:log-group:/jumptotech/real-estate:*"
      ]
    }
  ]
}

Replace ACCOUNT_ID with the actual AWS account ID. The instructor must configure the trust relationship and attach this policy to the collector's dedicated role.

Create the log group:

aws logs create-log-group \
  --log-group-name /jumptotech/real-estate \
  --region "$AWS_REGION"

If the log group already exists, AWS returns an already-exists error; that does not mean logging is broken.

Set a short retention period for the lab:

aws logs put-retention-policy \
  --log-group-name /jumptotech/real-estate \
  --retention-in-days 7 \
  --region "$AWS_REGION"

10.3 Install Fluent Bit with Helm

For a real centralized-logging lab, use the maintained Fluent Bit Helm chart rather than writing an incomplete logging DaemonSet from scratch.

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

helm show values fluent/fluent-bit > fluent-default-values.yaml

The Helm chart must be configured for your EKS runtime log paths, CloudWatch output and IAM integration. Do not assume that the default values automatically send logs to CloudWatch.

Create fluent-values.yaml:

kind: DaemonSet

serviceAccount:
  create: true
  name: fluent-bit-realestate

config:
  service: |
    [SERVICE]
        Flush         5
        Daemon        Off
        Log_Level     info
        Parsers_File  /fluent-bit/etc/parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

  inputs: |
    [INPUT]
        Name              tail
        Path              /var/log/containers/*real-estate-lab*.log
        Tag               kube.*
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        DB                /var/log/flb_realestate.db

  filters: |
    [FILTER]
        Name   kubernetes
        Match  kube.*
        Kube_Tag_Prefix kube.var.log.containers.
        Merge_Log On
        Keep_Log Off

  outputs: |
    [OUTPUT]
        Name cloudwatch_logs
        Match kube.*
        region us-east-1
        log_group_name /jumptotech/real-estate
        log_stream_prefix fluentbit-
        auto_create_group false

Install only after the Fluent Bit ServiceAccount has been connected to the appropriate AWS IAM role. With EKS Pod Identity, this requires a Pod Identity association for the ServiceAccount; with IRSA, the ServiceAccount needs the role annotation and a valid OIDC trust policy.

The chart's default hostPath mounts must also be verified against your node runtime. The above values are a reference configuration to be validated, not a guarantee that every EKS node image exposes identical log paths.

helm upgrade --install realestate-logs \
  fluent/fluent-bit \
  -n logging --create-namespace \
  -f fluent-values.yaml

Check:

kubectl get daemonsets -n logging
kubectl get pods -n logging -o wide
kubectl logs -n logging \
  -l app.kubernetes.io/name=fluent-bit \
  --tail=50

10.4 Verify centralized logs

Generate some backend traffic:

curl "http://$ALB_DNS/api/properties"

Then inspect CloudWatch:

aws logs describe-log-streams \
  --log-group-name /jumptotech/real-estate \
  --region "$AWS_REGION"

Read recent logs:

aws logs tail /jumptotech/real-estate \
  --since 10m \
  --region "$AWS_REGION"

If no logs appear, inspect the Fluent Bit Pod logs, IAM permissions, log paths, filter configuration and output plugin errors.

Student question: Why do we use DaemonSet rather than a Deployment with three replicas for a node-level log collector?

Answer: A DaemonSet ties desired Pod placement to eligible nodes. A Deployment controls replica count but does not guarantee one collector on every node.

11. Phase 9 — Terraform and infrastructure ownership

Terraform is an infrastructure-as-code tool. It can provision the AWS infrastructure that Kubernetes applications depend on.

For this lab, divide ownership as follows:

Resource	Managed by	Responsible team
VPC, subnets, security groups	Terraform / AWS	Cloud / DevOps
EKS cluster and node groups	Terraform / AWS	Platform / DevOps
ECR repositories	Terraform / AWS	DevOps
RDS instance and subnet group	Terraform / AWS	Cloud / DBA
AWS IAM roles and policies	Terraform / AWS	Cloud Security / DevOps
Frontend and backend code	Git	Application developers
Deployments and Services	Kubernetes manifests / Helm	DevOps
PostgreSQL StatefulSet and PVC	Kubernetes	DevOps / DBA
EBS volumes	EBS CSI Driver / AWS	Kubernetes storage integration
Fluent Bit	Helm / DaemonSet	Platform / SRE
Database schema and migrations	SQL migration tooling	Backend / DBA

Terraform example: RDS security group

This example demonstrates how to restrict PostgreSQL access to the EKS worker-node security group.

Create terraform/rds-security.tf in a separate Terraform project:

variable "vpc_id" {
  type = string
}

variable "eks_node_security_group_id" {
  type = string
}

resource "aws_security_group" "rds" {
  name_prefix = "realestate-rds-"
  description = "RDS access from EKS worker nodes"
  vpc_id      = var.vpc_id

  tags = {
    Project = "JumpToTech-RealEstate"
  }
}

resource "aws_vpc_security_group_ingress_rule" "postgres" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = var.eks_node_security_group_id
  ip_protocol                  = "tcp"
  from_port                    = 5432
  to_port                      = 5432
}

resource "aws_vpc_security_group_egress_rule" "rds_egress" {
  security_group_id = aws_security_group.rds.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

This is only one component of the RDS Terraform configuration, not a complete standalone infrastructure deployment. The VPC, provider, DB subnet group, RDS instance, credential management and Terraform backend still need to be configured.

Important: Students must never run terraform apply against an existing shared EKS infrastructure project without understanding its state and proposed changes.

12. Phase 10 — Production troubleshooting exercises

Each student must investigate these scenarios independently and explain the findings.

Scenario	Symptoms	First checks
ImagePullBackOff	Pod cannot start	Image URI, ECR tag, pull permissions
CrashLoopBackOff	Container repeatedly exits	`kubectl logs --previous`, events
Backend not Ready	Readiness probe fails	`/readyz`, RDS connectivity, credentials
Service unavailable	No backend endpoints	Selectors, Pod labels, EndpointSlices
ALB 502	ALB cannot get valid backend response	Target health, ports, app process
ALB 503	No available targets or other service issue	Target groups, endpoints, health
ALB 504	Backend response timeout	Application latency, networking, RDS
PVC Pending	PostgreSQL cannot obtain storage	StorageClass, CSI driver, IAM, scheduling
PostgreSQL Pod Pending	Pod cannot schedule	Node capacity, AZ constraints, volume events
Fluent Bit no logs	CloudWatch log group empty	IAM, input path, collector logs, output errors

Troubleshooting commands

kubectl get all -n real-estate-lab

kubectl get pods -n real-estate-lab -o wide

kubectl describe deployment realestate-backend \
  -n real-estate-lab

kubectl logs deployment/realestate-backend \
  -n real-estate-lab --tail=100

kubectl get svc -n real-estate-lab

kubectl get endpointslices -n real-estate-lab

kubectl get events -n real-estate-lab \
  --sort-by=.lastTimestamp

kubectl get pvc -n real-estate-lab

kubectl describe pvc postgres-data-postgres-0 \
  -n real-estate-lab

kubectl get daemonsets -A

kubectl get ingress -n real-estate-lab

Simulated production incident

Your instructor will introduce a problem.

Incident: Customers can open the Real Estate Company website, but submitting an inquiry fails.

Students must:

Verify whether the frontend is available.
Check the backend HTTP response.
Inspect backend Pod readiness.
Check Service selectors and EndpointSlices.
Examine application logs.
Test PostgreSQL connectivity.
Identify the root cause.
Propose a safe correction.
Verify that inquiries can be saved again.
Write a short incident report.

Do not delete a namespace or restart all workloads as your first troubleshooting action.

13. Final student assessment

Student completion checklist

0/12 complete

Frontend and backend container images pushed to ECR

Frontend and backend Deployments running

ClusterIP Services and EndpointSlices verified

ALB routes frontend and backend traffic

RDS PostgreSQL connected to Backend

Customer inquiry saved and retrieved

PostgreSQL StatefulSet created

PVC Bound to an EBS-backed PV

PostgreSQL data survives Pod replacement

Fluent Bit DaemonSet deployed

CloudWatch receives application logs

One incident investigated and documented

Interview questions

Students should be able to answer these questions in English without reading notes.

What is the difference between Deployment and StatefulSet?
Why does the frontend use a Deployment?
Why do we use ClusterIP Services?
What is the difference between Amazon RDS and PostgreSQL in a StatefulSet?
What happens if the PostgreSQL Pod is deleted?
What is the difference between PV, PVC and StorageClass?
What does the EBS CSI Driver do?
Why is a DaemonSet suitable for a log collector?
What is the difference between a DaemonSet controller and kubelet?
How do you troubleshoot a backend Pod that is Running but not Ready?
What is the purpose of AWS IAM in this architecture?
How does a customer HTTP request travel from ALB to PostgreSQL?

14. Cleanup — prevent unnecessary AWS charges

Do not delete resources from a shared or production cluster. Confirm the namespace and ownership before cleanup.

First inspect:

kubectl get all -n real-estate-lab
kubectl get pvc -n real-estate-lab
kubectl get ingress -n real-estate-lab

Remove the ALB Ingress and verify that the AWS Load Balancer Controller has deleted the corresponding ALB:

kubectl delete ingress realestate-ingress \
  -n real-estate-lab

Uninstall Fluent Bit if it was installed specifically for this lab:

helm uninstall realestate-logs -n logging

Remove the StatefulSet:

kubectl delete statefulset postgres \
  -n real-estate-lab

The StatefulSet's PVC is generally retained when the StatefulSet is deleted. Inspect the PVC and PV before deleting storage:

kubectl get pvc -n real-estate-lab
kubectl get pv

If the PostgreSQL data is disposable and the PVC belongs to this lab:

kubectl delete pvc postgres-data-postgres-0 \
  -n real-estate-lab

With the lab's Delete reclaim policy, this can cause the backing EBS volume to be deleted. Verify the actual EBS resource state in AWS.

After verifying that all resources in the namespace belong to this lab:

kubectl delete namespace real-estate-lab

Remove the lab-specific RDS instance, snapshots, CloudWatch log group, ECR images, IAM associations and other AWS resources only after verifying their ownership and backup requirements. Deleting the Kubernetes namespace does not automatically delete Amazon RDS.

Do not delete the EKS cluster if other students or projects use it.

15. What students must submit

Each student must submit a GitHub repository link and a short lab report containing evidence of the deployment, networking, database persistence, logging and troubleshooting.

The report should explain:

Why the frontend and backend use Deployments.
Why RDS is separate from Kubernetes.
How StatefulSet, PVC, PV, StorageClass and EBS CSI work together.
Why the log collector uses a DaemonSet.
Which Kubernetes controllers are responsible for maintaining the desired state.
What failed during troubleshooting and how the student verified recovery.

Students must not commit passwords, .env secrets, kubeconfig credentials or AWS access keys.

Final teaching outcome

By the end of this project, you should be able to explain the architecture of a multi-tier application on AWS EKS and demonstrate how its components work together.

The goal is not simply to create Pods. The goal is to understand application deployment, networking, database persistence, AWS permissions, observability and production troubleshooting.

Instructor implementation note: The application and Kubernetes examples above form a reference lab, but several infrastructure prerequisites are environment-specific: EKS IAM configuration, ALB controller installation, RDS networking and credentials, EBS CSI installation, and Fluent Bit AWS permissions. These must be validated against the actual student cluster before calling the deployment fully operational.

THRIVE MARKET SRE – TECHNICAL INTERVIEW PREP, questions and answers for Aisel for the next step

Aisalkyn Aidarova — Mon, 02 Nov 2026 22:08:46 +0000

PART 1. WHAT YOU KNOW ABOUT THRIVE (use this in your answers)

Company is 10+ years old, everything runs on AWS.
Core app is PHP, mostly APIs for mobile apps, plus smaller APIs.
They moved from a Puppet-based system to Kubernetes on EKS.
CI is GitHub Actions, CD is Argo CD.
Still have legacy systems in different AWS accounts. They want to move them into the main prod account (caches, ALBs, services).
Biggest priorities: cost, legacy migration, Kubernetes work.
Most incidents are app-related: DB slowness or cache issues make PHP-FPM workers fill up, Nginx upstream queues grow, error rate goes up.
One past infra issue: CPU steal on Kubernetes nodes.
They care about early signals, SLOs, and error rate. They are starting to use AI for monitoring.
They scale on PHP-FPM busy workers, not CPU, because CPU is a late signal.

Use these words in your answers: early signals, user impact, step by step, reduce cost, reduce risk, root cause.

PART 2. KEEP YOUR STORY CONSISTENT

Current/last company: FitTech (Senior SRE, Jan 2024 – now). Update the resume.
Home Depot: DevOps Engineer, 2022–2023.
XPO Logistics: Cloud/DevOps Engineer, 2020–2021.
Cost savings: 28% everywhere.
Be ready to explain the 28%: rightsizing EC2 and EKS nodes, tuning pod requests, deleting unused EBS/snapshots/ALBs, shutting down dev/test at night, Terraform defaults, cost alerts.

PART 3. KUBERNETES

Troubleshooting order (say it out loud like this):

kubectl get pods -n ns – check status.
kubectl describe pod – read Events at the bottom.
kubectl logs pod, and kubectl logs pod --previous if it crashed.
kubectl get svc, kubectl get endpoints – does the Service selector match pod labels?
Check Ingress / ALB target health.
Check what changed in Git / Argo CD. If urgent, roll back first.

Common pod states:

Pending: not enough CPU/memory on nodes, node selector or taint problem, PVC not bound.
ImagePullBackOff: wrong image name/tag, no ECR permission.
CrashLoopBackOff: app starts and crashes. Check logs, env vars, secrets, config.
OOMKilled: memory limit too low or memory leak.
Running but not Ready: readiness probe failing, so no traffic.

Probes:

Readiness: is the pod ready to get traffic? If it fails, pod is removed from Service.
Liveness: is the app stuck? If it fails, container restarts.
Startup: for slow-starting apps, protects from liveness killing it too early.
Tip: liveness should NOT check the database. If DB is slow, all pods restart and make it worse.

Requests and limits:

Requests: what the scheduler reserves. Used for placement and cost.
Limits: the max. Over memory limit = OOMKilled. Over CPU limit = throttled.
Requests too high = wasted money. Too low = noisy neighbors and evictions.

Scaling:

HPA: scales pods. Default on CPU/memory, but can use custom metrics.
Custom metrics: Prometheus Adapter, or KEDA (easier, scales on Prometheus queries).
Cluster Autoscaler or Karpenter: adds/removes nodes. Karpenter is faster and picks cheaper instance types.
PDB (PodDisruptionBudget): keeps a minimum number of pods during node drains and upgrades.

Deployments:

Rolling update: maxSurge and maxUnavailable control speed.
Blue/green: two versions, switch traffic at once, easy rollback.
Canary: small % of traffic first (Argo Rollouts can do this).

Networking:

Service types: ClusterIP (internal), NodePort, LoadBalancer.
Ingress with AWS Load Balancer Controller creates ALBs. Use IngressGroup to share one ALB across many services. This saves money (good for their consolidation).
DNS problems: check CoreDNS pods.

Security:

RBAC for access, namespaces per team.
IRSA (IAM Roles for Service Accounts) or EKS Pod Identity, so pods get AWS access without static keys.
Network Policies to limit pod-to-pod traffic.
Secrets: AWS Secrets Manager with External Secrets Operator.
Image scanning in CI (Trivy) and ECR scanning.

PART 4. PHP-FPM AND NGINX (very important for Thrive)

How it breaks:
DB or cache gets slow → PHP-FPM workers wait → all workers busy → requests wait in listen queue → Nginx upstream timeouts → 502/504 errors → users feel it.

What to monitor:

PHP-FPM: active processes, idle processes, max children reached, listen queue, slow requests.
Nginx: upstream response time, 5xx rate, active connections.
Tool: php-fpm status page + php-fpm exporter for Prometheus.

How to scale:

Scale on busy worker % (for example 70–80%) or listen queue, not CPU.
Use KEDA or HPA with Prometheus custom metrics.
Warning: if DB is the root cause, more PHP pods = more DB connections = worse. Use connection limits, pooling (RDS Proxy for MySQL/Postgres), and check DB latency first.

Key settings:

pm.max_children: max workers per pod. Size it by memory: pod memory / memory per worker.
request_terminate_timeout and Nginx timeouts: so stuck requests don't hold workers forever.
Timeouts to DB and cache in the app, so one slow dependency doesn't take everything down.

PART 5. OBSERVABILITY AND SLOs

Four golden signals: latency, traffic, errors, saturation.

SLI, SLO, error budget:

SLI: what you measure. Example: % of requests that succeed in under 300 ms.
SLO: the target. Example: 99.9% over 30 days.
Error budget: 0.1% allowed failures. If used up, slow down releases and fix reliability.

Burn rate alerts (impress them with this):

Burn rate = how fast you use the error budget.
Fast burn: for example 14x over 1 hour → page right away.
Slow burn: for example 2–3x over 6 hours → ticket or warning.
Better than fixed thresholds: fewer false alerts, catches real problems early.

Leading vs lagging signals:

Leading (early): PHP-FPM queue growing, cache hit rate dropping, DB latency going up, connection count rising.
Lagging (late): CPU, error rate, users complaining.

Alerting rules:

Page only on user impact (errors, latency, SLO burn).
Everything else goes to dashboards or warnings.
Every alert must be actionable and have a runbook.

Monitoring a web app + cache + RDS:

App: request rate, 5xx rate, p95/p99 latency, pod restarts.
Cache (ElastiCache Redis): hit rate, evictions, memory, connections, latency.
RDS: CPU, FreeableMemory, FreeStorageSpace, DatabaseConnections, Read/Write latency, IOPS, DiskQueueDepth, ReplicaLag. Turn on Performance Insights for slow queries. RDS events to SNS.
ALB: UnHealthyHostCount, HTTPCode_ELB_5XX, TargetResponseTime.
Nodes: CPU steal time (their past issue).

CPU steal:

Means the hypervisor is giving CPU time to other VMs.
Common on burstable instances (t3) when credits run out.
Fix: use non-burstable types (m5/m6i/m7i), or check t3 unlimited, and monitor node_cpu_seconds_total{mode="steal"}.

AI in monitoring:

Anomaly detection (Datadog Watchdog, CloudWatch anomaly detection) catches unusual patterns.
AI helps summarize incidents and logs, but humans still decide.

PART 6. AWS

Multi-account consolidation (their big project):

Inventory what runs in each account (services, caches, ALBs, DBs, costs).
Plan the order: low-risk services first.
Networking: VPC peering or Transit Gateway so accounts can talk during migration.
Move service into the main EKS cluster with Terraform + Argo CD.
Share ALBs with IngressGroup, combine caches where safe.
Shift traffic slowly with Route 53 weighted records.
Monitor, then turn off the old resources.

Cost optimization list:

Rightsize EC2 and EKS nodes, tune pod requests.
Karpenter for better node packing and cheaper instances.
Spot instances for stateless workloads and dev/test.
Savings Plans / Reserved Instances for steady prod usage.
Graviton (ARM) instances: about 20% cheaper.
Delete unused EBS volumes, snapshots, old ALBs, unattached Elastic IPs.
Move gp2 to gp3 volumes (cheaper).
NAT Gateway data costs: use VPC endpoints for S3 and ECR.
S3 lifecycle policies.
Shut down dev/test at night and weekends.
Visibility: Cost Explorer, tags, Kubecost for cost per service/namespace.

RDS basics:

Multi-AZ for high availability (automatic failover).
Read replicas for read scaling.
Automated backups and snapshots.
Private subnets, security groups, least privilege.
RDS Proxy for connection pooling.

PART 7. TERRAFORM

Remote state in S3 with DynamoDB locking.
Modules to reuse code.
terraform plan before apply, review in pull request.
Run in CI (GitHub Actions) with plan output in the PR.
terraform import to bring legacy resources into code (useful for their legacy accounts).
Drift: when someone changes things by hand. Detect with scheduled plan.
Never put secrets in code. Use Secrets Manager.

PART 8. CI/CD

GitHub Actions: build, test, scan image, push to ECR, update Helm values or image tag in Git.
Argo CD: watches Git and syncs to cluster (GitOps). Git is the source of truth.
Rollback: revert the Git commit, or Argo CD rollback.
Use OIDC from GitHub Actions to AWS, no long-lived keys.
Safe deploys: small changes, canary or rolling, health checks, automatic rollback.

PART 9. INCIDENT RESPONSE

Steps:

Detect (alert).
Assess impact: how many users, which service.
Communicate: incident channel, status updates.
Mitigate first: rollback, scale, restart, failover. Fix users first, find root cause later.
Find root cause.
Blameless postmortem: timeline, root cause, action items.
Follow up on action items.

Key words: MTTR, blameless, mitigation first, action items.

PART 10. PRACTICE SCENARIOS (they may ask live)

Error rate suddenly goes up. What do you do?
Check what changed (deploys in Argo CD). Check dashboards: is it one service or all? Check PHP-FPM workers, Nginx upstream, cache hit rate, RDS latency and connections. If a recent deploy, roll back. Then root cause.
Latency is high but CPU is low.
Workers are waiting on something: DB, cache, or external API. Check RDS slow queries and Performance Insights, cache hit rate, PHP-FPM listen queue. CPU is low because workers are waiting, not working.
Pods keep restarting.
describe pod: OOMKilled or liveness failure? Logs with --previous. If OOM, raise memory or fix leak. If liveness, check if probe is too strict or checks the DB.
A node is NotReady.
kubectl describe node, check kubelet, disk pressure, memory pressure. Cordon and drain if needed. Karpenter or ASG replaces it.
RDS connections are at max.
Find which app opens too many connections. Lower pm.max_children or pod count, add RDS Proxy, check for connection leaks. Short-term: kill idle connections carefully.
Cache goes down.
All traffic hits the DB, DB slows, PHP workers fill up. Protect the DB: rate limit, timeouts, ElastiCache Multi-AZ with replicas.
Reduce AWS cost by 20% in 3 months.
Get visibility first (Cost Explorer, Kubecost, tags). Quick wins: unused resources, gp3, dev/test schedules. Then rightsize, Karpenter, Spot for stateless, Savings Plans, Graviton. Track monthly.
Migrate a legacy service from another account to EKS.
See Part 6 steps. Containerize, add probes and monitoring, deploy with Argo CD, shift traffic with Route 53 weights, keep rollback ready.

PART 11. LINUX BASICS (quick review)

top / htop: CPU and memory.
df -h: disk space. du -sh: folder size.
free -m: memory.
ss -tulpn or netstat: open ports.
curl -v: test HTTP.
dig / nslookup: DNS.
journalctl -u service: service logs.
ps aux | grep php-fpm: processes.
Load average vs number of CPUs.

PART 12. QUESTIONS TO ASK (tech interview)

Are you scaling PHP-FPM with KEDA or HPA custom metrics?
Do you use burn-rate alerts on your SLOs today?
What is the next legacy account or service to migrate?
Are you using Karpenter or Cluster Autoscaler, and any Spot?
What would success look like in the first 90 days?

PART 13. TIPS FOR THE DAY

If you don't know something, say: "I haven't used that directly, but here is how I would approach it." Then explain your thinking.
Think out loud in troubleshooting questions. They want to see your process.
Always say: mitigate first, then root cause.
Connect answers to their problems: PHP-FPM, cost, migration, early signals.
Keep answers short, then give one real example.

THRIVE MARKET SRE – QUESTIONS AND ANSWERS

TELL ME ABOUT YOURSELF

Hi, my name is Aisel. I'm a Site Reliability Engineer with over 6 years of experience in AWS, Kubernetes, and production support.

In my current role at FitTech, I manage production workloads on Amazon EKS. I build CI/CD pipelines with Jenkins and GitHub Actions, deploy with Helm and Argo CD, and manage AWS infrastructure with Terraform. I use Prometheus, Grafana, Datadog, and CloudWatch for monitoring, and I take part in incident response and root cause analysis. I also helped reduce our cloud costs by 28%.

Before that, I was a DevOps Engineer at Home Depot, supporting a Kubernetes microservices platform, and at XPO Logistics, building AWS infrastructure with Terraform.

I enjoy troubleshooting, finding root causes, and automating work so problems don't come back. This role is a great match because it focuses on Kubernetes migration, cost, and reliability, which is exactly what I've been doing.

WHAT ARE YOUR DAY-TO-DAY RESPONSIBILITIES?

My day usually starts with checking dashboards and alerts in Datadog and Grafana to make sure production is healthy. Then I review any issues from the night before.

During the day I work on a mix of things: supporting deployments with Argo CD, fixing or improving CI/CD pipelines in GitHub Actions and Jenkins, writing Terraform for new infrastructure, and helping developers troubleshoot problems in Kubernetes.

I also work on improvement projects, like cost optimization, better alerting, and automation with Bash and Python.

When I'm on call, I respond to incidents, restore service, and then do root cause analysis and write the postmortem.

TELL ME ABOUT A PROJECT YOU'RE PROUD OF (COST OPTIMIZATION)

At FitTech, our AWS bill was growing every month, and I was asked to reduce it.

First, I looked at the data in AWS Cost Explorer, and at real CPU and memory usage in CloudWatch and Datadog. I found three main problems: oversized EC2 instances and EKS nodes, unused resources like old EBS volumes, snapshots, and load balancers, and dev and test environments running 24/7.

Then I fixed them step by step. I rightsized instances and EKS node groups, tuned CPU and memory requests for pods, wrote Python and Bash scripts to find and clean unused resources, and scheduled dev and test to shut down at night and on weekends. I moved EBS volumes from gp2 to gp3. I also put the right sizes into Terraform, so new resources were created correctly, and set up cost alerts.

The result was a 28% reduction in monthly cloud cost, with no impact on performance.

TELL ME ABOUT ANOTHER PROJECT (MONITORING / SLOs)

At FitTech, our alerting was noisy. I had added many alerts on CPU, memory, and restarts, and on-call engineers were getting alert fatigue.

I reviewed all the alerts with the team and removed or tuned the noisy ones. Then we moved to SLI and SLO based alerting, focused on what users feel: error rate and latency. We used burn-rate alerts, so we got paged when the error budget was being used too fast. CPU and memory moved to dashboards for troubleshooting.

The result was far fewer alerts, almost every page was a real issue, and our MTTR went down because the team trusted the alerts.

TELL ME ABOUT A PROJECT (KUBERNETES RELIABILITY AT HOME DEPOT)

At Home Depot, many incidents happened during releases because pods received traffic before the app was ready, or stayed running when the app was stuck.

I collected data from past incidents and proposed adding readiness and liveness probes. I tested it on one service first, showed the results, and then worked with developers to roll it out to other services. I also set up the ALB Ingress Controller and supported rolling and blue/green deployments.

The result was fewer failed deployments and safer releases.

TELL ME ABOUT AN INCIDENT YOU HANDLED

At FitTech, one afternoon we got alerts that the error rate and latency on one of our main APIs were going up fast. Users were getting timeouts.

First, I joined the incident channel and checked what changed. There was no recent deployment. In the dashboards, CPU on the pods looked normal, but latency was high, which told me the app was waiting on something. I checked the database and saw DatabaseConnections on RDS was almost at the max.

To mitigate, we scaled down a background worker service that was opening too many connections, and the errors dropped within a few minutes.

For root cause, we found a new batch job had been released earlier that day. It opened a new DB connection for every task and didn't close them. The developers fixed the connection handling, and we added connection pooling.

After the postmortem, I added alerts on DB connections and latency as early warning signals, and we added connection limits for that service. We didn't have that problem again.

What I learned: CPU can look fine while users are suffering, so watch dependencies like the database, and always mitigate first, then find the root cause.

TELL ME ABOUT A TIME YOU COLLABORATED WITH DEVELOPERS

At FitTech, a development team was having frequent deployment failures, and they felt it was an infrastructure problem.

Instead of arguing about whose fault it was, I set up a meeting and we looked at the data together: Argo CD history, pod events, and logs. We found that most failures were from missing config values and wrong resource limits.

We worked together on a fix. I added validation steps in the GitHub Actions pipeline to check Helm values before deploy, and they updated their resource settings. I also wrote a short runbook for them.

Deployment failures dropped a lot, and the relationship between our teams got much better. I learned that good collaboration starts with shared data, not blame.

TELL ME ABOUT A TIME YOU DISAGREED WITH YOUR MANAGER

At FitTech, my manager wanted to deploy a big change to production all at once to save time. I was worried because it touched many services, and rollback would be hard.

I talked to him one on one and explained the risk. I suggested deploying in smaller parts, starting with one low-risk service, using Argo CD for fast rollback. It only added about a day.

He agreed. In the first small deploy, we found a config problem and fixed it in minutes with no customer impact. After that, smaller step-by-step deploys became our normal process.

I learned to disagree with data and a clear solution, and to respect the final decision.

TELL ME ABOUT A FAILURE OR MISTAKE

At FitTech, I added many alerts with low thresholds, thinking more alerts would catch more problems. It created too much noise, and the team started getting alert fatigue.

I owned it, reviewed the alerts with the team, removed the noisy ones, added time windows, and moved to SLO-based alerting on user impact.

I learned that more alerts don't mean better monitoring. Every alert must be actionable.

TELL ME ABOUT A TIME YOU TOOK INITIATIVE

At Home Depot, my team's plan was only to support deployments. But I noticed many incidents were caused by missing health checks.

It wasn't in our plan, but I collected incident data, proposed readiness and liveness probes, tested on one service, showed the results, and then rolled it out with developers. It reduced failed deployments and made releases safer.

I learned: when I see a problem, I collect data, start small, show results, and then get the team on board.

TELL ME ABOUT A TIME YOU HELPED A JUNIOR ENGINEER

At Home Depot, a junior engineer deployed a service to Kubernetes and it wasn't working. I sat with him and we went step by step: get pods showed CrashLoopBackOff, logs showed a missing environment variable because of a typo in the secret name. After fixing it, the app still wasn't reachable, and we found the Service selector didn't match the pod labels.

I didn't just fix it for him. I guided him, and then asked him to write a short runbook. Other new engineers started using it too.

HOW DO YOU HANDLE PRESSURE DURING AN OUTAGE?

I stay calm and focus on facts. First, I check impact and what changed. I communicate clearly in the incident channel, even if it's just "still investigating." I focus on restoring service first, then root cause later. Having good runbooks and dashboards also helps a lot under pressure.

HOW DO YOU PRIORITIZE WHEN MANY THINGS HAPPEN AT ONCE?

Production incidents and user impact always come first. Then I look at risk and urgency: security issues and things that could cause an outage next. Then planned project work. If priorities are not clear, I talk to my manager and the team so we agree on what's most important.

HOW DO YOU KEEP LEARNING?

I learn by doing: I set up small labs to test new tools like Karpenter and KEDA. I read AWS and Kubernetes documentation and blogs, and learn from postmortems at other companies. I also use AI tools to help me learn faster and write scripts.

WHAT ARE YOUR STRENGTHS?

I'm very good at troubleshooting. I stay calm, follow the data, and find the root cause. I also like automation, so I fix problems in a way that prevents them from coming back.

WHAT IS YOUR WEAKNESS?

Earlier in my career, I sometimes tried to fix everything myself instead of asking for help quickly. I learned that during incidents, asking early and sharing information saves time. Now I communicate early and involve the right people quickly.

HOW DO YOU USE AI IN YOUR WORK?

I use AI coding assistants to help write Bash and Python scripts, Terraform, and GitHub Actions workflows faster. I also use them to help explain errors and logs during troubleshooting. But I always review and test the output before using it in production.

WHY THRIVE MARKET?

I like Thrive Market's mission of making healthy and sustainable products more accessible. And technically, this role matches my experience very well: migrating legacy systems to Kubernetes, AWS cost optimization, and improving reliability with better signals and SLOs. I'd love to help the team with those projects.

WHY ARE YOU LOOKING FOR A NEW JOB?

(Choose the one that's true.)
If the contract ended: My contract ended, and I'm looking for a long-term role where I can grow with a strong team.
If still working: My contract is ending soon, and I'm looking for a long-term role where I can grow with a strong team.

SALARY EXPECTATIONS

Based on the role and my experience, I'm targeting a base salary of $145,000 to $160,000. I'm flexible depending on the full compensation package.

WHERE DO YOU SEE YOURSELF IN 3 TO 5 YEARS?

I want to grow into a senior or lead SRE role, where I help design reliable systems, guide other engineers, and drive bigger projects like platform improvements and cost strategy.

DO YOU HAVE ON-CALL EXPERIENCE?

Yes. At FitTech I'm part of the on-call rotation. I respond to alerts, restore service, communicate in the incident channel, and then help with root cause analysis and postmortems. I also work to reduce noisy alerts so on-call is sustainable for the team.

QUESTIONS TO ASK THEM
What would success look like in the first 90 days?
What is the next legacy system or account you plan to migrate?
Are you using KEDA or HPA with custom metrics to scale PHP-FPM?
Do you use burn-rate alerts on your SLOs?
How is on-call organized, and how big is the team?
CLOSING

Thank you for your time today. I'm really excited about this role. The Kubernetes migration, cost work, and reliability improvements are exactly what I enjoy and have done before. I believe I can help the team move faster and make the systems more reliable and cost-efficient. What are the next steps?

Tip: practice each answer out loud 2 or 3 times. Don't memorize word by word, just remember the main steps of each story. If you want, I can quiz you one question at a time.

interview questions and answers based on real interview for Aizada

Aisalkyn Aidarova — Mon, 02 Nov 2026 22:03:29 +0000

TELL ME ABOUT YOURSELF

Hi, thank you for having me. I am a DevOps engineer focused on AWS, Kubernetes and automation. My education is in finance, so I understand budgets, cost and business value. Later I moved into technology because I love solving problems and automating things.

In my work I build and support cloud infrastructure for development teams. I use Terraform to build reusable modules, so teams can create VPCs, EKS clusters, RDS and S3 in a standard and secure way. For delivery, I build CI/CD pipelines with GitHub Actions and Jenkins, and I use Helm and Argo CD for GitOps on EKS. I add security scanning in pipelines. For monitoring I use Prometheus, Grafana, CloudWatch and Datadog. I join production incidents, find root cause and fix it so it does not repeat.

I have AWS Solutions Architect Associate and Terraform Associate certifications. I am excited about SOA because you are moving to Salesforce, building platform engineering and quality engineering, and focusing on safe delivery and reliability. That is exactly the work I enjoy.

DAY TO DAY RESPONSIBILITIES

Morning I check dashboards in Grafana and Datadog, alerts from the night, failed pipelines and open incidents.
Standup with the team.
Work on Jira tickets, like new environments, databases, S3 buckets or access requests. I do all of it with Terraform, never by hand in the console.
Build and improve CI/CD pipelines, help developers when builds or deployments fail.
Kubernetes work, update Helm charts, manage Argo CD apps, tune CPU and memory, HPA, RBAC, EKS upgrades.
Security work, fix scan findings, rotate secrets, review IAM permissions.
Add dashboards and alerts for new services, clean noisy alerts.
Review Terraform and pipeline pull requests from teammates.
On call and incidents, then root cause analysis.
Write runbooks and documentation so developers can self serve.

Short version: half my time is building and automating, half is supporting developers and keeping production healthy.

MAIN PROJECT STORY

I worked on a platform with more than 100 microservices running on Amazon EKS.

Problem: teams deployed in different ways, some with Jenkins scripts, some with kubectl by hand. Deployments took 40 to 45 minutes, failed often, and it was hard to know what version was in production.

What I did: I built EKS clusters with Terraform modules, VPC, node groups, IAM roles and add ons. I created a standard Helm chart template for all services. I set up Argo CD for GitOps, so merge to Git means automatic deploy. I improved pipelines with Docker layer caching, multi stage builds, dependency caching and parallel tests. I added Trivy image scanning and SonarQube. I set up HPA autoscaling, RBAC, and secrets in AWS Secrets Manager. I set up Prometheus, Grafana and alerts.

Result: deployments went from about 40 minutes to about 12 minutes, around 70 percent faster. Fewer failed deployments, rollback became one git revert, and teams could deploy without waiting for DevOps.

My part versus others: I owned the platform and the delivery path. Developers owned their code, Dockerfiles and their Helm values. QA owned testing. Security reviewed scan results and policies.

TECHNICAL QUESTIONS WITH SHORT ANSWERS

AWS

Q: How do you design a highly available app on AWS?
A: VPC with public and private subnets in at least two or three availability zones. ALB in public subnets, apps in private subnets on EKS or EC2 with auto scaling, RDS Multi AZ, NAT gateway for outbound, Route 53 for DNS, backups and CloudWatch alarms.

Q: Difference between security group and NACL?
A: Security group is on the instance level and is stateful, return traffic is allowed automatically. NACL is on the subnet level and is stateless, you must allow both directions. Security groups have only allow rules, NACL has allow and deny.

Q: Public subnet versus private subnet?
A: Public subnet has a route to the internet gateway. Private subnet has no direct internet access, it goes out through a NAT gateway.

Q: How do you give an app in EKS access to S3 safely?
A: IRSA or EKS Pod Identity. The pod gets an IAM role with least privilege, no access keys in code.

Q: How do you reduce AWS cost?
A: Right size instances, auto scaling, savings plans or reserved instances, spot for non critical workloads, delete unused resources, S3 lifecycle rules, and tags to see cost by team. Show reports with Cost Explorer.

TERRAFORM

Q: What is Terraform state and where do you keep it?
A: State is the file that maps Terraform code to real resources. I keep it in S3 with encryption and versioning, and DynamoDB or S3 native locking so two people cannot change it at the same time.

Q: How do you structure Terraform for many environments?
A: Reusable modules for VPC, EKS, RDS, S3. Separate folders or workspaces for dev, stage and prod, each with its own state and variables. Modules are versioned in Git.

Q: What happens if someone changes a resource by hand in the console?
A: That is drift. terraform plan shows it. I either import the change into code or apply to bring it back. To prevent it, limit console write access and run plan regularly in a pipeline.

Q: How do you run Terraform in CI/CD?
A: On pull request, run fmt, validate, tflint, Checkov or tfsec, and plan, and post the plan in the PR. After approval and merge, run apply. Prod needs manual approval.

Q: terraform import, taint, and lifecycle?
A: Import brings an existing resource into state. Taint, now replace flag, forces recreate. Lifecycle lets me use prevent_destroy, create_before_destroy and ignore_changes.

KUBERNETES

Q: Explain Deployment, Service, Ingress.
A: Deployment manages pods and rolling updates. Service gives a stable IP and DNS for pods. Ingress routes outside HTTP traffic to services, on EKS usually with the AWS Load Balancer Controller.

Q: Pod is in CrashLoopBackOff, what do you do?
A: kubectl describe pod to see events, kubectl logs and logs with previous flag. Common causes are wrong config or secret, app error, failed health probe, or OOMKilled because of low memory limit.

Q: Pod is Pending?
A: Usually not enough CPU or memory on nodes, node selector or taints do not match, or PVC not bound. Check describe pod events, then add nodes or fix requests.

Q: Requests versus limits?
A: Request is what the scheduler reserves. Limit is the maximum. If memory goes over the limit, the pod is OOMKilled. CPU over the limit is throttled.

Q: How does HPA work?
A: It watches metrics like CPU or memory from metrics server, or custom metrics, and adds or removes pods to keep the target. For nodes, Cluster Autoscaler or Karpenter adds nodes.

Q: Liveness versus readiness probe?
A: Readiness says if the pod can get traffic. Liveness says if the pod is stuck and should restart.

Q: How do you manage secrets in Kubernetes?
A: Not plain in Git. I use AWS Secrets Manager with External Secrets Operator or Secrets Store CSI driver, and RBAC to limit who can read them.

Q: How do you upgrade EKS?
A: Read release notes and check deprecated APIs, upgrade in dev first, upgrade control plane, then add ons, then node groups with rolling replacement. Use PodDisruptionBudgets so apps stay up.

CI/CD AND GITOPS

Q: Describe a good pipeline.
A: Build, unit tests, code scan, dependency scan, build Docker image, image scan, push to ECR, update Helm values in Git, Argo CD deploys to dev, then integration and smoke tests, approval, then stage and prod.

Q: Blue green versus canary?
A: Blue green runs two full environments and switches all traffic at once, easy rollback. Canary sends a small percent of traffic to the new version first, like 10 percent, watches metrics, then increases. I can do canary with Argo Rollouts.

Q: What is GitOps?
A: Git is the single source of truth. Argo CD watches the repo and syncs the cluster to match Git. Every change has history and rollback is a git revert.

Q: How do you roll back a bad deployment?
A: With Argo CD, revert the commit or roll back to the previous synced version. With Helm, helm rollback. With canary, stop and abort the rollout.

Q: How do you keep secrets safe in GitHub Actions?
A: GitHub secrets or better, OIDC to assume an AWS role, so no long term AWS keys are stored at all.

Q: Have you used Azure DevOps?
A: My main experience is GitHub Actions and Jenkins. Azure DevOps pipelines use the same ideas, YAML stages, jobs, approvals and service connections, so I can pick it up quickly.

SECURITY AND DEVSECOPS

Q: What is SAST, DAST and SCA?
A: SAST scans source code, like SonarQube. DAST tests the running app from outside, like OWASP ZAP. SCA scans third party libraries for known vulnerabilities, like Snyk or Dependabot.

Q: How do you add security to pipelines without slowing teams?
A: Run fast scans on every PR, deeper scans nightly. Block only critical and high issues, report the rest. Give developers clear results in the PR.

Q: How do you handle IAM?
A: Least privilege, roles not users, SSO with MFA, no root usage, regular access review, and CloudTrail for audit.

Q: SSO and federation?
A: Users log in once with an identity provider like Okta or Azure AD, and AWS or apps trust it through SAML or OIDC. In AWS I use IAM Identity Center connected to the identity provider.

OBSERVABILITY AND SRE

Q: Metrics, logs and traces?
A: Metrics show numbers over time like CPU and error rate. Logs show detailed events. Traces show one request across many services. Together they help find problems fast.

Q: What is SLI, SLO, SLA?
A: SLI is the measurement, like percent of successful requests. SLO is our internal target, like 99.9 percent. SLA is the promise to customers with consequences. Error budget is how much failure we allow, if we use it up we slow down releases and fix reliability.

Q: How do you avoid alert fatigue?
A: Alert on symptoms users feel, like error rate and latency, not every CPU spike. Every alert must have an action and a runbook. Route by severity in PagerDuty or Opsgenie.

Q: What are golden signals?
A: Latency, traffic, errors and saturation.

Q: New Relic or Splunk experience?
A: I used Datadog, Prometheus and Grafana. New Relic is the same idea, APM, metrics, logs and alerts, so I can learn it fast.

Q: What is AIOps and how do you use AI?
A: AIOps uses machine learning to find anomalies, group related alerts and predict problems, like Datadog Watchdog or New Relic AI. Personally I use AI tools like Copilot and Claude to write Terraform and scripts faster, explain errors and summarize logs. I always review the output before using it.

SCRIPTING

Q: What do you automate with Bash or Python?
A: Cleanup of old images and snapshots, health checks, log parsing, reports for cost or unused resources, small tools with boto3 for AWS tasks.

Q: Example Python with boto3?
A: A script that lists EC2 instances without an owner tag or stopped for more than 30 days and sends a report, so we can clean up and save cost.

SALESFORCE

Q: How would you support DevOps for Salesforce?
A: Keep Salesforce metadata in Git, use Salesforce CLI in GitHub Actions to validate and deploy to sandboxes, run Apex tests, then deploy to production with approval. Tools like Copado or Gearset also exist. I am honest that my Salesforce experience is limited, but the CI/CD ideas are the same.

SCENARIO AND TROUBLESHOOTING QUESTIONS

Q: Website is slow in production, what do you do?
A: Check dashboards first, latency, error rate, CPU, memory. Check what changed recently, a deployment or config. Look at traces to find the slow service or database query. Check RDS performance and connections. If a recent deployment caused it, roll back first, then investigate.

Q: Deployment worked in dev but fails in prod?
A: Compare config, secrets, IAM permissions, resource limits, network rules and data. Usually it is a missing secret, permission or different environment variable.

Q: A node in EKS is NotReady?
A: Describe the node, check disk pressure, memory pressure, kubelet logs, network plugin. Cordon and drain it, and let the node group replace it.

Q: Traffic spike during exam registration, how do you prepare?
A: Load test before, set HPA and cluster autoscaling, scale RDS or use read replicas, add caching and CDN with CloudFront, set alerts, and have the team ready on call that day.

Q: Someone deleted a production resource by mistake?
A: Restore from backup or snapshot, recreate with Terraform. Then prevent it with prevent_destroy, limited access, and reviews.

Q: How do you do disaster recovery?
A: Define RTO and RPO with business. Backups with AWS Backup, cross region copies, infrastructure as code so we can rebuild fast, and test the restore regularly.

BEHAVIORAL QUESTIONS

Q: Tell me about a production incident you handled.
A: An app started returning errors after a deployment. I joined the call, saw the error rate in Grafana, and found a new release used a wrong secret name. We rolled back with Argo CD in a few minutes. After, I added a check in the pipeline and wrote root cause analysis so it would not happen again.

Q: Tell me about a time you disagreed with a teammate.
A: A developer wanted direct kubectl access in prod to fix things fast. I explained the risk, and we agreed on GitOps plus a break glass process for emergencies. We both got what we needed, speed and safety.

Q: How do you mentor junior engineers?
A: Pair work, give small real tasks, code reviews with explanation, and good documentation. I let them own something so they grow confidence.

Q: Tell me about a tool or process you introduced.
A: Argo CD and GitOps. I started with a pilot team, showed results, wrote guides, did demos, and helped teams move one by one. Most teams moved within a few months.

Q: How do you handle many priorities?
A: Production issues first, then things blocking other teams, then planned work. I talk with my manager when priorities conflict.

Q: Why are you leaving or why this role?
A: I want a senior role with more ownership, platform engineering and mentoring, and a mission I care about. SOA is a strong organization in Chicago and the Salesforce migration is a great challenge.

GAPS AND HOW TO ANSWER HONESTLY

Azure and Azure DevOps: main experience is AWS, but the concepts are the same, I can learn fast.
New Relic and Splunk: used Datadog and Prometheus, same ideas.
PagerDuty and Opsgenie: say what you used for alerting and on call, same process.
HashiCorp Vault: used AWS Secrets Manager, same purpose.
Okta or Azure AD: understand SSO and federation with SAML and OIDC.
Salesforce: limited, but understand CI/CD for it.
Never say I do not know only. Say I have not used that exact tool, but I used this similar one, and here is how I would approach it.

QUESTIONS TO ASK THEM

What does success look like in the first 90 days?
Is your main cloud AWS, Azure, or both?
Do you use GitHub Actions or Azure DevOps today?
How do you deploy Salesforce changes now?
Are you running Kubernetes today, or mostly VMs?
Do you have SLOs and on call set up, and which tools?
How does DevOps work with the cybersecurity and quality engineering teams?
How are you using AI tools today?
What is the biggest pain point right now?
What are the next steps in the process?

Good luck, you are well prepared.

interview questions and answers based on real interview for Jibek

Aisalkyn Aidarova — Mon, 02 Nov 2026 21:59:48 +0000

TELL ME ABOUT YOURSELF

My name is Zhibek. I am a DevOps Engineer with more than 5 years of experience in AWS cloud. I started at Yelp in 2021, where I worked 4 years on high traffic consumer apps. I built AWS infrastructure, ran Kubernetes on EKS, and improved our CI CD pipelines with GitHub Actions. With blue green deployments and automatic rollback, I made deployments 35 percent faster. I also mentored junior engineers and wrote runbooks for the team. Now I work at Zulily, where I build Terraform for dev, staging and production, run EKS with autoscaling, and build secure pipelines with approval steps. I have AWS Solutions Architect Associate and CKAD certifications. Now I am ready for the next step, to lead a team as a player coach, and that is why this role at Herc Rentals is very interesting for me.

DAY TO DAY RESPONSIBILITIES

My day starts with checking CloudWatch dashboards and alerts to see if anything broke at night. Then we have a standup with the team. Most of the day I work on Terraform changes, like new environments, network changes or new services. I review pull requests from other engineers. I help developers when their pipeline fails or their app does not scale well on EKS. I fix production issues when they happen and do root cause analysis after. I also update documents and runbooks so the team knows how to handle things. Sometimes I meet with security or app teams to plan new projects.

TELL ME ABOUT YOUR PROJECT

At Zulily the team needed a private database that is easy to repeat in all environments. I wrote Terraform modules for Amazon DocumentDB, put it in private subnets, and used security groups and IAM so only approved apps on EKS can connect. I added it to GitHub Actions with plan, review and approval before production. I set CloudWatch alarms for CPU, connections and storage. The result was the same setup in dev, staging and production, no public access, and new environments now are built much faster.

Second project if they ask, at Yelp. Deployments were slow and risky. I optimized GitHub Actions with caching and parallel jobs, added blue green deployments, and added automatic rollback if health checks fail. Deployments became 35 percent faster and we had fewer failed releases.

TECHNICAL QUESTIONS AND SHORT ANSWERS

What is Terraform state and why remote state
State is a file where Terraform keeps what it created. I keep it in S3 so the whole team uses the same state, and DynamoDB locks it so two people cannot change at the same time.

How do you structure Terraform for many environments
I use reusable modules and separate folders or workspaces for dev, staging and production, with different variable files. Same code, different values.

What is Terraform drift and how do you handle it
Drift is when someone changes things by hand in the console. I run terraform plan regularly to find it, and then fix it in code, not by hand.

Terraform vs CloudFormation vs Bicep
CloudFormation is only AWS, Bicep is only Azure, Terraform works on many clouds. For a multi cloud company like Herc, Terraform is the best choice for one standard.

What is a landing zone
It is a ready secure base for cloud accounts, with account structure, network, IAM, logging and guardrails. In AWS it is Control Tower and Organizations. New teams get a safe account fast.

How does Kubernetes autoscaling work
HPA adds more pods when CPU or memory is high. Cluster Autoscaler adds more nodes when pods cannot fit. Together they handle traffic changes.

A pod keeps crashing, what do you do
I run kubectl describe pod and kubectl logs. Common reasons are wrong config, missing secret, out of memory, or failing health checks. Then I fix and redeploy.

What is Helm
Helm is a package manager for Kubernetes. It uses templates and values files so we deploy the same app to many environments in a standard way.

Blue green vs canary
Blue green has two full environments and you switch all traffic at once, easy rollback. Canary sends small traffic first to the new version, then more if it is healthy.

How do you design a CI CD pipeline
Build, unit tests, security scan, terraform plan, approval for production, deploy, health check, and automatic rollback if it fails.

How do you keep secrets safe in pipelines
Never put secrets in code. Use AWS Secrets Manager or Parameter Store, and GitHub OIDC to get short time AWS credentials instead of long keys.

Explain VPC design
Public subnets for load balancers, private subnets for apps and databases, NAT gateway for outbound internet, security groups to control traffic, many availability zones for high availability.

Security group vs NACL
Security group works on instance level and is stateful. NACL works on subnet level and is stateless, so you need rules both ways.

What is hybrid connectivity
Connecting on premise data center to the cloud with Site to Site VPN or Direct Connect in AWS, ExpressRoute in Azure, FastConnect in OCI. Transit Gateway helps connect many VPCs.

What is IAM least privilege
Give only the permissions needed, nothing more. Use roles, not users with long keys, and review permissions often.

How do you do disaster recovery
First know RTO and RPO, how fast to recover and how much data you can lose. Then backups, multi AZ, cross region copies, and test the recovery regularly, not only on paper.

How do you reduce cloud cost
Right size instances, autoscaling, Savings Plans or Reserved Instances, spot for non critical work, delete unused resources, tagging for each team, and monthly cost reports to leadership.

How do you handle a production outage
First stop the impact, maybe rollback. Communicate to stakeholders. Find root cause with logs and metrics. After that write a postmortem with no blame and fix so it does not happen again.

Do you have Azure or OCI experience
Be honest. My deep experience is AWS, but the concepts are the same. VPC is VNet, IAM is Entra ID and RBAC, EKS is AKS or OKE. Terraform works on all of them, so I can learn fast. I already started learning Azure basics.

How do you use AI in your work
I use Claude AI to help draft pipelines, review Terraform and write documentation. I always review and test before using it. I think AI can help the team move faster when used carefully.

MANAGER QUESTIONS

How do you handle a low performer
Talk early in one on one, understand the reason, set clear goals with time, give support, and follow up. If no change, work with HR.

How do you prioritize when everything is urgent
Production issues and security first, then business priorities agreed with leadership. I make the work visible so everyone sees the tradeoffs.

How do you manage contractors
Clear scope, same standards and code review as full time, and documentation so knowledge stays in the company.

Two engineers disagree on a design, what do you do
Let both explain, compare by facts like cost, risk and simplicity, maybe do a small test, then decide and move forward together.

What will you do in first 90 days
First 30 days listen and learn the team, systems and pain points. Next 30 days find quick wins like standards and runbooks. Last 30 days make a roadmap with leadership.

How do you measure team success
Deployment frequency, failed changes, time to recover, uptime, cloud cost, and team happiness.

2-2 The Midnight Payment Processing Incident — Kubernetes, PostgreSQL, Terraform, CI/CD, Grafana and PagerDuty

Aisalkyn Aidarova — Fri, 30 Oct 2026 02:04:16 +0000

Student hands-on guide · 3 classes × 3 hours · AWS EKS training environment

Your mission

You are a DevOps/SRE engineer supporting a banking payment-processing platform. At midnight, PagerDuty notifies the primary on-call engineer that payment transactions are failing.

You open Grafana and see increasing HTTP 503 errors. But when you inspect Kubernetes, the application Pods are all Running and Ready.

How can the application be healthy according to Kubernetes while customers cannot complete payments?

You will investigate the application, its PostgreSQL database, Kubernetes networking, deployment history, monitoring, and the infrastructure changes that could have caused the incident.

This is a realistic fictional production case study, supported by a real, runnable training lab. You must not claim that it is an actual incident at a named bank.

1. Incident ticket

INC-2026-1047

SEV-1 · Scenario

Payment Processing API — Elevated 5xx Error Rate

PagerDuty · Payments Production · 12:05 AM

Reported impact: Payment submissions failing across the customer-facing banking application.

Monitoring: HTTP 503 responses increasing, payment success rate declining, API Pods still Ready.

Initial hypothesis: Application dependency failure, database connectivity, or recent infrastructure change.

Assigned: Primary On-Call SRE

Escalation: Incident Commander, Backend Engineering, DBA, Cloud Platform, Release Engineering, Security.

In the hypothetical production architecture, the application runs on AWS EKS across three Availability Zones and connects to Amazon RDS PostgreSQL. Terraform manages security groups and networking; GitHub Actions handles application and infrastructure pipelines.

For the actual student lab, we use two Payment API replicas and PostgreSQL inside Kubernetes to keep costs and resource requirements low.

2. Production architecture and team responsibilities

Reference production architecture. The classroom lab replaces RDS and AWS security-group failure with an in-cluster database and NetworkPolicy.

Team	Responsibility during incident
Primary On-Call SRE	Acknowledge alert, assess impact, investigate, mitigate and communicate
Incident Commander	Coordinate teams, severity, updates and escalation
Backend/Application Team	Investigate application errors, connection pooling, configuration and code
Database Administrator	Check database availability, active connections, locks, CPU, I/O and query performance
Cloud/Platform Team	Investigate EKS, VPC, security groups, routing and network policies
Release/CI/CD Team	Check recent releases, GitHub Actions workflows and Terraform changes
Security Team	Review and approve necessary access-rule changes

A primary on-call engineer may initially perform several of these functions, especially on a small team. Do not wait for a meeting before checking the impact and following your incident-response runbook.

3. What you will build

The lab creates:

Namespace payments-incident-lab
PostgreSQL database with a simulated_payments table
Python Payment API with two replicas
Kubernetes Services and health probes
Prometheus metrics for requests, errors and latency
A controlled database connectivity incident
Recovery and verification procedures

The lab uses synthetic payments only. It does not charge cards or connect to real payment processors.

4. Requirements

You need a training EKS cluster, working kubectl, Docker with permission to push images to a registry, and enough capacity for three small Pods in addition to monitoring.

For the NetworkPolicy incident, the cluster must have NetworkPolicy enforcement enabled. Installing a NetworkPolicy resource without an enforcing CNI will not necessarily block traffic.

First, verify your environment

kubectl config current-context
kubectl get nodes -o wide
kubectl get namespaces
kubectl get pods -n monitoring
kubectl get storageclass

Do not perform this exercise in a real production namespace.

Part A — Create all application and Kubernetes files

Create the project folders:

mkdir -p payment-incident-lab/{app,k8s,scripts,grafana,terraform}
cd payment-incident-lab

File 1 — `app/requirements.txt`

flask==3.1.1
psycopg[binary]==3.2.9
prometheus-client==0.22.1

File 2 — `app/app.py`

This is the application students will troubleshoot. Notice that /readyz checks the application process, but deliberately does not check database connectivity. This allows Pods to remain Ready while payment requests fail.

import os        try:            with connect() as conn:                with conn.cursor() as cur:                    cur.execute("SELECT 1")                    cur.fetchone()            REQ.labels("/api/db-check", "200").inc()            return jsonify(database="reachable"), 200        except Exception as exc:            app.logger.error(                "database connectivity failed: %s",                type(exc).__name__,            )            DB_ERRORS.inc()            REQ.labels("/api/db-check", "503").inc()            return jsonify(error="database_unavailable"), 503@app.post("/api/payments")def payment():    with DURATION.labels("/api/payments").time():        data = request.get_json(silent=True) or {}        key = request.headers.get("Idempotency-Key", "").strip()        if (            not key            or len(key) > 128            or not isinstance(data.get("amount_cents"), int)            or isinstance(data.get("amount_cents"), bool)            or not 1 <= data["amount_cents"] <= 100000        ):            REQ.labels("/api/payments", "400").inc()            return jsonify(                error="Idempotency-Key and integer "                      "amount_cents 1..100000 required"            ), 400        try:            with connect() as conn:                with conn.cursor() as cur:                    cur.execute(                        """                        INSERT INTO simulated_payments                        (idempotency_key, amount_cents)                        VALUES (%s, %s)                        ON CONFLICT (idempotency_key)                        DO NOTHING                        """,                        (key, data["amount_cents"]),                    )                    cur.execute(                        """                        SELECT id, amount_cents, status                        FROM simulated_payments                        WHERE idempotency_key = %s                        """,                        (key,),                    )                    row = cur.fetchone()

File 3 — `app/Dockerfile`

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

RUN useradd --uid 10001 --create-home appuser

USER 10001

EXPOSE 8080

CMD ["python", "app.py"]

File 4 — `k8s/00-namespace.yaml`

apiVersion: v1
kind: Namespace
metadata:
  name: payments-incident-lab

File 5 — `k8s/01-postgres.yaml`

This database is disposable. Its storage uses emptyDir, so deleting or replacing the database Pod can destroy its data.

apiVersion: v1
kind: Secret
metadata:
  name: postgres-auth
  namespace: payments-incident-lab
type: Opaque
stringData:
  POSTGRES_PASSWORD: classroom-only-not-production
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-init
  namespace: payments-incident-lab
data:
  001-init.sql: |
    CREATE TABLE IF NOT EXISTS simulated_payments (
      id BIGSERIAL PRIMARY KEY,
      idempotency_key TEXT NOT NULL UNIQUE,
      amount_cents INTEGER NOT NULL
        CHECK (amount_cents > 0),
      status TEXT NOT NULL
        DEFAULT 'simulated_approved',
      created_at TIMESTAMPTZ NOT NULL DEFAULT now()
    );
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: lab-postgres
  template:
    metadata:
      labels:
        app: lab-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: payments
            - name: POSTGRES_USER
              value: labuser
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-auth
                  key: POSTGRES_PASSWORD
          volumeMounts:
            - name: init
              mountPath: /docker-entrypoint-initdb.d
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U labuser -d payments
            periodSeconds: 5
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              memory: 512Mi
      volumes:
        - name: init
          configMap:
            name: postgres-init
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  selector:
    app: lab-postgres
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

File 6 — `k8s/02-payment-api.yaml`

Replace REPLACE_WITH_YOUR_IMAGE with the container image you build.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: payments-incident-lab
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      containers:
        - name: payment-api
          image: REPLACE_WITH_YOUR_IMAGE
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: DB_HOST
              value: lab-postgres
            - name: DB_NAME
              value: payments
            - name: DB_USER
              value: labuser
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-auth
                  key: POSTGRES_PASSWORD
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments-incident-lab
  labels:
    app: payment-api
spec:
  selector:
    app: payment-api
  ports:
    - name: http
      port: 8080
      targetPort: http

File 7 — `k8s/03-networkpolicy-outage.yaml`

This is the controlled failure. It blocks new incoming connections to the PostgreSQL Pod from the API.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-postgres-ingress
  namespace: payments-incident-lab
spec:
  podSelector:
    matchLabels:
      app: lab-postgres
  policyTypes:
    - Ingress
  ingress: []

Important: This only works when your cluster enforces Kubernetes NetworkPolicy. On EKS, check the Amazon VPC CNI NetworkPolicy configuration before running this exercise.

File 8 — `k8s/04-servicemonitor-optional.yaml`

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-api
  namespace: monitoring
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - payments-incident-lab
  selector:
    matchLabels:
      app: payment-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

The release: monitoring label must match the selector of your Prometheus installation. This file requires the Prometheus Operator ServiceMonitor CRD.

Part B — Build, deploy and verify

Step 1. Build and push the Docker image

Run from inside payment-incident-lab.

Replace the Docker Hub username with yours.

export IMAGE='YOUR_DOCKERHUB_USERNAME/jumptotech-payment-api:v1'

docker buildx build \
  --platform linux/amd64 \
  -t "$IMAGE" \
  --push ./app

If your EKS worker nodes are ARM64, use linux/arm64 instead.

Step 2. Deploy PostgreSQL

kubectl apply -f k8s/00-namespace.yaml

kubectl apply -f k8s/01-postgres.yaml

kubectl rollout status deployment/lab-postgres \
  -n payments-incident-lab \
  --timeout=180s

Check:

kubectl get pods -n payments-incident-lab
kubectl get svc -n payments-incident-lab

The PostgreSQL Pod should show 1/1 Running.

Step 3. Deploy Payment API

sed "s|REPLACE_WITH_YOUR_IMAGE|$IMAGE|" \
  k8s/02-payment-api.yaml | kubectl apply -f -

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=180s

kubectl get deployments -n payments-incident-lab
kubectl get pods -n payments-incident-lab -o wide

Expected baseline:

NAME           READY   UP-TO-DATE   AVAILABLE
lab-postgres   1/1     1            1
payment-api    2/2     2            2

Step 4. Verify the database table

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c '\dt'

You should see the simulated_payments table.

Check the number of synthetic payments:

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c 'SELECT COUNT(*) FROM simulated_payments;'

Step 5. Test the application

Create scripts/02-smoke-test.sh:

#!/usr/bin/env bash
set -euo pipefail

NS=payments-incident-lab

kubectl run "smoke-$(date +%s)" \
  -n "$NS" \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- sh -c '
    echo "READINESS:"
    curl -sS -i --max-time 10 \
      http://payment-api:8080/readyz

    echo
    echo "DB CHECK:"
    curl -sS -i --max-time 10 \
      http://payment-api:8080/api/db-check

    echo
    echo "SYNTHETIC PAYMENT:"
    curl -sS -i --max-time 10 \
      -X POST \
      http://payment-api:8080/api/payments \
      -H "Content-Type: application/json" \
      -H "Idempotency-Key: classroom-$(date +%s)" \
      -d "{\"amount_cents\":1250}"
'

Run:

bash scripts/02-smoke-test.sh

Expected results:

Endpoint	Expected HTTP status
`/readyz`	200
`/api/db-check`	200
`/api/payments`	200

The temporary curl Pod may display an attach warning after completing its request. Focus on the actual HTTP response.

Part C — Configure Grafana and Prometheus

First verify the monitoring components:

kubectl get pods -n monitoring

kubectl get crd servicemonitors.monitoring.coreos.com

kubectl get prometheus -n monitoring

Inspect the ServiceMonitor selectors:

kubectl get prometheus -n monitoring -o yaml \
  | grep -A12 serviceMonitorSelector

If the ServiceMonitor label matches your Prometheus configuration:

kubectl apply -f k8s/04-servicemonitor-optional.yaml

kubectl get servicemonitor -n monitoring

In Grafana, select your Prometheus data source and create the following panels.

Panel 1 — Payment API request rate by HTTP status

sum by (status) (
  rate(payment_http_requests_total[5m])
)

This distinguishes successful requests from HTTP 400, 409 and 503 responses.

Panel 2 — HTTP 5xx error rate

sum(
  rate(payment_http_requests_total{status=~"5.."}[5m])
) or vector(0)

Panel 3 — Database errors over five minutes

sum(
  increase(payment_db_errors_total[5m])
) or vector(0)

Panel 4 — Available replicas

kube_deployment_status_replicas_available{
  namespace="payments-incident-lab",
  deployment="payment-api"
}

Panel 5 — Desired replicas

kube_deployment_spec_replicas{
  namespace="payments-incident-lab",
  deployment="payment-api"
}

Panel 6 — Pod readiness

sum by (condition) (
  kube_pod_status_ready{
    namespace="payments-incident-lab",
    pod=~"payment-api-.*"
  }
)

Panel 7 — Payment request p95 latency

histogram_quantile(
  0.95,
  sum by (le) (
    rate(
      payment_http_request_duration_seconds_bucket{
        endpoint="/api/payments"
      }[5m]
    )
  )
)

Generate test traffic before expecting request metrics to appear. Check that Prometheus is actually scraping the Payment API target. The metrics may remain absent if the ServiceMonitor selector is incorrect.

Key learning point

A dashboard can show:

Available Pods

2 / 2

Kubernetes sees healthy processes

Payment API

503

Database-dependent requests fail

This is not contradictory. Kubernetes readiness and application business functionality are different signals.

Part D — Create the incident

Class 2: Midnight outage simulation

Your instructor announces:

It is 12:05 AM. PagerDuty has triggered a critical alert. Customers cannot complete payment transactions. Grafana shows elevated HTTP 503 errors, but all Payment API Pods appear healthy. You are the primary on-call engineer. Investigate the incident and identify the failing dependency.

Step 1. Confirm the healthy baseline

kubectl get pods -n payments-incident-lab

bash scripts/02-smoke-test.sh

Do not continue until the database check and payment request return HTTP 200.

Step 2. Introduce the controlled failure

Create scripts/03-inject-outage.sh:

#!/usr/bin/env bash
set -euo pipefail

kubectl apply -f k8s/03-networkpolicy-outage.yaml

kubectl get networkpolicy \
  -n payments-incident-lab

printf '\nRun the smoke test and investigate.\n'

Run:

bash scripts/03-inject-outage.sh

Then:

bash scripts/02-smoke-test.sh

When NetworkPolicy enforcement is active, expected behavior is:

Component	State
Payment API Pods	Running, Ready
PostgreSQL Pod	Running, Ready
API readiness endpoint	HTTP 200
Database connectivity endpoint	HTTP 503
Synthetic payment request	HTTP 503
Application logs	Database connectivity errors

If requests still succeed, verify that NetworkPolicy enforcement is enabled. Do not assume the incident is active simply because kubectl apply succeeded.

Step 3. Investigate the incident

Start with the Kubernetes resources:

kubectl get pods -n payments-incident-lab -o wide

kubectl get deployments -n payments-incident-lab

kubectl get svc -n payments-incident-lab

kubectl get endpointslices -n payments-incident-lab

Inspect the API Deployment:

kubectl describe deployment payment-api \
  -n payments-incident-lab

Inspect the API Pods:

kubectl describe pods \
  -l app=payment-api \
  -n payments-incident-lab

Inspect application logs:

kubectl logs deployment/payment-api \
  -n payments-incident-lab \
  --tail=100 \
  --max-log-requests=10

Look for errors indicating the application cannot communicate with PostgreSQL.

Inspect the database:

kubectl get pods \
  -l app=lab-postgres \
  -n payments-incident-lab

kubectl logs deployment/lab-postgres \
  -n payments-incident-lab \
  --tail=100

Check the NetworkPolicies:

kubectl get networkpolicy \
  -n payments-incident-lab

kubectl describe networkpolicy block-postgres-ingress \
  -n payments-incident-lab

kubectl get networkpolicy \
  -n payments-incident-lab \
  -o yaml

Check rollout history:

kubectl rollout history deployment/payment-api \
  -n payments-incident-lab

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab

Important: In this particular incident, the API Deployment may have had no recent rollout at all. The problem is a networking configuration change, not necessarily a new container version.

Student investigation script

Create scripts/04-investigate.sh:

#!/usr/bin/env bash
set -euo pipefail

NS=payments-incident-lab

kubectl get pods,deploy,svc,endpointslices -n "$NS"

kubectl describe deployment payment-api -n "$NS"

kubectl logs deployment/payment-api \
  -n "$NS" \
  --tail=60 \
  --max-log-requests=10

kubectl get networkpolicy -n "$NS" -o yaml

kubectl rollout history deployment/payment-api -n "$NS"

Run:

bash scripts/04-investigate.sh

Root-cause analysis

In this lab, the NetworkPolicy selects the PostgreSQL Pod and allows no incoming traffic. The application can still start and answer /readyz, but new PostgreSQL connections fail.

The root cause is therefore database connectivity blocked by Kubernetes NetworkPolicy, not a crashed application Pod.

A blind kubectl rollout undo of the Payment API is not an appropriate fix because the network policy would remain in effect.

Part E — Restore service

After identifying the policy and following the classroom change procedure, remove the offending rule.

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Check that the application is still available:

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=120s

Run the payment tests again:

bash scripts/02-smoke-test.sh

Verify the database contains the successful synthetic payments:

kubectl exec -n payments-incident-lab \
  deployment/lab-postgres -- \
  psql -U labuser -d payments \
  -c 'SELECT id, amount_cents, status, created_at
      FROM simulated_payments
      ORDER BY id DESC
      LIMIT 10;'

Create scripts/05-recover.sh:

#!/usr/bin/env bash
set -euo pipefail

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab \
  --ignore-not-found

kubectl rollout status deployment/payment-api \
  -n payments-incident-lab \
  --timeout=120s

bash scripts/02-smoke-test.sh

Run:

bash scripts/05-recover.sh

Recovery is verified when new database checks and synthetic payment requests succeed. In a real payment system, the application and database teams would also need to verify transaction integrity, retries and reconciliation.

Part F — Terraform and AWS RDS investigation

Why Terraform matters

In a production environment, PostgreSQL may run on Amazon RDS rather than inside Kubernetes.

Suppose a GitHub Actions workflow applies a Terraform change that accidentally removes the security-group rule allowing the EKS application to reach RDS on TCP port 5432.

The symptoms could resemble our lab: Kubernetes Pods are Ready, but application requests fail because database connections cannot be established.

The two failure mechanisms are different:

Classroom lab	Production reference scenario
Kubernetes NetworkPolicy blocks PostgreSQL ingress	AWS security group blocks EKS-to-RDS traffic
PostgreSQL runs inside EKS	PostgreSQL runs on Amazon RDS
Fix the Kubernetes NetworkPolicy	Restore the correct AWS security-group rule
Investigate `kubectl get networkpolicy`	Investigate AWS VPC, security groups, CloudTrail and Terraform changes

File 9 — `terraform/EXAMPLE_ONLY.tf`

This is a teaching example. Do not run terraform apply on this file against an existing AWS environment. The correct security group and identity must first be established.

variable "rds_security_group_id" {
  type = string
}

variable "approved_eks_application_security_group_id" {
  type = string
}

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id = var.rds_security_group_id

  referenced_security_group_id = (
    var.approved_eks_application_security_group_id
  )

  ip_protocol = "tcp"

  from_port = 5432
  to_port   = 5432

  description = "Approved EKS application traffic to PostgreSQL"
}

Terraform investigation commands

For an actual managed Terraform project, first identify the correct repository, workspace, state and pipeline.

terraform version

terraform workspace show

terraform state list

terraform plan

Review the plan and the last GitHub Actions run. Determine whether a rule was removed, replaced, or changed.

Do not assume that the latest Terraform deployment caused the incident merely because it happened recently. Correlate the change timestamp with application errors, RDS connection failures, AWS configuration history and network evidence.

CI/CD investigation

Students should be able to explain this workflow:

Developer creates pull request

GitHub Actions: test, build, security checks

Terraform fmt, validate, plan and review

Approved infrastructure change

Application deployment and post-deploy tests

Prometheus / Grafana validation and alerting

An important production lesson: a pipeline can finish successfully while the system is broken. Successful execution of terraform apply only proves Terraform applied the requested changes; it does not prove payment processing still works.

Part G — Three-class teaching schedule

Class	Duration	Activities
Class 1	3 hours	Architecture, team roles, Docker, PostgreSQL, EKS Deployment, Services, API testing, Grafana
Class 2	3 hours	PagerDuty ticket, incident injection, logs, database investigation, NetworkPolicy, Terraform and CI/CD analysis
Class 3	3 hours	Recovery, verification, incident report, prevention, mock interview and follow-up questions

Class 1 — Build the platform

Students must finish with a working Payment API, two healthy replicas, PostgreSQL connectivity, successful synthetic payments and visible metrics.

Instructor questions: What is the difference between a Deployment and a Service? Why do we use a Secret? What does a readiness probe check? How does the API connect to PostgreSQL? Why might database availability differ from Pod readiness?

Class 2 — Investigate the outage

Do not immediately tell students the root cause. Present the incident ticket and ask them to gather evidence.

Students should identify that the Pods are Ready, the database Pod is running, the API reports database connectivity errors, and a NetworkPolicy is blocking the connection.

Instructor questions: Is this an application crash? Why are there no container restarts? Why does the HTTP 503 error appear? Why would restarting the Pods not fix the problem? What evidence would connect this to a Terraform change in AWS?

Class 3 — Recover and explain

Students must explain the root cause, restore connectivity, validate synthetic payments, review Grafana, and present an incident report.

They should distinguish between mitigation (restoring service), root cause (identifying the failing rule), and prevention (adding change controls and connectivity tests).

Part H — Interview preparation

These are the questions students should practice aloud, with technically accurate answers.

1. Your Pods are Running and Ready, but customers receive HTTP 503. What do you investigate?

Sample answer:

I would first assess the customer impact using Grafana or another monitoring tool. I would check the HTTP 5xx error rate, latency, and affected endpoints. Then I would inspect the Kubernetes Deployments, Pods, Services and application logs. Since the Pods are Ready, I would not assume Kubernetes itself is healthy end to end. I would investigate downstream dependencies such as the database, networking, authentication and external APIs.

2. How can all Pods be Ready while payment processing is failing?

Kubernetes readiness only checks what the configured readiness probe verifies. If the probe checks only the application process, the Pod can remain Ready even when the database is unreachable. That's why I would also monitor transaction success rates and perform synthetic end-to-end checks.

3. What would you check first to determine whether the database is the problem?

I would examine the application logs for connection timeouts, authentication failures, connection-pool exhaustion and query errors. Then I would check database health and connectivity. I would also review database metrics such as CPU, memory, connections, I/O and wait events.

4. How would you prove that a Terraform change caused the outage?

I would correlate the incident start time with the Terraform pipeline execution. Then I would review the Terraform plan, the applied changes and AWS configuration history. If the change removed a required RDS security-group rule, I would verify that the affected application could no longer reach the database. I would use that evidence to establish causation rather than relying only on timing.

5. Would you immediately run `kubectl rollout undo`?

Not automatically. I would first determine whether the application deployment caused the failure. If the root cause is a network rule or database configuration, rolling back the application might not help. I would follow the incident runbook and select the safest mitigation supported by the evidence.

6. What is the difference between HTTP 502, 503 and 504?

Code	Meaning	Common example
502 Bad Gateway	Gateway received an invalid response from an upstream server	Reverse proxy cannot obtain a valid upstream response
503 Service Unavailable	Service cannot currently handle the request	Payment API deliberately returns 503 when database is unavailable
504 Gateway Timeout	Gateway did not receive an upstream response in time	ALB or proxy times out waiting for an application response

A database timeout does not automatically mean the customer receives 504. In this lab, the application catches the error and explicitly returns 503.

7. Who would you involve in a SEV-1 incident?

I would follow the company's incident-response procedure. Depending on the impact, I would involve the incident commander, application engineers, database team, platform engineers and release team. I would communicate the symptoms, current impact, evidence collected, and mitigation plan so the teams could work in parallel.

8. How would you investigate a delayed failure after a successful release?

I would compare the release timestamp with the first customer errors and infrastructure changes. I would investigate whether the failure was triggered by connection-pool exhaustion, database load, expired credentials, a scheduled job, network changes or traffic growth. I would also check whether the application had existing connections that continued working temporarily before new connections started failing.

9. What database metrics would you inspect?

I would review CPU utilization, available memory, connection counts, disk I/O, read and write latency, active sessions, slow queries, locks and wait events. I would use those metrics to distinguish between database overload, slow queries, connection exhaustion and network connectivity failures.

10. What is a database connection pool?

A connection pool maintains reusable database connections so an application doesn't need to establish a new connection for every request. If the pool is exhausted or connections become invalid, requests may wait, time out or fail. I would check active connections, pool utilization, wait time and timeout settings.

11. How would you restore service without creating additional risk?

I would first identify the failing dependency and the last known good configuration. I would coordinate the mitigation through the incident process, restore only the necessary access, and verify recovery using application requests, database checks, monitoring and synthetic transactions. I would avoid broad security-group changes or unnecessary restarts.

12. What would you do after recovery?

I would confirm that error rates and payment success metrics returned to baseline. Then I would document the incident timeline, impact, root cause, mitigation and prevention actions. I would recommend post-deployment database connectivity checks, better dependency monitoring, Terraform plan review and appropriate change approvals.

Part I — Student incident report

Every student must complete this report after Class 3.

Incident report template

Incident ID: INC-2026-1047

Severity: SEV-1 (training scenario)

Affected service: Payment Processing API

Detection: PagerDuty / Grafana

Start time: Record actual lab injection time

Recovery time: Record actual successful verification time

Customer impact: Synthetic payment requests failed in the lab; production customer impact is hypothetical

Observed symptoms: API Pods Ready, database-dependent requests returned HTTP 503

Root cause: PostgreSQL ingress blocked by Kubernetes NetworkPolicy

Mitigation: Removed the offending policy in the isolated lab

Verification: Database checks and synthetic payments returned HTTP 200

Prevention: Network change reviews, dependency checks, monitoring, and automated connectivity tests

Evidence: Attach command outputs, relevant logs, Grafana screenshots and the exact configuration change.

File 10 — `scripts/01-check.sh`

#!/usr/bin/env bash
set -euo pipefail

kubectl get nodes
kubectl get ns
kubectl get pods -n monitoring

kubectl get pods,deploy,svc \
  -n payments-incident-lab

kubectl get networkpolicy \
  -n payments-incident-lab

File 11 — `scripts/06-cleanup.sh`

Use only after the classes are finished. This deletes the entire training namespace and its synthetic database.

#!/usr/bin/env bash
set -euo pipefail

read -r -p \
  'Delete ENTIRE payments-incident-lab namespace? Type DELETE: ' \
  answer

if [[ "$answer" == "DELETE" ]]; then
  kubectl delete namespace payments-incident-lab
else
  echo "Canceled."
fi

To clean up the optional ServiceMonitor separately:

kubectl delete servicemonitor payment-api \
  -n monitoring \
  --ignore-not-found

Final student challenge

Your instructor will ask:

"It is 12:05 AM. Your Payment API has two healthy Pods, PostgreSQL is Running, and your application deployment has not changed. However, payment requests are returning HTTP 503. What are your first actions, how do you identify the root cause, who do you involve, and how do you restore service safely?"

You must be able to explain the investigation in your own words, demonstrate the relevant commands, identify the exact failed dependency, and verify the recovery.

Your goal is not to memorize commands. Your goal is to think and communicate like a DevOps/SRE engineer responding to a production incident.

JumpToTech DevOps Engineering · Practical Incident Response Training

JumpToTech DevOps Production Incident Lab #2 The Midnight Payment Processing Incident

Aisalkyn Aidarova — Fri, 30 Oct 2026 01:05:13 +0000

Student training handbook | 3 classes × 3 hours | AWS EKS · Terraform · GitHub Actions · PostgreSQL RDS · Prometheus · Grafana · PagerDuty

Students, welcome to your next production incident investigation.

In our previous incident, you investigated Kubernetes Pods that became unavailable after a deployment. This time, the challenge is different.

All Kubernetes Pods are healthy. The deployment succeeded. The application is reachable. But customers cannot complete payments.

This is the type of scenario that tests whether you understand the entire production system—not just Kubernetes commands.

You will work as an on-call DevOps/SRE engineer at a fictional enterprise payment platform. The scenario is designed to resemble incidents that could occur in large financial technology companies, banks, and e-commerce organizations. All company details, ticket numbers, figures, and incident events below are fictional training data.

By the end of three classes, you should be able to investigate a multi-system incident, identify an infrastructure change, coordinate with other engineering teams, restore service safely, explain database connectivity, and defend your decisions during a technical interview.

INC-2026-1047 — Payment Authorization Failures

SEV-1

Payment success rate

41%

Baseline: 99.95%

HTTP 5xx rate

37%

On payment API

Kubernetes Pods

12/12

Ready and Running

Database

Available

RDS instance is running

Incident clue: A Terraform deployment completed approximately 20 minutes before the payment failures began.

The critical interview question is:

“If all Kubernetes Pods are healthy and the RDS database is available, why are customers receiving payment failures?”

You are not allowed to answer, “I would restart the Pods,” without first explaining what evidence justifies that action.

Your three-class mission

Class	Duration	Your objective
Class 1 — Detection and triage	3 hours	Understand the ticket, production architecture, incident roles, and evidence collection
Class 2 — Deep troubleshooting	3 hours	Investigate Kubernetes, CI/CD, Terraform, networking, and PostgreSQL; identify the root cause
Class 3 — Recovery and interviews	3 hours	Restore service, verify transactions, complete the postmortem, and pass a mock technical interview

You will not be given the root cause at the beginning. Your instructor will reveal additional evidence as you progress.

1. The company and production environment

You work for NorthStar Payments, a fictional payment technology company processing transactions for online retailers.

NorthStar provides payment authorization, order management, transaction history, refunds, and customer account services.

The production system runs on AWS. The platform team manages infrastructure through Terraform, developers deploy applications through GitHub Actions, and SRE engineers monitor the platform using Prometheus, Grafana, CloudWatch, and PagerDuty.

Production architecture

Conceptual production architecture. AWS networking and service placement are simplified for training.

The system has these components:

Component	Purpose	Responsible team
Amazon EKS	Runs containerized services	Platform / SRE
Payment API	Receives and processes payment requests	Backend application team
Payment workers	Process asynchronous events	Backend application team
Amazon RDS PostgreSQL	Stores transaction and payment records	Database / DBA team
AWS Security Groups	Control permitted network connections	Cloud platform / security
Terraform	Provisions and changes AWS infrastructure	Cloud platform
GitHub Actions	Tests and deploys infrastructure and applications	DevOps / release engineering
Prometheus and Grafana	Collect and visualize service metrics	SRE / observability
CloudWatch	AWS infrastructure logs and metrics	Cloud platform / SRE
PagerDuty	Routes urgent alerts to on-call responders	SRE / incident management
Jira Service Management	Tracks incident tickets and investigation	Service management

Example workload sizes

For this case study, the production environment has 12 Payment API Pods, 6 background workers, 3 Availability Zones, and one Multi-AZ PostgreSQL RDS deployment. These are illustrative values, not actual figures from any bank.

The Payment API has a readiness endpoint at /ready, which checks whether the HTTP process is responsive. It does not check whether the application can reach PostgreSQL. That design choice will become important.

2. Your production engineering team

An enterprise incident is not normally handled by one engineer working alone. You may be the first person paged, but different teams own different systems.

Incident response team structure

Incident Commander

Coordinates response and decisions

Primary SRE

Triage, metrics, Kubernetes

Platform Engineer

Terraform, VPC, security groups

Backend Engineer

Payment API, application logs

Database Engineer

RDS, SQL, connection health

Release Engineer

CI/CD, change history

Security Engineer

Network access, policy review

Customer support and business stakeholders receive coordinated status updates through the incident communication process.

Who is responsible for what?

Primary on-call DevOps/SRE engineer — your role

You acknowledge the alert, establish the initial technical picture, identify impacted components, collect evidence, coordinate with the incident team, and execute approved operational actions within your access and authority.

Incident Commander

Owns coordination, severity management, escalation, communications, and ensuring someone is assigned to each workstream. The incident commander does not necessarily perform the technical fix.

Application engineer

Investigates payment request failures, database client errors, connection pool behavior, retry logic, and application code.

Cloud platform engineer

Investigates AWS networking, Terraform state and plans, security groups, IAM, EKS networking, and infrastructure changes. This engineer normally owns the corrective Terraform pull request.

Database engineer

Checks database availability, active connections, PostgreSQL errors, query performance, locks, replication or failover state, and database-side access settings.

Release engineer

Finds the GitHub Actions deployment run, commit SHA, pull request, approvers, change window, and any simultaneous releases.

Security engineer

Reviews whether a proposed security group change is permitted and ensures recovery does not introduce unnecessary network exposure.

Customer support and communications

Collect customer reports and distribute approved incident updates. They should not receive raw database credentials or sensitive customer records.

Interview question

“If you are the primary on-call engineer, are you personally responsible for fixing every component?”

A strong answer:

“I am responsible for responding to the incident, understanding the impact, and driving technical triage within my role. I would engage the correct service owners for specialized components such as RDS, networking, and application code. We work together under the incident response process to restore service safely.”

3. Where the production ticket comes from

In this scenario, Prometheus detects the elevated error rate and sends the alert to Alertmanager. Alertmanager routes it to PagerDuty. PagerDuty pages the primary on-call engineer. An incident ticket is then created in Jira Service Management through the company's configured integration or incident workflow.

PagerDuty — Production Alert

12:15 AM · Payment Platform · Primary On-Call

High Payment Failure Rate

Service: payment-api-prod

Environment: production

Cluster: northstar-prod-eks

Namespace: payments-prod

Trigger: HTTP 5xx rate exceeds the configured threshold for 5 minutes.

Customer payment authorization is degraded.

Example Jira incident ticket

Jira Service Management · INC-2026-1047

Payment authorization failures — customer impact

Status	Investigating
Assigned group	Production SRE
First detected	12:15 AM
Customer symptoms	Payment errors and timeouts
Related change	CHG-2026-318 — Terraform network update
Impacted service	Payment API

Description

Payment success rate has dropped from approximately 99.95% to 41%. Payment API 5xx errors have increased. Kubernetes reports all 12 API Pods as Ready. RDS instance status is Available. Investigate application dependencies, infrastructure changes, and customer impact.

Initial assignment: Primary on-call SRE.

Where else might you learn about the incident?

You may see the same incident in the PagerDuty mobile app, Slack incident channel, Jira queue, Grafana dashboard, CloudWatch alarms, or customer support escalation. Different companies use different integrations.

Student question: “What is the difference between an alert, an incident ticket, and a change ticket?”

An alert is a monitoring signal that requires attention.
An incident ticket tracks the unexpected service disruption and response.
A change ticket documents an intended production change, its approvals, timing, and rollback plan.

4. The incident timeline

Time	Event
11:20 PM	GitHub Actions infrastructure pipeline starts
11:28 PM	Terraform plan is reviewed and approved
11:40 PM	Terraform apply completes successfully
11:45 PM	Infrastructure pipeline reports success
12:00 AM	Payment API error rate begins increasing
12:10 AM	Payment success rate drops sharply
12:15 AM	PagerDuty pages the primary on-call engineer
12:17 AM	SRE acknowledges and begins triage
12:22 AM	Incident Commander engaged
12:30 AM	Database connectivity errors identified
12:40 AM	Recent infrastructure change becomes the leading suspect
12:50 AM	Recovery change is prepared and reviewed
1:05 AM	Connectivity restored; application recovery monitored
1:20 AM	Payment success and error metrics return to baseline
1:35 AM	Incident moves into observation and post-incident follow-up

These timestamps are the planned case-study narrative, not measured outcomes from the student lab.

5. The mystery: why healthy Pods cannot process payments

At first, you check Kubernetes:

kubectl get pods -n payments-prod

The output shows:

NAME                           READY   STATUS    RESTARTS
payment-api-7d8f5b6c9-a1b2c    1/1     Running   0
payment-api-7d8f5b6c9-d3e4f    1/1     Running   0
payment-api-7d8f5b6c9-g5h6i    1/1     Running   0
...
12 payment-api Pods Ready

Then you inspect the Deployment:

kubectl get deployment payment-api -n payments-prod

NAME          READY   UP-TO-DATE   AVAILABLE
payment-api   12/12   12           12

You inspect RDS in AWS and see:

DB instance: northstar-payments-prod
Status: Available
Engine: PostgreSQL
Port: 5432

Yet the payment API returns:

HTTP/1.1 503 Service Unavailable

Application logs contain:

ERROR payment authorization failed
ERROR could not establish database connection
ERROR connection timeout to postgres endpoint:5432

This is the central lesson of the incident:

Kubernetes Pod readiness, application dependency health, and successful business transactions are three different things.

A database can be operational while a particular application cannot connect to it.

Class 1 — Detect, assess, and investigate

3 hours · Students act as the primary on-call SRE

Class 1 schedule

Time	Activity
0:00–0:25	Read incident ticket, identify business impact
0:25–0:50	Understand team structure and incident communication
0:50–1:20	Investigate Kubernetes health
1:20–1:50	Investigate Grafana and HTTP errors
1:50–2:20	Check application logs and database connectivity
2:20–2:45	Identify next investigation steps
2:45–3:00	Mock interview and handover

Task 1 — Your first five minutes

You receive the PagerDuty alert. You are the primary on-call engineer.

Your interviewer asks:

“What would you do immediately?”

A model response:

“First, I would acknowledge the PagerDuty alert. Then I would open our monitoring dashboards to validate the impact, including payment success rate, HTTP 5xx errors, latency, and Kubernetes availability. I would determine whether the issue affects all customers or a subset, identify the impacted services, and follow our incident escalation procedure. I would not immediately restart Pods because the evidence does not yet show that restarting them would help.”

What you should ask the incident team

Which payment operations are failing: authorization, capture, refund, or all of them?
When did the errors begin?
Are all customers affected or only certain regions or merchants?
Did an application release, infrastructure change, or database maintenance occur?
Are failures isolated to one service or shared by multiple services?
Are transactions being rejected, timing out, or potentially duplicated?
Is there a known-good recovery procedure?

Important: During a payment incident, do not blindly retry failed transactions. A timeout does not necessarily mean a payment was never processed. The application and payment teams must verify transaction status and idempotency before retrying.

Task 2 — Kubernetes investigation

Run the following commands in the designated training namespace. In the production case study, the namespace is payments-prod.


# Confirm cluster context before running commands
kubectl config current-context

# Inspect deployments
kubectl get deployments -n payments-prod

# Inspect Pods and their nodes
kubectl get pods -n payments-prod -o wide

# Inspect Services
kubectl get svc -n payments-prod

# Inspect Service endpoints
kubectl get endpointslices -n payments-prod

# Inspect nodes
kubectl get nodes -o wide

# Inspect the payment Deployment
kubectl describe deployment payment-api -n payments-prod

# Inspect one payment Pod
kubectl describe pod POD_NAME -n payments-prod

# Review application logs
kubectl logs POD_NAME -n payments-prod --tail=100

What should you conclude?

If all Pods are Ready, you have evidence that Kubernetes considers them eligible to serve traffic. You do not yet have evidence that the payment workflow is functioning correctly.

Check the readiness probe:

kubectl get deployment payment-api \
  -n payments-prod -o yaml

Suppose you discover:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5

The endpoint checks whether the application process can respond, but it does not test PostgreSQL connectivity.

Interview follow-up: “Should a readiness probe always connect to the database?”

No. Dependency-aware readiness checks can be useful, but they require careful design. If every Pod fails readiness during a shared database outage, the Service may lose all endpoints, potentially worsening recovery. Teams often separate lightweight process readiness, dependency health, and end-to-end business transaction monitoring.

Task 3 — Grafana investigation

Your instructor should show four monitoring panels.

Example incident dashboard

Available API replicas

12 / 12

Stable

Payment success rate

41%

Severely degraded

HTTP 5xx error rate

37%

Elevated

Database connection failures

Increasing

Application-side metric

PromQL examples

These queries assume that the required Kubernetes and application metrics are actually collected. Metric names such as http_requests_total and payment_transactions_total are example application instrumentation; they do not appear automatically from kube-state-metrics.

Available replicas

kube_deployment_status_replicas_available{
  namespace="payments-prod",
  deployment="payment-api"
}

Desired replicas

kube_deployment_spec_replicas{
  namespace="payments-prod",
  deployment="payment-api"
}

HTTP 5xx percentage

100 *
sum(rate(http_requests_total{
  service="payment-api",
  status=~"5.."
}[5m]))
/
clamp_min(
  sum(rate(http_requests_total{
    service="payment-api"
  }[5m])),
  0.001
)

Payment success percentage

100 *
sum(rate(payment_transactions_total{
  result="success"
}[5m]))
/
clamp_min(
  sum(rate(payment_transactions_total[5m])),
  0.001
)

95th percentile HTTP latency

histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{
      service="payment-api"
    }[5m])
  )
)

For HTTP status-code metrics, instrument the application or collect appropriate ingress/load-balancer metrics. Prometheus cannot derive HTTP 503 counts from Pod readiness alone.

Interviewer asks

“Grafana shows all 12 Pods healthy but the payment success rate is 41%. Where do you investigate next?”

A strong response:

“I would investigate the application request path and its dependencies. I would review payment API logs, trace failed requests, check database connection metrics, and test connectivity from the application network. I would also review recent infrastructure and application changes.”

Task 4 — Application logs

Example logs:

00:11:42 INFO  POST /api/payments/authorize
00:11:42 ERROR database connection timeout
00:11:42 ERROR SQLSTATE 08001
00:11:42 WARN  payment authorization unavailable
00:11:42 INFO  response_status=503 duration_ms=5002

Explain to the instructor:

What does SQLSTATE 08001 suggest?
Does this prove the database server has crashed?
What is the difference between a connection timeout and an authentication failure?
Why might the application return 503 while its Pod remains Ready?
Which team should investigate the database, and which team should investigate the application?

A useful answer to question 2:

“No. A database connection failure can result from networking, security groups, routing, DNS, authentication, database capacity, or a database outage. I would test each layer before deciding which component is responsible.”

Class 1 communication exercise

Students should practice this update to the incident commander:

Incident update — Payment API

We have confirmed significant customer-facing payment failures. Payment success rate has dropped to approximately 41%, and HTTP 5xx errors are elevated.

All 12 Payment API Pods remain Ready, and the RDS instance reports Available. However, application logs show PostgreSQL connection timeouts.

We are investigating the application-to-database network path and reviewing recent infrastructure changes with the platform and database teams.

The root cause is not yet confirmed. Our priority is restoring payment authorization safely and preventing duplicate transactions.

Class 1 homework

Students should submit a short incident update, the commands they would use, their initial hypotheses, and an explanation of why healthy Pods do not guarantee healthy payment processing.

Class 2 — Terraform, CI/CD, networking, and database

3 hours · Deep technical investigation

Class 2 schedule

Time	Activity
0:00–0:25	Review Class 1 evidence
0:25–0:55	Trace the payment request to PostgreSQL
0:55–1:25	Investigate database connectivity and metrics
1:25–1:55	Review GitHub Actions and Terraform change history
1:55–2:25	Compare security group rules
2:25–2:45	Confirm the cause and propose recovery
2:45–3:00	Technical interview challenge

Task 1 — Understand the database connection

The payment application uses PostgreSQL to store payment transactions.

Example database tables:


CREATE TABLE payments (
    payment_id UUID PRIMARY KEY,
    merchant_id VARCHAR(50) NOT NULL,
    amount_cents BIGINT NOT NULL,
    currency CHAR(3) NOT NULL,
    status VARCHAR(20) NOT NULL,
    idempotency_key VARCHAR(100) UNIQUE NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE payment_events (
    event_id BIGSERIAL PRIMARY KEY,
    payment_id UUID NOT NULL REFERENCES payments(payment_id),
    event_type VARCHAR(50) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

Example payment statuses are PENDING, AUTHORIZED, DECLINED, and FAILED.

In a real payment platform, financial transaction data needs additional controls, auditing, access restrictions, and carefully designed state transitions. This simplified schema is for learning connectivity and SQL troubleshooting, not a complete payment ledger.

Normal payment request flow

Customer sends POST /api/payments/authorize.
ALB routes the request to the Payment API in EKS.
Payment API validates the request and checks its idempotency key.
Application opens or reuses a PostgreSQL connection.
Application reads or writes payment state and coordinates the external authorization workflow.
Application returns the appropriate response and emits metrics.

What happens if step 4 fails?

The application may be unable to read or persist the payment state. It must not blindly repeat external authorization requests or assume that a timed-out transaction did not succeed.

Task 2 — Test database connectivity

The database engineer confirms that RDS is Available. Your next question is whether the Payment API network can reach the database endpoint on TCP port 5432.

For the production case study, you would use approved access through the existing application Pod or an authorized diagnostic Pod in the same network context.


# Check the ServiceAccount and Pod configuration
kubectl get pod POD_NAME \
  -n payments-prod -o yaml

# Inspect relevant application logs
kubectl logs POD_NAME \
  -n payments-prod --tail=100

# Check database DNS resolution
getent hosts DB_ENDPOINT

# Check TCP connectivity
nc -vz -w 5 DB_ENDPOINT 5432

The last two commands must run in an authorized environment with equivalent network access to the affected application, not simply on an unrelated EC2 management host. A successful connection from a bastion does not prove that an EKS Pod can connect.

Expected investigation evidence:

DNS resolution: successful

TCP connection:
connect to DB_ENDPOINT port 5432:
Connection timed out

This suggests a network-path problem but does not yet identify the exact AWS rule responsible.

Interviewer follow-ups

“What if DNS fails?”

Investigate CoreDNS, VPC DNS configuration, DNS records, resolver access, and the endpoint being used.

“What if DNS works but TCP times out?”

Investigate security groups, Network ACLs, routing, NetworkPolicies, database endpoint reachability, and possible network packet drops.

“What if TCP succeeds but PostgreSQL authentication fails?”

Investigate database credentials, IAM database authentication if configured, Secrets Manager, PostgreSQL roles, SSL requirements, and authentication settings.

“What if TCP succeeds but SQL queries are slow?”

Investigate query plans, locks, active sessions, connection pools, CPU, I/O, and database wait events.

Task 3 — Inspect PostgreSQL health

The database team runs approved diagnostic queries:


-- Verify database responds
SELECT 1;

-- Count current connections
SELECT count(*) AS active_connections
FROM pg_stat_activity;

-- Inspect connection states
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;

-- Check configured connection limit
SHOW max_connections;

-- Review sessions waiting on resources
SELECT
    pid,
    state,
    wait_event_type,
    wait_event
FROM pg_stat_activity
WHERE wait_event IS NOT NULL;

If the database is responsive and the connection count is low, but Payment API Pods cannot connect, the team should investigate the network path.

A low connection count alone does not prove a security group problem. It becomes useful evidence when combined with timeouts, recent changes, and network configuration.

AWS RDS checks

aws rds describe-db-instances \
  --db-instance-identifier northstar-payments-prod \
  --query 'DBInstances[0].{Status:DBInstanceStatus,Endpoint:Endpoint.Address,Port:Endpoint.Port,VpcSecurityGroups:VpcSecurityGroups}'

Check CloudWatch or RDS monitoring for database connections, CPU utilization, free memory, storage, and relevant logs. Enhanced Monitoring and Performance Insights/Database Insights may provide deeper diagnostics if configured.

Task 4 — Investigate the CI/CD pipeline

The release engineer discovers a GitHub Actions workflow:

GitHub Actions — Production Infrastructure

Succeeded

Workflow: terraform-production.yml · Completed 11:45 PM

Checkout repository
Terraform format and validate
Terraform plan
Production approval
Terraform apply

Interviewer: “If the Terraform pipeline succeeded, how could it have caused the incident?”

Answer:

“A successful Terraform apply means the requested infrastructure changes were applied. It does not prove that application connectivity or business functionality remained healthy. The pipeline may lack post-deployment integration tests or application-level validation.”

Production workflow example

Students should understand this pipeline structure:


name: Production Terraform

on:
  workflow_dispatch:

permissions:
  contents: read
  id-token: write

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Configure approved AWS OIDC credentials
      # Install a pinned, approved Terraform version
      - run: terraform fmt -check
      - run: terraform init -input=false
      - run: terraform validate
      - run: terraform plan -out=tfplan

  # Production apply should be a separate approved job.
  # Use a protected environment, artifact integrity
  # checks, and the reviewed saved plan.

This is a teaching skeleton, not a complete production-ready workflow. In a real pipeline, the saved plan must be securely transferred between jobs, access controlled, and applied only after approval.

Questions students must answer

What is the difference between terraform plan and terraform apply?
What does Terraform state store?
Why use a remote backend and state locking?
Why should GitHub Actions authenticate to AWS using OIDC rather than long-lived access keys?
What does a protected GitHub environment accomplish?
Why is manual approval useful but not sufficient?
What post-apply checks would have detected the payment outage?
Why can an infrastructure change fail without any new application image?

Task 5 — The Terraform change

The platform engineer compares the previous and current security group configuration.

Before the change

The RDS security group allows PostgreSQL access from the EKS application security group.

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = aws_security_group.eks_application.id

  ip_protocol = "tcp"
  from_port   = 5432
  to_port     = 5432
}

After the change

A Terraform refactor accidentally replaces the application security group reference with an unrelated security group.

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = aws_security_group.management.id

  ip_protocol = "tcp"
  from_port   = 5432
  to_port     = 5432
}

Important: The code is illustrative. Actual EKS Pod network traffic may use node security groups or Pod security groups depending on the VPC CNI and security-groups-for-Pods configuration. The effective source security group must be verified in the actual architecture.

What happened?

The authorized source for PostgreSQL traffic was changed.

The database remained healthy, and the EKS Pods remained Ready, but new database connections from the application network were no longer permitted by the intended ingress rule.

Existing established connections may continue temporarily because AWS security groups are stateful and connection tracking can preserve some flows. As connections close, recycle, or time out, application failures may increase.

This explains why the outage can appear after the Terraform apply rather than exactly when the change is made.

Compare the infrastructure change

git log --oneline -10

git diff PREVIOUS_COMMIT CURRENT_COMMIT -- terraform/

terraform plan

For AWS-side verification:

aws ec2 describe-security-groups \
  --group-ids sg-EXAMPLE \
  --query 'SecurityGroups[*].IpPermissions'

Do not assume a security group ID from a screenshot or example. Identify the actual RDS security group and the effective source security group first.

Root cause evidence chain

Terraform changed the RDS ingress source security group.
The infrastructure change occurred before the payment errors.
Payment API logs show database connection timeouts.
RDS remains operational and responsive from an authorized network.
The application network cannot establish new TCP connections to port 5432.
Restoring the approved application security group rule restores connectivity and payment processing.

This combination of change history, network testing, application logs, and successful recovery provides much stronger causal evidence than timing alone.

Class 2 interview challenge

“How would you prove the Terraform change caused the payment failures rather than a PostgreSQL outage?”

Expected answer:

“I would compare the Terraform plan and AWS security group configuration with the last known-good state, verify that the database itself is responsive, and test connectivity from the affected application network. I would correlate the network failures with the infrastructure change and confirm that restoring the correct rule resolves the connectivity issue.”

Class 3 — Recovery, prevention, and interview mastery

3 hours · Students act as incident responders and interview candidates

Class 3 schedule

Time	Activity
0:00–0:30	Decide on mitigation and evaluate rollback risk
0:30–1:00	Restore database connectivity
1:00–1:30	Verify Kubernetes, database, and customer transactions
1:30–2:00	Write the incident report and preventive actions
2:00–2:40	Advanced technical interview
2:40–3:00	Final individual interview assessment

Task 1 — How do you restore production?

The platform engineer has identified the incorrect RDS security group ingress rule.

The incident commander asks:

“What is the safest and fastest way to restore service?”

The team should evaluate two possible recovery approaches.

Option A — Revert the incorrect Terraform change. Restore the previously approved application security group rule in code, review the plan, obtain the necessary approval, and apply the corrective configuration.

Option B — Emergency network change. If the approved emergency procedure allows an immediate AWS-side correction, an authorized engineer can restore the exact known-good rule and subsequently reconcile Terraform code and state. This must be documented and reviewed to avoid configuration drift.

Do not open PostgreSQL to 0.0.0.0/0, disable all security groups, or expose RDS publicly as a troubleshooting shortcut.

Example corrected Terraform configuration

resource "aws_vpc_security_group_ingress_rule" "postgres_from_eks" {
  security_group_id            = aws_security_group.rds.id
  referenced_security_group_id = aws_security_group.eks_application.id

  ip_protocol = "tcp"
  from_port   = 5432
  to_port     = 5432

  description = "PostgreSQL access from approved EKS application SG"
}

Interviewer asks

“Would you simply run terraform apply again?”

No. First inspect the planned changes. Applying the same incorrect configuration again would not solve the incident.

“Would you run terraform destroy?”

No. Destroying production infrastructure would be highly dangerous and unrelated to this targeted connectivity issue.

“Would you roll back the Kubernetes Deployment?”

Not automatically. If the root cause is an RDS security group rule, reverting the application image may have no effect. The corrective action should target the confirmed cause.

Task 2 — Verify recovery at every layer

After the approved rule is restored, verify the application network can connect to PostgreSQL.


# Verify application Pod health
kubectl get pods -n payments-prod

# Verify Deployment availability
kubectl get deployment payment-api -n payments-prod

# Verify Service endpoints
kubectl get endpointslices -n payments-prod

# Check recent application errors
kubectl logs deployment/payment-api \
  -n payments-prod --tail=100

# Confirm application rollout state
kubectl rollout status \
  deployment/payment-api \
  -n payments-prod

Then verify database connectivity using an approved diagnostic path, confirm the application can perform its expected queries, and run a controlled synthetic payment test using a dedicated nonfinancial test transaction or payment-provider sandbox.

Recovery criteria

Students must demonstrate all five checks

Payment API can establish PostgreSQL connections

Database queries succeed without exposing sensitive records

Customer-facing 5xx rate returns to normal

Payment success rate returns to expected baseline

No unintended duplicate payments or unresolved transaction states

0 of 5 recovery checks completed

Remember: A green Grafana dashboard alone does not establish that customer transactions are correct.

Task 3 — Write the post-incident report

Postmortem — INC-2026-1047

Incident summary

Payment authorization was degraded following a production Terraform networking change.

Customer impact

The illustrative payment success rate declined from 99.95% to 41%, with elevated HTTP 5xx errors and latency.

Technical root cause

An infrastructure refactor changed the source security group allowed to reach RDS PostgreSQL on TCP port 5432. Application Pods remained Ready because their readiness checks did not verify database-dependent business functionality.

Contributing factors

Terraform validation confirmed configuration syntax but did not test application-to-database connectivity.
The CI/CD pipeline lacked a post-apply payment workflow smoke test.
The alerting design detected payment failures but did not immediately identify the affected network dependency.
The readiness probe was insufficient as an indicator of end-to-end payment health.

Mitigation

The team restored the approved database ingress rule and verified application connectivity, transaction processing, and recovery metrics.

Preventive actions

Introduce security group policy checks, infrastructure change review, post-deployment connectivity testing, synthetic payment monitoring, and clearer incident runbooks.

Who owns the preventive actions?

Action	Owner	Verification
Restore and review security group configuration	Platform engineer	Terraform plan and AWS configuration
Add network policy tests to CI/CD	Platform / security	Automated policy checks
Add post-apply connectivity smoke test	DevOps / SRE	Successful synthetic check
Improve payment error metrics	Backend team	Grafana dashboard and alerts
Improve transaction reconciliation	Backend / database	Controlled recovery tests
Update incident runbook	SRE	Reviewed incident exercise
Review production change approvals	Release engineering	Change audit

Practical hands-on lab — Build the failure safely

Students, you will reproduce the application-to-database connectivity failure in your own training EKS cluster.

You will not modify an actual bank, customer database, or production security group.

The safest classroom version uses a PostgreSQL Pod inside a dedicated namespace and a Kubernetes NetworkPolicy to block connectivity. This reproduces the symptom and troubleshooting workflow without changing live AWS infrastructure.

For the Terraform portion, you will separately review an example security group change and its plan. An optional isolated AWS RDS exercise can be added later when your instructor has prepared a dedicated account, budget, and recovery procedure.

Lab architecture

Step 1 — Create an isolated namespace

kubectl create namespace payments-incident-lab

Step 2 — Deploy PostgreSQL

This database is temporary and contains no real customer data.


apiVersion: v1
kind: Secret
metadata:
  name: lab-postgres-secret
  namespace: payments-incident-lab
type: Opaque
stringData:
  POSTGRES_PASSWORD: "lab-only-change-me"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: lab-postgres
  template:
    metadata:
      labels:
        app: lab-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: payments
            - name: POSTGRES_USER
              value: labuser
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: lab-postgres-secret
                  key: POSTGRES_PASSWORD
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U labuser -d payments
            periodSeconds: 5
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: lab-postgres
  namespace: payments-incident-lab
spec:
  selector:
    app: lab-postgres
  ports:
    - port: 5432
      targetPort: 5432

Save this as postgres-lab.yaml and apply:

kubectl apply -f postgres-lab.yaml

kubectl rollout status deployment/lab-postgres \
  -n payments-incident-lab

This lab uses ephemeral PostgreSQL storage. Data is lost when the database Pod is replaced. That is acceptable for a disposable connectivity exercise but not for a production database.

Step 3 — Verify normal connectivity

Use a temporary PostgreSQL client Pod.

kubectl run payment-db-client \
  -n payments-incident-lab \
  --image=postgres:16 \
  --restart=Never \
  --env="PGPASSWORD=lab-only-change-me" \
  --command -- sleep 3600

Wait until it is Running:

kubectl wait --for=condition=Ready \
  pod/payment-db-client \
  -n payments-incident-lab \
  --timeout=120s

Run a database query:

kubectl exec -n payments-incident-lab \
  payment-db-client -- \
  psql -h lab-postgres \
  -U labuser -d payments \
  -c "SELECT 1;"

Expected:

 ?column?
----------
        1
(1 row)

This proves that the client can reach PostgreSQL before the incident is introduced.

The example password is deliberately disposable. Never use real database credentials in shared class terminals or recorded commands.

Step 4 — Inject the incident

Apply a NetworkPolicy that selects the PostgreSQL Pod and allows no incoming connections.


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-postgres-ingress
  namespace: payments-incident-lab
spec:
  podSelector:
    matchLabels:
      app: lab-postgres
  policyTypes:
    - Ingress
  ingress: []

Save as block-postgres.yaml:

kubectl apply -f block-postgres.yaml

Prerequisite: The cluster must have a network policy implementation that enforces Kubernetes NetworkPolicy, such as a properly configured Amazon VPC CNI network policy agent or another compatible CNI. Otherwise, the policy may exist without actually blocking traffic.

Test again with a timeout:

kubectl exec -n payments-incident-lab \
  payment-db-client -- \
  sh -c 'PGCONNECT_TIMEOUT=5 psql -h lab-postgres -U labuser -d payments -c "SELECT 1;"'

Expected on a correctly enforcing cluster:

connection to server at "lab-postgres" failed:
Connection timed out

Students should now investigate why PostgreSQL is Running and Ready but the client cannot connect.

kubectl get pods -n payments-incident-lab

kubectl get svc -n payments-incident-lab

kubectl get networkpolicy -n payments-incident-lab

kubectl describe networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Student challenge

You are not told that the instructor introduced a NetworkPolicy.

Your job is to identify the exact resource that changed and explain how it prevents database access.

Step 5 — Restore service

Once you have identified and documented the cause, remove the injected policy:

kubectl delete networkpolicy block-postgres-ingress \
  -n payments-incident-lab

Test again:

kubectl exec -n payments-incident-lab \
  payment-db-client -- \
  psql -h lab-postgres \
  -U labuser -d payments \
  -c "SELECT 1;"

Expected: SELECT 1 succeeds.

Clean up the client:

kubectl delete pod payment-db-client \
  -n payments-incident-lab

When the entire lab is complete, the instructor can delete the dedicated namespace.

Important difference between the lab and production

The production case study involves an AWS security group changed by Terraform. The hands-on exercise uses Kubernetes NetworkPolicy to reproduce blocked database traffic safely.

Both teach network connectivity troubleshooting, but they are different control planes. Students must be able to explain that distinction during an interview.

Also, this initial lab tests connectivity directly; it does not yet deploy a complete payment API, transaction simulator, or the custom payment metrics used in the fictional dashboard. Those would be additional components if the class needs a fully instrumented end-to-end payment workflow.

Advanced interview question bank

Practice the following questions without looking at the answers first.

Incident response

1 / 17

You receive a payment failure alert. What are your first five actions?

Show model answerFollow-up

PreviousNext

Final mock interview: Tell me about a critical production incident

Students, your final task is to explain the entire incident in approximately three minutes. This is a model answer for discussing the case study. Adapt it to your actual experience rather than claiming to have personally resolved an incident you only practiced.

Production Incident — Payment API Database Connectivity Failure

One complex production incident scenario I worked through involved a customer-facing payment processing application running on AWS EKS with PostgreSQL on Amazon RDS.

The environment used Terraform for infrastructure management, GitHub Actions for CI/CD, and Prometheus, Grafana, CloudWatch, and PagerDuty for monitoring and alerting.

The incident began when the payment success rate dropped significantly, from approximately 99.95% to 41%. Customers were experiencing payment failures and HTTP 5xx errors.

My first step would be to acknowledge the PagerDuty alert and investigate the customer impact through Grafana. I would check the error rate, latency, request volume, and Kubernetes availability.

Interestingly, all 12 Payment API Pods were Running and Ready, and the RDS database itself reported Available.

This indicated that the problem might be outside basic Kubernetes Pod health.

I would inspect the application logs, where I would find PostgreSQL connection timeouts. I would then work with the database and cloud platform teams to investigate database health and the network path from EKS to RDS.

I would check DNS resolution, TCP connectivity on port 5432, security group rules, and any relevant Kubernetes NetworkPolicies.

At the same time, I would review recent production changes. The release engineering team would identify a successful Terraform infrastructure deployment shortly before the errors started.

By comparing the Terraform change with the previous known-good configuration, we would identify that the RDS security group ingress rule had been changed to reference the wrong source security group.

The database itself was operational, but the application could no longer reliably establish new database connections.

I would coordinate with the incident commander and platform team to restore the previously approved security group rule using the authorized emergency change procedure.

After the correction, I would verify connectivity from the application network, confirm successful database operations, and monitor payment success rate, HTTP errors, latency, and transaction reconciliation.

I would not declare recovery based only on Kubernetes Pod status. I would verify that the customer-facing payment workflow was functioning correctly.

Following recovery, I would participate in the postmortem and recommend stronger Terraform policy checks, post-deployment connectivity testing, synthetic payment monitoring, and improved incident runbooks.

The main lesson from this incident is that healthy Kubernetes Pods and a healthy database do not necessarily mean the complete application is healthy. As a DevOps engineer, I need to investigate the full request path and work across infrastructure, application, database, and security teams to restore service safely.

Final evaluation rubric

Use this scorecard to assess each student during Class 3.

Mock interview score

Adjust scores as the student answers

0 / 100

Needs more practice

Incident impact and first response

0 / 15

Kubernetes investigation

0 / 15

Database and networking diagnosis

0 / 20

Terraform and CI/CD investigation

0 / 20

Mitigation and recovery verification

0 / 15

Communication and postmortem

0 / 15

Reset for next student

What interviewers are really evaluating

This incident is especially useful for DevOps Engineer, Site Reliability Engineer, Platform Engineer, Cloud Infrastructure Engineer, and DevSecOps interviews because it tests several competencies together:

Kubernetes: Understanding that Pod readiness is not the same as business functionality.
AWS: Knowing how VPC networking, security groups, EKS, and RDS interact.
Terraform: Understanding change management, infrastructure drift, and safe recovery.
CI/CD: Recognizing that a successful pipeline does not guarantee a successful production outcome.
Database: Knowing how to distinguish connectivity, authentication, performance, and availability failures.
Monitoring: Using technical and business metrics together.
Incident management: Communicating, escalating, mitigating, verifying, and documenting.

Students should be able to explain not just which command they would run, but what question that command answers.

Your final assignment: Explain why the Payment API could have 12 healthy Kubernetes Pods, an Available PostgreSQL RDS instance, and a successful Terraform deployment while customer payments were failing—and describe the evidence needed to identify and fix the real cause.

That is the central engineering challenge for these three classes.

JumpToTech DevOps Bootcamp Kubernetes Production Support Lab — From Deployment to HTTP Request Troubleshooting

Aisalkyn Aidarova — Thu, 29 Oct 2026 15:48:47 +0000

Level: Beginner to Intermediate\
Role: DevOps Engineer / Cloud Engineer / Site Reliability Engineer\
Project: Restaurant Company\
Environment: Kubernetes (Amazon EKS, kind, or minikube)\
Estimated time: 3–4 hours\
Assignment type: Hands-on Lab + Simulated Production Incident + Interview Preparation

PART 1 — YOUR ROLE AS A DEVOPS ENGINEER

Welcome to Restaurant Company

Imagine you have recently joined Restaurant Company as a DevOps Engineer.

Restaurant Company has a web application that customers use to view restaurant information and place orders.

The application runs in Kubernetes.

Your responsibility is to help ensure that the application is:

Available to customers.
Running on healthy Kubernetes Pods.
Accessible through the correct network routes.
Deployed reliably through CI/CD pipelines.
Monitored for failures.
Recoverable when incidents occur.

You are also responsible for communicating with developers, other DevOps engineers, SREs, engineering managers, and incident responders.

Your job is not simply to execute commands. Your job is to investigate problems, understand the impact, restore service safely, and communicate clearly.

1.1 Tools used by your company

For this simulated environment, assume your company uses:

Tool	Purpose
GitHub	Application source code
GitHub Actions	CI/CD automation
Docker	Container image packaging
Amazon ECR	Container image registry
Amazon EKS	Managed Kubernetes
kubectl	Kubernetes administration
Jira / ServiceNow	Incident tickets
Slack / Microsoft Teams	Team communication
Prometheus	Application and infrastructure metrics
Grafana	Monitoring dashboards
Amazon CloudWatch	AWS monitoring and logs
CoreDNS	Internal Kubernetes DNS
AWS ALB	External HTTP/HTTPS routing

Different companies may use different tools. The concepts remain similar.

PART 2 — UNDERSTAND THE APPLICATION ARCHITECTURE

Before troubleshooting production issues, you must understand how the application reaches Kubernetes.

2.1 How an application is deployed

A developer writes application code and pushes it to GitHub.

A typical CI/CD workflow then performs the following steps:

Developer pushes code to a GitHub repository.
GitHub Actions starts a CI/CD pipeline.
The pipeline checks out the source code.
The pipeline installs dependencies and runs tests.
Docker builds a container image.
The image is pushed to Amazon ECR.
The deployment workflow updates the Kubernetes Deployment to use the new image.
The Kubernetes API Server accepts the updated desired state.
The Deployment Controller manages the ReplicaSet.
The ReplicaSet Controller maintains the required number of Pods.
The Scheduler selects suitable Worker Nodes for new Pods.
kubelet on each assigned Worker Node works with the container runtime to start the containers.
The CNI implementation configures Pod networking.
Readiness probes determine whether the application Pods are ready to receive traffic.
Kubernetes Service endpoints are updated to reflect eligible backends.

Important: This is the typical production deployment architecture. In this lab, we will create Kubernetes resources manually using YAML and kubectl. We are not building a complete GitHub Actions/ECR pipeline today.

2.2 Kubernetes component responsibilities

API Server

The API Server is the entry point for Kubernetes API requests. When you execute kubectl apply, kubectl sends the requested configuration to the API Server.

etcd

etcd stores Kubernetes cluster state, including the desired state of Kubernetes resources.

Deployment Controller

The Deployment Controller manages ReplicaSets and application rollouts.

ReplicaSet Controller

The ReplicaSet Controller ensures that the desired number of Pods exists.

Scheduler

The Scheduler selects an appropriate Worker Node for a Pod that has not yet been assigned to a Node.

kubelet

kubelet is the agent running on a Kubernetes Node. It works with the container runtime to maintain assigned Pods.

Container Runtime

The container runtime, such as containerd, manages container image pulling and container execution.

Linux Kernel

The Linux Kernel provides the underlying operating system functionality, including processes, networking, namespaces, cgroups, and resource management.

CNI

The Container Network Interface implementation configures Pod networking. Depending on the CNI plugin, this includes interfaces, IP allocation, and routes.

CoreDNS

CoreDNS resolves Kubernetes Service DNS names into their corresponding Service IP addresses.

EndpointSlices

EndpointSlices publish backend endpoint information associated with Kubernetes Services.

kube-proxy

In clusters using kube-proxy, it watches Services and EndpointSlices and configures the Service networking dataplane. In iptables mode, Linux Netfilter processes packets according to the programmed rules.

Other Kubernetes clusters may use different networking implementations, such as eBPF-based Service routing.

PART 3 — YOUR FIRST PRODUCTION TICKET

Incident Ticket: INC-2026-1042

Title: Restaurant Backend Service Unavailable\
Priority: P2 — High (simulated)\
Environment: Kubernetes\
Assigned Team: DevOps / Platform Engineering\
Status: Open — Investigating

Incident description

Customers are reporting that the Restaurant application cannot retrieve backend information.

Monitoring indicates that application requests are failing.

The Kubernetes Deployment appears healthy, but requests to the backend Service are unsuccessful.

Business impact

Customers may be unable to view restaurant information or complete certain application workflows.

Your task is to investigate the issue, identify the root cause, restore connectivity, and document the resolution.

3.1 Where will you receive the ticket?

In a real company, incidents may be reported through:

Jira or ServiceNow.
PagerDuty or another on-call alerting system.
Grafana or Prometheus alerts.
CloudWatch alarms.
Slack or Microsoft Teams.
Customer support escalation.
Developer reports.

For example:

A monitoring system detects failed HTTP requests and creates an alert.

The alert reaches the on-call engineer.

The engineer acknowledges the alert, reviews the impact, and opens or updates the incident ticket.

The incident is then investigated according to the company's incident-management procedures.

3.2 What should you do first?

Before executing troubleshooting commands:

Acknowledge the ticket.
Identify the affected application and environment.
Determine the reported symptoms and business impact.
Check whether there is an active incident bridge or incident commander.
Review recent deployments and infrastructure changes.
Start with safe, read-only diagnostic commands.
Communicate your investigation status.

Do not immediately restart applications, delete Pods, or modify production configuration without understanding the situation and following company procedures.

3.3 Your first Slack message

Example:

Hi team,

I have acknowledged INC-2026-1042 regarding failed requests to the Restaurant backend service.

I am reviewing the Kubernetes Deployment, Pod readiness, Service configuration, and backend endpoints to determine the cause.

I will provide another update within 10 minutes.

Please let me know if there were any recent application deployments or infrastructure changes.

Thank you.

3.4 What information should you collect?

Before troubleshooting, identify:

Application name.
Namespace.
Cluster.
Environment.
Time the issue started.
User-facing symptoms.
Error codes, if available.
Recent deployments.
Monitoring alerts.
Relevant logs.
Current incident owner.

For our lab:

Application: Restaurant Company\
Namespace: jumptotech-lab\
Deployment: restaurant\
Service: restaurant-service\
Port: 80\
Replicas: 2

PART 4 — BUILD YOUR KUBERNETES ENVIRONMENT FROM ZERO

Step 1 — Verify Kubernetes access

Open your terminal.

If you are using Amazon EKS, make sure your AWS credentials, cluster access, and kubeconfig are configured.

If you are using kind or minikube, ensure your local cluster is running.

Execute:

kubectl cluster-info
kubectl get nodes -o wide
kubectl config current-context

Expected result:

Kubernetes API Server is reachable.
At least one Node is Ready.
Your current context points to the intended lab cluster.

Important: Never assume you are connected to the correct cluster. Always verify your Kubernetes context before making changes.

If you receive a connection error, stop and troubleshoot cluster access before continuing.

Step 2 — Create your project directory

mkdir -p ~/jumptotech-k8s-network-lab
cd ~/jumptotech-k8s-network-lab
pwd

Step 3 — Create a Namespace

kubectl create namespace jumptotech-lab

Verify:

kubectl get namespaces

Interview question: Why do we use Kubernetes Namespaces?

Answer:

Namespaces provide logical separation of namespaced Kubernetes resources within a cluster. Teams commonly use them to organize applications and environments, apply access controls, and manage resource quotas.

Namespaces alone are not a complete security boundary.

PART 5 — CREATE THE RESTAURANT DEPLOYMENT

Step 4 — Create deployment.yaml

Execute:

cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: restaurant
  namespace: jumptotech-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: restaurant
  template:
    metadata:
      labels:
        app: restaurant
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "50m"
            memory: "64Mi"
          limits:
            cpu: "250m"
            memory: "128Mi"
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 2
          periodSeconds: 5
EOF

Step 5 — Apply the Deployment

kubectl apply -f deployment.yaml

Check rollout status:

kubectl rollout status deployment/restaurant \
  -n jumptotech-lab --timeout=180s

Check resources:

kubectl get deployments -n jumptotech-lab

kubectl get replicasets -n jumptotech-lab

kubectl get pods -n jumptotech-lab -o wide

Expected result:

Deployment exists.
ReplicaSet exists.
Two Restaurant Pods are Running and Ready.

If the Pods are not Ready, do not continue blindly.

Investigate:

kubectl describe deployment restaurant \
  -n jumptotech-lab

kubectl describe pods \
  -n jumptotech-lab -l app=restaurant

kubectl get events -n jumptotech-lab \
  --sort-by=.lastTimestamp

Step 6 — Understand what happened

When you executed kubectl apply -f deployment.yaml, you submitted the desired application configuration to Kubernetes.

The API Server processed the request.

The Deployment Controller managed the ReplicaSet.

The ReplicaSet Controller created the required Pods.

The Scheduler selected Worker Nodes.

kubelet worked with the container runtime to start the containers.

The CNI implementation configured networking for the Pods.

Student questions:

What is a Deployment?
What is a ReplicaSet?
What is a Pod?
Who selects the Worker Node?
Who starts the container?
What is the role of the Linux Kernel?
Why does the Pod need a network interface and IP address?

Be prepared to explain every answer without reading your notes.

PART 6 — CREATE THE KUBERNETES SERVICE

Step 7 — Create service.yaml

cat > service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: restaurant-service
  namespace: jumptotech-lab
spec:
  type: ClusterIP
  selector:
    app: restaurant
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
EOF

Apply:

kubectl apply -f service.yaml

Verify:

kubectl get svc -n jumptotech-lab

kubectl describe svc restaurant-service \
  -n jumptotech-lab

Check EndpointSlices:

kubectl get endpointslices \
  -n jumptotech-lab \
  -l kubernetes.io/service-name=restaurant-service \
  -o wide

Step 8 — Understand Service and Pod labels

Your Service has this selector:

selector:
  app: restaurant

Your Pods have this label:

labels:
  app: restaurant

The matching labels allow the EndpointSlice Controller to publish the correct backend endpoints.

The Service provides a stable virtual IP address, called a ClusterIP.

Pods can be replaced and receive new IP addresses, while the Service can retain its ClusterIP.

Interview question: Why do we need a Service if Pods already have IP addresses?

Answer:

Pod IP addresses may change when Pods are recreated. A Kubernetes Service provides a stable endpoint and Service discovery mechanism for reaching eligible backend Pods.

PART 7 — CREATE A DIAGNOSTIC CLIENT POD

Step 9 — Create client.yaml

cat > client.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: jumptotech-lab
spec:
  containers:
  - name: netshoot
    image: nicolaka/netshoot:latest
    command: ["sleep", "36000"]
    resources:
      requests:
        cpu: "25m"
        memory: "32Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"
EOF

Apply:

kubectl apply -f client.yaml

Wait:

kubectl wait \
  --for=condition=Ready \
  pod/client \
  -n jumptotech-lab \
  --timeout=180s

Enter the Pod:

kubectl exec -it client \
  -n jumptotech-lab -- bash

Your terminal prompt should change to something similar to:

client:~#

Important: You are now inside the Client Pod, not on your EC2 management host.

PART 8 — INVESTIGATE POD NETWORKING

Step 10 — Check the Pod hostname

Inside Client Pod:

hostname

Step 11 — Check network interfaces

ip addr

Find the eth0 interface and its IPv4 address.

Record the IP address.

Step 12 — Check routing

ip route

Look for the default route.

In AWS VPC CNI, you may see a gateway such as:

default via 169.254.1.1 dev eth0

Other Kubernetes network implementations may have different routing configurations.

Step 13 — Check DNS configuration

cat /etc/resolv.conf

Identify the configured DNS server and search domains.

Interview question: What is a Linux Network Namespace?

Answer:

A Network Namespace provides an isolated view of networking resources, including network interfaces, IP addresses, routing tables, and ports.

Containers in the same Kubernetes Pod normally share one Network Namespace.

Interview question: What does CNI do?

Answer:

The CNI implementation configures networking for Pods. Depending on the plugin, this includes network interfaces, IP addresses, and routing.

PART 9 — TRACE AN HTTP REQUEST

Step 14 — DNS resolution

Still inside Client Pod:

nslookup restaurant-service

Then:

nslookup restaurant-service.jumptotech-lab.svc.cluster.local

Expected:

The DNS lookup returns the ClusterIP assigned to restaurant-service.

Record the IP address.

What happened?

The Client Pod's resolver queried cluster DNS.

CoreDNS resolved the Kubernetes Service name.

The DNS response provided the Service IP address.

CoreDNS does not forward the HTTP request itself.

Step 15 — Send HTTP request

curl -v http://restaurant-service

Then:

curl -I http://restaurant-service

Expected HTTP status:

HTTP/1.1 200 OK

The default NGINX welcome page should be returned.

Step 16 — Understand the complete internal request flow

Client Pod initiates an HTTP request.
DNS resolves restaurant-service to its ClusterIP.
The client establishes a TCP connection to the Service IP on port 80.
The Kubernetes Service dataplane selects an eligible backend Pod.
In iptables-based kube-proxy mode, Linux Netfilter applies Service forwarding and address translation rules.
The packet reaches the selected Restaurant Pod.
NGINX receives the HTTP request on port 80.
NGINX returns the HTTP response.
The response travels back to the Client Pod.

Important distinction:

kube-proxy programs forwarding rules in its supported modes. In iptables mode, the Linux Kernel processes packets using those rules.

The exact implementation depends on the cluster networking configuration.

Step 17 — Compare direct Pod IP requests

Exit the Client Pod:

exit

Find the current Restaurant Pod IP addresses:

kubectl get pods -n jumptotech-lab \
  -l app=restaurant -o wide

Record both Pod IPs.

Enter Client Pod again:

kubectl exec -it client \
  -n jumptotech-lab -- bash

Run the following, replacing the placeholders with the real IP addresses:

curl -I http://POD_1_IP:80
curl -I http://POD_2_IP:80

For example, if a Pod IP is 172.31.46.36:

curl -I http://172.31.46.36:80

Interview question: What is the difference between a Service request and a direct Pod IP request?

Answer:

A Service request uses a stable Service endpoint and the cluster's Service routing implementation to reach an eligible backend.

A direct Pod IP request targets one specific Pod and bypasses Service load balancing.

PART 10 — REPRODUCE THE PRODUCTION INCIDENT

Now we will simulate a real Kubernetes networking incident.

We will intentionally create a Service with an incorrect selector.

We will not modify the healthy Restaurant Service.

Step 18 — Exit Client Pod

exit

Step 19 — Create a broken Service

cat > broken-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: restaurant-broken
  namespace: jumptotech-lab
spec:
  type: ClusterIP
  selector:
    app: wrong-restaurant
  ports:
  - name: http
    port: 80
    targetPort: 80
EOF

Apply:

kubectl apply -f broken-service.yaml

Step 20 — Investigate the broken Service

Check:

kubectl get svc -n jumptotech-lab

Describe:

kubectl describe svc restaurant-broken \
  -n jumptotech-lab

Check Pod labels:

kubectl get pods -n jumptotech-lab \
  --show-labels

Check EndpointSlices:

kubectl get endpointslices \
  -n jumptotech-lab \
  -l kubernetes.io/service-name=restaurant-broken \
  -o wide

Observation:

The Service selector uses app: wrong-restaurant.

The Restaurant Pods use app: restaurant.

Because these values do not match, the Service has no eligible backend endpoints.

Step 21 — Reproduce the failed HTTP request

Test DNS from Client Pod:

kubectl exec client -n jumptotech-lab -- \
  nslookup restaurant-broken

The DNS name should resolve to the broken Service's ClusterIP.

Now test HTTP:

kubectl exec client -n jumptotech-lab -- \
  curl -v --max-time 5 http://restaurant-broken

The request should fail because the Service has no eligible backend endpoints.

The exact failure may be connection refused, timeout, or another network-level error depending on the cluster dataplane.

Do not automatically describe this as HTTP 503. A Service without endpoints does not necessarily return an HTTP response.

Step 22 — Identify the root cause

Compare:

Service selector:

app: wrong-restaurant

Pod label:

app: restaurant

Root cause:

The Kubernetes Service selector did not match the Restaurant Pod labels.

As a result, no eligible backend endpoints were available for that Service.

The application Pods were healthy, but traffic could not reach them through the broken Service.

PART 11 — COMMUNICATE YOUR FINDINGS

Before applying the fix, write an investigation update.

Example Slack message:

Hi team,

I identified a likely root cause for INC-2026-1042.

The Restaurant application Pods are Running and Ready, but the affected Kubernetes Service has no eligible backend endpoints.

The Service selector is configured with app: wrong-restaurant, while the application Pods use app: restaurant.

I reproduced the connectivity failure from a diagnostic Pod.

I am preparing a selector correction and will verify the backend endpoints and HTTP response after the change.

PART 12 — FIX THE INCIDENT

Step 23 — Correct the Service selector

For our isolated lab, execute:

kubectl patch svc restaurant-broken \
  -n jumptotech-lab \
  -p '{"spec":{"selector":{"app":"restaurant"}}}'

In real production, follow change authorization and GitOps procedures. If the Service is managed through Git, Helm, or Argo CD, update the source of truth to prevent the incorrect configuration from returning.

Step 24 — Verify the fix

kubectl describe svc restaurant-broken \
  -n jumptotech-lab

Check EndpointSlices:

kubectl get endpointslices \
  -n jumptotech-lab \
  -l kubernetes.io/service-name=restaurant-broken \
  -o wide

Confirm that backend Pod IPs appear.

Then:

kubectl exec client -n jumptotech-lab -- \
  curl -I --max-time 5 http://restaurant-broken

Expected:

HTTP/1.1 200 OK

Step 25 — Verify the original healthy Service

kubectl exec client -n jumptotech-lab -- \
  curl -I --max-time 5 http://restaurant-service

Expected:

HTTP/1.1 200 OK

For a real production incident, one successful curl is not enough to establish full recovery.

You should also verify monitoring metrics, error rates, readiness, backend health, and the actual user-facing path.

PART 13 — RESOLUTION COMMUNICATION

After verifying recovery, write a resolution update.

Example Slack message:

Hi team,

INC-2026-1042 has been resolved in our lab environment.

Root cause: The Kubernetes Service selector did not match the application Pod labels, leaving the Service without eligible backend endpoints.

Actions taken:

Confirmed the application Pods were Running and Ready.
Reviewed Service configuration and Pod labels.
Inspected EndpointSlices.
Reproduced the failed HTTP request from a diagnostic Pod.
Corrected the Service selector.
Verified backend endpoints were populated.
Confirmed HTTP 200 responses after the fix.

I recommend adding automated validation for Service selectors and backend endpoint availability to help prevent similar configuration errors.

I will document the investigation and resolution in the incident ticket.

Important: Only use this resolution message after completing and verifying the corresponding actions.

PART 14 — COMPLETE YOUR INCIDENT TICKET

Incident Report

Incident ID: INC-2026-1042

Application: Restaurant Company

Environment: Kubernetes Lab

Issue: Backend Service connectivity failure.

Impact: Simulated inability to reach the backend application through the affected Service.

Detection: Simulated monitoring alert and failed HTTP connectivity test.

Investigation:

Reviewed Kubernetes Deployment, Pod status, Service configuration, Pod labels, and EndpointSlices.

Confirmed the application Pods were Running and Ready.

Observed that the affected Service did not have eligible backend endpoints.

Root Cause:

Service selector mismatch.

The Service used app: wrong-restaurant, while the Pods used app: restaurant.

Resolution:

Updated the Service selector to match the Pod labels.

Verification:

Confirmed backend endpoints appeared and the Service returned HTTP 200 from the diagnostic Client Pod.

Preventive Actions:

Add deployment validation for Service selectors.
Monitor Service backend availability.
Review Kubernetes manifests before deployment.
Use GitOps and peer-reviewed changes where applicable.
Maintain clear incident runbooks.

Status: Resolved — after successful verification.

PART 15 — BONUS LAB: KUBERNETES SELF-HEALING

This lab demonstrates how Kubernetes maintains the desired number of Pods.

Step 26 — Check existing Pods

kubectl get pods -n jumptotech-lab \
  -l app=restaurant

Step 27 — Delete one Pod

Choose an actual Restaurant Pod name.

kubectl delete pod POD_NAME \
  -n jumptotech-lab

Replace POD_NAME with the real name.

Step 28 — Watch Kubernetes create a replacement

kubectl get pods -n jumptotech-lab -w

Press Ctrl+C when the new Pod becomes Ready.

Step 29 — Verify replicas

kubectl get deployment restaurant \
  -n jumptotech-lab

Expected: Two Ready replicas after recovery.

Interview question: What happens when a Pod is deleted?

Answer:

The ReplicaSet Controller detects that the number of existing Pods is below the desired replica count and creates a replacement Pod.

The Scheduler assigns the new Pod to a suitable Node, and kubelet starts its containers.

This is an example of Kubernetes reconciliation and self-healing.

PART 16 — BONUS LAB: ACCESS THE APPLICATION FROM A BROWSER

From the machine where kubectl is configured:

kubectl port-forward \
  -n jumptotech-lab \
  svc/restaurant-service \
  8080:80

Open:

http://localhost:8080

Expected: NGINX welcome page.

If kubectl is running on a remote EC2 instance, localhost:8080 refers to that EC2 instance, not your laptop. Use an approved SSH tunnel or another authorized access method if you need to view it locally.

Important: Port-forwarding is a diagnostic tool. It does not demonstrate the production AWS ALB and Ingress request path.

PART 17 — UNDERSTAND THE EXTERNAL HTTP REQUEST FLOW

A real customer usually does not access the application's internal Kubernetes Service name.

Consider a production architecture using AWS ALB.

A typical request flow is:

Customer opens the Restaurant Company website.
The browser resolves the public domain through DNS.
The browser connects to the public application endpoint, commonly over HTTPS.
AWS ALB receives the request.
ALB evaluates listener and routing rules.
The request is forwarded to a configured target group.
The target group sends the request to a healthy backend target.
The application processes the request.
The HTTP response returns to the browser.

With AWS Load Balancer Controller, ALB can target Pod IPs directly in IP target mode or NodePorts in instance target mode.

For a separate internal backend call, a frontend server Pod may resolve a Kubernetes Service name using CoreDNS and send traffic through the internal Service dataplane.

A browser-hosted React application may instead call a public backend API through ALB.

You must know which architecture your application actually uses.

Important HTTP status codes

Code	Meaning	What DevOps investigates
200	OK	Successful response
201	Created	Resource creation succeeded
301 / 302	Redirect	Redirect configuration
400	Bad Request	Invalid request or API validation
401	Unauthorized	Authentication
403	Forbidden	Authorization
404	Not Found	Missing route or resource
500	Internal Server Error	Application/backend failure
502	Bad Gateway	Gateway or upstream response failure
503	Service Unavailable	Backend availability or overload
504	Gateway Timeout	Upstream timeout

Important: A 404 does not necessarily mean the entire application is unavailable. A 400 does not automatically mean the application is down. HTTP status codes must be interpreted in context.

PART 18 — HOW TO TROUBLESHOOT DIFFERENT ERRORS

Scenario A — Pod is Pending

Check:

kubectl describe pod POD_NAME -n jumptotech-lab

Investigate:

Insufficient CPU or memory.
Scheduling constraints.
Node availability.
Taints and tolerations.
Persistent volume requirements.

Scenario B — ImagePullBackOff

Check:

kubectl describe pod POD_NAME -n jumptotech-lab

Investigate:

Incorrect image name.
Missing image tag.
Registry access.
Image pull permissions.
Registry connectivity.

Scenario C — CrashLoopBackOff

Check:

kubectl logs POD_NAME \
  -n jumptotech-lab --previous

Also inspect Pod events and container exit codes.

Investigate:

Application startup failure.
Incorrect configuration.
Missing dependencies.
Resource limits.
Probe failures.

Scenario D — Service DNS resolution fails

Check from a diagnostic Pod:

nslookup restaurant-service

Investigate:

Service existence.
Namespace and DNS name.
CoreDNS health.
Pod DNS configuration.
DNS connectivity and network policies.

Scenario E — DNS works but HTTP fails

Check:

kubectl get svc -n jumptotech-lab

kubectl get endpointslices \
  -n jumptotech-lab

kubectl get pods -n jumptotech-lab \
  --show-labels

Investigate:

Service selector.
Pod readiness.
Target port.
Application listener.
Network policies.
Service dataplane behavior.

Scenario F — External users receive 503

Investigate:

AWS ALB target health.
Ingress configuration.
Target groups.
Service and endpoint availability.
Pod readiness.
Application health checks.

The investigation depends on which component generated the 503.

PART 19 — HOW TO ANSWER IN A DEVOPS INTERVIEW

Interview Question 1: Tell me about yourself

Sample answer:

I am building hands-on experience in DevOps and cloud infrastructure, focusing on Linux, AWS, Docker, Kubernetes, Terraform, and CI/CD automation.

Through my Restaurant Company project, I have practiced deploying containerized applications to Kubernetes, configuring Deployments and Services, troubleshooting Pod networking, and investigating application connectivity issues.

I have also worked on simulated production incidents, where I followed a structured process to identify problems, verify fixes, and communicate technical findings.

I am especially interested in cloud infrastructure automation, Kubernetes operations, and production reliability.

Interview Question 2: Explain how an application gets deployed to Kubernetes

Sample answer:

In a typical CI/CD workflow, developers push their code to GitHub, which triggers an automated pipeline.

The pipeline builds and tests the application, creates a Docker image, and pushes it to a container registry such as Amazon ECR.

The deployment process updates the Kubernetes Deployment with the new image version.

The Kubernetes API Server accepts the desired configuration, and the Deployment Controller manages the rollout through ReplicaSets.

The Scheduler assigns new Pods to suitable Worker Nodes, and kubelet works with the container runtime to start the containers.

The CNI implementation configures Pod networking, and readiness probes help determine when the application is ready to receive traffic.

Interview Question 3: Explain how an HTTP request reaches a Kubernetes application

Sample answer:

For an external request, the client first resolves the public domain through DNS and connects to the application's public endpoint.

In an AWS environment, the request may reach an Application Load Balancer.

The ALB evaluates its listener rules and forwards traffic to a healthy backend target.

For internal communication between Kubernetes applications, a client Pod can use the backend Service DNS name.

CoreDNS resolves the Service name to its ClusterIP.

The Kubernetes Service dataplane then forwards the connection to an eligible backend Pod.

The application processes the HTTP request and sends the response back to the client.

Interview Question 4: Tell me about a Kubernetes incident you resolved

Sample answer:

During my DevOps training, I investigated a simulated production incident involving a Restaurant Company application running in Kubernetes.

The application Pods were healthy, but HTTP requests through a Kubernetes Service were failing.

I started by reviewing Pod readiness, the Service configuration, Pod labels, and EndpointSlices.

I reproduced the connectivity failure from a diagnostic Client Pod.

DNS resolution was successful, which helped me separate DNS issues from Service routing problems.

I found that the Service selector did not match the application Pod labels, leaving the Service without eligible backend endpoints.

I corrected the selector, verified that the expected endpoints appeared, and confirmed the application returned HTTP 200.

I also practiced communicating the incident status, documenting the root cause, and recommending preventive checks.

This exercise helped me understand how Kubernetes networking works and how to approach connectivity issues systematically.

Interview Question 5: How do you troubleshoot a Kubernetes application that is unavailable?

Sample answer:

I first confirm the scope and impact of the issue.

I check monitoring alerts and determine whether the problem affects all users or only specific requests.

Then I inspect the Kubernetes Deployment, Pod readiness, events, and application logs.

If the Pods are healthy, I investigate Service configuration, EndpointSlices, DNS resolution, and network connectivity.

For externally exposed applications, I also check Ingress configuration, load balancer target health, and relevant AWS networking components.

Once I identify the likely root cause, I coordinate the appropriate mitigation, verify recovery, monitor for recurrence, and document the incident.

Interview Question 6: What is the difference between CoreDNS, CNI, and kube-proxy?

Sample answer:

CoreDNS provides DNS resolution for Kubernetes Service names.

CNI is the interface through which the networking implementation configures Pod connectivity, including IP addressing and interfaces.

kube-proxy, when used, implements Kubernetes Service networking by programming forwarding rules based on Service and EndpointSlice information.

These components have different responsibilities, but they work together to support communication between Kubernetes applications.

Interview Question 7: How do you communicate during a production incident?

Sample answer:

I acknowledge the incident and confirm the business impact.

I provide regular updates to the incident channel, including what has been verified, what I am investigating, and when the next update is expected.

I avoid presenting assumptions as confirmed root causes.

If the incident requires escalation or a potentially disruptive change, I coordinate with the incident lead and relevant teams.

After service recovery, I document the root cause, resolution, verification results, and preventive actions.

PART 20 — YOUR FINAL HOMEWORK

Submit the following:

Screenshot showing Kubernetes Nodes in Ready status.
Screenshot showing your Restaurant Deployment and two Ready Pods.
Screenshot showing your Kubernetes Service and ClusterIP.
Screenshot showing the EndpointSlices.
Screenshot showing successful CoreDNS resolution.
Screenshot showing HTTP 200 from the healthy Service.
Screenshot showing the broken Service selector.
Screenshot showing missing backend endpoints.
Screenshot showing the failed HTTP request.
Screenshot showing the corrected selector.
Screenshot showing HTTP 200 after the fix.
Your first Slack incident acknowledgment.
Your investigation update.
Your resolution update.
Your completed incident report.
A written or recorded interview explanation of the incident.

Final practical assessment

Your instructor may ask you to share your screen and demonstrate:

How to find a Deployment.
How to check Pod health.
How to find the Pod IP.
How to inspect a Service.
How to inspect EndpointSlices.
How to check DNS resolution.
How to test HTTP connectivity.
How to identify a selector mismatch.
How to correct the configuration.
How to verify recovery.

You should explain what every command does and why you are using it.

PART 21 — CLEANUP

After your instructor confirms that the assignment is complete, remove the lab resources:

kubectl delete namespace jumptotech-lab

This removes the resources created in that Namespace.

Do not run this command until you have saved your screenshots and completed the assignment.

FINAL MESSAGE TO STUDENTS

Your goal is not to memorize Kubernetes commands. Your goal is to understand the full lifecycle of an application and how to troubleshoot it when something goes wrong.

A strong DevOps Engineer should be able to explain:

How application code moves from GitHub to Kubernetes.
How Kubernetes schedules and runs containers.
How Pods receive network connectivity.
How Services and EndpointSlices support application communication.
How DNS resolution differs from HTTP routing.
How an HTTP request reaches an application.
How to investigate a production incident safely.
How to communicate findings and progress.
How to verify that a problem is resolved.
How to explain the work clearly in a technical interview.

Remember the incident response structure:

Impact → Investigation → Evidence → Root Cause → Mitigation → Verification → Communication → Prevention

Practice explaining your project in your own words.

Be clear about which activities you performed in a lab and which tools or processes belong to the example production architecture. Never present a training exercise as actual employment experience.

JumpToTech DevOps Bootcamp — Learn It. Build It. Troubleshoot It. Explain It.

You can copy this entire handbook directly into your student group or development platform. It includes the complete lab, the simulated ticket, step-by-step commands, workplace communication, incident reporting, and interview preparation.

JumpToTech DevOps/SRE Bootcamp Production Incident Response Lab — Banking Application on AWS EKS

Aisalkyn Aidarova — Wed, 28 Oct 2026 21:17:39 +0000

Kubernetes Incident Response — Banking Application on AWS EKS

One of the most critical incidents I handled involved a customer-facing banking application running on AWS EKS.

Our production environment had approximately 60 frontend replicas distributed across three Availability Zones, with an AWS Application Load Balancer handling incoming traffic. We used Prometheus and Grafana for monitoring and alerting.

During a production release, we started receiving alerts that the banking application's HTTP 5xx error rate had increased significantly. The error rate reached approximately 18%, and customers were experiencing intermittent failures when accessing their accounts.

I was the primary SRE responsible for investigating the incident and coordinating the technical recovery.

My first step was to assess the impact. I checked Grafana dashboards for error rates, request latency, and available replicas. I noticed that available frontend replicas had dropped from 60 to 38, which explained why the remaining Pods were struggling to handle traffic.

I then checked the EKS cluster using kubectl get nodes, kubectl get pods, and kubectl get deployments. All worker nodes were healthy, so I ruled out a cluster-wide infrastructure failure.

When I inspected the affected Pods using kubectl describe pod, I found repeated readiness-probe failures. The new containers were Running, but they were not becoming Ready.

I checked the Kubernetes events and application logs and found that the readiness probes were returning HTTP 404.

After comparing the new deployment configuration with the previous release, I identified the root cause: the readiness probe had been changed from / to /healthz, but the application did not expose the new endpoint.

I also found that the deployment configuration allowed too many healthy Pods to become unavailable during the rollout, which contributed to the loss of serving capacity.

At that point, my priority was restoring service rather than troubleshooting the defective release further.

I rolled back the deployment using kubectl rollout undo deployment/banking-frontend and monitored the recovery using kubectl rollout status.

As the previous application version came back online, available replicas increased from 38 to 60, and the HTTP 5xx error rate returned to its normal baseline.

I verified recovery by checking Kubernetes Service endpoints, application health checks, load balancer metrics, and Grafana dashboards.

The application was restored within approximately 12 minutes of incident detection.

After the incident, I completed a root cause analysis and recommended three improvements: automated readiness-probe validation in CI/CD, safer RollingUpdate settings with maxUnavailable: 0, and canary deployments to detect faulty releases before they affect the entire application.

The outcome was that I restored full application availability, reduced the immediate customer impact, and helped establish deployment safeguards to prevent the same failure from recurring.

This incident reinforced the importance of understanding Kubernetes readiness probes, deployment strategies, monitoring, and rapid rollback procedures in a high-availability banking environment.

Numbers in this example
Metric Value
Frontend replicas 60
Availability Zones 3
Available replicas during incident 38
Unavailable replicas 22
Peak HTTP 5xx error rate 18%
Readiness-probe error HTTP 404
Recovery time 12 minutes
Replicas after recovery 60
Recovery method Kubernetes rollback

These numbers form a consistent hypothetical production incident, not a historical incident verified at JPMorgan Chase. In a real interview, students should use the metrics from their own incident evidence rather than present these figures as their personal employment history.

What makes this a senior-level answer: It demonstrates that the engineer assessed customer impact, distinguished application failure from infrastructure failure, investigated Kubernetes events and logs, identified the configuration defect, performed a controlled rollback, verified recovery, and implemented preventive measures.
Student Lab Guide · Day 2

Scenario: Banking Application Outage After a Failed Production Deployment\
Environment: AWS EKS, Kubernetes, Prometheus, Grafana, NGINX\
Role: DevOps / Site Reliability Engineer\
Difficulty: Intermediate\
Estimated duration: 3 hours\
Incident severity: Simulated SEV-2 / P2

Important note for students

This lab uses a fictional banking incident inspired by the types of production problems that can occur in large financial institutions such as JPMorgan Chase. It is not a documented incident at Chase, and you must not claim that you worked at Chase or resolved this incident in its actual production environment.

You will, however, perform a real hands-on investigation in your own Kubernetes cluster. You can discuss that experience confidently in interviews as an independently resolved production-style lab incident.

1. Your role and responsibilities

You are a DevOps/SRE Engineer supporting a banking application running on Amazon EKS.

The application allows customers to access banking services. Your team is responsible for keeping the infrastructure reliable, deploying application releases, monitoring service health, and restoring service when an incident occurs.

During this lab, you are responsible for:

Monitoring: Detect when application availability decreases.
Incident response: Assess the impact and prioritize recovery.
Troubleshooting: Investigate Deployments, ReplicaSets, Pods, Services, probes, events, and logs.
Mitigation: Restore the last known healthy application version.
Verification: Confirm application availability and healthy Kubernetes status.
Communication: Provide clear updates to the incident commander and application team.
Root cause analysis: Explain why the failure occurred and how to prevent recurrence.

Your objective is not simply to restart a Pod. Your objective is to restore customer-facing service safely, understand the failure, and reduce the chance of recurrence.

2. The incident you will investigate

INC-2026-001

Simulated SEV-2

Banking frontend unavailable after a Kubernetes deployment

Business impact: Customers may be unable to access the banking frontend during the failed rollout.

Technical symptoms: Deployment unavailable, Pods not Ready, application health checks failing.

Expected root cause: A newly released application version fails its readiness checks.

Recovery strategy: Roll back to the previous known-good Deployment revision.

What happened?

The development team released a new version of the banking frontend.

The new container starts, but a configuration defect causes its HTTP readiness endpoint to fail. Kubernetes therefore cannot mark the new Pods as Ready.

As the rollout progresses, the old working Pods are replaced. Eventually, the application has no Ready endpoints, and customers cannot access the service.

Prometheus detects the decrease in available replicas. Grafana displays the outage and can trigger an alert.

You are the engineer responsible for diagnosing the failure and restoring the previous working release.

What success looks like

At the end of the incident:

The Deployment is restored to its previous healthy revision.
Two frontend replicas are Ready.
The Kubernetes Service has healthy endpoints.
The application returns HTTP 200.
Monitoring shows recovery.
You have recorded the root cause, commands, evidence, and preventive actions.

Do not invent recovery times, financial losses, customer counts, or performance improvements. Measure your actual lab results and use those numbers in your interview explanation.

3. Understand the production architecture

Banking customers

Customer requests

Application access layer

Production would typically use HTTPS and an ALB or Ingress

Kubernetes Service

banking-service · ClusterIP · Port 80

Frontend Pod 1

NGINX · Port 80

Frontend Pod 2

NGINX · Port 80

Prometheus collects Kubernetes health metrics. Grafana visualizes them and evaluates alert rules.

In this lab, the frontend is represented by an NGINX container. It is not a complete banking system with authentication, payment processing, or a database. The incident concerns the frontend availability layer.

4. Lab prerequisites

From the previous lesson, you should already have:

Resource	Expected state
EKS cluster	Active
Worker nodes	Ready
Namespace	`banking-incident-lab`
Deployment	`banking-frontend`
Replicas	2/2 Ready
Service	`banking-service`
Prometheus	Running
Grafana	Running

Use the same cluster and namespace as yesterday. Do not delete or reinstall the monitoring stack.

Part A — Establish the healthy production baseline

Step 1. Verify your AWS and Kubernetes environment

Run these commands on the machine where your kubectl is configured.

kubectl config current-context

kubectl get nodes -o wide

kubectl get pods -n monitoring

kubectl get deployment -n banking-incident-lab

kubectl get pods -n banking-incident-lab -o wide

kubectl get svc -n banking-incident-lab

Expected Deployment:

NAME               READY   UP-TO-DATE   AVAILABLE
banking-frontend   2/2     2            2

Student task: Explain why READY 2/2 matters and how Kubernetes determines whether a Pod is ready.

Step 2. Inspect the readiness probe

kubectl describe deployment banking-frontend \
  -n banking-incident-lab

Look for:

Readiness: http-get http://:80/

The readiness probe is configured to request / on port 80.

The kubelet runs the probe. Successful HTTP responses (status codes 200–399) allow the container to become Ready.

A failed readiness probe does not necessarily restart the container. Instead, the Pod becomes Not Ready and is excluded from normal Service traffic.

Step 3. Verify the Service endpoints

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o wide

Then:

kubectl get endpoints banking-service \
  -n banking-incident-lab

You should see two Pod addresses associated with the Service.

The EndpointSlice output is the preferred way to inspect endpoint readiness in modern Kubernetes. The legacy Endpoints command is included because it is still common in interviews and older runbooks.

Step 4. Verify the application responds

kubectl run baseline-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -sS -o /dev/null \
  -w "HTTP %{http_code}\n" \
  --max-time 10 \
  http://banking-service

Expected:

HTTP 200

A temporary warning about attaching to the diagnostic Pod is acceptable if the curl command still returns HTTP 200.

Evidence to save: Terminal output showing two Ready replicas and a successful HTTP response.

Step 5. Save the healthy Deployment configuration

This is an essential production habit: preserve the current state before making changes.

mkdir -p ~/banking-incident-lab/evidence

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o yaml \
  > ~/banking-incident-lab/evidence/deployment-before.yaml

kubectl rollout history deployment/banking-frontend \
  -n banking-incident-lab

Do not manually edit the saved baseline file during the incident.

Part B — Configure Prometheus and Grafana

Step 6. Open Grafana

Use the Grafana instance you configured in the previous lesson.

If Grafana is not accessible, start the port-forward on the management machine:

kubectl port-forward \
  -n monitoring \
  svc/monitoring-grafana \
  3000:80 \
  --address 127.0.0.1

If you access Grafana through an EC2 management machine, use your existing SSH tunnel to reach its localhost port securely.

Step 7. Create a production monitoring dashboard

In Grafana, go to Dashboards → New → New dashboard → Add visualization and select your Prometheus data source.

Create these four panels.

Panel	Visualization	Purpose
Available replicas	Stat	Number of available frontend replicas
Desired replicas	Stat	Number Kubernetes should maintain
Unavailable replicas	Stat	Detect loss of Deployment availability
Container restarts	Time series	Track application restarts

Panel 1 — Available replicas

kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Healthy value: 2

Panel 2 — Desired replicas

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Healthy value: 2

Panel 3 — Unavailable replicas

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
-
kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Healthy value: 0

Panel 4 — Container restarts

sum(
  kube_pod_container_status_restarts_total{
    namespace="banking-incident-lab",
    pod=~"banking-frontend-.*"
  }
) or vector(0)

Healthy initial value: normally 0.

Save the dashboard as:

JumpToTech — Banking Production Incident Monitoring

If a query returns No Data, verify the Prometheus data source, kube-state-metrics, namespace, and Deployment labels before proceeding. Do not treat missing metrics as a healthy result.

Step 8. Create an incident alert

Navigate to Alerting → Alert rules → New alert rule.

Configure:

Field	Value
Alert name	`BankingFrontendUnavailable`
Data source	Prometheus
Evaluation interval	1 minute
Pending period	1 minute
Condition	Last value greater than 0
Severity label	`warning`
Folder	`Banking Production Lab`
Notification	Your configured email contact point

Use this PromQL expression:

kube_deployment_spec_replicas{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}
-
kube_deployment_status_replicas_available{
  namespace="banking-incident-lab",
  deployment="banking-frontend"
}

Set the alert condition to Last IS ABOVE 0. Depending on your Grafana version, this may require a Reduce expression and a Threshold expression.

Use these annotations:

Summary: Banking frontend has unavailable replicas.

Description: The banking-frontend Deployment has fewer available replicas than desired. Investigate Pod readiness, Deployment rollout status, Kubernetes events, and Service endpoints.

For the lab, use No Data = No Data and Error = Error so that missing metrics or evaluation failures are not silently treated as healthy. These states should be monitored separately in a real production alerting design.

Before proceeding, verify the alert is saved and that the email contact point is selected by the notification policy.

Part C — Simulate the production incident

Controlled incident simulation

The next commands deliberately make the banking frontend unavailable. Execute them only in your dedicated training namespace, never in a real production banking environment.

Verify that kubectl config current-context points to your training cluster before continuing.

Step 9. Understand the failure before creating it

Our healthy NGINX container responds to:

GET /
HTTP 200 OK

We will introduce a bad readiness probe that requests a nonexistent endpoint:

GET /healthz
HTTP 404 Not Found

NGINX will still be running, but the kubelet will mark the container Not Ready.

To make the full outage reproducible, we will temporarily use the Deployment's Recreate strategy. This intentionally removes the old Pods before starting replacements. This is not a recommended production rollout strategy for a high-availability banking frontend.

This combination demonstrates two failures: a bad health-check configuration and a rollout strategy that allows total service interruption.

Step 10. Record the incident start time

date -u +"Incident started: %Y-%m-%dT%H:%M:%SZ"

Record the output in your incident worksheet.

Step 11. Configure the controlled rollout strategy

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=merge \
  -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

This changes the rollout strategy without changing the application template.

Confirm:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.strategy.type}{"\n"}'

Expected:

Recreate

Step 12. Introduce the faulty readiness probe

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"frontend","readinessProbe":{"httpGet":{"path":"/healthz","port":80},"initialDelaySeconds":5,"periodSeconds":5}}]}}}}'

This creates a new Deployment revision with a bad readiness endpoint.

Wait approximately 30–60 seconds.

Step 13. Observe the production failure

Run:

kubectl get pods -n banking-incident-lab

kubectl get deployment banking-frontend \
  -n banking-incident-lab

kubectl rollout status deployment/banking-frontend \
  -n banking-incident-lab \
  --timeout=30s

Expected symptoms may resemble:

NAME                                READY   STATUS
banking-frontend-xxxxxxxxxx-aaaaa   0/1     Running
banking-frontend-xxxxxxxxxx-bbbbb   0/1     Running

NAME               READY   UP-TO-DATE   AVAILABLE
banking-frontend   0/2     2            0

The rollout status command should time out because the new Pods never become available.

Important: A Pod can have STATUS Running while showing READY 0/1. These fields describe different aspects of Pod health.

Step 14. Confirm customer-facing impact

Inspect the endpoints:

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o yaml

Look for:

conditions:
  ready: false

Now test the application:

kubectl run incident-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -v --max-time 10 \
  http://banking-service

Expected: the request fails or times out because the Service has no Ready backend endpoints.

The exact network error can vary with the cluster networking implementation.

Record the actual error. Do not assume every failed Service request returns HTTP 503.

Part D — Investigate like a production SRE

Step 15. Establish the scope

First determine whether the problem affects only the banking application or the whole Kubernetes cluster.

kubectl get nodes

kubectl get pods -n banking-incident-lab -o wide

kubectl get pods -n monitoring

kubectl get deployment banking-frontend \
  -n banking-incident-lab

Ask yourself:

Are Worker Nodes Ready?
Are Prometheus and Grafana healthy?
Are the banking Pods Running?
Are the banking Pods Ready?
Is the issue isolated to the frontend?

Do not immediately restart every Pod. That can destroy useful evidence without fixing the underlying configuration problem.

Step 16. Examine Kubernetes events

kubectl get events \
  -n banking-incident-lab \
  --sort-by=.lastTimestamp

Look for an event resembling:

Warning  Unhealthy  Readiness probe failed:
HTTP probe failed with statuscode: 404

This is a strong clue that the application is running but failing its health check.

Step 17. Describe the affected Pods

kubectl describe pods \
  -n banking-incident-lab \
  -l app=banking-frontend

Find the Readiness field and the Events section.

You should observe that the probe is requesting /healthz instead of /.

Step 18. Examine application logs

kubectl logs \
  -n banking-incident-lab \
  -l app=banking-frontend \
  -c frontend \
  --tail=100 \
  --prefix=true

Look for NGINX requests to /healthz returning HTTP 404.

This is the evidence that connects the Kubernetes probe failures to the application endpoint.

Step 19. Compare the deployment revisions

kubectl rollout history deployment/banking-frontend \
  -n banking-incident-lab

Inspect the live readiness configuration:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}'

Expected:

/healthz

Compare it with the baseline configuration saved before the incident, where the path was /.

Root cause analysis

Primary configuration defect

The new Deployment revision used /healthz as its readiness endpoint, but the NGINX application did not serve that path.

Failure mechanism

Kubelet received HTTP 404, marked the new Pods Not Ready, and the Service had no Ready endpoints.

Outage-enabling rollout configuration

The intentionally selected Recreate strategy removed the healthy old Pods before the replacement Pods passed readiness checks.

Business impact in the simulation

The banking frontend became unavailable to clients using its Kubernetes Service.

Part E — Recover production service

Step 20. Perform a Kubernetes rollback

We have identified the defective Deployment revision. Rather than manually editing the new release during the outage, restore the previous working version.

kubectl rollout undo deployment/banking-frontend \
  -n banking-incident-lab

Check the recovery:

kubectl rollout status deployment/banking-frontend \
  -n banking-incident-lab \
  --timeout=180s

Expected:

deployment "banking-frontend" successfully rolled out

A Deployment rollback restores the previous Pod template. It does not automatically restore the Deployment's rollout strategy, so we must correct that separately.

Step 21. Restore the safer RollingUpdate strategy

kubectl patch deployment banking-frontend \
  -n banking-incident-lab \
  --type=merge \
  -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"}}}}'

With two replicas, this configuration allows a surge Pod while rounding the permitted unavailable count down to zero during normal rolling updates.

Verify:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.strategy.type}{"\n"}'

Expected:

RollingUpdate

Step 22. Verify recovery

kubectl get pods -n banking-incident-lab

kubectl get deployment banking-frontend \
  -n banking-incident-lab

kubectl get endpointslice \
  -n banking-incident-lab \
  -l kubernetes.io/service-name=banking-service \
  -o wide

The Deployment should return to 2/2 available.

Verify the readiness endpoint was restored:

kubectl get deployment banking-frontend \
  -n banking-incident-lab \
  -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}'

Expected:

Finally, test the Service again:

kubectl run recovery-client \
  -n banking-incident-lab \
  --image=curlimages/curl:8.11.1 \
  --restart=Never \
  --rm -i \
  --command -- \
  curl -sS -o /dev/null \
  -w "HTTP %{http_code}\n" \
  --max-time 10 \
  http://banking-service

Expected:

HTTP 200

Step 23. Record the recovery time

date -u +"Service restored: %Y-%m-%dT%H:%M:%SZ"

Calculate the elapsed time between the incident start and recovery.

This is your measured lab recovery duration. It is not automatically the same as production MTTR, which is typically calculated across multiple incidents.

Part F — Verify monitoring and notification

Return to the JumpToTech — Banking Production Incident Monitoring dashboard.

The expected pattern is:

Illustrative availability timeline

Example only. Replace these values with observations from your Grafana dashboard.

Desired replicasAvailable replicas

00.511.52HealthyFailureInvestigationRollbackRecovered

Check whether the alert transitioned through its expected states:

Normal → Pending → Firing → Resolved

Because alert evaluations run periodically, a short outage may not last long enough to enter Firing. If no alert fired, record that observation rather than claiming that an email was sent.

If an email was delivered, preserve the timestamp and alert name as evidence.

Part G — Your incident report

Every student must complete this worksheet using their own terminal outputs and observations.

Incident report — INC-2026-001

Complete after the hands-on exercise. Copy the report into your lab submission.

Engineer name

Incident start time (UTC)

How was the incident detected?

Observed application impact

Key commands and evidence

Root cause

Mitigation and recovery actions

Service restoration time (UTC)

Measured recovery duration

Preventive actions

Copy completed incident report

Prevention recommendations

A strong SRE does more than restore service. They identify engineering improvements.

For this incident, recommend:

Add automated readiness-endpoint checks to the CI/CD pipeline before deployment.
Use RollingUpdate with an appropriate availability budget for customer-facing applications.
Test application health checks in staging before promoting to production.
Monitor desired versus available replicas and alert on sustained unavailability.
Introduce canary deployments or progressive delivery for high-risk releases.
Maintain a documented rollback runbook and test it regularly.

Part H — Interview preparation

The question interviewers may ask

Tell me about a production incident you independently investigated and resolved. What was the impact, root cause, and outcome?

Sample answer in English

Production-Style Incident Response — AWS EKS Banking Application

During a hands-on production incident simulation, I was responsible for investigating and restoring a banking frontend application deployed on AWS EKS.

The application was running with two replicas behind a Kubernetes ClusterIP Service. We used Prometheus and Grafana to monitor Deployment availability, Pod readiness, and container restarts.

During a deployment, the frontend became unavailable. My initial investigation showed that the desired replica count was two, but zero replicas were available.

I checked the Deployment rollout status, inspected the Pods, reviewed Kubernetes events, and examined the application logs.

The containers were running, but their readiness probes were failing with HTTP 404 errors.

I traced the issue to an incorrect readiness probe path introduced in the new Deployment revision. The application was configured to check /healthz, but NGINX was only serving the root endpoint. The rollout also used a Recreate strategy, which removed the healthy Pods before the replacement Pods were ready.

After identifying the root cause, I rolled back the Deployment to the previous working revision and restored the RollingUpdate strategy.

I verified that both replicas returned to Ready, the Service had healthy endpoints, and the application responded with HTTP 200.

The key lesson was that a running container does not necessarily mean the application is ready to serve traffic.

To prevent recurrence, I recommended automated health-check validation in CI/CD, safer rollout strategies, and Prometheus/Grafana alerting for unavailable replicas.

This exercise gave me practical experience in Kubernetes incident investigation, root cause analysis, rollback, service recovery, and production reliability practices.

This is an interview practice script for a lab incident. Personalize it with your actual evidence and recovery time. Do not present the simulation as an incident you handled while employed by a bank.

Follow-up interview questions you must master

12 technical interview questions

Tap to reveal answers

1.Why were the Pods Running but not Ready?

2.What is the difference between readiness and liveness probes?

3.Why didn't Kubernetes automatically fix the incident?

4.What did the Service do when both Pods were Not Ready?

5.Why did the deployment cause a full outage?

6.Would the default RollingUpdate strategy behave the same way?

7.Why use kubectl rollout undo?

8.Did the rollback restore every Deployment setting?

9.How did Prometheus detect the problem?

10.What is the difference between mitigation and root cause analysis?

11.What would you communicate to stakeholders?

12.What is the most important prevention?

Part I — Student submission checklist

Evidence and deliverables

0/12 completed

Deployment healthy before incident: 2/2

Baseline application test: HTTP 200

Grafana dashboard with banking metrics

Screenshot showing failed readiness probes

Kubernetes events showing HTTP 404

Application logs showing failed /healthz requests

Deployment rollout history

Successful rollback output

Deployment recovered: 2/2

Application recovered: HTTP 200

Incident report with measured recovery time

Interview explanation practiced aloud

Final learning outcome

By completing this lab, you should be able to explain the entire incident lifecycle:

Detect

Investigate

Mitigate

Verify

Prevent

Your goal is not to memorize a fictional story. Your goal is to reproduce the failure, understand every command, independently restore the application, and explain your own results clearly in an interview.

This is the difference between knowing Kubernetes commands and demonstrating practical DevOps/SRE incident-response skills.

JumpToTech DevOps/SRE Bootcamp — Production Incident Response Lab, Day 2