JumpToTech DevOps/SRE Capstone Project EstateFlow — Enterprise AWS EKS Production Environment

1. The complete architecture

EstateFlow — target deployment architecture

Conceptual architecture. The database and Redis shown here are lab workloads; production environments would normally require stronger availability, backups, and disaster-recovery design.

What students will actually implement

Area	Implementation	What students must explain
Application	React + Node.js + PostgreSQL + Redis	Request flow, API, database dependencies
Infrastructure	Terraform, VPC, EKS, ECR, IAM	Infrastructure as Code
CI	GitHub Actions, tests, SonarQube, Trivy	Quality and security gates
CD	Helm + Argo CD	GitOps, drift, rollback
Networking	Services, ALB, Ingress, DNS	External and internal traffic
Security	RBAC, IAM, Pod Identity, NetworkPolicy	Authorization and isolation
Storage	StatefulSet, PVC, EBS CSI	Persistence and recovery
Scheduling	Affinity, anti-affinity, taints, tolerations	Pod placement
Reliability	HPA, PDB, probes, rolling updates	Availability and scaling
Operations	Prometheus, Grafana, alerting	Incident detection and diagnosis
Incidents	503, Pending, CrashLoopBackOff, PVC failures	Root cause, mitigation, postmortem

Important architecture decisions

For this lab, PostgreSQL and Redis run inside Kubernetes so students can learn StatefulSets, persistent volumes, and storage troubleshooting. For a higher-availability production design, I would normally evaluate Amazon RDS/Aurora and ElastiCache instead.

We will demonstrate the built-in workload controllers and selected AWS controllers, not indiscriminately install every Kubernetes controller. Each controller must have a reason to exist.

The application will have two deployment environments, estateflow-staging and estateflow-prod, plus dedicated infrastructure namespaces. We will only call the second one “prod” as a classroom environment.

2. Repository structure

Students will use one application repository and one GitOps repository.

jumptotech-estateflow/
├── frontend/
│   ├── src/
│   ├── Dockerfile
│   ├── package.json
│   └── nginx.conf
├── backend/
│   ├── src/
│   │   ├── server.js
│   │   └── db.js
│   ├── tests/
│   ├── Dockerfile
│   └── package.json
├── database/
│   └── init.sql
├── terraform/
│   ├── versions.tf
│   ├── providers.tf
│   ├── main.tf
│   ├── variables.tf
│   ├── outputs.tf
│   └── terraform.tfvars.example
├── helm/
│   └── estateflow/
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
├── k8s/
│   ├── rbac/
│   ├── security/
│   ├── scheduling/
│   ├── autoscaling/
│   └── storage/
├── argocd/
├── monitoring/
├── incidents/
├── scripts/
├── docs/
├── .github/
│   └── workflows/
├── docker-compose.yml
└── README.md

jumptotech-estateflow-gitops/
├── applications/
├── environments/
│   ├── staging/
│   └── prod/
└── charts/

We will use the following naming conventions consistently:

AWS_REGION=us-east-1
CLUSTER_NAME=estateflow-lab
APP_NAME=estateflow
STAGING_NAMESPACE=estateflow-staging
PROD_NAMESPACE=estateflow-prod

The original payments-incident-lab must not be included in any cleanup script.

3. Phase 0 — Environment and safety checks

Every student runs these commands before provisioning anything.

git --version
docker --version
aws --version
kubectl version --client
terraform version
helm version --short

aws sts get-caller-identity
aws configure get region

kubectl config get-contexts
kubectl get namespaces
kubectl get nodes -o wide

Students must confirm they are using their intended AWS account.

Cost warning: EKS, EC2 worker nodes, load balancers, EBS volumes, NAT gateways, public IPv4 addresses, and monitoring resources can incur charges. The cluster must be deliberately sized and destroyed after the project. A multi-AZ production-like environment is not a free-tier lab.

4. Phase 1 — Create the application

We will first build a working backend and frontend locally. Kubernetes must not be used to hide application defects.

4.1 Backend — Node.js API

mkdir -p jumptotech-estateflow/backend/src
cd jumptotech-estateflow/backend

cat > package.json <<'EOF'
{
  "name": "estateflow-backend",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "start": "node src/server.js",
    "test": "node --test"
  },
  "dependencies": {
    "express": "^4.21.2",
    "pg": "^8.13.1",
    "redis": "^4.7.0",
    "prom-client": "^15.1.3"
  }
}
EOF

npm install

Create backend/src/server.js:

import express from "express";
import pg from "pg";
import { createClient } from "redis";
import client from "prom-client";

const app = express();
app.use(express.json());

const pool = new pg.Pool({
  host: process.env.DB_HOST || "localhost",
  port: Number(process.env.DB_PORT || 5432),
  database: process.env.DB_NAME || "estateflow",
  user: process.env.DB_USER || "estateflow",
  password: process.env.DB_PASSWORD,
  connectionTimeoutMillis: 3000
});

const redis = createClient({
  url: process.env.REDIS_URL || "redis://localhost:6379",
  socket: { reconnectStrategy: false }
});

redis.on("error", err =>
  console.error("Redis error:", err.message)
);

client.collectDefaultMetrics();

const httpRequests = new client.Counter({
  name: "estateflow_http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "route", "status"]
});

app.use((req, res, next) => {
  res.on("finish", () => {
    httpRequests.inc({
      method: req.method,
      route: req.route?.path || req.path,
      status: String(res.statusCode)
    });
  });
  next();
});

app.get("/healthz", (_req, res) => {
  res.json({ status: "alive" });
});

app.get("/readyz", (_req, res) => {
  res.json({ status: "ready" });
});

app.get("/api/db-check", async (_req, res) => {
  try {
    await pool.query("SELECT 1");
    res.json({ database: "reachable" });
  } catch (err) {
    console.error("Database check failed:", err.message);
    res.status(503).json({ error: "database unavailable" });
  }
});

app.get("/api/properties", async (_req, res) => {
  try {
    const result = await pool.query(
      "SELECT id, title, city, price FROM properties ORDER BY id"
    );
    res.json(result.rows);
  } catch (err) {
    console.error("Property query failed:", err.message);
    res.status(503).json({ error: "database unavailable" });
  }
});

app.post("/api/bookings", async (req, res) => {
  const propertyId = Number(req.body.property_id);
  const customer = req.body.customer_name;

  if (!Number.isInteger(propertyId) || propertyId < 1 ||
      typeof customer !== "string" || !customer.trim()) {
    return res.status(400).json({ error: "invalid booking request" });
  }

  try {
    const result = await pool.query(
      `INSERT INTO bookings(property_id, customer_name)
       VALUES ($1, $2)
       RETURNING id, property_id, customer_name, status`,
      [propertyId, customer.trim()]
    );
    res.status(201).json(result.rows[0]);
  } catch (err) {
    console.error("Booking failed:", err.message);
    res.status(503).json({ error: "booking unavailable" });
  }
});

app.get("/api/cache-check", async (_req, res) => {
  try {
    if (!redis.isOpen) await redis.connect();
    await redis.ping();
    res.json({ redis: "reachable" });
  } catch (err) {
    console.error("Redis check failed:", err.message);
    res.status(503).json({ error: "cache unavailable" });
  }
});

app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
});

app.listen(8080, "0.0.0.0", () => {
  console.log("EstateFlow API listening on port 8080");
});

Teaching point: /healthz and /readyz deliberately do not require database access. A database failure can therefore produce HTTP 503 on business endpoints while Pods remain Running and Ready. This is central to our incident scenario.

For an actual production application, readiness behavior must be designed around whether a Pod can serve its intended traffic; this simplified probe is chosen to demonstrate dependency failures.

4.2 Database schema

Create database/init.sql:

CREATE TABLE IF NOT EXISTS properties (
    id SERIAL PRIMARY KEY,
    title VARCHAR(200) NOT NULL,
    city VARCHAR(100) NOT NULL,
    price NUMERIC(12,2) NOT NULL
);

CREATE TABLE IF NOT EXISTS bookings (
    id SERIAL PRIMARY KEY,
    property_id INTEGER NOT NULL
      REFERENCES properties(id),
    customer_name VARCHAR(200) NOT NULL,
    status VARCHAR(30) NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

INSERT INTO properties(title, city, price)
VALUES
('Chicago Downtown Apartment', 'Chicago', 2500),
('Schaumburg Family House', 'Schaumburg', 3200),
('Naperville Townhome', 'Naperville', 2800);

For repeated initialization, students should ensure seed data is not inserted twice; a later database migration will replace this development seed.

4.3 Backend Dockerfile

Create backend/Dockerfile:

FROM node:22-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --omit=dev

COPY src ./src

ENV NODE_ENV=production
EXPOSE 8080

USER node

CMD ["node", "src/server.js"]

4.4 Frontend

Create the React application:

cd ../

npm create vite@latest frontend -- \
  --template react-ts

cd frontend
npm install

Replace frontend/src/App.tsx:

import { useEffect, useState } from "react";

type Property = {
  id: number;
  title: string;
  city: string;
  price: string;
};

export default function App() {
  const [properties, setProperties] = useState<Property[]>([]);
  const [message, setMessage] = useState("");

  useEffect(() => {
    fetch("/api/properties")
      .then(r => {
        if (!r.ok) throw new Error("API unavailable");
        return r.json();
      })
      .then(setProperties)
      .catch(() => setMessage("Unable to load properties"));
  }, []);

  async function book(id: number) {
    const response = await fetch("/api/bookings", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        property_id: id,
        customer_name: "Lab Student"
      })
    });

    setMessage(response.ok
      ? "Booking submitted successfully"
      : "Booking service unavailable");
  }

  return (
    <main style={{ maxWidth: 850, margin: "auto", padding: 30 }}>
      <h1>EstateFlow</h1>
      <p>Find your next home.</p>
      <p role="status">{message}</p>

      {properties.map(property => (
        <article key={property.id}
          style={{ border: "1px solid #ccc", padding: 20, margin: 12 }}>
          <h2>{property.title}</h2>
          <p>{property.city} — ${property.price}/month</p>
          <button onClick={() => book(property.id)}>
            Request booking
          </button>
        </article>
      ))}
    </main>
  );
}

Create frontend/nginx.conf:

server {
    listen 8080;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /healthz {
        access_log off;
        return 200 "ok";
    }

    location /api/ {
        proxy_pass http://backend:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For
            $proxy_add_x_forwarded_for;
    }
}

The backend hostname above is for local Docker Compose. In Kubernetes, the frontend configuration will use the estateflow-api Service name.

Create frontend/Dockerfile:

FROM node:22-alpine AS build

WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginxinc/nginx-unprivileged:stable-alpine

COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/dist /usr/share/nginx/html

EXPOSE 8080

4.5 Local Docker Compose

Create docker-compose.yml in the repository root:

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: estateflow
      POSTGRES_USER: estateflow
      POSTGRES_PASSWORD: local-dev-only
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./database/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U estateflow"]
      interval: 5s
      retries: 10

  redis:
    image: redis:7-alpine

  backend:
    build: ./backend
    environment:
      DB_HOST: postgres
      DB_NAME: estateflow
      DB_USER: estateflow
      DB_PASSWORD: local-dev-only
      REDIS_URL: redis://redis:6379
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started

  frontend:
    build: ./frontend
    ports:
      - "8088:8080"
    depends_on:
      - backend

volumes:
  postgres_data:

Run:

docker compose up --build -d

curl -i http://localhost:8088/healthz

curl -i http://localhost:8088/api/properties

curl -i http://localhost:8088/api/db-check

curl -i -X POST \
  http://localhost:8088/api/bookings \
  -H "Content-Type: application/json" \
  -d '{"property_id":1,"customer_name":"JumpToTech Student"}'

docker compose ps

Expected healthy baseline: frontend HTTP 200, property list HTTP 200, database check HTTP 200, booking HTTP 201.

The students must save the actual responses.

5. Phase 2 — Terraform and AWS infrastructure

We will create a separate EKS cluster named estateflow-lab, rather than risking the existing training cluster.

For classroom work, use a dedicated AWS account if possible. The reference infrastructure uses two Availability Zones, managed node groups, ECR, and EKS Pod Identity.

Terraform provider

terraform/versions.tf

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

terraform/providers.tf

provider "aws" {
  region = var.aws_region
}

terraform/variables.tf

variable "aws_region" {
  type    = string
  default = "us-east-1"
}

variable "cluster_name" {
  type    = string
  default = "estateflow-lab"
}

variable "student_id" {
  type = string
}

VPC, EKS and ECR

terraform/main.tf

data "aws_availability_zones" "available" {
  state = "available"
}

locals {
  name = "${var.student_id}-${var.cluster_name}"

  azs = slice(
    data.aws_availability_zones.available.names,
    0,
    2
  )

  tags = {
    Project     = "EstateFlow"
    Environment = "training"
    ManagedBy   = "Terraform"
    Owner       = var.student_id
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = "${local.name}-vpc"
  cidr = "10.60.0.0/16"

  azs = local.azs

  private_subnets = [
    "10.60.1.0/24",
    "10.60.2.0/24"
  ]

  public_subnets = [
    "10.60.101.0/24",
    "10.60.102.0/24"
  ]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  tags = local.tags
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = local.name
  kubernetes_version = "1.34"

  endpoint_public_access = true

  enable_cluster_creator_admin_permissions = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  addons = {
    coredns = {}
    kube-proxy = {}

    vpc-cni = {
      before_compute = true
      configuration_values = jsonencode({
        enableNetworkPolicy = "true"
      })
    }

    eks-pod-identity-agent = {}

    aws-ebs-csi-driver = {
      most_recent = true
    }
  }

  eks_managed_node_groups = {
    general = {
      instance_types = ["t3.large"]

      min_size     = 2
      max_size     = 4
      desired_size = 2

      labels = {
        workload = "general"
      }
    }
  }

  tags = local.tags
}

resource "aws_ecr_repository" "frontend" {
  name                 = "${local.name}-frontend"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  tags = local.tags
}

resource "aws_ecr_repository" "backend" {
  name                 = "${local.name}-backend"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  tags = local.tags
}

terraform/outputs.tf

output "cluster_name" {
  value = module.eks.cluster_name
}

output "frontend_repository" {
  value = aws_ecr_repository.frontend.repository_url
}

output "backend_repository" {
  value = aws_ecr_repository.backend.repository_url
}

terraform/terraform.tfvars.example

aws_region  = "us-east-1"
cluster_name = "estateflow-lab"
student_id   = "student01"

Before applying: Confirm the requested Kubernetes version is currently supported by EKS and compatible with the selected add-ons, AWS provider, and Terraform module. The example version is a starting target, not a substitute for compatibility validation.

The initial lab uses a single NAT gateway to limit cost. This is not a fully resilient multi-AZ NAT design. For a higher-availability deployment, use appropriate per-AZ egress architecture or VPC endpoints.

Terraform commands

cd terraform

cp terraform.tfvars.example terraform.tfvars

terraform init

terraform fmt -recursive

terraform validate

terraform plan -out=estateflow.tfplan

Instructor approval checkpoint: Review cost, AWS quotas, subnet allocation, cluster version, add-on permissions, node capacity, and Terraform plan before applying.

terraform apply estateflow.tfplan

Then:

aws eks update-kubeconfig \
  --region us-east-1 \
  --name STUDENT01-estateflow-lab

kubectl get nodes -o wide

kubectl get pods -n kube-system

kubectl get daemonsets -n kube-system

Replace the cluster name with the exact Terraform output; it will include the student's identifier.

Infrastructure items that must be completed before application deployment

The Terraform baseline above is not yet the entire AWS platform. Students must also configure:

• EBS CSI driver IAM permissions through EKS Pod Identity or a supported IAM integration.
• AWS Load Balancer Controller IAM role, Pod Identity association, Helm installation, and webhook readiness.
• GitHub Actions OIDC identity with narrowly scoped ECR push permissions.
• ECR lifecycle policies and image retention.
• Terraform remote state in an S3 backend with locking and encryption.
• Restricted EKS API access, rather than leaving unrestricted public endpoint access for production.
• DNS, TLS certificates, and ALB listener rules.
• Worker-node sizing for monitoring, Argo CD, database, and application workloads.

Do not assume that creating the EKS cluster automatically installs the AWS Load Balancer Controller or grants EBS CSI storage permissions.

6. Phase 3 — Kubernetes workloads

First create the application namespaces:

kubectl create namespace estateflow-staging
kubectl create namespace estateflow-prod

Do not deploy the same StatefulSet database to both environments using shared volumes. Each environment must have its own data and storage.

6.1 PostgreSQL StatefulSet

Save as k8s/storage/postgres.yaml:

apiVersion: v1
kind: Service
metadata:
  name: estateflow-postgres
  namespace: estateflow-staging
spec:
  clusterIP: None
  selector:
    app: estateflow-postgres
  ports:
    - port: 5432
      name: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: estateflow-postgres
  namespace: estateflow-staging
spec:
  serviceName: estateflow-postgres
  replicas: 1
  selector:
    matchLabels:
      app: estateflow-postgres
  template:
    metadata:
      labels:
        app: estateflow-postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: estateflow
            - name: POSTGRES_USER
              value: estateflow
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: estateflow-db
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - pg_isready -U estateflow -d estateflow
            periodSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 512Mi
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: gp3
        resources:
          requests:
            storage: 10Gi

Create a database password without storing it in Git:

read -s "DB_PASSWORD?Database password: "
echo

kubectl create secret generic estateflow-db \
  -n estateflow-staging \
  --from-literal=password="$DB_PASSWORD"

unset DB_PASSWORD

For a classroom deployment, this is acceptable as an introductory Secret exercise. In a production design, use an external secret-management solution, encryption controls, and rotation. Do not commit Secret manifests containing plaintext passwords.

Before applying the StatefulSet, verify that the gp3 StorageClass exists and that the EBS CSI driver is authorized:

kubectl get storageclass

kubectl get csidrivers

kubectl get pods -n kube-system | grep ebs

If gp3 is absent, create a CSI-backed StorageClass with ebs.csi.aws.com, appropriate binding mode, and encryption. Do not proceed until the storage controller is operational.

kubectl apply -f k8s/storage/postgres.yaml

kubectl get statefulsets -n estateflow-staging

kubectl get pvc -n estateflow-staging

kubectl get pods -n estateflow-staging -o wide

Instructor explanation: StatefulSet gives PostgreSQL a stable identity and persistent volume association. It does not automatically provide database replication, backups, or high availability. Our single-replica PostgreSQL is deliberately a learning environment.

6.2 Redis StatefulSet

Create a separate Redis StatefulSet with one replica, its own headless Service, persistent volume, and resource limits. Redis must not share the PostgreSQL PVC.

Students must explain why Redis is not the source of truth for booking records.

6.3 Backend Deployment and Service

Save as k8s/backend.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: estateflow-api
  namespace: estateflow-staging
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: estateflow-api
  template:
    metadata:
      labels:
        app: estateflow-api
    spec:
      containers:
        - name: api
          image: REPLACE_WITH_BACKEND_ECR_IMAGE
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: DB_HOST
              value: estateflow-postgres
            - name: DB_NAME
              value: estateflow
            - name: DB_USER
              value: estateflow
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: estateflow-db
                  key: password
            - name: REDIS_URL
              value: redis://estateflow-redis:6379
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 10
          startupProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 5
            failureThreshold: 12
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
---
apiVersion: v1
kind: Service
metadata:
  name: estateflow-api
  namespace: estateflow-staging
spec:
  type: ClusterIP
  selector:
    app: estateflow-api
  ports:
    - name: http
      port: 8080
      targetPort: http

Replace the image placeholder with the actual immutable ECR image URI before applying. The final GitOps deployment will use Helm rather than manually editing this file.

6.4 Frontend Deployment

The frontend needs a separate Deployment and ClusterIP Service.

Use three initial replicas for the classroom deployment only if node capacity allows. Configure:

• Port 8080
• /healthz readiness and liveness checks
• Resource requests and limits
• RollingUpdate
• Pod anti-affinity or topology spread
• A Service named estateflow-frontend

The Kubernetes NGINX configuration must proxy /api/ to http://estateflow-api:8080, not to the Docker Compose hostname backend.

6.5 HPA

Save as k8s/autoscaling/api-hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: estateflow-api-hpa
  namespace: estateflow-staging
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: estateflow-api
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Prerequisites:

kubectl top nodes
kubectl top pods -n estateflow-staging

If metrics are unavailable, fix Metrics Server before expecting CPU-based HPA scaling.

6.6 PodDisruptionBudget

k8s/autoscaling/api-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: estateflow-api-pdb
  namespace: estateflow-staging
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: estateflow-api

Explain that a PDB limits certain voluntary disruptions, such as node drains. It does not prevent all crashes, node failures, or application bugs.

6.7 Scheduling — affinity, taints and tolerations

Add this to a workload Pod specification only when the cluster has eligible nodes:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: estateflow-api
          topologyKey: kubernetes.io/hostname

For AZ distribution:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: estateflow-api

For the taints lab, create a separate dedicated node group before applying any taint.

Example:

kubectl taint node NODE_NAME \
  dedicated=estateflow:NoSchedule

Matching Pod toleration:

tolerations:
  - key: dedicated
    operator: Equal
    value: estateflow
    effect: NoSchedule

A toleration allows scheduling onto a tainted node; it does not force scheduling there. To target that node group, use appropriate node affinity or a node selector as well.

Never taint the only available worker node in a shared classroom cluster. That could disrupt CoreDNS, monitoring, and other workloads.

6.8 NetworkPolicy

The application requires a deliberate communication model:

Frontend

Backend API

PostgreSQL

Frontend

Direct PostgreSQL access

NetworkPolicy controls permitted network traffic. It does not delete Pod IP addresses or remove Pods.

k8s/security/postgres-networkpolicy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-api-only
  namespace: estateflow-staging
spec:
  podSelector:
    matchLabels:
      app: estateflow-postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: estateflow-api
      ports:
        - protocol: TCP
          port: 5432

This isolates PostgreSQL ingress and permits connections from matching API Pods in the same namespace. Other applicable policies can add permissions, so inspect all NetworkPolicies before claiming isolation.

On AWS EKS, NetworkPolicy enforcement requires a compatible network-policy-capable implementation. With AWS VPC CNI, verify that NetworkPolicy support is enabled and the policy agent is running.

kubectl get daemonset aws-node -n kube-system

kubectl get networkpolicy -n estateflow-staging

kubectl describe networkpolicy \
  postgres-allow-api-only \
  -n estateflow-staging

6.9 RBAC

k8s/rbac/read-only.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: estateflow-observer
  namespace: estateflow-staging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: estateflow-read-only
  namespace: estateflow-staging
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: estateflow-observer-binding
  namespace: estateflow-staging
subjects:
  - kind: ServiceAccount
    name: estateflow-observer
    namespace: estateflow-staging
roleRef:
  kind: Role
  name: estateflow-read-only
  apiGroup: rbac.authorization.k8s.io

Verify:

kubectl auth can-i list pods \
  --as=system:serviceaccount:estateflow-staging:estateflow-observer \
  -n estateflow-staging

kubectl auth can-i delete pods \
  --as=system:serviceaccount:estateflow-staging:estateflow-observer \
  -n estateflow-staging

Expected: listing allowed, deletion denied.

6.10 IAM and Pod Identity

Students must distinguish:

Kubernetes RBAC controls what an identity can do through the Kubernetes API.

AWS IAM controls access to AWS APIs and resources.

EKS Pod Identity allows an authorized Kubernetes ServiceAccount to obtain AWS credentials through an IAM role association.

For the lab, create a dedicated ServiceAccount for a workload that reads a narrowly scoped S3 object or CloudWatch resource. Associate it with a least-privilege IAM role using Terraform and EKS Pod Identity. Do not grant application Pods administrator permissions.

6.11 Ingress and AWS ALB

After the AWS Load Balancer Controller is installed and healthy, create:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: estateflow-ingress
  namespace: estateflow-staging
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: estateflow-api
                port:
                  number: 8080
          - path: /
            pathType: Prefix
            backend:
              service:
                name: estateflow-frontend
                port:
                  number: 8080

The ALB controller reconciles this Ingress into AWS load-balancer resources. The ALB routes requests to the selected backend target group. The Kubernetes Service defines the backend Pods; the exact packet forwarding depends on the chosen target mode and networking implementation.

Before calling this production-ready, add HTTPS, an ACM certificate, Route 53 DNS, access logs, health-check configuration, and appropriate security groups.

7. Phase 4 — GitHub Actions CI

The delivery sequence will be:

Developer opens Pull Request
          |
          v
    GitHub Actions
          |
          +-- npm install / test / build
          |
          +-- SonarQube quality analysis
          |
          +-- Trivy vulnerability scan
          |
          v
     Merge approved
          |
          v
    Build Docker images
          |
          v
    Push to Amazon ECR
          |
          v
   Update GitOps image tags
          |
          v
      Argo CD sync
          |
          v
     Kubernetes rollout

GitHub permissions

Create an AWS IAM role that trusts GitHub's OIDC provider. Restrict its trust policy to the intended repository, branch or GitHub environment, and audience sts.amazonaws.com.

Grant only the ECR permissions required to authenticate and push to the two EstateFlow repositories.

Do not put AWS access keys in GitHub Secrets. Use OIDC federation.

CI workflow

Save as .github/workflows/ci.yml:

name: EstateFlow CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  backend:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: backend
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
          cache-dependency-path: backend/package-lock.json

      - run: npm ci
      - run: npm test

  frontend:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
          cache-dependency-path: frontend/package-lock.json

      - run: npm ci
      - run: npm run build

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Trivy filesystem scan
        uses: aquasecurity/trivy-action@0.33.1
        with:
          scan-type: fs
          scan-ref: .
          severity: HIGH,CRITICAL
          exit-code: "1"

The backend currently has no committed test suite. Before making npm test a required branch-protection check, students must add tests for /healthz, input validation, and booking/database error handling.

For a production pipeline, pin GitHub Actions to reviewed commit SHAs and establish dependency-update procedures.

SonarQube

Create sonar-project.properties:

sonar.projectKey=estateflow
sonar.projectName=EstateFlow
sonar.sources=frontend/src,backend/src
sonar.sourceEncoding=UTF-8

Configure a SonarQube server, token, and quality gate. Add a dedicated CI job that runs SonarScanner and waits for the quality-gate result.

Students must understand that SonarQube performs static code analysis; it does not replace unit tests, dependency scanning, image scanning, or runtime monitoring.

CD — ECR push workflow

After CI passes on main, a protected publishing workflow must:

1. Assume the AWS IAM role through GitHub OIDC.
2. Authenticate to ECR.
3. Build both Docker images.
4. Scan the built images.
5. Push immutable image tags derived from the Git commit SHA.
6. Record the image digests.
7. Open a pull request updating the GitOps repository.

Example build commands:

docker build \
  -t "$FRONTEND_REPO:$GITHUB_SHA" \
  ./frontend

docker build \
  -t "$BACKEND_REPO:$GITHUB_SHA" \
  ./backend

docker push "$FRONTEND_REPO:$GITHUB_SHA"

docker push "$BACKEND_REPO:$GITHUB_SHA"

These variables must be populated by the authenticated CI job. The image repositories must already exist.

Important: Argo CD should not be given AWS credentials to build or push images. Its responsibility is reconciling the approved Kubernetes configuration from Git.

8. Phase 5 — Helm

Helm packages the application's Kubernetes resources into reusable, parameterized charts.

Create:

helm create helm/estateflow

Remove or replace the generated example templates that do not match our application.

helm/estateflow/Chart.yaml:

apiVersion: v2
name: estateflow
description: EstateFlow Kubernetes application
type: application
version: 0.1.0
appVersion: "1.0.0"

Example helm/estateflow/values.yaml:

frontend:
  replicas: 2
  image:
    repository: REPLACE_FRONTEND_ECR_URI
    tag: REPLACE_WITH_GIT_SHA

backend:
  replicas: 2
  image:
    repository: REPLACE_BACKEND_ECR_URI
    tag: REPLACE_WITH_GIT_SHA

database:
  host: estateflow-postgres
  name: estateflow
  user: estateflow

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 512Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 6
  targetCPUUtilizationPercentage: 60

ingress:
  enabled: true
  className: alb

Students will convert the backend Deployment, frontend Deployment, Services, Ingress, HPA, PDB, and application configuration into Helm templates.

Validate before deployment:

helm lint helm/estateflow

helm template estateflow \
  helm/estateflow \
  --namespace estateflow-staging \
  > /tmp/estateflow-rendered.yaml

kubectl apply \
  --dry-run=client \
  -f /tmp/estateflow-rendered.yaml

Do not use a Helm chart containing unreviewed duplicate resources already managed by Terraform or another controller.

9. Phase 6 — Argo CD GitOps

Install Argo CD into its own namespace after checking cluster capacity and the supported installation version.

Create an Argo CD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: estateflow-staging
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_ORG/jumptotech-estateflow-gitops.git
    targetRevision: main
    path: charts/estateflow
    helm:
      valueFiles:
        - ../../environments/staging/values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: estateflow-staging
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The chart path and values-file path must be verified against the actual GitOps repository layout. A common simplification is to keep environment values inside the chart directory, avoiding cross-directory Helm value-file restrictions.

For production-style operation, create an Argo CD AppProject with explicit source-repository, namespace, and resource restrictions instead of relying on the unrestricted default project.

Students must verify:

kubectl get applications -n argocd

kubectl get deployments -n estateflow-staging

kubectl get pods -n estateflow-staging

kubectl rollout status deployment/estateflow-api \
  -n estateflow-staging

Argo CD incident demonstration

Change the approved image tag in the GitOps repository to a nonexistent tag.

Expected possible result: new Pods enter ImagePullBackOff, the rollout stalls, and Argo CD reports an unhealthy application.

Restore the previous known-good image tag in Git, synchronize Argo CD, and verify the rollout.

For GitOps-managed applications, the durable correction must be committed to Git. A manual kubectl rollout undo may be overwritten by Argo CD reconciliation.

10. Phase 7 — Monitoring and alerting

Install a compatible Prometheus Operator stack, including Prometheus, Grafana, and Alertmanager. If the existing classroom cluster already has monitoring, do not install a duplicate stack without checking capacity and ownership.

Application metrics

Our backend exposes:

/metrics

Prometheus can scrape the backend using a ServiceMonitor when the operator and ServiceMonitor CRD are installed.

Example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: estateflow-api
  namespace: estateflow-staging
  labels:
    release: monitoring
spec:
  namespaceSelector:
    matchNames:
      - estateflow-staging
  selector:
    matchLabels:
      app: estateflow-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

The Service must expose a port named http and have the matching label. The Prometheus instance must also be configured to select this ServiceMonitor.

Grafana dashboards

Students will create dashboards showing:

Metric	PromQL example
Available API replicas	kube_deployment_status_replicas_available{namespace="estateflow-staging",deployment="estateflow-api"}
Desired replicas	kube_deployment_spec_replicas{namespace="estateflow-staging",deployment="estateflow-api"}
Pod restarts	sum(increase(kube_pod_container_status_restarts_total{namespace="estateflow-staging"}[5m]))
HTTP 503 rate	sum(rate(estateflow_http_requests_total{status="503"}[5m]))
API request rate	sum(rate(estateflow_http_requests_total[5m]))

For the application HTTP metrics, add the Prometheus scrape labels actually present in your deployment to avoid mixing staging and production.

Alerts

Create alerts for:

• API HTTP 503 errors above a defined threshold.
• Available replicas below desired replicas.
• Persistent CrashLoopBackOff.
• Excessive Pod restarts.
• High CPU utilization.
• PostgreSQL connectivity failures.
• PVC capacity or availability issues.

A real SRE team should tune thresholds using observed traffic, error budgets, and service-level objectives. A classroom alert can be deliberately sensitive to make the incident visible.

PagerDuty

Connect a dedicated training PagerDuty service through Alertmanager or Grafana.

Students should experience the full sequence:

Prometheus detects condition
        ↓
Alertmanager / Grafana Alerting
        ↓
PagerDuty incident
        ↓
Primary on-call acknowledges
        ↓
Incident investigation
        ↓
Recovery verification
        ↓
Incident resolution
        ↓
Postmortem

Do not send classroom alerts to a real production on-call service.

11. Phase 8 — Main production incident: HTTP 503

This is the incident every student will investigate and present in their capstone.

SEV-1 training incident

Service degraded

EstateFlow booking requests are failing

Incident start

2:13 AM — scenario

Customer-facing symptom

HTTP 503

Kubernetes

Pods Running / Ready

Suspected dependency

PostgreSQL

The website is reachable, but booking requests cannot complete. The on-call engineer must identify the actual failure before attempting a rollback.

11.1 Incident team

Role	Responsibility
Primary SRE	Acknowledge PagerDuty, investigate, coordinate recovery
Incident Commander	Set priorities, coordinate responders, maintain timeline
Backend Engineer	Review API behavior, exceptions, connection pools
DBA	Verify PostgreSQL health, connectivity, and database state
Cloud/Platform Engineer	Investigate EKS, CNI, nodes, networking, IAM
Security Engineer	Review NetworkPolicies and security implications
Release Engineer	Review deployments, Argo CD history, recent changes
Communications Lead	Provide stakeholder updates

For a small organization, one person may hold multiple roles. Students should learn the responsibilities, not assume every incident has eight people online.

11.2 Confirm healthy baseline

Before injecting the failure:

kubectl get pods -n estateflow-staging

kubectl get deployments -n estateflow-staging

kubectl get svc -n estateflow-staging

kubectl get endpointslices -n estateflow-staging

Start a local port-forward:

kubectl port-forward \
  -n estateflow-staging \
  svc/estateflow-api \
  18080:8080

In another Terminal:

curl -i http://127.0.0.1:18080/healthz

curl -i http://127.0.0.1:18080/readyz

curl -i http://127.0.0.1:18080/api/db-check

curl -i http://127.0.0.1:18080/api/properties

Expected baseline: HTTP 200 on all four endpoints.

Record the actual timestamp, screenshots, Pod names, Deployment revisions, and API responses.

11.3 Introduce the controlled outage

Only in estateflow-staging, apply a NetworkPolicy that selects PostgreSQL and denies new ingress connections:

incidents/01-block-postgres.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: incident-block-postgres
  namespace: estateflow-staging
spec:
  podSelector:
    matchLabels:
      app: estateflow-postgres
  policyTypes:
    - Ingress
  ingress: []

Important: If postgres-allow-api-only is already present, this deny policy will not override it. Kubernetes NetworkPolicy allow rules are additive. For this controlled test, first inspect the existing policies and temporarily remove the allow policy through the approved staging change process.

kubectl get networkpolicy \
  -n estateflow-staging

kubectl describe networkpolicy \
  postgres-allow-api-only \
  -n estateflow-staging

Then, after recording the original state:

kubectl delete networkpolicy \
  postgres-allow-api-only \
  -n estateflow-staging

kubectl apply \
  -f incidents/01-block-postgres.yaml

If Argo CD manages the policy, pause or coordinate reconciliation for this controlled experiment and restore the intended policy through GitOps afterward.

Existing established database connections may survive policy changes temporarily. To test the failure reliably, force a new database connection from the application in the staging environment. Do not restart a production database or delete unrelated Pods.

11.4 Investigate the outage

First check the API:

curl -i http://127.0.0.1:18080/readyz

curl -i http://127.0.0.1:18080/api/db-check

Expected result after enforcement and new connection attempts:

GET /readyz
HTTP/1.1 200 OK

GET /api/db-check
HTTP/1.1 503 SERVICE UNAVAILABLE

{"error":"database unavailable"}

Now investigate Kubernetes:

kubectl get pods \
  -n estateflow-staging \
  -o wide

kubectl describe deployment estateflow-api \
  -n estateflow-staging

kubectl logs \
  -n estateflow-staging \
  deployment/estateflow-api \
  --all-pods=true \
  --tail=100

kubectl get networkpolicy \
  -n estateflow-staging

kubectl describe networkpolicy \
  incident-block-postgres \
  -n estateflow-staging

kubectl get events \
  -n estateflow-staging \
  --sort-by=.lastTimestamp

Students must identify the difference between a healthy Pod and a healthy business transaction.

A Pod can be Running and Ready while its application cannot connect to PostgreSQL.

11.5 The incident investigation decision tree

If the database is running and healthy, investigate connectivity and policy enforcement. If PostgreSQL itself is unhealthy, investigate storage, resource pressure, process logs, and database recovery instead.

11.6 Communicate during the incident

Students should practice realistic, concise communication.

Initial PagerDuty acknowledgement:

Acknowledged the EstateFlow booking failure alert. Investigating the HTTP 503 increase. Initial Kubernetes checks show the API Pods are Running and Ready. I am checking the database dependency and recent changes.

Message to the backend engineer:

Can you confirm whether the booking API is failing on database connection attempts? Please review recent application errors and connection-pool behavior.

Message to the platform/security engineer:

PostgreSQL appears healthy, but new API-to-database connections are failing. Please help verify NetworkPolicy enforcement, AWS VPC CNI policy-agent health, and any recent networking changes.

Incident Commander update:

We have isolated the failure to application connectivity with PostgreSQL. We are reviewing the active ingress policy and preparing a controlled restoration. Customer booking requests remain affected.

Only communicate findings that the team has actually verified.

11.7 Restore service

Remove the incident policy:

kubectl delete networkpolicy \
  incident-block-postgres \
  -n estateflow-staging

Restore the intended least-privilege allow policy:

kubectl apply \
  -f k8s/security/postgres-networkpolicy.yaml

If Argo CD owns the policy, restore the approved GitOps manifest and verify synchronization.

Test:

curl -i http://127.0.0.1:18080/api/db-check

curl -i -X POST \
  http://127.0.0.1:18080/api/bookings \
  -H "Content-Type: application/json" \
  -d '{"property_id":1,"customer_name":"Recovery Test"}'

Expected after recovery:

Database check: HTTP 200
Booking creation: HTTP 201

Also verify the newly created booking exists in PostgreSQL, application error rates decline, and alerts return to normal.

Do not close the incident based only on Pods showing 1/1 Running.

11.8 Postmortem template

EstateFlow Incident Postmortem

Incident ID: EF-INC-001\
Severity: SEV-1 (training scenario)\
Environment: estateflow-staging\
Incident Commander: [Student / instructor]\
Primary SRE: [Student]

Summary

The EstateFlow booking API experienced HTTP 503 errors because application-to-PostgreSQL connectivity was blocked by a Kubernetes NetworkPolicy.

Customer impact

[Record which synthetic transactions failed, the observed error rate, and the duration measured during the lab.]

Detection

The incident was detected through [Grafana alert / PagerDuty / synthetic monitoring]. Record the actual alert timestamp.

Timeline

Time	Event
[Time]	Healthy baseline verified
[Time]	NetworkPolicy change applied
[Time]	First HTTP 503 observed
[Time]	Alert acknowledged
[Time]	Application logs investigated
[Time]	Root cause confirmed
[Time]	Policy corrected
[Time]	HTTP 200 and booking HTTP 201 verified
[Time]	Incident resolved

Root cause

A NetworkPolicy selected the PostgreSQL Pod and blocked new ingress connections from the EstateFlow API.

The API's readiness endpoint did not depend on PostgreSQL, so Kubernetes continued reporting the application Pods as Ready even though database-dependent business requests failed.

Mitigation

The incident policy was removed and the intended least-privilege PostgreSQL access rule was restored.

Verification

• Database check returned HTTP 200.
• Synthetic booking returned HTTP 201.
• Booking record was verified in PostgreSQL.
• Error-rate metrics recovered.
• Monitoring alerts resolved.

Preventive actions

1. Add NetworkPolicy connectivity tests before deployment.
2. Require review of security-policy changes.
3. Add synthetic booking monitoring.
4. Improve runbooks for HTTP 503 investigation.
5. Add GitOps validation and staged rollout controls.

Evidence

Attach Terminal outputs, Grafana screenshots, PagerDuty event details, Git commit history, and recovery timestamps.

12. Phase 9 — Additional production-style incident labs

After students complete the main outage, they must investigate the following scenarios. Every scenario must have its own baseline, failure injection, diagnosis, recovery, and evidence.

Incident	Injected failure	Primary investigation	Expected recovery
01 — HTTP 503	PostgreSQL NetworkPolicy block	Logs, connectivity, policies	Restore permitted DB traffic
02 — ImagePullBackOff	Invalid ECR image tag	describe pod, image URI, ECR	Correct GitOps image reference
03 — CrashLoopBackOff	Application exits on startup	Pod logs, env, events	Correct configuration/code
04 — Readiness failure	Invalid /readyz probe path	describe pod, probe output	Correct probe
05 — HPA overload	CPU pressure and insufficient capacity	kubectl top, HPA events	Tune capacity and scaling
06 — PVC Pending	Unsupported StorageClass	PVC events, CSI controller	Restore correct storage configuration
07 — Node NotReady	Dedicated training node failure	Node conditions, kubelet, events	Recover/replace node
08 — RBAC Forbidden	Observer attempts unauthorized action	kubectl auth can-i	Grant only justified permissions
09 — IAM AccessDenied	Missing AWS role permission	AWS API error, Pod Identity	Correct least-privilege IAM policy
10 — Scheduling Pending	Taint without toleration	Scheduler events	Correct scheduling constraints
11 — Affinity failure	Required placement unavailable	Node labels, affinity, events	Restore eligible placement
12 — Argo CD OutOfSync	Manual resource change	Argo CD diff and history	Reconcile from Git
13 — ALB 503	No healthy application targets	Target health, Ingress, endpoints	Restore healthy targets
14 — DNS failure	DNS resolution problem	CoreDNS, Service DNS	Correct DNS issue
15 — Database recovery	Controlled database restart	StatefulSet, PVC, logs	Restore database and validate data
16 — PDB and node drain	Drain constrained by PDB	Eviction events, PDB	Safely adjust capacity
17 — RollingUpdate	New release fails readiness	ReplicaSets, rollout status	Revert GitOps release
18 — Secret failure	Missing Secret key	Pod events, environment refs	Restore Secret
19 — Resource limits	Memory exhaustion	OOMKilled, memory metrics	Correct memory sizing/leak
20 — CronJob failure	Backup job fails	Job logs, events, permissions	Repair and rerun backup

For the CronJob exercise, students must verify that the backup is recoverable by restoring into a separate database. A successful backup command alone does not prove recoverability.

13. Phase 10 — Kubernetes controllers: what to demonstrate

Students should understand the controller's responsibility and why the workload needs it.

Controller / resource	EstateFlow use
Deployment	Frontend and API stateless workloads
ReplicaSet	Maintains desired Pods through Deployment
StatefulSet	PostgreSQL and Redis lab workloads
DaemonSet	VPC CNI, node monitoring, log agents
Job	One-time database migration or verification
CronJob	Scheduled backup and maintenance
HPA	Scale API replicas using resource metrics
Ingress controller	Reconcile ALB routing
EBS CSI controller	Provision and attach persistent volumes
Argo CD controller	Reconcile desired state from Git
Node controller	Detect node health changes
EndpointSlice controller	Maintain Service endpoint information
Scheduler	Select eligible nodes for unscheduled Pods

Also demonstrate ServiceAccounts, Roles, RoleBindings, ClusterRoles, ClusterRoleBindings, ResourceQuotas, LimitRanges, PriorityClasses, Pod Security Admission, PodDisruptionBudgets, topology spread, and storage reclamation policies.

Not every resource should be used in the same application workload. For example, a ClusterRole should not replace a namespaced Role when only namespace-scoped permissions are required.

14. Phase 11 — Production-readiness requirements

Before students present EstateFlow as a production-style platform, complete the following checklist.

Project readiness

0 / 10 completed

Application

Automated backend tests, frontend build, input validation, dependency checks

Container security

Non-root runtime, image scanning, immutable tags, reviewed base images

AWS infrastructure

Terraform remote state, IAM least privilege, restricted cluster access

Networking

ALB controller, HTTPS, DNS, security groups, verified NetworkPolicy

Kubernetes

Requests/limits, probes, HPA, PDB, placement, safe rolling updates

Storage

EBS CSI IAM, encryption, backup, restore test, storage retention

GitOps

Separate GitOps repository, environment values, Argo CD project restrictions

Monitoring

Metrics, dashboards, alert routing, synthetic business transactions

Incident response

On-call roles, escalation, rollback, recovery validation, postmortem

Cost and cleanup

Budget alerts, resource inventory, cleanup approval and verification

This checklist is a teaching gate, not a certification that the system meets a particular organization's production standards.

15. Phase 12 — Interview preparation

Students should learn to describe the incident using the STAR method: Situation, Task, Action, Result.

DevOps/SRE Interview — EstateFlow Incident

Question: Tell me about a Kubernetes production incident you investigated.

During my DevOps/SRE capstone, I worked on EstateFlow, a production-style real-estate application deployed on AWS EKS.

The platform used React for the frontend, Node.js for the backend, PostgreSQL for persistent data, and Redis for caching. We used Terraform for AWS infrastructure, GitHub Actions for CI, Amazon ECR for container images, and Helm with Argo CD for GitOps deployments.

One of the incidents I investigated involved HTTP 503 errors on the booking API.

The interesting part was that Kubernetes reported the application Pods as Running and Ready, but customers in our synthetic test could not complete booking requests.

I first checked the Deployment, Pods, Services, and EndpointSlices using kubectl. Then I reviewed the application logs and tested the health and database endpoints separately.

The readiness endpoint returned HTTP 200, but the database check returned HTTP 503.

That told me the application process was running, but a downstream dependency was unavailable.

I checked PostgreSQL health and the active Kubernetes NetworkPolicies. We found that a policy was blocking new ingress connections to the PostgreSQL Pod.

I coordinated the investigation using the roles of on-call SRE, backend engineer, platform engineer, and security engineer.

We restored the intended least-privilege NetworkPolicy and tested the application again.

After the fix, the database endpoint returned HTTP 200 and the booking endpoint successfully created a booking with HTTP 201.

We also checked the monitoring dashboards and documented the incident timeline, root cause, and preventive actions.

The main lesson was that healthy Kubernetes Pods do not necessarily mean healthy business transactions. Application-level monitoring and dependency checks are essential.

If asked whether this occurred at a real employer: This was a hands-on production-style lab incident that I reproduced, investigated, and recovered in AWS EKS. It was not a customer outage at a bank.

Follow-up questions students must answer

1. Why were the Pods Running while the API returned HTTP 503?
2. What is the difference between a readiness probe and a liveness probe?
3. How does a NetworkPolicy select Pods?
4. How does AWS VPC CNI enforce network isolation?
5. Why would rolling back the API Deployment not necessarily fix a NetworkPolicy issue?
6. How does an ALB find healthy Kubernetes targets?
7. What is the difference between IAM and Kubernetes RBAC?
8. Why use a StatefulSet for PostgreSQL?
9. What happens if a PVC remains Pending?
10. How does Argo CD detect configuration drift?
11. What is the difference between maxSurge and maxUnavailable?
12. Why does an HPA need Metrics Server for CPU-based scaling?
13. How do taints and tolerations differ from node affinity?
14. What does a PodDisruptionBudget protect against?
15. How do you prove a business service has recovered?

Every student should answer with a command they used, an output they observed, and an explanation of what that output meant.

16. Phase 13 — Final project demonstration

Each student must demonstrate:

1. Application: Open EstateFlow, list properties, and create a booking.
2. AWS: Show Terraform configuration, EKS nodes, and ECR repositories.
3. CI/CD: Show a GitHub Actions run, image publishing, and Argo CD deployment.
4. Kubernetes: Explain Deployments, StatefulSets, Services, Ingress, HPA, PDB, and scheduling.
5. Security: Demonstrate RBAC authorization and PostgreSQL NetworkPolicy isolation.
6. Monitoring: Show Grafana metrics and an alert.
7. Incident: Reproduce HTTP 503, identify the cause, restore connectivity, and verify a booking.
8. Documentation: Submit a postmortem with actual timestamps and screenshots.

17. Phase 14 — Safe cleanup

Cleanup is a separate, instructor-approved activity. It must never be an unrestricted kubectl delete --all or terraform destroy against an unverified environment.

Before cleanup, students must:

• Export their incident evidence and final reports.
• Verify the active AWS account and Kubernetes context.
• Confirm that all resources belong to their dedicated EstateFlow project.
• Back up any data they need to retain.
• Identify ALBs, EBS volumes, ECR images, and other resources that might remain billable.

Then remove the EstateFlow GitOps Applications through their approved process, allow AWS controllers to clean up managed load balancers, and verify no resources are stranded.

Only after confirming Terraform manages the intended dedicated infrastructure should the student run:

cd terraform

terraform plan -destroy

Review the entire destroy plan with the instructor.

For a dedicated, disposable EstateFlow cluster only:

terraform destroy

Never run this against infrastructure that hosts payments-incident-lab or another student's resources.

18. Instructor implementation notes — important gaps to finish

The material above is a complete curriculum and implementation blueprint, with runnable examples for the core application and main incident. It is not yet a fully assembled, tested, one-command deployment repository. I want to distinguish that clearly before students copy it into their environments.

These remaining implementation tasks require actual files and validation:

Component	What still needs to be completed
Frontend	Kubernetes-specific NGINX configuration and final Helm template
Redis	StatefulSet, Service, PVC, authentication and configuration
Database	Schema migration Job, backup CronJob, restore procedure
Terraform	EBS CSI IAM, ALB Controller IAM, Pod Identity, GitHub OIDC, remote state
AWS ALB	Controller installation, HTTPS, DNS, security groups
GitHub Actions	Complete ECR publishing workflow, SonarQube job, GitOps PR automation
Helm	Full rendered templates for every application resource
Argo CD	Tested repository paths, AppProject, environment promotion
Security	Pod Security Admission, external secrets, complete policy tests
Monitoring	Tested ServiceMonitor selection, alert rules, PagerDuty routing
Incidents	Executable injection and recovery scripts for all 20 scenarios
Validation	Integration tests, failure recovery, capacity and cost checks

Those gaps matter. Students should not be told that copying this document alone creates a complete production environment.

Recommended classroom sequence

Week 1

Application, PostgreSQL, Redis, Docker Compose

Week 2

Terraform, VPC, EKS, IAM, ECR

Week 3

Kubernetes Deployments, StatefulSets, storage, Services

Week 4

ALB, Ingress, DNS, RBAC, NetworkPolicy, scheduling

Week 5

GitHub Actions, SonarQube, Trivy, Helm, Argo CD

Week 6

Prometheus, Grafana, PagerDuty, incident labs

Week 7

Incident simulation, postmortems, interview presentations

My recommendation for JumpToTech: Use this as the master student syllabus and architecture guide, but keep implementation in a real GitHub repository with tested source files. Students should build and verify each phase before moving on, rather than paste a large unvalidated script into AWS.

The final standard is straightforward: every student must be able to show the working application, demonstrate the failure, explain why it happened, restore service, and prove recovery using real evidence.