Instructions to students: Read each section, explain the purpose of every component, execute the practical commands, save evidence of the results, and practice answering the interview questions without reading the answers.
PART 1 — YOUR FIRST DAY AS A DEVOPS ENGINEER
1.1 Business scenario
Congratulations! You have joined Real Estate Company, a fictional American real estate technology company, as a DevOps Engineer.
The company provides an online platform where customers can:
- Browse properties available for sale or rent.
- Search for properties by location and price.
- View property details.
- Submit inquiries to real estate agents.
- Create accounts and manage their information, where those features are implemented.
Your responsibility is not to develop every feature yourself. Your responsibility is to build and maintain the infrastructure and delivery systems that allow developers to release their software safely.
The company has frontend developers, backend developers, QA engineers, cloud engineers, security engineers, database administrators, SRE engineers, and DevOps engineers.
You work with all these teams.
1.2 What do you do every day?
Imagine this is Monday morning.
At 9:00 AM, a frontend developer creates a pull request containing a new property search feature.
At 9:10 AM, GitHub Actions starts testing the application.
At 9:15 AM, Trivy discovers a HIGH vulnerability in a backend dependency.
At 9:20 AM, the developer asks why the pipeline is failing.
You investigate the logs, identify the vulnerable dependency, and collaborate with the backend developer to fix it.
At 10:00 AM, the tests pass and the pull request is approved.
After the merge, GitHub Actions attempts to publish Docker images to Amazon ECR.
However, the workflow fails with:
Not authorized to perform sts:AssumeRoleWithWebIdentity
You investigate the GitHub OIDC token and the AWS IAM role's trust policy.
You collaborate with the Cloud Security team to authorize the correct repository and branch.
Once the role assumption succeeds, the pipeline publishes images to ECR.
Argo CD then deploys the approved application version to Kubernetes.
Later, a Pod enters ImagePullBackOff.
You investigate the image tag and discover that the Kubernetes Deployment references an image that does not exist in ECR.
This is a realistic DevOps workday.
1.3 Responsibilities and collaboration
| Team | Responsibilities |
|---|---|
| Frontend | React components, UI, browser API calls |
| Backend | Express API, business logic, database queries |
| QA | Automated testing and acceptance criteria |
| DevOps | CI/CD, Docker, ECR, GitOps, deployment automation |
| Cloud / Platform | VPC, IAM, EKS, infrastructure provisioning |
| Security | IAM approvals, security controls, vulnerability review |
| DBA | PostgreSQL, schema, migrations, backup and recovery |
| SRE | Monitoring, incident response, SLOs, reliability |
In a small company, a DevOps Engineer may handle several of these responsibilities. In a larger company, ownership is divided across specialized teams.
Interview question: What are your responsibilities as a DevOps Engineer?
Answer:
"In my role, I support the software delivery lifecycle from source code to production. I maintain GitHub Actions pipelines, integrate quality and security checks, build and publish Docker images to Amazon ECR, and manage Kubernetes deployments using Argo CD. I collaborate with application developers, security engineers, cloud engineers, and SRE teams to troubleshoot deployment failures and maintain reliable services."
PART 2 — UNDERSTANDING THE REAL ESTATE APPLICATION
2.1 What is frontend?
Our frontend uses React and Vite.
React builds the user interface.
Vite provides development tooling and generates optimized production assets.
When a customer opens the real estate website, the frontend displays property cards, search fields, buttons, and forms.
The production build typically generates:
frontend/
src/
public/
package.json
package-lock.json
Dockerfile
dist/
dist/ contains the compiled static assets produced by the build. In a containerized deployment, Nginx can serve those files.
2.2 What is backend?
Our earlier Real Estate Company lab uses Node.js and Express.
The backend processes API requests, executes business logic, and can communicate with PostgreSQL.
Example request:
GET /api/properties HTTP/1.1
Host: realestate.example.com
Example response:
{
"properties": [
{
"id": 101,
"city": "Chicago",
"price": 350000
}
]
}
This JSON is an illustrative API contract, not a verified response from the current repository.
The backend might contain:
backend/
src/
server.js
routes/
controllers/
package.json
package-lock.json
Dockerfile
The exact files and endpoints must be checked in the repository.
2.3 How does frontend communicate with backend?
Example JavaScript:
const response = await fetch("/api/properties");
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const properties = await response.json();
console.log(properties);
The browser sends an HTTP request. The backend returns a response, usually JSON. React uses the response to update the interface.
Interview question: What is the difference between frontend and backend?
Answer:
"The frontend is responsible for the user interface and client-side interactions. The backend handles API requests, business logic, and database access. In our project, React communicates with a Node.js Express backend through HTTP APIs."
PART 3 — UNDERSTANDING THE REPOSITORY
Before modifying a project, inspect it.
git clone https://github.com/DariaBelts/real-estate-company.git
cd real-estate-company
pwd
ls -la
git status
git branch -a
find . -maxdepth 3 -type f
The GitHub URL above comes from the live student lab. Use your assigned fork if your instructor has provided one.
Inspect the application:
cat README.md
cat frontend/package.json
cat backend/package.json
find . -name Dockerfile
ls -la .github/workflows
ls -la k8s
Do not assume that every command works in every repository. First examine the scripts sections of the package files.
3.1 Frontend validation
cd frontend
npm ci
npm run lint
npm run build
Expected result: linting succeeds and the Vite build produces dist/.
3.2 Backend validation
cd ../backend
npm ci
npm run lint
npm test
If npm test is not defined, check package.json rather than inventing a test command.
For a backend that provides the start script:
npm start
The earlier lab used port 4000, so a locally running backend can be checked with:
curl -i http://localhost:4000/health
Use the actual health endpoint defined by the code.
3.3 What do these commands mean?
| Command | Explanation |
|---|---|
npm ci |
Installs dependencies from the lockfile reproducibly |
npm run lint |
Checks code against configured lint rules |
npm test |
Runs the project's test script |
npm run build |
Produces a production build when configured |
npm start |
Executes the application's start script |
Interview follow-up: Why use npm ci instead of npm install in CI?
"npm ci uses the committed lockfile, requires package and lockfile consistency, and performs a clean installation. This makes dependency installation more predictable across pipeline runs."
PART 4 — GIT AND THE SOFTWARE DELIVERY PROCESS
Developers should not normally make unreviewed changes directly to the protected production branch.
A common workflow is:
main
|
+-- feature-property-search
|
+-- code changes
+-- commit
+-- push
+-- pull request
+-- CI checks
+-- review
+-- merge to main
Practical lab
git switch main
git pull --ff-only origin main
git switch -c feature-property-search
git status
git add .
git commit -m "feat: improve property search"
git push -u origin feature-property-search
Create a pull request in GitHub.
Required checks should pass before merge.
Actual incident: Git push rejected
During our live class, a push failed:
! [rejected] feature-estate -> feature-estate (fetch first)
Updates were rejected because the remote contains work
that you do not have locally.
The student investigated and ran:
git fetch origin
git rebase origin/feature-estate
The rebase completed successfully.
The next step is to verify:
git status
git log --oneline -5
git push origin feature-estate
If the remote branch changed again, investigate before retrying. Do not use git push --force casually on shared branches.
Interview answer:
"We encountered a non-fast-forward Git push rejection because the remote branch contained commits that were missing locally. I fetched the remote changes, rebased my local commits, verified the branch state, and pushed the updated history. This preserved the remote changes without overwriting teammates' work."
PART 5 — CONTINUOUS INTEGRATION WITH GITHUB ACTIONS
5.1 What is CI?
Continuous Integration means automatically validating code changes when developers push commits or open pull requests.
In our Real Estate Company project, the workflow includes:
- Frontend lint and build.
- Backend lint and tests.
- Trivy filesystem security scan.
- Docker image build and ECR publishing after the required checks pass.
SonarQube can be added as a separate quality job or integrated with the existing workflow.
5.2 Example workflow structure
The following is a teaching example. It must be adapted to the repository's actual workflow, scripts, Node.js version, and AWS permissions.
name: Real Estate CI
on:
pull_request:
branches: [main]
push:
branches: [main]
permissions:
contents: read
jobs:
frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Frontend checks
working-directory: frontend
run: |
npm ci
npm run lint
npm run build
backend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: backend/package-lock.json
- name: Backend checks
working-directory: backend
run: |
npm ci
npm run lint
npm test
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Trivy filesystem scan
run: |
docker run --rm \
-v "$PWD:/workspace" \
aquasec/trivy:latest \
fs --scanners vuln \
--severity HIGH,CRITICAL \
--exit-code 1 /workspace
For a production pipeline, pin actions and scanner images to reviewed immutable versions or digests, and define a controlled vulnerability database/update strategy. latest is used here only to illustrate the scanner invocation.
5.3 What happens if one job fails?
Suppose the security scan finds a HIGH vulnerability.
The security job exits with code 1.
If publishing is configured to depend on the security job, Docker images must not be published.
That is expected security-gate behavior.
A failed security gate is not necessarily a broken pipeline. It can mean the pipeline successfully prevented an unsafe release.
PART 6 — THE REAL TRIVY INCIDENT
In our class, Trivy identified a HIGH-severity vulnerability involving the body-parser dependency in:
backend/package-lock.json
The training material records the vulnerable version as 1.20.8 and the finding as CVE-2026-11181. Students should verify current advisory details and compatible fixed versions before changing dependencies.
6.1 Troubleshooting
cd backend
npm ls body-parser
npm audit
Determine whether body-parser is a direct dependency or comes from another package.
Then inspect compatible updates:
npm outdated
npm update
npm test
After remediation:
cd ..
trivy fs \
--scanners vuln \
--severity HIGH,CRITICAL \
--exit-code 1 .
If the vulnerability remains, inspect the dependency chain and update the compatible parent package with the backend developer.
Do not suppress a HIGH finding merely to make the pipeline green.
6.2 Interview scenario
Question: Tell me about a CI/CD security incident.
STAR answer:
Situation: "During a pull request validation, our GitHub Actions security job detected a HIGH-severity vulnerability in a backend dependency."
Task: "I needed to investigate the finding and ensure that the pipeline did not publish an affected Docker image."
Action: "I reviewed the Trivy logs, identified the affected package in the lockfile, checked the dependency tree, coordinated a compatible update, and reran application tests and the security scan."
Result: "The workflow enforced the security gate, and we could proceed with the release only after the required checks passed."
Follow-up: Why didn't you disable Trivy?
"Because the scan was an intentional release control. Disabling it would hide the risk rather than address it. Any temporary exception would require documented security approval and compensating controls."
PART 7 — DOCKER AND AMAZON ECR
7.1 Why Docker?
Docker packages the application and its runtime into a portable image.
A Docker image is a packaged artifact.
A container is a running instance of that image.
For this project, we build separate images:
Frontend image → React static files + Nginx
Backend image → Node.js + Express application
7.2 Build locally
docker build \
-t real-estate-company-backend:v1 \
./backend
docker build \
-t real-estate-company-frontend:v1 \
./frontend
docker images
Run the backend:
docker run -d \
--name real-estate-backend \
-p 4000:4000 \
real-estate-company-backend:v1
This assumes the backend listens on port 4000 and has no missing database dependency.
Test:
docker ps
docker logs real-estate-backend
curl -i http://localhost:4000/health
7.3 Why ECR?
Amazon Elastic Container Registry stores container images.
Kubernetes pulls the required image from ECR when creating a Pod.
GitHub Actions
|
v
Docker build
|
v
Amazon ECR
|
v
Kubernetes worker node
|
v
Pod running application
7.4 Push images to ECR
The live student environment used AWS account 992734380025 in us-east-1. Use the account and region returned by your own AWS CLI session, not another student's credentials.
aws sts get-caller-identity
aws configure get region
Set variables for the lab:
export AWS_REGION=us-east-1
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity \
--query Account --output text)
export ECR_REGISTRY="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
Verify the repositories exist:
aws ecr describe-repositories \
--region "$AWS_REGION"
Authenticate Docker:
aws ecr get-login-password \
--region "$AWS_REGION" |
docker login \
--username AWS \
--password-stdin "$ECR_REGISTRY"
Tag and push the backend:
docker tag \
real-estate-company-backend:v1 \
"$ECR_REGISTRY/real-estate-company-backend:v1"
docker push \
"$ECR_REGISTRY/real-estate-company-backend:v1"
Repeat for frontend using the exact existing repository name.
Production practice: use a unique immutable image tag, such as the Git commit SHA, rather than reusing latest.
PART 8 — GITHUB OIDC AND AWS IAM
This was one of the most important real incidents in our class.
8.1 Why do we need OIDC?
GitHub Actions needs AWS permissions to push Docker images to ECR.
One older approach is to store AWS access keys as GitHub secrets.
A better approach is to use OpenID Connect (OIDC) and temporary AWS credentials.
GitHub provides a signed OIDC token. AWS Security Token Service validates the token against the IAM role's trust conditions and issues temporary credentials if the request is authorized.
8.2 The actual error
Our GitHub Actions workflow failed with:
Not authorized to perform sts:AssumeRoleWithWebIdentity
This means AWS did not authorize the requested web-identity role assumption.
It does not automatically mean that the ECR permissions policy is wrong.
We must first investigate the IAM role's trust relationship.
8.3 Trust policy versus permissions policy
Trust policy: Who is allowed to assume the role?
Permissions policy: What can the role do after it has been assumed?
For example:
GitHub repository + main branch
|
v
IAM Trust Policy
|
v
AssumeRoleWithWebIdentity
|
v
Temporary AWS Credentials
|
v
ECR Permissions Policy
|
v
Push Docker Image
8.4 OIDC claims from the student lab
The live workflow displayed a subject containing repository IDs:
repo:Medetbekovaa-hue@327037543/real-estate-company@1420924172:ref:refs/heads/main
The IAM trust policy had to match the actual subject issued by GitHub.
The important lesson is that GitHub's OIDC subject can be customized. Never assume every repository uses the standard name-only subject format.
8.5 AWS console troubleshooting
Open:
AWS Console
→ IAM
→ Roles
→ GitHubActions-RealEstate-ECR-Push
→ Trust relationships
→ Edit trust policy
Verify:
- Correct AWS account.
- Correct IAM role.
- GitHub OIDC identity provider.
- Audience
sts.amazonaws.com. - Exact authorized subject from the workflow.
- Branch restriction.
- Any additional organizational conditions.
The trust policy must match the actual token claims and the security team's approved identity model.
Do not replace a production trust policy with a generic wildcard.
8.6 GitHub Actions permissions
The ECR publishing job needs permission to request an OIDC token:
permissions:
contents: read
id-token: write
Then the AWS credentials action can assume the approved role:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::ACCOUNT_ID:role/GitHubActions-RealEstate-ECR-Push
aws-region: us-east-2
- name: Verify AWS identity
run: aws sts get-caller-identity
Replace ACCOUNT_ID, role name, and region with the values belonging to the current student's AWS environment.
Do not give OIDC permission to every job unnecessarily. Keep it scoped to the publishing job.
8.7 What if you need more IAM permissions?
Imagine that the OIDC role assumption now works, but the workflow receives:
AccessDeniedException:
not authorized to perform ecr:PutImage
The trust policy is no longer the primary issue. Now investigate the role's effective permissions, repository policy, and any organizational restrictions.
A DevOps Engineer should not request AdministratorAccess to fix an ECR push problem.
Instead, prepare a specific request for the Cloud Security team.
Hello Cloud Security Team,
Our Real Estate Company GitHub Actions pipeline successfully authenticates to AWS using OIDC, but the Docker image publishing step is failing with an AccessDenied error for ecr:PutImage.
Could you review the IAM permissions for the GitHub Actions publishing role?
Business purpose: Publish approved frontend and backend container images to our designated Amazon ECR repositories after CI and security checks pass.
Requested access: The minimum ECR permissions required for image publishing, scoped to the approved repositories.
Environment: Development/Training
Evidence: GitHub Actions run URL, AWS account, role ARN, repository ARNs, timestamp, and exact error message will be included in the access ticket.
Please let us know if additional approval or a change request is required.
Thank you,\
DevOps Engineering Team
8.8 How does this work in a real company?
A typical process is:
- DevOps opens a ServiceNow or Jira access request.
- The request identifies the role, AWS account, resources, required actions, and business reason.
- Cloud Security reviews least-privilege requirements.
- The IAM/Platform team updates the approved policy through Terraform or another controlled infrastructure process.
- The change is reviewed and deployed.
- DevOps reruns the failed pipeline and verifies the outcome.
The exact approval workflow depends on the organization.
Interview answer:
"Our GitHub Actions pipeline failed while assuming an AWS IAM role through OIDC. I inspected the workflow's OIDC claims and compared them with the IAM role trust conditions. We identified a mismatch in the expected repository subject format. I coordinated the trust-policy correction with the security team while preserving the repository and branch restrictions. After authentication was restored, we separately validated ECR publishing permissions."
PART 9 — KUBERNETES FUNDAMENTALS
9.1 Why Kubernetes?
Docker runs containers.
Kubernetes orchestrates containers across worker nodes.
It provides:
- Scheduling.
- Desired-state reconciliation.
- Pod replacement.
- Service discovery.
- Scaling.
- Rolling updates.
- Resource management.
- Integration with persistent storage and networking.
9.2 Kubernetes components
| Component | Purpose |
|---|---|
| Cluster | Kubernetes environment |
| Control plane | Manages desired state and cluster APIs |
| Worker node | Runs application workloads |
| Pod | Smallest deployable unit |
| Deployment | Manages stateless application replicas |
| ReplicaSet | Maintains the requested Pod count |
| Service | Provides stable access to selected Pods |
| EndpointSlice | Tracks backend endpoints for Services |
| Ingress | Defines HTTP/HTTPS routing |
| ConfigMap | Non-secret configuration |
| Secret | Sensitive configuration data |
| StatefulSet | Manages stateful workloads with stable identities |
| PVC | Requests persistent storage |
| RBAC | Controls Kubernetes API access |
| NetworkPolicy | Controls supported Pod network traffic |
9.3 Verify AWS and cluster context
aws sts get-caller-identity
aws eks list-clusters --region us-east-1
kubectl config current-context
kubectl get nodes -o wide
kubectl get namespaces
Never modify a shared cluster until you know which AWS account, region, and cluster you are connected to.
9.4 Deployments and Pods
Our demonstrated application used two backend Pods and two frontend Pods.
Example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: real-estate-company
spec:
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: backend
image: ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/real-estate-company-backend:VERSION
ports:
- containerPort: 4000
This is a simplified illustration. Production deployments should also define resource requests and limits, probes, security context, and application configuration.
9.5 Why two replicas?
Two replicas allow the application to tolerate some individual Pod failures and can support rolling updates with reduced interruption.
But two Pods do not automatically guarantee high availability.
For better resilience, replicas should be distributed across appropriate worker nodes and Availability Zones, and the application must have suitable readiness checks and dependency handling.
Interview question: What happens if a Pod crashes?
"The Deployment controller detects that the desired replica count is not satisfied and uses a ReplicaSet to create a replacement Pod. The scheduler assigns the new Pod to an eligible node, and the kubelet starts its containers."
PART 10 — KUBERNETES SERVICES AND HTTP REQUESTS
10.1 Why do we need Services?
Pod IP addresses can change.
A Service provides a stable virtual endpoint and selects matching Pods using labels.
Example backend Service:
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: real-estate-company
spec:
type: ClusterIP
selector:
app: backend
ports:
- name: http
port: 4000
targetPort: 4000
Apply only after verifying that the Deployment's Pod labels match the selector.
10.2 Test from inside Kubernetes
Start a temporary diagnostic Pod:
kubectl run curl-client \
-n real-estate-company \
--image=curlimages/curl:8.12.1 \
--restart=Never \
--command -- sleep 3600
Wait until the Pod is Running:
kubectl wait \
--for=condition=Ready pod/curl-client \
-n real-estate-company \
--timeout=120s
Test backend DNS and HTTP:
kubectl exec -n real-estate-company curl-client -- \
curl -i http://backend:4000/health
The response depends on the application's actual health endpoint and whether the container is healthy.
Delete the temporary Pod after testing:
kubectl delete pod curl-client \
-n real-estate-company
10.3 How does the request travel?
Client Pod
|
| DNS lookup: backend
v
CoreDNS
|
| Returns Service ClusterIP
v
Service virtual IP
|
| kube-proxy or another service dataplane
v
Selected Backend Pod IP
|
v
Express API
CoreDNS resolves the Service name.
EndpointSlices record eligible backend endpoints.
The cluster's service dataplane forwards traffic toward a selected backend endpoint.
The exact forwarding implementation depends on the cluster's networking mode.
10.4 HTTP status codes
| Code | Meaning |
|---|---|
| 200 | Request succeeded |
| 201 | Resource created |
| 400 | Bad request |
| 401 | Authentication required or invalid |
| 403 | Request forbidden |
| 404 | Resource not found |
| 500 | Internal server error |
| 502 | Gateway received an invalid upstream response |
| 503 | Service unavailable |
| 504 | Gateway timed out waiting for upstream |
10.5 Curl commands
curl -i http://localhost:4000/health
Displays response headers and body.
curl -I http://localhost:4000/health
Sends a HEAD request. It is useful only if the server supports HEAD for that endpoint.
curl -v http://localhost:4000/health
Displays verbose connection and request information.
curl -sS -o /dev/null -w '%{http_code}\n' \
http://localhost:4000/health
Prints the HTTP status code.
Interview follow-up: Why test from inside the cluster?
"Testing from a Pod helps isolate internal DNS, Service routing, network policies, and backend connectivity. A successful external browser test does not tell us exactly which internal components are healthy, and an external failure may occur before traffic reaches the backend."
PART 11 — ARGO CD AND GITOPS
11.1 Why do we use Argo CD?
GitHub Actions is responsible for Continuous Integration and artifact publishing in our design.
Argo CD is responsible for Continuous Delivery through GitOps.
Instead of giving GitHub Actions broad Kubernetes deployment permissions, we store the desired Kubernetes configuration in Git.
Argo CD compares that desired configuration with the actual cluster state.
When differences exist, it identifies the application as OutOfSync.
If automated synchronization is enabled, Argo CD can reconcile the cluster automatically.
11.2 CI versus CD
| GitHub Actions CI | Argo CD |
|---|---|
| Runs tests | Watches Git desired state |
| Performs linting | Detects configuration drift |
| Runs Trivy | Applies Kubernetes resources |
| Builds images | Tracks sync and health |
| Pushes images to ECR | Supports GitOps rollback workflows |
11.3 Our real classroom result
Initially, the Argo CD application displayed:
Health: Healthy
Sync: OutOfSync
After synchronization, it displayed:
Health: Healthy
Sync: Synced
Last Sync: Sync OK
The application tree later showed healthy frontend and backend Deployments, ReplicaSets, Services, and Pods.
This is evidence that the Kubernetes deployment converged to the desired state at that point.
11.4 Argo CD workflow
Developer merges PR
|
v
CI builds image
|
v
Image pushed to ECR
|
v
GitOps repository updated
|
v
Argo CD detects new Git revision
|
v
Argo CD synchronizes EKS
|
v
Kubernetes performs rolling update
Important: pushing an image to ECR does not automatically update a Kubernetes Deployment. The GitOps configuration must reference the new image tag or digest.
11.5 Verify the application
kubectl get applications -n argocd
kubectl get pods -n real-estate-company
kubectl get deployments -n real-estate-company
kubectl get services -n real-estate-company
If the Argo CD CLI is installed and configured:
argocd app get real-estate-company
argocd app history real-estate-company
11.6 Interview question: Why Argo CD instead of deploying directly from GitHub Actions?
Answer:
"Argo CD provides a GitOps deployment model where Git is the source of truth for Kubernetes configuration. GitHub Actions handles validation and image publishing, while Argo CD reconciles the cluster against the approved Git configuration. This improves deployment traceability, drift detection, and separation of CI and CD permissions."
PART 12 — KUBERNETES RBAC: WHO CAN ACCESS WHAT?
12.1 What is RBAC?
RBAC stands for Role-Based Access Control.
RBAC controls which authenticated users, groups, and service accounts can perform actions against the Kubernetes API.
Imagine Real Estate Company has 30 engineers.
Not every engineer should be allowed to delete production Deployments, read Secrets, or modify cluster-wide security settings.
We need different access levels.
For example:
| Employee | Example authorized access |
|---|---|
| Frontend developer | Read frontend Pods and logs |
| Backend developer | Read backend Pods and logs |
| DevOps engineer | Manage approved application resources |
| SRE on-call | Diagnose incidents and execute approved recovery actions |
| Security engineer | Audit permissions and security configuration |
| Platform administrator | Manage cluster-wide infrastructure |
Important distinction: Kubernetes RBAC controls API operations. It does not, by itself, control whether one Pod can send HTTP traffic to another Pod. NetworkPolicy handles that separate problem.
12.2 The four important RBAC objects
- Role: Defines permissions within a namespace.
- RoleBinding: Grants a Role or ClusterRole to subjects within a namespace.
- ClusterRole: Defines permissions for cluster-scoped resources or reusable rules.
- ClusterRoleBinding: Grants a ClusterRole across the cluster.
A ServiceAccount is an identity used by workloads. A user or group can also be the subject of an RBAC binding.
LAB 12A — Create a read-only troubleshooting Role
Our namespace:
real-estate-company
First verify it exists:
kubectl get namespace real-estate-company
Create rbac-readonly.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: real-estate-observer
namespace: real-estate-company
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: application-observer
namespace: real-estate-company
rules:
- apiGroups: [""]
resources:
- pods
- pods/log
- services
- endpoints
- events
verbs:
- get
- list
- watch
- apiGroups: ["apps"]
resources:
- deployments
- replicasets
- statefulsets
verbs:
- get
- list
- watch
- apiGroups: ["discovery.k8s.io"]
resources:
- endpointslices
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: application-observer-binding
namespace: real-estate-company
subjects:
- kind: ServiceAccount
name: real-estate-observer
namespace: real-estate-company
roleRef:
kind: Role
name: application-observer
apiGroup: rbac.authorization.k8s.io
Apply:
kubectl apply -f rbac-readonly.yaml
Check:
kubectl get serviceaccount \
-n real-estate-company
kubectl get role \
-n real-estate-company
kubectl get rolebinding \
-n real-estate-company
LAB 12B — Test RBAC
The following commands simulate the ServiceAccount identity. The person running them must already have permission to impersonate that ServiceAccount.
kubectl auth can-i list pods \
-n real-estate-company \
--as=system:serviceaccount:real-estate-company:real-estate-observer
Expected:
yes
Now test deletion:
kubectl auth can-i delete pods \
-n real-estate-company \
--as=system:serviceaccount:real-estate-company:real-estate-observer
Expected:
no
Test reading Secrets:
kubectl auth can-i get secrets \
-n real-estate-company \
--as=system:serviceaccount:real-estate-company:real-estate-observer
Expected:
no
This is intentional.
An engineer investigating an application usually does not need permission to read every database password.
LAB 12C — Demonstrate RBAC denial
kubectl get pods \
-n real-estate-company \
--as=system:serviceaccount:real-estate-company:real-estate-observer
This should succeed for an authorized impersonating operator.
Then:
kubectl delete deployment backend \
-n real-estate-company \
--as=system:serviceaccount:real-estate-company:real-estate-observer
Expected:
Error from server (Forbidden)
This is a controlled negative test. Do not execute it under a different, privileged identity.
12.3 Why should we not grant cluster-admin?
Because cluster-admin effectively provides unrestricted Kubernetes API access.
If credentials are compromised, the attacker may be able to alter workloads, read sensitive resources, or disrupt the cluster.
Production companies generally use least privilege, group-based authorization, controlled elevation, and audit logs.
For EKS, AWS IAM authentication and Kubernetes authorization are separate stages. EKS access entries or other configured authentication mechanisms determine how an AWS principal gains cluster access; Kubernetes RBAC then determines the permitted Kubernetes operations.
12.4 Interview questions
Q: What is the difference between Role and ClusterRole?
"A Role defines permissions within one namespace. A ClusterRole can define cluster-wide permissions and can also be bound within a namespace using a RoleBinding."
Q: What is a RoleBinding?
"A RoleBinding associates permissions with a user, group, or ServiceAccount within a namespace."
Q: What does Forbidden mean?
"It means the Kubernetes API rejected the requested operation during authorization. I would check the authenticated identity, the requested resource and verb, namespace, and applicable RBAC bindings."
Q: How would you request additional Kubernetes permissions?
"I would identify the exact API resources and verbs needed, provide the namespace and business justification, and request a least-privilege Role or RoleBinding through the platform security process. I would avoid requesting cluster-admin unless the responsibility genuinely requires cluster-wide administration."
PART 13 — KUBERNETES NETWORKPOLICY: WHICH PODS CAN COMMUNICATE?
13.1 What is NetworkPolicy?
RBAC controls access to the Kubernetes API.
NetworkPolicy controls permitted network connections to and from selected Pods, provided the cluster's networking implementation enforces Kubernetes NetworkPolicy.
Imagine our application has three components:
Frontend
Backend
PostgreSQL
We want the following connections:
Frontend → Backend ALLOW
Backend → PostgreSQL ALLOW
Frontend → PostgreSQL DENY
Unknown Pod → PostgreSQL DENY
Why?
The frontend does not need direct database access.
If a frontend container is compromised, we want to reduce the attacker's ability to reach PostgreSQL.
This is called network segmentation or microsegmentation.
13.2 Important EKS requirement
Before creating NetworkPolicies, verify that the cluster uses a CNI/network-policy implementation that actually enforces them.
A Kubernetes API can accept a NetworkPolicy object even when no network enforcement is active.
On EKS, this may involve an appropriately configured Amazon VPC CNI network-policy feature or a compatible implementation such as Calico or Cilium.
Students must verify the actual cluster configuration rather than assuming that creating YAML automatically blocks traffic.
LAB 13A — Inspect labels and networking
kubectl get pods \
-n real-estate-company \
--show-labels
kubectl get networkpolicy \
-n real-estate-company
kubectl get pods -n kube-system
Verify the frontend, backend, and database labels.
Our example uses:
app: frontend
app: backend
app: postgres
If the existing Deployment uses different labels, adapt the policies before applying them.
LAB 13B — Default deny ingress
Create networkpolicy-default-deny.yaml:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: real-estate-company
spec:
podSelector: {}
policyTypes:
- Ingress
Apply:
kubectl apply -f networkpolicy-default-deny.yaml
This selects all Pods in the namespace and denies ingress unless another NetworkPolicy explicitly allows it.
Warning: This can interrupt an existing application. Use an isolated lab namespace or approved maintenance window, and prepare the allow rules before enabling enforcement. Do not apply it blindly to a shared environment.
LAB 13C — Allow frontend to backend
Create allow-frontend-to-backend.yaml:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: real-estate-company
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 4000
This allows backend ingress from Pods labeled app: frontend in the same namespace.
However, our earlier architecture also supports direct browser traffic through an ALB /api route. That path may enter from outside the frontend Pods.
Therefore, this policy alone may not permit the actual ALB-to-backend traffic. The team must inspect the AWS Load Balancer Controller target type, source addressing, and CNI enforcement behavior, then design a compatible ingress allowance.
For a production ALB, also use appropriate security groups and VPC network controls.
LAB 13D — Allow backend to PostgreSQL
Create allow-backend-to-postgres.yaml:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-postgres
namespace: real-estate-company
spec:
podSelector:
matchLabels:
app: postgres
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 5432
Apply:
kubectl apply -f allow-backend-to-postgres.yaml
The rule permits backend Pods to initiate connections to PostgreSQL Pods.
It does not grant database username/password permissions. PostgreSQL authentication is a separate security control.
LAB 13E — Verify the policy
kubectl get networkpolicy \
-n real-estate-company
kubectl describe networkpolicy \
allow-backend-to-postgres \
-n real-estate-company
From a backend Pod, test PostgreSQL connectivity using the correct Service name:
kubectl exec \
-n real-estate-company \
deploy/backend -- \
node -e '
const net = require("net");
const s = net.connect(
5432,
"postgres.real-estate-company.svc.cluster.local"
);
s.on("connect", () => {
console.log("TCP connection succeeded");
s.end();
});
s.on("error", e => {
console.error(e.message);
process.exit(1);
});
s.setTimeout(3000, () => {
console.error("Connection timed out");
s.destroy();
process.exit(1);
});
'
This requires the backend container to include Node.js and an accessible shell/exec environment.
A successful TCP connection proves that the database port is reachable. It does not prove that SQL authentication or queries work.
13.3 Egress policies
Ingress controls incoming connections.
Egress controls outgoing connections.
For example, an egress-restricted backend may need explicit allowances for PostgreSQL and DNS.
Without DNS egress, an application may fail to resolve:
postgres.real-estate-company.svc.cluster.local
A production egress policy must account for the actual DNS resolver endpoints, database destination, external dependencies, and network-policy implementation.
13.4 NetworkPolicy incident simulation
Scenario: A security change accidentally blocks backend access to PostgreSQL.
Expected symptoms may include:
Website loads
Backend Pods Running
Backend readiness may remain green
Database-dependent requests fail
Investigation:
kubectl get pods \
-n real-estate-company
kubectl get networkpolicy \
-n real-estate-company
kubectl describe networkpolicy \
-n real-estate-company
kubectl logs \
-n real-estate-company \
deploy/backend \
--tail=100
Check database TCP connectivity and application logs.
Do not immediately restart every Pod. A network-policy problem generally will not be fixed by restarting containers.
Interview question: What is the difference between RBAC and NetworkPolicy?
"RBAC controls who can perform operations through the Kubernetes API. NetworkPolicy controls which network connections are allowed between selected Pods and other destinations. They protect different layers and are both part of a defense-in-depth security model."
PART 14 — POSTGRESQL STATEFULSET AND PERSISTENT STORAGE
14.1 Why do we need a database?
Our real estate application needs to store business information.
Examples:
Properties
Agents
Customers
Inquiries
Appointments
For example, when a customer submits an inquiry, the backend can save it to PostgreSQL.
A database must preserve information even when a Pod restarts.
This is why persistent storage matters.
14.2 Deployment versus StatefulSet
| Deployment | StatefulSet |
|---|---|
| Common for stateless workloads | Common for stateful workloads |
| Pods are interchangeable | Pods have stable ordinal identities |
| Usually no unique persistent disk per replica | Supports stable per-Pod volume claims |
| Frontend and backend | PostgreSQL lab |
A StatefulSet does not automatically provide database replication, backup, or high availability.
14.3 Production database choice
For many AWS production systems, Amazon RDS or Aurora PostgreSQL is a practical choice because AWS manages significant portions of database infrastructure and operations.
However, students should understand how PostgreSQL works inside Kubernetes.
We will create a single-instance PostgreSQL StatefulSet for training.
This is not a production high-availability database architecture.
14.4 Check the storage system
kubectl get storageclass
kubectl get csidrivers
kubectl get nodes -o wide
On EKS with EBS-backed persistent volumes, verify the EBS CSI driver and its IAM permissions.
Check the available StorageClass and provisioner.
Do not assume every cluster uses the same StorageClass name.
LAB 14A — Create a database Secret
For the training lab, create a dedicated database password outside Git:
kubectl create secret generic postgres-credentials \
-n real-estate-company \
--from-literal=POSTGRES_USER=realestate \
--from-literal=POSTGRES_PASSWORD='REPLACE_WITH_STRONG_PASSWORD' \
--from-literal=POSTGRES_DB=realestate
Replace the placeholder with a unique strong password. Avoid putting a real password into shared screenshots, Git history, or recorded terminal output.
For production, use an approved secret-management solution, such as AWS Secrets Manager with a suitable workload integration.
Kubernetes Secret values are not protected merely because they are Base64 encoded.
LAB 14B — Create PostgreSQL Service and StatefulSet
Create postgres-statefulset.yaml.
Replace STORAGE_CLASS_NAME with a verified StorageClass that supports the cluster's EBS provisioning.
apiVersion: v1
kind: Service
metadata:
name: postgres-headless
namespace: real-estate-company
spec:
clusterIP: None
selector:
app: postgres
ports:
- name: postgres
port: 5432
targetPort: 5432
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: real-estate-company
spec:
type: ClusterIP
selector:
app: postgres
ports:
- name: postgres
port: 5432
targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: real-estate-company
spec:
serviceName: postgres-headless
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
securityContext:
fsGroup: 999
containers:
- name: postgres
image: postgres:16
ports:
- containerPort: 5432
name: postgres
envFrom:
- secretRef:
name: postgres-credentials
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
readinessProbe:
exec:
command:
- sh
- -c
- pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
initialDelaySeconds: 10
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 1Gi
volumeClaimTemplates:
- metadata:
name: postgres-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: STORAGE_CLASS_NAME
resources:
requests:
storage: 10Gi
Use a reviewed, pinned PostgreSQL image digest for production. This manifest is a teaching baseline, not a hardened production database specification.
LAB 14C — Deploy PostgreSQL
kubectl apply -f postgres-statefulset.yaml
Verify:
kubectl get statefulsets \
-n real-estate-company
kubectl get pods \
-n real-estate-company
kubectl get pvc \
-n real-estate-company
kubectl get pv
Expected StatefulSet Pod name:
postgres-0
Expected claim naming pattern:
postgres-data-postgres-0
14.5 How does storage work?
PostgreSQL Container
|
v
Volume Mount
|
v
PersistentVolumeClaim
|
v
PersistentVolume
|
v
AWS EBS Volume
A PVC requests storage.
A PV represents the storage resource.
The StorageClass controls dynamic provisioning.
The EBS CSI driver connects Kubernetes storage operations to AWS EBS.
EBS volumes are tied to an Availability Zone, so Pod scheduling and volume placement must be compatible.
LAB 14D — Create a database table
Connect to PostgreSQL:
kubectl exec -it \
-n real-estate-company \
postgres-0 -- \
psql -U realestate -d realestate
Create a table:
CREATE TABLE properties (
id SERIAL PRIMARY KEY,
address VARCHAR(255) NOT NULL,
city VARCHAR(100) NOT NULL,
price NUMERIC(12,2) NOT NULL
);
Insert a sample record:
INSERT INTO properties (address, city, price)
VALUES (
'123 Training Street',
'Chicago',
350000
);
Read:
SELECT * FROM properties;
Exit:
\q
LAB 14E — Test persistence
Delete the PostgreSQL Pod:
kubectl delete pod postgres-0 \
-n real-estate-company
The StatefulSet controller should create a replacement Pod with the same identity.
Wait:
kubectl rollout status \
statefulset/postgres \
-n real-estate-company
Reconnect:
kubectl exec -it \
-n real-estate-company \
postgres-0 -- \
psql -U realestate -d realestate
Run:
SELECT * FROM properties;
If the record remains, the test demonstrates persistence across Pod recreation.
It does not prove that backup, disaster recovery, or database failover are configured.
14.6 Connect the backend to PostgreSQL
The backend application must support PostgreSQL. Merely setting environment variables will not make an application store records in a database.
First inspect:
grep -R "DATABASE_URL\|PGHOST\|pg\|postgres" \
backend/src backend/package.json
If the backend uses the Node.js pg package, the application developer can configure a connection pool similar to:
const { Pool } = require("pg");
const pool = new Pool({
host: process.env.DB_HOST,
port: Number(process.env.DB_PORT || 5432),
database: process.env.DB_NAME,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
});
module.exports = pool;
Install the dependency only if it is required and approved by the application team.
For the backend Deployment, the environment can be wired to the database Secret:
env:
- name: DB_HOST
value: postgres.real-estate-company.svc.cluster.local
- name: DB_PORT
value: "5432"
- name: DB_NAME
valueFrom:
secretKeyRef:
name: postgres-credentials
key: POSTGRES_DB
- name: DB_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: POSTGRES_USER
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: POSTGRES_PASSWORD
Add this to the existing backend container specification through a reviewed GitOps change. Do not replace the whole Deployment without inspecting its existing configuration.
Then verify:
kubectl rollout status deployment/backend \
-n real-estate-company
kubectl logs deployment/backend \
-n real-estate-company \
--tail=100
14.7 End-to-end database test
The ideal functional test is:
Customer submits inquiry
|
v
Frontend sends POST /api/inquiries
|
v
Backend validates request
|
v
Backend INSERT into PostgreSQL
|
v
Database commits transaction
|
v
Backend returns HTTP 201
|
v
Frontend displays confirmation
The /api/inquiries endpoint is an example. Students must implement or verify it with the backend developer before running the following test.
curl -i -X POST \
http://localhost:4000/api/inquiries \
-H "Content-Type: application/json" \
-d '{
"propertyId": 101,
"customerName": "Test Customer",
"message": "I would like more information."
}'
A successful HTTP response alone is not enough. Verify that the intended record was actually committed to PostgreSQL.
Interview question: Why use StatefulSet for PostgreSQL?
"StatefulSet provides stable Pod identity and persistent volume claim association. For our training lab, we used it to demonstrate that database records survive Pod recreation. In a production AWS environment, I would also evaluate RDS or Aurora because database availability, backup, patching, and recovery require more than a StatefulSet."
PART 15 — AWS ALB INGRESS AND EXTERNAL HTTP TRAFFIC
15.1 Why do we need an Ingress?
A Kubernetes ClusterIP Service is normally reachable inside the cluster, not directly from the public internet.
Customers need an external HTTPS endpoint.
In AWS EKS, we can use the AWS Load Balancer Controller to provision and manage an Application Load Balancer based on Kubernetes Ingress configuration.
The intended architecture is:
Customer Browser
|
| HTTPS :443
v
AWS Application Load Balancer
|
+---- / --------> Frontend Service
| |
| v
| Frontend Pods
|
+---- /api/* ---> Backend Service
|
v
Backend Pods
|
v
PostgreSQL
15.2 Prerequisites
Before applying an ALB Ingress, verify that:
- The AWS Load Balancer Controller is installed and healthy.
- The controller has the required AWS IAM permissions.
- The VPC subnets are correctly configured and discoverable.
- Security groups allow approved traffic.
- The Services and Pods are healthy.
- An ACM certificate and DNS configuration are available for production HTTPS.
- The ALB target type matches the Service and networking configuration.
Check:
kubectl get deployment \
-n kube-system \
aws-load-balancer-controller
kubectl get svc \
-n real-estate-company
kubectl get endpointslices \
-n real-estate-company
LAB 15A — Example Ingress
This is a starting point for an EKS environment with a properly configured AWS Load Balancer Controller and IP targets.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: real-estate-ingress
namespace: real-estate-company
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: backend
port:
number: 4000
- path: /
pathType: Prefix
backend:
service:
name: frontend
port:
number: 80
This HTTP-only example is for an isolated lab. For a customer-facing production service, configure HTTPS, ACM, appropriate inbound restrictions, and the application's required health-check behavior.
LAB 15B — Check ALB creation
kubectl get ingress \
-n real-estate-company
kubectl describe ingress real-estate-ingress \
-n real-estate-company
After the controller successfully provisions the ALB, the Ingress should display an address.
Test the actual assigned endpoint:
curl -i http://ALB_DNS_NAME/
For a verified API endpoint:
curl -i http://ALB_DNS_NAME/api/properties
Do not assume the API endpoint exists without checking the backend routes.
Interview question: What is the difference between Service and Ingress?
Answer:
"A Kubernetes Service provides stable network access to a group of Pods. Ingress defines HTTP and HTTPS routing rules to Services. In our EKS environment, the AWS Load Balancer Controller translates Ingress configuration into an Application Load Balancer."
PART 16 — THE REAL IMAGEPULLBACKOFF INCIDENT
16.1 What happened during class?
After Argo CD deployed the Real Estate application, the Pods failed to start.
The Kubernetes events showed:
Failed to pull image
rpc error: code = NotFound
Error: ErrImagePull
Error: ImagePullBackOff
The Deployment referenced an image similar to:
992734380025.dkr.ecr.us-east-1.amazonaws.com/real-estate-company-backend:latest
However, the ECR repository contained commit-based image tags rather than latest.
That is the important mismatch.
16.2 How to investigate
First check the Pods:
kubectl get pods \
-n real-estate-company
Then:
kubectl describe pod POD_NAME \
-n real-estate-company
Read the Events section.
Check which image Kubernetes is trying to pull:
kubectl get deployment backend \
-n real-estate-company \
-o jsonpath='{.spec.template.spec.containers[*].image}'
Check ECR:
aws ecr describe-images \
--repository-name real-estate-company-backend \
--region us-east-1 \
--query 'imageDetails[].imageTags' \
--output json
Compare the exact registry, repository, tag, and digest.
16.3 Why did this happen?
The CI pipeline published a valid image, but the Kubernetes Deployment referenced a different tag.
For example:
ECR contains:
backend:f260c920...
Kubernetes requests:
backend:latest
ECR cannot return an image for a tag that does not exist.
16.4 How to fix it using GitOps
Update the backend image reference in the Git-managed Deployment manifest to a tag or digest confirmed in ECR.
Example:
containers:
- name: backend
image: 992734380025.dkr.ecr.us-east-1.amazonaws.com/real-estate-company-backend:VERIFIED_TAG
Replace VERIFIED_TAG with the actual published tag.
Then:
git add k8s/backend-deployment.yaml
git commit -m "fix: use published backend image tag"
git push origin feature-estate
Create or update the pull request.
After approval and merge, synchronize Argo CD if automatic sync is not enabled.
Verify:
kubectl rollout status \
deployment/backend \
-n real-estate-company
kubectl get pods \
-n real-estate-company
In the live class, a later Argo CD screenshot showed healthy running frontend and backend Pods. That supports the observed recovery of the deployment, although it does not establish that every external business workflow was tested.
16.5 Interview answer using STAR
Situation: "During an EKS deployment, our application Pods entered ImagePullBackOff."
Task: "I needed to identify why Kubernetes could not start the containers and restore the deployment."
Action: "I inspected Pod events, identified the requested ECR image reference, and compared it with the tags actually published by GitHub Actions. I found that the Deployment referenced latest, while the CI pipeline published commit-based tags. We corrected the GitOps manifest to reference an existing image."
Result: "After Argo CD synchronized the corrected configuration, the deployment recovered and the Pods became healthy."
Follow-up: What if the image exists but ImagePullBackOff continues?
"I would investigate ECR authentication and authorization, the worker node or Pod identity used for image pulling, repository policies, network access to ECR and S3, the image architecture, and the exact error in Pod events."
PART 17 — MONITORING WITH PROMETHEUS AND GRAFANA
17.1 Why monitoring matters
A Deployment can show two running Pods while customers still experience errors.
For example:
Frontend Pods: Running
Backend Pods: Running
Database: Unreachable
Customer requests: HTTP 503
Therefore, Pod health alone is not sufficient.
We need visibility into application behavior and infrastructure.
17.2 Four important observability signals
| Signal | Example |
|---|---|
| Latency | API response duration |
| Traffic | Requests per second |
| Errors | HTTP 5xx rate |
| Saturation | CPU, memory, connection pool utilization |
17.3 Example PromQL
These examples assume Prometheus is installed and the relevant metrics are actually collected.
Running Pods
sum(
kube_pod_status_phase{
namespace="real-estate-company",
phase="Running"
}
)
Container restart count
sum by (pod) (
kube_pod_container_status_restarts_total{
namespace="real-estate-company"
}
)
Deployment unavailable replicas
kube_deployment_status_replicas_unavailable{
namespace="real-estate-company"
}
CPU usage
sum by (pod) (
rate(container_cpu_usage_seconds_total{
namespace="real-estate-company",
container!="",
container!="POD"
}[5m])
)
HTTP 5xx rate
sum(
rate(
http_requests_total{
status=~"5.."
}[5m]
)
)
The HTTP query requires application instrumentation with a metric named http_requests_total and a status label. It will not return data automatically for every Express application.
17.4 What dashboards should DevOps maintain?
A useful Real Estate Company dashboard could display:
- Frontend and backend availability.
- Number of ready Pods.
- Container restarts.
- CPU and memory usage.
- HTTP request rate.
- HTTP 4xx and 5xx errors.
- API latency.
- Database connection errors.
- Deployment versions and recent changes.
17.5 Example alert
An alert can detect a backend Deployment with unavailable replicas:
groups:
- name: real-estate-alerts
rules:
- alert: RealEstateBackendUnavailable
expr: |
kube_deployment_status_replicas_unavailable{
namespace="real-estate-company",
deployment="backend"
} > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Real Estate backend has unavailable replicas"
This rule requires a correctly configured Prometheus rule-loading mechanism and kube-state-metrics.
In production, alerts should be routed through the approved notification or incident-management system.
17.6 Interview question: What is the difference between monitoring and logging?
Answer:
"Monitoring uses metrics and alerts to identify changes in system health and performance. Logging provides detailed records of application and infrastructure events. Metrics help identify when and where a problem is occurring, while logs help investigate its cause."
PART 18 — PRODUCTION INCIDENT RESPONSE
Scenario: Customers cannot submit inquiries
At 2:15 PM, Grafana shows increasing HTTP 503 errors.
The frontend website loads.
Kubernetes reports:
Frontend Pods: 2/2 Ready
Backend Pods: 2/2 Ready
However, customers cannot submit inquiries.
Step 1 — Acknowledge the incident
The on-call SRE or DevOps engineer acknowledges the alert and evaluates its severity.
Determine:
- Are all customers affected?
- Is the issue limited to inquiries?
- When did errors begin?
- Was there a recent deployment?
- Is there evidence of data loss?
- What is the business impact?
Step 2 — Check application health
kubectl get pods \
-n real-estate-company
kubectl get deployments \
-n real-estate-company
kubectl get services \
-n real-estate-company
Step 3 — Inspect logs
kubectl logs \
-n real-estate-company \
deployment/backend \
--since=15m
Look for:
Connection timeout
Database connection refused
Authentication failed
Too many connections
Query timeout
These are possible symptoms, not confirmed messages from the classroom incident.
Step 4 — Check database
kubectl get statefulset postgres \
-n real-estate-company
kubectl get pod postgres-0 \
-n real-estate-company
kubectl get pvc \
-n real-estate-company
If using RDS, investigate the RDS instance and its CloudWatch metrics instead.
Step 5 — Check NetworkPolicy
kubectl get networkpolicy \
-n real-estate-company
kubectl describe networkpolicy \
-n real-estate-company
Determine whether a recent network-policy change blocked backend-to-database traffic.
Step 6 — Check recent releases
kubectl rollout history \
deployment/backend \
-n real-estate-company
Also inspect the Argo CD application history.
If a recent deployment caused the incident, follow the approved GitOps rollback process.
Do not repeatedly issue manual kubectl rollout undo commands while Argo CD is configured to reconcile a different desired state. Git and the live cluster may otherwise fight each other.
Step 7 — Restore and validate
After implementing an approved mitigation:
kubectl get pods \
-n real-estate-company
kubectl get endpointslices \
-n real-estate-company
Test the affected business operation.
Check error rate, latency, and database writes.
A recovery is not complete simply because Pods are Running.
Step 8 — Document the incident
A useful incident report includes:
Incident ID:
Severity:
Start time:
Detection source:
Customer impact:
Affected services:
Timeline:
Root cause:
Mitigation:
Recovery time:
Evidence:
Preventive actions:
Owners:
Interview answer: How do you troubleshoot HTTP 503 when Pods are Ready?
"I first confirm customer impact and inspect metrics to identify the affected endpoint and timeframe. I check the ingress, Service endpoints, backend logs, and dependencies such as PostgreSQL. Ready Pods do not guarantee that every business transaction works. I also review recent deployments and NetworkPolicy changes. After isolating the cause, I coordinate the safest mitigation, verify the customer request path, and document the root cause and preventive actions."
PART 19 — HOW LARGE IS A REAL PRODUCTION ENVIRONMENT?
Students often ask:
"How many microservices does a real company have?"
"How many Pods do DevOps engineers manage?"
"How many Kubernetes clusters does a company use?"
There is no universal number.
A startup may have one cluster and a few services. A large enterprise may operate many clusters and hundreds or thousands of services.
For interview practice, we will use a hypothetical mid-sized real estate technology company.
These are scenario assumptions, not measurements of the student project.
3
EKS clusters
40
Microservices
180
Running Pods
19.1 Example company architecture
| Environment | Clusters | Example workloads |
|---|---|---|
| Development | 1 | Developer integration and testing |
| Staging | 1 | Release validation |
| Production | 1 | Customer-facing applications |
The hypothetical company operates 40 microservices, including property search, customer accounts, listings, inquiries, agent management, notifications, payments, and reporting.
There may be 180 Pods across the three clusters because each microservice can have multiple replicas, and clusters also run infrastructure Pods.
19.2 Does one DevOps engineer manage all 180 Pods manually?
No.
Kubernetes controllers maintain desired state.
The DevOps team manages the platform, deployment configuration, automation, policies, and operational processes.
For example:
40 Microservices
|
v
GitHub Repositories
|
v
CI Pipelines
|
v
ECR Images
|
v
Argo CD Applications
|
v
Kubernetes Deployments
|
v
ReplicaSets
|
v
Pods
If a Deployment requires four replicas, Kubernetes maintains that desired count.
The DevOps engineer does not manually create four individual Pods every morning.
19.3 How do engineers manage many services?
Common practices include:
- Terraform for infrastructure.
- Helm or Kustomize for reusable Kubernetes configuration.
- Argo CD for GitOps deployment.
- Namespace and RBAC boundaries.
- NetworkPolicies and cloud network controls.
- Resource requests, limits, and quotas.
- Horizontal Pod Autoscalers.
- Prometheus and Grafana.
- Centralized logging.
- CI/CD templates.
- Change approval and incident-management processes.
19.4 Interview answer: How many clusters and services have you supported?
For this hypothetical company scenario, a student can practice:
"In our training scenario, the platform contains three EKS clusters for development, staging, and production, supporting approximately 40 microservices and 180 Pods. The DevOps team manages deployment automation, cluster configuration, observability, security policies, and incident response using Terraform, GitHub Actions, Argo CD, and Kubernetes."
Important: In a real job interview, describe these numbers as lab or scenario experience unless you actually supported such an environment professionally. Do not present simulated production scale as employment history.
PART 20 — ADDITIONAL PRODUCTION ENGINEERING TOPICS
20.1 Horizontal Pod Autoscaler
An HPA adjusts replica counts using observed metrics.
Example:
kubectl autoscale deployment backend \
-n real-estate-company \
--cpu-percent=60 \
--min=2 \
--max=6
This requires working resource metrics and appropriate CPU requests.
If Argo CD manages the Deployment and autoscaling resources, add the HPA through GitOps rather than creating unmanaged drift.
Interview question: What is the difference between HPA and Cluster Autoscaler?
"HPA scales application replicas, while a node autoscaler adds or removes compute capacity when workloads cannot be scheduled efficiently. They solve different scaling problems."
20.2 Resource requests and limits
Example:
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
Requests help the scheduler place Pods.
Limits constrain resource consumption according to Kubernetes and container-runtime behavior.
20.3 Readiness versus liveness
Readiness probe: Determines whether a Pod should receive traffic through Services.
Liveness probe: Determines whether a container should be restarted because it is unhealthy.
Startup probe: Gives slow-starting applications additional time before other probe behavior takes effect.
Interview follow-up: Why shouldn't liveness depend on every external database query?
"Because a temporary database outage could trigger unnecessary restarts across otherwise healthy application containers. I would design readiness, liveness, and dependency checks according to their different purposes."
20.4 Rolling updates
Suppose the backend has two replicas.
With:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
Kubernetes can create an additional new Pod while keeping the required number of available replicas during the update.
This depends on available cluster capacity and the new Pod becoming ready.
20.5 Database backup
Persistent storage is not the same as backup.
Deleting a Pod should not delete its persistent database records, but storage corruption, accidental deletion, or application mistakes can still destroy data.
Production database planning should include backup schedules, retention, recovery objectives, restore testing, and access controls.
PART 21 — COMPLETE END-TO-END VALIDATION CHECKLIST
Students should finish the lab by verifying every stage, rather than assuming a successful deployment means the entire system works.
Student lab completion
0 / 15
Repository inspected
README, package scripts, Dockerfiles, workflow and manifests reviewed
Frontend validated
Lint and production build successful
Backend validated
Lint, tests and health endpoint checked
Security gate passed
Trivy findings reviewed and resolved
OIDC verified
AWS STS assumes the approved role
ECR verified
Frontend and backend images exist with exact tags
Kubernetes verified
Deployments, Services and Pods healthy
Argo CD verified
Application Synced and Healthy
RBAC verified
Read-only identity allowed reads but denied deletion
NetworkPolicy verified
Approved traffic works and prohibited traffic is blocked
PostgreSQL verified
SQL insert and select successful
Persistence verified
Data survives PostgreSQL Pod recreation
HTTP path verified
Customer request reaches the intended backend
Monitoring verified
Metrics, logs and alert rules produce useful evidence
Incident practiced
Diagnosis, mitigation and RCA documented
Copy checklist
PART 22 — FINAL INTERVIEW PRACTICE
Students should be able to answer these questions directly.
| Interview question | Key points to include |
|---|---|
| Explain your CI/CD pipeline. | PR, lint, tests, security scan, image build, ECR, GitOps |
| Why do you use OIDC? | Short-lived credentials, identity federation, no stored AWS keys |
| What is an IAM trust policy? | Defines who may assume a role and under which conditions |
| What is an IAM permissions policy? | Defines actions the role may perform |
| Why did the OIDC incident happen? | Actual GitHub subject did not match the expected trust condition |
| Why use Argo CD? | GitOps, desired-state reconciliation, drift detection |
| What does OutOfSync mean? | Live state differs from desired Git state |
| What is ImagePullBackOff? | Kubernetes repeatedly fails to obtain a container image |
| What is a StatefulSet? | Controller for workloads requiring stable identity and storage association |
| Why use a PVC? | Requests persistent storage independent of a Pod lifecycle |
| What is RBAC? | Kubernetes API authorization |
| What is NetworkPolicy? | Controls selected Pod network connections |
| Why can Pods be Ready while customers get 503? | Business dependencies may fail even when basic readiness passes |
| How do you troubleshoot DNS? | Check Service name, CoreDNS, resolver behavior, endpoints, policies |
| How do you troubleshoot a failed deployment? | Events, logs, image, probes, dependencies, recent changes |
| How do you request IAM access? | Exact action, resource scope, business reason, security approval |
| How do you rollback? | Restore approved GitOps state and verify recovery |
| How do you handle a production incident? | Assess impact, acknowledge, investigate, mitigate, validate, document |
Final interview question: Walk me through your Real Estate Company project.
Suggested student answer:
"I worked on an end-to-end DevOps training project for a real estate web application consisting of a React frontend and a Node.js Express backend.
We used GitHub for source control and GitHub Actions for Continuous Integration. Our pipeline ran frontend linting and builds, backend tests, and Trivy vulnerability scanning. After the required checks passed, the pipeline built Docker images and published them to Amazon ECR.
We integrated GitHub OIDC with AWS IAM to use temporary credentials instead of long-lived AWS access keys. During the lab, we encountered an OIDC role-assumption failure and investigated the mismatch between GitHub token claims and IAM trust conditions.
We deployed the application to Amazon EKS and used Argo CD for GitOps-based Continuous Delivery. We also investigated an ImagePullBackOff incident caused by a mismatch between the Kubernetes image reference and the image tags available in ECR.
We studied Kubernetes Deployments, Services, Ingress, RBAC, and NetworkPolicies, and extended the lab to include PostgreSQL with a StatefulSet and persistent storage.
The project helped me understand not only how to deploy an application but also how DevOps engineers collaborate with developers, security teams, database engineers, and SRE teams to troubleshoot incidents and maintain reliable delivery pipelines."
Students should adjust the final paragraph to reflect the labs they personally completed and verified.
Final assignment for every student
Each student must demonstrate the application delivery process from Git to Kubernetes, explain one CI or OIDC incident, show Argo CD synchronization, demonstrate one RBAC permission denial, verify an allowed and blocked NetworkPolicy connection, and prove PostgreSQL data persistence.
They must also explain the difference between an application being Running, Ready, Synced, and functionally working for customers.
The most important outcome is not memorizing YAML. It is being able to answer:
What happened? Why did it happen? How did you investigate? How did you fix it safely? How did you verify recovery? How would you prevent it from happening again?
That is the foundation of a strong production DevOps interview.
Top comments (0)