DEV Community: Aisalkyn Aidarova

Docker Image Kubernetes Browser

Aisalkyn Aidarova — Wed, 09 Sep 2026 02:39:55 +0000

Goal

In this lab, we will follow a very common DevOps workflow:

Developer code → Docker Image → Docker Registry → Kubernetes Deployment → Kubernetes Service → Browser

By the end of this lab, you should understand:

how to create a small application
how to create a Docker image
how to test the container
how to push the Docker image to Docker Hub
how Kubernetes downloads the image
how Kubernetes creates Pods
how a Service exposes the application
how a user opens the application in a browser

Architecture

Application files
      ↓
Dockerfile
      ↓
docker build
      ↓
Docker Image
      ↓
Docker Hub
      ↓
Kubernetes Deployment
      ↓
Pod
      ↓
Service
      ↓
Browser

Part 1 — Create the project

Do this on your local computer or EC2 instance where Docker and kubectl are installed.

Create a project directory:

mkdir docker-k8s-lab
cd docker-k8s-lab

Check where you are:

pwd

You should now be inside:

docker-k8s-lab

Part 2 — Create a simple website

Create a file:

nano index.html

Add:

<!DOCTYPE html>
<html>
<head>
    <title>Docker Kubernetes Lab</title>
</head>
<body>
    <h1>Hello from Docker and Kubernetes!</h1>
    <p>My application is running inside Kubernetes.</p>
</body>
</html>

Save the file.

If you use nano:

CTRL + O
Enter
CTRL + X

Check the file:

cat index.html

What did we do?

We created the application.

For this lab, our application is only one HTML page.

In a real company, this could be:

React application
Java application
Python application
Node.js application
.NET application

As DevOps engineers, we usually do not write the business application itself.

Developers create the application.

Our responsibility is to package it, deploy it, automate it, monitor it, and keep it running.

Part 3 — Create the Dockerfile

Create:

nano Dockerfile

Add:

FROM nginx:alpine

COPY index.html /usr/share/nginx/html/index.html

Save it.

Check:

cat Dockerfile

Explanation

FROM nginx:alpine

This means:

Use the Nginx image as the base image.

Nginx will work as our web server.

Then:

COPY index.html /usr/share/nginx/html/index.html

This copies our HTML page inside the Docker image.

The image will contain:

Linux
Nginx
Our index.html

This is what Docker packaging means.

Part 4 — Build the Docker image

Run:

docker build -t docker-k8s-lab:v1 .

Check the image:

docker images

You should see something similar to:

REPOSITORY        TAG
docker-k8s-lab    v1

What happened?

Docker read the:

Dockerfile

and created a Docker image.

Think of the image as a package or template.

We have not deployed the application yet.

We only created the package.

Part 5 — Test the Docker image locally

Before sending our image to Kubernetes, we should test it.

Run:

docker run -d -p 8080:80 --name docker-k8s-test docker-k8s-lab:v1

Check:

docker ps

Open:

http://localhost:8080

If you are using an EC2 instance, use:

http://PUBLIC-IP:8080

Make sure port 8080 is allowed in the Security Group if using EC2.

You should see:

Hello from Docker and Kubernetes!

What does this mean?

-p 8080:80

means:

Computer port 8080
        ↓
Container port 80

Nginx listens on port:

Part 6 — Stop the test container

We only used this container for testing.

Run:

docker stop docker-k8s-test

Remove it:

docker rm docker-k8s-test

Check:

docker ps

Part 7 — Push the image to Docker Hub

Kubernetes must be able to download the Docker image.

Your local image:

docker-k8s-lab:v1

exists only on your machine.

Kubernetes usually downloads images from a registry such as:

Docker Hub
AWS ECR
Azure ACR
Google Artifact Registry

For this lab we will use Docker Hub.

Part 8 — Login to Docker Hub

Run:

docker login

Enter your Docker Hub username and password/token.

Part 9 — Tag the image

Replace:

YOUR_DOCKERHUB_USERNAME

with your real username.

Run:

docker tag docker-k8s-lab:v1 YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v1

Example:

docker tag docker-k8s-lab:v1 aisalkyn/docker-k8s-lab:v1

Check:

docker images

Now you should see both names.

Part 10 — Push the image

Run:

docker push YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v1

Example:

docker push aisalkyn/docker-k8s-lab:v1

Now the image is stored in Docker Hub.

DevOps concept

This is an important production idea.

Developer code
      ↓
Docker build
      ↓
Container Registry
      ↓
Kubernetes

In AWS, instead of Docker Hub, we usually use:

Amazon ECR

Part 11 — Check Kubernetes

Before deployment, make sure Kubernetes works.

Run:

kubectl get nodes

You should see:

NAME      STATUS
node-1    Ready

The important word is:

Ready

Also check:

kubectl cluster-info

Part 12 — Create deployment.yaml

Now we tell Kubernetes:

"Run my Docker image."

Create:

nano deployment.yaml

Add:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: docker-k8s-app

spec:
  replicas: 2

  selector:
    matchLabels:
      app: docker-k8s-app

  template:
    metadata:
      labels:
        app: docker-k8s-app

    spec:
      containers:
        - name: docker-k8s-app
          image: YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v1
          ports:
            - containerPort: 80

Replace:

YOUR_DOCKERHUB_USERNAME

with your username.

Example:

image: aisalkyn/docker-k8s-lab:v1

Part 13 — Understand deployment.yaml

The first part:

apiVersion: apps/v1
kind: Deployment

means:

Create a Kubernetes Deployment.

The name:

metadata:
  name: docker-k8s-app

means our Deployment will be called:

docker-k8s-app

Then:

replicas: 2

means:

Run two copies of the application.

So Kubernetes will create approximately:

Pod 1
Pod 2

Both run the same Docker image.

Then:

image: YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v1

means:

Kubernetes will go to Docker Hub and download this image.

Then:

containerPort: 80

means:

The application inside the container listens on port 80.

Part 14 — Deploy to Kubernetes

Run:

kubectl apply -f deployment.yaml

Expected:

deployment.apps/docker-k8s-app created

Check:

kubectl get deployments

Then:

kubectl get pods

You should see something similar to:

docker-k8s-app-xxxx   1/1   Running
docker-k8s-app-yyyy   1/1   Running

What does 1/1 mean?

1/1

means:

1 container expected
1 container ready

The Pod is ready.

Part 15 — Check more details

Run:

kubectl get pods -o wide

This shows additional information:

Pod IP
Node
Status

You can also run:

kubectl describe pod POD-NAME

Example:

kubectl describe pod docker-k8s-app-xxxxx

Part 16 — Look at the logs

Run:

kubectl logs POD-NAME

Example:

kubectl logs docker-k8s-app-xxxxx

For Nginx, logs may appear after someone accesses the website.

Logs are very important for DevOps engineers because we use them when troubleshooting applications.

Part 17 — Why can't we open the Pod directly?

We now have:

Pod 1
Pod 2

But Pods are temporary.

Their IP addresses can change.

For example:

Pod dies
↓
Kubernetes creates a new Pod
↓
New Pod gets a different IP

Because of this, users should not connect directly to Pods.

We need a:

Service

Part 18 — Create service.yaml

Create:

nano service.yaml

Add:

apiVersion: v1
kind: Service

metadata:
  name: docker-k8s-service

spec:
  selector:
    app: docker-k8s-app

  ports:
    - port: 80
      targetPort: 80

  type: LoadBalancer

Save it.

Part 19 — Understand the Service

This line:

selector:
  app: docker-k8s-app

means:

Send traffic to Pods that have this label:

app: docker-k8s-app

Remember our Deployment also has:

labels:
  app: docker-k8s-app

This is how the Service finds the Pods.

Part 20 — Understand the ports

port: 80
targetPort: 80

Think:

User
 ↓
Service port 80
 ↓
Pod port 80
 ↓
Nginx

Part 21 — Create the Service

Run:

kubectl apply -f service.yaml

Check:

kubectl get services

or:

kubectl get svc

You may see:

NAME                 TYPE           EXTERNAL-IP
docker-k8s-service   LoadBalancer   ...

Part 22 — Open the application

If you are using a cloud Kubernetes cluster such as EKS, AKS, or GKE:

Run:

kubectl get svc

Find:

EXTERNAL-IP

It may look like:

abc123.us-east-1.elb.amazonaws.com

Open it in your browser:

http://EXTERNAL-IP

Now you should see:

Hello from Docker and Kubernetes!

Congratulations.

Your application flow is now:

Browser
   ↓
LoadBalancer
   ↓
Kubernetes Service
   ↓
Pod
   ↓
Container
   ↓
Nginx
   ↓
index.html

If you are using Minikube

Instead of LoadBalancer, run:

minikube service docker-k8s-service

Or:

minikube service docker-k8s-service --url

Then open the URL.

If you are using kind

LoadBalancer normally does not automatically give you a public IP.

For a simple lab, use port-forward:

kubectl port-forward service/docker-k8s-service 8080:80

Then open:

http://localhost:8080

Keep that terminal open while testing.

Part 23 — Watch how traffic reaches different Pods

Check Pods:

kubectl get pods

You should have two Pods.

The Service can send requests to either Pod.

Architecture:

                  ┌── Pod 1
Browser → Service ┤
                  └── Pod 2

This is one reason Kubernetes is useful.

Part 24 — Scale the application

Change:

replicas: 2

to:

replicas: 4

Then run:

kubectl apply -f deployment.yaml

Check:

kubectl get pods

You should now have four Pods.

You can also scale using a command:

kubectl scale deployment docker-k8s-app --replicas=4

Check:

kubectl get pods

What did Kubernetes do?

You told Kubernetes:

I want 4 application instances.

Kubernetes created enough Pods to reach the desired state.

Part 25 — Test self-healing

Check Pods:

kubectl get pods

Choose one Pod.

Delete it:

kubectl delete pod POD-NAME

Immediately run:

kubectl get pods

You should see Kubernetes creating a new Pod.

Why?

Because Deployment says:

replicas: 4

Kubernetes constantly compares:

Desired state = 4 Pods
Actual state = 3 Pods

So Kubernetes creates another Pod.

This is called:

self-healing

Part 26 — Update the application

Change the website:

nano index.html

Change:

<h1>Hello from Docker and Kubernetes!</h1>

to:

<h1>Version 2 is running in Kubernetes!</h1>

Now build a new image:

docker build -t docker-k8s-lab:v2 .

Tag it:

docker tag docker-k8s-lab:v2 YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v2

Push it:

docker push YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v2

Part 27 — Update Kubernetes

Open:

nano deployment.yaml

Change:

image: YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v1

to:

image: YOUR_DOCKERHUB_USERNAME/docker-k8s-lab:v2

Apply:

kubectl apply -f deployment.yaml

Watch the rollout:

kubectl rollout status deployment/docker-k8s-app

Check Pods:

kubectl get pods

Refresh the browser.

You should now see:

Version 2 is running in Kubernetes!

This is a basic Kubernetes rolling deployment.

Part 28 — See rollout history

Run:

kubectl rollout history deployment/docker-k8s-app

This shows deployment revisions.

If the new version has a problem, DevOps engineers may need to roll back.

Part 29 — Roll back

Run:

kubectl rollout undo deployment/docker-k8s-app

Then:

kubectl rollout status deployment/docker-k8s-app

Refresh your browser.

The previous version should return.

Part 30 — Troubleshooting commands every DevOps engineer should know

Check Pods:

kubectl get pods

Check Deployment:

kubectl get deployment

Check Service:

kubectl get svc

See more information:

kubectl get pods -o wide

Describe Pod:

kubectl describe pod POD-NAME

View logs:

kubectl logs POD-NAME

View Deployment:

kubectl describe deployment docker-k8s-app

View Service:

kubectl describe service docker-k8s-service

Watch Pods:

kubectl get pods -w

Common problems

ImagePullBackOff

If you see:

ImagePullBackOff

check:

kubectl describe pod POD-NAME

Possible reasons:

Wrong Docker image name
Wrong tag
Image was not pushed
Private Docker Hub repository
Authentication problem

CrashLoopBackOff

If you see:

CrashLoopBackOff

check:

kubectl logs POD-NAME

This usually means the application starts and crashes repeatedly.

Pending

If Pod status is:

Pending

run:

kubectl describe pod POD-NAME

Possible reasons:

Not enough CPU
Not enough memory
Scheduling problem
No available node

Browser does not open

Check:

kubectl get svc

Then:

kubectl describe svc docker-k8s-service

Also check the Pods:

kubectl get pods

They should be:

Running

and:

1/1

Clean up

When you finish the lab:

kubectl delete -f service.yaml

Then:

kubectl delete -f deployment.yaml

Check:

kubectl get pods
kubectl get svc

Final DevOps workflow

Remember this flow:

1. Developer writes application
2. DevOps creates/builds Docker image
3. DevOps tests Docker image
4. Image is pushed to registry
5. Kubernetes Deployment pulls image
6. Deployment creates Pods
7. Service finds Pods using labels
8. Service exposes the application
9. User opens application in browser
10. DevOps monitors, troubleshoots, scales, updates, and rolls back

The most important architecture to remember is:

Developer
   ↓
GitHub
   ↓
CI/CD
   ↓
Docker Image
   ↓
Registry
   ↓
Kubernetes Deployment
   ↓
Pods
   ↓
Service
   ↓
Load Balancer / Ingress
   ↓
Browser

WEEKEND DEVOPS LAB KUBERNETES CONTINUOUS DEPLOYMENT WITH JENKINS

Aisalkyn Aidarova — Sat, 05 Sep 2026 03:03:59 +0000

PROJECT:
Restaurant Company

DEADLINE:
Be ready to present this lab in the next class.

==================================================

LAB OBJECTIVE

The goal of this lab is NOT to copy commands.

The goal is to understand the complete Continuous Deployment process:

GitHub
↓
Jenkins
↓
AWS
↓
EKS
↓
Kubernetes Deployment
↓
ReplicaSet
↓
Pods
↓
Service
↓
Application

During this lab you will:

Verify your Kubernetes environment
Deploy an application
Understand Deployment, ReplicaSet, and Pods
Test Kubernetes self-healing
Scale an application
Perform a Rolling Update
Investigate Kubernetes Events
Work with health probes
Intentionally break a deployment
Troubleshoot ImagePullBackOff
Troubleshoot application failures
Practice rollback
Run the deployment through Jenkins
Present and explain what you learned

==================================================

IMPORTANT RULE

DO NOT immediately ask AI to fix an error.

When you receive an error:

Read the error
Check Pod status
Run describe
Read Events
Check logs
Explain what you think happened
Then research the solution

Your instructor will ask:

"How did you know what the problem was?"

You must be able to answer.

==================================================

PART 1 — VERIFY YOUR ENVIRONMENT

Before deploying anything, verify your environment.

Check AWS:

aws sts get-caller-identity

Check AWS CLI:

aws --version

Check kubectl:

kubectl version --client

Check your Kubernetes context:

kubectl config current-context

Check cluster nodes:

kubectl get nodes

EXPECTED RESULT:

Your Kubernetes nodes should be:

Ready

Take a screenshot.

Screenshot name:

01-cluster-nodes

QUESTION TO ANSWER:

What does Ready mean for a Kubernetes Node?

==================================================

PART 2 — VERIFY THE NAMESPACE

Check namespaces:

kubectl get namespaces

Check our production namespace:

kubectl get namespace restaurant-prod

If the namespace does not exist and you have namespace.yaml:

kubectl apply -f namespace.yaml

Now run:

kubectl get pods

Then run:

kubectl get pods -n restaurant-prod

QUESTION:

Why can these two commands return different results?

YOUR ANSWER SHOULD EXPLAIN:

default namespace

restaurant-prod namespace

Take a screenshot:

02-namespace

==================================================

PART 3 — DEPLOY THE APPLICATION

From your CD repository, inspect the Kubernetes files.

You should understand files such as:

namespace.yaml
configmap.yaml
secrets.yaml
serviceaccount.yaml
deployment.yaml
service.yaml
pdb.yaml
networkpolicy.yaml
ingress.yaml

Your repository may not contain every optional resource.

Apply the required resources.

Example:

kubectl apply -f namespace.yaml

kubectl apply -f configmap.yaml -n restaurant-prod

kubectl apply -f secrets.yaml -n restaurant-prod

kubectl apply -f serviceaccount.yaml -n restaurant-prod

kubectl apply -f deployment.yaml -n restaurant-prod

kubectl apply -f service.yaml -n restaurant-prod

If your project includes additional working resources, apply them as instructed.

Now check:

kubectl get all -n restaurant-prod

Then:

kubectl get pods -n restaurant-prod -o wide

EXPECTED RESULT:

Your application Pods should eventually show:

Running

and:

READY 1/1

Take a screenshot:

03-application-running

==================================================

PART 4 — UNDERSTAND THE KUBERNETES HIERARCHY

Run:

kubectl get deployment -n restaurant-prod

Then:

kubectl get replicasets -n restaurant-prod

Then:

kubectl get pods -n restaurant-prod

Study the output.

You should understand:

Deployment
↓
ReplicaSet
↓
Pods
↓
Containers

QUESTION:

Who creates the ReplicaSet?

QUESTION:

Who maintains the required number of Pods?

QUESTION:

Does Jenkins continuously manage individual Pods?

Be prepared to explain these answers.

Take a screenshot:

04-deployment-replicaset-pods

==================================================

PART 5 — INSPECT THE DEPLOYMENT

Run:

kubectl describe deployment restaurant-company \
-n restaurant-prod

Find:

Replicas

Image

Labels

Selector

Strategy

Conditions

Events

Now inspect the Deployment YAML.

Find:

replicas

image

containerPort

resources

startupProbe

readinessProbe

livenessProbe

strategy

Write one sentence explaining each one.

Example:

replicas:
Defines how many application Pods Kubernetes should maintain.

DO NOT copy definitions from the internet.

Explain them in your own words.

==================================================

PART 6 — SELF-HEALING LAB

Check your Pods:

kubectl get pods -n restaurant-prod

Choose ONE Pod.

Copy its name.

Delete it:

kubectl delete pod POD-NAME -n restaurant-prod

Immediately run:

kubectl get pods -n restaurant-prod -w

WATCH WHAT HAPPENS.

You should see:

Old Pod deleted
↓
Kubernetes detects missing replica
↓
New Pod created
↓
Container starts
↓
New Pod becomes Ready

Stop watch mode with:

Ctrl + C

Take BEFORE and AFTER screenshots.

05-self-healing-before

06-self-healing-after

QUESTION:

Why did the Pod come back?

QUESTION:

Did Jenkins create the replacement Pod?

QUESTION:

What Kubernetes component was responsible for maintaining the desired number of Pods?

==================================================

PART 7 — MANUAL SCALING

Check current replicas:

kubectl get deployment restaurant-company \
-n restaurant-prod

Now scale to 5:

kubectl scale deployment restaurant-company \
--replicas=5 \
-n restaurant-prod

Watch:

kubectl get pods -n restaurant-prod -w

Wait until all Pods are Ready.

Press:

Ctrl + C

Verify:

kubectl get deployment restaurant-company \
-n restaurant-prod

QUESTION:

What was the desired state before scaling?

QUESTION:

What is the desired state now?

Now scale back to 3:

kubectl scale deployment restaurant-company \
--replicas=3 \
-n restaurant-prod

Verify:

kubectl get pods -n restaurant-prod

Take screenshot:

07-scaling

==================================================

PART 8 — SERVICE AND PODS

Check Service:

kubectl get service -n restaurant-prod

Describe the Service:

kubectl describe service restaurant-company \
-n restaurant-prod

Now run:

kubectl get pods -n restaurant-prod --show-labels

Look at:

Selector

and:

Labels

QUESTION:

How does the Service know which Pods should receive traffic?

Now check endpoints:

kubectl get endpoints -n restaurant-prod

You should understand:

Service
↓
Selector
↓
Pod Labels
↓
Matching Pods

Take screenshot:

08-service

==================================================

PART 9 — HEALTH PROBES

Open:

deployment.yaml

Find:

startupProbe

readinessProbe

livenessProbe

For each probe, write what question it answers.

STARTUP PROBE:

Has my application finished starting?

READINESS PROBE:

Is my application ready to receive traffic?

LIVENESS PROBE:

Is my application still healthy?

Now inspect a Pod:

kubectl describe pod POD-NAME -n restaurant-prod

Find information about:

Readiness

Liveness

Startup

QUESTION:

Can a container be Running but the Pod still show:

0/1 Ready?

Explain why.

==================================================

PART 10 — ROLLING UPDATE LAB

Before restarting, open another terminal.

Run:

kubectl get pods -n restaurant-prod -w

In the first terminal run:

kubectl rollout restart deployment/restaurant-company \
-n restaurant-prod

WATCH CAREFULLY.

You should see Kubernetes gradually create new Pods and terminate old Pods.

After the rollout:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

Then:

kubectl get pods -n restaurant-prod

QUESTION:

Why didn't Kubernetes delete every Pod at the same time?

Open deployment.yaml and find:

strategy:
type: RollingUpdate

Find:

maxSurge

maxUnavailable

Explain both values in your own words.

Take screenshot:

09-rolling-update

==================================================

PART 11 — BREAK THE DEPLOYMENT

Now we are going to intentionally create an incident.

THIS IS PART OF THE LAB.

Open:

deployment.yaml

FIRST:

Save your current working image value somewhere.

Example:

ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/restaurant-company:latest

Now intentionally change the tag to something that does not exist.

Example:

ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/restaurant-company:broken-version

Save the file.

Apply:

kubectl apply -f deployment.yaml \
-n restaurant-prod

Now watch:

kubectl get pods -n restaurant-prod

Wait.

You may see:

ErrImagePull

or:

ImagePullBackOff

DO NOT FIX IT YET.

==================================================

PART 12 — TROUBLESHOOT IMAGEPULLBACKOFF

Imagine this is production.

You are the DevOps Engineer.

You cannot simply say:

"It doesn't work."

STEP 1:

kubectl get pods -n restaurant-prod

Copy the failing Pod name.

STEP 2:

kubectl describe pod POD-NAME \
-n restaurant-prod

Go to:

Events

Read the error.

QUESTION:

What exactly is Kubernetes trying to do?

QUESTION:

Why is the image pull failing?

QUESTION:

Is this an application-code problem or an image/deployment problem?

Take screenshot:

10-imagepullbackoff

Now fix deployment.yaml.

Restore the correct image.

Apply again:

kubectl apply -f deployment.yaml \
-n restaurant-prod

Check:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

Then:

kubectl get pods -n restaurant-prod

Everything should return to:

Running

1/1 Ready

Take screenshot:

11-fixed-deployment

==================================================

PART 13 — PRACTICE LOG TROUBLESHOOTING

Choose a healthy Pod:

kubectl get pods -n restaurant-prod

View its logs:

kubectl logs POD-NAME \
-n restaurant-prod

Now understand this command:

kubectl logs POD-NAME \
-n restaurant-prod \
--previous

QUESTION:

When would you use --previous?

ANSWER IN YOUR OWN WORDS.

You should understand the difference:

kubectl describe
↓
Kubernetes-level information and Events

kubectl logs
↓
Application/container output

==================================================

PART 14 — CHECK KUBERNETES EVENTS

Run:

kubectl get events \
-n restaurant-prod \
--sort-by=.metadata.creationTimestamp

Study the events.

Look for events related to:

Scheduling

Image pulling

Container creation

Container start

Pod deletion

Scaling

Rolling updates

QUESTION:

Why are Kubernetes Events useful during troubleshooting?

Take screenshot:

12-events

==================================================

PART 15 — ROLLOUT HISTORY

Run:

kubectl rollout history deployment/restaurant-company \
-n restaurant-prod

Study the revisions.

Then check:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

QUESTION:

What is a rollout?

QUESTION:

What is a revision?

QUESTION:

Why is deployment history useful?

Take screenshot:

13-rollout-history

==================================================

PART 16 — ROLLBACK PRACTICE

Before doing this section, make sure your application is healthy.

Check:

kubectl get pods -n restaurant-prod

Then:

kubectl rollout history deployment/restaurant-company \
-n restaurant-prod

Practice the rollback command:

kubectl rollout undo deployment/restaurant-company \
-n restaurant-prod

Watch:

kubectl get pods -n restaurant-prod -w

Then verify:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

IMPORTANT:

After the exercise, make sure the Deployment is using the intended working image/configuration.

QUESTION:

When would you perform a rollback in production?

QUESTION:

Why might rollback be better than trying to fix a broken release directly in production?

==================================================

PART 17 — JENKINS CD

Now that you understand the Kubernetes commands manually, use Jenkins.

Open Jenkins.

Run the Restaurant Company CD pipeline.

Observe each stage.

You should understand what Jenkins is doing during:

Checkout

Verify Tools

AWS Authentication

Connect to EKS

Create Namespace

Apply Configuration

Deploy Application

Apply Production Resources

Apply Ingress

Verify Rollout

Verify Deployment

IMPORTANT QUESTION:

Why did we learn the kubectl commands manually if Jenkins can do everything automatically?

Expected idea:

Automation is useful only when you understand what is being automated.

If Jenkins fails, a DevOps Engineer must know how to troubleshoot the underlying AWS and Kubernetes operations.

Take screenshot:

14-jenkins-success

==================================================

PART 18 — VERIFY AFTER JENKINS

A green Jenkins pipeline does not mean you should stop checking.

Run:

kubectl get deployment -n restaurant-prod

kubectl get pods -n restaurant-prod

kubectl get services -n restaurant-prod

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

Optional:

kubectl get ingress -n restaurant-prod

Your goal:

Verify that the deployment is actually healthy.

==================================================

PART 19 — FINAL ARCHITECTURE

You must be able to draw and explain this WITHOUT reading notes:

Developer
↓
GitHub
↓
CI
↓
Tests
↓
SonarQube
↓
Trivy
↓
Docker Build
↓
AWS ECR
↓
Docker Image

========================

CD STARTS

Jenkins
↓
AWS Authentication
↓
AWS EKS
↓
Namespace
↓
ConfigMap / Secret
↓
ServiceAccount
↓
Deployment
↓
ReplicaSet
↓
Pods
↓
Containers
↓
Service
↓
Ingress / Load Balancer
↓
Users

==================================================

PART 20 — INCIDENT CHALLENGE

For each scenario, explain:

What does the error mean?
What command would you run first?
What would you investigate?
How would you fix or recover?

SCENARIO 1:

Pod status:

ImagePullBackOff

SCENARIO 2:

Pod status:

CrashLoopBackOff

SCENARIO 3:

Pod status:

Pending

SCENARIO 4:

Pod status:

Running

but:

READY 0/1

SCENARIO 5:

All Pods are:

Running 1/1

but users cannot access the application.

SCENARIO 6:

Jenkins deployed a new version successfully, but customers immediately report application errors.

For Scenario 6, think about:

Rollout status

Logs

Events

Application health

Rollback

==================================================

PART 21 — PRESENTATION ASSIGNMENT

NEXT CLASS:

EVERY STUDENT MUST PRESENT THEIR LAB.

Presentation time:

5–7 minutes per student.

Do NOT read definitions from Google.

Do NOT read this document during your entire presentation.

Explain the project as if you are a DevOps Engineer explaining your deployment to your team.

Your presentation should include:

ARCHITECTURE

Draw:

GitHub
↓
CI
↓
ECR
↓
Jenkins
↓
EKS
↓
Deployment
↓
ReplicaSet
↓
Pods
↓
Service
↓
Users

DEPLOYMENT

Explain:

What Jenkins does

What Kubernetes does

What ECR does

SELF-HEALING

Show:

You deleted one Pod.

Explain:

Why Kubernetes created another one.

SCALING

Show:

3 Pods
↓
5 Pods
↓
3 Pods

Explain:

What desired state means.

ROLLING UPDATE

Explain:

How Kubernetes replaces application versions without intentionally taking every replica down at once.

TROUBLESHOOTING

Show your ImagePullBackOff screenshot.

Explain:

What you broke

What status you saw

What command you used

What Events showed

How you identified the root cause

How you fixed it

ROLLBACK

Explain:

When a DevOps Engineer would roll back a production deployment.

==================================================

PART 22 — SCREENSHOTS TO SUBMIT

Submit these screenshots:

01-cluster-nodes

02-namespace

03-application-running

04-deployment-replicaset-pods

05-self-healing-before

06-self-healing-after

07-scaling

08-service

09-rolling-update

10-imagepullbackoff

11-fixed-deployment

12-events

13-rollout-history

14-jenkins-success

Do not submit screenshots only.

You must understand what each screenshot demonstrates.

==================================================

PART 23 — README

Create a file in your repository:

WEEKEND-LAB.md

Document:

Your name

Project name

Architecture

What you deployed

What each major Kubernetes resource does

Self-healing test

Scaling test

Rolling update test

The error you intentionally created

How you troubleshot it

How you fixed it

Rollback test

What you learned

==================================================

PART 24 — FINAL QUESTIONS

Before coming to class, make sure you can answer these without searching:

What is CI?

What is CD?

What is Jenkins responsible for?

What is ECR responsible for?

What is EKS?

What is a Kubernetes Deployment?

What is a ReplicaSet?

What is a Pod?

What is desired state?

What is self-healing?

What happens when a Pod dies?

What is a Service?

How does Service find Pods?

What is ConfigMap?

What is Secret?

What is ServiceAccount?

What is Startup Probe?

What is Readiness Probe?

What is Liveness Probe?

What is Rolling Update?

What is maxSurge?

What is maxUnavailable?

What is HPA?

What is PDB?

What is NetworkPolicy?

What is ImagePullBackOff?

What is CrashLoopBackOff?

What is the difference between describe and logs?

How do you check Kubernetes Events?

How do you check rollout status?

How do you perform rollback?

Why should we verify an application after Jenkins reports SUCCESS?

==================================================

FINAL GOAL

By the end of this weekend lab, you should NOT say:

"I know how to run kubectl commands."

You should be able to say:

"I understand the Kubernetes deployment lifecycle.

I can deploy an application through Jenkins to EKS.

I understand how Deployment, ReplicaSet, Pods, and Services work together.

I can verify application health.

I can test Kubernetes self-healing and scaling.

I understand Rolling Updates.

I can use kubectl get, describe, logs, events, and rollout commands for troubleshooting.

I can identify an ImagePullBackOff problem.

I understand how to investigate CrashLoopBackOff.

I can recover from a bad deployment using rollback.

Most importantly, I can explain WHY I am performing each step."

==================================================

CLEANUP / COST REMINDER

AWS resources can generate charges.

Follow the instructor's cleanup instructions for the shared/class environment.

Do NOT delete a shared EKS cluster, VPC, Load Balancer, ECR repository, or other shared AWS resource unless you were specifically instructed to do so.

If you created your own temporary AWS resources for this lab, verify what is still running after you finish.

KUBERNETES CONTINUOUS DEPLOYMENT

Aisalkyn Aidarova — Sat, 05 Sep 2026 02:43:40 +0000

Goal

After reading this material, you should understand what happens after CI creates a Docker image and how a DevOps engineer deploys, verifies, troubleshoots, and rolls back an application in Kubernetes.

==================================================

1. FROM CI TO CD

Continuous Integration (CI) prepares and validates the application.

Typical CI flow:

Developer
↓
GitHub
↓
GitHub Actions / Jenkins
↓
Lint
↓
Unit Tests
↓
SonarQube
↓
Security Scanning
↓
Trivy
↓
Docker Build
↓
AWS ECR

At the end of CI, we have a tested and scanned Docker image stored in a container registry.

Example:

restaurant-company:

AWS ECR is a container registry.

ECR stores Docker images.

ECR does NOT deploy the application.

After the image is ready, Continuous Delivery / Continuous Deployment begins.

==================================================

2. WHAT IS CONTINUOUS DEPLOYMENT?

The purpose of CD is to take an approved artifact and deploy it into an environment.

In our project:

Artifact:
Docker Image

Container Registry:
AWS ECR

CD Tool:
Jenkins

Container Platform:
Kubernetes

Managed Kubernetes Service:
AWS EKS

Our CD flow:

AWS ECR
↓
Docker Image
↓
Jenkins
↓
AWS Authentication
↓
AWS EKS
↓
Kubernetes Manifests
↓
Deployment
↓
ReplicaSet
↓
Pods
↓
Containers
↓
Service
↓
Ingress / Load Balancer
↓
Users

IMPORTANT:

Jenkins does not run our application containers.

Jenkins automates the deployment process.

Kubernetes is responsible for running and maintaining the application.

==================================================

3. WHAT DOES JENKINS DO?

In our project, Jenkins performs tasks such as:

Checkout the CD repository
Verify required tools

AWS CLI
kubectl
Git

Authenticate to AWS
Connect to the EKS cluster
Create or verify the Kubernetes namespace
Apply application configuration
Apply Kubernetes Deployment
Apply Kubernetes Service
Apply additional production resources
Verify the Kubernetes rollout
Verify that the application resources are healthy

Without Jenkins, a DevOps engineer could perform these commands manually.

Jenkins allows us to automate the same process consistently.

==================================================

4. KUBERNETES DESIRED STATE

One of the most important Kubernetes concepts is:

DESIRED STATE

We describe what we want.

Kubernetes continuously works to make the actual environment match that desired state.

Example:

replicas: 3

This means:

Desired State = 3 Pods

Current state:

Pod 1 → Running
Pod 2 → Running
Pod 3 → Running

Now imagine Pod 2 crashes.

Actual State = 2 Pods

Desired State = 3 Pods

Kubernetes detects:

Actual State != Desired State

Kubernetes creates another Pod.

This behavior is called:

SELF-HEALING

==================================================

5. DEPLOYMENT, REPLICASET, POD, CONTAINER

Understand this relationship:

Deployment
↓
ReplicaSet
↓
Pod
↓
Container

Deployment

Defines the desired application state and deployment strategy.

ReplicaSet

Maintains the required number of Pods.

Pod

The smallest deployable Kubernetes workload.

Container

Runs the actual application process inside the Pod.

Jenkins does NOT directly create and maintain individual Pods.

Jenkins applies the Deployment configuration.

Kubernetes controllers handle the rest.

==================================================

6. NAMESPACE

Namespaces logically separate Kubernetes resources.

Examples:

restaurant-dev
restaurant-qa
restaurant-staging
restaurant-prod

Our production namespace:

restaurant-prod

To see Pods in that namespace:

kubectl get pods -n restaurant-prod

If you run:

kubectl get pods

without specifying a namespace, kubectl normally checks the default namespace.

Therefore:

No resources found in default namespace

does NOT necessarily mean that your application does not exist.

You may simply be looking in the wrong namespace.

==================================================

7. DOCKER IMAGE AND ECR

The Kubernetes Deployment references a Docker image.

Example:

image: ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/restaurant-company:

The basic process is:

Kubernetes creates a Pod
↓
Scheduler selects a Node
↓
Node receives the workload
↓
Node contacts ECR
↓
Node pulls Docker image
↓
Container is created
↓
Application starts

The Docker image is NOT pushed directly into Kubernetes.

The image is stored in ECR.

Kubernetes Nodes pull the required image from ECR.

==================================================

8. WHY IMMUTABLE IMAGE VERSIONS MATTER

For learning, you may see:

restaurant-company:latest

In production, immutable version tags are generally better.

Example:

restaurant-company:a84f12c

The value could represent a Git commit SHA.

This gives us traceability.

Git Commit
a84f12c
↓
CI Pipeline
↓
Docker Image
restaurant-company:a84f12c
↓
ECR
↓
CD
↓
Kubernetes

If a production incident happens, we can identify exactly which code version was deployed.

==================================================

9. IMAGEPULLBACKOFF

A common Kubernetes error is:

ImagePullBackOff

This means Kubernetes is having trouble pulling the container image.

Possible causes include:

Wrong image repository

Wrong image tag

Image does not exist

Registry permissions

Authentication problem

Incorrect AWS account or region

CPU architecture incompatibility

The first command:

kubectl get pods -n restaurant-prod

Then investigate the Pod:

kubectl describe pod POD-NAME -n restaurant-prod

Pay special attention to:

Events

Events may show:

Failed to pull image

Image not found

Access denied

Authentication error

No match for platform in manifest

==================================================

10. THE ARCHITECTURE PROBLEM FROM OUR LAB

During our lab, we encountered:

no match for platform in manifest

The Docker image was built for:

linux/arm64

The Kubernetes worker environment required:

linux/amd64

This can happen when building containers on an Apple Silicon Mac.

We rebuilt the image for AMD64:

docker buildx build \
--platform linux/amd64 \
-t IMAGE \
--push .

The important lesson is NOT:

ImagePullBackOff always means ARM64.

It does not.

The correct lesson is:

ImagePullBackOff
↓
Investigate
↓
kubectl describe
↓
Read Events
↓
Find the actual root cause

==================================================

11. CRASHLOOPBACKOFF

Another common error is:

CrashLoopBackOff

This is different from ImagePullBackOff.

ImagePullBackOff:

Kubernetes tries to pull image
↓
Image cannot be pulled
↓
Container cannot start

CrashLoopBackOff:

Image successfully pulled
↓
Container created
↓
Container starts
↓
Process fails
↓
Container exits
↓
Kubernetes restarts it
↓
Process fails again
↓
CrashLoopBackOff

This usually means we need to investigate what is happening INSIDE the container.

==================================================

12. KUBECTL LOGS

Check container logs:

kubectl logs POD-NAME -n restaurant-prod

If the container already restarted:

kubectl logs POD-NAME -n restaurant-prod --previous

This is especially useful for:

CrashLoopBackOff

Application exceptions

Configuration problems

Permission errors

Startup failures

A useful troubleshooting idea:

kubectl get pods
↓
What is happening?

kubectl describe pod
↓
What does Kubernetes know about the problem?

kubectl logs
↓
What is the application/container reporting?

==================================================

13. BASIC TROUBLESHOOTING PROCESS

Do not randomly change YAML files when something fails.

Use a systematic process.

STEP 1

Check resources:

kubectl get pods -n restaurant-prod

STEP 2

Identify the status.

STEP 3

Describe the failing resource:

kubectl describe pod POD-NAME -n restaurant-prod

STEP 4

Read Events.

STEP 5

Check application logs:

kubectl logs POD-NAME -n restaurant-prod

STEP 6

For restarting containers:

kubectl logs POD-NAME -n restaurant-prod --previous

STEP 7

Identify the root cause.

STEP 8

Fix the configuration or application.

STEP 9

Deploy again.

STEP 10

Verify the rollout and application health.

==================================================

14. STARTUP PROBE

Startup Probe answers:

HAS THE APPLICATION FINISHED STARTING?

Some applications need time to initialize.

They may need to:

Load configuration

Connect to dependencies

Initialize caches

Start application services

Startup Probe protects slow-starting applications from being treated as unhealthy too early.

==================================================

15. READINESS PROBE

Readiness Probe answers:

IS THIS POD READY TO RECEIVE USER TRAFFIC?

A container may be running while the application is not yet ready.

Example:

Container started
↓
Application initializing
↓
Database connection starting
↓
Application not ready

During this time, Kubernetes should not send user traffic to that Pod.

Once readiness succeeds:

Pod becomes Ready
↓
Service can send traffic to it

==================================================

16. LIVENESS PROBE

Liveness Probe answers:

IS THE APPLICATION STILL HEALTHY?

Imagine:

Container = Running

but

Application = Frozen

The process may exist but no longer work correctly.

Liveness Probe can detect this condition.

After repeated failures, Kubernetes can restart the container.

Remember:

Startup Probe
→ Did the application finish starting?

Readiness Probe
→ Can it receive traffic?

Liveness Probe
→ Is it still healthy?

==================================================

17. SERVICE

Pods are temporary.

A Pod can be deleted and recreated with:

A new Pod name

A new Pod IP

Therefore, applications should not depend on individual Pod IP addresses.

Kubernetes Service provides a stable network endpoint.

Traffic flow:

User
↓
Load Balancer / Ingress
↓
Service
↓
Pod 1
Pod 2
Pod 3

Service discovers Pods using:

LABELS AND SELECTORS

Example Pod label:

app: restaurant-company

Example Service selector:

app: restaurant-company

These values must match.

If they do not match:

Service exists
↓
Pods exist
↓
But Service cannot find the Pods

==================================================

18. CONFIGMAP

ConfigMap stores non-sensitive application configuration.

Examples:

APP_ENV=production

LOG_LEVEL=info

API_URL=...

The purpose is to separate configuration from the application image.

==================================================

19. SECRET

Secrets are used for sensitive configuration.

Examples:

Database password

API token

API key

Credentials

Important:

Do not assume that simply creating a Kubernetes Secret solves all production secret-management requirements.

Production environments may integrate Kubernetes with systems such as:

AWS Secrets Manager

HashiCorp Vault

External Secrets solutions

==================================================

20. SERVICEACCOUNT

A ServiceAccount provides an identity for workloads running inside Kubernetes.

Think about it as:

Pod
↓
ServiceAccount
↓
Workload identity / permissions

Applications should receive only the permissions they actually require.

This follows the principle of:

LEAST PRIVILEGE

==================================================

21. ROLLING UPDATE

Imagine production currently runs:

Pod 1 → Version 1

Pod 2 → Version 1

Pod 3 → Version 1

A new Version 2 is released.

We do not want to delete every Version 1 Pod at the same time.

That could cause downtime.

Rolling Update gradually replaces the old Pods.

Example:

v1
v1
v1

↓

v1
v1
v1
v2

↓

v1
v1
v2

↓

v1
v2
v2

↓

v2
v2
v2

This allows the application to remain available while the release is being deployed.

==================================================

22. MAXSURGE AND MAXUNAVAILABLE

Example:

maxUnavailable: 0

maxSurge: 1

means Kubernetes can temporarily create one additional Pod during the rollout.

If desired replicas = 3:

Normal:

3 Pods

During rollout:

Up to 4 Pods

maxUnavailable: 0

means the rollout is configured to avoid intentionally reducing available replicas below the desired count during the update, subject to the Pods actually becoming Ready.

==================================================

23. ROLLING UPDATE LAB COMMANDS

Restart the Deployment:

kubectl rollout restart deployment/restaurant-company \
-n restaurant-prod

Watch the Pods:

kubectl get pods -n restaurant-prod -w

Observe:

New Pod created
↓
Container starts
↓
Readiness succeeds
↓
New Pod becomes Ready
↓
Old Pod terminates
↓
Process continues

==================================================

24. MANUAL SCALING

Suppose the Deployment currently has:

3 replicas

We can manually scale it:

kubectl scale deployment restaurant-company \
--replicas=5 \
-n restaurant-prod

Then:

kubectl get pods -n restaurant-prod

Kubernetes creates additional Pods to reach the desired state.

==================================================

25. HPA

HPA means:

Horizontal Pod Autoscaler

Instead of manually changing:

3 → 5 → 10 → 5 → 3

HPA can automatically adjust the number of replicas based on supported metrics.

Simplified example:

Application load increases
↓
Metrics increase
↓
HPA calculates more replicas are needed
↓
Deployment replica target increases
↓
Kubernetes creates additional Pods

When demand decreases, HPA can scale the workload down according to its configuration.

==================================================

26. PDB

PDB means:

PodDisruptionBudget

PDB helps protect application availability during voluntary disruptions.

Example:

3 application Pods

PDB:

minAvailable: 2

During supported voluntary disruptions, Kubernetes considers this budget when evicting Pods.

Important:

PDB does NOT guarantee that Pods can never fail.

It does not protect the application from every type of outage.

==================================================

27. NETWORKPOLICY

NetworkPolicy controls allowed network communication between workloads when supported by the cluster networking implementation.

Without appropriate controls:

Many workloads may be able to communicate with each other.

Production goal:

Only allow required communication.

Example:

Frontend
↓
Backend
↓
Database

We may want:

Frontend → Backend

Backend → Database

but NOT:

Random Pod → Database

This supports the principle of least privilege.

==================================================

28. ROLLOUT STATUS

After deployment, do not assume success simply because:

kubectl apply

completed successfully.

Check the rollout:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

Then check Pods:

kubectl get pods -n restaurant-prod

Then check Services:

kubectl get services -n restaurant-prod

A production deployment should be verified.

==================================================

29. ROLLBACK

Imagine:

10:00 AM

Version 2 deployed

10:05 AM

Customers report that the application is failing.

First investigate the impact.

Check rollout:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

Check history:

kubectl rollout history deployment/restaurant-company \
-n restaurant-prod

If the new release is causing production impact and rollback is the appropriate recovery action:

kubectl rollout undo deployment/restaurant-company \
-n restaurant-prod

Then verify:

kubectl rollout status deployment/restaurant-company \
-n restaurant-prod

The priority during a serious production incident is often:

RESTORE SERVICE FIRST

then

INVESTIGATE ROOT CAUSE

==================================================

30. COMMON KUBERNETES STATUS TROUBLESHOOTING

PENDING

Think about:

Scheduling

Worker Nodes

CPU

Memory

Taints / Tolerations

Node selectors / affinity

Storage requirements

IMAGEPULLBACKOFF

Think about:

Image repository

Image tag

Registry

ECR permissions

Authentication

Architecture

CRASHLOOPBACKOFF

Think about:

Application logs

Configuration

Environment variables

Permissions

Startup command

Dependencies

Use:

kubectl logs POD-NAME -n restaurant-prod

kubectl logs POD-NAME -n restaurant-prod --previous

RUNNING BUT 0/1 READY

Think about:

Readiness Probe

Application health

Port

Health endpoint

Dependencies

SERVICE EXISTS BUT APPLICATION IS NOT REACHABLE

Think about:

Service selector

Pod labels

Service port

targetPort

Endpoints

Ingress / Load Balancer

==================================================

31. COMMANDS YOU SHOULD KNOW

kubectl get nodes

kubectl get namespaces

kubectl get pods -n restaurant-prod

kubectl get pods -n restaurant-prod -o wide

kubectl get deployments -n restaurant-prod

kubectl get replicasets -n restaurant-prod

kubectl get services -n restaurant-prod

kubectl get ingress -n restaurant-prod

kubectl describe pod POD-NAME -n restaurant-prod

kubectl logs POD-NAME -n restaurant-prod

kubectl logs POD-NAME -n restaurant-prod --previous

kubectl get events -n restaurant-prod

kubectl rollout status deployment/restaurant-company -n restaurant-prod

kubectl rollout history deployment/restaurant-company -n restaurant-prod

kubectl rollout undo deployment/restaurant-company -n restaurant-prod

Do NOT only memorize these commands.

Understand:

WHEN to use them

WHY to use them

WHAT information they provide

==================================================

32. COMPLETE PROJECT ARCHITECTURE

Developer
↓
GitHub
↓
CI Pipeline
↓
Lint
↓
Unit Tests
↓
SonarQube
↓
Security Scanning
↓
Trivy
↓
Docker Build
↓
AWS ECR

==============================

CONTINUOUS DEPLOYMENT

==================================================

33. INTERVIEW QUESTION

QUESTION:

Walk me through how you deploy an application to Kubernetes.

SAMPLE ANSWER:

Our CI pipeline validates the application through testing, code-quality checks, and security scanning.

It then builds the Docker image and publishes the approved image to our container registry.

Our CD pipeline authenticates to the target environment and deploys the Kubernetes manifests to EKS.

The Deployment defines the desired replicas, image, resource requirements, health probes, and rollout strategy.

Kubernetes creates and manages the ReplicaSets and Pods, while Services provide stable networking.

We monitor the rollout and verify Pod and application health after deployment.

If the deployment fails, I check the Pod status, Kubernetes events, container logs, configuration, probes, and image information to identify the root cause.

If a new release is causing production impact, we can roll back to the previous stable version and verify service health before continuing the root-cause investigation.

==================================================

34. QUESTIONS YOU SHOULD BE ABLE TO ANSWER

What is CI?
What is CD?
What does Jenkins do in our CD pipeline?
What does AWS ECR do?
Does ECR deploy the application?
What does a Kubernetes Deployment do?
What is desired state?
What is a ReplicaSet?
What is a Pod?
What happens if a Pod dies?
What is Kubernetes self-healing?
What is a Service?
How does a Service find Pods?
What is the difference between ConfigMap and Secret?
What is a ServiceAccount?
What is Startup Probe?
What is Readiness Probe?
What is Liveness Probe?
What is ImagePullBackOff?
What is CrashLoopBackOff?
How do you check Pod logs?
Why would you use kubectl describe?
What is Rolling Update?
What is maxSurge?
What is maxUnavailable?
What is HPA?
What is PDB?
What is NetworkPolicy?
How do you check rollout status?
How do you roll back a bad deployment?

==================================================

35. HOMEWORK

Do not just read this material.

Open your Kubernetes YAML files and identify where each concept exists.

Find:

namespace.yaml

deployment.yaml

service.yaml

configmap.yaml

secrets.yaml

serviceaccount.yaml

hpa.yaml if available

pdb.yaml

networkpolicy.yaml

ingress.yaml

Jenkinsfile

For each file, be able to explain:

What is this resource?
Why do we need it?
What problem does it solve?
What happens if it is configured incorrectly?
How would you verify that it is working?

Finally, practice explaining this complete flow without reading:

Developer
↓
GitHub
↓
CI
↓
Docker Image
↓
ECR
↓
Jenkins CD
↓
EKS
↓
Deployment
↓
ReplicaSet
↓
Pods
↓
Service
↓
Ingress / Load Balancer
↓
User

HOMEWORK GOAL:

Do not memorize Kubernetes.

Understand how the components work together.

A DevOps engineer must be able to:

DEPLOY

VERIFY

TROUBLESHOOT

ROLL BACK

and explain WHY each step is necessary.

DevOps + AI — Day 1 Notes

Aisalkyn Aidarova — Tue, 01 Sep 2026 17:21:43 +0000

Today we started from the very beginning: What is a computer, how does it work, what is hardware, what is software, what is an operating system, and how do we communicate with a computer through Terminal?

The most important idea from today is this:

YOU
 ↓
give instructions
 ↓
COMPUTER
 ↓
executes instructions
 ↓
RESULT

A computer does not guess what you mean. It follows the instructions you give it.

That is why, when we work in DevOps, our commands and scripts need to be clear and correct.

1. Hardware

Hardware means the physical parts of a computer.

These are things you can physically touch.

Examples:

CPU
RAM
SSD / HDD
Motherboard
Network Card
Keyboard
Mouse
Monitor
GPU

CPU

CPU means:

Central Processing Unit

You can think of the CPU as the part of the computer that processes instructions and performs calculations.

For example:

You click something
       ↓
Software gives instructions
       ↓
CPU processes instructions
       ↓
You get a result

2. RAM

RAM means:

Random Access Memory

RAM is temporary working memory.

Imagine you open:

Chrome
Zoom
VS Code
Slack

While those applications are running, the computer needs temporary memory to work with them.

That is RAM.

Simple example:

RAM = your working desk

The bigger your desk is, the more things you can work with at the same time.

When you restart or shut down the computer, the temporary information in RAM disappears.

3. Storage — SSD / HDD

Storage keeps information for a longer time.

Examples:

Photos
Documents
Videos
Applications
Operating System
Code

Today most computers use:

SSD

Older computers commonly used:

HDD

Simple comparison:

RAM = temporary

SSD/HDD = permanent storage

4. Software

Software means programs and instructions that tell the computer what to do.

Examples:

Chrome
Zoom
Microsoft Word
Slack
VS Code
Docker
Git
Jenkins

A very simple way to remember:

Hardware = physical computer

Software = instructions/programs

5. Operating System

One very important type of software is the Operating System.

Examples:

Windows
macOS
Linux

The operating system manages the computer's resources.

For example:

CPU
RAM
Disk
Network
Processes
Devices

Applications normally do not directly control all of the hardware themselves.

A simplified picture is:

USER
 ↓
APPLICATION
 ↓
OPERATING SYSTEM
 ↓
KERNEL
 ↓
HARDWARE

6. What is Kernel?

The Kernel is the core part of the operating system.

It helps manage things such as:

CPU
Memory
Processes
Disk
Network
Devices

Simplified:

Applications
      ↓
Operating System
      ↓
Kernel
      ↓
CPU / RAM / Disk / Network

One important fact:

Linux is technically a kernel.

Ubuntu, Debian, Red Hat and others build complete operating-system environments around the Linux kernel.

We will talk much more about this later.

7. Why do DevOps Engineers learn Linux?

A large amount of infrastructure and many servers run Linux.

As DevOps engineers, we will eventually work with technologies such as:

AWS
Docker
Kubernetes
Jenkins
Terraform
Ansible
Git
Nginx

Linux knowledge is therefore one of our foundations.

Today we started learning how to communicate with an operating system using the Terminal.

8. What is Terminal?

Terminal is an application where we can give instructions to our computer by typing commands.

Instead of clicking:

Finder
→ Documents
→ Folder
→ File

we can tell the computer what we want using commands.

For example:

pwd

ls

cd Documents

This is called using the Command Line Interface — CLI.

You will hear this term many times:

CLI = Command Line Interface

9. Important: The Terminal Prompt

On Mac you may see something like:

aisalkyn@MacBook-Air ~ %

The % is the Terminal prompt.

You do not type the % when I give you commands.

For example, if your screen shows:

aisalkyn@MacBook-Air ~ %

and I give you:

pwd

you only type:

pwd

and press Enter.

10. First Command — `whoami`

Run:

whoami

whoami asks:

Which user am I currently using?

Example output:

aisalkyn

It means you are currently logged in as that user.

Remember:

whoami = who am I?

11. `pwd` — Where Am I?

Run:

pwd

PWD means:

Print Working Directory

This command tells you where you currently are in the filesystem.

Example:

/Users/aisalkyn

Think about it like GPS.

If you are lost, run:

pwd

You are asking:

Computer, where am I?

12. `ls` — What Is Here?

Run:

ls

ls means list.

It shows files and directories inside your current location.

Example:

Applications
Desktop
Documents
Downloads
Library
Movies
Pictures

Think:

pwd = Where am I?

ls = What is here?

13. `ls -l` — Detailed List

Run:

ls -l

This gives more information about files and directories.

You may see something like:

drwxr-xr-x   5 user staff 160 Sep 1 Documents
-rw-r--r--   1 user staff 200 Sep 1 hello.txt

Don't worry about understanding all of this today.

Later we will learn:

permissions
owner
group
file size
date

For now remember:

ls

Simple list.

ls -l

Detailed list.

14. What Is a Directory?

A directory is basically what you normally call a folder.

These two words are commonly used:

Folder
Directory

In Linux and DevOps you will hear directory very often.

Example:

Downloads
Documents
Desktop

These are directories.

15. `cd` — Change Directory

cd means:

Change Directory

It allows us to move from one directory to another.

For example:

cd Downloads

Now check:

pwd

You may see:

/Users/yourname/Downloads

You moved from:

/Users/yourname

to:

/Users/yourname/Downloads

16. `cd ..` — Go Back One Level

Run:

cd ..

The two dots:

..

mean:

parent directory

or simply:

Go one directory back/up.

Example:

/Users/student/Downloads

Run:

cd ..

Now:

/Users/student

17. `~` — Your Home Directory

This symbol:

represents your home directory.

For example:

cd ~

takes you home.

Then run:

pwd

You may see:

/Users/student

Therefore:

cd ~

means:

Take me to my home directory.

18. A Useful Shortcut

Instead of:

cd Downloads

you can write:

cd ~/Downloads

This means:

Go to my home directory
then
go to Downloads

19. `mkdir` — Create a Directory

mkdir means:

Make Directory

Let's create our first DevOps directory.

Run:

mkdir devops

Now check:

ls

You should see:

devops

Congratulations — you created a directory using the command line.

20. Enter the Directory

Run:

cd devops

Now check where you are:

pwd

You should see something similar to:

/Users/yourname/devops

21. `touch` — Create a File

Let's create our first file.

Run:

touch day1.txt

Now:

ls

You should see:

day1.txt

We created an empty file.

22. `cat` — Read a File

Run:

cat day1.txt

Nothing may appear because the file is currently empty.

cat can be used to display the contents of a file.

Remember:

cat = show me what is inside this file

23. `echo` — Print Text

Run:

echo "Hello DevOps"

You should see:

Hello DevOps

The computer printed the text back to you.

24. Put Text Into a File

Now run:

echo "Hello DevOps" > day1.txt

We are telling the computer:

Take:
Hello DevOps

and put it into:
day1.txt

Now read the file:

cat day1.txt

You should see:

Hello DevOps

25. What Does `>` Mean?

This symbol:

is called redirection.

For now, understand it simply as:

Send this output into this file.

Example:

echo "I am learning DevOps" > student.txt

Then:

cat student.txt

Output:

I am learning DevOps

We will learn redirection in much more detail later.

26. `cp` — Copy

cp means:

copy

Create a file:

touch engineer.txt

Copy it:

cp engineer.txt engineer-backup.txt

Now:

ls

You should see:

engineer.txt
engineer-backup.txt

The original file still exists.

We created a copy.

27. `mv` — Move or Rename

mv means:

move

but it can also be used to rename something.

Example:

mv engineer.txt devops-engineer.txt

Now run:

ls

Instead of:

engineer.txt

you should see:

devops-engineer.txt

We renamed the file.

28. Move a File Into Another Directory

Create directories:

mkdir developers
mkdir devops-team

Create a file:

touch employee.txt

Move the file:

mv employee.txt devops-team/

Now check:

ls

The file disappeared from the current directory because we moved it.

Check inside:

ls devops-team

You should see:

employee.txt

29. `rm` — Delete a File

rm means:

remove

Create a test file:

touch delete-me.txt

Check:

ls

Now delete it:

rm delete-me.txt

Check again:

ls

The file is gone.

⚠️ Be careful with rm.

Terminal does not always behave like Finder with a Trash folder.

Do not randomly use commands like:

rm -rf

You will learn that command later when you understand exactly what it does.

30. Our Main Commands From Today

Keep this cheat sheet:

whoami     Who am I?

pwd        Where am I?

ls         What is here?

ls -l      Show detailed information

cd         Change directory

cd ..      Go back one directory

cd ~       Go to home directory

mkdir      Create directory

touch      Create file

echo       Print/write text

cat        Read file

cp         Copy file

mv         Move or rename

rm         Remove file

31. Think of Commands Like English Instructions

This was the idea behind our game today.

You told the "computer":

Stand up
Turn right
Walk
Stop
Pick something up

Terminal is the same idea.

Instead of:

Create a folder.

we say:

mkdir folder

Instead of:

Go inside the folder.

we say:

cd folder

Instead of:

Tell me where I am.

we say:

pwd

Instead of:

Show me what is here.

we say:

ls

This is why learning commands becomes easier when you understand what you're asking the computer to do.

32. Practice Lab — Do This at Home

Open Terminal.

First go home:

cd ~

Check:

pwd

Create a practice directory:

mkdir devops-practice

Enter it:

cd devops-practice

Check:

pwd

Create three directories:

mkdir developers
mkdir devops
mkdir managers

Check:

ls

You should see:

developers
devops
managers

Enter DevOps:

cd devops

Create a file:

touch student.txt

Add text:

echo "I am learning DevOps" > student.txt

Read it:

cat student.txt

Copy it:

cp student.txt student-backup.txt

Check:

ls

Rename it:

mv student.txt devops-engineer.txt

Check:

ls

Delete the backup:

rm student-backup.txt

Check again:

ls

Go back:

cd ..

Check:

pwd

Then:

ls

33. Final Structure

If everything worked, you should have something similar to:

devops-practice/
│
├── developers/
│
├── devops/
│   └── devops-engineer.txt
│
└── managers/

And inside:

devops-engineer.txt

you should have:

I am learning DevOps

34. Challenge — Don't Look at the Answers First

Try this by yourself:

Create a directory called:

company

Inside company, create:

frontend
backend
devops

Inside devops, create:

engineer.txt

Put this text inside:

I want to become a DevOps Engineer

Then:

Display the contents of the file.
Make a copy called engineer-backup.txt.
Rename engineer.txt to senior-devops.txt.
Delete the backup.
Show all remaining files.

The commands you will need are:

mkdir
cd
pwd
ls
touch
echo
cat
cp
mv
rm

Try to solve it without looking at today's examples first.

35. Important Mac Issue We Saw Today

Some students using macOS received:

ls: .: Operation not permitted

For example:

cd ~/Downloads
ls

and macOS blocked access.

This does not necessarily mean your ls command is wrong.

macOS has privacy protections that may prevent Terminal from reading folders such as:

Downloads
Documents
Desktop

If this happens, go to:

System Settings
→ Privacy & Security
→ Files and Folders

Look for Terminal and allow access.

If Terminal is not listed or the problem continues, check:

System Settings
→ Privacy & Security
→ Full Disk Access

Add/enable:

Terminal.app

Then completely quit Terminal:

Command + Q

Open Terminal again and retry:

cd ~/Downloads
ls

This was actually our first real troubleshooting example.

The command was correct, but the operating system's security settings prevented the command from accessing the folder.

36. Most Important Lesson From Day 1

Do not memorize commands without understanding them.

Before typing anything, ask yourself:

Where am I?

Where do I want to go?

What do I want the computer to do?

What result am I expecting?

Then choose the command.

Today:

Human instruction
       ↓
Computer instruction
       ↓
Linux/Unix command
       ↓
Result

Later we will go from individual commands to:

Commands
   ↓
Scripts
   ↓
Automation
   ↓
CI/CD
   ↓
Cloud
   ↓
Docker
   ↓
Kubernetes
   ↓
Production

But everything starts with the same principle we learned in our game:

Give the computer clear instructions, understand what those instructions do, and verify the result.

Homework

Repeat the commands from today at least 2–3 times. Do not simply copy and paste everything. Type the commands yourself so your hands and brain start remembering them.

Tomorrow, before class, you should be comfortable answering:

What is hardware?
What is software?
What is an operating system?
What is a kernel?
What is Terminal?
What does pwd do?
What does ls do?
What does cd do?
What does mkdir do?
What does touch do?
What does cat do?
What does cp do?
What does mv do?
What does rm do?

If you can explain these in your own words and complete the practice lab without looking at the answers, Day 1 is complete.

DevOps Lab: CI/CD Deployment to AWS ECS

Aisalkyn Aidarova — Tue, 25 Aug 2026 02:55:34 +0000

Scenario

You are the DevOps Engineer.

Jules is your Software Developer.

Ask Jules to create a professional website for a product of your choice. Jules is responsible for application development. You are responsible for the entire DevOps lifecycle.

The final result must be:

https://yourdomain.com

The browser must display the name of your product.

Part 1 — Application Development

Jules acts as the developer.

The application must have:

Professional homepage
Product name
Multiple sections
Responsive design
Working application

Application code must eventually be stored in your own GitHub repository.

Architecture begins:

Jules
  ↓
Application Code
  ↓
GitHub Repository

Part 2 — AWS Infrastructure

Create a production-style AWS environment.

VPC
10.0.0.0/16

├── Public Subnet — us-east-1a
├── Public Subnet — us-east-1b
│
├── Private Subnet — us-east-1a
└── Private Subnet — us-east-1b

The architecture must support:

Internet
    ↓
Internet Gateway
    ↓
Application Load Balancer
    ↓
Public Subnets
    ↓
Target Group
    ↓
ECS
    ↓
Private Subnets

Private workloads must have appropriate outbound connectivity.

Create:

VPC
Public/private subnets
Route tables
Internet Gateway
NAT/outbound connectivity
Security groups
ECR repository
ECS cluster
ECS task definition
ECS service
Target group
Application Load Balancer

Part 3 — Continuous Integration (CI)

CI starts when application code changes.

Use GitHub Actions for Continuous Integration.

Developer
    ↓
git push
    ↓
GitHub
    ↓
GitHub Actions

Checkout
    ↓
Install Dependencies
    ↓
Test
    ↓
SonarQube
    ↓
Docker Build
    ↓
Trivy Scan
    ↓
Authenticate to AWS
    ↓
Push Docker Image
    ↓
Amazon ECR

CI responsibility

CI answers:

Is this application safe and ready to deploy?

The CI pipeline must perform:

Source code checkout

Dependency installation

Application tests

SonarQube code-quality analysis

Docker image build

Trivy vulnerability scanning

Docker image tagging

AWS authentication

ECR authentication

Push image to ECR

Use versioned/immutable image tags such as:

product-name:a83fd91

or:

product-name:v1.0.0

Do not depend only on:

latest

CI finishes here

GitHub
   ↓
GitHub Actions
   ↓
SonarQube
   ↓
Docker
   ↓
Trivy
   ↓
ECR

Do not deploy to ECS inside the CI section.

Part 4 — Continuous Deployment (CD)

CD begins after a validated container image exists in ECR.

Use Jenkins for Continuous Deployment.

Amazon ECR
     ↓
Jenkins CD Pipeline
     ↓
New ECS Task Definition Revision
     ↓
ECS Service Update
     ↓
ECS Rolling Deployment
     ↓
Target Group Health Check
     ↓
Application Load Balancer

CD responsibility

CD answers:

How do we safely release the approved application to the environment?

The Jenkins deployment should use the specific image produced by CI.

For example:

ECR

product-name:a83fd91
        ↓
Jenkins
        ↓
ECS Task Definition :7
        ↓
ECS Service
        ↓
Rolling Deployment

During deployment, observe:

OLD VERSION

Task 1 → v1
Task 2 → v1


DEPLOYMENT STARTS

Task 1 → v1
Task 2 → v1
Task 3 → v2


HEALTH CHECK PASSES

Task 1 → v2
Task 2 → v2


OLD TASKS TERMINATED

Students must observe the deployment instead of simply clicking Update and considering the lab finished.

Part 5 — ALB

The Application Load Balancer must be Internet-facing.

ALB:

Public Subnet 1A
+
Public Subnet 1B

ECS:

Private Subnet 1A
+
Private Subnet 1B

Traffic:

Customer
   ↓
ALB
   ↓
Listener
   ↓
Target Group
   ↓
ECS Service
   ↓
Container

The target group must report:

Healthy

Part 6 — Domain and HTTPS

Configure:

Domain
   ↓
DNS
   ↓
ALB

Request/configure an SSL/TLS certificate and configure the ALB HTTPS listener.

Final traffic:

https://yourdomain.com
          ↓
        DNS
          ↓
       ALB :443
          ↓
     Target Group
          ↓
     ECS Service
          ↓
      Container

The browser must show a valid HTTPS connection and display your product name.

Final Architecture

                    DEVELOPER
                       │
                     Jules
                       │
                       ▼
                     GitHub
                       │
             ┌─────────▼─────────┐
             │       CI          │
             │  GitHub Actions   │
             └─────────┬─────────┘
                       │
                  SonarQube
                       │
                  Docker Build
                       │
                   Trivy Scan
                       │
                       ▼
                     ECR
                       │
            CI ENDS ───┼─── CD STARTS
                       │
                       ▼
                    Jenkins
                       │
                       ▼
               ECS Task Revision
                       │
                       ▼
                 ECS Service
                 Private Subnets
                       │
                       ▼
                  Target Group
                       │
                       ▼
                       ALB
                  Public Subnets
                       │
                       ▼
                   HTTPS :443
                       │
                       ▼
                     Domain
                       │
                       ▼
                    Customer

Lab is successful when

CI
GitHub push triggers GitHub Actions
SonarQube analysis passes
Docker image builds
Trivy scans the image
Versioned image reaches ECR

CD
Jenkins deploys the ECR image
New ECS task definition revision is created
ECS performs a rolling deployment
New tasks become healthy
Old tasks terminate
Target group reports healthy

PRODUCTION
ALB is public
ECS workload is private
Domain points to ALB
HTTPS works
https://yourdomain.com opens successfully
Product name appears in the browser

The most important boundary for students to understand is:

CI = Code → tested/scanned container image in ECR

CD = ECR image → running application in ECS → ALB → HTTPS → customer.

Lab: From LLM RAG MCP

Aisalkyn Aidarova — Thu, 13 Aug 2026 15:18:15 +0000

STAGE 1 — LLM
User → LLM → Answer
              ↑
       model knowledge only


STAGE 2 — RAG
User → Retrieval → Internal Runbook
          ↓
         LLM → Answer
               ↑
        private knowledge


STAGE 3 — MCP
User → LLM → MCP Client → MCP Server → DevOps Tool
                                         ↓
                                   live/simulated data

Use this scenario throughout:

We are DevOps engineers supporting payment-service.

And ask:

"Why is payment-service failing?"

Part 0 — Create the project

Students create:

mkdir llm-rag-mcp-lab
cd llm-rag-mcp-lab

python3 -m venv venv
source venv/bin/activate

Create:

llm-rag-mcp-lab/
│
├── 01_llm.py
├── 02_rag.py
├── 03_mcp_server.py
├── 04_mcp_client.py
│
├── knowledge/
│   └── production_runbook.txt
│
└── requirements.txt

The important teaching point is that every stage adds one capability.

PART 1 — Demonstrate the LLM itself

Before using RAG or MCP, make the LLM answer something it cannot possibly know.

Create:

01_llm.py

For a classroom lab, you can use an API-backed model or a local model. The important experiment is the same.

Conceptually:

question = """
What database does our internal payment-service use,
and what should we check if it returns HTTP 500?
"""

response = llm(question)

print(response)

Ask students before running it

"Will the LLM know our company's payment-service?"

No.

It may answer:

The payment service may use PostgreSQL, MySQL,
or another database.

For HTTP 500 errors, check:
- application logs
- database connectivity
- environment variables
...

It can give a reasonable answer.

But ask:

"Where did PostgreSQL come from?"

It doesn't know.

Maybe PostgreSQL. Maybe MySQL. Maybe DynamoDB.

This demonstrates the first concept:

                  LLM
                   │
         ┌─────────┴─────────┐
         │                   │
      KNOWS              DOESN'T KNOW
         │                   │
   Kubernetes          our passwords
   Terraform           our runbooks
   Python              today's deployment
   AWS concepts        our Jira tickets
                       our production state

Experiment that makes this very obvious

Ask:

What is the database hostname for
JumpToTech payment-service production?

The model cannot reliably know it.

Student conclusion #1

Write this on the board:

LLM = reasoning/generation over the information available in its context and learned model behavior.

It does not automatically know your private environment.

Now tell them:

"We need to give our AI company knowledge."

That introduces RAG naturally.

PART 2 — Demonstrate RAG

Now create private information that the LLM has never seen.

Create:

knowledge/production_runbook.txt

Put:

JUMPToTECH BANKING PLATFORM
PRODUCTION RUNBOOK

Application:
payment-service

Environment:
production

Kubernetes Cluster:
banking-prod-eks

AWS Region:
us-east-1

Database:
PostgreSQL

Database Host:
payments-prod.internal

Common Incident:
Payment service returns HTTP 500 when
DB_HOST is configured incorrectly.

Troubleshooting procedure:

1. Check payment-service pod status.
2. Check application logs.
3. Verify DB_HOST.
4. Test connectivity to PostgreSQL.
5. Check Kubernetes Secret payment-db-secret.

Rollback command:

helm rollback payment-service 3

Tell students:

"We invented this information five seconds ago. The LLM was never trained on it."

That's what makes the demonstration powerful.

Step 2.1 — Ask the LLM again

Same question:

What database does our payment-service use?

Without providing the document, the LLM still doesn't know.

Now we build retrieval.

Step 2.2 — Understand RAG before coding

Take the document:

production_runbook.txt

Split it:

            DOCUMENT
                │
                ▼
             CHUNKS

┌────────────────────────────┐
│ payment-service            │
│ environment: production    │
└────────────────────────────┘

┌────────────────────────────┐
│ Database: PostgreSQL       │
│ Host: payments-prod...     │
└────────────────────────────┘

┌────────────────────────────┐
│ HTTP 500 occurs when       │
│ DB_HOST is incorrect       │
└────────────────────────────┘

Then:

Chunks
   ↓
Embeddings
   ↓
Vectors
   ↓
Vector/Search Store

Student asks:

Why is payment-service returning HTTP 500?

Convert the question to a searchable representation and find the most relevant chunk.

Conceptually:

Question
   ↓
Retrieval
   ↓

Result #1
"HTTP 500 occurs when DB_HOST
is configured incorrectly."

   ↓

Question + Result
   ↓
LLM

Now the model has evidence.

Step 2.3 — Make retrieval visible

This is critical for teaching.

Don't just display the final answer.

Your program should print something like:

===============================
USER QUESTION
===============================

Why is payment-service returning HTTP 500?


===============================
RETRIEVED CONTEXT
===============================

Common Incident:
Payment service returns HTTP 500 when
DB_HOST is configured incorrectly.

Troubleshooting:
Verify DB_HOST.
Test PostgreSQL connectivity.


===============================
LLM ANSWER
===============================

According to the production runbook,
a common cause is an incorrectly configured
DB_HOST.

Check DB_HOST and verify connectivity
to PostgreSQL.

Now stop.

Ask students:

"Where did the answer come from?"

Not just the LLM.

Question
   ↓
RETRIEVE
   ↓
AUGMENT
   ↓
GENERATE

That's literally the name:

R = Retrieval
A = Augmented
G = Generation

Student conclusion #2

RAG retrieves relevant information and adds it to the LLM's context before generation.

PART 3 — Demonstrate why RAG still isn't enough

Now ask:

How many payment-service pods are currently running?

Your runbook doesn't contain that.

Even with RAG:

RAG
 ↓
Search documents
 ↓
???

Why?

Because this isn't really a static-document question.

You need current system information.

Normally a DevOps engineer might run:

kubectl get pods

This gives us the transition to tools/MCP.

Tell students:

"Reading knowledge is different from interacting with an external system."

PART 4 — Demonstrate MCP

For the first class, do not connect production AWS/EKS.

Create a simulated DevOps tool.

Your MCP server exposes something conceptually like:

get_pods()

And it returns:

payment-service-7d8f9    Running
payment-service-2ks91    Running
payment-service-8sa21    CrashLoopBackOff

Architecture:

                   AI APPLICATION
                         │
                         ▼
                       LLM
                         │
                  "I need pod data."
                         │
                         ▼
                    MCP CLIENT
                         │
                  MCP protocol
                         │
                         ▼
                    MCP SERVER
                         │
                    get_pods()
                         │
                         ▼
                 Kubernetes/API

For the classroom, the bottom can initially be simulated.

Step 4.1 — Make the MCP server obvious

Have the server expose three safe tools:

get_pods()

get_logs(pod)

get_deployment(service)

For example:

get_pods()

returns:

NAME                        STATUS
payment-service-abc         Running
payment-service-def         Running
payment-service-xyz         CrashLoopBackOff

And:

get_logs("payment-service-xyz")

returns:

ERROR:
connection to payments-prod.internal:5432 failed

FATAL:
database connection refused

Step 4.2 — Let the LLM choose the tool

Student asks:

Why is payment-service failing?

Now trace every operation on screen:

USER
│
│ Why is payment-service failing?
│
▼
LLM

"I need current information."

│
▼
MCP TOOL CALL

get_pods()

│
▼

payment-service-xyz
CrashLoopBackOff

│
▼
LLM

"I need logs."

│
▼
MCP TOOL CALL

get_logs("payment-service-xyz")

│
▼

database connection refused

│
▼
LLM

"The payment-service pod is crashing
because it cannot connect to the database."

This is where students usually understand MCP.

The LLM didn't magically execute Kubernetes commands.

A tool was made available through the integration.

PART 5 — The best experiment: compare all three

Now put three terminals/windows next to each other.

Window 1 — LLM

Ask:

Why is payment-service failing?

Result:

Possible causes include:
CPU,
memory,
database,
networking,
configuration...

Tell students:

Generic reasoning.

Window 2 — RAG

Same question.

Retrieved:

Company Runbook:

HTTP 500 can occur because of incorrect DB_HOST.

Answer:

According to the runbook,
check DB_HOST and PostgreSQL connectivity.

Tell students:

Company knowledge.

Window 3 — MCP

Same question.

AI calls:

get_pods()

Result:

CrashLoopBackOff

Then:

get_logs()

Result:

database connection refused

Answer:

The current payment-service pod is failing
because its database connection is refused.

Tell students:

Tool/system information.

PART 6 — Then combine RAG + MCP

This is the final "aha" moment.

Student asks:

Payment-service is failing.

Investigate the problem and tell me
how our company runbook says to fix it.

Now both capabilities are required.

                         USER
                          │
                          ▼
                         LLM
                     /          \
                    /            \
                   ▼              ▼
                 MCP             RAG
                  │               │
          Current system       Company
             state             knowledge
                  │               │
                  ▼               ▼
             get_pods()       Runbook
                  │
                  ▼
             get_logs()
                  │
                  └──────┬────────┘
                         ▼
                        LLM
                         │
                         ▼
                    FINAL ANSWER

MCP tells us:

Current pod:
CrashLoopBackOff

Current logs:
database connection refused

RAG tells us:

Company procedure:

1. Verify DB_HOST
2. Test PostgreSQL connectivity
3. Check payment-db-secret
4. If required:
   helm rollback payment-service 3

LLM combines them:

The current payment-service pod is in
CrashLoopBackOff because the database
connection is being refused.

According to the production runbook:

1. Verify DB_HOST.
2. Check payment-db-secret.
3. Test connectivity to PostgreSQL.
4. If the issue was introduced by the latest
   deployment, rollback payment-service.

Now you can ask:

"Which component generated the explanation?"

LLM.

"Which component found our company's troubleshooting procedure?"

RAG.

"Which component let the AI application obtain current pod/log information through exposed tools?"

MCP/tool integration.

That's the distinction you want students to leave class understanding.

One important improvement for your class

I would actually make this a real executable lab, not just conceptual Python.

We can build it so students have:

llm-rag-mcp-lab/

01-llm/
   app.py

02-rag/
   app.py
   runbook.txt

03-mcp/
   server.py
   client.py

04-complete-ai-devops/
   app.py

And when they run:

python app.py

they see clearly labeled output:

🤖 LLM
🔎 RAG RETRIEVAL
🔧 MCP TOOL CALL
📄 TOOL RESULT
🤖 FINAL ANSWER

LAB: Build an AI DevOps Incident Assistant with LLM + RAG + MCP

Aisalkyn Aidarova — Mon, 10 Aug 2026 14:10:13 +0000

What we are building

Imagine your company has this problem:

Developer:
"payment-service is returning 503.
Why is it failing and what should I do?"

An ordinary LLM might know general Kubernetes troubleshooting, but it does not automatically know:

our company's payment-service
our runbook
our deployment version
our internal incidents
our current pod status
our current application logs
our company procedures

So we will build this:

                         USER
                          |
                          |
                          v
                 "Why is payment-service
                     returning 503?"
                          |
                          v
                +-------------------+
                |   Python AI App   |
                +-------------------+
                    /           \
                   /             \
                  v               v
             +---------+       +---------+
             |   RAG   |       |   MCP   |
             +---------+       +---------+
                  |                |
                  |                |
                  v                v
         Company runbooks      Live operational
         documentation         information
                  |                |
                  v                v
             embeddings        MCP tools
                  |                |
                  +-------+--------+
                          |
                          v
                     +---------+
                     |   LLM   |
                     +---------+
                          |
                          v
               Explanation + action

The most important lesson of the entire lab is:

LLM
=
generates/reasons over text

RAG
=
provides relevant KNOWLEDGE to the LLM

MCP
=
provides standardized access to DATA and CAPABILITIES

DevOps
=
deploys, secures, monitors, scales and operates
all of the above

PART 0 — Before writing code: what exactly is an LLM?

Do this explanation on the whiteboard before opening VS Code.

0.1 Artificial Intelligence

AI is the broad category.

Artificial Intelligence

Machines performing tasks
that appear intelligent

Examples:

recommendations
computer vision
speech recognition
fraud detection
chatbots
autonomous systems

AI is the umbrella.

0.2 Machine Learning

Traditional programming looks like this:

RULES + DATA
      |
      v
    OUTPUT

For example:

if cpu > 90:
    print("High CPU")

A human explicitly wrote the rule.

Machine learning changes the idea:

DATA + EXPECTED RESULTS
          |
          v
        TRAINING
          |
          v
         MODEL

Instead of manually writing every rule, the machine learns patterns from examples.

For your students, use this analogy:

Traditional programming:

Engineer writes:
IF CPU > 90
THEN alert


Machine learning:

Give system thousands of examples:

CPU
Memory
Latency
Requests
Failures

and tell it:

NORMAL
ABNORMAL
NORMAL
ABNORMAL

The model learns relationships.

0.3 Deep Learning

Deep learning is a type of machine learning using neural networks with many layers.

You don't need to teach the mathematics first.

Tell students:

Machine Learning
      |
      v
Deep Learning
      |
      v
Neural Networks

0.4 What is a model?

This word confuses beginners.

A model is the learned mathematical system produced through training.

Think:

training data
     |
     v
 training process
     |
     v
   MODEL

A deployed model can then receive new input.

That phase is called:

Inference

USER INPUT
    |
    v
 pretrained model
    |
    v
 generated output

In this lab we are not training GPT.

We are doing inference.

That's a critical distinction.

0.5 What does LLM mean?

LLM:

Large
Language
Model

Large

Large amounts of:

parameters
training data
compute

Language

It processes language representations.

Examples:

English
Russian
Python
Java
YAML
Terraform
JSON
Kubernetes manifests
logs

Model

It is a trained mathematical model.

0.6 The most simplified mental model

When you type:

Kubernetes pod is crashing because...

an LLM essentially predicts what text should follow based on learned patterns.

Conceptually:

INPUT

"The Kubernetes pod is..."

        |
        v

      LLM

        |
        v

possible continuation probabilities

crashing      32%
running       18%
failing       15%
unable        11%
...

        |
        v

generated output

Real modern LLMs are much more sophisticated than this simplified diagram, but this is the right first mental model.

0.7 What is a token?

The LLM doesn't operate on your sentence exactly as humans see it.

Text is divided into tokens.

For teaching purposes:

"Kubernetes deployment failed"

could conceptually become pieces such as:

Kuber
netes
deployment
failed

Do not tell students one word always equals one token.

It doesn't.

The important idea is:

text
 ↓
tokens
 ↓
numbers
 ↓
model processing
 ↓
tokens
 ↓
text

0.8 Why can an LLM hallucinate?

Because the LLM is generating a likely response.

It is not automatically doing:

SELECT *
FROM company_database
WHERE truth = true;

An LLM may produce something that sounds correct but isn't grounded in your company's actual data.

That is one reason RAG exists.

PART 1 — Create the project

We will work on a Mac/Linux terminal.

Open Terminal.

Run:

mkdir ai-devops-rag-mcp
cd ai-devops-rag-mcp

What did we do?

mkdir means:

make directory

We created:

ai-devops-rag-mcp/

Then:

cd ai-devops-rag-mcp

means:

change directory

Now we're inside the project.

Run:

pwd

Expected result on a Mac might look like:

/Users/yourname/ai-devops-rag-mcp

PART 2 — Create a Python virtual environment

Run:

python3 -m venv .venv

Then:

source .venv/bin/activate

Your terminal should change to something similar to:

(.venv) user@macbook ai-devops-rag-mcp %

Why do we need `.venv`?

Think like a DevOps engineer.

Application A might need:

openai version X
mcp version Y

Application B might require different versions.

Instead of installing everything globally on the laptop:

Mac
├── application A dependencies
├── application B dependencies
├── application C dependencies

we isolate them:

Project
 |
 +-- .venv
      |
      +-- openai
      +-- mcp
      +-- python-dotenv
      +-- numpy

This is similar conceptually to dependency isolation you already understand from containers.

PART 3 — Open project in VS Code

Run:

code .

If code isn't configured, open VS Code manually:

VS Code
→ File
→ Open Folder
→ ai-devops-rag-mcp

PART 4 — Create project structure

Create:

ai-devops-rag-mcp/
│
├── .env
├── .gitignore
├── requirements.txt
│
├── knowledge/
│   ├── payment-runbook.txt
│   ├── kubernetes-runbook.txt
│   └── company-architecture.txt
│
├── 01_llm.py
├── 02_embeddings.py
├── 03_rag.py
├── mcp_server.py
└── 05_final_assistant.py

Why these names?

Because I want students to see the evolution:

01_llm.py
     ↓
basic intelligence

02_embeddings.py
     ↓
meaning as numbers

03_rag.py
     ↓
private knowledge

mcp_server.py
     ↓
external capability

05_final_assistant.py
     ↓
combine everything

PART 5 — Install dependencies

Open:

requirements.txt

Paste:

openai
python-dotenv
numpy
mcp

Save.

Now terminal:

pip install -r requirements.txt

The official MCP Python SDK can be installed with the mcp package and supports building MCP servers and clients. (GitHub)

PART 6 — API key

Create:

.env

Paste:

OPENAI_API_KEY=YOUR_API_KEY_HERE

Do not commit this.

Create:

.gitignore

Paste:

.env
.venv/
__pycache__/

This is a DevOps/security lesson.

Never do this:

api_key = "sk-real-secret-key"

inside committed source code.

Instead:

source code
       +
environment/config
       =
running application

Later production architecture might use:

AWS Secrets Manager
Kubernetes Secrets
Vault
GitHub Actions Secrets
OIDC/workload identity

Also remember: an API key alone isn't enough if the API account/project has no available billing quota. An authentication problem and a quota problem are different failures.

PART 7 — Our first LLM call

Create:

01_llm.py

Paste:

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()

question = "Explain Kubernetes CrashLoopBackOff in simple DevOps language."

response = client.responses.create(
    model="gpt-5.6",
    input=question
)

print(response.output_text)

OpenAI's current documentation shows the Responses API as the primary pattern for model interaction, including current examples using client.responses.create(...). (OpenAI Platform)

Now run:

python 01_llm.py

You should receive an explanation of CrashLoopBackOff.

The exact wording can differ from run to run.

STOP HERE AND EXPLAIN EVERY LINE

Line 1

from openai import OpenAI

Meaning:

Python, please import the OpenAI client class from the OpenAI package.

We installed that package earlier with:

pip install openai

Without the import Python doesn't know what OpenAI means.

Line 2

from dotenv import load_dotenv

This lets Python load values from:

.env

Line 4

load_dotenv()

This reads:

OPENAI_API_KEY=...

and makes it available as an environment variable.

Line 6

client = OpenAI()

This is important.

We create a client object.

Think:

OUR PYTHON PROGRAM
       |
       | OpenAI client
       |
       v
   OpenAI API

The variable name could technically be:

banana = OpenAI()

and Python wouldn't care.

But we call it:

client

because it represents the API client.

Names are selected by programmers for readability.

Line 8

question = "Explain Kubernetes CrashLoopBackOff in simple DevOps language."

We create a variable named:

question

The value is a string.

A string is text.

Line 10

response = client.responses.create(

Break this into pieces.

client

Our OpenAI connection object.

.responses

We are using the Responses API.

.create()

Create a new model response.

So conceptually:

send request
     ↓
model processes request
     ↓
receive response

This line

model="gpt-5.6",

answers:

Which model do we want to use?

Current OpenAI MCP/Responses documentation uses gpt-5.6 in its examples. (OpenAI Platform)

This line

input=question

means:

send the value stored inside variable question

So Python replaces:

input=question

conceptually with:

input="Explain Kubernetes CrashLoopBackOff..."

Finally

print(response.output_text)

response contains the API result.

output_text gives us the generated text.

print() displays it in Terminal.

What students have proven

This is now a functioning LLM application:

Python
  |
  | request
  v
OpenAI API
  |
  v
LLM
  |
  | response
  v
Python
  |
  v
Terminal

Ask your students:

Where is the intelligence?

Not here:

print()

Not here:

question =

The trained model provides the learned language capability.

Our code orchestrates access to it.

That separation is extremely important for DevOps engineers.

PART 8 — Demonstrate the LLM's limitation

Now change:

question =

to:

question = """
What is our company's exact procedure for payment-service
when database connection pool utilization exceeds 90%?
"""

Run:

python 01_llm.py

The LLM may provide a reasonable general answer.

But ask:

How could the model know our internal company procedure?

It doesn't automatically have our private runbook.

Here is the problem:

                   LLM
                    |
          General learned knowledge
                    |
                    X
                    |
         private company runbook

This is where RAG enters.

PART 9 — Create private company knowledge

Create:

knowledge/payment-runbook.txt

Paste:

PAYMENT SERVICE PRODUCTION RUNBOOK

Service: payment-service
Team: Payments Platform

Incident: HTTP 503 errors

Common causes:

1. Database connection pool exhaustion.
2. payment-service pods not ready.
3. Upstream authentication-service unavailable.
4. Deployment configuration errors.

Company-specific procedure:

If database connection pool utilization exceeds 90 percent:

1. Check payment-service pod logs.
2. Verify PostgreSQL connectivity.
3. Check active database connections.
4. Scale payment-service from 3 replicas to 6 replicas.
5. Do not restart the PostgreSQL database without approval.
6. Notify the Payments Platform team.
7. Open incident severity SEV-2 if errors continue for more than 10 minutes.

Rollback command:

kubectl rollout undo deployment/payment-service -n banking

Healthy replica count:

3 minimum.

Production namespace:

banking

Now:

knowledge/kubernetes-runbook.txt

Paste:

KUBERNETES TROUBLESHOOTING RUNBOOK

CrashLoopBackOff procedure:

1. Run kubectl get pods -n banking.
2. Run kubectl describe pod POD_NAME -n banking.
3. Run kubectl logs POD_NAME -n banking.
4. Inspect environment variables.
5. Inspect Kubernetes Secrets and ConfigMaps.
6. Check readiness and liveness probes.
7. Review the latest deployment.
8. Roll back only if the newest deployment caused the incident.

ImagePullBackOff procedure:

1. Confirm image name.
2. Confirm image tag.
3. Confirm Amazon ECR image exists.
4. Validate imagePullSecrets when applicable.
5. Validate IAM permissions.

And:

knowledge/company-architecture.txt

Paste:

BANKING PLATFORM ARCHITECTURE

Environment: AWS

Container platform:
Amazon EKS

Production namespace:
banking

Services:
authentication-service
customer-service
account-service
payment-service
notification-service

CI/CD:
GitHub Actions

GitOps:
Argo CD

Container registry:
Amazon ECR

Infrastructure as Code:
Terraform

Monitoring:
Prometheus
Grafana
CloudWatch

Database:
Amazon RDS PostgreSQL

Now the important question:

How do we give these documents to the LLM?

We could put all documents into every prompt.

But imagine:

10 documents
100 documents
10,000 documents
1,000,000 documents

We need retrieval.

PART 10 — What does RAG mean?

RAG:

Retrieval-Augmented Generation

Break the name down.

Retrieval

Find relevant information.

Augmented

Add that information to the model's context.

Generation

The LLM generates an answer.

Therefore:

QUESTION
   |
   v
RETRIEVE
relevant information
   |
   v
AUGMENT
the prompt
   |
   v
GENERATE
answer with LLM

That's RAG.

Very important

RAG does not mean:

train the LLM again

We are not changing the model's weights.

We are doing:

pretrained LLM
      +
retrieved context
      =
better grounded response

PART 11 — But how can a computer search by meaning?

This introduces:

Embeddings

Suppose our user asks:

"What should I do when payment DB connections are full?"

Our document says:

"If database connection pool utilization exceeds 90 percent..."

Exact words differ.

A simple keyword search may struggle.

But semantically:

DB connections full

and

database connection pool exceeds 90%

are related.

Embeddings represent text as vectors—lists of numbers—which can be compared to estimate semantic relatedness. OpenAI's current documentation shows text-embedding-3-small returning a numeric embedding vector; by default that model produces a 1,536-dimensional vector. (OpenAI Platform)

PART 12 — See an embedding yourself

Create:

02_embeddings.py

Paste:

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()

text = "Kubernetes pod cannot connect to PostgreSQL"

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=text
)

vector = response.data[0].embedding

print("Original text:")
print(text)

print("\nFirst 10 numbers:")
print(vector[:10])

print("\nNumber of dimensions:")
print(len(vector))

Run:

python 02_embeddings.py

Expected shape:

Original text:
Kubernetes pod cannot connect to PostgreSQL

First 10 numbers:
[-0.00..., 0.01..., -0.02..., ...]

Number of dimensions:
1536

The exact values will vary.

OpenAI documents the same embedding call pattern and notes that text-embedding-3-small defaults to 1,536 dimensions. (OpenAI Platform)

What happened?

Input:

"Kubernetes pod cannot connect to PostgreSQL"

became something conceptually like:

[
 -0.0132,
  0.0211,
 -0.0043,
 ...
]

Not random numbers.

They represent learned semantic characteristics in a high-dimensional space.

Why numbers?

Computers can compare numbers efficiently.

Consider simplified 3-dimensional vectors:

"database error"

[0.9, 0.2, 0.7]

and:

"PostgreSQL connection failure"

[0.88, 0.19, 0.72]

They're geometrically close.

But:

"best pizza recipe"

[-0.2, 0.8, -0.5]

would be farther away.

Real embeddings have many more dimensions.

PART 13 — Similarity

We need to compare vectors.

For this lab we will use:

Cosine similarity

Conceptually:

vector A ↘
          \ angle
           \
vector B --->

Smaller semantic angle → generally higher similarity.

Scores are commonly interpreted comparatively rather than as magical absolute truth.

For the class:

query vs payment runbook      0.82
query vs Kubernetes runbook   0.57
query vs architecture         0.35

Choose highest.

That's retrieval.

PART 14 — Build RAG manually

Now create:

03_rag.py

Paste:

import os
import numpy as np

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()


def read_documents():
    documents = []

    for filename in os.listdir("knowledge"):
        if filename.endswith(".txt"):
            path = os.path.join("knowledge", filename)

            with open(path, "r") as file:
                text = file.read()

            documents.append({
                "filename": filename,
                "text": text
            })

    return documents


def create_embedding(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )

    return response.data[0].embedding


def cosine_similarity(vector_a, vector_b):
    a = np.array(vector_a)
    b = np.array(vector_b)

    return np.dot(a, b) / (
        np.linalg.norm(a) * np.linalg.norm(b)
    )


documents = read_documents()

print(f"Loaded {len(documents)} documents.")

for document in documents:
    document["embedding"] = create_embedding(document["text"])

question = """
payment-service returns 503 and database connections
are above 90 percent. What should I do?
"""

question_embedding = create_embedding(question)

for document in documents:
    document["score"] = cosine_similarity(
        question_embedding,
        document["embedding"]
    )


documents = sorted(
    documents,
    key=lambda document: document["score"],
    reverse=True
)

best_document = documents[0]

print("\nSimilarity results:")

for document in documents:
    print(
        document["filename"],
        round(document["score"], 4)
    )

print("\nBest document:")
print(best_document["filename"])

context = best_document["text"]

prompt = f"""
You are a Senior DevOps Incident Assistant.

Answer the question using ONLY the company context below.

If the answer is not contained in the context,
say that you do not have enough company information.

COMPANY CONTEXT:
{context}

QUESTION:
{question}
"""

response = client.responses.create(
    model="gpt-5.6",
    input=prompt
)

print("\nAI ANSWER:")
print(response.output_text)

PART 15 — Understand every section of the RAG code

Start here:

import os

os helps Python work with:

files
folders
paths
operating system functionality

Then:

import numpy as np

NumPy gives us numerical operations.

We use it to compare vectors.

np is just a commonly used shorter alias.

This:

def read_documents():

means:

Define a reusable function called read_documents.

A function is a block of reusable logic.

Think of Terraform:

module "vpc" {
}

You don't want to copy the same infrastructure logic everywhere.

Programming functions serve a similar reuse principle.

Inside:

documents = []

creates an empty Python list.

Think:

documents
[
]

Later:

[
 payment-runbook,
 kubernetes-runbook,
 company-architecture
]

This:

for filename in os.listdir("knowledge"):

means:

Go through every file in the knowledge folder.

Like:

ls knowledge

but programmatically.

This:

if filename.endswith(".txt"):

means:

Only process text files.

This:

with open(path, "r") as file:

means:

open this file
in read mode

r = read.

This:

text = file.read()

reads file contents into memory.

Then:

documents.append({
    "filename": filename,
    "text": text
})

We store:

filename
+
document contents

Example:

{
    "filename": "payment-runbook.txt",
    "text": "PAYMENT SERVICE PRODUCTION RUNBOOK..."
}

Now the embedding function

def create_embedding(text):

We created our own reusable function.

Input:

text

Output:

embedding vector

Inside:

response = client.embeddings.create(

asks OpenAI's embedding model to convert text into a vector.

The official OpenAI example uses the same API shape with text-embedding-3-small. (OpenAI Platform)

Then:

return response.data[0].embedding

return sends the result back to whoever called the function.

Example:

vector = create_embedding("hello")

becomes conceptually:

vector = [0.02, -0.01, ...]

Now similarity

def cosine_similarity(vector_a, vector_b):

takes two vectors.

Then:

a = np.array(vector_a)
b = np.array(vector_b)

converts regular Python lists to NumPy arrays.

Then:

np.dot(a, b)

calculates their dot product.

And:

np.linalg.norm(a)

calculates vector magnitude.

This implements the cosine similarity formula.

You do not need to teach linear algebra deeply yet.

Senior DevOps students need to understand:

query embedding
       |
       | compare
       v
document embeddings

highest similarity
       |
       v
most relevant document

PART 16 — Run RAG

Execute:

python 03_rag.py

Expected style of output:

Loaded 3 documents.

Similarity results:
payment-runbook.txt 0.72
company-architecture.txt 0.55
kubernetes-runbook.txt 0.49

Best document:
payment-runbook.txt

Your actual similarity numbers can differ.

Then an answer should include company-specific information such as:

Check payment-service pod logs.
Verify PostgreSQL connectivity.
Check active connections.
Scale payment-service from 3 to 6 replicas.
Do not restart PostgreSQL without approval.
Notify the Payments Platform team.
Open SEV-2 if errors continue over 10 minutes.

Now ask:

Did we retrain GPT?

No.

What happened?

User question
      |
      v
embedding
      |
      v
similarity search
      |
      v
payment-runbook found
      |
      v
runbook inserted into prompt
      |
      v
LLM
      |
      v
grounded answer

That is RAG.

PART 17 — Show students the actual augmented prompt

Before:

response = client.responses.create(

add:

print("\nFINAL PROMPT SENT TO LLM:")
print(prompt)

Run:

python 03_rag.py

Now students can literally see:

You are a Senior DevOps Incident Assistant.

COMPANY CONTEXT:

PAYMENT SERVICE PRODUCTION RUNBOOK
...

QUESTION:

payment-service returns 503...

This is the moment RAG usually clicks.

There is no mysterious magic.

The retrieval system found text and put it into the LLM's context.

PART 18 — Our simplified RAG architecture

What we built:

FILES
 |
 +-- payment-runbook
 +-- kubernetes-runbook
 +-- architecture
          |
          v
      embeddings
          |
          v
 vectors in memory
          ^
          |
question → embedding
          |
          v
 cosine similarity
          |
          v
 relevant document
          |
          v
      prompt/context
          |
          v
         LLM
          |
          v
        answer

PART 19 — Why would production RAG be more complicated?

Our lab embeds the same files every execution.

That's intentionally inefficient.

We do it because students need to see the mechanics.

Production might look like:

                INGESTION PIPELINE

PDF / Confluence / GitHub / S3
             |
             v
          parsing
             |
             v
          chunking
             |
             v
         embeddings
             |
             v
        vector store

Then separately:

                QUERY PIPELINE

user question
      |
      v
query embedding
      |
      v
vector search
      |
      v
top relevant chunks
      |
      v
prompt
      |
      v
LLM

PART 20 — What is chunking?

Imagine a 300-page PDF.

We don't necessarily create one vector for the entire book.

Instead:

300-page PDF
      |
      v
split
      |
      +-- chunk 1
      +-- chunk 2
      +-- chunk 3
      +-- chunk 4
      ...

For example:

chunk 1:
payment-service architecture

chunk 2:
database failure procedures

chunk 3:
rollback procedure

chunk 4:
monitoring

Then retrieval can return the specific relevant portion.

Why chunk?

If the user asks:

How do I rollback payment-service?

we would rather retrieve:

rollback section

than feed a huge unrelated document.

PART 21 — What is a vector database?

Our vectors currently live only in Python memory.

Python process
     |
     v
vectors

When the process stops, our generated vectors disappear.

Production can use a vector-capable retrieval system.

Conceptually:

document
   |
embedding
   |
   v
+----------------+
| vector store   |
|----------------|
| vector         |
| text           |
| metadata       |
+----------------+

Possible metadata:

service = payment-service
environment = production
document = runbook
team = payments
version = 4

Then retrieve:

Top 5 most semantically related chunks

OpenAI's embedding documentation explicitly describes saving embedding vectors in a vector database for retrieval/search use cases. (OpenAI Platform)

PART 22 — We still have a problem

RAG can tell us:

company procedure
architecture
documentation
previous knowledge

But suppose the user asks:

How many payment-service replicas are running RIGHT NOW?

Our runbook says:

Healthy replica count: 3 minimum

But that doesn't tell us:

current live replicas

We need external system access.

For example:

kubectl get deployment payment-service -n banking

Or:

kubectl get pods

Or CloudWatch.

Or AWS.

Or GitHub.

This is where MCP becomes useful.

PART 23 — What is MCP?

MCP stands for:

Model Context Protocol

The official MCP project describes it as an open standard for connecting AI applications to external systems—analogous to a standardized connector for AI applications. MCP servers can expose capabilities such as tools and resources to clients. (Model Context Protocol)

Think about USB-C.

Without a standard connector:

Device A → custom cable
Device B → different cable
Device C → different cable
Device D → different cable

With USB-C:

Device A ─┐
Device B ─┼── USB-C
Device C ─┤
Device D ─┘

MCP tries to provide a standardized interface between AI applications and external systems.

Traditional integration problem

Without MCP you might write custom integrations:

AI application
 |
 +-- custom Kubernetes integration
 |
 +-- custom GitHub integration
 |
 +-- custom Jira integration
 |
 +-- custom AWS integration
 |
 +-- custom database integration

MCP provides a common protocol model:

              AI HOST
                 |
             MCP CLIENT
                 |
      +----------+---------+
      |          |         |
      v          v         v
   MCP K8s    MCP GitHub  MCP AWS
   server      server      server

PART 24 — Important MCP concepts

There are several pieces students need to understand.

MCP HOST
   |
MCP CLIENT
   |
MCP SERVER
   |
TOOLS / RESOURCES
   |
external systems

Host

The AI application/environment.

Client

Handles the MCP connection.

Server

Exposes capabilities.

Tool

Something the model/application can invoke.

Official MCP documentation describes tools as server-exposed capabilities that can interact with external systems such as APIs, databases, or computations. (Model Context Protocol)

Examples:

get_pods()
restart_service()
get_logs()
create_ticket()
query_database()

Resource

Data the MCP server exposes as contextual information, such as files, schemas, or application-specific information. (Model Context Protocol)

Examples:

runbook://payment
config://production
schema://database

PART 25 — Build our first MCP server

For safety, we will simulate Kubernetes first.

That is intentional.

Do not give beginner agents permission to delete production resources.

Create:

mcp_server.py

Paste:

from mcp.server import MCPServer

mcp = MCPServer("Banking DevOps MCP")


@mcp.tool()
def get_service_status(service_name: str) -> str:
    """
    Return the current simulated Kubernetes status
    for a banking service.
    """

    services = {
        "payment-service": {
            "desired_replicas": 3,
            "ready_replicas": 2,
            "status": "DEGRADED"
        },
        "authentication-service": {
            "desired_replicas": 3,
            "ready_replicas": 3,
            "status": "HEALTHY"
        }
    }

    service = services.get(service_name)

    if service is None:
        return f"Service {service_name} was not found."

    return (
        f"Service: {service_name}\n"
        f"Desired replicas: {service['desired_replicas']}\n"
        f"Ready replicas: {service['ready_replicas']}\n"
        f"Status: {service['status']}"
    )


@mcp.tool()
def get_recent_logs(service_name: str) -> str:
    """
    Return simulated recent production logs.
    """

    if service_name == "payment-service":
        return """
2026-08-10T13:02:14Z ERROR database connection pool exhausted
2026-08-10T13:02:15Z ERROR timeout acquiring PostgreSQL connection
2026-08-10T13:02:16Z WARN readiness probe failed
"""

    return f"No critical logs found for {service_name}."


@mcp.resource("runbook://payment-service")
def payment_resource() -> str:
    """
    Return basic payment-service operational metadata.
    """

    return """
Service: payment-service
Namespace: banking
Team: Payments Platform
Environment: production
"""


if __name__ == "__main__":
    mcp.run()

The current official Python MCP SDK's v2 examples use MCPServer, and the SDK supports both server tools and resources. (GitHub)

PART 26 — Understand MCP code

This:

from mcp.server import MCPServer

imports the MCP server implementation.

This:

mcp = MCPServer("Banking DevOps MCP")

creates our MCP server.

We named it:

Banking DevOps MCP

The key line

@mcp.tool()

This is a Python decorator.

For beginners, don't go deep into decorator internals.

Explain:

This tells the MCP server: expose the function below as an MCP tool.

Without:

@mcp.tool()

it's just a normal Python function.

With it:

Python function
       +
MCP registration
       =
MCP tool

Then:

def get_service_status(service_name: str) -> str:

Let's break it apart.

def

Define function.

get_service_status

Function name.

service_name

Input parameter.

: str

Input should be text.

-> str

Function returns text.

Why descriptive function names matter enormously in AI tools

Compare:

def x(a):

with:

def get_service_status(service_name: str):

The second describes what it does.

Tool names and descriptions matter because the AI system needs to understand which capability is appropriate.

This dictionary

services = {

is simply simulated Kubernetes data.

We're pretending:

payment-service

desired = 3
ready = 2
status = DEGRADED

Later we can replace this with:

kubectl

or the Kubernetes Python API.

The MCP interface can stay conceptually similar.

This is powerful:

today

MCP tool
    |
simulated dictionary


tomorrow

MCP tool
    |
Kubernetes API

The consumer still thinks:

get_service_status

PART 27 — Tool versus resource

We created:

@mcp.tool()

and:

@mcp.resource(...)

What's the difference?

Simplified teaching version:

RESOURCE
≈ information/context

TOOL
≈ callable capability

Example resource:

runbook://payment-service

Example tool:

get_service_status("payment-service")

MCP's specification treats resources as server-exposed contextual data and tools as model-invocable capabilities. (Model Context Protocol)

PART 28 — Why MCP isn't the LLM

Very important.

Our MCP server contains:

get_service_status()
get_recent_logs()

But it does not itself understand natural language like:

Why is checkout broken?

It exposes capabilities.

The LLM provides language interpretation/reasoning.

Think:

LLM
"I think I need logs."

       |
       v

MCP

"Here is get_recent_logs."

       |
       v

external system

       |
       v

log result

       |
       v

LLM

"I can now explain the incident."

PART 29 — RAG vs MCP

Students MUST be able to answer this.

RAG

Question:

What does our runbook say?

Use RAG.

documents
   |
retrieval
   |
context
   |
LLM

MCP

Question:

What is happening right now?

MCP might access:

Kubernetes
AWS
GitHub
Datadog
database
Jira

through exposed tools/resources.

Memorize this

RAG = KNOW

MCP = CONNECT / ACCESS / ACT

LLM = UNDERSTAND + GENERATE

It's simplified, but extremely useful.

PART 30 — Build final combined assistant

For a classroom lab, I want students to see the mechanics explicitly rather than hiding everything behind an agent framework.

Create:

05_final_assistant.py

Paste:

import os
import numpy as np

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI()


# --------------------------------------------------
# SECTION 1: RAG KNOWLEDGE
# --------------------------------------------------

def read_documents():
    documents = []

    for filename in os.listdir("knowledge"):
        if filename.endswith(".txt"):

            path = os.path.join("knowledge", filename)

            with open(path, "r") as file:
                text = file.read()

            documents.append({
                "filename": filename,
                "text": text
            })

    return documents


def create_embedding(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )

    return response.data[0].embedding


def cosine_similarity(vector_a, vector_b):
    a = np.array(vector_a)
    b = np.array(vector_b)

    return np.dot(a, b) / (
        np.linalg.norm(a) * np.linalg.norm(b)
    )


def retrieve_company_knowledge(question):
    documents = read_documents()

    question_embedding = create_embedding(question)

    for document in documents:

        document_embedding = create_embedding(
            document["text"]
        )

        document["score"] = cosine_similarity(
            question_embedding,
            document_embedding
        )

    documents = sorted(
        documents,
        key=lambda doc: doc["score"],
        reverse=True
    )

    return documents[0]


# --------------------------------------------------
# SECTION 2: SIMULATED LIVE OPERATIONS
# These are the same capabilities our MCP server exposes.
# --------------------------------------------------

def get_service_status(service_name):

    services = {
        "payment-service": {
            "desired_replicas": 3,
            "ready_replicas": 2,
            "status": "DEGRADED"
        }
    }

    service = services.get(service_name)

    if service is None:
        return "Service not found."

    return f"""
Service: {service_name}
Desired replicas: {service['desired_replicas']}
Ready replicas: {service['ready_replicas']}
Status: {service['status']}
"""


def get_recent_logs(service_name):

    if service_name == "payment-service":
        return """
2026-08-10T13:02:14Z ERROR database connection pool exhausted
2026-08-10T13:02:15Z ERROR timeout acquiring PostgreSQL connection
2026-08-10T13:02:16Z WARN readiness probe failed
"""

    return "No critical logs found."


# --------------------------------------------------
# SECTION 3: USER QUESTION
# --------------------------------------------------

question = """
payment-service is returning HTTP 503 errors.

Explain:
1. What is happening right now?
2. What is the likely root cause?
3. What does our company runbook tell me to do?
"""


# --------------------------------------------------
# SECTION 4: RAG RETRIEVAL
# --------------------------------------------------

retrieved_document = retrieve_company_knowledge(
    question
)

print("\n--- RAG RETRIEVAL ---")
print(
    "Retrieved:",
    retrieved_document["filename"]
)

print(
    "Similarity:",
    round(retrieved_document["score"], 4)
)


# --------------------------------------------------
# SECTION 5: OPERATIONAL TOOL DATA
# --------------------------------------------------

service_status = get_service_status(
    "payment-service"
)

logs = get_recent_logs(
    "payment-service"
)

print("\n--- LIVE TOOL DATA ---")
print(service_status)

print("\n--- LOGS ---")
print(logs)


# --------------------------------------------------
# SECTION 6: AUGMENT THE LLM CONTEXT
# --------------------------------------------------

prompt = f"""
You are a Senior DevOps Incident Assistant.

Your task is to analyze a production incident.

Use the following sources.

SOURCE 1 - COMPANY KNOWLEDGE RETRIEVED BY RAG:

{retrieved_document["text"]}


SOURCE 2 - CURRENT SERVICE STATUS:

{service_status}


SOURCE 3 - CURRENT APPLICATION LOGS:

{logs}


USER QUESTION:

{question}


Respond using this format:

CURRENT STATUS:
Explain what is happening.

LIKELY ROOT CAUSE:
Explain the most likely cause using evidence.

RUNBOOK ACTIONS:
List the company-approved actions.

EVIDENCE:
Explain which supplied information supports your conclusion.

SAFETY:
Do not recommend destructive production actions
that are not explicitly supported by the runbook.
"""


# --------------------------------------------------
# SECTION 7: LLM INFERENCE
# --------------------------------------------------

response = client.responses.create(
    model="gpt-5.6",
    input=prompt
)


# --------------------------------------------------
# SECTION 8: FINAL RESPONSE
# --------------------------------------------------

print("\n--- AI INCIDENT ANALYSIS ---")

print(response.output_text)

PART 31 — Run final application

Execute:

python 05_final_assistant.py

You should see roughly:

--- RAG RETRIEVAL ---

Retrieved: payment-runbook.txt
Similarity: ...

Then:

--- LIVE TOOL DATA ---

Service: payment-service
Desired replicas: 3
Ready replicas: 2
Status: DEGRADED

Then:

--- LOGS ---

ERROR database connection pool exhausted
ERROR timeout acquiring PostgreSQL connection
WARN readiness probe failed

Then the LLM should conclude something similar to:

CURRENT STATUS:

payment-service is degraded.
Only 2 of 3 replicas are ready.

LIKELY ROOT CAUSE:

The evidence indicates database connection pool
exhaustion and PostgreSQL connection acquisition
timeouts.

RUNBOOK ACTIONS:

1. Check pod logs.
2. Verify PostgreSQL connectivity.
3. Check active database connections.
4. Scale payment-service from 3 to 6 replicas.
5. Do not restart PostgreSQL without approval.
6. Notify the Payments Platform team.
7. Open SEV-2 if errors continue over 10 minutes.

Now the student can see the entire AI architecture.

PART 32 — Exactly what happened?

Walk through it slowly.

User asks:

payment-service is returning 503

Step 1:

question

gets converted into:

embedding vector

Step 2:

We compare it against:

runbook embedding
Kubernetes document embedding
architecture embedding

Step 3:

Highest semantic similarity:

payment-runbook

Step 4:

RAG retrieves:

company procedure

Step 5:

Operational tools provide:

desired replicas = 3
ready replicas = 2
status = degraded

Step 6:

Logs provide:

database connection pool exhausted

Step 7:

We combine:

QUESTION

+

RAG CONTEXT

+

OPERATIONAL CONTEXT

+

INSTRUCTIONS

Step 8:

Send all of that to the LLM.

Step 9:

LLM generates an incident explanation.

PART 33 — Final architecture students should draw

At the end of class, erase everything and ask students to draw this from memory:

                         USER
                          |
                          v
                   AI APPLICATION
                          |
              +-----------+-----------+
              |                       |
              v                       v
             RAG                     MCP
              |                       |
              |                       |
        KNOWLEDGE ACCESS        SYSTEM ACCESS
              |                       |
              v                       v
        embeddings/vector          MCP server
            retrieval             /    |     \
              |                  /     |      \
              v                 v      v       v
          runbooks            K8s    AWS    GitHub
              \                  \     |      /
               \                  \    |     /
                +------------------+---+
                           |
                           v
                         PROMPT
                           |
                           v
                          LLM
                           |
                           v
                       RESPONSE

PART 34 — But our final Python file isn't actually invoking MCP yet. Why?

This distinction is pedagogically important.

In mcp_server.py, we built the actual MCP interface.

In 05_final_assistant.py, we call equivalent Python functions directly so students can see:

data enters here
↓
prompt changes here
↓
LLM receives this

If we immediately bury everything behind an MCP client/agent orchestration layer, beginners often learn:

copy framework code

rather than:

understand AI architecture

Once they understand the flow, the production form becomes:

AI application
      |
      v
MCP client
      |
      v
MCP server
      |
      v
get_service_status

OpenAI's Responses API can connect to remote MCP servers using its built-in mcp tool type, while MCP itself standardizes how those servers expose tools/context. (OpenAI Platform)

PART 35 — What real OpenAI + remote MCP looks like

Once an MCP server is available over a supported remote transport and reachable by the API, the architecture can become:

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-5.6",

    tools=[
        {
            "type": "mcp",
            "server_label": "banking_devops",
            "server_description": "Banking production operations tools",
            "server_url": "https://YOUR-MCP-SERVER/mcp",
            "require_approval": "always"
        }
    ],

    input="""
    Check payment-service and explain
    why it is unhealthy.
    """
)

print(response.output_text)

This follows the current Responses API pattern for remote MCP servers: an MCP tool configuration includes fields such as a server label, server URL, and approval behavior. (OpenAI Platform)

Notice this:

"require_approval": "always"

For production DevOps, this idea is extremely important.

You don't want:

User:
"Maybe restart everything."

AI:
"Sure."

kubectl delete ...

without governance.

PART 36 — The senior DevOps security lesson

Imagine an MCP tool:

@mcp.tool()
def delete_namespace(namespace):

You have effectively given an AI-accessible system the capability to remove Kubernetes workloads.

The problem is no longer only:

Can the LLM answer correctly?

Now it becomes:

Who can invoke the tool?

Which namespaces?

Read or write?

Production or staging?

Is approval required?

Are calls audited?

Where are credentials stored?

Can prompt injection trigger tools?

Can a compromised MCP server exfiltrate data?

OpenAI's current MCP guidance explicitly supports approval controls for MCP tool calls, and the MCP specification also defines authorization for HTTP-based transports. (OpenAI Platform)

For senior DevOps engineers, this is where the topic becomes very serious.

PART 37 — Read-only MCP first

Start production adoption with:

READ

not:

WRITE

For example:

GOOD FIRST TOOLS

get_pods
get_deployments
get_logs
get_events
get_service_status
get_alarms
get_cpu_metrics
get_argocd_status
get_git_commit

Later:

CONTROLLED WRITE TOOLS

scale_deployment
rollback_deployment
create_incident
restart_pod
update_ticket

And dangerous operations should have strong:

authentication
authorization
least privilege
approval
audit logging
environment boundaries

PART 38 — Replace simulated MCP data with Kubernetes

Later our tool:

def get_service_status(service_name):

could execute:

kubectl get deployment

Conceptually:

import subprocess


def get_service_status(service_name):

    command = [
        "kubectl",
        "get",
        "deployment",
        service_name,
        "-n",
        "banking"
    ]

    result = subprocess.run(
        command,
        capture_output=True,
        text=True
    )

    return result.stdout

Then:

LLM
 |
MCP
 |
Python
 |
kubectl
 |
Kubernetes API
 |
EKS

But for a first AI lab, simulated data is much safer and easier to understand.

PART 39 — What is the DevOps engineer responsible for?

This is the part I would emphasize most to senior engineers.

The ML team might build:

model behavior
embedding strategy
evaluation

The application team might build:

AI application
API
UI
business logic

But DevOps/Platform/SRE still owns major parts of:

infrastructure
deployment
security
reliability
observability
scalability
cost
CI/CD
secrets
networking
availability
incident response

Your production stack may become:

                    USER
                     |
                     v
                 Route 53
                     |
                     v
                   ALB
                     |
                     v
             AI APPLICATION
                  on EKS
                     |
          +----------+----------+
          |                     |
          v                     v
      OpenAI API             RAG SERVICE
                                |
                                v
                            vector DB
                                |
                                v
                               S3

          +
          |
          v
       MCP SERVICE
          |
      +---+----+------+
      |        |      |
      v        v      v
     EKS      AWS   GitHub

Then observability:

Prometheus
Grafana
CloudWatch
OpenTelemetry
centralized logs
tracing

PART 40 — New metrics DevOps engineers must care about

Traditional application metrics:

CPU
memory
requests/sec
5xx
latency
pod count
disk

AI introduces additional concerns:

model latency
token usage
API errors
rate limits
retrieval latency
retrieval quality
embedding latency
MCP tool latency
MCP failures
tool-call rate
vector DB latency
cost per request

And one extremely important distinction:

SYSTEM HEALTH

HTTP 200
pod healthy
CPU fine

does NOT necessarily mean

AI QUALITY

correct answer
correct retrieval
correct tool usage

That is one of the biggest conceptual shifts for SRE/DevOps.

PART 41 — A healthy AI application can still be wrong

Imagine:

API: 200 OK

CPU: 20%

Memory: 40%

Pods: 3/3

Latency: 500ms

Everything looks green.

But the chatbot answers:

Production database password is XYZ...

or:

Restart PostgreSQL immediately.

when the runbook explicitly says not to.

Infrastructure is healthy.

AI behavior is bad.

Therefore AI systems need:

traditional monitoring
+
quality evaluation
+
security evaluation

PART 42 — What are evals?

An evaluation is essentially a way to test whether the AI behaves correctly.

Create test questions:

Question:
What do we do when DB pool > 90%?

Expected:
scale payment-service 3 → 6
do not restart PostgreSQL
notify Payments Platform

Another:

Question:
Which namespace hosts payment-service?

Expected:
banking

Another:

Question:
Should PostgreSQL be restarted automatically?

Expected:
No.

Now we can build automated checks.

This starts looking familiar to DevOps engineers:

application tests

+

AI behavior tests

PART 43 — AI CI/CD

Traditional pipeline:

git push
   |
   v
unit test
   |
   v
security scan
   |
   v
Docker build
   |
   v
ECR
   |
   v
deploy

AI pipeline:

git push
   |
   v
unit tests
   |
   v
RAG tests
   |
   v
prompt/evaluation tests
   |
   v
security tests
   |
   v
Docker build
   |
   v
ECR
   |
   v
EKS
   |
   v
smoke test

This is why senior DevOps engineers need to understand LLM architecture even if they're not ML engineers.

PART 44 — Where Terraform fits

Terraform might create:

VPC
private subnets
EKS
IAM
security groups
load balancers
S3
RDS
Secrets Manager
monitoring
DNS

Potential RAG infrastructure:

vector database
object storage
ingestion workers
queues

Potential MCP infrastructure:

MCP service
IAM roles
network policy
authentication
secrets
audit logs

PART 45 — Where Kubernetes fits

You might deploy:

ai-api
rag-api
document-ingestion-worker
mcp-server
frontend

For example:

EKS

namespace: ai-platform

├── ai-api
├── rag-service
├── ingestion-worker
├── mcp-kubernetes
└── frontend

PART 46 — Why not put the LLM inside Kubernetes?

Important distinction.

If you're using OpenAI's API:

EKS application
      |
      | HTTPS
      v
OpenAI API
      |
      v
hosted model

Your application is running on EKS.

That does not mean GPT itself is running in your EKS cluster.

Self-hosted models are a separate architecture.

PART 47 — RAG versus fine-tuning

Students will ask this.

Use this simplified answer.

RAG:

change KNOWLEDGE supplied at runtime

Fine-tuning:

adapt MODEL BEHAVIOR/weights through additional training

For changing company documentation frequently:

RAG

usually makes much more conceptual sense than retraining every time a runbook changes.

Example:

Monday:
scale to 6 replicas

Tuesday:
policy changed → scale to 8

With RAG:

update document
re-index

You don't need to retrain the underlying LLM for each documentation update.

PART 48 — MCP versus API

Another common question:

Isn't MCP just an API?

MCP still operates using normal software/networking concepts, but its value is the standardized protocol/interface for AI applications.

Traditional:

Our AI application
   |
custom code
   |
AWS API

Another integration:

Our AI application
   |
different custom code
   |
GitHub API

MCP creates a more standardized abstraction:

AI
 |
MCP
 |
server
 |
external system

The official MCP SDK itself describes MCP as somewhat like a web API designed specifically for LLM interactions. (GitHub)

PART 49 — LLM versus agent

Another very important distinction.

LLM:

input
  |
model
  |
output

An agentic system adds a loop around the model:

USER
 |
 v
LLM
 |
 +---- Need tool?
 |        |
 |        v
 |      TOOL
 |        |
 |        v
 |      RESULT
 |        |
 +--------+
 |
 v
final answer

The LLM is part of the agent system.

They are not synonyms.

PART 50 — The entire system in one sentence

Have every student say:

An LLM generates and reasons over language; RAG retrieves relevant external knowledge and puts it into the model's context; MCP standardizes access to external tools and resources; and DevOps makes the complete AI system secure, reliable, scalable, observable and deployable.

If they can explain that accurately after the lab, the lab worked.

PART 51 — Final classroom challenge

After demonstrating everything, don't let them simply go home.

Give this assignment.

Change the incident from:

payment-service

to:

authentication-service

Create:

knowledge/authentication-runbook.txt

with:

AUTHENTICATION SERVICE RUNBOOK

If users receive HTTP 401 unexpectedly:

1. Check authentication-service logs.
2. Check JWT signing configuration.
3. Validate Secrets Manager secret.
4. Check token expiration configuration.
5. Verify latest deployment.
6. Roll back if incident started immediately after release.
7. Notify Identity Platform team.

Then add MCP status:

"authentication-service": {
    "desired_replicas": 3,
    "ready_replicas": 3,
    "status": "HEALTHY"
}

And logs:

ERROR JWT signing key mismatch
WARN token validation failed

Ask:

Users suddenly receive 401 after today's deployment.
What is happening and what should I do?

The student must explain which parts came from:

LLM
RAG
MCP/tool data

Not just produce an answer.

PART 52 — Questions I would ask students at the end

Use these orally:

Did we train GPT in this lab?
No. We used a pretrained model for inference.
What does an embedding do?
Converts text into a numeric vector useful for semantic comparison.
Why do we embed both documents and the question?
So they can be compared in the same vector space.
What is retrieval?
Finding the most relevant external information for the question.
What does RAG stand for?
Retrieval-Augmented Generation.
Does RAG modify GPT's model weights?
No.
Where is company knowledge stored?
In our external knowledge source, not magically inside the LLM.
What does MCP stand for?
Model Context Protocol.
Why use MCP?
To standardize how AI applications connect to tools/resources and external systems.
RAG versus MCP?
RAG primarily retrieves knowledge; MCP exposes standardized external capabilities/context.
Can MCP perform actions?
An MCP server can expose tools that call external systems, subject to the permissions and controls you design. (Model Context Protocol)
Why is MCP dangerous for DevOps if designed badly?
Because tools may expose powerful infrastructure operations.
Why should production MCP use least privilege?
The AI should have no more capability than necessary.
What does an LLM do?
Processes context and generates model output; it is not a company database or Kubernetes API.
Why can a perfectly healthy AI pod still produce a bad product experience?
Infrastructure availability and AI answer quality are different dimensions.

PART 53 — The complete mental model

This is the diagram I would put on the final slide or whiteboard:

                         ┌──────────────────┐
                         │      USER        │
                         └────────┬─────────┘
                                  │
                                  │ natural language
                                  ▼
                         ┌──────────────────┐
                         │  AI APPLICATION  │
                         └────────┬─────────┘
                                  │
                    ┌─────────────┴────────────┐
                    │                          │
                    ▼                          ▼
             ┌──────────────┐           ┌──────────────┐
             │     RAG      │           │     MCP      │
             │              │           │              │
             │  KNOWLEDGE   │           │ CAPABILITY   │
             └──────┬───────┘           └──────┬───────┘
                    │                          │
          ┌─────────▼────────┐       ┌────────▼─────────┐
          │ Embedding Model  │       │    MCP Server    │
          └─────────┬────────┘       └────────┬─────────┘
                    │                         │
                    ▼                    ┌────┼──────────┐
          ┌──────────────────┐           │    │          │
          │ Vector Retrieval │           ▼    ▼          ▼
          └────────┬─────────┘          EKS  AWS       GitHub
                   │
                   ▼
          ┌──────────────────┐
          │ Relevant Runbook │
          └────────┬─────────┘
                   │
                   └───────────┐
                               │
                               ▼
                       ┌───────────────┐
                       │    CONTEXT    │
                       │               │
                       │ question      │
                       │ + RAG data    │
                       │ + tool data   │
                       │ + instruction │
                       └───────┬───────┘
                               │
                               ▼
                       ┌───────────────┐
                       │      LLM      │
                       │   INFERENCE   │
                       └───────┬───────┘
                               │
                               ▼
                       ┌───────────────┐
                       │ FINAL ANSWER  │
                       └───────────────┘

The single most important takeaway

Do not teach students:

LLM + LangChain + vector DB + MCP + Kubernetes

as five tools they have to memorize.

Teach the problem each component solves:

"We need language intelligence."
                 ↓
                LLM


"But the LLM doesn't know our private runbook."
                 ↓
                RAG


"But the runbook doesn't know what is happening
in Kubernetes right now."
                 ↓
                MCP


"But now we have an AI system touching production."
                 ↓
        DEVOPS / PLATFORM / SRE

That's the point where a senior DevOps engineer stops seeing LLMs as “a chatbot API” and starts seeing the complete system architecture.

Lab 1 — Your First Jenkins Job

Aisalkyn Aidarova — Sat, 08 Aug 2026 04:53:24 +0000

Goal

By the end of this lab, you will understand:

Jenkins → Job → Build → Console Output → Workspace

You will create a job that Jenkins executes automatically.

Prerequisites

Make sure Jenkins is running. On your Mac, you can check:

brew services list | grep jenkins

If Jenkins is stopped:

brew services start jenkins-lts

Then open:

http://localhost:8080

You should see the Jenkins Dashboard.

Step 1 — Create Your First Job

On the Jenkins Dashboard:

New Item → Enter a name:

my-first-job

Choose:

Freestyle project

Click:

OK

What is a Jenkins Job?

A job is a task that Jenkins knows how to execute.

For example, later your jobs can:

Pull code from GitHub
        ↓
Run tests
        ↓
Run security scan
        ↓
Build Docker image
        ↓
Push image to ECR
        ↓
Deploy to Kubernetes

But today we'll start with a simple command.

Step 2 — Add Your First Build Step

Scroll down to:

Build → Add build step → Execute shell

Paste:

echo "Hello from Jenkins!"
echo "This is my first Jenkins job."
echo "Jenkins is running commands automatically."

Click:

Save

You should now be on the job page.

Step 3 — Run the Job

Click:

Build Now

On the left side you should see:

Build History

#1

Click:

#1

Then click:

Console Output

Expected output:

Started by user
Building in workspace ...

Hello from Jenkins!
This is my first Jenkins job.
Jenkins is running commands automatically.

Finished: SUCCESS

If you see:

Finished: SUCCESS

your first Jenkins job worked.

Step 4 — Understand What Just Happened

You manually clicked:

Build Now

Jenkins then:

1. Created a build
        ↓
2. Created/used a workspace
        ↓
3. Started a shell
        ↓
4. Executed your commands
        ↓
5. Collected the output
        ↓
6. Determined SUCCESS/FAILURE

This is the foundation of CI/CD.

Step 5 — Make the Job More Useful

Go to:

my-first-job → Configure

Replace the shell commands with:

echo "================================"
echo "      JENKINS DEVOPS LAB"
echo "================================"

echo ""

echo "Current user:"
whoami

echo ""

echo "Current directory:"
pwd

echo ""

echo "Files:"
ls -la

echo ""

echo "Git version:"
git --version

echo ""

echo "Java version:"
java -version

echo ""

echo "Current date:"
date

echo ""

echo "================================"
echo "BUILD COMPLETED"
echo "================================"

Save it.

Click:

Build Now

Then open:

Build History → #2 → Console Output

You should see information about the Jenkins environment.

Step 6 — Understand the Jenkins Workspace

One very important DevOps concept is the workspace.

When Jenkins runs a job, it needs a directory in which to work.

Your command:

pwd

may show something similar to:

/Users/Shared/Jenkins/Home/workspace/my-first-job

or another Jenkins workspace location depending on your installation.

Think of it as:

Jenkins
│
└── workspace
      │
      └── my-first-job
            │
            ├── source code
            ├── build files
            ├── test files
            └── generated artifacts

Later, when Jenkins clones a GitHub repository, the repository will normally appear in this workspace.

Step 7 — Learn SUCCESS vs FAILURE

Now let's intentionally break the build.

Go to:

my-first-job → Configure → Build → Execute shell

Use:

echo "Starting application deployment..."

echo "Checking application..."

exit 1

echo "Deployment completed."

Save and click:

Build Now

Open:

#3 → Console Output

You should get:

Finished: FAILURE

Why?

Because:

exit 1

means the command failed.

In Linux:

exit 0  = SUCCESS
exit 1+ = FAILURE

This concept is extremely important for CI/CD.

For example:

Jenkins
   ↓
Run tests
   ↓
Tests return exit 0
   ↓
Continue deployment

But:

Jenkins
   ↓
Run tests
   ↓
Tests return exit 1
   ↓
STOP
   ↓
Build FAILED

This prevents bad code from reaching production.

Step 8 — Fix the Build

Go back to Configure and replace the script with:

echo "Starting Jenkins build..."

echo "Checking Git..."
git --version

echo "Checking Java..."
java -version

echo "Running application test..."

echo "Test passed!"

echo "Build completed successfully."

exit 0

Run another build.

Expected result:

Finished: SUCCESS

Step 9 — Create a Small Project

Now let's make the lab closer to real DevOps work.

On your Mac Terminal:

mkdir jenkins-first-project
cd jenkins-first-project

Create a file:

touch app.sh

Open it:

nano app.sh

Paste:

#!/bin/bash

echo "================================"
echo "My Application"
echo "================================"

echo "Application started successfully!"

echo "Environment: Development"

echo "Version: 1.0.0"

Save with:

CTRL + O
ENTER
CTRL + X

Give it permission:

chmod +x app.sh

Run:

./app.sh

Expected:

================================
My Application
================================
Application started successfully!
Environment: Development
Version: 1.0.0

Step 10 — Put It on GitHub

Initialize Git:

git init
git add .
git commit -m "Initial Jenkins lab"

Create a GitHub repository called:

jenkins-first-project

Then connect your local project to it:

git branch -M main
git remote add origin YOUR_GITHUB_REPOSITORY_URL
git push -u origin main

Now you have:

Developer
   ↓
GitHub
   ↓
jenkins-first-project

Step 11 — Connect Jenkins to GitHub

Go back to Jenkins.

Create another job:

New Item

Name:

jenkins-github-job

Choose:

Freestyle project

Click OK.

Under:

Source Code Management

select:

Git

Repository URL:

YOUR_GITHUB_REPOSITORY_URL

For a public repository, credentials aren't necessary.

Set branch to:

*/main

Step 12 — Tell Jenkins What to Run

Go to:

Build → Add build step → Execute shell

Paste:

echo "================================"
echo "JENKINS CI BUILD STARTED"
echo "================================"

echo ""
echo "Workspace:"
pwd

echo ""
echo "Source code:"
ls -la

echo ""
echo "Running application..."
chmod +x app.sh
./app.sh

echo ""
echo "================================"
echo "BUILD SUCCESSFUL"
echo "================================"

Save.

Click:

Build Now

Open:

Build History
→ #1
→ Console Output

You should see Jenkins clone your GitHub repository and execute app.sh.

What You Built

Your architecture is now:

┌──────────────┐
│  Developer   │
│     Mac      │
└──────┬───────┘
       │
       │ git push
       ▼
┌──────────────┐
│    GitHub    │
│ Repository   │
└──────┬───────┘
       │
       │ git clone / checkout
       ▼
┌──────────────┐
│   Jenkins    │
│     Job      │
└──────┬───────┘
       │
       │ Execute shell
       ▼
┌──────────────┐
│    app.sh    │
└──────┬───────┘
       │
       ▼
   SUCCESS

What your students should understand after Lab 1

They should be able to explain these concepts:

Concept	Meaning
Jenkins	Automation server
Job	Task Jenkins executes
Build	One execution of a job
Workspace	Directory where Jenkins works
Console Output	Logs from the build
Execute Shell	Run Linux/macOS commands
Exit 0	Success
Exit 1	Failure
Git	Source-code version control
GitHub	Remote repository
CI	Automatically build/test code changes

The most important takeaway is:

Jenkins takes work that a DevOps engineer could perform manually and executes it in a repeatable, automated way.

For your next lab, I recommend moving directly to Pipeline Job + first Jenkinsfile. That is where students start learning Jenkins the way it's commonly used in DevOps: GitHub → Jenkinsfile → stages → build → test → deploy.

install jenkins

Aisalkyn Aidarova — Sat, 08 Aug 2026 02:26:30 +0000

1. Check Homebrew

Open Terminal and run:

brew --version

If you see a version number, continue.

If brew: command not found, install Homebrew first:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Install Java

brew install openjdk@21

Then:

java -version

You should see something similar to:

openjdk version "21..."

If java isn't found, run:

sudo ln -sfn $(brew --prefix)/opt/openjdk@21/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-21.jdk

Then check again:

java -version

3. Install Jenkins

Install the current LTS version:

brew install jenkins-lts

4. Start Jenkins

brew services start jenkins-lts

Check it:

brew services list

You want to see something like:

jenkins-lts    started

5. Open Jenkins

In your browser, open:

Jenkins on localhost

You should get the Unlock Jenkins screen.

6. Get the Jenkins password

Run:

cat ~/.jenkins/secrets/initialAdminPassword

Copy the password and paste it into the browser.

If that file doesn't exist, use:

find ~/.jenkins -name initialAdminPassword 2>/dev/null

7. Finish setup

Choose Install suggested plugins, wait for installation, then create your Jenkins admin user.

After that you'll have:

Mac
 │
 ├── Java 21
 │
 └── Jenkins
      │
      └── localhost:8080

Useful commands:

# Stop Jenkins
brew services stop jenkins-lts

# Start Jenkins
brew services start jenkins-lts

# Restart Jenkins
brew services restart jenkins-lts

If you paste the output of Step 1 (brew --version), I can take you through the installation one command at a time.

Advanced LLM Lab 1

Aisalkyn Aidarova — Fri, 07 Aug 2026 14:29:02 +0000

Build and Operate a DevOps LLM Incident Assistant

What students will understand

At the end of this lab, a student should be able to explain this architecture without memorizing definitions:

DevOps Engineer
      |
      | "Why is my Kubernetes pod restarting?"
      v
+----------------------+
| Python Application   |
+----------------------+
      |
      | builds prompt
      v
+----------------------+
| System Instructions  |
| User Prompt          |
| Context              |
+----------------------+
      |
      v
+----------------------+
| Tokenizer            |
| text -> token IDs    |
+----------------------+
      |
      v
+----------------------+
| LLM API              |
+----------------------+
      |
      v
+----------------------+
| LLM Inference        |
| Transformer          |
| Attention            |
| Neural Network       |
+----------------------+
      |
      | predicts tokens
      v
+----------------------+
| Generated Response   |
+----------------------+
      |
      v
DevOps Engineer

Meanwhile:

Secrets    -> protect API key
Logs       -> record requests/errors
Metrics    -> latency/token usage/errors
Docker     -> package application
CI/CD      -> deploy application
AWS/K8s    -> run application
Monitoring -> observe application + LLM API

This is the mental model I want students to have.

OpenAI also describes text as being processed as tokens, not simply whole words, and provides tiktoken for programmatic tokenization. (OpenAI Platform)

PART 0 — What Are We Building?

"Today we are an AI Platform/DevOps team. A development team wants an AI assistant that helps troubleshoot Kubernetes incidents. Our responsibility is to build the service, understand how the LLM works, protect its credentials, observe its behavior, package it, and eventually deploy it."

The final application will work like this:

$ python app.py

========================================
 DEVOPS LLM INCIDENT ASSISTANT
========================================

Describe your incident:

> Pod payment-service is in CrashLoopBackOff

Analyzing incident...

LLM RESPONSE
----------------------------------------

CrashLoopBackOff means Kubernetes is repeatedly
starting the container and the container is failing.

Start with:

1. kubectl get pods
2. kubectl describe pod payment-service
3. kubectl logs payment-service
4. kubectl logs payment-service --previous

Check:
- application errors
- missing environment variables
- Secrets
- database connectivity
- probes
- memory limits

----------------------------------------

Prompt tokens: 96
Response time: 2.31 seconds

Now the student immediately understands:

We're not building ChatGPT. We're building an application that uses an LLM as one component.

PART 1 — Create the Project

Open Terminal.

Go to your Projects directory:

cd ~/Projects

Create the lab:

mkdir devops-llm-lab
cd devops-llm-lab

Open it in VS Code:

code .

If code doesn't work, open VS Code manually:

File → Open Folder → devops-llm-lab

PART 2 — Create This Exact Structure

Inside VS Code create:

devops-llm-lab/
│
├── app.py
├── tokenizer_demo.py
├── llm_client.py
├── config.py
├── logger.py
│
├── prompts/
│   └── incident_prompt.txt
│
├── logs/
│
├── .env
├── .gitignore
├── requirements.txt
└── Dockerfile

Do not create random files.

Each file has a responsibility.

app.py
    Application entry point

config.py
    Configuration

llm_client.py
    Communication with LLM

tokenizer_demo.py
    Shows how text becomes tokens

logger.py
    Observability

prompts/
    Prompt management

.env
    Secrets/configuration

requirements.txt
    Python dependencies

Dockerfile
    Containerization

DevOps reason

This is already a DevOps lesson.

We do separation of concerns.

You don't want:

500 lines inside app.py

containing:

API keys
prompts
logging
API calls
application logic
configuration

Production applications need maintainable structure.

PART 3 — Create Python Environment

In Terminal, make sure you're inside:

pwd

Expected something similar to:

/Users/yourname/Projects/devops-llm-lab

Create virtual environment:

python3 -m venv .venv

Activate it:

source .venv/bin/activate

Your terminal should change to:

(.venv) user@macbook devops-llm-lab %

Why?

A Python application may need:

openai 2.x
tiktoken
python-dotenv

Another application may require different versions.

The virtual environment isolates dependencies.

DevOps equivalent:

Python venv
       ↓

Docker container
       ↓

Kubernetes Pod

Same fundamental idea:

isolate the runtime environment.

PART 4 — Create requirements.txt

Open:

requirements.txt

Paste:

openai
python-dotenv
tiktoken

Save.

Then run:

pip install -r requirements.txt

Verify:

pip list

You should see packages including:

openai
python-dotenv
tiktoken

Why does DevOps care?

Dependencies are part of the application's software supply chain.

Later in CI/CD we can scan them for vulnerabilities.

For example:

Developer
   ↓
requirements.txt
   ↓
CI pipeline
   ↓
dependency scan
   ↓
Docker build
   ↓
container scan
   ↓
deployment

PART 5 — First Understand Tokens WITHOUT Calling an LLM

This is important.

Do not call OpenAI yet.

We first want to see what an LLM receives.

Open:

tokenizer_demo.py

Paste:

import tiktoken

text = """
The payment-service Kubernetes pod is in CrashLoopBackOff.
"""

encoding = tiktoken.get_encoding("o200k_base")

tokens = encoding.encode(text)

print("ORIGINAL TEXT")
print("----------------")
print(text)

print("TOKEN IDs")
print("----------------")
print(tokens)

print()

print("NUMBER OF TOKENS")
print("----------------")
print(len(tokens))

print()

print("DECODE EACH TOKEN")
print("----------------")

for token in tokens:
    print(
        token,
        "->",
        repr(encoding.decode([token]))
    )

Run:

python tokenizer_demo.py

You will see output conceptually like:

ORIGINAL TEXT
----------------

The payment-service Kubernetes pod is in CrashLoopBackOff.


TOKEN IDs
----------------
[198, 976, 7522, ...]

NUMBER OF TOKENS
----------------
14

DECODE EACH TOKEN
----------------
198 -> '\n'
976 -> 'The'
...

Your exact token IDs/count can vary with encoding and text.

STOP THE LAB HERE AND EXPLAIN

Students usually think:

LLM reads:

"Kubernetes pod is failing."

It doesn't literally operate on that sentence as English words.

The simplified pipeline is:

"Kubernetes pod is failing"

             ↓

          TOKENIZER

             ↓

["Kubernetes", " pod", " is", " failing"]

             ↓

[12345, 9382, 382, 12451]

             ↓

         EMBEDDINGS

             ↓

vectors/numerical representations

             ↓

       TRANSFORMER

             ↓

       probabilities

             ↓

         next token

This is extremely important.

PART 6 — What Does "Generate" Actually Mean?

Ask the students:

Suppose the prompt is:

Kubernetes is a container

The LLM might calculate probabilities somewhat conceptually like:

next token:

orchestration      0.61
platform           0.23
system             0.09
technology         0.04
banana             0.00001

It chooses a token.

Then:

Kubernetes is a container orchestration

Now it predicts again:

platform     0.69
system       0.18
tool         0.08

Then again.

And again.

So:

Prompt
  ↓
Tokens
  ↓
Neural network
  ↓
Probability distribution
  ↓
Next token
  ↓
Probability distribution
  ↓
Next token
  ↓
...

That process is called:

Inference

Why does a DevOps engineer need to know this?

Because inference consumes:

CPU/GPU
memory
network
time
money

Therefore AI infrastructure engineers care about:

latency
throughput
tokens/sec
input tokens
output tokens
GPU utilization
API errors
rate limits
cost

Now monitoring an LLM makes sense.

PART 7 — Understand Embeddings Conceptually

Before the transformer works with tokens, numerical representations are used.

Simplified:

"Kubernetes"

↓

token ID

↓

vector

[
  0.123,
 -0.733,
  0.091,
  ...
]

A vector contains many dimensions.

Conceptually:

Kubernetes
    ↓
[0.21, 0.91, -0.32, ...]

Docker
    ↓
[0.24, 0.87, -0.29, ...]

banana
    ↓
[-0.81, 0.12, 0.76, ...]

"Kubernetes" and "Docker" may be semantically more related than "Kubernetes" and "banana".

This becomes extremely useful later for:

RAG
semantic search
vector databases
document retrieval
incident similarity
knowledge bases

We'll make that a separate advanced lab because it deserves its own experiment.

PART 8 — Transformer and Attention

Now draw this on your board.

Input

"The Kubernetes pod cannot connect to database"

                    ↓

                 TOKENS

                    ↓

               EMBEDDINGS

                    ↓

        +-----------------------+
        |      TRANSFORMER      |
        |                       |
        |  Self Attention       |
        |  Feed Forward         |
        |  Normalization        |
        |  Multiple Layers      |
        +-----------------------+

                    ↓

             Next-token scores

                    ↓

                  TOKEN

                    ↓

                 OUTPUT

The important component is:

Attention

Consider:

The Kubernetes pod cannot connect to the database
because its password secret is incorrect.

When processing:

incorrect

the model can pay different levels of attention to:

Kubernetes
pod
database
password
secret

This helps it understand relationships in context.

You don't need your DevOps students calculating attention matrices in Lab 1.

They need to understand:

Attention allows the model to use relationships among tokens in its context.

PART 9 — Context Window

This part is essential for AI DevOps.

An LLM request conceptually contains:

+--------------------------------------+
|              CONTEXT                 |
|                                      |
| System instructions                  |
| Previous conversation                |
| Retrieved documentation              |
| User question                        |
| Tool results                         |
|                                      |
+--------------------------------------+

That all consumes tokens.

This explains why:

MORE CONTEXT

      ↓

MORE TOKENS

      ↓

MORE PROCESSING

      ↓

MORE LATENCY / COST

And why you cannot simply dump:

2 GB of CloudWatch logs

into every request.

Later we solve this with:

filtering
chunking
embeddings
retrieval
RAG
summarization

Now students understand why RAG exists before they ever install a vector database.

PART 10 — Add the API Key

Create:

.env

Paste:

OPENAI_API_KEY=PASTE_YOUR_KEY_HERE
OPENAI_MODEL=gpt-5

Replace:

PASTE_YOUR_KEY_HERE

with your key.

Do not put quotes around it.

PART 11 — Protect the Secret

Create:

.gitignore

Paste:

.env
.venv/
__pycache__/
logs/
*.pyc

Run:

git init

Then:

git status

You should not see .env as a file that Git intends to commit.

Stop here.

Ask the students:

Why aren't we putting the API key into app.py?

Because:

BAD

app.py
OPENAI_API_KEY="sk-xxxxxxxx"

If pushed:

GitHub
   ↓
secret exposed
   ↓
someone uses API
   ↓
financial/security incident

Local environment:

.env

CI/CD:

GitHub Secrets
Jenkins Credentials

AWS:

AWS Secrets Manager

Kubernetes:

External Secrets Operator
Secrets

This is a DevOps responsibility.

PART 12 — Centralize Configuration

Open:

config.py

Paste:

import os
from dotenv import load_dotenv

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-5")

if not OPENAI_API_KEY:
    raise ValueError(
        "OPENAI_API_KEY is missing. "
        "Add it to the .env file."
    )

Why separate config?

Bad architecture:

api_key = "..."
model = "..."
timeout = "..."
region = "..."

scattered across 20 files.

Better:

Environment

     ↓

config.py

     ↓

Application

This follows twelve-factor application ideas commonly used in DevOps.

PART 13 — Create the System Prompt

Go to:

prompts/incident_prompt.txt

Paste:

You are a Senior DevOps and Site Reliability Engineer.

Your responsibility is to help troubleshoot production infrastructure incidents.

When analyzing an incident:

1. Explain what the error means.
2. List the most likely causes.
3. Provide diagnostic commands.
4. Explain what each command verifies.
5. Recommend the safest remediation.
6. Never recommend destructive production actions without warning.
7. Clearly separate investigation from remediation.

Focus on:

- Kubernetes
- Docker
- AWS
- Linux
- Terraform
- CI/CD
- Networking
- Observability

Why put prompts in files?

Because prompts become application artifacts.

In production you may need:

Prompt v1
Prompt v2
Prompt v3

You want:

Git history
pull requests
testing
review
rollback

Think of prompt management like:

Helm values
Terraform modules
application configuration

We shouldn't randomly change production prompts.

PART 14 — Create Logging

Open:

logger.py

Paste:

import logging
import os

os.makedirs("logs", exist_ok=True)

logging.basicConfig(
    level=logging.INFO,
    format=(
        "%(asctime)s | "
        "%(levelname)s | "
        "%(message)s"
    ),
    handlers=[
        logging.FileHandler("logs/llm-app.log"),
        logging.StreamHandler()
    ]
)

logger = logging.getLogger("devops-llm")

Why?

Because when something fails in production:

"It doesn't work"

is useless.

We need:

timestamp
request
model
latency
status
error
token usage

Same reason we log:

Kubernetes applications
ECS tasks
Lambda functions
Jenkins pipelines

AI applications need observability too.

PART 15 — Create the LLM Client

Open:

llm_client.py

Paste:

import time
from openai import OpenAI

from config import OPENAI_API_KEY, OPENAI_MODEL
from logger import logger


client = OpenAI(
    api_key=OPENAI_API_KEY
)


def load_system_prompt():
    with open(
        "prompts/incident_prompt.txt",
        "r"
    ) as file:
        return file.read()


def analyze_incident(user_input):

    system_prompt = load_system_prompt()

    logger.info(
        "Sending request to LLM model=%s",
        OPENAI_MODEL
    )

    start_time = time.time()

    try:

        response = client.responses.create(
            model=OPENAI_MODEL,
            instructions=system_prompt,
            input=user_input
        )

        latency = time.time() - start_time

        logger.info(
            "LLM request successful latency=%.2fs",
            latency
        )

        return {
            "answer": response.output_text,
            "latency": latency
        }

    except Exception as error:

        logger.exception(
            "LLM request failed: %s",
            error
        )

        raise

STOP AGAIN

Show the architecture.

The application now has:

                 config.py
                    |
                    | API key/model
                    v

User → app.py → llm_client.py → OpenAI API
                    ^
                    |
          incident_prompt.txt

                    |
                    v

                 logger.py
                    |
                    v

            logs/llm-app.log

This is much closer to a real application.

PART 16 — Create the Main Application

Open:

app.py

Paste:

import tiktoken

from llm_client import analyze_incident


def count_tokens(text):

    encoding = tiktoken.get_encoding(
        "o200k_base"
    )

    return len(
        encoding.encode(text)
    )


print()
print("=" * 55)
print("DEVOPS LLM INCIDENT ASSISTANT")
print("=" * 55)
print()

incident = input(
    "Describe your incident:\n\n> "
)

print()
print("Analyzing incident...")
print()

input_tokens = count_tokens(incident)

result = analyze_incident(incident)

print("=" * 55)
print("LLM RESPONSE")
print("=" * 55)

print(result["answer"])

print()
print("=" * 55)
print("OBSERVABILITY")
print("=" * 55)

print(
    f"Input tokens: {input_tokens}"
)

print(
    f"Response latency: "
    f"{result['latency']:.2f} seconds"
)

Save everything.

PART 17 — Run the Real Application

Terminal:

python app.py

You should see:

=======================================================
DEVOPS LLM INCIDENT ASSISTANT
=======================================================

Describe your incident:

>

Paste:

The payment-service pod is in CrashLoopBackOff after a deployment.

Press Enter.

Expected structure:

Analyzing incident...

=======================================================
LLM RESPONSE
=======================================================

CrashLoopBackOff indicates that the container
starts, exits, and Kubernetes repeatedly attempts
to restart it.

Investigation:

1. Check pod state

kubectl get pods

2. Inspect Kubernetes events

kubectl describe pod payment-service

3. Check application logs

kubectl logs payment-service

4. Check previous container logs

kubectl logs payment-service --previous

...

=======================================================
OBSERVABILITY
=======================================================
Input tokens: 15
Response latency: 2.63 seconds

The exact answer, token count, and latency will vary.

That variation itself is part of teaching LLM behavior.

PART 18 — Examine the Logs

Run:

cat logs/llm-app.log

You should see something like:

2026-08-07 09:30:21 | INFO | Sending request to LLM model=gpt-5
2026-08-07 09:30:23 | INFO | LLM request successful latency=2.31s

Now connect it to production.

Locally:

Python logging
      ↓
llm-app.log

ECS:

Container
   ↓
stdout
   ↓
CloudWatch Logs

EKS:

Pod
 ↓
stdout
 ↓
Fluent Bit
 ↓
CloudWatch / Loki

PART 19 — Deliberately Break the Application

This is the DevOps part.

Working applications teach less than broken applications.

Change .env:

OPENAI_API_KEY=wrong-key

Run:

python app.py

Ask:

Pod is CrashLoopBackOff

You should receive an authentication/API error.

Then:

cat logs/llm-app.log

Notice:

ERROR
LLM request failed

Students learn:

User says:
"AI isn't working"

DevOps investigation:

Application running?
        ↓
Configuration correct?
        ↓
Secret correct?
        ↓
DNS working?
        ↓
Network working?
        ↓
API reachable?
        ↓
Authentication valid?
        ↓
Rate limited?
        ↓
Provider available?
        ↓
Model available?

This is AI operations.

PART 20 — Understand the COMPLETE REQUEST

Now students can understand what happened when they typed:

payment-service is CrashLoopBackOff

Layer 1 — User Input

"payment-service is CrashLoopBackOff"

Layer 2 — Application

Python receives:

incident

Layer 3 — Prompt

Application adds instructions:

You are a Senior DevOps Engineer...

plus:

payment-service is CrashLoopBackOff

Layer 4 — Tokenization

text
 ↓
tokens
 ↓
token IDs

Layer 5 — API request

Python
   ↓

HTTPS

   ↓

OpenAI API

Layer 6 — Inference infrastructure

Conceptually:

tokens

  ↓

embeddings

  ↓

transformer layers

  ↓

attention

  ↓

neural network computations

  ↓

probability distribution

  ↓

next token

repeated many times.

Layer 7 — Response

tokens
 ↓
text
 ↓
HTTP response

Layer 8 — Application

response.output_text

Layer 9 — User

Terminal prints answer.

PART 21 — Containerize It

Now we're doing actual DevOps.

Create:

Dockerfile

Paste:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install \
    --no-cache-dir \
    -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Build:

docker build -t devops-llm:v1 .

Verify:

docker images

Expected:

REPOSITORY     TAG
devops-llm     v1

PART 22 — NEVER COPY `.env` INTO THE IMAGE

We need one more file.

Create:

.dockerignore

Paste:

.env
.venv
.git
logs
__pycache__

Why?

Without this you risk:

.env
 ↓
docker build
 ↓
Docker image layer
 ↓
ECR
 ↓
credential exposure

This is extremely important.

PART 23 — Run the Container

Run:

docker run \
  --rm \
  -it \
  --env-file .env \
  devops-llm:v1

Notice what we did.

We did not store credentials in the container.

We injected configuration at runtime:

Docker Image
     +
Environment Secret
     ↓
Running Container

That same design maps directly to:

ECS Task Definition
        +
AWS Secrets Manager

or:

Kubernetes Deployment
        +
External Secrets

PART 24 — The DevOps Engineer's LLM Architecture

Now students should draw this themselves:

                         USER
                           |
                           v
                 +------------------+
                 | AI Application   |
                 +------------------+
                           |
            +--------------+--------------+
            |                             |
            v                             v
     System Prompt                   User Input
            |                             |
            +--------------+--------------+
                           |
                           v
                      Tokenization
                           |
                           v
                      Context Window
                           |
                           v
                       LLM API
                           |
                           v
                 +-------------------+
                 | Model Inference   |
                 |                   |
                 | Embeddings        |
                 | Attention         |
                 | Transformer       |
                 | Neural Network    |
                 +-------------------+
                           |
                           v
                     Output Tokens
                           |
                           v
                       Response
                           |
                           v
                    AI Application
                           |
                           v
                         USER


DEVOPS LAYER
====================================================

GitHub
   |
CI/CD
   |
Docker
   |
ECR
   |
ECS / EKS
   |
Load Balancer
   |
Route 53
   |
HTTPS

Security:
Secrets Manager
IAM
Network policies

Observability:
CloudWatch
Prometheus
Grafana
Logs
Alerts

LLM Metrics:
Latency
Requests
Errors
Input Tokens
Output Tokens
Cost
Rate Limits

This is why a DevOps engineer needs LLM knowledge

We're not learning LLMs so a DevOps engineer can ask ChatGPT questions.

We're learning them because companies are putting LLMs inside production systems.

The infrastructure team now has to operate something like:

Application
       |
       +---- PostgreSQL
       |
       +---- Redis
       |
       +---- Kafka
       |
       +---- OpenAI/Claude/etc.
       |
       +---- Vector Database
       |
       +---- Agent tools
       |
       +---- internal APIs

A traditional DevOps engineer monitors:

CPU
memory
disk
network
5xx
latency
pods

An AI DevOps/Platform engineer additionally cares about:

input tokens

output tokens

time to first token

tokens per second

LLM latency

API rate limits

model failures

prompt versions

model versions

hallucination/evaluation failures

RAG retrieval quality

vector DB health

embedding jobs

AI cost

GPU utilization

That is the bridge between LLM engineering and DevOps.

What We Intentionally Did NOT Add Yet

Do not throw LangChain, ChromaDB, Pinecone, Redis, agents and Kubernetes into this first lab.

Your students first need this mental model:

                  LLM APPLICATION

Prompt
  ↓
Tokens
  ↓
Context
  ↓
Model
  ↓
Inference
  ↓
Tokens
  ↓
Response


                 APPLICATION LAYER

Python
 ↓
SDK
 ↓
REST API
 ↓
LLM Provider


                  DEVOPS LAYER

Secrets
Logging
Monitoring
Docker
CI/CD
Deployment
Security
Cost
Reliability

Only after they completely understand this should Lab 2 introduce embeddings, vector similarity and RAG.

The course sequence I recommend from here

Lab	Build	What students actually learn
1 — LLM Fundamentals	The application above	Tokens, inference, transformer, context, API, secrets, logging, Docker
2 — Embeddings	DevOps incident similarity search	Embeddings, vectors, cosine similarity
3 — RAG	Ask questions against Kubernetes/AWS documentation	Chunking, embedding, retrieval, vector DB, grounding
4 — LLM Memory	Multi-turn DevOps assistant	Stateless vs stateful AI, conversation context, Redis/Postgres
5 — Tool Calling	AI executes safe read-only `kubectl`/AWS diagnostics	Agents, tools, structured outputs
6 — MCP/Integrations	Connect AI with operational systems	AI-to-tool integration architecture
7 — LLM Evaluation	Test answers automatically	Hallucinations, quality gates, regression testing
8 — AI Observability	Prometheus/Grafana AI dashboard	tokens, cost, latency, failures, traces
9 — AI Security	Attack the assistant	prompt injection, secret leakage, permissions, guardrails
10 — Production AI Platform	EKS + Terraform + CI/CD + RAG + monitoring	Complete AI DevOps architecture

This Lab 1 should be the foundation. Lab 2 should not simply "use embeddings"; we should actually print vectors, calculate similarity ourselves, compare Kubernetes/Docker/banana sentences mathematically, then build a small DevOps knowledge search engine. That will make embeddings → vector DB → RAG feel obvious instead of magical.

vpc creation

Aisalkyn Aidarova — Fri, 07 Aug 2026 01:37:00 +0000

Step 1. Create a VPC

Name: restaurant-vpc
CIDR: 10.0.0.0/16

Result:

AWS
└── VPC
    └── 10.0.0.0/16

Step 2. Create an Internet Gateway (IGW)

restaurant-igw

Attach it to the VPC.

Result:

Internet
    │
Internet Gateway
    │
restaurant-vpc

Step 3. Create Public Subnets

Create two public subnets in different Availability Zones.

public-subnet-1a
AZ: us-east-1a
CIDR: 10.0.1.0/24

public-subnet-1b
AZ: us-east-1b
CIDR: 10.0.2.0/24

Enable:

Auto Assign Public IP = Yes

Result:

VPC
├── Public Subnet 1A
└── Public Subnet 1B

Step 4. Create Private Subnets

private-subnet-1a
AZ: us-east-1a
CIDR: 10.0.11.0/24

private-subnet-1b
AZ: us-east-1b
CIDR: 10.0.12.0/24

Do NOT enable public IP.

Result:

VPC
├── Public 1A
├── Public 1B
├── Private 1A
└── Private 1B

Step 5. Create Public Route Table

restaurant-public-rt

Add route:

Destination      Target

0.0.0.0/0  --->  Internet Gateway

Associate it with:

Public Subnet 1A
Public Subnet 1B

Step 6. Create Elastic IP

VPC → Elastic IP

Allocate one Elastic IP.

Step 7. Create NAT Gateway

Inside:

public-subnet-1a

Attach:

Elastic IP

Wait until status becomes Available.

Step 8. Create Private Route Table

restaurant-private-rt

Route:

0.0.0.0/0  ---> NAT Gateway

Associate:

Private Subnet 1A
Private Subnet 1B

Step 9. Create Security Groups

For example:

ALB SG

HTTP 80
HTTPS 443

Source:
0.0.0.0/0

ECS SG

80

Source:
ALB Security Group

Database SG (later)

5432

Source:
ECS Security Group

Step 10. Launch EC2 (optional Bastion)

Place it in:

Public Subnet

Assign Public IP.

Step 11. Create ECS Cluster

restaurant-cluster

Step 12. Create Task Definition

Docker image

CPU

Memory

Logs

Execution Role

Step 13. Create ECS Service

Choose:

Launch Type:
Fargate

Network:

Private Subnet 1A
Private Subnet 1B

Security Group:

ecs-sg

Public IP:

Disabled

Step 14. Create ALB

Choose:

Internet Facing

Subnets:

Public 1A
Public 1B

Security Group:

alb-sg

Step 15. Create Target Group

Type:

IP

Health Check:

/health

Step 16. Connect ECS Service to ALB

Listener:

Target Group:

restaurant-tg

Step 17. Create Route 53

Hosted Zone

Create:

A Record
Alias → ALB

Step 18. Create ACM Certificate

Request:

restaurant-company.com

Validate with Route 53.

Attach to ALB.

Step 19. Enable Auto Scaling

Minimum:

Desired:

Maximum:

Policy:

CPU 60%

Step 20. Enable Monitoring

CloudWatch Logs
CloudWatch Metrics
CloudWatch Alarm
SNS Notification

Final Architecture

                 Internet
                     │
             Internet Gateway
                     │
        ┌─────────────────────────┐
        │          VPC            │
        │      10.0.0.0/16        │
        └─────────────────────────┘
            │                  │
    Public Subnet A      Public Subnet B
      10.0.1.0/24          10.0.2.0/24
            │                  │
        ALB (HTTPS)─────────────┘
            │
       Target Group
            │
    ┌───────┴────────┐
    │                │
Private Subnet A  Private Subnet B
10.0.11.0/24      10.0.12.0/24
    │                │
 ECS Task 1       ECS Task 2
    │                │
    └───────┬────────┘
            │
      Private Route Table
            │
       NAT Gateway
            │
      Elastic IP
            │
        Internet

This sequence gives students a clear understanding of how networking is built from the ground up before deploying applications.

FDE Lab 1: Build a DevOps AI Assistant

Aisalkyn Aidarova — Thu, 06 Aug 2026 15:50:27 +0000

Lab objective

Students will build a browser-based AI assistant that can answer questions about:

AWS
Linux
Git
Docker
Kubernetes
Terraform
CI/CD
Helm
Argo CD
Monitoring
Networking
Security

The application will include:

Browser frontend
        ↓
FastAPI backend
        ↓
OpenAI API
        ↓
Large Language Model
        ↓
DevOps answer

Students will learn:

How an application communicates with an LLM
How system instructions control model behavior
How prompts are sent from a frontend
How to keep API keys secure
How to create REST API endpoints
How to maintain short conversation history
How to validate requests
How to test an AI application
How to package the application with Docker

1. Final project structure

fde-devops-ai-assistant/
├── app/
│   ├── static/
│   │   ├── index.html
│   │   ├── app.js
│   │   └── styles.css
│   ├── __init__.py
│   ├── config.py
│   ├── main.py
│   ├── models.py
│   └── services.py
├── tests/
│   ├── __init__.py
│   └── test_api.py
├── .dockerignore
├── .env.example
├── .gitignore
├── Dockerfile
├── Makefile
├── README.md
└── requirements.txt

2. Create the project

Open Terminal.

mkdir fde-devops-ai-assistant
cd fde-devops-ai-assistant

Create the folders:

mkdir -p app/static
mkdir -p tests

Create the files:

touch app/__init__.py
touch app/config.py
touch app/main.py
touch app/models.py
touch app/services.py

touch app/static/index.html
touch app/static/app.js
touch app/static/styles.css

touch tests/__init__.py
touch tests/test_api.py

touch requirements.txt
touch .env.example
touch .gitignore
touch .dockerignore
touch Dockerfile
touch Makefile
touch README.md

Check the structure:

find . -maxdepth 3 -type f

3. Create `requirements.txt`

fastapi>=0.115,<1.0
uvicorn[standard]>=0.34,<1.0
openai>=1.65,<3.0
python-dotenv>=1.0,<2.0
pydantic-settings>=2.7,<3.0
pytest>=8.3,<9.0
httpx>=0.28,<1.0

Explanation

fastapi creates the REST API.

uvicorn runs the FastAPI application.

openai connects the application to the OpenAI API.

python-dotenv allows local environment variables to be loaded from .env.

pydantic-settings validates configuration.

pytest runs automated tests.

httpx is used by FastAPI testing tools.

4. Create `.env.example`

OPENAI_API_KEY=replace_with_your_api_key
OPENAI_MODEL=gpt-5
MAX_HISTORY_MESSAGES=10

Students will copy this file to .env.

The .env.example file can be committed to Git because it does not contain a real secret.

The .env file must never be committed.

5. Create `.gitignore`

.venv/
__pycache__/
*.py[cod]
.pytest_cache/
.env
.DS_Store
coverage.xml
htmlcov/

6. Create `.dockerignore`

.venv
.git
.gitignore
.env
__pycache__
.pytest_cache
tests
*.pyc
.DS_Store

This prevents unnecessary or sensitive files from being copied into the Docker image.

7. Create `app/config.py`

from functools import lru_cache

from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    """
    Application configuration.

    Values are loaded from environment variables or from a local .env file.
    """

    openai_api_key: str = ""
    openai_model: str = "gpt-5"
    max_history_messages: int = 10

    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        case_sensitive=False,
        extra="ignore",
    )


@lru_cache
def get_settings() -> Settings:
    """
    Return a cached Settings object.

    Caching prevents the application from repeatedly reading the environment
    file for every request.
    """

    return Settings()

What this file does

This file manages the application configuration.

The application expects these environment variables:

OPENAI_API_KEY
OPENAI_MODEL
MAX_HISTORY_MESSAGES

Pydantic automatically converts environment variable names such as:

OPENAI_API_KEY

into the Python field:

openai_api_key

8. Create `app/models.py`

from pydantic import BaseModel, Field, field_validator


class ChatRequest(BaseModel):
    """
    Request body sent by the browser to POST /api/chat.
    """

    session_id: str = Field(
        min_length=1,
        max_length=100,
    )

    message: str = Field(
        min_length=1,
        max_length=4000,
    )

    @field_validator("session_id", "message")
    @classmethod
    def strip_whitespace(cls, value: str) -> str:
        """
        Remove whitespace from the beginning and end.

        Reject values that contain only spaces.
        """

        cleaned = value.strip()

        if not cleaned:
            raise ValueError("Value must not be empty.")

        return cleaned


class ChatResponse(BaseModel):
    """
    Response returned by POST /api/chat.
    """

    session_id: str
    answer: str
    model: str
    history_messages: int


class HealthResponse(BaseModel):
    """
    Response returned by GET /api/health.
    """

    status: str
    service: str
    model: str


class DeleteHistoryResponse(BaseModel):
    """
    Response returned by DELETE /api/history/{session_id}.
    """

    session_id: str
    deleted: bool

Why models are important

Models validate incoming and outgoing data.

For example, this request will be rejected because the message is empty:

{
  "session_id": "student-1",
  "message": ""
}

This request will also be rejected:

{
  "session_id": "",
  "message": "Explain Docker."
}

FastAPI automatically returns HTTP status code 422 when validation fails.

9. Create `app/services.py`

import logging
from collections import defaultdict
from threading import Lock
from typing import Protocol

from openai import OpenAI

from app.config import Settings


logger = logging.getLogger(__name__)


SYSTEM_INSTRUCTIONS = """
You are a Senior DevOps Engineer and patient technical instructor.

Your responsibilities:

1. Answer questions about Linux, Git, Docker, CI/CD, AWS, Terraform,
   Kubernetes, Helm, Argo CD, monitoring, networking, reliability,
   cloud infrastructure, and security.

2. Begin with a direct and simple explanation.

3. Use beginner-friendly language first, then add technical depth.

4. Include commands, YAML, Terraform, Docker, or configuration examples
   when they are useful.

5. Clearly label commands that can change, restart, or delete infrastructure.

6. Never claim that you executed a command.

7. Never invent logs, metrics, deployment results, AWS resources,
   Kubernetes resources, or monitoring data.

8. When information is missing, explain exactly what the student should inspect.

9. When uncertain, clearly say that verification is required.

10. Keep the response focused on the student's question.

11. For troubleshooting questions, organize the answer using:

    - What the problem means
    - Most likely causes
    - Commands to run
    - How to interpret the output
    - Safe next steps

12. Warn the student before providing destructive commands such as:

    - kubectl delete
    - terraform destroy
    - aws resource deletion commands
    - database deletion commands
""".strip()


class AIService(Protocol):
    """
    Interface used by the FastAPI application.

    Using a protocol makes it easy to replace the real OpenAI service
    with a fake service during tests.
    """

    def answer(
        self,
        session_id: str,
        message: str,
    ) -> tuple[str, int]:
        ...

    def clear_history(
        self,
        session_id: str,
    ) -> bool:
        ...


class OpenAIDevOpsService:
    """
    OpenAI-backed DevOps assistant.

    Conversation history is stored in memory.

    This approach is acceptable for a classroom lab but is not appropriate
    for a production application with multiple replicas.
    """

    def __init__(self, settings: Settings):
        if not settings.openai_api_key:
            raise ValueError(
                "OPENAI_API_KEY is missing. "
                "Copy .env.example to .env and add a valid API key."
            )

        self.settings = settings

        self.client = OpenAI(
            api_key=settings.openai_api_key
        )

        self._history: dict[
            str,
            list[dict[str, str]]
        ] = defaultdict(list)

        self._lock = Lock()

    def answer(
        self,
        session_id: str,
        message: str,
    ) -> tuple[str, int]:
        """
        Send the user's message and recent conversation history to the model.

        Returns:
            tuple:
                - generated answer
                - number of messages stored in history
        """

        with self._lock:
            previous_messages = list(
                self._history[session_id]
            )

        model_input = previous_messages + [
            {
                "role": "user",
                "content": message,
            }
        ]

        try:
            response = self.client.responses.create(
                model=self.settings.openai_model,
                instructions=SYSTEM_INSTRUCTIONS,
                input=model_input,
            )

        except Exception:
            logger.exception(
                "OpenAI request failed for session %s",
                session_id,
            )
            raise

        answer = response.output_text.strip()

        if not answer:
            answer = (
                "The model returned an empty response. "
                "Please try again."
            )

        with self._lock:
            history = self._history[session_id]

            history.extend(
                [
                    {
                        "role": "user",
                        "content": message,
                    },
                    {
                        "role": "assistant",
                        "content": answer,
                    },
                ]
            )

            max_messages = max(
                2,
                self.settings.max_history_messages,
            )

            if len(history) > max_messages:
                self._history[session_id] = history[
                    -max_messages:
                ]

            history_count = len(
                self._history[session_id]
            )

        return answer, history_count

    def clear_history(
        self,
        session_id: str,
    ) -> bool:
        """
        Delete history for one browser session.

        Returns True when the session existed.
        """

        with self._lock:
            existed = session_id in self._history
            self._history.pop(session_id, None)

        return existed

10. Understanding the service

The most important code is:

response = self.client.responses.create(
    model=self.settings.openai_model,
    instructions=SYSTEM_INSTRUCTIONS,
    input=model_input,
)

The model specifies which model to use.

The instructions define how the assistant should behave.

The input contains the current user request and conversation history.

The model returns text through:

response.output_text

11. How conversation history works

Suppose the first user message is:

What is Kubernetes?

The history becomes:

[
    {
        "role": "user",
        "content": "What is Kubernetes?"
    },
    {
        "role": "assistant",
        "content": "Kubernetes is a container orchestration platform..."
    }
]

The user then asks:

How does it perform scaling?

The application sends the previous conversation plus the new question:

[
    {
        "role": "user",
        "content": "What is Kubernetes?"
    },
    {
        "role": "assistant",
        "content": "Kubernetes is a container orchestration platform..."
    },
    {
        "role": "user",
        "content": "How does it perform scaling?"
    }
]

The model understands that the word it refers to Kubernetes.

12. Create `app/main.py`

import logging
from contextlib import asynccontextmanager
from pathlib import Path

from fastapi import (
    Depends,
    FastAPI,
    HTTPException,
    Request,
    status,
)
from fastapi.responses import FileResponse
from fastapi.staticfiles import StaticFiles

from app.config import Settings, get_settings
from app.models import (
    ChatRequest,
    ChatResponse,
    DeleteHistoryResponse,
    HealthResponse,
)
from app.services import (
    AIService,
    OpenAIDevOpsService,
)


logging.basicConfig(
    level=logging.INFO,
    format=(
        "%(asctime)s "
        "%(levelname)s "
        "%(name)s "
        "%(message)s"
    ),
)


logger = logging.getLogger(__name__)


STATIC_DIR = Path(__file__).parent / "static"


def create_ai_service(
    settings: Settings,
) -> AIService:
    """
    Create the real OpenAI service.

    Keeping service creation in a separate function makes the application
    easier to test and extend.
    """

    return OpenAIDevOpsService(settings)


@asynccontextmanager
async def lifespan(app: FastAPI):
    """
    Run application startup and shutdown logic.

    The AI service is created once when the application starts.
    """

    settings = get_settings()

    try:
        app.state.ai_service = create_ai_service(
            settings
        )

        logger.info(
            "AI service initialized with model %s",
            settings.openai_model,
        )

    except ValueError as exc:
        app.state.ai_service = None

        logger.warning(
            "AI service is not configured: %s",
            exc,
        )

    yield


app = FastAPI(
    title="FDE DevOps AI Assistant",
    description=(
        "A student lab demonstrating a "
        "production-shaped LLM application."
    ),
    version="1.0.0",
    lifespan=lifespan,
)


app.mount(
    "/static",
    StaticFiles(directory=STATIC_DIR),
    name="static",
)


def get_ai_service(
    request: Request,
) -> AIService:
    """
    Dependency that retrieves the initialized AI service.
    """

    service = getattr(
        request.app.state,
        "ai_service",
        None,
    )

    if service is None:
        raise HTTPException(
            status_code=(
                status.HTTP_503_SERVICE_UNAVAILABLE
            ),
            detail=(
                "AI service is not configured. "
                "Set OPENAI_API_KEY and restart "
                "the application."
            ),
        )

    return service


@app.get(
    "/",
    include_in_schema=False,
)
def home() -> FileResponse:
    """
    Return the browser frontend.
    """

    return FileResponse(
        STATIC_DIR / "index.html"
    )


@app.get(
    "/api/health",
    response_model=HealthResponse,
)
def health(
    settings: Settings = Depends(get_settings),
) -> HealthResponse:
    """
    Application health endpoint.

    This endpoint does not call the OpenAI API.
    """

    return HealthResponse(
        status="UP",
        service="devops-ai-assistant",
        model=settings.openai_model,
    )


@app.post(
    "/api/chat",
    response_model=ChatResponse,
)
def chat(
    payload: ChatRequest,
    service: AIService = Depends(
        get_ai_service
    ),
    settings: Settings = Depends(
        get_settings
    ),
) -> ChatResponse:
    """
    Send a user question to the DevOps AI assistant.
    """

    try:
        answer, history_count = service.answer(
            session_id=payload.session_id,
            message=payload.message,
        )

    except HTTPException:
        raise

    except Exception as exc:
        logger.exception(
            "Chat request failed"
        )

        raise HTTPException(
            status_code=(
                status.HTTP_502_BAD_GATEWAY
            ),
            detail=(
                "The AI provider request failed. "
                "Check the server logs."
            ),
        ) from exc

    return ChatResponse(
        session_id=payload.session_id,
        answer=answer,
        model=settings.openai_model,
        history_messages=history_count,
    )


@app.delete(
    "/api/history/{session_id}",
    response_model=DeleteHistoryResponse,
)
def delete_history(
    session_id: str,
    service: AIService = Depends(
        get_ai_service
    ),
) -> DeleteHistoryResponse:
    """
    Delete one session's conversation history.
    """

    deleted = service.clear_history(
        session_id
    )

    return DeleteHistoryResponse(
        session_id=session_id,
        deleted=deleted,
    )

13. API endpoints

The application has four endpoints.

Home page

GET /

Returns the browser interface.

Health check

GET /api/health

Example response:

{
  "status": "UP",
  "service": "devops-ai-assistant",
  "model": "gpt-5"
}

Chat endpoint

POST /api/chat

Request:

{
  "session_id": "student-1",
  "message": "Explain Docker."
}

Response:

{
  "session_id": "student-1",
  "answer": "Docker is a platform used to package applications...",
  "model": "gpt-5",
  "history_messages": 2
}

Delete conversation history

DELETE /api/history/student-1

Response:

{
  "session_id": "student-1",
  "deleted": true
}

14. Create `app/static/index.html`

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">

  <meta
    name="viewport"
    content="width=device-width, initial-scale=1.0"
  >

  <title>DevOps AI Assistant</title>

  <link
    rel="stylesheet"
    href="/static/styles.css"
  >
</head>

<body>
  <main class="app-shell">

    <header class="hero">
      <div>
        <p class="eyebrow">
          FDE LAB 1
        </p>

        <h1>
          DevOps AI Assistant
        </h1>

        <p class="subtitle">
          Ask questions about AWS, Docker,
          Kubernetes, Terraform, CI/CD,
          monitoring, Linux, and Git.
        </p>
      </div>

      <button
        id="clearButton"
        class="secondary-button"
        type="button"
      >
        Clear history
      </button>
    </header>

    <section
      class="status-bar"
      aria-live="polite"
    >
      <span
        id="statusDot"
        class="status-dot"
      ></span>

      <span id="statusText">
        Checking API health...
      </span>

      <span id="modelText"></span>
    </section>

    <section
      id="messages"
      class="messages"
      aria-live="polite"
    >
      <article class="message assistant">

        <div class="message-label">
          Assistant
        </div>

        <div class="message-content">
          Welcome. Ask me a DevOps question.
          For example: “Why is my Kubernetes
          Pod in CrashLoopBackOff?”
        </div>

      </article>
    </section>

    <form
      id="chatForm"
      class="composer"
    >
      <label for="messageInput">
        Your question
      </label>

      <textarea
        id="messageInput"
        rows="4"
        maxlength="4000"
        placeholder="Explain the difference between readiness and liveness probes..."
        required
      ></textarea>

      <div class="composer-footer">

        <span id="characterCount">
          0 / 4000
        </span>

        <button
          id="sendButton"
          type="submit"
        >
          Ask assistant
        </button>

      </div>
    </form>

  </main>

  <script src="/static/app.js"></script>
</body>
</html>

15. Create `app/static/app.js`

const chatForm =
  document.getElementById("chatForm");

const messageInput =
  document.getElementById("messageInput");

const messages =
  document.getElementById("messages");

const sendButton =
  document.getElementById("sendButton");

const clearButton =
  document.getElementById("clearButton");

const characterCount =
  document.getElementById("characterCount");

const statusText =
  document.getElementById("statusText");

const statusDot =
  document.getElementById("statusDot");

const modelText =
  document.getElementById("modelText");


const sessionId =
  localStorage.getItem("fde-session-id") ||
  (
    crypto.randomUUID
      ? crypto.randomUUID()
      : `session-${Date.now()}`
  );


localStorage.setItem(
  "fde-session-id",
  sessionId
);


function addMessage(role, text) {
  const article =
    document.createElement("article");

  article.className =
    `message ${role}`;


  const label =
    document.createElement("div");

  label.className =
    "message-label";

  label.textContent =
    role === "user"
      ? "You"
      : "Assistant";


  const content =
    document.createElement("div");

  content.className =
    "message-content";

  content.textContent = text;


  article.append(
    label,
    content
  );

  messages.appendChild(article);

  messages.scrollTop =
    messages.scrollHeight;

  return article;
}


function setLoading(isLoading) {
  sendButton.disabled =
    isLoading;

  messageInput.disabled =
    isLoading;

  sendButton.textContent =
    isLoading
      ? "Thinking..."
      : "Ask assistant";
}


async function checkHealth() {
  try {
    const response =
      await fetch("/api/health");

    if (!response.ok) {
      throw new Error(
        `Health check returned ${response.status}`
      );
    }

    const data =
      await response.json();

    statusDot.classList.add(
      "healthy"
    );

    statusText.textContent =
      `${data.service} is ${data.status}`;

    modelText.textContent =
      `Model: ${data.model}`;

  } catch (error) {
    statusDot.classList.add(
      "unhealthy"
    );

    statusText.textContent =
      "API health check failed";

    modelText.textContent = "";
  }
}


messageInput.addEventListener(
  "input",
  () => {
    characterCount.textContent =
      `${messageInput.value.length} / 4000`;
  }
);


chatForm.addEventListener(
  "submit",
  async (event) => {
    event.preventDefault();

    const message =
      messageInput.value.trim();

    if (!message) {
      return;
    }

    addMessage(
      "user",
      message
    );

    messageInput.value = "";

    characterCount.textContent =
      "0 / 4000";

    setLoading(true);

    const pending = addMessage(
      "assistant",
      "Thinking..."
    );

    try {
      const response = await fetch(
        "/api/chat",
        {
          method: "POST",

          headers: {
            "Content-Type":
              "application/json",
          },

          body: JSON.stringify({
            session_id: sessionId,
            message: message,
          }),
        }
      );

      const data =
        await response.json();

      if (!response.ok) {
        const detail =
          typeof data.detail === "string"
            ? data.detail
            : "The request failed.";

        throw new Error(detail);
      }

      pending
        .querySelector(".message-content")
        .textContent = data.answer;

      modelText.textContent =
        `Model: ${data.model} | ` +
        `History messages: ${data.history_messages}`;

    } catch (error) {
      pending
        .querySelector(".message-content")
        .textContent =
          `Error: ${error.message}`;

      pending.classList.add(
        "error"
      );

    } finally {
      setLoading(false);
      messageInput.focus();
    }
  }
);


clearButton.addEventListener(
  "click",
  async () => {
    try {
      const response = await fetch(
        `/api/history/${
          encodeURIComponent(sessionId)
        }`,
        {
          method: "DELETE",
        }
      );

      if (!response.ok) {
        throw new Error(
          "Could not clear history."
        );
      }

      messages.innerHTML = "";

      addMessage(
        "assistant",
        "Conversation history was cleared. " +
        "Start a new question."
      );

      modelText.textContent = "";

    } catch (error) {
      addMessage(
        "assistant",
        `Error: ${error.message}`
      );
    }
  }
);


checkHealth();
messageInput.focus();

16. How the frontend works

When the user enters a question, JavaScript sends this request:

fetch("/api/chat", {
  method: "POST",
  headers: {
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    session_id: sessionId,
    message: message
  })
});

The browser does not contain the OpenAI API key.

This is important.

The secure flow is:

Browser
   ↓
Our backend
   ↓
OpenAI API

The insecure flow would be:

Browser containing secret key
   ↓
OpenAI API

Anyone can inspect browser JavaScript. Therefore, secret keys must remain on the backend.

17. Create `app/static/styles.css`

:root {
  color-scheme: light;

  font-family:
    Inter,
    ui-sans-serif,
    system-ui,
    -apple-system,
    BlinkMacSystemFont,
    "Segoe UI",
    sans-serif;

  background: #f4f7fb;
  color: #172033;
}


* {
  box-sizing: border-box;
}


body {
  margin: 0;
  min-height: 100vh;

  background:
    radial-gradient(
      circle at top left,
      #dbeafe 0,
      transparent 35%
    ),
    #f4f7fb;
}


button,
textarea {
  font: inherit;
}


.app-shell {
  width: min(
    960px,
    calc(100% - 32px)
  );

  margin: 32px auto;
}


.hero {
  display: flex;
  justify-content: space-between;
  gap: 24px;
  align-items: flex-start;

  padding: 28px;

  background: #ffffff;

  border:
    1px solid #dbe3ef;

  border-radius:
    20px 20px 0 0;
}


.eyebrow {
  margin:
    0 0 8px;

  font-size: 0.76rem;
  font-weight: 800;
  letter-spacing: 0.16em;
  color: #3159a6;
}


h1 {
  margin: 0;

  font-size:
    clamp(
      2rem,
      5vw,
      3.25rem
    );

  line-height: 1;
}


.subtitle {
  max-width: 680px;

  margin:
    14px 0 0;

  color: #526078;
  line-height: 1.6;
}


.status-bar {
  display: flex;
  gap: 10px;
  align-items: center;

  min-height: 48px;

  padding:
    0 28px;

  background: #f8fafc;

  border-right:
    1px solid #dbe3ef;

  border-left:
    1px solid #dbe3ef;

  color: #526078;
  font-size: 0.9rem;
}


#modelText {
  margin-left: auto;
}


.status-dot {
  width: 10px;
  height: 10px;

  border-radius: 50%;

  background: #9ca3af;
}


.status-dot.healthy {
  background: #16a34a;
}


.status-dot.unhealthy {
  background: #dc2626;
}


.messages {
  height: 460px;
  overflow-y: auto;

  padding: 28px;

  background: #ffffff;

  border:
    1px solid #dbe3ef;
}


.message {
  max-width: 82%;
  margin-bottom: 22px;
}


.message.user {
  margin-left: auto;
}


.message-label {
  margin-bottom: 6px;

  font-size: 0.78rem;
  font-weight: 800;

  color: #526078;

  text-transform: uppercase;
  letter-spacing: 0.08em;
}


.message-content {
  padding:
    15px 17px;

  border-radius: 16px;

  line-height: 1.6;

  white-space: pre-wrap;
  overflow-wrap: anywhere;
}


.message.assistant
.message-content {
  background: #eef3fb;
}


.message.user
.message-content {
  background: #172033;
  color: #ffffff;
}


.message.error
.message-content {
  background: #fee2e2;
  color: #991b1b;
}


.composer {
  padding:
    24px 28px 28px;

  background: #ffffff;

  border:
    1px solid #dbe3ef;

  border-top: 0;

  border-radius:
    0 0 20px 20px;
}


.composer label {
  display: block;

  margin-bottom: 8px;

  font-weight: 700;
}


textarea {
  width: 100%;

  resize: vertical;

  min-height: 100px;

  padding: 14px;

  border:
    1px solid #b9c5d8;

  border-radius: 12px;

  outline: none;
}


textarea:focus {
  border-color: #3159a6;

  box-shadow:
    0 0 0 3px
    rgba(49, 89, 166, 0.14);
}


.composer-footer {
  display: flex;
  justify-content: space-between;
  align-items: center;
  gap: 16px;

  margin-top: 14px;
}


#characterCount {
  color: #667085;
  font-size: 0.85rem;
}


button {
  border: 0;
  border-radius: 10px;

  cursor: pointer;
  font-weight: 800;
}


#sendButton {
  padding:
    12px 20px;

  background: #3159a6;
  color: #ffffff;
}


#sendButton:disabled {
  cursor: not-allowed;
  opacity: 0.6;
}


.secondary-button {
  padding:
    10px 14px;

  background: #e8eef8;
  color: #243b68;

  white-space: nowrap;
}


@media (
  max-width: 700px
) {
  .app-shell {
    width: 100%;
    margin: 0;
  }

  .hero {
    flex-direction: column;
    border-radius: 0;
  }

  .messages {
    height: 52vh;
  }

  .composer {
    border-radius: 0;
  }

  #modelText {
    display: none;
  }
}

18. Create `app/init.py`

Leave this file empty:

Its presence tells Python that app is a Python package.

19. Create `tests/init.py`

Leave this file empty:

20. Create `tests/test_api.py`

from fastapi.testclient import TestClient

from app.main import (
    app,
    get_ai_service,
)


class FakeAIService:
    """
    Fake AI service used during testing.

    It does not call OpenAI and does not consume API credits.
    """

    def __init__(self):
        self.history: dict[
            str,
            list[str]
        ] = {}

    def answer(
        self,
        session_id: str,
        message: str,
    ) -> tuple[str, int]:
        self.history.setdefault(
            session_id,
            [],
        ).extend(
            [
                message,
                f"Mock answer: {message}",
            ]
        )

        return (
            f"Mock answer: {message}",
            len(self.history[session_id]),
        )

    def clear_history(
        self,
        session_id: str,
    ) -> bool:
        existed = (
            session_id in self.history
        )

        self.history.pop(
            session_id,
            None,
        )

        return existed


fake_service = FakeAIService()


def override_ai_service():
    return fake_service


app.dependency_overrides[
    get_ai_service
] = override_ai_service


client = TestClient(app)


def test_health_endpoint():
    response = client.get(
        "/api/health"
    )

    assert response.status_code == 200

    body = response.json()

    assert body["status"] == "UP"

    assert (
        body["service"]
        == "devops-ai-assistant"
    )


def test_home_page():
    response = client.get("/")

    assert response.status_code == 200

    assert (
        "DevOps AI Assistant"
        in response.text
    )


def test_chat_endpoint():
    response = client.post(
        "/api/chat",
        json={
            "session_id":
                "test-session",

            "message":
                "What is Docker?",
        },
    )

    assert response.status_code == 200

    body = response.json()

    assert (
        body["answer"]
        == "Mock answer: What is Docker?"
    )

    assert (
        body["history_messages"]
        == 2
    )


def test_empty_message_is_rejected():
    response = client.post(
        "/api/chat",
        json={
            "session_id":
                "test-session",

            "message":
                "   ",
        },
    )

    assert response.status_code == 422


def test_delete_history():
    client.post(
        "/api/chat",
        json={
            "session_id":
                "delete-session",

            "message":
                "What is Terraform?",
        },
    )

    response = client.delete(
        "/api/history/delete-session"
    )

    assert response.status_code == 200

    assert response.json() == {
        "session_id":
            "delete-session",

        "deleted":
            True,
    }

21. Why we use a fake service in tests

We do not want automated tests to call the real OpenAI API.

Real API calls would:

Consume money
Depend on the internet
Be slower
Produce variable answers
Possibly fail because of rate limits

Instead, this test replaces the real service with:

class FakeAIService:

When the application calls:

service.answer(...)

the test returns:

Mock answer: What is Docker?

This is called dependency injection.

22. Create the `Dockerfile`

FROM python:3.12-slim


ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1


WORKDIR /app


RUN addgroup \
      --system \
      appgroup \
    && adduser \
      --system \
      --ingroup appgroup \
      appuser


COPY requirements.txt .


RUN pip install \
      --upgrade pip \
    && pip install \
      -r requirements.txt


COPY app ./app


USER appuser


EXPOSE 8000


HEALTHCHECK \
  --interval=30s \
  --timeout=5s \
  --start-period=10s \
  --retries=3 \
  CMD python -c \
  "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health')" \
  || exit 1


CMD [
  "uvicorn",
  "app.main:app",
  "--host",
  "0.0.0.0",
  "--port",
  "8000"
]

23. Dockerfile explanation

Base image

FROM python:3.12-slim

Uses a smaller Python image.

Environment variables

ENV PYTHONDONTWRITEBYTECODE=1

Prevents Python from creating unnecessary .pyc files.

PYTHONUNBUFFERED=1

Makes logs appear immediately.

PIP_NO_CACHE_DIR=1

Prevents pip from keeping its download cache.

Working directory

WORKDIR /app

All following commands run inside /app.

Non-root user

RUN addgroup --system appgroup \
    && adduser --system --ingroup appgroup appuser

Creates a non-root user.

Later:

USER appuser

The application does not run as root.

Health check

Docker calls:

http://localhost:8000/api/health

to verify the application is responding.

24. Create `Makefile`

Make sure commands below each target begin with a real tab.

.PHONY: install run test docker-build docker-run


install:
    python -m pip install --upgrade pip
    pip install -r requirements.txt


run:
    uvicorn app.main:app --reload --host 0.0.0.0 --port 8000


test:
    pytest -v


docker-build:
    docker build -t fde-devops-ai-assistant:v1 .


docker-run:
    docker run --rm --name fde-ai -p 8000:8000 --env-file .env fde-devops-ai-assistant:v1

Students can now use:

make install

make run

make test

make docker-build

make docker-run

25. Create `README.md`

# FDE Lab 1: DevOps AI Assistant

## Overview

This project is a browser-based DevOps AI assistant built with:

- Python
- FastAPI
- OpenAI API
- HTML
- CSS
- JavaScript
- Docker
- Pytest

## Architecture

```text
Browser
   |
   | HTTP
   v
FastAPI backend
   |
   | OpenAI Responses API
   v
Large Language Model

Features

DevOps-focused AI assistant
Browser chat interface
Conversation history
Clear-history function
Request validation
Health endpoint
Error handling
Automated testing
Docker container
Non-root container user
Container health check

Local setup

Create a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Copy the environment file:

cp .env.example .env

Add your API key:

OPENAI_API_KEY=your_real_api_key
OPENAI_MODEL=gpt-5
MAX_HISTORY_MESSAGES=10

Run:

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Open:

http://localhost:8000

API documentation

Open:

http://localhost:8000/docs

Health endpoint

curl http://localhost:8000/api/health

Test the chat API

curl -X POST \
  http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "student-1",
    "message": "Explain Docker images and containers."
  }'

Run tests

pytest -v

Build Docker image

docker build \
  -t fde-devops-ai-assistant:v1 \
  .

Run Docker container

docker run \
  --rm \
  --name fde-ai \
  -p 8000:8000 \
  --env-file .env \
  fde-devops-ai-assistant:v1

Definition of done

The project is complete when:

The browser page loads
Health endpoint returns 200
User can ask a DevOps question
Assistant returns a response
Follow-up questions use history
Clear History works
Tests pass
Docker image builds
Docker container runs
.env is not committed


---

# 26. Create a virtual environment

From the project root:

```bash
python3 -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Activate it on Windows PowerShell:

.venv\Scripts\Activate.ps1

After activation, Terminal should show something similar to:

(.venv) student@computer fde-devops-ai-assistant %

27. Install dependencies

python -m pip install --upgrade pip

pip install -r requirements.txt

Verify FastAPI:

pip show fastapi

Verify OpenAI:

pip show openai

28. Create the real `.env` file

macOS or Linux:

cp .env.example .env

Windows PowerShell:

Copy-Item .env.example .env

Open .env:

OPENAI_API_KEY=your_real_api_key
OPENAI_MODEL=gpt-5
MAX_HISTORY_MESSAGES=10

Do not add quotation marks unless they are part of the secret.

Correct:

OPENAI_API_KEY=sk-example

Avoid:

OPENAI_API_KEY="sk-example"

29. Confirm `.env` is ignored

Run:

git init
git status

The .env file should not appear as an untracked file.

You should see .env.example, but not .env.

30. Run the application

uvicorn app.main:app \
  --reload \
  --host 0.0.0.0 \
  --port 8000

Expected output:

INFO:     Will watch for changes in these directories
INFO:     Uvicorn running on http://0.0.0.0:8000
INFO:     Started reloader process
INFO:     Started server process
INFO:     Application startup complete

Open in the browser:

http://localhost:8000

31. Open automatic API documentation

FastAPI automatically creates Swagger documentation.

Open:

http://localhost:8000/docs

You should see:

GET    /
GET    /api/health
POST   /api/chat
DELETE /api/history/{session_id}

32. Test the health endpoint

Browser:

http://localhost:8000/api/health

Terminal:

curl http://localhost:8000/api/health

Expected response:

{
  "status": "UP",
  "service": "devops-ai-assistant",
  "model": "gpt-5"
}

33. Test the chat endpoint

curl -X POST \
  http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "student-1",
    "message": "Explain the difference between Docker images and containers."
  }'

Expected structure:

{
  "session_id": "student-1",
  "answer": "A Docker image is a reusable template...",
  "model": "gpt-5",
  "history_messages": 2
}

The exact answer can vary.

LLM output is not always identical.

34. Test conversation history

First request:

curl -X POST \
  http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "student-history",
    "message": "What is Kubernetes?"
  }'

Second request using the same session ID:

curl -X POST \
  http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "student-history",
    "message": "How does it perform autoscaling?"
  }'

Because the session ID is the same, the model receives the previous conversation.

Now use a different session ID:

curl -X POST \
  http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "different-student",
    "message": "How does it perform autoscaling?"
  }'

The model may not know what it means because this session has no previous context.

35. Clear conversation history

curl -X DELETE \
  http://localhost:8000/api/history/student-history

Expected response:

{
  "session_id": "student-history",
  "deleted": true
}

Calling it again may return:

{
  "session_id": "student-history",
  "deleted": false
}

That means there was no remaining history.

36. Run automated tests

pytest -v

Expected output:

tests/test_api.py::test_health_endpoint PASSED
tests/test_api.py::test_home_page PASSED
tests/test_api.py::test_chat_endpoint PASSED
tests/test_api.py::test_empty_message_is_rejected PASSED
tests/test_api.py::test_delete_history PASSED

37. Build the Docker image

Stop the local application with:

Control + C

Build:

docker build \
  -t fde-devops-ai-assistant:v1 \
  .

Verify:

docker images

You should see:

fde-devops-ai-assistant   v1

38. Run the Docker container

docker run \
  --rm \
  --name fde-ai \
  -p 8000:8000 \
  --env-file .env \
  fde-devops-ai-assistant:v1

Open:

http://localhost:8000

39. Check Docker container status

In another Terminal:

docker ps

You should see the container:

fde-ai

After the health check runs, Docker should eventually show:

healthy

Inspect health status:

docker inspect \
  --format='{{json .State.Health}}' \
  fde-ai

View logs:

docker logs fde-ai

Follow logs:

docker logs -f fde-ai

40. Test the browser application

Ask:

What is Terraform?

Then ask:

Why do DevOps teams use it?

The second question should use the first answer as context.

Ask:

My Kubernetes Pod is in CrashLoopBackOff. What should I check?

A good response should include commands such as:

kubectl get pods

kubectl describe pod <pod-name>

kubectl logs <pod-name>

kubectl logs <pod-name> --previous

The assistant should not claim it executed those commands.

41. Lab experiment: system prompt

Open:

app/services.py

Find:

SYSTEM_INSTRUCTIONS

Change it temporarily to:

SYSTEM_INSTRUCTIONS = """
You are a DevOps instructor.

Explain every topic using a restaurant analogy.

Keep answers under 200 words.
""".strip()

Restart the application.

Ask:

What is Kubernetes?

Observe how the system instruction changes the answer.

Then restore the original system instruction.

42. Lab experiment: context

Ask:

What is an AWS Application Load Balancer?

Follow with:

Can it route traffic based on URL paths?

Then ask:

Show me an example.

The assistant should understand all three questions are connected.

Press:

Clear history

Then ask:

Show me an example.

Now the assistant does not have enough context.

This demonstrates why conversation history matters.

43. Lab experiment: user prompt quality

Compare these prompts.

Prompt 1

Docker

Prompt 2

Explain Docker.

Prompt 3

Explain Docker to a beginner DevOps student.
Include images, containers, Dockerfiles, registries,
volumes, and networking. Use one restaurant analogy.

Students should record:

Which answer is clearest
Which answer is most detailed
Which answer is easiest to understand
Why the third prompt performs better

44. Lab experiment: hallucination and uncertainty

Ask:

Show me the exact CPU usage of my Kubernetes Pod.

The assistant should explain that it cannot know the real CPU usage without access to the cluster or monitoring system.

It should recommend commands such as:

kubectl top pod <pod-name>

This demonstrates an important rule:

An LLM should not invent real infrastructure state.

45. Student exercise 1: Add a question counter

Add this inside index.html near the model information:

<span id="questionCount">
  Questions: 0
</span>

In app.js, add:

const questionCount =
  document.getElementById(
    "questionCount"
  );

let totalQuestions = 0;

After a successful user submission, add:

totalQuestions += 1;

questionCount.textContent =
  `Questions: ${totalQuestions}`;

When history is cleared:

totalQuestions = 0;

questionCount.textContent =
  "Questions: 0";

46. Student exercise 2: Add an explain-error endpoint

Add these models to app/models.py:

class ExplainErrorRequest(BaseModel):
    error: str = Field(
        min_length=1,
        max_length=4000,
    )


class ExplainErrorResponse(BaseModel):
    explanation: str
    model: str

Update imports in app/main.py:

from app.models import (
    ChatRequest,
    ChatResponse,
    DeleteHistoryResponse,
    ExplainErrorRequest,
    ExplainErrorResponse,
    HealthResponse,
)

Add the endpoint:

@app.post(
    "/api/explain-error",
    response_model=ExplainErrorResponse,
)
def explain_error(
    payload: ExplainErrorRequest,
    service: AIService = Depends(
        get_ai_service
    ),
    settings: Settings = Depends(
        get_settings
    ),
) -> ExplainErrorResponse:
    prompt = f"""
Analyze this DevOps error:

{payload.error}

Return:

1. What the error means
2. Most likely causes
3. Commands to run
4. How to interpret the output
5. Safe next steps
""".strip()

    try:
        answer, _ = service.answer(
            session_id="error-analysis",
            message=prompt,
        )

    except Exception as exc:
        raise HTTPException(
            status_code=502,
            detail=(
                "The AI provider request failed."
            ),
        ) from exc

    return ExplainErrorResponse(
        explanation=answer,
        model=settings.openai_model,
    )

Test:

curl -X POST \
  http://localhost:8000/api/explain-error \
  -H "Content-Type: application/json" \
  -d '{
    "error": "CrashLoopBackOff"
  }'

47. Student exercise 3: Add output length instructions

In SYSTEM_INSTRUCTIONS, add:

Keep normal responses under 500 words unless the user asks for a detailed explanation.

Test the difference.

48. Student exercise 4: Add request logging

In app/main.py, add:

import time

Add middleware before the endpoints:

@app.middleware("http")
async def log_requests(
    request: Request,
    call_next,
):
    start_time = time.perf_counter()

    response = await call_next(
        request
    )

    duration_ms = (
        time.perf_counter()
        - start_time
    ) * 1000

    logger.info(
        "%s %s status=%s duration_ms=%.2f",
        request.method,
        request.url.path,
        response.status_code,
        duration_ms,
    )

    return response

Now logs may look like:

POST /api/chat status=200 duration_ms=2450.31

49. Student exercise 5: Add Kubernetes endpoints

Create these endpoints:

POST /api/kubernetes/troubleshoot
POST /api/docker/explain
POST /api/terraform/review

Example request:

{
  "message": "Pod is Pending"
}

The Kubernetes endpoint should ask the model to return:

Meaning
Possible causes
Investigation commands
Expected outputs
Resolution options

50. Student exercise 6: Add rate limiting concept

Ask students:

Why should one user not be allowed to make unlimited requests?

Reasons:

API cost
Abuse
Denial-of-service risk
Provider rate limits
Resource consumption

A future version can add Redis-based rate limiting.

51. Student exercise 7: Replace memory with Redis

Current memory:

self._history

is stored inside one Python process.

Problems:

Lost after restart
Not shared between containers
Not persistent
Cannot support multiple replicas correctly

Production design:

Browser
   ↓
FastAPI Replica 1 ──┐
FastAPI Replica 2 ──┼── Redis
FastAPI Replica 3 ──┘

Redis would store conversation history by session ID.

52. Student exercise 8: Add authentication

Current application allows anyone to ask questions.

Production architecture should include:

User
  ↓
Login
  ↓
JWT or secure session
  ↓
FastAPI
  ↓
Authorized AI request

Each conversation should belong to an authenticated user.

53. Troubleshooting

Error: `OPENAI_API_KEY is missing`

Check:

ls -la

Confirm .env exists.

Check the file:

cat .env

It should contain:

OPENAI_API_KEY=your_real_key

Restart Uvicorn after changing .env.

Error: `ModuleNotFoundError`

Activate the environment:

source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Error: port 8000 already in use

macOS or Linux:

lsof -i :8000

Use another port:

uvicorn app.main:app \
  --reload \
  --port 8001

Open:

http://localhost:8001

Error: API returns 401

Possible reasons:

Invalid API key
Expired or revoked key
Spaces inside .env
Wrong environment variable name

Correct:

OPENAI_API_KEY=your_key

Incorrect:

OPEN_AI_KEY=your_key

Error: quota or billing problem

The API key may be valid, but the account may not have available API billing or credits.

API usage and ChatGPT subscriptions are generally separate services.

Error: Docker cannot access API key

Make sure you used:

docker run \
  --env-file .env \
  -p 8000:8000 \
  fde-devops-ai-assistant:v1

Do not copy .env into the Docker image.

Error: Docker container exits

Check logs:

docker logs fde-ai

Run without automatically removing the container:

docker run \
  --name fde-ai \
  -p 8000:8000 \
  --env-file .env \
  fde-devops-ai-assistant:v1

Then inspect:

docker ps -a

Error: empty response

Possible reasons:

Temporary API issue
Unsupported model
Provider error
Network issue

Check application logs.

54. Production limitations

This lab uses in-memory conversation history.

That is acceptable for learning, but not production.

Production improvements should include:

PostgreSQL or Redis
Authentication
Authorization
Rate limiting
Request timeouts
Retry logic
Centralized logging
Prometheus metrics
Distributed tracing
Cost tracking
Token tracking
Input safety checks
Output validation
Prompt-injection protection
Secret management
HTTPS
Audit logging
Persistent storage
Automated LLM evaluations
Kubernetes deployment
Horizontal autoscaling

55. Complete production architecture

                        User
                          │
                          ▼
                  React or Web UI
                          │
                          ▼
                 Application Load Balancer
                          │
                          ▼
                     FastAPI API
                          │
           ┌──────────────┼──────────────┐
           │              │              │
           ▼              ▼              ▼
        Redis         PostgreSQL     OpenAI API
           │
           ▼
   Conversation history

Additional components:

- AWS Secrets Manager
- CloudWatch
- Prometheus
- Grafana
- WAF
- Cognito or another identity provider
- GitHub Actions
- ECR
- ECS or EKS

56. Final result

Students should be able to open:

http://localhost:8000

They will see:

FDE LAB 1

DevOps AI Assistant

Ask questions about AWS, Docker, Kubernetes,
Terraform, CI/CD, monitoring, Linux, and Git.

Example question:

Why is my Kubernetes Pod in CrashLoopBackOff?

Expected answer structure:

What the problem means

CrashLoopBackOff means Kubernetes starts the container,
the container crashes, and Kubernetes waits before restarting it.

Most likely causes

- Application startup error
- Missing environment variable
- Invalid command
- Failed health probe
- Missing secret
- Database connection failure
- Insufficient permissions

Commands to run

kubectl get pods

kubectl describe pod <pod-name>

kubectl logs <pod-name>

kubectl logs <pod-name> --previous

How to interpret the output

Check exit codes, events, application errors, missing secrets,
probe failures, and resource problems.

Safe next steps

Correct the configuration, update the deployment, and verify
the rollout.

57. Definition of done

The lab is complete when:

The project structure is correct
Python virtual environment works
Dependencies are installed
.env contains the API key
.env is ignored by Git
The home page loads
/api/health returns HTTP 200
The user can ask a question
The assistant returns an answer
Follow-up questions use conversation history
Clear History works
Swagger documentation works
All tests pass
Docker image builds
Docker container starts
Docker health check becomes healthy
The application does not expose the API key
The container runs as a non-root user

58. Homework

Students must extend the project into an AI Kubernetes Troubleshooting Assistant.

Requirements:

Add a new page or section called:

Kubernetes Troubleshooter

Accept:

Pod status
kubectl describe output
kubectl logs output

Generate:

Problem summary
Likely root cause
Evidence
Commands to run
Recommended resolution
Risk warning

Add an endpoint:

POST /api/kubernetes/troubleshoot

Add at least three automated tests.
Build a new Docker image:

docker build \
  -t kubernetes-ai-assistant:v1 \
  .

Add screenshots and instructions to the README.
Push the project to GitHub without committing .env.