Build and Operate a DevOps LLM Incident Assistant
What students will understand
At the end of this lab, a student should be able to explain this architecture without memorizing definitions:
DevOps Engineer
|
| "Why is my Kubernetes pod restarting?"
v
+----------------------+
| Python Application |
+----------------------+
|
| builds prompt
v
+----------------------+
| System Instructions |
| User Prompt |
| Context |
+----------------------+
|
v
+----------------------+
| Tokenizer |
| text -> token IDs |
+----------------------+
|
v
+----------------------+
| LLM API |
+----------------------+
|
v
+----------------------+
| LLM Inference |
| Transformer |
| Attention |
| Neural Network |
+----------------------+
|
| predicts tokens
v
+----------------------+
| Generated Response |
+----------------------+
|
v
DevOps Engineer
Meanwhile:
Secrets -> protect API key
Logs -> record requests/errors
Metrics -> latency/token usage/errors
Docker -> package application
CI/CD -> deploy application
AWS/K8s -> run application
Monitoring -> observe application + LLM API
This is the mental model I want students to have.
OpenAI also describes text as being processed as tokens, not simply whole words, and provides tiktoken for programmatic tokenization. (OpenAI Platform)
PART 0 — What Are We Building?
"Today we are an AI Platform/DevOps team. A development team wants an AI assistant that helps troubleshoot Kubernetes incidents. Our responsibility is to build the service, understand how the LLM works, protect its credentials, observe its behavior, package it, and eventually deploy it."
The final application will work like this:
$ python app.py
========================================
DEVOPS LLM INCIDENT ASSISTANT
========================================
Describe your incident:
> Pod payment-service is in CrashLoopBackOff
Analyzing incident...
LLM RESPONSE
----------------------------------------
CrashLoopBackOff means Kubernetes is repeatedly
starting the container and the container is failing.
Start with:
1. kubectl get pods
2. kubectl describe pod payment-service
3. kubectl logs payment-service
4. kubectl logs payment-service --previous
Check:
- application errors
- missing environment variables
- Secrets
- database connectivity
- probes
- memory limits
----------------------------------------
Prompt tokens: 96
Response time: 2.31 seconds
Now the student immediately understands:
We're not building ChatGPT. We're building an application that uses an LLM as one component.
PART 1 — Create the Project
Open Terminal.
Go to your Projects directory:
cd ~/Projects
Create the lab:
mkdir devops-llm-lab
cd devops-llm-lab
Open it in VS Code:
code .
If code doesn't work, open VS Code manually:
File → Open Folder → devops-llm-lab
PART 2 — Create This Exact Structure
Inside VS Code create:
devops-llm-lab/
│
├── app.py
├── tokenizer_demo.py
├── llm_client.py
├── config.py
├── logger.py
│
├── prompts/
│ └── incident_prompt.txt
│
├── logs/
│
├── .env
├── .gitignore
├── requirements.txt
└── Dockerfile
Do not create random files.
Each file has a responsibility.
app.py
Application entry point
config.py
Configuration
llm_client.py
Communication with LLM
tokenizer_demo.py
Shows how text becomes tokens
logger.py
Observability
prompts/
Prompt management
.env
Secrets/configuration
requirements.txt
Python dependencies
Dockerfile
Containerization
DevOps reason
This is already a DevOps lesson.
We do separation of concerns.
You don't want:
500 lines inside app.py
containing:
API keys
prompts
logging
API calls
application logic
configuration
Production applications need maintainable structure.
PART 3 — Create Python Environment
In Terminal, make sure you're inside:
pwd
Expected something similar to:
/Users/yourname/Projects/devops-llm-lab
Create virtual environment:
python3 -m venv .venv
Activate it:
source .venv/bin/activate
Your terminal should change to:
(.venv) user@macbook devops-llm-lab %
Why?
A Python application may need:
openai 2.x
tiktoken
python-dotenv
Another application may require different versions.
The virtual environment isolates dependencies.
DevOps equivalent:
Python venv
↓
Docker container
↓
Kubernetes Pod
Same fundamental idea:
isolate the runtime environment.
PART 4 — Create requirements.txt
Open:
requirements.txt
Paste:
openai
python-dotenv
tiktoken
Save.
Then run:
pip install -r requirements.txt
Verify:
pip list
You should see packages including:
openai
python-dotenv
tiktoken
Why does DevOps care?
Dependencies are part of the application's software supply chain.
Later in CI/CD we can scan them for vulnerabilities.
For example:
Developer
↓
requirements.txt
↓
CI pipeline
↓
dependency scan
↓
Docker build
↓
container scan
↓
deployment
PART 5 — First Understand Tokens WITHOUT Calling an LLM
This is important.
Do not call OpenAI yet.
We first want to see what an LLM receives.
Open:
tokenizer_demo.py
Paste:
import tiktoken
text = """
The payment-service Kubernetes pod is in CrashLoopBackOff.
"""
encoding = tiktoken.get_encoding("o200k_base")
tokens = encoding.encode(text)
print("ORIGINAL TEXT")
print("----------------")
print(text)
print("TOKEN IDs")
print("----------------")
print(tokens)
print()
print("NUMBER OF TOKENS")
print("----------------")
print(len(tokens))
print()
print("DECODE EACH TOKEN")
print("----------------")
for token in tokens:
print(
token,
"->",
repr(encoding.decode([token]))
)
Run:
python tokenizer_demo.py
You will see output conceptually like:
ORIGINAL TEXT
----------------
The payment-service Kubernetes pod is in CrashLoopBackOff.
TOKEN IDs
----------------
[198, 976, 7522, ...]
NUMBER OF TOKENS
----------------
14
DECODE EACH TOKEN
----------------
198 -> '\n'
976 -> 'The'
...
Your exact token IDs/count can vary with encoding and text.
STOP THE LAB HERE AND EXPLAIN
Students usually think:
LLM reads:
"Kubernetes pod is failing."
It doesn't literally operate on that sentence as English words.
The simplified pipeline is:
"Kubernetes pod is failing"
↓
TOKENIZER
↓
["Kubernetes", " pod", " is", " failing"]
↓
[12345, 9382, 382, 12451]
↓
EMBEDDINGS
↓
vectors/numerical representations
↓
TRANSFORMER
↓
probabilities
↓
next token
This is extremely important.
PART 6 — What Does "Generate" Actually Mean?
Ask the students:
Suppose the prompt is:
Kubernetes is a container
The LLM might calculate probabilities somewhat conceptually like:
next token:
orchestration 0.61
platform 0.23
system 0.09
technology 0.04
banana 0.00001
It chooses a token.
Then:
Kubernetes is a container orchestration
Now it predicts again:
platform 0.69
system 0.18
tool 0.08
Then again.
And again.
So:
Prompt
↓
Tokens
↓
Neural network
↓
Probability distribution
↓
Next token
↓
Probability distribution
↓
Next token
↓
...
That process is called:
Inference
Why does a DevOps engineer need to know this?
Because inference consumes:
CPU/GPU
memory
network
time
money
Therefore AI infrastructure engineers care about:
latency
throughput
tokens/sec
input tokens
output tokens
GPU utilization
API errors
rate limits
cost
Now monitoring an LLM makes sense.
PART 7 — Understand Embeddings Conceptually
Before the transformer works with tokens, numerical representations are used.
Simplified:
"Kubernetes"
↓
token ID
↓
vector
[
0.123,
-0.733,
0.091,
...
]
A vector contains many dimensions.
Conceptually:
Kubernetes
↓
[0.21, 0.91, -0.32, ...]
Docker
↓
[0.24, 0.87, -0.29, ...]
banana
↓
[-0.81, 0.12, 0.76, ...]
"Kubernetes" and "Docker" may be semantically more related than "Kubernetes" and "banana".
This becomes extremely useful later for:
RAG
semantic search
vector databases
document retrieval
incident similarity
knowledge bases
We'll make that a separate advanced lab because it deserves its own experiment.
PART 8 — Transformer and Attention
Now draw this on your board.
Input
"The Kubernetes pod cannot connect to database"
↓
TOKENS
↓
EMBEDDINGS
↓
+-----------------------+
| TRANSFORMER |
| |
| Self Attention |
| Feed Forward |
| Normalization |
| Multiple Layers |
+-----------------------+
↓
Next-token scores
↓
TOKEN
↓
OUTPUT
The important component is:
Attention
Consider:
The Kubernetes pod cannot connect to the database
because its password secret is incorrect.
When processing:
incorrect
the model can pay different levels of attention to:
Kubernetes
pod
database
password
secret
This helps it understand relationships in context.
You don't need your DevOps students calculating attention matrices in Lab 1.
They need to understand:
Attention allows the model to use relationships among tokens in its context.
PART 9 — Context Window
This part is essential for AI DevOps.
An LLM request conceptually contains:
+--------------------------------------+
| CONTEXT |
| |
| System instructions |
| Previous conversation |
| Retrieved documentation |
| User question |
| Tool results |
| |
+--------------------------------------+
That all consumes tokens.
This explains why:
MORE CONTEXT
↓
MORE TOKENS
↓
MORE PROCESSING
↓
MORE LATENCY / COST
And why you cannot simply dump:
2 GB of CloudWatch logs
into every request.
Later we solve this with:
filtering
chunking
embeddings
retrieval
RAG
summarization
Now students understand why RAG exists before they ever install a vector database.
PART 10 — Add the API Key
Create:
.env
Paste:
OPENAI_API_KEY=PASTE_YOUR_KEY_HERE
OPENAI_MODEL=gpt-5
Replace:
PASTE_YOUR_KEY_HERE
with your key.
Do not put quotes around it.
PART 11 — Protect the Secret
Create:
.gitignore
Paste:
.env
.venv/
__pycache__/
logs/
*.pyc
Run:
git init
Then:
git status
You should not see .env as a file that Git intends to commit.
Stop here.
Ask the students:
Why aren't we putting the API key into
app.py?
Because:
BAD
app.py
OPENAI_API_KEY="sk-xxxxxxxx"
If pushed:
GitHub
↓
secret exposed
↓
someone uses API
↓
financial/security incident
Local environment:
.env
CI/CD:
GitHub Secrets
Jenkins Credentials
AWS:
AWS Secrets Manager
Kubernetes:
External Secrets Operator
Secrets
This is a DevOps responsibility.
PART 12 — Centralize Configuration
Open:
config.py
Paste:
import os
from dotenv import load_dotenv
load_dotenv()
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-5")
if not OPENAI_API_KEY:
raise ValueError(
"OPENAI_API_KEY is missing. "
"Add it to the .env file."
)
Why separate config?
Bad architecture:
api_key = "..."
model = "..."
timeout = "..."
region = "..."
scattered across 20 files.
Better:
Environment
↓
config.py
↓
Application
This follows twelve-factor application ideas commonly used in DevOps.
PART 13 — Create the System Prompt
Go to:
prompts/incident_prompt.txt
Paste:
You are a Senior DevOps and Site Reliability Engineer.
Your responsibility is to help troubleshoot production infrastructure incidents.
When analyzing an incident:
1. Explain what the error means.
2. List the most likely causes.
3. Provide diagnostic commands.
4. Explain what each command verifies.
5. Recommend the safest remediation.
6. Never recommend destructive production actions without warning.
7. Clearly separate investigation from remediation.
Focus on:
- Kubernetes
- Docker
- AWS
- Linux
- Terraform
- CI/CD
- Networking
- Observability
Why put prompts in files?
Because prompts become application artifacts.
In production you may need:
Prompt v1
Prompt v2
Prompt v3
You want:
Git history
pull requests
testing
review
rollback
Think of prompt management like:
Helm values
Terraform modules
application configuration
We shouldn't randomly change production prompts.
PART 14 — Create Logging
Open:
logger.py
Paste:
import logging
import os
os.makedirs("logs", exist_ok=True)
logging.basicConfig(
level=logging.INFO,
format=(
"%(asctime)s | "
"%(levelname)s | "
"%(message)s"
),
handlers=[
logging.FileHandler("logs/llm-app.log"),
logging.StreamHandler()
]
)
logger = logging.getLogger("devops-llm")
Why?
Because when something fails in production:
"It doesn't work"
is useless.
We need:
timestamp
request
model
latency
status
error
token usage
Same reason we log:
Kubernetes applications
ECS tasks
Lambda functions
Jenkins pipelines
AI applications need observability too.
PART 15 — Create the LLM Client
Open:
llm_client.py
Paste:
import time
from openai import OpenAI
from config import OPENAI_API_KEY, OPENAI_MODEL
from logger import logger
client = OpenAI(
api_key=OPENAI_API_KEY
)
def load_system_prompt():
with open(
"prompts/incident_prompt.txt",
"r"
) as file:
return file.read()
def analyze_incident(user_input):
system_prompt = load_system_prompt()
logger.info(
"Sending request to LLM model=%s",
OPENAI_MODEL
)
start_time = time.time()
try:
response = client.responses.create(
model=OPENAI_MODEL,
instructions=system_prompt,
input=user_input
)
latency = time.time() - start_time
logger.info(
"LLM request successful latency=%.2fs",
latency
)
return {
"answer": response.output_text,
"latency": latency
}
except Exception as error:
logger.exception(
"LLM request failed: %s",
error
)
raise
STOP AGAIN
Show the architecture.
The application now has:
config.py
|
| API key/model
v
User → app.py → llm_client.py → OpenAI API
^
|
incident_prompt.txt
|
v
logger.py
|
v
logs/llm-app.log
This is much closer to a real application.
PART 16 — Create the Main Application
Open:
app.py
Paste:
import tiktoken
from llm_client import analyze_incident
def count_tokens(text):
encoding = tiktoken.get_encoding(
"o200k_base"
)
return len(
encoding.encode(text)
)
print()
print("=" * 55)
print("DEVOPS LLM INCIDENT ASSISTANT")
print("=" * 55)
print()
incident = input(
"Describe your incident:\n\n> "
)
print()
print("Analyzing incident...")
print()
input_tokens = count_tokens(incident)
result = analyze_incident(incident)
print("=" * 55)
print("LLM RESPONSE")
print("=" * 55)
print(result["answer"])
print()
print("=" * 55)
print("OBSERVABILITY")
print("=" * 55)
print(
f"Input tokens: {input_tokens}"
)
print(
f"Response latency: "
f"{result['latency']:.2f} seconds"
)
Save everything.
PART 17 — Run the Real Application
Terminal:
python app.py
You should see:
=======================================================
DEVOPS LLM INCIDENT ASSISTANT
=======================================================
Describe your incident:
>
Paste:
The payment-service pod is in CrashLoopBackOff after a deployment.
Press Enter.
Expected structure:
Analyzing incident...
=======================================================
LLM RESPONSE
=======================================================
CrashLoopBackOff indicates that the container
starts, exits, and Kubernetes repeatedly attempts
to restart it.
Investigation:
1. Check pod state
kubectl get pods
2. Inspect Kubernetes events
kubectl describe pod payment-service
3. Check application logs
kubectl logs payment-service
4. Check previous container logs
kubectl logs payment-service --previous
...
=======================================================
OBSERVABILITY
=======================================================
Input tokens: 15
Response latency: 2.63 seconds
The exact answer, token count, and latency will vary.
That variation itself is part of teaching LLM behavior.
PART 18 — Examine the Logs
Run:
cat logs/llm-app.log
You should see something like:
2026-08-07 09:30:21 | INFO | Sending request to LLM model=gpt-5
2026-08-07 09:30:23 | INFO | LLM request successful latency=2.31s
Now connect it to production.
Locally:
Python logging
↓
llm-app.log
ECS:
Container
↓
stdout
↓
CloudWatch Logs
EKS:
Pod
↓
stdout
↓
Fluent Bit
↓
CloudWatch / Loki
PART 19 — Deliberately Break the Application
This is the DevOps part.
Working applications teach less than broken applications.
Change .env:
OPENAI_API_KEY=wrong-key
Run:
python app.py
Ask:
Pod is CrashLoopBackOff
You should receive an authentication/API error.
Then:
cat logs/llm-app.log
Notice:
ERROR
LLM request failed
Students learn:
User says:
"AI isn't working"
DevOps investigation:
Application running?
↓
Configuration correct?
↓
Secret correct?
↓
DNS working?
↓
Network working?
↓
API reachable?
↓
Authentication valid?
↓
Rate limited?
↓
Provider available?
↓
Model available?
This is AI operations.
PART 20 — Understand the COMPLETE REQUEST
Now students can understand what happened when they typed:
payment-service is CrashLoopBackOff
Layer 1 — User Input
"payment-service is CrashLoopBackOff"
Layer 2 — Application
Python receives:
incident
Layer 3 — Prompt
Application adds instructions:
You are a Senior DevOps Engineer...
plus:
payment-service is CrashLoopBackOff
Layer 4 — Tokenization
text
↓
tokens
↓
token IDs
Layer 5 — API request
Python
↓
HTTPS
↓
OpenAI API
Layer 6 — Inference infrastructure
Conceptually:
tokens
↓
embeddings
↓
transformer layers
↓
attention
↓
neural network computations
↓
probability distribution
↓
next token
repeated many times.
Layer 7 — Response
tokens
↓
text
↓
HTTP response
Layer 8 — Application
response.output_text
Layer 9 — User
Terminal prints answer.
PART 21 — Containerize It
Now we're doing actual DevOps.
Create:
Dockerfile
Paste:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install \
--no-cache-dir \
-r requirements.txt
COPY . .
CMD ["python", "app.py"]
Build:
docker build -t devops-llm:v1 .
Verify:
docker images
Expected:
REPOSITORY TAG
devops-llm v1
PART 22 — NEVER COPY .env INTO THE IMAGE
We need one more file.
Create:
.dockerignore
Paste:
.env
.venv
.git
logs
__pycache__
Why?
Without this you risk:
.env
↓
docker build
↓
Docker image layer
↓
ECR
↓
credential exposure
This is extremely important.
PART 23 — Run the Container
Run:
docker run \
--rm \
-it \
--env-file .env \
devops-llm:v1
Notice what we did.
We did not store credentials in the container.
We injected configuration at runtime:
Docker Image
+
Environment Secret
↓
Running Container
That same design maps directly to:
ECS Task Definition
+
AWS Secrets Manager
or:
Kubernetes Deployment
+
External Secrets
PART 24 — The DevOps Engineer's LLM Architecture
Now students should draw this themselves:
USER
|
v
+------------------+
| AI Application |
+------------------+
|
+--------------+--------------+
| |
v v
System Prompt User Input
| |
+--------------+--------------+
|
v
Tokenization
|
v
Context Window
|
v
LLM API
|
v
+-------------------+
| Model Inference |
| |
| Embeddings |
| Attention |
| Transformer |
| Neural Network |
+-------------------+
|
v
Output Tokens
|
v
Response
|
v
AI Application
|
v
USER
DEVOPS LAYER
====================================================
GitHub
|
CI/CD
|
Docker
|
ECR
|
ECS / EKS
|
Load Balancer
|
Route 53
|
HTTPS
Security:
Secrets Manager
IAM
Network policies
Observability:
CloudWatch
Prometheus
Grafana
Logs
Alerts
LLM Metrics:
Latency
Requests
Errors
Input Tokens
Output Tokens
Cost
Rate Limits
This is why a DevOps engineer needs LLM knowledge
We're not learning LLMs so a DevOps engineer can ask ChatGPT questions.
We're learning them because companies are putting LLMs inside production systems.
The infrastructure team now has to operate something like:
Application
|
+---- PostgreSQL
|
+---- Redis
|
+---- Kafka
|
+---- OpenAI/Claude/etc.
|
+---- Vector Database
|
+---- Agent tools
|
+---- internal APIs
A traditional DevOps engineer monitors:
CPU
memory
disk
network
5xx
latency
pods
An AI DevOps/Platform engineer additionally cares about:
input tokens
output tokens
time to first token
tokens per second
LLM latency
API rate limits
model failures
prompt versions
model versions
hallucination/evaluation failures
RAG retrieval quality
vector DB health
embedding jobs
AI cost
GPU utilization
That is the bridge between LLM engineering and DevOps.
What We Intentionally Did NOT Add Yet
Do not throw LangChain, ChromaDB, Pinecone, Redis, agents and Kubernetes into this first lab.
Your students first need this mental model:
LLM APPLICATION
Prompt
↓
Tokens
↓
Context
↓
Model
↓
Inference
↓
Tokens
↓
Response
APPLICATION LAYER
Python
↓
SDK
↓
REST API
↓
LLM Provider
DEVOPS LAYER
Secrets
Logging
Monitoring
Docker
CI/CD
Deployment
Security
Cost
Reliability
Only after they completely understand this should Lab 2 introduce embeddings, vector similarity and RAG.
The course sequence I recommend from here
| Lab | Build | What students actually learn |
|---|---|---|
| 1 — LLM Fundamentals | The application above | Tokens, inference, transformer, context, API, secrets, logging, Docker |
| 2 — Embeddings | DevOps incident similarity search | Embeddings, vectors, cosine similarity |
| 3 — RAG | Ask questions against Kubernetes/AWS documentation | Chunking, embedding, retrieval, vector DB, grounding |
| 4 — LLM Memory | Multi-turn DevOps assistant | Stateless vs stateful AI, conversation context, Redis/Postgres |
| 5 — Tool Calling | AI executes safe read-only kubectl/AWS diagnostics |
Agents, tools, structured outputs |
| 6 — MCP/Integrations | Connect AI with operational systems | AI-to-tool integration architecture |
| 7 — LLM Evaluation | Test answers automatically | Hallucinations, quality gates, regression testing |
| 8 — AI Observability | Prometheus/Grafana AI dashboard | tokens, cost, latency, failures, traces |
| 9 — AI Security | Attack the assistant | prompt injection, secret leakage, permissions, guardrails |
| 10 — Production AI Platform | EKS + Terraform + CI/CD + RAG + monitoring | Complete AI DevOps architecture |
This Lab 1 should be the foundation. Lab 2 should not simply "use embeddings"; we should actually print vectors, calculate similarity ourselves, compare Kubernetes/Docker/banana sentences mathematically, then build a small DevOps knowledge search engine. That will make embeddings → vector DB → RAG feel obvious instead of magical.
Top comments (0)