DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

JumpToTech DevOps + AI Bootcamp Hands-On Lab: Kubernetes Monitoring, Grafana Persistence, Gmail SMTP, and Production Alerts

#aws #devops #kubernetes #monitoring

Environment: AWS EKS

Namespace: dev

Tools: Kubernetes, Helm, Prometheus, Grafana, AWS EBS CSI Driver, Gmail SMTP

Estimated duration: 2–3 hours

Difficulty: Intermediate

Learning objectives

By the end of this lab, students will be able to:

  1. Deploy Prometheus and Grafana into Kubernetes.
  2. Configure persistent storage for Grafana using AWS EBS.
  3. Understand the difference between emptyDir and PersistentVolumeClaims.
  4. Configure Gmail SMTP securely using Kubernetes Secrets.
  5. Create Grafana email contact points and notification policies.
  6. Create an ImagePullBackOff alert using PromQL.
  7. Simulate a Kubernetes incident and receive an email notification.
  8. Verify Grafana data survives a Pod restart.

Important architecture decision

Students must use their own development EKS cluster or an instructor-approved isolated lab environment.

Do not modify the existing production monitoring release.

The namespace dev isolates Kubernetes objects, but it does not provide complete security or resource isolation. Students must use their own credentials and must not share passwords.

Part 1 — Verify the Kubernetes environment

Step 1: Verify AWS identity 

aws sts get-caller-identity
Enter fullscreen mode Exit fullscreen mode

This confirms which AWS account and IAM identity the CLI is using.

Step 2: Connect to EKS

Replace the cluster name and region with your assigned development cluster.

export AWS_REGION="us-east-1"
export EKS_CLUSTER="YOUR-DEV-CLUSTER"

aws eks update-kubeconfig \
  --region "$AWS_REGION" \
  --name "$EKS_CLUSTER"
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl config current-context
kubectl get nodes -o wide
kubectl cluster-info
Enter fullscreen mode Exit fullscreen mode

Expected result: At least one worker node is Ready.

Step 3: Create the dev namespace 

kubectl create namespace dev \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl get namespaces
Enter fullscreen mode Exit fullscreen mode

All application and monitoring resources in this lab will be deployed into dev.

Part 2 — Verify AWS EBS storage

AWS EBS provides persistent block storage for Kubernetes workloads.

Step 1: Check the EBS CSI driver 

aws eks describe-addon \
  --cluster-name "$EKS_CLUSTER" \
  --addon-name aws-ebs-csi-driver \
  --region "$AWS_REGION" \
  --query 'addon.{status:status,version:addonVersion,issues:health.issues}'
Enter fullscreen mode Exit fullscreen mode

Expected:

ACTIVE
Enter fullscreen mode Exit fullscreen mode

Also check:

kubectl get pods -n kube-system | grep ebs-csi
Enter fullscreen mode Exit fullscreen mode

If the add-on is missing, stop and ask the instructor to install it with the correct IAM permissions. Do not create an EBS CSI IAM role or install the add-on independently in a shared cluster.

Step 2: Check StorageClasses 

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

Look for a class provisioned by:

ebs.csi.aws.com
Enter fullscreen mode Exit fullscreen mode

For this lab, we will create a dedicated gp3 StorageClass if it does not already exist.

cat > dev-gp3-storageclass.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dev-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f dev-gp3-storageclass.yaml
kubectl get storageclass dev-gp3
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • gp3 is the AWS EBS volume type.
  • encrypted: true enables EBS encryption.
  • Delete allows Kubernetes to delete the EBS volume when the PVC is deleted, subject to the provisioner and cloud cleanup succeeding.
  • WaitForFirstConsumer helps schedule the Pod and EBS volume in the same availability zone.

StorageClasses are cluster-scoped, so this step requires instructor approval in shared environments.

Part 3 — Install Prometheus and Grafana in dev

Step 1: Install Helm

Check first:

helm version
Enter fullscreen mode Exit fullscreen mode

If Helm is not installed, use the approved installation method for your operating system.

Step 2: Add the Helm repository 

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts

helm repo update
Enter fullscreen mode Exit fullscreen mode

Step 3: Create a working directory 

mkdir -p ~/jumptotech-monitoring-lab
cd ~/jumptotech-monitoring-lab
Enter fullscreen mode Exit fullscreen mode

Step 4: Create monitoring values 

cat > monitoring-values.yaml <<'EOF'
alertmanager:
  enabled: false

grafana:
  persistence:
    enabled: true
    type: pvc
    storageClassName: dev-gp3
    accessModes:
      - ReadWriteOnce
    size: 5Gi

  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 300m
      memory: 512Mi

prometheus:
  prometheusSpec:
    retention: 6h
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 500m
        memory: 1Gi
EOF
Enter fullscreen mode Exit fullscreen mode

We are enabling persistent storage from the beginning, so students do not need to migrate an existing Grafana database.

Step 5: Install the monitoring stack 

helm upgrade --install dev-monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 92.7.0 \
  --namespace dev \
  -f monitoring-values.yaml \
  --wait \
  --timeout 15m
Enter fullscreen mode Exit fullscreen mode

Verify:

helm list -n dev

kubectl get pods -n dev

kubectl get deployments -n dev

kubectl get pvc -n dev
Enter fullscreen mode Exit fullscreen mode

Expected:

  • Grafana Pod running
  • Prometheus Pod running
  • Grafana PVC bound to EBS
  • Helm release deployed

If the installation fails, inspect Pod events and resource capacity before retrying.

Part 4 — Verify Grafana persistent storage

Step 1: Find the Grafana Deployment 

kubectl get deployment -n dev \
  -l app.kubernetes.io/name=grafana
Enter fullscreen mode Exit fullscreen mode

Expected Deployment name:

dev-monitoring-grafana
Enter fullscreen mode Exit fullscreen mode

Step 2: Inspect the storage volume 

kubectl get deployment dev-monitoring-grafana \
  -n dev \
  -o jsonpath='{.spec.template.spec.volumes}'
echo
Enter fullscreen mode Exit fullscreen mode

Look for a persistentVolumeClaim storage volume.

Step 3: Verify the PVC 

kubectl get pvc -n dev
kubectl describe pvc -n dev
Enter fullscreen mode Exit fullscreen mode

The Grafana claim should be Bound.

Discussion question: What would happen to Grafana dashboards if the application used emptyDir instead of a PVC?

Part 5 — Access Grafana

Step 1: Retrieve the Grafana admin password

Do not paste the password into chat or a shared document.

kubectl get secret dev-monitoring-grafana \
  -n dev \
  -o jsonpath='{.data.admin-password}' \
  | base64 --decode
echo
Enter fullscreen mode Exit fullscreen mode

Step 2: Start port-forwarding 

kubectl port-forward \
  -n dev \
  svc/dev-monitoring-grafana \
  3000:80
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:3000
Enter fullscreen mode Exit fullscreen mode

If Kubernetes is managed from a remote EC2 machine, use an SSH tunnel to forward port 3000 to your own computer. Do not expose Grafana publicly just to complete this lab.

Login:

Username: admin
Password: password retrieved from Kubernetes Secret
Enter fullscreen mode Exit fullscreen mode

Step 3: Verify Grafana health

From a second terminal, run:

curl -sS http://localhost:3000/api/health
Enter fullscreen mode Exit fullscreen mode

Expected:

{
  "database": "ok"
}
Enter fullscreen mode Exit fullscreen mode

Additional fields may be returned.

Part 6 — Configure Gmail SMTP

Step 1: Create a Google app password

Students must use their own Gmail account.

Open:

https://myaccount.google.com/apppasswords

Requirements:

  • Google 2-Step Verification enabled
  • A new app password created for the Grafana lab

Never use a normal Gmail login password.

Step 2: Create a Kubernetes Secret

Run:

read -r -p "Enter Gmail address: " GMAIL_ADDRESS

read -r -s -p "Enter Gmail app password: " GMAIL_APP_PASSWORD
echo

kubectl create secret generic grafana-smtp \
  -n dev \
  --from-literal=username="$GMAIL_ADDRESS" \
  --from-literal=password="$GMAIL_APP_PASSWORD"

unset GMAIL_APP_PASSWORD
Enter fullscreen mode Exit fullscreen mode

Verify without revealing credentials:

kubectl get secret grafana-smtp -n dev
Enter fullscreen mode Exit fullscreen mode

Expected:

grafana-smtp   Opaque   2
Enter fullscreen mode Exit fullscreen mode

Step 3: Create SMTP Helm values 

cat > grafana-smtp-values.yaml <<'EOF'
grafana:
  env:
    GF_SMTP_ENABLED: "true"
    GF_SMTP_HOST: "smtp.gmail.com:587"
    GF_SMTP_STARTTLS_POLICY: "MandatoryStartTLS"
    GF_SMTP_FROM_NAME: "JumpToTech Dev Alerts"
  envValueFrom:
    GF_SMTP_USER:
      secretKeyRef:
        name: grafana-smtp
        key: username
    GF_SMTP_PASSWORD:
      secretKeyRef:
        name: grafana-smtp
        key: password
    GF_SMTP_FROM_ADDRESS:
      secretKeyRef:
        name: grafana-smtp
        key: username
EOF
Enter fullscreen mode Exit fullscreen mode

This uses the same Gmail address for authentication and the email sender.

Step 4: Preview the Helm deployment 

helm template dev-monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 92.7.0 \
  --namespace dev \
  -f monitoring-values.yaml \
  -f grafana-smtp-values.yaml \
  > monitoring-preview.yaml
Enter fullscreen mode Exit fullscreen mode

Inspect the Grafana SMTP environment variables and Secret references in the rendered Deployment.

Step 5: Upgrade Helm 

helm upgrade dev-monitoring \
  prometheus-community/kube-prometheus-stack \
  --version 92.7.0 \
  --namespace dev \
  -f monitoring-values.yaml \
  -f grafana-smtp-values.yaml \
  --wait \
  --timeout 15m
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl get pods -n dev \
  -l app.kubernetes.io/name=grafana

kubectl get deployment dev-monitoring-grafana \
  -n dev \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="grafana")].env[*]}{.name}{"\n"}{end}' \
  | grep GF_SMTP
Enter fullscreen mode Exit fullscreen mode

Students should see seven SMTP environment variable names.

Part 7 — Configure email notifications in Grafana

Open Grafana in the browser.

Step 1: Create a contact point

Navigate to:

Alerting → Notification configuration → Contact points

Click New contact point.

Enter:

  • Name: Student-Dev-Email
  • Integration: Email
  • Address: your own Gmail address

Save.

Step 2: Send a test notification

Click Test → Send test notification.

Expected:

Test notification sent successfully
Enter fullscreen mode Exit fullscreen mode

Check Gmail inbox and Spam.

The test may arrive with a subject containing FIRING. This is a simulated alert and does not necessarily indicate a real incident.

Step 3: Configure the notification policy

Navigate to:

Alerting → Notification configuration → Notification policies

For an isolated student Grafana installation:

  1. Open the Default policy menu.
  2. Select Edit.
  3. Change the default contact point to Student-Dev-Email.
  4. Keep grouping and timing settings unchanged.
  5. Click Update policy.

Verify:

Delivered to Student-Dev-Email
Enter fullscreen mode Exit fullscreen mode

Do not change a shared organization's default notification policy.

Part 8 — Create a real ImagePullBackOff incident

We will deliberately deploy an application with a nonexistent container image.

Use only the dev namespace.

Step 1: Create the broken Deployment 

cat > broken-app.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-restaurant
  namespace: dev
  labels:
    app: broken-restaurant
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-restaurant
  template:
    metadata:
      labels:
        app: broken-restaurant
    spec:
      containers:
        - name: restaurant
          image: nginx:jumptotech-image-does-not-exist-2026
          imagePullPolicy: Always
          ports:
            - containerPort: 80
EOF
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f broken-app.yaml
Enter fullscreen mode Exit fullscreen mode

Step 2: Observe the failure 

kubectl get pods -n dev

kubectl describe deployment broken-restaurant -n dev

kubectl get events -n dev \
  --sort-by=.lastTimestamp \
  | tail -20
Enter fullscreen mode Exit fullscreen mode

The Pod should eventually show:

ImagePullBackOff
Enter fullscreen mode Exit fullscreen mode

Step 3: Verify the Prometheus metric

In Grafana, open Explore, select the Prometheus data source, and enter:

kube_pod_container_status_waiting_reason{
  namespace="dev",
  reason="ImagePullBackOff"
}
Enter fullscreen mode Exit fullscreen mode

Run the query in Instant mode.

Expected: a matching series with value 1 while the image-pull failure is present.

If no series appears, verify that kube-state-metrics is running and that Prometheus is scraping it before continuing.

Part 9 — Create the ImagePullBackOff alert

Navigate to:

Alerting → Alert rules → New alert rule

Step 1: Alert name 

DEV - Kubernetes ImagePullBackOff
Enter fullscreen mode Exit fullscreen mode

Step 2: Prometheus query A 

kube_pod_container_status_waiting_reason{
  namespace="dev",
  reason="ImagePullBackOff"
}
Enter fullscreen mode Exit fullscreen mode

Use an Instant query.

Step 3: Alert condition

Configure Grafana expressions:

  • Query A: Prometheus metric
  • Expression B: Reduce, Last of A
  • Expression C: Threshold, B is above 0.5
  • Set C as the alert condition

This evaluates each matching container series and fires when its value is 1.

Step 4: Evaluation

  • Evaluation interval: 1 minute
  • Pending period: 2 minutes
  • Keep firing for: 0 minutes

Use a dedicated folder and evaluation group, such as:

Folder: JumpToTech-Dev
Evaluation group: dev-kubernetes-alerts
Enter fullscreen mode Exit fullscreen mode

Step 5: Notification routing

Use the Grafana notification policy configured earlier:

Default policy → Student-Dev-Email
Enter fullscreen mode Exit fullscreen mode

Step 6: Alert message

Summary:

ImagePullBackOff detected in dev namespace
Enter fullscreen mode Exit fullscreen mode

Description:

Kubernetes cannot pull a container image.
Check the Pod image, image registry, credentials, and Kubernetes events.
Enter fullscreen mode Exit fullscreen mode

Step 7: Save the alert

Wait for the alert to evaluate.

Expected progression:

Normal → Pending → Firing
Enter fullscreen mode Exit fullscreen mode

The email notification may arrive later because Grafana's notification policy also has grouping and delivery delays.

Part 10 — Fix the incident

Once the alert is firing, students must troubleshoot and resolve the problem.

Step 1: Inspect the Pod 

kubectl get pods -n dev

kubectl describe deployment broken-restaurant -n dev

kubectl get events -n dev \
  --sort-by=.lastTimestamp \
  | tail -20
Enter fullscreen mode Exit fullscreen mode

Step 2: Fix the container image 

kubectl set image deployment/broken-restaurant \
  restaurant=nginx:1.27 \
  -n dev
Enter fullscreen mode Exit fullscreen mode

Step 3: Verify recovery 

kubectl rollout status deployment/broken-restaurant \
  -n dev \
  --timeout=180s

kubectl get pods -n dev
Enter fullscreen mode Exit fullscreen mode

Expected:

Running
1/1
Enter fullscreen mode Exit fullscreen mode

Step 4: Verify alert recovery

Return to Grafana.

The alert should transition back toward Normal after Prometheus and Grafana observe that the image-pull problem has cleared.

Students should also check whether a resolved email notification arrives.

Part 11 — Test Grafana persistence

Students must prove that Grafana can retain data after Pod replacement.

Step 1: Create a test dashboard

In Grafana:

  1. Create a new dashboard.
  2. Name it JumpToTech Dev Persistence Test.
  3. Add a simple Prometheus panel.
  4. Save the dashboard.

Step 2: Restart Grafana 

kubectl rollout restart deployment/dev-monitoring-grafana \
  -n dev

kubectl rollout status deployment/dev-monitoring-grafana \
  -n dev \
  --timeout=180s
Enter fullscreen mode Exit fullscreen mode

Step 3: Verify the PVC 

kubectl get pvc -n dev
kubectl get pods -n dev \
  -l app.kubernetes.io/name=grafana
Enter fullscreen mode Exit fullscreen mode

Step 4: Check Grafana

Log back into Grafana and verify:

  • The test dashboard still exists.
  • The email contact point still exists.
  • The alert rule still exists.
  • The notification policy is unchanged.

Expected result: The Grafana database and its saved objects survive the Pod restart.

Part 12 — Troubleshooting

Problem What to investigate
Grafana PVC Pending EBS CSI driver, IAM permissions, scheduling, storage class
Grafana Pod Pending Node capacity, volume zone, Pod events
Grafana HTTP 401 Authentication or API permissions
Gmail 535 error Incorrect or revoked Gmail app password
SMTP timeout Outbound connectivity, port 587, security controls
No email received Contact point, policy routing, SMTP logs, Gmail Spam
No Prometheus metric kube-state-metrics, scrape targets, namespace labels
Alert stays Normal Query results, threshold, evaluation group
Dashboard disappears PVC mount, storage configuration, database errors

Useful troubleshooting commands:

kubectl get pods -n dev

kubectl get pvc -n dev

kubectl get events -n dev \
  --sort-by=.lastTimestamp \
  | tail -30

kubectl logs -n dev \
  deployment/dev-monitoring-grafana \
  -c grafana \
  --tail=100

helm status dev-monitoring -n dev
Enter fullscreen mode Exit fullscreen mode

Never paste credentials or decoded Kubernetes Secret values into reports.

Part 13 — Cleanup and cost control

After the instructor approves cleanup, delete the deliberately broken application:

kubectl delete deployment broken-restaurant \
  -n dev \
  --ignore-not-found
Enter fullscreen mode Exit fullscreen mode

If this is a disposable, dedicated student cluster, the instructor may authorize removal of the monitoring release:

helm uninstall dev-monitoring -n dev
Enter fullscreen mode Exit fullscreen mode

Helm uninstall does not guarantee deletion of all PVCs or cloud volumes.

Before deleting any PVC, identify its exact PV and EBS volume, verify that it belongs to this lab, and confirm the data is no longer needed.

Do not delete shared namespaces, StorageClasses, EBS CSI drivers, or production volumes.

AWS EBS volumes and EKS resources may incur charges while they exist.

Homework submission requirements

Students must submit a report containing:

  1. AWS EKS cluster and node verification.
  2. dev namespace creation.
  3. Helm monitoring installation output.
  4. Grafana PVC showing Bound.
  5. Grafana HTTP health result.
  6. SMTP environment variable names (no passwords).
  7. Grafana email contact point screenshot.
  8. Successful test notification screenshot.
  9. Broken Kubernetes Pod showing ImagePullBackOff.
  10. Prometheus query result.
  11. Grafana alert showing Firing.
  12. Gmail alert notification screenshot.
  13. Deployment recovery showing Running.
  14. Grafana dashboard persistence verification.

Student reflection questions

Question 1: Why do we use a PVC for Grafana instead of emptyDir?

Question 2: Why is a Kubernetes Secret better than putting an SMTP password directly into Helm values?

Question 3: What is the difference between ImagePullBackOff and CrashLoopBackOff?

Question 4: What does Prometheus do, and what does Grafana do?

Question 5: Why do we configure both an alert rule and a notification policy?

Question 6: Why can an alert be Firing before an email arrives?

Question 7: Why should monitoring alerts include namespace, Pod, and container labels?

Question 8: What are the limitations of storing Grafana's SQLite database on a single EBS volume?

Final expected architecture

Kubernetes application → kube-state-metrics → Prometheus → Grafana alert evaluation → Grafana notification policy → Gmail SMTP → Student email.

Completion criteria: The student must demonstrate a real Kubernetes image-pull failure, receive an alert, fix the Deployment, and verify recovery while Grafana's configuration persists across a Pod restart.

Top comments (0)