JumpToTech — Production Kubernetes Complete Lab

Level: Beginner → Intermediate
Platform: AWS EKS
Duration: ~3 hours
Language: English
Goal: Build a production-style Kubernetes deployment from scratch and troubleshoot it.

---

1. What We Will Build

By the end of the lab:

                         INTERNET
                            │
                            ▼
                           ALB
                            │
                            ▼
                         INGRESS
                            │
                            ▼
                         SERVICE
                       ClusterIP :80
                            │
                    selector: app=web
                            │
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
            POD 1          POD 2         POD 3
              │             │             │
          Container     Container     Container
              │             │             │
              └─────────────┼─────────────┘
                            ▲
                        ReplicaSet
                            ▲
                            │
                        Deployment
                            │
             ┌──────────────┼───────────────┐
             │              │               │
         ConfigMap        Secret      ServiceAccount

Additional protection:

Requests/Limits → Resource management
Readiness       → Traffic readiness
Liveness        → Container health
Startup Probe   → Startup protection
PDB             → Disruption availability
HPA             → Automatic scaling
NetworkPolicy   → Network restrictions
RBAC            → Kubernetes authorization

Everything will be deployed into:

jumptotech-prod

---

2. Project Structure

Create the project:

mkdir production-k8s-lab
cd production-k8s-lab

Our final directory:

production-k8s-lab/
│
├── 01-namespace.yaml
├── 02-configmap.yaml
├── 03-secret.yaml
├── 04-serviceaccount.yaml
├── 05-deployment.yaml
├── 06-service.yaml
├── 07-pdb.yaml
├── 08-hpa.yaml
├── 09-ingress.yaml
├── 10-networkpolicy.yaml
├── 11-role.yaml
└── 12-rolebinding.yaml

This numbering is intentional because students can apply resources in a logical order.

---

3. Verify the Cluster

Before touching YAML:

kubectl get nodes

Expected:

NAME                              STATUS
ip-172-31-10-20.ec2.internal     Ready
ip-172-31-20-30.ec2.internal     Ready

Explain:

A Kubernetes cluster contains a Control Plane and Worker Nodes. On EKS, AWS manages the control plane. Our application workloads normally run on worker nodes.

Check where existing Pods are running:

kubectl get pods -A -o wide

Architecture:

EKS
│
├── AWS-managed Control Plane
│
└── Worker Nodes
      │
      ├── kubelet
      ├── container runtime
      └── Pods

---

4. Namespace

Create:

nano 01-namespace.yaml

apiVersion: v1
kind: Namespace

metadata:
  name: jumptotech-prod

  labels:
    environment: production
    team: jumptotech

Apply:

kubectl apply -f 01-namespace.yaml

Verify:

kubectl get namespaces

What is a Namespace?

A Namespace creates a logical scope inside a cluster.

EKS Cluster
│
├── default
│
├── kube-system
│
├── dev
│
├── test
│
└── jumptotech-prod

It does not create another Kubernetes cluster.

It helps organize and scope resources, permissions, quotas, names, and policies.

For the rest of the lab:

kubectl config set-context --current --namespace=jumptotech-prod

Check:

kubectl config view --minify | grep namespace

Now our commands default to jumptotech-prod.

---

5. ConfigMap

Applications require configuration.

For example:

APP_NAME
APP_ENV
LOG_LEVEL

These should not necessarily be hard-coded into the Docker image.

Create:

nano 02-configmap.yaml

apiVersion: v1
kind: ConfigMap

metadata:
  name: web-config
  namespace: jumptotech-prod

data:
  APP_NAME: "JumpToTech"
  APP_ENV: "production"
  LOG_LEVEL: "info"

Apply:

kubectl apply -f 02-configmap.yaml

Check:

kubectl get configmap

Then:

kubectl describe configmap web-config

Explain:

Docker Image
    │
    ├── application code
    └── dependencies

ConfigMap
    │
    └── environment-specific
        non-sensitive configuration

---

6. Secret

Should this go into ConfigMap?

DB_PASSWORD=my-secret-password

No.

Kubernetes provides the Secret resource.

Create:

nano 03-secret.yaml

For our lab only:

apiVersion: v1
kind: Secret

metadata:
  name: web-secret
  namespace: jumptotech-prod

type: Opaque

stringData:
  DB_USERNAME: "admin"
  DB_PASSWORD: "student-demo-password"

Apply:

kubectl apply -f 03-secret.yaml

Check:

kubectl get secrets

Do not print real secrets during a production troubleshooting session.

Teach:

ConfigMap
    ↓
Non-sensitive configuration


Secret
    ↓
Sensitive configuration

Important production lesson:

Kubernetes Secret is a Kubernetes object designed for sensitive data, but simply putting a value in a Secret does not make every secret-management problem disappear.

And:

Base64 ≠ Encryption

For production GitOps, do not commit real plaintext passwords like this lab example.

---

7. ServiceAccount

Our application should have a Kubernetes workload identity.

Create:

nano 04-serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount

metadata:
  name: web-sa
  namespace: jumptotech-prod

Apply:

kubectl apply -f 04-serviceaccount.yaml

Check:

kubectl get serviceaccount

Teach:

ServiceAccount
      ↓
WHO AM I?

RBAC
      ↓
WHAT AM I ALLOWED TO DO?

A ServiceAccount does not automatically give the Pod administrator permissions.

---

8. Production Deployment

Now create the central object.

nano 05-deployment.yaml

Replace YOUR_ECR_IMAGE with your actual ECR image.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: web-deployment
  namespace: jumptotech-prod

  labels:
    app: web

spec:

  replicas: 3

  strategy:
    type: RollingUpdate

    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

  selector:
    matchLabels:
      app: web

  template:

    metadata:
      labels:
        app: web
        environment: production

    spec:

      serviceAccountName: web-sa

      terminationGracePeriodSeconds: 30

      containers:

        - name: web-container

          image: YOUR_ECR_IMAGE

          imagePullPolicy: IfNotPresent

          ports:
            - name: http
              containerPort: 80
              protocol: TCP

          envFrom:
            - configMapRef:
                name: web-config

          env:

            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: web-secret
                  key: DB_USERNAME

            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: web-secret
                  key: DB_PASSWORD

          resources:

            requests:
              cpu: "100m"
              memory: "128Mi"

            limits:
              cpu: "500m"
              memory: "256Mi"

          startupProbe:

            httpGet:
              path: /
              port: http

            periodSeconds: 5
            failureThreshold: 12

          readinessProbe:

            httpGet:
              path: /
              port: http

            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3

          livenessProbe:

            httpGet:
              path: /
              port: http

            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

Don't apply yet.

Explain the important pieces first.

---

9. Deployment → ReplicaSet → Pod

We said:

replicas: 3

This represents desired state.

Deployment
     │
     │ desired replicas = 3
     ▼
ReplicaSet
     │
     ├──────────┬──────────┐
     ▼          ▼          ▼
   Pod 1      Pod 2      Pod 3

Deployment primarily handles things such as:

Desired application state
Rollouts
Rollbacks
ReplicaSet management

ReplicaSet maintains the desired Pod replica count.

---

10. Labels and Selectors

Our Pod template has:

labels:
  app: web

Deployment selector:

selector:
  matchLabels:
    app: web

Later our Service will also use:

selector:
  app: web

Students should memorize the relationship, not the syntax:

              app=web

Deployment ────────────────┐
                           │
                    ┌──────▼──────┐
                    │    POD      │
                    │             │
                    │ app = web   │
                    └──────▲──────┘
                           │
Service ───────────────────┘
selector app=web

---

11. Requests

We have:

requests:
  cpu: "100m"
  memory: "128Mi"

When a Pod needs scheduling:

New Pod
   │
   │ requests
   │ CPU: 100m
   │ RAM: 128Mi
   ▼
Scheduler
   │
   ├── Node A
   ├── Node B
   └── Node C

The Scheduler uses requests when determining whether the Pod fits on a Node.

CPU:

1000m = 1 CPU
500m  = 0.5 CPU
250m  = 0.25 CPU
100m  = 0.1 CPU

---

12. Limits

We have:

limits:
  cpu: "500m"
  memory: "256Mi"

Simple classroom explanation:

REQUEST
   ↓
Scheduler capacity calculation


LIMIT
   ↓
Runtime resource ceiling

If CPU demand exceeds an enforced CPU limit:

CPU
 ↓
throttling

If memory usage exceeds an enforced memory limit:

Memory
  ↓
OOM
  ↓
process/container may be killed
  ↓
OOMKilled

---

13. Startup Probe

Now we have three probes.

First:

startupProbe:

Question:

Has my application successfully started?

This is especially useful for applications that can take time to initialize.

Container starts
       │
       ▼
 Startup Probe
       │
       ├── failing → still starting
       │
       └── succeeds
               │
               ▼
       Liveness/Readiness
       take over

Our configuration:

periodSeconds: 5
failureThreshold: 12

gives the application roughly up to 60 seconds of startup checking before startup failure reaches the configured threshold.

---

14. Readiness Probe

readinessProbe:
  httpGet:
    path: /
    port: http

Question:

Should this Pod receive traffic?

When you run:

kubectl get pods

you might see:

NAME                 READY   STATUS
web-abc123            1/1    Running

1/1 means:

1 container Ready
───────────────
1 readiness-counted container in this Pod

If:

0/1 Running

the Pod can still:

Exist             YES
Be Running        YES
Have an IP        YES

but:

Ready             NO

Therefore it should not be used as a normal ready Service backend.

---

15. Liveness Probe

Question:

Is this application still healthy enough to keep running?

Repeated failures:

Liveness Probe
      │
      ▼
    FAIL
      │
    FAIL
      │
    FAIL
      │
      ▼
failureThreshold reached
      │
      ▼
    kubelet
      │
      ▼
restart container

The important distinction:

READINESS
"Should I receive traffic?"


LIVENESS
"Should this container keep running?"


STARTUP
"Has the application successfully started?"

---

16. RollingUpdate Strategy

We added:

strategy:
  type: RollingUpdate

  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

Suppose:

Current version = v1
Desired version = v2

Kubernetes does not necessarily kill all v1 Pods simultaneously.

Conceptually:

v1   v1   v1

        ↓

v1   v1   v1   v2

        ↓

v1   v1   v2

        ↓

v1   v2   v2

        ↓

v2   v2   v2

maxUnavailable controls how many desired Pods may be unavailable during the update.

maxSurge controls how many extra Pods above the desired replica count may temporarily be created.

---

17. Apply Deployment

Now:

kubectl apply -f 05-deployment.yaml

Watch:

kubectl get pods -w

Then:

kubectl get deployment

kubectl get rs

kubectl get pods -o wide

Students should physically see:

Deployment
   ↓
ReplicaSet
   ↓
Pods

---

18. Inspect a Pod

Get a Pod name:

kubectl get pods

Then:

kubectl describe pod POD_NAME

Find:

Node
Labels
IP
Service Account

Container:
  Image
  Port

Requests
Limits

Startup
Readiness
Liveness

Events

This is a very important production troubleshooting command.

---

19. Verify ConfigMap

kubectl exec POD_NAME -- printenv APP_NAME

Expected:

JumpToTech

Then:

kubectl exec POD_NAME -- printenv APP_ENV

Expected:

production

We have now proved:

ConfigMap
    ↓
Deployment
    ↓
Pod
    ↓
Container
    ↓
Environment Variable

---

20. Service

Pods are disposable.

Today:

Pod A
10.0.1.10

Tomorrow:

Pod B
10.0.2.25

Applications should not rely on individual Pod IP addresses.

Create:

nano 06-service.yaml

apiVersion: v1
kind: Service

metadata:
  name: web-service
  namespace: jumptotech-prod

spec:

  type: ClusterIP

  selector:
    app: web

  ports:

    - name: http
      protocol: TCP
      port: 80
      targetPort: http

Apply:

kubectl apply -f 06-service.yaml

Check:

kubectl get svc

---

21. Understand port vs targetPort

We have:

port: 80
targetPort: http

Our named container port is:

ports:
  - name: http
    containerPort: 80

Conceptually:

Client
   │
   ▼
Service
port 80
   │
   ▼
targetPort http
   │
   ▼
Pod
containerPort 80

---

22. EndpointSlice

Run:

kubectl get pods -o wide

Look at the Pod IPs.

Then:

kubectl get endpointslice \
-l kubernetes.io/service-name=web-service

For more detail:

kubectl get endpointslice \
-l kubernetes.io/service-name=web-service \
-o yaml

Architecture:

Service
   │
selector app=web
   │
   ▼
matching Pods
   │
   ▼
EndpointSlice

This is one of the most useful troubleshooting relationships:

SERVICE HAS NO EXPECTED BACKENDS?

Check:

Service selector
       ↓
Pod labels
       ↓
Pod readiness
       ↓
EndpointSlice

---

23. Test Service Before ALB

Run:

kubectl port-forward service/web-service 8080:80

Open:

http://localhost:8080

If it works:

Browser
   ↓
localhost:8080
   ↓
port-forward
   ↓
Service
   ↓
Pod
   ↓
Container

This proves the application works before we add Ingress/ALB.

---

24. Production Troubleshooting Exercise — Break Readiness

Change:

readinessProbe:
  httpGet:
    path: /

to:

readinessProbe:
  httpGet:
    path: /does-not-exist

Apply:

kubectl apply -f 05-deployment.yaml

Watch:

kubectl get pods -w

You should see affected new Pods become:

0/1 Running

Now:

kubectl describe pod POD_NAME

Then:

kubectl get pods -o wide

Show students:

Running: YES
IP:      YES
Ready:   NO

Check:

kubectl get endpointslice \
-l kubernetes.io/service-name=web-service \
-o yaml

Then fix the readiness path:

path: /

and apply again.

---

25. Production Troubleshooting Exercise — Scheduler

Now break requests:

requests:
  cpu: "100"
  memory: "100Gi"

Apply:

kubectl apply -f 05-deployment.yaml

Check:

kubectl get pods

New Pods may remain:

Pending

Why?

Pod
requests:
100 CPU
100Gi memory
      │
      ▼
Scheduler
      │
      ├── Node 1 ❌
      ├── Node 2 ❌
      └── Node 3 ❌
              │
              ▼
            PENDING

Prove it:

kubectl describe pod POD_NAME

Look at:

Events

You may see:

Insufficient cpu
Insufficient memory

Restore:

requests:
  cpu: "100m"
  memory: "128Mi"

Apply again.

---

26. PodDisruptionBudget

We have three replicas.

During supported voluntary disruptions, we want to preserve application availability.

Create:

nano 07-pdb.yaml

apiVersion: policy/v1
kind: PodDisruptionBudget

metadata:
  name: web-pdb
  namespace: jumptotech-prod

spec:

  minAvailable: 2

  selector:
    matchLabels:
      app: web

Apply:

kubectl apply -f 07-pdb.yaml

Check:

kubectl get pdb

Then:

kubectl describe pdb web-pdb

Think:

3 replicas

[Pod 1] [Pod 2] [Pod 3]

        PDB

 minAvailable = 2

Important:

PDB does not prevent every crash or Node failure.

It primarily constrains voluntary disruptions that use Kubernetes' eviction mechanisms.

---

27. Horizontal Pod Autoscaler

Check metrics first:

kubectl top pods

If metrics are available, create:

nano 08-hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler

metadata:
  name: web-hpa
  namespace: jumptotech-prod

spec:

  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-deployment

  minReplicas: 3
  maxReplicas: 10

  behavior:

    scaleDown:
      stabilizationWindowSeconds: 300

  metrics:

    - type: Resource

      resource:

        name: cpu

        target:
          type: Utilization
          averageUtilization: 70

Apply:

kubectl apply -f 08-hpa.yaml

Check:

kubectl get hpa

Watch:

kubectl get hpa -w

Concept:

Metrics
   │
   ▼
 HPA
   │
   ▼
Deployment desired replicas
   │
   ▼
ReplicaSet
   │
   ▼
Pods


Low demand
↓
3 Pods


High CPU demand
↓
HPA
↓
more replicas
↓
up to 10 Pods

Important connection:

For CPU utilization-based HPA, CPU requests matter because utilization is evaluated relative to requested CPU.

---

28. Ingress + AWS ALB

Before creating it:

kubectl get deployment \
aws-load-balancer-controller \
-n kube-system

If the AWS Load Balancer Controller is installed, create:

nano 09-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress

metadata:
  name: web-ingress
  namespace: jumptotech-prod

  annotations:

    alb.ingress.kubernetes.io/scheme: internet-facing

    alb.ingress.kubernetes.io/target-type: ip

spec:

  ingressClassName: alb

  rules:

    - http:

        paths:

          - path: /
            pathType: Prefix

            backend:

              service:

                name: web-service

                port:
                  number: 80

Apply:

kubectl apply -f 09-ingress.yaml

Watch:

kubectl get ingress -w

Eventually you may receive an ALB address.

---

29. Who Actually Creates the ALB?

This is important.

The flow is:

kubectl apply
       ↓
Ingress object
       ↓
Kubernetes API
       ↓
AWS Load Balancer Controller
       ↓
AWS API
       ↓
ALB
Listeners
Rules
Target Groups

The controller reconciles Kubernetes configuration into AWS infrastructure.

But customer runtime traffic is different:

CUSTOMER
    │
    ▼
   ALB
    │
    ▼
configured targets
    │
    ▼
  PODS

The AWS Load Balancer Controller does not sit in the middle of every HTTP request.

---

30. NetworkPolicy

Now introduce network security.

Without network policy enforcement, workloads may be able to communicate more broadly than you intend.

Create:

nano 10-networkpolicy.yaml

Example policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy

metadata:
  name: web-ingress-policy
  namespace: jumptotech-prod

spec:

  podSelector:
    matchLabels:
      app: web

  policyTypes:
    - Ingress

  ingress:

    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: jumptotech-prod

      ports:
        - protocol: TCP
          port: 80

Explain:

NetworkPolicy
      ↓
Which network connections
are allowed to/from selected Pods?

However, tell students something very important:

Creating a NetworkPolicy object only has an enforcement effect when the cluster's networking implementation supports and enforces Kubernetes NetworkPolicy. Verify your EKS networking configuration before treating this as protection.

Also, this particular example is for teaching namespace-scoped ingress and may need adjustment for the actual ALB traffic path in your cluster. Don't blindly deploy a restrictive policy to production without testing.

For that reason, in class I would inspect this file first, then apply it only if your cluster's NetworkPolicy enforcement and ALB path are already verified.

---

31. RBAC — Role

ServiceAccount gave us identity.

Now we can demonstrate authorization.

Create:

nano 11-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role

metadata:
  name: pod-reader
  namespace: jumptotech-prod

rules:

  - apiGroups:
      - ""

    resources:
      - pods

    verbs:
      - get
      - list
      - watch

This says:

Allowed resource:
Pods

Allowed actions:
GET
LIST
WATCH

It does not say:

delete Pods
create Pods
delete Deployments

---

32. RoleBinding

Now connect:

ServiceAccount
      +
    Role

Create:

nano 12-rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding

metadata:
  name: web-pod-reader
  namespace: jumptotech-prod

subjects:

  - kind: ServiceAccount
    name: web-sa
    namespace: jumptotech-prod

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: Role

  name: pod-reader

Apply:

kubectl apply -f 11-role.yaml
kubectl apply -f 12-rolebinding.yaml

Now teach:

ServiceAccount
web-sa
      │
      ▼
RoleBinding
      │
      ▼
Role
pod-reader
      │
      ▼
get
list
watch
Pods

Test authorization:

kubectl auth can-i list pods \
--as=system:serviceaccount:jumptotech-prod:web-sa \
-n jumptotech-prod

Expected:

yes

Try:

kubectl auth can-i delete pods \
--as=system:serviceaccount:jumptotech-prod:web-sa \
-n jumptotech-prod

Expected:

no

This is a fantastic live demonstration of least privilege.

---

33. Final Production Verification

Now run:

kubectl get deployment
kubectl get rs
kubectl get pods -o wide
kubectl get svc
kubectl get endpointslice
kubectl get configmap
kubectl get serviceaccount
kubectl get pdb
kubectl get hpa
kubectl get ingress
kubectl get networkpolicy
kubectl get role
kubectl get rolebinding

And:

kubectl get all

Important teaching point:

kubectl get all does not literally mean every Kubernetes resource type.

For example, you still separately inspect objects such as:

kubectl get ingress
kubectl get configmap
kubectl get secrets
kubectl get networkpolicy
kubectl get pdb

---

34. Production Troubleshooting Method

Teach students not to randomly type commands.

For a website that is unavailable, troubleshoot layer by layer:

Internet
   ↓
ALB
   ↓
Ingress
   ↓
Service
   ↓
EndpointSlice
   ↓
Pod Ready?
   ↓
Container
   ↓
Application

Use:

kubectl get ingress
kubectl describe ingress web-ingress

kubectl get svc
kubectl describe svc web-service

kubectl get endpointslice \
-l kubernetes.io/service-name=web-service

kubectl get pods -o wide

kubectl describe pod POD_NAME

kubectl logs POD_NAME

This troubleshooting order is more important than memorizing 50 kubectl commands.

---

35. What Each Production Feature Solves

Have students explain this without looking at notes:

Feature	Question it answers
Namespace	Where are these resources logically scoped?
Deployment	What workload state do I want?
ReplicaSet	How many matching Pods should exist?
Pod	Where does my workload execute?
Service	How do clients reach replaceable Pods through a stable service abstraction?
EndpointSlice	Which endpoints back the Service?
ConfigMap	Where is non-sensitive configuration?
Secret	Where is sensitive configuration represented?
ServiceAccount	What Kubernetes identity does the workload use?
Role	What namespaced Kubernetes API actions are permitted?
RoleBinding	Who receives those Role permissions?
Requests	What resources should scheduling account for?
Limits	What runtime resource ceiling applies?
Startup Probe	Has the application started?
Readiness Probe	Should it receive traffic?
Liveness Probe	Should kubelet restart an unhealthy container?
RollingUpdate	How should a Deployment replace versions?
PDB	How much availability should voluntary disruptions preserve?
HPA	Should replica count scale automatically?
NetworkPolicy	Which Pod network traffic should be permitted?
Ingress	What HTTP/HTTPS routing should exist?

---

36. Final Architecture Students Should Be Able to Draw

                              INTERNET
                                  │
                                  ▼
                                 ALB
                                  │
                                  ▼
                               INGRESS
                                  │
                                  ▼
                            web-service
                             ClusterIP
                                  │
                         selector app=web
                                  │
                    ┌─────────────┼─────────────┐
                    ▼             ▼             ▼
                  POD 1         POD 2         POD 3
                    │             │             │
                Container     Container     Container
                    ▲             ▲             ▲
                    └─────────────┼─────────────┘
                                  │
                              ReplicaSet
                                  ▲
                                  │
                              Deployment
                                  │
             ┌────────────────────┼────────────────────┐
             │                    │                    │
             ▼                    ▼                    ▼
         ConfigMap              Secret          ServiceAccount
                                                      │
                                                      ▼
                                               RoleBinding
                                                      │
                                                      ▼
                                                    Role


              ┌───────────────────────────────────────────┐
              │          PRODUCTION CONTROLS              │
              │                                           │
              │ Requests/Limits → Resources               │
              │ Startup        → Startup protection       │
              │ Readiness      → Traffic readiness        │
              │ Liveness       → Health/restart           │
              │ RollingUpdate  → Deployment strategy      │
              │ PDB            → Disruption availability  │
              │ HPA            → Scaling                  │
              │ NetworkPolicy  → Network security         │
              └───────────────────────────────────────────┘