DEV Community: Ajinkya Singh

Kubernetes Core Workloads: Everything You Need to Know

Ajinkya Singh — Fri, 03 Apr 2026 07:52:31 +0000

If you've made it past the basics of Kubernetes — you know what a cluster is, you've spun up a local environment with kind or minikube — the next wall you hit is understanding the workload objects. There are nine of them. They look similar in YAML. They all involve containers. But they each exist for a very different reason.

The Quick Reference: Which Workload Do You Need?

Before diving in, here's the decision tree you'll use every day:

Workload	Use When
Deployment	Running stateless apps, APIs, web servers, workers
StatefulSet	Databases, message brokers, anything needing stable identity
DaemonSet	Log agents, monitoring — one pod per node
Job	One-off batch tasks: migrations, reports, cleanup
CronJob	Scheduled recurring tasks
Pod (bare)	Almost never in production — use one of the above

Now let's understand why.

Pods: The Atomic Unit

Everything in Kubernetes ultimately runs as a Pod — the smallest deployable unit. You don't run containers directly; you run Pods that contain containers.

Think of a Pod as a tiny apartment that one or more containers share:

They share a single IP address
Containers within a Pod talk to each other via localhost
They can share volumes (disk storage)
They always land on the same node — never split

apiVersion: v1
kind: Pod
metadata:
  name: api-server-pod
spec:
  containers:
    - name: api
      image: mycompany/api:2.1.0
      resources:
        requests:
          memory: "256Mi"
          cpu: "250m"
        limits:
          memory: "512Mi"
          cpu: "1000m"
      livenessProbe:
        httpGet:
          path: /health
          port: 3000
        initialDelaySeconds: 15
        periodSeconds: 20
      readinessProbe:
        httpGet:
          path: /ready
          port: 3000
        initialDelaySeconds: 5
        periodSeconds: 10

The Most Important Thing About Pods

Pods are ephemeral. When a Pod dies, a new one is created with a completely new name and a completely new IP. Any code that hardcoded the old IP breaks. This is why Services exist (Phase 3) — but it's the mental model you need to carry through everything else.

Two Probes You Can't Skip

Liveness probe — "Is this container still alive?" If it fails, Kubernetes restarts the container. Use for crash detection and deadlocks.

Readiness probe — "Is this container ready for traffic?" If it fails, the pod is removed from the Service's endpoints — stops receiving traffic — but is not restarted. Use for startup warmup and temporary overload.

These two probes are doing different jobs. Never confuse them.

Pod Status Lifecycle

Pending → Running → Succeeded
              ↓
         Failed / CrashLoopBackOff / OOMKilled

When you see CrashLoopBackOff, run kubectl logs my-pod --previous — that's where your crash output lives.

Quick Debugging Playbook

# Pod stuck in Pending?
kubectl describe pod my-pod   # read the Events section
# "Insufficient CPU" → reduce requests or add nodes

# ImagePullBackOff?
kubectl describe pod my-pod   # wrong image name? typo? private registry?

# CrashLoopBackOff?
kubectl logs my-pod --previous   # read the crash

# OOMKilled?
kubectl describe pod my-pod   # see Last State → increase memory limit

# Running but not working?
kubectl exec -it my-pod -- /bin/bash   # investigate from inside

🧪 Practice — Pods

Lab 1: Your first pod

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hello-nginx
  labels:
    app: hello
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "200m"
EOF

kubectl get pods --watch
kubectl describe pod hello-nginx    # read the Events section top to bottom
kubectl exec -it hello-nginx -- /bin/bash
# inside: curl localhost
kubectl port-forward pod/hello-nginx 8080:80 &
curl http://localhost:8080
kill %1

Lab 2: Multi-container pod — shared localhost

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: multi-pod
spec:
  containers:
    - name: app
      image: nginx:1.25
    - name: sidecar
      image: busybox
      command: ['sh', '-c', 'while true; do echo sidecar running; sleep 5; done']
EOF

kubectl logs multi-pod -c app
kubectl logs multi-pod -c sidecar
kubectl exec -it multi-pod -c sidecar -- /bin/sh
# inside: wget -q -O- localhost   ← hits the nginx container via shared IP

Lab 3: Trigger and observe CrashLoopBackOff

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crasher
spec:
  containers:
    - name: app
      image: busybox
      command: ['sh', '-c', 'echo crashing; exit 1']
EOF

kubectl get pods --watch    # watch it go into CrashLoopBackOff
kubectl logs crasher --previous   # see the crash output

Cleanup

kubectl delete pod hello-nginx multi-pod crasher

Namespaces: Logical Isolation

A Namespace is a virtual cluster inside your real cluster. It's how you divide one Kubernetes cluster into isolated sections — by team, environment, or application.

Four namespaces exist by default:

default — where objects land if you don't specify
kube-system — Kubernetes system components (etcd, apiserver, coredns)
kube-public — publicly readable data (rarely used)
kube-node-lease — node heartbeat objects (internal, ignore)

What Namespaces Give You

Name reuse — you can have a webapp Deployment in both staging and production with no conflict.

RBAC scoping — give team A access only to their namespace.

Resource quotas — cap a namespace to 4 CPUs and 8Gi memory total.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    pods: "50"
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi

One Critical Misconception

Namespaces do not provide network isolation by default. A pod in dev can still reach a pod in prod if it knows the IP or DNS name. For actual network isolation, you need NetworkPolicies.

Cross-namespace DNS format: <service>.<namespace>.svc.cluster.local

🧪 Practice — Namespaces

Lab: Isolation, quotas, and the -n flag

# Create two environments
kubectl create namespace dev
kubectl create namespace prod

# Deploy the same app name in both — no conflict
kubectl create deployment webapp --image=nginx:1.25 -n dev
kubectl create deployment webapp --image=nginx:1.25 -n prod

kubectl get deployments -A    # see both side-by-side

# Apply a quota to prod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: prod
spec:
  hard:
    pods: "5"
    requests.cpu: "2"
    requests.memory: 4Gi
EOF

# Try to exceed it
kubectl scale deployment webapp --replicas=10 -n prod
kubectl get pods -n prod              # stops at quota limit
kubectl describe resourcequota prod-quota -n prod   # usage vs hard

# Set dev as your default namespace
kubectl config set-context --current --namespace=dev
kubectl get pods    # now implicitly reads from dev

# Switch back
kubectl config set-context --current --namespace=default

Cleanup

kubectl delete namespace dev prod

Labels, Selectors & Annotations

Labels are key-value pairs on any Kubernetes object. They're the connective tissue — how Services find Pods, how Deployments track their ReplicaSets, how you filter resources.

Annotations are also key-value pairs but for non-identifying metadata: descriptions, tool config, URLs. They cannot be used in selectors.

metadata:
  labels:
    app: api
    env: production
    version: "2.1.0"   # always quote values that look like numbers
  annotations:
    description: "Payments API server"
    prometheus.io/scrape: "true"

How a Service Uses Labels

apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  selector:
    app: api           # matches pods with app=api
    env: production    # AND env=production
  ports:
    - port: 80
      targetPort: 3000

The Service routes traffic to any pod with both labels. Remove a label from a pod and it's immediately removed from the Service's endpoints. Add it back and it rejoins. This is live — no restart required.

Set-Based Selectors

Beyond simple equality, you can use expressions:

selector:
  matchExpressions:
    - key: env
      operator: In
      values: [production, staging]
    - key: tier
      operator: NotIn
      values: [frontend]
    - key: app
      operator: Exists

One gotcha: always quote version numbers in YAML. version: 2.0 is parsed as a float, stored as "2" — breaking your selectors silently.

🧪 Practice — Labels & Selectors

Lab: Live label manipulation and Service routing

# Create three pods with different label combos
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: frontend-v1
  labels:
    app: frontend
    env: prod
    version: "1.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.25
---
apiVersion: v1
kind: Pod
metadata:
  name: backend-v1
  labels:
    app: backend
    env: prod
    version: "1.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.24
---
apiVersion: v1
kind: Pod
metadata:
  name: backend-staging
  labels:
    app: backend
    env: staging
    version: "2.0"
spec:
  containers:
    - name: nginx
      image: nginx:1.26
EOF

# Filter practice
kubectl get pods --show-labels
kubectl get pods -l app=backend
kubectl get pods -l app=backend,env=prod
kubectl get pods -l 'env in (prod,staging)'
kubectl get pods -l 'version notin (1.0)'

# Create a Service that selects app=backend + env=prod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
spec:
  selector:
    app: backend
    env: prod
  ports:
    - port: 80
      targetPort: 80
EOF

kubectl describe service backend-svc   # check Endpoints section

# Live label surgery — remove env from backend-v1
kubectl label pod backend-v1 env-
kubectl describe service backend-svc   # endpoint disappears immediately

# Add it back
kubectl label pod backend-v1 env=prod
kubectl describe service backend-svc   # endpoint returns

Cleanup

kubectl delete pod frontend-v1 backend-v1 backend-staging
kubectl delete service backend-svc

ReplicaSets: Self-Healing

A ReplicaSet ensures a specified number of identical Pods are always running.

spec.replicas: 3

Pod-2 crashes → ReplicaSet creates Pod-4 → back to 3
Rogue pod added → ReplicaSet deletes one → back to 3

The RS continuously counts pods matching its selector. Actual count ≠ desired → create or delete.

Why You Almost Never Create ReplicaSets Directly

ReplicaSets don't support rolling updates or rollbacks. That's exactly what Deployments add. In practice, you create a Deployment and it creates/manages a ReplicaSet for you.

You need to understand ReplicaSets because:

Deployments own them under the hood
Debugging often means inspecting the RS
It explains the self-healing behavior you see

The RS name format when created by a Deployment: <deployment-name>-<pod-template-hash>. That hash changes every time the pod template changes — which is exactly how rolling updates work.

🧪 Practice — ReplicaSets

Lab: Self-healing in action

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: demo-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
EOF

kubectl get pods -l app=demo

# Open a watch in a second terminal: kubectl get pods --watch

# Kill one pod — RS heals immediately
kubectl delete pod $(kubectl get pods -l app=demo -o jsonpath='{.items[0].metadata.name}')
kubectl get pods -l app=demo    # new pod already created

# Scale up
kubectl scale rs demo-rs --replicas=5
kubectl get pods -l app=demo

# Try creating a stray pod with the same label
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: stray-pod
  labels:
    app: demo        # matches RS selector!
spec:
  containers:
    - name: nginx
      image: nginx:1.25
EOF

kubectl get pods -l app=demo    # RS sees 6, wants 5 → deletes one

Cleanup

kubectl delete rs demo-rs

Deployments: The Standard Way to Run Apps

A Deployment is what you use 90% of the time. It wraps a ReplicaSet and adds rolling updates, rollbacks, and version history.

You → Deployment → manages → ReplicaSets → manages → Pods
                      │
                      ├── RS v1 (old image)  → 0 pods
                      └── RS v2 (new image)  → 3 pods ✅

How a Rolling Update Actually Works

With maxSurge: 1, maxUnavailable: 0 and 3 replicas:

[v1][v1][v1]               # start

[v1][v1][v1][v2]           # spin up 1 new pod (surge)
[v1][v1][v2]               # v2 passes readiness → kill 1 v1

[v1][v1][v2][v2]           # spin up another v2
[v1][v2][v2]               # v2 passes → kill v1

[v1][v2][v2][v2]           # final new pod
[v2][v2][v2]               # all updated, zero downtime ✅

The readiness probe gates each step. Without it, Kubernetes can't tell when a new pod is actually ready.

Production Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0   # never go below desired count
  minReadySeconds: 10
  revisionHistoryLimit: 5
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: mycompany/orders-api:3.0.1
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5

The Commands You'll Use Daily

# Update image → triggers rolling update
kubectl set image deployment/my-app app=nginx:1.26

# Watch it happen
kubectl rollout status deployment/my-app

# Something went wrong?
kubectl rollout undo deployment/my-app

# Check history
kubectl rollout history deployment/my-app

# Restart all pods (rolling, with zero downtime)
kubectl rollout restart deployment/my-app

kubectl rollout undo is your emergency brake. Run it first, debug second.

🧪 Practice — Deployments

Lab: Full deployment lifecycle — deploy, update, break, rollback

# Deploy v1
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  annotations:
    kubernetes.io/change-cause: "initial deployment nginx 1.24"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
        - name: nginx
          image: nginx:1.24
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
EOF

kubectl get pods --watch    # Ctrl+C when all 3 Running
kubectl get rs              # see the RS created by the Deployment

# Rolling update to 1.25
kubectl set image deployment/webapp nginx=nginx:1.25
kubectl annotate deployment/webapp kubernetes.io/change-cause="upgrade to nginx 1.25" --overwrite
kubectl rollout status deployment/webapp
kubectl rollout history deployment/webapp   # see 2 revisions
kubectl get rs    # old RS now has 0 pods but still exists for rollback

# Simulate a bad update — nonexistent image
kubectl set image deployment/webapp nginx=nginx:NONEXISTENT
kubectl rollout status deployment/webapp    # hangs — new pods can't start
kubectl get pods    # old pods still serving! new pods in ErrImagePull

# Emergency rollback
kubectl rollout undo deployment/webapp
kubectl rollout status deployment/webapp    # watch recovery
kubectl get pods    # all healthy again

# Scale
kubectl scale deployment webapp --replicas=6
kubectl scale deployment webapp --replicas=2

Cleanup

kubectl delete deployment webapp

Imperative vs Declarative: The Mental Model

There are two ways to work with Kubernetes.

Imperative — tell Kubernetes what to do step by step:

kubectl create deployment web --image=nginx:1.25
kubectl scale deployment web --replicas=3
kubectl set image deployment/web nginx=nginx:1.26

Declarative — tell Kubernetes what you want, let it figure out how:

kubectl apply -f web.yaml
# edit the file
kubectl apply -f web.yaml   # kubernetes computes the diff

The key advantage of declarative: idempotency.

# Imperative — fails on second run
kubectl create deployment web --image=nginx
# Error: deployments.apps "web" already exists

# Declarative — safe to run forever
kubectl apply -f web.yaml
# deployment.apps/web created      (first run)
# deployment.apps/web unchanged    (no changes)
# deployment.apps/web configured   (file changed)

In production: always declarative. Store YAML in Git. CI/CD runs kubectl apply. This is GitOps — Git is the source of truth, the cluster always mirrors it.

Generate YAML Fast

kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

--dry-run=client -o yaml generates the YAML locally without sending anything to the API server. Edit it, then apply. This is extremely useful in CKA/CKAD exams.

🧪 Practice — Imperative vs Declarative

Lab: Generate, edit, and apply

# Generate Deployment YAML without applying
kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

cat deployment.yaml    # inspect what was generated

# Add resources block manually, then apply
kubectl apply -f deployment.yaml
kubectl apply -f deployment.yaml    # safe to run again — "unchanged"

# Edit the file — change replicas to 5
sed -i 's/replicas: 3/replicas: 5/' deployment.yaml
kubectl apply -f deployment.yaml    # "configured" — only diff applied
kubectl get deployment web

# Generate Pod and Service YAML for practice
kubectl run my-pod --image=busybox --dry-run=client -o yaml > pod.yaml
kubectl expose deployment web --port=80 --type=ClusterIP \
  --dry-run=client -o yaml > service.yaml

Cleanup

kubectl delete deployment web

DaemonSets: One Pod per Node

A DaemonSet ensures exactly one Pod runs on every node in the cluster. New node joins → pod auto-created. Node removed → pod garbage collected.

Node 1      Node 2      Node 3      Node 4 (new)
[log]       [log]       [log]       [log] ← auto-created

When to Use DaemonSets

Use Case	Examples
Log collection	Fluentd, Filebeat, Promtail
Monitoring agents	Prometheus node-exporter, Datadog agent
Network plugins	Calico, Cilium, kube-proxy
Security agents	Falco, Sysdig

kube-proxy itself is a DaemonSet. Verify: kubectl get ds -n kube-system

Running on Control-Plane Nodes

By default, DaemonSet pods won't run on control-plane nodes because of a taint: node-role.kubernetes.io/control-plane:NoSchedule. If you need them there, add a toleration:

spec:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

Update Strategies

RollingUpdate (default): automatically updates pods one node at a time
OnDelete: only updates pods when you manually delete them — useful for per-node maintenance windows

🧪 Practice — DaemonSets

Lab: Per-node coverage and system DaemonSets

# See existing DaemonSets in the cluster
kubectl get ds -n kube-system -o wide
# kube-proxy is a DaemonSet — one pod per node

# Create a simple node-monitoring DaemonSet
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      containers:
        - name: monitor
          image: busybox
          command:
            - sh
            - -c
            - while true; do echo "Node $(NODE_NAME) alive at $(date)"; sleep 10; done
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              memory: "20Mi"
              cpu: "10m"
EOF

# Verify one pod per worker node
kubectl get pods -l app=node-monitor -o wide

# Check logs from one
POD=$(kubectl get pods -l app=node-monitor -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD

Cleanup

kubectl delete ds node-monitor

Jobs & CronJobs: Batch and Scheduled Work

Job — runs pods to completion. Once finished successfully, stops. For one-off batch tasks.

CronJob — runs a Job on a schedule (cron syntax). For recurring tasks.

Deployment:  "Run this forever"
Job:         "Run this ONCE until it succeeds"
CronJob:     "Run this Job every day at 2am"

Job YAML

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 3              # retry up to 3 times
  activeDeadlineSeconds: 300   # kill after 5 minutes regardless
  ttlSecondsAfterFinished: 60  # auto-delete 60s after completion
  template:
    spec:
      restartPolicy: OnFailure  # REQUIRED — Always is invalid for Jobs
      containers:
        - name: migration
          image: mycompany/api:3.0.1
          command: ["node", "scripts/migrate.js"]

The restartPolicy: OnFailure (or Never) is mandatory — Jobs cannot use Always because they'd never complete.

Set ttlSecondsAfterFinished or finished Jobs pile up in your cluster forever.

CronJob YAML

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"        # every day at 2:00 AM
  concurrencyPolicy: Forbid    # skip if previous run still running
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 300
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: reporter
              image: mycompany/reporter:1.0
              command: ["python", "generate_report.py"]

Cron Schedule Quick Reference

0 2 * * *       Every day at 2:00 AM
*/5 * * * *     Every 5 minutes
0 9 * * 1       Every Monday at 9:00 AM
0 0 1 * *       1st of every month at midnight
@daily          Shortcut for 0 0 * * *
@hourly         Shortcut for 0 * * * *

Trigger a CronJob Manually

kubectl create job --from=cronjob/nightly-report manual-run-01

This creates a Job immediately using the CronJob's template, without affecting scheduled runs.

🧪 Practice — Jobs & CronJobs

Lab 1: Simple one-off Job

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  backoffLimit: 2
  ttlSecondsAfterFinished: 60
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: hello
          image: busybox
          command: ['sh', '-c', 'echo "Job running at $(date)"; sleep 5; echo Done!']
EOF

kubectl get pods --watch    # watch pod go: Pending → Running → Completed
kubectl logs -l job-name=hello-job
kubectl get job hello-job   # COMPLETIONS should show 1/1

Lab 2: CronJob that runs every minute

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: minute-job
spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 60
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: echo
              image: busybox
              command: ['sh', '-c', 'echo "Scheduled run at $(date)"']
EOF

# Wait 1-2 minutes
kubectl get jobs
kubectl get cronjob minute-job

# Trigger it manually right now
kubectl create job --from=cronjob/minute-job manual-run-01
kubectl logs -l job-name=manual-run-01

# Suspend future runs without deleting
kubectl patch cronjob minute-job -p '{"spec":{"suspend":true}}'
kubectl get cronjob minute-job    # SUSPEND should show True

Cleanup

kubectl delete cronjob minute-job

StatefulSets: Stateful Applications

A StatefulSet is like a Deployment, but for applications that need three guarantees:

Stable, predictable names — pods are always pod-0, pod-1, pod-2
Stable storage — each pod gets its own PVC that sticks with it across restarts
Ordered startup/shutdown — pods start in order (0→1→2) and stop in reverse

DEPLOYMENT:              STATEFULSET:
my-app-abc12 (random)    mysql-0  (always 0)
my-app-def34 (random)    mysql-1  (always 1)
my-app-ghi56 (random)    mysql-2  (always 2)

Pod dies → new name      Pod dies → comes back as mysql-1
Pod dies → new IP        Pod dies → same storage re-attached
Starts in any order      Starts 0 first, then 1, then 2

The Headless Service Requirement

StatefulSets require a headless Service (clusterIP: None) to provide stable DNS names per pod:

mysql-0.mysql-svc.default.svc.cluster.local → 10.244.1.5
mysql-1.mysql-svc.default.svc.cluster.local → 10.244.2.8

A normal Service load-balances across pods. A headless Service returns individual DNS records for each pod — so applications can say "connect to the primary at mysql-0.mysql-svc" and mean it.

StatefulSet YAML

# Headless Service — create this FIRST
apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
spec:
  clusterIP: None    # ← this makes it headless
  selector:
    app: mysql
  ports:
    - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-svc    # links to headless service above
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:     # ← key difference from Deployment
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

volumeClaimTemplates creates individual PVCs per pod: data-mysql-0, data-mysql-1, data-mysql-2.

The PVC Retention Gotcha

When you delete a StatefulSet, the PVCs are not deleted. This is intentional — data is precious. But it means they stay around until you clean them up manually:

kubectl delete sts mysql
kubectl delete pvc -l app=mysql   # must do this separately

Canary Updates with partition

The partition field in rollingUpdate is powerful for staged rollouts:

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2    # only update pods with ordinal >= 2

Set partition: 2 to update only mysql-2 first. Test it. Lower to 1. Lower to 0 to complete. This is native canary deployment for StatefulSets.

🧪 Practice — StatefulSets

Lab: Stable identity, persistent storage, and ordered startup

# Create headless service + StatefulSet
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - port: 80
      name: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web-svc
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
EOF

# Watch ordered startup — web-0 first, then web-1, then web-2
kubectl get pods --watch

# Confirm stable names
kubectl get pods -l app=web   # always web-0, web-1, web-2

# See the per-pod PVCs created
kubectl get pvc   # www-web-0, www-web-1, www-web-2

# Write data to web-0's volume
kubectl exec web-0 -- sh -c 'echo "pod-0 data" > /usr/share/nginx/html/index.html'

# Delete web-0 — watch it come back with the SAME name
kubectl delete pod web-0
kubectl get pods --watch    # web-0 comes back, not a random name

# Data persists — PVC was reattached
kubectl exec web-0 -- cat /usr/share/nginx/html/index.html   # still "pod-0 data"

# Reach a specific pod by stable DNS from another pod
kubectl run tmp --image=busybox --rm -it --restart=Never -- \
  wget -q -O- web-0.web-svc.default.svc.cluster.local

Cleanup — note PVCs must be deleted separately

kubectl delete sts web
kubectl delete svc web-svc
kubectl delete pvc -l app=web    # PVCs are NOT auto-deleted

The Master Cheat Sheet

# LIST
kubectl get pods / deploy / rs / ds / sts / jobs / cronjobs

# USEFUL FLAGS
kubectl get pods -o wide              # +IP +NODE
kubectl get pods --show-labels        # show label columns
kubectl get pods -l app=my-app        # filter by label
kubectl get pods -A                   # all namespaces

# INSPECT (always read the Events section)
kubectl describe pod/deploy/sts <n>

# LOGS
kubectl logs <pod>
kubectl logs <pod> --previous         # after a crash
kubectl logs <pod> -c <container>     # multi-container pod

# SHELL
kubectl exec -it <pod> -- /bin/bash

# DEPLOYMENTS
kubectl scale deploy <n> --replicas=5
kubectl set image deploy/<n> <container>=<image>:tag
kubectl rollout status deploy/<n>
kubectl rollout undo deploy/<n>
kubectl rollout restart deploy/<n>

# JOBS
kubectl create job test --from=cronjob/<n>   # manual trigger
kubectl patch cronjob <n> -p '{"spec":{"suspend":true}}'

# GENERATE YAML BOILERPLATE
kubectl create deployment web --image=nginx:1.25 --replicas=3 \
  --dry-run=client -o yaml > deployment.yaml

API Versions Quick Reference

Workload	apiVersion	kind
Pod	`v1`	`Pod`
Namespace	`v1`	`Namespace`
ReplicaSet	`apps/v1`	`ReplicaSet`
Deployment	`apps/v1`	`Deployment`
DaemonSet	`apps/v1`	`DaemonSet`
StatefulSet	`apps/v1`	`StatefulSet`
Job	`batch/v1`	`Job`
CronJob	`batch/v1`	`CronJob`

Restart Policy Rules

Policy	Use In	Behaviour
`Always`	Deployments, DaemonSets	Restart on any exit — default
`OnFailure`	Jobs	Restart only on non-zero exit
`Never`	Jobs	Never restart — new pod on each retry

🧪 Final Practice — End-to-End Scenario

This lab ties everything together. You'll deploy a full application stack using every concept from this article.

Scenario: Deploy a web application with a background worker, a nightly cleanup job, and per-node logging.

# Step 1: Create a dedicated namespace
kubectl create namespace myapp

# Step 2: Deploy the web app (Deployment)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: myapp
  labels:
    app: web
    tier: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        tier: frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          resources:
            requests: { memory: "64Mi", cpu: "50m" }
            limits:   { memory: "128Mi", cpu: "200m" }
          readinessProbe:
            httpGet: { path: /, port: 80 }
            initialDelaySeconds: 3
EOF

# Step 3: Deploy a background worker (Deployment, different labels)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: myapp
  labels:
    app: worker
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: worker
  template:
    metadata:
      labels:
        app: worker
        tier: backend
    spec:
      containers:
        - name: worker
          image: busybox
          command: ['sh', '-c', 'while true; do echo "worker processing..."; sleep 10; done']
          resources:
            requests: { memory: "32Mi", cpu: "25m" }
EOF

# Step 4: Per-node log collector (DaemonSet)
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: myapp
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
        - name: collector
          image: busybox
          command: ['sh', '-c', 'while true; do echo "collecting logs from $(NODE_NAME)"; sleep 15; done']
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests: { memory: "20Mi", cpu: "10m" }
EOF

# Step 5: Nightly cleanup Job (CronJob)
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cleanup
  namespace: myapp
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  jobTemplate:
    spec:
      ttlSecondsAfterFinished: 120
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: cleanup
              image: busybox
              command: ['sh', '-c', 'echo "cleaning up old data..."; sleep 3; echo done']
EOF

# Inspect the whole namespace
kubectl get all -n myapp

# Check labels
kubectl get pods -n myapp --show-labels

# Filter by tier
kubectl get pods -n myapp -l tier=frontend
kubectl get pods -n myapp -l tier=backend

# Trigger cleanup job manually
kubectl create job --from=cronjob/cleanup manual-cleanup-01 -n myapp
kubectl logs -l job-name=manual-cleanup-01 -n myapp

# Simulate rolling update on web
kubectl set image deployment/web nginx=nginx:1.26 -n myapp
kubectl rollout status deployment/web -n myapp

# Scale worker up
kubectl scale deployment worker --replicas=3 -n myapp
kubectl get pods -n myapp

Cleanup

kubectl delete namespace myapp    # deletes everything inside it

Kubernetes Foundations — Architecture, Components & kubectl

Ajinkya Singh — Sun, 22 Mar 2026 19:05:21 +0000

Why Kubernetes Exists
Cluster Architecture
Control Plane
Worker Node Components
kubectl
Local Cluster Setup
Cheat Sheet

Why Kubernetes Exists

Before Kubernetes, running containers in production meant: SSH into servers, start and stop containers by hand, write fragile shell scripts for restarts, and scale by spinning up machines one by one.

Kubernetes is a container orchestration system. It solves five problems that every team hits at scale.

🔴 Problem 1 — Single Point of Failure

Without Kubernetes:
  [App] ──► [Server A] ──✕──► App is DOWN until someone manually fixes it

With Kubernetes:
             ┌──► [Node A] ──✕── dies
  [App] ─────┤──► [Node B]  ✓  still running
             └──► [Node C]  ✓  still running
                            K8s auto-reschedules pod from A → B or C

Kubernetes detects the dead node and automatically reschedules all pods to healthy ones. No pager. No SSH.

📈 Problem 2 — Manual Scaling

Without Kubernetes:
  Traffic spike → someone notices → SSH → docker run ×10 → 20 minutes wasted

With Kubernetes:
  Traffic spike → HPA detects high CPU → pods spin up automatically → seconds, no human

The Horizontal Pod Autoscaler watches metrics and adds or removes pods automatically.

🚀 Problem 3 — Deployment Downtime

Without Kubernetes:
  Deploy v2:   stop v1 ──► [gap — users see errors] ──► start v2

With Kubernetes (rolling update):
  [v1][v1][v1]
       ↓
  [v2][v1][v1]   ← v2 started and health-checked first
       ↓
  [v2][v2][v1]
       ↓
  [v2][v2][v2]   ← v1 fully gone. Zero downtime.

New pods are health-checked before old ones are removed. Users never see an error page.

🔑 Problem 4 — Config & Secrets Chaos

Without Kubernetes:
  DB_PASSWORD="hunter2"  ← hardcoded in source code
  Different .env files per environment ← drift guaranteed

With Kubernetes:
  ConfigMaps + Secrets   ← config lives outside the image
  Same binary            ← works in dev / staging / prod

Config is decoupled from the container image entirely.

🌐 Problem 5 — Service Discovery Breakage

Without Kubernetes:
  Service A ──hardcoded──► IP 10.0.0.45 (Service B)
  Service B restarts → new IP → Service A breaks

With Kubernetes:
  Service A ──DNS──► "payment-service"
  Kubernetes routes to the right pods — IPs are irrelevant

Pods are ephemeral and get new IPs on restart. Kubernetes Services provide stable DNS names that always resolve correctly.

Common Misconceptions

Misconception	Reality
Kubernetes replaces Docker	No — K8s orchestrates containers; Docker/containerd runs them
Only for large companies	Works equally well for a 2-person startup
Cloud-only	Runs on laptops, on-prem hardware, and cloud equally
Replaces CI/CD pipelines	K8s is the deployment target, not the pipeline

Cluster Architecture

A Kubernetes cluster has two types of machines:

Control Plane — the brain. Makes all scheduling and reconciliation decisions.
Worker Nodes — the muscle. Runs your actual application containers.

All communication goes through the control plane. You never interact with worker nodes directly.

┌──────────────────────────────────────────────────────────────────────┐
│                        KUBERNETES CLUSTER                            │
│                                                                      │
│   You (kubectl)                                                      │
│        │  HTTPS                                                      │
│        ▼                                                             │
│   ┌──────────────────────────────────────────────────────────────┐  │
│   │                     CONTROL PLANE                            │  │
│   │                                                              │  │
│   │   kube-apiserver  ◄──────────────►  etcd                    │  │
│   │   (front door · port 6443)          (cluster database)      │  │
│   │                                                              │  │
│   │   kube-scheduler                  controller-manager        │  │
│   │   (pod placement)                 (reconciliation loops)    │  │
│   └───────────────┬──────────────────────────┬────────────────  ┘  │
│                   │                          │                      │
│      ┌────────────▼──────┐  ┌────────────────▼──┐  ┌────────────┐  │
│      │   Worker Node 1   │  │   Worker Node 2   │  │  Worker 3  │  │
│      │  Pod  Pod         │  │   Pod             │  │  Pod  Pod  │  │
│      │  kubelet          │  │   kubelet         │  │  kubelet   │  │
│      │  kube-proxy       │  │   kube-proxy      │  │  kube-proxy│  │
│      │  containerd       │  │   containerd      │  │  containerd│  │
│      └───────────────────┘  └───────────────────┘  └────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

▶ Full flow: kubectl create deployment → running container

1.   kubectl          reads ~/.kube/config → finds API server URL
2.   kube-apiserver   authenticates, RBAC check, validates request
3.   kube-apiserver   writes Deployment object to etcd
4.   controller-mgr   detects new Deployment → creates ReplicaSet → unscheduled Pod
5.   kube-scheduler   filters + scores nodes → binds pod to best node
6.   kubelet          sees assignment → calls containerd via CRI
7.   containerd       pulls image → calls runc → starts container
8.   kubelet          reports "Running" to API server
9.   etcd             pod status updated
10.  kubectl get pods → my-app  1/1  Running ✓

Control Plane

kube-apiserver

The single entry point for everything — kubectl, CI/CD pipelines, dashboards, and internal components. Validates requests, enforces RBAC, persists state to etcd. Port 6443 over HTTPS.

▶ The 4-gate request pipeline

Incoming request
       │
       ▼
┌─ GATE 1: AUTHENTICATION ─────────────────────────────────────────┐
│  "Who are you?"                                                  │
│  TLS client certificates · Bearer tokens · OIDC                 │
│  Failure → 401 Unauthorized                                      │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 2: AUTHORIZATION (RBAC) ───────────────────────────────────┐
│  "Are you allowed to do this?"                                   │
│  Checks Roles and ClusterRoles bound to the user                 │
│  Failure → 403 Forbidden                                         │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 3: ADMISSION CONTROL ──────────────────────────────────────┐
│  Mutating webhooks   → can modify the request                    │
│  Validating webhooks → can reject the request                    │
│  Failure → 400 / 422                                             │
└──────────────────────────────────────┬───────────────────────────┘
                                       │
                                       ▼
┌─ GATE 4: PERSIST TO etcd ────────────────────────────────────────┐
│  Object written → 201 Created returned ✓                         │
└──────────────────────────────────────────────────────────────────┘

Controllers and kubectl --watch use long-lived HTTP watch streams, not polling:

# GET /api/v1/namespaces/default/pods?watch=true
# Server streams:
# ADDED    my-pod   Pending
# MODIFIED my-pod   Running
# DELETED  my-pod

etcd

Distributed key-value store. The single source of truth for all cluster state. Losing etcd without a backup = losing all cluster configuration.

/registry/pods/default/my-pod         →  { pod spec }
/registry/deployments/default/my-app  →  { deployment spec }
/registry/services/default/my-svc     →  { service config }

etcd uses Raft consensus — a write commits only when a majority quorum agrees.

▶ Raft quorum explained

3-node etcd cluster:

  Leader ──heartbeat──► Follower 1
         ──heartbeat──► Follower 2

Write committed when majority (2 of 3) agree ✓
Leader + one Follower offline → 1 of 3 → no majority → writes halt ✗

Cluster Size	Quorum	Failures Tolerated
1	1	0
3	2	1 ← production minimum
5	3	2 ← recommended
7	4	3

Always use an odd number. Even numbers risk a 50/50 split → no quorum → all writes halt.

kube-scheduler

Decides which node a new pod runs on. Does not start the pod — only decides placement.

▶ How the scheduler picks a node

New pod (unscheduled)
       │
       ▼
PHASE 1 — FILTER: "Which nodes can run this pod?"
  Node 1: 90% CPU → skip
  Node 2: 20% CPU → feasible ✓
  Node 3: 55% CPU → feasible ✓

PHASE 2 — SCORE: "Which is the best node?"
  Node 2: score 85  (less loaded, image already cached)
  Node 3: score 62

PHASE 3 — BIND:
  Pod assigned to Node 2 → kubelet starts the container

Diagnosing a pod stuck in Pending:

kubectl describe pod my-pod
# Events:
# Warning  FailedScheduling  0/3 nodes are available:
#   3 Insufficient memory.

controller-manager

Runs all built-in controllers in a single binary. Each one watches for drift between desired and actual state and corrects it — forever.

while True:
    desired = api_server.get_desired_state()
    actual  = api_server.get_actual_state()
    if desired != actual:
        take_corrective_action()   # create / delete / update
    sleep(brief_interval)

▶ Built-in controllers

Controller	Responsibility
Deployment Controller	Manages ReplicaSets to maintain desired replicas
ReplicaSet Controller	Ensures the correct number of Pods exist
Node Controller	Detects node failures, evicts pods from dead nodes
Job Controller	Runs pods to completion for batch workloads
CronJob Controller	Creates Jobs on a cron schedule
Endpoint Controller	Updates Service endpoints when backing pods change
Namespace Controller	Cleans up resources when a namespace is deleted

Production note: Run 3 or 5 control plane nodes for HA. Back up etcd on a schedule. Never run user workloads on control plane nodes.

Worker Node Components

kubelet

Runs on every worker node as a systemd service — not a Kubernetes pod. If it crashes, systemd restarts it.

▶ What kubelet manages

Responsibility	Detail
PodSpecs	Downloads pod definitions from the API server
Container lifecycle	Start, stop, restart via the container runtime
Health checks	Runs liveness, readiness, and startup probes
Resource reporting	Reports CPU/memory usage to the API server
Volume mounting	Mounts ConfigMaps, Secrets, and PVCs into containers
Static pods	Reads YAML from `/etc/kubernetes/manifests/` directly

▶ Static pods — how the control plane bootstraps itself

Static pods allow kubelet to start containers without a running API server, by reading YAML files from disk. This is how K8s starts itself from scratch:

/etc/kubernetes/manifests/
  etcd.yaml
  kube-apiserver.yaml
  kube-controller-manager.yaml
  kube-scheduler.yaml

Bootstrap order:
  systemd starts kubelet
    → kubelet reads manifests → starts etcd
    → starts kube-apiserver
    → everything else connects normally

# Inspect in a kind cluster:
docker exec kind-control-plane ls /etc/kubernetes/manifests/

kube-proxy

Programs iptables or IPVS rules into the Linux kernel for Service routing. Not in the packet data path at runtime — it writes the rules and steps aside.

Client → GET http://my-service:80
       ↓
Linux kernel intercepts (rules written by kube-proxy)
       ├── 33% ──► Pod 10.244.1.5 (Node 1)
       ├── 33% ──► Pod 10.244.2.8 (Node 2)
       └── 34% ──► Pod 10.244.3.2 (Node 3)

▶ Proxy modes comparison

Mode	Mechanism	Lookup	Use Case
`iptables` (default)	Linux iptables chains	O(n)	< ~1,000 Services
`ipvs`	Kernel hash table	O(1)	Large clusters
eBPF (Cilium)	Bypasses iptables entirely	Best-in-class	Performance-critical

containerd

The default container runtime. kubelet calls it via the CRI (Container Runtime Interface).

kubelet → (gRPC / CRI) → containerd → (OCI) → runc → container

▶ The "Docker was removed" myth — clarified

What was removed in K8s 1.24: dockershim (an adapter between kubelet and Docker's API).

What was NOT removed:
  ✓  Docker images and Dockerfiles → still work perfectly
  ✓  docker build                  → still valid locally

Before K8s 1.24:  kubelet → dockershim → Docker → containerd → runc
After  K8s 1.24:  kubelet → containerd → runc

Docker was a middleman. containerd — which Docker itself uses internally — always did the actual work.

Runtime	Used By	Notes
containerd	GKE, EKS, AKS, kind	Default — lightweight, fast
CRI-O	OpenShift, bare-metal	Built specifically for Kubernetes

kubectl

kubectl reads ~/.kube/config to find your cluster's API server URL and credentials, then talks to kube-apiserver over HTTPS.

Managing Multiple Clusters

kubectl config view
kubectl config get-contexts
kubectl config use-context kind-k8s-lab
kubectl config current-context
kubectl config set-context --current --namespace=my-team
kubectl get pods --context=production-cluster

Command Structure

kubectl  [verb]    [resource]   [name]      [flags]
         get       pods         my-pod      -n production
         describe  deployment   my-dep
         delete    service      my-svc
         apply                  -f app.yaml
         exec      -it my-pod   --          /bin/bash
         logs      my-pod                   -f --previous

▶ Getting information

kubectl get pods
kubectl get pods -n kube-system        # specific namespace
kubectl get pods -A                    # all namespaces
kubectl get pods -o wide               # with IP and NODE columns
kubectl get pods -o yaml               # full YAML spec
kubectl get pods --watch               # live stream
kubectl get pods -l app=my-app         # filter by label
kubectl get all -n my-namespace

kubectl describe pod my-pod            # full details + Events ← debug here
kubectl describe node my-node
kubectl describe deployment my-dep

> Always read the Events section in kubectl describe — it is the primary debugging signal in Kubernetes.

▶ Creating and updating resources

kubectl apply -f app.yaml              # create or update (idempotent) ← always prefer this
kubectl apply -f ./manifests/
kubectl create deployment my-app --image=nginx
kubectl create namespace my-team

kubectl apply is idempotent — creates if absent, updates if present. kubectl create fails if the resource already exists. Use apply in all scripts and pipelines.

▶ Deleting resources

kubectl delete pod my-pod
kubectl delete -f app.yaml
kubectl delete deployment my-app                        # removes ReplicaSet and pods too
kubectl delete pod my-pod --force --grace-period=0      # immediate (stuck pods)

▶ Logs

kubectl logs my-pod
kubectl logs my-pod -f                      # stream live
kubectl logs my-pod --previous              # logs before last restart ← use after crashes
kubectl logs my-pod -c my-container         # specific container
kubectl logs my-pod --tail=100
kubectl logs my-pod --since=1h
kubectl logs -l app=my-app                  # all matching pods

▶ Exec into a pod

kubectl exec -it my-pod -- /bin/bash
kubectl exec -it my-pod -- /bin/sh          # if bash is unavailable
kubectl exec my-pod -- ls /app
kubectl exec -it my-pod -c sidecar -- /bin/sh

# Ephemeral debug pod — auto-deleted on exit
kubectl run debug --image=busybox --rm -it --restart=Never -- /bin/sh

▶ Port forwarding

kubectl port-forward pod/my-pod 8080:80
kubectl port-forward service/my-svc 8080:80
kubectl port-forward deployment/my-app 8080:3000

▶ Scaling and rollouts

kubectl scale deployment my-app --replicas=5
kubectl set image deployment/my-app nginx=nginx:1.25
kubectl rollout status deployment/my-app
kubectl rollout history deployment/my-app
kubectl rollout undo deployment/my-app
kubectl rollout undo deployment/my-app --to-revision=3

▶ JSONPath — extract specific fields

kubectl get pod my-pod -o jsonpath='{.status.podIP}'
kubectl get pods -o jsonpath='{.items[*].metadata.name}'

# All pods: namespace, name, IP
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'

Common Mistakes

Mistake	Correct Approach
Using `kubectl create` in scripts	Use `kubectl apply` — idempotent
Forgetting `-n <namespace>`	Use `-A` or set a default namespace in the context
Deleting a pod and expecting it to stay gone	Delete the Deployment — the ReplicaSet recreates pods
Skipping the Events section in `describe`	Events are the primary debugging signal
Not using `--previous` after a crash	`kubectl logs my-pod --previous` shows pre-crash logs

Local Cluster Setup

▶ Tool comparison

Tool	Nodes	Use Case
kind (recommended)	Multi-node	CI/CD, multi-node testing, closest to production
minikube	Single-node	Quick experiments, rich addons

Install

# macOS
brew install kubectl kind

# Linux — kubectl
curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# Linux — kind
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
chmod +x kind && sudo mv kind /usr/local/bin/

Create a Multi-Node Cluster

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: k8s-lab
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --config kind-config.yaml

kubectl get nodes
# k8s-lab-control-plane  Ready  control-plane
# k8s-lab-worker         Ready  <none>
# k8s-lab-worker2        Ready  <none>

Verify Health

kubectl get nodes                   # all nodes → Ready
kubectl get pods -n kube-system     # all system pods → Running
kubectl cluster-info

▶ First application lab

# Deploy
kubectl create deployment my-nginx --image=nginx:1.25 --replicas=2
kubectl get pods -o wide

# Expose
kubectl expose deployment my-nginx --port=80 --name=my-nginx-service
kubectl describe service my-nginx-service

# Access
kubectl port-forward service/my-nginx-service 8080:80 &amp;
curl http://localhost:8080

# Scale
kubectl scale deployment my-nginx --replicas=4

# Rolling update
kubectl set image deployment/my-nginx nginx=nginx:1.26
kubectl rollout status deployment/my-nginx

# Rollback
kubectl rollout undo deployment/my-nginx

# Clean up
kill %1
kubectl delete deployment my-nginx
kubectl delete service my-nginx-service

kind Management

kind get clusters
kind create cluster --config kind-config.yaml
kind delete cluster --name k8s-lab
kind load docker-image my-app:latest --name k8s-lab
kubectl config use-context kind-k8s-lab

Cheat Sheet

▶ Architecture & key concepts

Control Plane:   kube-apiserver · etcd · kube-scheduler · controller-manager
Worker Nodes:    kubelet · kube-proxy · containerd
Full flow:       kubectl → apiserver → etcd/scheduler → kubelet → container

Reconciliation:  desired ≠ actual → controller fixes it → repeat forever
Static pods:     /etc/kubernetes/manifests/ → kubelet reads directly
kubeconfig:      ~/.kube/config → cluster URLs + credentials
etcd quorum:     always odd — 3 or 5 in production
Scheduler:       filter → score → bind to winner
kubelet:         systemd service (not a pod); bootstraps the control plane

▶ kubectl quick reference

# Get
kubectl get pods / svc / deploy / nodes
kubectl get pods -A                      # all namespaces
kubectl get pods -o wide                 # +IP +NODE
kubectl get pod  -o yaml             # full spec
kubectl get pods --watch                 # live stream
kubectl describe pod                # details + Events ← debug here
kubectl explain pod.spec.containers      # built-in docs

# Apply / Delete
kubectl apply -f file.yaml
kubectl delete -f file.yaml
kubectl delete pod  --force

# Logs
kubectl logs  -f
kubectl logs  --previous            # after crash ← important
kubectl logs  -c 
kubectl logs -l app=my-app

# Exec / Port-forward
kubectl exec -it  -- /bin/bash
kubectl port-forward svc/ 8080:80

# Scale / Rollout
kubectl scale deployment  --replicas=5
kubectl set image deployment/ =<img>:tag
kubectl rollout status deployment/
kubectl rollout history deployment/
kubectl rollout undo deployment/

# Context
kubectl config get-contexts
kubectl config use-context 
kubectl config set-context --current --namespace=

▶ kind quick reference

kind create cluster --config kind-config.yaml
kind get clusters
kind load docker-image  --name 
kind delete cluster --name

🤖 AI-generated · Personal reference only · Not original content"

📚 Complete Java Microservices Guide: Dropwizard & Ecosystem

Ajinkya Singh — Thu, 26 Feb 2026 17:39:46 +0000

A Step-by-Step Beginner-to-Intermediate Learning Path

How to use this guide: Read chapter by chapter. Every concept builds on the previous one. Code examples are practical and runnable. Chapters marked 🔲 are TODO — topics to study next.

📋 Table of Contents

#	Chapter	Status
1	What is Dropwizard & Why Use It?	✅
2	Project Setup & Fat JAR	✅
3	Configuration with YAML	✅
4	Building REST Resources (JAX-RS)	✅
5	Bean Validation (Hibernate Validator)	✅
6	Jackson – JSON Serialization	✅
7	Database Access with Hibernate/JDBI	✅
8	Liquibase – Database Migrations	✅
9	Authentication (Basic Auth & OAuth2)	✅
10	Health Checks & Metrics	✅
11	Testing in Dropwizard	✅
12	Lombok – Boilerplate Killer	✅
13	MapStruct – Object Mapping	✅
14	Google Guice – Dependency Injection	✅
15	OkHttp – HTTP Client	✅
16	Hystrix – Circuit Breaker	✅
17	TODO: Advanced Topics	🔲

Chapter 1: What is Dropwizard & Why Use It?

🤔 The Problem It Solves

Imagine you want to build a Java REST API. You'd need to:

Pick a web server (Jetty, Tomcat?)
Pick a JSON library (Jackson, Gson?)
Pick a validation library
Pick a database library
Pick a metrics/monitoring library
Make sure they all work together (version conflicts are a nightmare!)

This research alone can take weeks and paralyze a developer.

✅ Dropwizard's Solution

Dropwizard is a curated collection of best-of-breed libraries glued together so they work out of the box.

Component	Library Used
Web Server	Jetty (embedded)
REST Framework	Jersey (JAX-RS)
JSON	Jackson
Validation	Hibernate Validator
Database Migrations	Liquibase
Metrics & Health	Metrics library
Logging	Logback + SLF4J
Config	YAML + Jackson

🫙 The Fat JAR Concept

Traditional Java apps rely on an app server (Tomcat, GlassFish) being installed on the machine. The problem:

"It works on my computer" 😤

Dropwizard packages everything — your code + all dependencies + the web server — into a single JAR file.

myapp.jar  ← Contains Jetty, Jersey, Jackson, your code... everything!

To run it:

java -jar myapp.jar server config.yml

That's it. No app server needed. No "works on my machine" problem.

🏗️ Microservices Context

Microservices = breaking a big monolithic app into small, independent services. Each service:

Does one thing well
Has its own database
Communicates via REST/HTTP
Can be deployed independently

Dropwizard is perfect for microservices because each service is a single JAR that starts up fast.

🆚 Dropwizard vs Spring Boot

Feature	Dropwizard	Spring Boot
Philosophy	Opinionated, curated	Flexible, extensive
Learning curve	Easier if you know JAX-RS	Easier if you know Spring
Startup speed	Fast	Moderate
Ecosystem size	Smaller	Huge
Best for	REST APIs, microservices	Anything Java

Which to choose? If you're building a focused REST microservice, Dropwizard is excellent. For complex enterprise apps, Spring Boot has more features.

Chapter 2: Project Setup & Fat JAR

🛠️ Prerequisites

Make sure you have installed:

Java 8+ (Dropwizard 1.x requires Java 8)
Maven 3.x
An IDE (IntelliJ IDEA recommended)

📁 Creating a Project with Maven Archetype

The fastest way to start:

mvn archetype:generate \
  -DarchetypeGroupId=io.dropwizard.archetypes \
  -DarchetypeArtifactId=java-simple \
  -DarchetypeVersion=1.3.29 \
  -DgroupId=com.example \
  -DartifactId=my-api \
  -Dversion=1.0-SNAPSHOT \
  -Dname=MyApi

⚠️ Important: The -Dname=MyApi parameter is used in generated file names. Always provide it!

📂 Generated Project Structure

my-api/
├── pom.xml
└── src/
    └── main/
        ├── java/
        │   └── com/example/
        │       ├── MyApiApplication.java   ← Main class (entry point)
        │       └── MyApiConfiguration.java ← Config class
        └── resources/
            └── config.yml                  ← Configuration file

📄 The `pom.xml` — Your Project's Heart

<project>
    <groupId>com.example</groupId>
    <artifactId>my-api</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <dropwizard.version>1.3.29</dropwizard.version>
    </properties>

    <dependencies>
        <!-- Core Dropwizard -->
        <dependency>
            <groupId>io.dropwizard</groupId>
            <artifactId>dropwizard-core</artifactId>
            <version>${dropwizard.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- This plugin creates the Fat JAR -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <configuration>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals><goal>shade</goal></goals>
                        <configuration>
                            <transformers>
                                <transformer implementation=
                                  "org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer implementation=
                                  "org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <!-- This tells Java which class has main() -->
                                    <mainClass>com.example.MyApiApplication</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

🚀 The Application Class

package com.example;

import io.dropwizard.Application;
import io.dropwizard.setup.Bootstrap;
import io.dropwizard.setup.Environment;

public class MyApiApplication extends Application<MyApiConfiguration> {

    // Java's main method — this starts everything
    public static void main(String[] args) throws Exception {
        new MyApiApplication().run(args);
    }

    @Override
    public String getName() {
        return "my-api";  // Name of your app
    }

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        // Register bundles (plugins) here
    }

    @Override
    public void run(MyApiConfiguration configuration, Environment environment) {
        // Register your resources (REST endpoints) here
        // This is where the app "comes alive"
    }
}

🏗️ Building & Running

# Build the Fat JAR
mvn clean package

# Run with server command
java -jar target/my-api-1.0-SNAPSHOT.jar server config.yml

# Check if it's alive (admin port 8081)
curl http://localhost:8081/ping

# Your API is on port 8080 by default
curl http://localhost:8080/hello

📋 Built-in Commands

# Validate your config file (catches errors before running!)
java -jar my-api.jar check config.yml

# Run the server
java -jar my-api.jar server config.yml

# Database commands (when Liquibase is added)
java -jar my-api.jar db migrate config.yml
java -jar my-api.jar db status config.yml
java -jar my-api.jar db drop-all --confirm-delete-everything config.yml

💡 Pro tip: Run check before server in deployment scripts to catch config errors early!

Chapter 3: Configuration with YAML

🤷 Why Configuration Matters

Hard-coding things like database URLs, passwords, and ports in code is bad practice. Configuration files let you:

Have different settings for dev/test/production
Change settings without recompiling
Keep secrets out of code

📄 The `config.yml` File

# Server ports
server:
  applicationConnectors:
    - type: http
      port: 8080
  adminConnectors:
    - type: http
      port: 8081

# Logging
logging:
  level: INFO
  loggers:
    com.example: DEBUG  # More verbose logging for your package

# Custom config (your own properties)
login: admin
password: secret123

# Database (added later with Hibernate bundle)
database:
  driverClass: com.mysql.jdbc.Driver
  url: jdbc:mysql://localhost:3306/mydb
  user: dbuser
  password: dbpassword

☕ The Configuration Class

Every key in config.yml maps to a field in your Configuration class:

package com.example;

import io.dropwizard.Configuration;
import com.fasterxml.jackson.annotation.JsonProperty;
import org.hibernate.validator.constraints.NotEmpty;

public class MyApiConfiguration extends Configuration {

    // @NotEmpty = Hibernate Validator annotation
    // If this field is empty/missing, app won't start!
    @NotEmpty
    private String login;

    @NotEmpty
    private String password;

    // @JsonProperty tells Jackson the YAML key name
    @JsonProperty
    public String getLogin() {
        return login;
    }

    @JsonProperty
    public void setLogin(String login) {
        this.login = login;
    }

    @JsonProperty
    public String getPassword() {
        return password;
    }

    @JsonProperty
    public void setPassword(String password) {
        this.password = password;
    }
}

🌍 Multiple Config Files (Best Practice)

# Development
java -jar my-api.jar server config-dev.yml

# Production  
java -jar my-api.jar server config-prod.yml

# Testing (uses H2 in-memory database)
java -jar my-api.jar server config-test.yml

Each config file can point to different databases, use different ports, etc.

🔐 Accessing Config in Your App

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Access config values
    String login = config.getLogin();
    String password = config.getPassword();

    // Pass them to classes that need them
    environment.jersey().register(new MyResource(login, password));
}

Chapter 4: Building REST Resources (JAX-RS)

🌐 What is JAX-RS?

JAX-RS is a Java standard (specification) for building REST APIs using annotations. Dropwizard uses Jersey as the JAX-RS implementation.

The key idea: annotate a regular Java class with special annotations, and Jersey turns it into a REST endpoint.

📌 Core Annotations

Annotation	Meaning
`@Path("/users")`	URL path this class/method handles
`@GET`	Responds to HTTP GET requests
`@POST`	Responds to HTTP POST requests
`@PUT`	Responds to HTTP PUT requests
`@DELETE`	Responds to HTTP DELETE requests
`@Produces(MediaType.APPLICATION_JSON)`	Returns JSON
`@Consumes(MediaType.APPLICATION_JSON)`	Accepts JSON input
`@PathParam("id")`	Gets value from URL: `/users/42` → id=42
`@QueryParam("name")`	Gets value from query string: `?name=John`

☕ Your First Resource Class

package com.example.resources;

import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.List;
import java.util.ArrayList;

@Path("/bookmarks")         // All methods in this class handle /bookmarks
@Produces(MediaType.APPLICATION_JSON)  // All responses are JSON
@Consumes(MediaType.APPLICATION_JSON)  // All requests accept JSON
public class BookmarkResource {

    private List<Bookmark> bookmarks = new ArrayList<>();

    // GET /bookmarks → returns all bookmarks
    @GET
    public List<Bookmark> getAllBookmarks() {
        return bookmarks;
    }

    // GET /bookmarks/42 → returns one bookmark by ID
    @GET
    @Path("/{id}")
    public Bookmark getBookmark(@PathParam("id") long id) {
        return bookmarks.stream()
            .filter(b -> b.getId() == id)
            .findFirst()
            .orElseThrow(() -> new NotFoundException("Bookmark not found: " + id));
    }

    // POST /bookmarks → create a new bookmark
    @POST
    public Response createBookmark(Bookmark bookmark) {
        bookmarks.add(bookmark);
        return Response.status(Response.Status.CREATED)
                       .entity(bookmark)
                       .build();
    }

    // DELETE /bookmarks/42 → delete a bookmark
    @DELETE
    @Path("/{id}")
    public Response deleteBookmark(@PathParam("id") long id) {
        bookmarks.removeIf(b -> b.getId() == id);
        return Response.noContent().build();  // 204 No Content
    }

    // GET /bookmarks/search?tag=java → filter by query param
    @GET
    @Path("/search")
    public List<Bookmark> searchByTag(@QueryParam("tag") String tag) {
        // filter logic here
        return bookmarks;
    }
}

📦 The Domain Model (Bookmark.java)

package com.example.core;

public class Bookmark {
    private long id;
    private String url;
    private String title;
    private String tag;

    // Default constructor (required for JSON deserialization)
    public Bookmark() {}

    public Bookmark(long id, String url, String title) {
        this.id = id;
        this.url = url;
        this.title = title;
    }

    // Getters and setters
    public long getId() { return id; }
    public void setId(long id) { this.id = id; }
    public String getUrl() { return url; }
    public void setUrl(String url) { this.url = url; }
    public String getTitle() { return title; }
    public void setTitle(String title) { this.title = title; }
    public String getTag() { return tag; }
    public void setTag(String tag) { this.tag = tag; }
}

📝 Register the Resource in Application

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Without this line, Dropwizard doesn't know about your resource!
    environment.jersey().register(new BookmarkResource());
}

🔧 HTTP Status Codes Quick Reference

Code	Meaning	When to use
200	OK	Successful GET, PUT
201	Created	Successful POST (new resource created)
204	No Content	Successful DELETE
400	Bad Request	Invalid input
401	Unauthorized	Not authenticated
403	Forbidden	Authenticated but no permission
404	Not Found	Resource doesn't exist
500	Server Error	Something blew up

🧑‍💻 Recommended Folder Structure

src/main/java/com/example/
├── MyApiApplication.java       ← Entry point
├── MyApiConfiguration.java     ← Config class
├── core/                       ← Domain models / entities
│   └── Bookmark.java
├── db/                         ← Database access (DAOs)
│   └── BookmarkDAO.java
├── resources/                  ← REST endpoints
│   └── BookmarkResource.java
└── auth/                       ← Authentication
    └── MyAuthenticator.java

Chapter 5: Bean Validation (Hibernate Validator)

❓ Why Validate?

Users send bad data. Always. Never trust input. Validation catches bad data before it hits your database.

Without validation:

{ "url": "", "title": null }

This would save garbage data to your database!

📦 Dependency (already included in dropwizard-core)

Dropwizard includes Hibernate Validator automatically. No extra dependency needed!

🏷️ Common Validation Annotations

import javax.validation.constraints.*;
import org.hibernate.validator.constraints.*;

public class Bookmark {

    private long id;

    @NotEmpty(message = "URL cannot be empty")
    @URL(message = "Must be a valid URL")
    private String url;

    @NotEmpty
    @Size(min = 1, max = 200, message = "Title must be 1-200 characters")
    private String title;

    @NotNull
    private String tag;

    @Min(value = 0, message = "Views cannot be negative")
    private int views;

    @Email(message = "Must be a valid email")
    private String authorEmail;

    @Pattern(regexp = "^[A-Z].*", message = "Must start with uppercase")
    private String category;
}

✅ Using `@Valid` in Resources

Add @Valid before the parameter — Dropwizard validates automatically!

@POST
public Response createBookmark(@Valid Bookmark bookmark) {
    // If bookmark fails validation, Dropwizard returns 422 Unprocessable Entity
    // with details about what's wrong — automatically!
    bookmarks.add(bookmark);
    return Response.status(201).entity(bookmark).build();
}

If validation fails, the client gets:

{
  "errors": [
    "bookmark.url may not be empty",
    "bookmark.title size must be between 1 and 200"
  ]
}

🛡️ Validating Config Too!

public class MyApiConfiguration extends Configuration {

    @NotEmpty  // App won't start if login is missing in config.yml!
    @JsonProperty
    private String login;

    @NotEmpty
    @JsonProperty
    private String password;
}

✍️ Custom Validator

// 1. Create the annotation
@Documented
@Constraint(validatedBy = NoSpacesValidator.class)
@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
public @interface NoSpaces {
    String message() default "Must not contain spaces";
    Class<?>[] groups() default {};
    Class<? extends Payload>[] payload() default {};
}

// 2. Create the validator logic
public class NoSpacesValidator implements ConstraintValidator<NoSpaces, String> {
    @Override
    public boolean isValid(String value, ConstraintValidatorContext ctx) {
        return value == null || !value.contains(" ");
    }
}

// 3. Use it
public class Bookmark {
    @NoSpaces
    private String slug;
}

Chapter 6: Jackson – JSON Serialization

🔄 What is Jackson?

Jackson converts between Java objects ↔ JSON. This is called serialization (Java → JSON) and deserialization (JSON → Java).

Dropwizard uses Jackson automatically for all REST responses. When your method returns a Bookmark object, Jackson converts it to JSON for you.

📦 Dependency

Already included in dropwizard-core!

🏷️ Key Jackson Annotations

import com.fasterxml.jackson.annotation.*;

public class Bookmark {

    // Changes the JSON key name
    @JsonProperty("bookmark_id")
    private long id;  // Java field "id" → JSON key "bookmark_id"

    // Include in JSON even if null
    @JsonInclude(JsonInclude.Include.ALWAYS)
    private String title;

    // Exclude from JSON output
    @JsonIgnore
    private String internalNotes;

    // Exclude null values from JSON
    @JsonInclude(JsonInclude.Include.NON_NULL)
    private String optionalField;

    // Custom date format
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss")
    private Date createdAt;

    // Map JSON → object (deserialization constructor)
    @JsonCreator
    public Bookmark(@JsonProperty("bookmark_id") long id,
                    @JsonProperty("url") String url) {
        this.id = id;
        this.url = url;
    }
}

🗺️ Working with Unknown Fields

// Ignore any extra JSON fields you don't have a field for
@JsonIgnoreProperties(ignoreUnknown = true)
public class Bookmark {
    private String url;
    // JSON might have "extra_field" — this annotation ignores it instead of throwing an error
}

⚙️ Customizing Jackson in Dropwizard

@Override
public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
    // Customize the ObjectMapper
    bootstrap.getObjectMapper()
        .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
        .setSerializationInclusion(JsonInclude.Include.NON_NULL);
}

🌐 Polymorphic Types (Advanced)

// Base class
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
    @JsonSubTypes.Type(value = WebBookmark.class, name = "web"),
    @JsonSubTypes.Type(value = BookBookmark.class, name = "book")
})
public abstract class Bookmark { }

public class WebBookmark extends Bookmark { private String url; }
public class BookBookmark extends Bookmark { private String isbn; }

// JSON will look like:
// { "type": "web", "url": "https://..." }
// { "type": "book", "isbn": "978-..." }

Chapter 7: Database Access with Hibernate/JDBI

🗃️ Your Two Options in Dropwizard

Option	Description	Best For
Hibernate (JPA)	ORM — maps Java objects to DB tables	Complex domain models
JDBI	Thin wrapper over JDBC — write SQL directly	Simple, fast queries

📦 Dependencies

<!-- For Hibernate -->
<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-hibernate</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

<!-- MySQL Driver (use 5.x to avoid Liquibase conflicts!) -->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.49</version>
</dependency>

🗄️ Part A: Hibernate

Step 1 — The Entity (maps to a DB table)

package com.example.core;

import javax.persistence.*;

@Entity                         // This Java class = a database table
@Table(name = "bookmarks")      // Table name in the database
@NamedQueries({
    // Pre-defined queries attached to this class
    @NamedQuery(
        name = "com.example.core.Bookmark.findAll",
        query = "SELECT b FROM Bookmark b"     // JPQL (not SQL!)
    ),
    @NamedQuery(
        name = "com.example.core.Bookmark.findByTag",
        query = "SELECT b FROM Bookmark b WHERE b.tag = :tag"
    )
})
public class Bookmark {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)  // Auto-increment
    private long id;

    @Column(name = "url", nullable = false)
    private String url;

    @Column(name = "title")
    private String title;

    @Column(name = "tag")
    private String tag;

    // Getters, setters, default constructor...
}

Step 2 — The DAO (Data Access Object)

package com.example.db;

import io.dropwizard.hibernate.AbstractDAO;
import com.example.core.Bookmark;
import org.hibernate.SessionFactory;
import org.hibernate.query.Query;
import java.util.List;
import java.util.Optional;

public class BookmarkDAO extends AbstractDAO<Bookmark> {

    public BookmarkDAO(SessionFactory factory) {
        super(factory);  // AbstractDAO handles Hibernate session management
    }

    // Get all bookmarks
    public List<Bookmark> findAll() {
        return list(namedQuery("com.example.core.Bookmark.findAll"));
    }

    // Get by ID — returns Optional (empty if not found)
    public Optional<Bookmark> findById(long id) {
        return Optional.ofNullable(get(id));
    }

    // Get by tag — using named query parameter
    public List<Bookmark> findByTag(String tag) {
        Query<Bookmark> query = namedQuery("com.example.core.Bookmark.findByTag");
        query.setParameter("tag", tag);
        return list(query);
    }

    // Create or update
    public Bookmark save(Bookmark bookmark) {
        return persist(bookmark);  // AbstractDAO's persist = INSERT or UPDATE
    }

    // Delete
    public void delete(Bookmark bookmark) {
        currentSession().delete(bookmark);
    }
}

Step 3 — Register Hibernate Bundle in Application

public class MyApiApplication extends Application<MyApiConfiguration> {

    // Create the Hibernate bundle — tell it about all your entities!
    private final HibernateBundle<MyApiConfiguration> hibernate =
        new HibernateBundle<MyApiConfiguration>(Bookmark.class) {  // list ALL entities here
            @Override
            public DataSourceFactory getDataSourceFactory(MyApiConfiguration config) {
                return config.getDataSourceFactory();
            }
        };

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        bootstrap.addBundle(hibernate);  // Register the bundle!
    }

    @Override
    public void run(MyApiConfiguration config, Environment environment) {
        // Create DAO with Hibernate session factory
        final BookmarkDAO dao = new BookmarkDAO(hibernate.getSessionFactory());

        // Pass DAO to resource
        environment.jersey().register(new BookmarkResource(dao));
    }
}

Step 4 — Config file addition

# config.yml
database:
  driverClass: com.mysql.jdbc.Driver
  url: jdbc:mysql://localhost:3306/bookmarks_db
  user: root
  password: secret
  properties:
    charSet: UTF-8
    hibernate.dialect: org.hibernate.dialect.MySQL5InnoDBDialect
  maxWaitForConnection: 1s
  validationQuery: "/* MyService Health Check */ SELECT 1"
  minSize: 2
  maxSize: 8
  checkConnectionWhileIdle: false

Step 5 — Add DataSourceFactory to Config class

public class MyApiConfiguration extends Configuration {

    @Valid
    @NotNull
    private DataSourceFactory database = new DataSourceFactory();

    @JsonProperty("database")
    public DataSourceFactory getDataSourceFactory() { return database; }

    @JsonProperty("database")
    public void setDataSourceFactory(DataSourceFactory factory) {
        this.database = factory;
    }
}

Step 6 — Use `@UnitOfWork` in Resource

public class BookmarkResource {
    private final BookmarkDAO dao;

    public BookmarkResource(BookmarkDAO dao) {
        this.dao = dao;
    }

    @GET
    // @UnitOfWork manages the Hibernate session + transaction for you!
    // Without it, you'd have to open/close sessions manually.
    @UnitOfWork
    public List<Bookmark> getAllBookmarks() {
        return dao.findAll();
        // ⚠️ WARNING: Lazy-loaded fields must be accessed INSIDE this method!
    }

    @POST
    @UnitOfWork
    public Bookmark createBookmark(@Valid Bookmark bookmark) {
        return dao.save(bookmark);
    }
}

Chapter 8: Liquibase – Database Migrations

🤔 The Problem Without Migrations

Imagine two developers both changing the database schema. Developer A adds a tag column, Developer B renames url to link. Who goes first? How do you track DB changes? How do you roll back?

Liquibase is like Git, but for your database schema.

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-migrations</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

⚙️ Register the Bundle

@Override
public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
    bootstrap.addBundle(new MigrationsBundle<MyApiConfiguration>() {
        @Override
        public DataSourceFactory getDataSourceFactory(MyApiConfiguration config) {
            return config.getDataSourceFactory();
        }
    });
}

📄 The Migration File (`migrations.xml`)

Place at src/main/resources/migrations.xml:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">

    <!-- Each changeSet = one atomic database change -->
    <!-- id + author combination must be globally unique! -->

    <changeSet id="1" author="mitri">
        <createTable tableName="bookmarks">
            <column name="id" type="BIGINT" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="url" type="VARCHAR(500)">
                <constraints nullable="false"/>
            </column>
            <column name="title" type="VARCHAR(255)"/>
            <column name="tag" type="VARCHAR(100)"/>
        </createTable>
        <!-- Liquibase can auto-generate rollback for createTable (just DROP TABLE) -->
    </changeSet>

    <changeSet id="2" author="mitri">
        <addColumn tableName="bookmarks">
            <column name="created_at" type="TIMESTAMP" defaultValueDate="CURRENT_TIMESTAMP"/>
        </addColumn>
    </changeSet>

    <changeSet id="3" author="mitri">
        <!-- Insert test data — must specify manual rollback! -->
        <insert tableName="bookmarks">
            <column name="url" value="https://dropwizard.io"/>
            <column name="title" value="Dropwizard Official Site"/>
            <column name="tag" value="java"/>
        </insert>
        <rollback>
            <!-- Tell Liquibase how to undo this changeSet -->
            <delete tableName="bookmarks">
                <where>url = 'https://dropwizard.io'</where>
            </delete>
        </rollback>
    </changeSet>

</databaseChangeLog>

🚀 Running Migrations

# Check current state of your database
java -jar my-api.jar db status config.yml

# Apply all pending migrations
java -jar my-api.jar db migrate config.yml

# Apply migrations up to a specific tag
java -jar my-api.jar db migrate --include-tag mytag config.yml

# Roll back last 1 change
java -jar my-api.jar db rollback --count 1 config.yml

# Danger zone! Drops everything. Only for dev!
java -jar my-api.jar db drop-all --confirm-delete-everything config.yml

🆚 Liquibase vs Flyway

Feature	Liquibase	Flyway
Migration format	XML, YAML, SQL, JSON	SQL (primarily)
Rollbacks	Built-in support	Manual in Flyway Pro
Author tracking	Yes (prevents conflicts)	No
DB independence	Yes (XML format)	SQL-only = DB specific
Dropwizard default	✅ Yes	Via third-party bundle

Why Liquibase wins for teams: The author field prevents naming collisions. Two developers can't accidentally create conflicting migrations.

Chapter 9: Authentication

🔐 Dropwizard's Two Built-in Auth Methods

Basic Auth — Username + Password sent with every request
OAuth 2.0 — Token-based auth (used by most modern APIs)

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-auth</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

👤 Step 1 — The User Class

Must extend java.security.Principal in Dropwizard 1.x:

package com.example.auth;

import java.security.Principal;

public class User implements Principal {
    private final String name;

    public User(String name) {
        this.name = name;
    }

    @Override
    public String getName() {
        return name;
    }
}

🔑 Step 2 — The Authenticator

package com.example.auth;

import io.dropwizard.auth.AuthenticationException;
import io.dropwizard.auth.Authenticator;
import io.dropwizard.auth.basic.BasicCredentials;
import java.util.Optional;

public class MyAuthenticator implements Authenticator<BasicCredentials, User> {

    private final String validLogin;
    private final String validPassword;

    public MyAuthenticator(String login, String password) {
        this.validLogin = login;
        this.validPassword = password;
    }

    @Override
    public Optional<User> authenticate(BasicCredentials credentials)
            throws AuthenticationException {

        // Check if credentials match
        if (validLogin.equals(credentials.getUsername()) &&
            validPassword.equals(credentials.getPassword())) {
            return Optional.of(new User(credentials.getUsername()));
        }

        // Return empty Optional = authentication failed
        // Dropwizard will return 401 Unauthorized automatically
        return Optional.empty();
    }
}

🏭 Step 3 — Register in Application

@Override
public void run(MyApiConfiguration config, Environment environment) {

    // Create the authenticator with credentials from config
    MyAuthenticator authenticator = new MyAuthenticator(
        config.getLogin(),
        config.getPassword()
    );

    // Register Basic Auth with the authenticator
    environment.jersey().register(
        new AuthDynamicFeature(
            new BasicCredentialAuthFilter.Builder<User>()
                .setAuthenticator(authenticator)
                .setRealm("My Secure API")  // Shown in browser login dialog
                .buildAuthFilter()
        )
    );

    // Enable @Auth annotation injection in resources
    environment.jersey().register(new AuthValueFactoryProvider.Binder<>(User.class));
}

🔒 Step 4 — Protect Your Endpoints

@GET
@Path("/{id}")
@UnitOfWork
// @Auth injects the authenticated user AND protects the endpoint
public Bookmark getBookmark(@Auth User user,  // user = logged in user
                             @PathParam("id") long id) {
    // Only authenticated users reach here
    return dao.findById(id)
              .orElseThrow(() -> new NotFoundException("Not found: " + id));
}

📡 Making a Request with Basic Auth

# Using curl — -u "username:password"
curl -u "admin:secret123" http://localhost:8080/bookmarks/1

# With HTTPS (ignore self-signed cert for dev)
curl -k -u "admin:secret123" https://localhost:8443/bookmarks

# Or manually with header
curl -H "Authorization: Basic YWRtaW46c2VjcmV0MTIz" http://localhost:8080/bookmarks

Chapter 10: Health Checks & Metrics

💓 Health Checks

Health checks answer: "Is my app healthy right now?"

The admin port (8081) exposes:

/ping → returns "pong" if app is up
/healthcheck → runs all registered health checks

Built-in Health Checks

Dropwizard provides one by default:

DeadlockHealthCheck — checks if JVM has thread deadlocks

When you add the Hibernate bundle, you get a free extra:

Database connectivity check — tries to ping the database

Writing a Custom Health Check

package com.example.health;

import com.codahale.metrics.health.HealthCheck;

public class ExternalServiceHealthCheck extends HealthCheck {

    private final String serviceUrl;

    public ExternalServiceHealthCheck(String serviceUrl) {
        this.serviceUrl = serviceUrl;
    }

    @Override
    protected Result check() throws Exception {
        // Try to call the external service
        try {
            // ... make HTTP call to serviceUrl ...
            boolean isReachable = pingService(serviceUrl);

            if (isReachable) {
                return Result.healthy("External service is UP");
            } else {
                return Result.unhealthy("External service is DOWN at: " + serviceUrl);
            }
        } catch (Exception e) {
            return Result.unhealthy("Exception connecting: " + e.getMessage());
        }
    }

    private boolean pingService(String url) {
        // Simplified — use OkHttp (Chapter 15) in real code
        return true;
    }
}

Registering a Health Check

@Override
public void run(MyApiConfiguration config, Environment environment) {
    // Register your custom health check
    environment.healthChecks().register(
        "external-service",
        new ExternalServiceHealthCheck("https://api.thirdparty.com/ping")
    );
}

Response When Healthy

curl http://localhost:8081/healthcheck

{
  "deadlocks": { "healthy": true },
  "hibernate": { "healthy": true },
  "external-service": { "healthy": true, "message": "External service is UP" }
}

Response When Unhealthy (returns 500)

{
  "external-service": {
    "healthy": false,
    "message": "External service is DOWN at: https://api.thirdparty.com/ping"
  }
}

📊 Metrics

Dropwizard's Metrics library lets you measure things in your app:

import com.codahale.metrics.*;

public class BookmarkResource {
    private final Meter requests;      // How many requests per second
    private final Timer latency;       // How long requests take
    private final Counter activeUsers; // Count something
    private final Histogram payloadSize; // Distribution of values

    public BookmarkResource(MetricRegistry metrics, BookmarkDAO dao) {
        this.requests = metrics.meter("bookmark.requests");
        this.latency = metrics.timer("bookmark.latency");
        this.activeUsers = metrics.counter("bookmark.active-users");
    }

    @GET
    public List<Bookmark> getAllBookmarks() {
        requests.mark();  // Record a request happened
        try (Timer.Context ctx = latency.time()) {  // Measure how long this takes
            return dao.findAll();
        }
    }
}

📈 Sending Metrics to Graphite

# config.yml
metrics:
  reporters:
    - type: graphite
      host: graphite.example.com
      port: 2003
      prefix: myapp
      frequency: 1 minute

Chapter 11: Testing in Dropwizard

🧪 Testing Philosophy

Dropwizard takes testing seriously and provides special tools that make it easy to test REST APIs without running a full server.

📦 Dependency

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-testing</artifactId>
    <version>${dropwizard.version}</version>
    <scope>test</scope>
</dependency>
<!-- Note: JUnit is included automatically! No need to add it separately. -->

🧩 Level 1: Unit Testing a Resource (ResourceTestRule)

Tests just your resource class + Jersey, without a real database:

package com.example.resources;

import io.dropwizard.testing.junit.ResourceTestRule;
import org.junit.*;
import static org.mockito.Mockito.*;
import static org.assertj.core.api.Assertions.*;
import javax.ws.rs.core.Response;
import java.util.List;

public class BookmarkResourceTest {

    // Mock the DAO — no real database needed!
    private static final BookmarkDAO dao = mock(BookmarkDAO.class);

    // Spins up an in-memory Jersey server for testing
    @ClassRule
    public static final ResourceTestRule resources = ResourceTestRule.builder()
            .addResource(new BookmarkResource(dao))
            .build();

    @Before
    public void setup() {
        // Define what the mock returns
        when(dao.findAll()).thenReturn(List.of(new Bookmark(1L, "https://example.com", "Example")));
    }

    @After
    public void tearDown() {
        reset(dao);  // Reset mock between tests
    }

    @Test
    public void testGetAllBookmarks() {
        // Make a GET request using the test client
        List<Bookmark> result = resources.client()
                .target("/bookmarks")
                .request()
                .get(new GenericType<List<Bookmark>>(){});

        assertThat(result).hasSize(1);
        assertThat(result.get(0).getUrl()).isEqualTo("https://example.com");
        verify(dao).findAll();  // Verify DAO was called
    }

    @Test
    public void testCreateBookmark_returns201() {
        Bookmark newBookmark = new Bookmark(0, "https://new.com", "New Site");
        when(dao.save(any(Bookmark.class))).thenReturn(newBookmark);

        Response response = resources.client()
                .target("/bookmarks")
                .request()
                .post(Entity.json(newBookmark));

        assertThat(response.getStatus()).isEqualTo(201);
    }

    @Test
    public void testGetBookmark_notFound_returns404() {
        when(dao.findById(99L)).thenReturn(Optional.empty());

        Response response = resources.client()
                .target("/bookmarks/99")
                .request()
                .get();

        assertThat(response.getStatus()).isEqualTo(404);
    }
}

🔗 Level 2: Integration Testing (DropwizardAppRule)

Starts the entire application — tests everything together:

package com.example;

import io.dropwizard.testing.junit.DropwizardAppRule;
import org.junit.*;
import javax.ws.rs.client.*;
import javax.ws.rs.core.Response;

public class BookmarkIntegrationTest {

    // Starts the ENTIRE app (real server, real database)
    @ClassRule
    public static final DropwizardAppRule<MyApiConfiguration> RULE =
        new DropwizardAppRule<>(
            MyApiApplication.class,
            "config-test.yml"  // Use test config (H2 in-memory DB)
        );

    private Client client;

    @Before
    public void setup() {
        client = ClientBuilder.newClient();
    }

    @Test
    public void testServerIsRunning() {
        Response response = client
            .target(String.format("http://localhost:%d/bookmarks",
                RULE.getLocalPort()))
            .request()
            .get();

        assertThat(response.getStatus()).isEqualTo(200);
    }

    @Test
    public void testAuthenticatedEndpoint() {
        Response response = client
            .target(String.format("https://localhost:%d/bookmarks",
                RULE.getLocalPort()))
            .request()
            // Add Basic Auth header
            .header("Authorization", "Basic " + Base64.encode("admin:secret"))
            .get();

        assertThat(response.getStatus()).isEqualTo(200);
    }
}

🗃️ Testing Database Queries (DAOTest)

public class BookmarkDAOTest {

    // Spins up real Hibernate with H2 in-memory database
    @ClassRule
    public static final DAOTestRule database = DAOTestRule.newBuilder()
        .addEntityClass(Bookmark.class)
        .build();

    private BookmarkDAO dao;

    @Before
    public void setUp() throws Exception {
        dao = new BookmarkDAO(database.getSessionFactory());
    }

    @Test
    public void testSaveAndFind() {
        Bookmark saved = database.inTransaction(() -> {
            Bookmark b = new Bookmark();
            b.setUrl("https://test.com");
            b.setTitle("Test");
            return dao.save(b);
        });

        assertThat(saved.getId()).isGreaterThan(0);
        assertThat(dao.findById(saved.getId())).isPresent();
    }
}

Chapter 12: Lombok – Boilerplate Killer

😤 The Problem

Java is verbose. For every field in a class, you write:

A getter
A setter
Constructor(s)
equals() and hashCode()
toString()

That's 30+ lines of boring code for a simple 5-field class.

✨ Lombok's Solution

Annotations that generate all that code at compile time. You write the fields. Lombok writes the rest.

📦 Dependency

<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.30</version>
    <scope>provided</scope>  <!-- Only needed at compile time -->
</dependency>

Also install the Lombok plugin in IntelliJ IDEA (Settings → Plugins → search "Lombok").

🏷️ Core Lombok Annotations

import lombok.*;
import lombok.extern.slf4j.Slf4j;

// @Data = @Getter + @Setter + @ToString + @EqualsAndHashCode + @RequiredArgsConstructor
@Data
public class Bookmark {
    private long id;
    private String url;
    private String title;
}
// Lombok generates: getters, setters, toString(), equals(), hashCode(), constructor

Breakdown of each annotation:

@Getter              // Generates getters for all fields
@Setter              // Generates setters for all fields
@ToString            // Generates toString() like: Bookmark(id=1, url=https://...)
@EqualsAndHashCode   // Generates equals() and hashCode() based on fields
@NoArgsConstructor   // Generates: public Bookmark() {}
@AllArgsConstructor  // Generates: public Bookmark(long id, String url, String title) {}
@RequiredArgsConstructor  // Constructor for @NonNull and final fields only
public class Bookmark {
    private long id;
    private String url;
    private String title;
}

🏗️ @builder Pattern

Immutable objects with a fluent builder API:

@Builder
@Getter  // @Builder doesn't auto-generate getters
public class BookmarkRequest {
    private final String url;
    private final String title;
    private final String tag;
}

// Usage:
BookmarkRequest request = BookmarkRequest.builder()
    .url("https://example.com")
    .title("Example")
    .tag("java")
    .build();

🔒 Immutable Objects with @Value

// @Value = immutable version of @Data
// All fields are private final, only getters generated (no setters!)
@Value
public class UserId {
    long id;
    String username;
}

📝 Logging with @Slf4j

@Slf4j  // Generates: private static final Logger log = LoggerFactory.getLogger(BookmarkResource.class);
public class BookmarkResource {

    @GET
    public List<Bookmark> getAll() {
        log.info("Fetching all bookmarks");  // Use log directly!
        log.debug("Debug details here");
        log.error("Something went wrong", exception);
        return dao.findAll();
    }
}

⚠️ Lombok + Jackson Integration

Jackson needs a default constructor for deserialization. Use both:

@Data
@NoArgsConstructor          // Jackson needs this
@AllArgsConstructor         // Useful for your code
@JsonIgnoreProperties(ignoreUnknown = true)
public class Bookmark {
    private long id;
    private String url;
    private String title;
}

Or use @Builder with @Jacksonized:

@Builder
@Jacksonized  // Tells Jackson to use the builder for deserialization
@Value
public class Bookmark {
    long id;
    String url;
    String title;
}

Chapter 13: MapStruct – Object Mapping

🗺️ The Mapping Problem

In real applications, you have different object types:

Entity — maps to database (has Hibernate annotations)
DTO (Data Transfer Object) — what you send/receive in API
Domain Model — internal business logic object

You need to convert between them constantly. Manually writing conversion code is error-prone and tedious.

// Manual mapping — boring, error-prone
public BookmarkDTO toDTO(Bookmark entity) {
    BookmarkDTO dto = new BookmarkDTO();
    dto.setId(entity.getId());
    dto.setUrl(entity.getUrl());
    dto.setTitle(entity.getTitle());
    // Forget one field and you have a bug!
    return dto;
}

✨ MapStruct Solution

MapStruct generates these conversion methods at compile time using annotations.

📦 Dependency

<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct</artifactId>
    <version>1.5.5.Final</version>
</dependency>
<dependency>
    <groupId>org.mapstruct</groupId>
    <artifactId>mapstruct-processor</artifactId>
    <version>1.5.5.Final</version>
    <scope>provided</scope>
</dependency>

🔧 Maven Plugin (required with Lombok)

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <annotationProcessorPaths>
            <path>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
                <version>1.18.30</version>
            </path>
            <path>
                <groupId>org.mapstruct</groupId>
                <artifactId>mapstruct-processor</artifactId>
                <version>1.5.5.Final</version>
            </path>
        </annotationProcessorPaths>
    </configuration>
</plugin>

📦 Define Your Objects

// Database Entity
@Entity
@Data
public class BookmarkEntity {
    @Id @GeneratedValue
    private Long id;
    private String url;
    private String title;
    private String tag;
    private Date createdAt;
    private String internalNotes;  // Should NOT be in API response!
}

// API DTO (Data Transfer Object)
@Data
@NoArgsConstructor
public class BookmarkDTO {
    private Long id;
    private String url;
    private String title;
    private String tag;
    // Notice: no internalNotes — we don't expose this!
    private String createdDate;  // Different format than entity!
}

🗺️ The Mapper Interface

package com.example.mapper;

import org.mapstruct.*;

@Mapper(componentModel = "default")  // Use "cdi" for CDI, "spring" for Spring
public interface BookmarkMapper {

    // Simple mapping — same field names map automatically!
    BookmarkDTO toDTO(BookmarkEntity entity);

    // Map from DTO back to entity
    BookmarkEntity toEntity(BookmarkDTO dto);

    // Custom field mapping
    @Mapping(source = "createdAt", target = "createdDate",
             dateFormat = "yyyy-MM-dd")  // Convert Date to formatted String
    @Mapping(target = "internalNotes", ignore = true)  // Don't map this field
    BookmarkDTO toDTOCustom(BookmarkEntity entity);

    // Map a list automatically!
    List<BookmarkDTO> toDTOList(List<BookmarkEntity> entities);
}

🏭 Getting the Mapper Instance

// Approach 1: Factory method (no DI)
BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);

// Approach 2: With Guice (Chapter 14)
// Bind in your module, inject where needed

✅ Using the Mapper in Resource

@Path("/bookmarks")
public class BookmarkResource {
    private final BookmarkDAO dao;
    private final BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);

    @GET
    @UnitOfWork
    public List<BookmarkDTO> getAllBookmarks() {
        List<BookmarkEntity> entities = dao.findAll();
        return mapper.toDTOList(entities);  // Entity → DTO, one line!
    }

    @POST
    @UnitOfWork
    public BookmarkDTO createBookmark(@Valid BookmarkDTO dto) {
        BookmarkEntity entity = mapper.toEntity(dto);  // DTO → Entity
        BookmarkEntity saved = dao.save(entity);
        return mapper.toDTO(saved);  // Entity → DTO for response
    }
}

Chapter 14: Google Guice – Dependency Injection

🤔 The Problem

As your app grows, you end up with this mess in your run() method:

public void run(Config config, Environment env) {
    HibernateBundle hib = hibernate;
    BookmarkDAO dao = new BookmarkDAO(hib.getSessionFactory());
    BookmarkMapper mapper = Mappers.getMapper(BookmarkMapper.class);
    MyAuthenticator auth = new MyAuthenticator(config.getLogin(), config.getPassword());
    BookmarkResource resource = new BookmarkResource(dao, mapper, auth);
    env.jersey().register(resource);
    // What if BookmarkResource needs 5 more dependencies?!
}

Every class manually creates its dependencies. Change one thing → change everywhere.

✨ Guice Solution

Inversion of Control: Instead of classes creating their dependencies, Guice creates them and injects them where needed.

📦 Dependency

<dependency>
    <groupId>com.google.inject</groupId>
    <artifactId>guice</artifactId>
    <version>5.1.0</version>
</dependency>
<!-- Dropwizard + Guice integration -->
<dependency>
    <groupId>ru.vyarus</groupId>
    <artifactId>dropwizard-guicey</artifactId>
    <version>5.7.1</version>
</dependency>

📝 Step 1 — Mark Dependencies with @Inject

// BookmarkDAO depends on SessionFactory
public class BookmarkDAO extends AbstractDAO<BookmarkEntity> {

    @Inject  // Guice will provide SessionFactory
    public BookmarkDAO(SessionFactory factory) {
        super(factory);
    }
}

// BookmarkResource depends on DAO and Mapper
public class BookmarkResource {

    private final BookmarkDAO dao;
    private final BookmarkMapper mapper;

    @Inject  // Guice will inject both!
    public BookmarkResource(BookmarkDAO dao, BookmarkMapper mapper) {
        this.dao = dao;
        this.mapper = mapper;
    }
}

📋 Step 2 — The Guice Module (Wiring Configuration)

package com.example;

import com.google.inject.AbstractModule;
import com.google.inject.Provides;
import com.google.inject.Singleton;

public class MyModule extends AbstractModule {

    @Override
    protected void configure() {
        // Bind interface → implementation
        // "Whenever someone asks for UserService, give them UserServiceImpl"
        bind(UserService.class).to(UserServiceImpl.class).in(Singleton.class);

        // Bind a class directly
        bind(BookmarkDAO.class).in(Singleton.class);

        // Bind mapper
        bind(BookmarkMapper.class).toInstance(Mappers.getMapper(BookmarkMapper.class));
    }

    // @Provides methods create instances that need configuration
    @Provides
    @Singleton
    public BookmarkDAO provideBookmarkDAO(SessionFactory factory) {
        return new BookmarkDAO(factory);
    }
}

🏭 Step 3 — Register with Dropwizard

Using dropwizard-guicey:

public class MyApiApplication extends GuiceApplication<MyApiConfiguration> {

    @Override
    public void initialize(Bootstrap<MyApiConfiguration> bootstrap) {
        super.initialize(bootstrap);
        bootstrap.addBundle(hibernate);
    }

    @Override
    public void run(MyApiConfiguration config, Environment environment) {
        // Resources are auto-discovered and injected!
    }

    @Override
    protected Injector createInjector(MyApiConfiguration config) {
        return Guice.createInjector(new MyModule());
    }
}

🏷️ Key Guice Annotations

@Inject     // Mark constructor/field/method for injection
@Singleton  // One instance for entire app lifetime
@Named("dbUrl")  // Disambiguate when multiple bindings of same type exist

// Usage:
@Inject @Named("dbUrl") String databaseUrl;

🔄 Guice Scopes

bind(BookmarkDAO.class).in(Singleton.class);    // One instance per app
bind(BookmarkDAO.class).in(RequestScoped.class); // New instance per HTTP request
bind(BookmarkDAO.class);                         // Default: new instance every time

Chapter 15: OkHttp – HTTP Client

🌐 Why OkHttp?

Microservices talk to each other over HTTP. You need an HTTP client to:

Call third-party APIs (payment, email, maps)
Call other microservices in your system

OkHttp is the most popular Java HTTP client — fast, simple, feature-rich.

📦 Dependency

<dependency>
    <groupId>com.squareup.okhttp3</groupId>
    <artifactId>okhttp</artifactId>
    <version>4.11.0</version>
</dependency>

🚀 Basic GET Request

import okhttp3.*;
import java.io.IOException;

public class ExternalApiClient {

    // IMPORTANT: Reuse the same OkHttpClient — it manages connection pooling!
    private final OkHttpClient client = new OkHttpClient();
    private final ObjectMapper objectMapper = new ObjectMapper();

    public String fetchData(String url) throws IOException {
        // Build the request
        Request request = new Request.Builder()
            .url(url)
            .header("Accept", "application/json")
            .header("Authorization", "Bearer my-token")
            .build();

        // Execute synchronously
        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) {
                throw new IOException("Unexpected response: " + response);
            }
            return response.body().string();
        }
    }
}

📤 POST Request with JSON Body

public static final MediaType JSON = MediaType.get("application/json; charset=utf-8");

public Bookmark createRemoteBookmark(BookmarkDTO dto) throws IOException {
    String json = objectMapper.writeValueAsString(dto);

    Request request = new Request.Builder()
        .url("https://api.other-service.com/bookmarks")
        .post(RequestBody.create(json, JSON))  // POST with JSON body
        .build();

    try (Response response = client.newCall(request).execute()) {
        String responseBody = response.body().string();
        return objectMapper.readValue(responseBody, Bookmark.class);
    }
}

⏱️ Timeouts & Configuration

OkHttpClient client = new OkHttpClient.Builder()
    .connectTimeout(10, TimeUnit.SECONDS)  // How long to wait to connect
    .readTimeout(30, TimeUnit.SECONDS)     // How long to wait for data
    .writeTimeout(15, TimeUnit.SECONDS)    // How long to wait for upload
    .retryOnConnectionFailure(true)        // Retry on network errors
    .build();

🔄 Async Requests

public void fetchDataAsync(String url, Callback callback) {
    Request request = new Request.Builder().url(url).build();

    // Non-blocking — callback is called when done
    client.newCall(request).enqueue(new Callback() {
        @Override
        public void onFailure(Call call, IOException e) {
            log.error("Request failed", e);
        }

        @Override
        public void onResponse(Call call, Response response) throws IOException {
            try (response) {
                String body = response.body().string();
                // Process response
            }
        }
    });
}

🔌 OkHttp Interceptors (Middleware)

Interceptors run for every request — perfect for logging, auth headers, etc.:

// Logging interceptor
OkHttpClient client = new OkHttpClient.Builder()
    .addInterceptor(chain -> {
        Request request = chain.request();
        log.info("→ {} {}", request.method(), request.url());

        long start = System.currentTimeMillis();
        Response response = chain.proceed(request);
        long duration = System.currentTimeMillis() - start;

        log.info("← {} {} ({}ms)", response.code(), request.url(), duration);
        return response;
    })
    .addInterceptor(chain -> {
        // Auth interceptor — add token to every request
        Request authenticated = chain.request().newBuilder()
            .header("Authorization", "Bearer " + getToken())
            .build();
        return chain.proceed(authenticated);
    })
    .build();

Chapter 16: Hystrix – Circuit Breaker

⚡ The Cascading Failure Problem

Service A calls Service B. Service B calls Service C. Service C goes down.

Without circuit breakers:

Service C times out after 30 seconds
Service B waits 30s, then times out
Service A waits 30s too = everything hangs for 30+ seconds!
Threads pile up, memory fills up, your whole system crashes

🔌 The Circuit Breaker Pattern

Like an electrical circuit breaker — when failures exceed a threshold, it "trips" and subsequent calls fail immediately instead of waiting.

States:

CLOSED — Normal operation, calls pass through
OPEN — Too many failures, all calls fail immediately
HALF-OPEN — Testing if service recovered; allows one call through

📦 Dependency

<dependency>
    <groupId>com.netflix.hystrix</groupId>
    <artifactId>hystrix-core</artifactId>
    <version>1.5.18</version>
</dependency>

☕ Wrapping Calls with HystrixCommand

package com.example.hystrix;

import com.netflix.hystrix.*;
import java.io.IOException;

public class FetchBookmarkCommand extends HystrixCommand<String> {

    private final String bookmarkId;
    private final ExternalApiClient apiClient;

    public FetchBookmarkCommand(String bookmarkId, ExternalApiClient apiClient) {
        super(HystrixCommandGroupKey.Factory.asKey("BookmarkService"));
        this.bookmarkId = bookmarkId;
        this.apiClient = apiClient;
    }

    @Override
    protected String run() throws Exception {
        // The actual (potentially failing) call
        return apiClient.fetchData("https://api.service.com/bookmarks/" + bookmarkId);
    }

    @Override
    protected String getFallback() {
        // This runs when:
        // 1. run() throws an exception
        // 2. Circuit is OPEN (previous failures)
        // 3. Timeout occurs
        return "{\"id\":\"" + bookmarkId + "\",\"status\":\"unavailable\"}";
    }
}

🎯 Using the Command

// Synchronous execution
String result = new FetchBookmarkCommand(bookmarkId, client).execute();

// Asynchronous (returns Future)
Future<String> future = new FetchBookmarkCommand(bookmarkId, client).queue();

// Reactive (returns Observable)
Observable<String> observable = new FetchBookmarkCommand(bookmarkId, client).observe();

⚙️ Configuring Hystrix

HystrixCommand.Setter config = HystrixCommand.Setter
    .withGroupKey(HystrixCommandGroupKey.Factory.asKey("BookmarkService"))
    .andCommandKey(HystrixCommandKey.Factory.asKey("FetchBookmark"))
    .andCommandPropertiesDefaults(
        HystrixCommandProperties.Setter()
            .withExecutionTimeoutInMilliseconds(3000)  // 3 second timeout
            .withCircuitBreakerRequestVolumeThreshold(10)  // Min requests before circuit can trip
            .withCircuitBreakerErrorThresholdPercentage(50)  // Trip if 50% fail
            .withCircuitBreakerSleepWindowInMilliseconds(5000)  // Wait 5s before trying again
    );

// Use in command:
public FetchBookmarkCommand(String id, ExternalApiClient client) {
    super(config);  // Pass config to super
    this.bookmarkId = id;
    this.apiClient = client;
}

🏎️ HystrixCommand + OkHttp (Combined Example)

@Slf4j
public class ExternalServiceClient {

    private final OkHttpClient httpClient;
    private final ObjectMapper objectMapper;

    @Inject
    public ExternalServiceClient(OkHttpClient httpClient, ObjectMapper objectMapper) {
        this.httpClient = httpClient;
        this.objectMapper = objectMapper;
    }

    public BookmarkDTO getBookmark(String id) {
        // Wrap the OkHttp call in Hystrix
        return new HystrixCommand<BookmarkDTO>(
            HystrixCommandGroupKey.Factory.asKey("ExternalBookmarks")) {

            @Override
            protected BookmarkDTO run() throws IOException {
                Request request = new Request.Builder()
                    .url("https://external.api.com/bookmarks/" + id)
                    .build();

                try (Response response = httpClient.newCall(request).execute()) {
                    return objectMapper.readValue(response.body().string(), BookmarkDTO.class);
                }
            }

            @Override
            protected BookmarkDTO getFallback() {
                log.warn("Fallback triggered for bookmark: {}", id);
                BookmarkDTO fallback = new BookmarkDTO();
                fallback.setTitle("Unavailable");
                return fallback;
            }
        }.execute();
    }
}

Chapter 17: 🔲 TODO — Advanced Topics to Study Next

These topics weren't covered in this guide but are important for production-ready applications:

🔐 Security

[ ] OAuth 2.0 — Token-based authentication (JWT tokens)
[ ] HTTPS/TLS — Configure SSL in Dropwizard config.yml
[ ] LDAP Authentication — Enterprise authentication via directory services
[ ] CORS — Cross-Origin Resource Sharing configuration

📊 Observability

[ ] Distributed Tracing — Zipkin/Jaeger for tracing requests across microservices
[ ] Prometheus — Modern metrics collection (alternative to Graphite)
[ ] Grafana Dashboards — Visualizing your metrics
[ ] Structured Logging — JSON logs for log aggregation (ELK Stack)
[ ] OpenTelemetry — Unified observability standard

🏗️ Architecture

[ ] Authorization — @RolesAllowed, @PermitAll annotations
[ ] Rate Limiting — Prevent API abuse
[ ] API Versioning — Strategies for versioning your REST API
[ ] HATEOAS — Hypermedia-driven REST (links in responses)
[ ] GraphQL — Alternative to REST for flexible queries
[ ] OpenAPI/Swagger — Auto-generate API documentation

🗄️ Database

[ ] JDBI — Lighter-weight alternative to Hibernate
[ ] Connection Pooling — HikariCP configuration
[ ] Read Replicas — Routing reads vs writes
[ ] Database Sharding — Horizontal scaling

🐳 Deployment

[ ] Docker — Containerizing your Fat JAR
[ ] Kubernetes — Orchestrating containers
[ ] CI/CD — GitHub Actions / Jenkins pipelines
[ ] Blue/Green Deployment — Zero-downtime deployments
[ ] Service Discovery — Consul, Eureka

⚡ Performance

[ ] Async Resources — @Suspended and AsyncResponse in JAX-RS
[ ] Caching — Redis/Memcached with Dropwizard
[ ] Connection Timeout Tuning — Jetty thread pool settings
[ ] GraalVM Native Image — Compile to native binary for faster startup

🧪 Advanced Testing

[ ] Consumer-Driven Contract Testing — Pact framework
[ ] WireMock — Mock external HTTP services in tests
[ ] TestContainers — Real databases in Docker for integration tests
[ ] Chaos Engineering — Testing failure scenarios (Chaos Monkey)

📨 Messaging

[ ] Apache Kafka — Event streaming
[ ] RabbitMQ — Message queuing
[ ] Dropwizard Kafka Bundle — Integration

Understanding Apache Kafka: Topics, Partitions, Delivery Guarantees & Replication 🔥

Ajinkya Singh — Tue, 25 Nov 2025 19:09:21 +0000

Topic Architecture: Building Your Data Highway
Delivery Guarantees: Never Lose, Never Duplicate
Replication & ISR: Zero Downtime Architecture

Topic Architecture: Building Your Data Highway

🎯 The Foundation: Understanding Topics

Think of a Kafka topic as a multi-lane highway where your data travels. Just like how more lanes allow more cars to drive simultaneously, more partitions allow more messages to be processed in parallel.

Creating Your First Topic

/opt/kafka/bin/kafka-topics.sh --create \
  --topic order-events \
  --partitions 4 \
  --replication-factor 3 \
  --bootstrap-server localhost:9092

What just happened?

✅ Created a topic called order-events
✅ Split it into 4 parallel lanes (partitions)
✅ Made 3 copies of each partition (for safety)

🛣️ Partitions: Your Performance Dial

Scenario: E-commerce Order Processing

You run an online store with these requirements:

5,000 orders per second during peak hours
Each processing server handles 1,250 orders/second

Calculate partitions needed:

Partitions = Target Throughput ÷ Consumer Throughput
Partitions = 5,000 ÷ 1,250 = 4 partitions

The Highway Analogy Visualized

1 Partition Topic (Bottleneck):

[═══════════════════════] Single lane - all traffic stuck

4 Partition Topic (Optimal):

[═══════════════════════] Partition 0 → Consumer A
[═══════════════════════] Partition 1 → Consumer B
[═══════════════════════] Partition 2 → Consumer C
[═══════════════════════] Partition 3 → Consumer D

🔑 Message Keys: The Secret to Ordering

The Problem

Without keys, messages for the same customer could process out of order, causing chaos!

❌ Without Keys (Random Distribution):

Customer #42 creates account    → Partition 1
Customer #42 updates email      → Partition 0  ← Different partition!
Customer #42 adds phone         → Partition 2  ← Processed out of order!

✅ With Keys (Guaranteed Order):

Key: "customer-42" → hash → Partition 2

Customer #42 creates account    → Partition 2
Customer #42 updates email      → Partition 2  ← Same partition!
Customer #42 adds phone         → Partition 2  ← Perfect order!

Real-World Example: Social Media Feed

Scenario: Building an Instagram-like feed processor

When users create posts, you need to ensure each user's posts are processed in order, but different users can be processed in parallel.

Key Strategy:

User ID as Key → All posts from same user → Same partition → Guaranteed order

Result: All of Sarah's posts stay in order, but different users' posts can be processed in parallel across partitions!

Key Strategy Decision Tree

Do you need message ordering?
├─ NO  → Use NO KEY (round-robin for max throughput)
│         Perfect for: Metrics, logs, sensor data
│
└─ YES → Use MESSAGE KEY
    ├─ User activity?     → key = user_id
    ├─ Order processing?  → key = order_id
    ├─ IoT devices?       → key = device_id
    └─ Multi-tenant app?  → key = tenant_id

🛡️ Replication: Your Safety Net

Scenario: Banking Application

You're building a payment processor. One question matters most:

"If a server crashes at 2 AM, do we lose transaction records?"

Answer depends on replication factor:

Replication Factor = 1 (❌ NEVER IN PRODUCTION)

Partition 0:
└── Broker 1 (Leader) ← Single point of failure
    If Broker 1 crashes → DATA PERMANENTLY LOST

Replication Factor = 3 (✅ Production Standard)

Partition 0:
├── Broker 1 (Leader)    ← Handles all reads/writes
├── Broker 2 (Follower)  ← Backup copy
└── Broker 3 (Follower)  ← Another backup

Broker 1 crashes? → Broker 2 becomes leader → NO DATA LOSS ✓

Real Failure Scenario: Diwali Sale

Timeline of a broker failure:

11:59 PM - Diwali sale starts, 10,000 orders/second
12:03 AM - Broker 2 crashes (hardware failure)
12:03 AM - Controller detects failure (3 seconds)
12:03 AM - Broker 3 promoted to leader
12:03 AM - Orders continue processing
         - Zero orders lost ✓
         - 3-second interruption only

⚙️ Advanced Configuration: Retention & Durability

Configuration Example: Multi-Tier Data Strategy

Real-time Analytics Topic (High Volume, Short Retention)

/opt/kafka/bin/kafka-topics.sh --create \
  --topic clickstream-events \
  --partitions 16 \
  --replication-factor 3 \
  --config retention.ms=86400000 \  # 1 day
  --config min.insync.replicas=1    # Lower durability for speed

Financial Transactions Topic (Critical, Long Retention)

/opt/kafka/bin/kafka-topics.sh --create \
  --topic payment-transactions \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=7776000000 \  # 90 days
  --config min.insync.replicas=2      # High durability

Event Sourcing Topic (Forever, Maximum Safety)

/opt/kafka/bin/kafka-topics.sh --create \
  --topic account-events \
  --partitions 8 \
  --replication-factor 5 \
  --config retention.ms=-1 \          # Forever
  --config min.insync.replicas=3      # Maximum durability

Delivery Guarantees: Never Lose, Never Duplicate

🎭 The Central Dilemma

In distributed systems, failures are inevitable. The question is:

When your consumer crashes mid-processing, what happens to that message?

You have two choices, each with trade-offs:

📊 At Most Once: "Fire and Forget"

Definition: Process each message 0 or 1 times (never more than once)

Priority: Prevent duplicates at all costs, even if it means losing messages

The Coffee Shop Analogy

You order coffee:
1. Barista writes order ✓
2. ✅ MARKS YOUR ORDER COMPLETE (commits offset)
3. 💥 Power outage before making coffee!
4. Power returns
5. Order marked complete → You never get your coffee

Result: Order processed ZERO times (at most once)

How It Works: Automatic Offset Commit

Consumer Configuration:

enable.auto.commit=true
auto.commit.interval.ms=5000  # Commits every 5 seconds

Processing Flow:

1. Fetch message from Kafka
2. ✅ Auto-commit saves offset (marks as done)
3. Start processing message
4. If crash here → Message LOST

Timeline of Message Loss

10:00:00 - Fetch message "user_clicked_buy_button"
10:00:01 - Auto-commit saves offset 1001 ✓
10:00:02 - Start processing...
10:00:03 - 💥 APPLICATION CRASHES
10:00:04 - Restart
10:00:05 - Ask Kafka: "Where was I?"
10:00:05 - Kafka: "You were at offset 1001"
10:00:06 - Skip to offset 1001
          - Message 1000 NEVER PROCESSED

🎯 At Least Once: "Never Lose Anything"

Definition: Process each message 1 or more times (never zero times)

Priority: Ensure every message is processed, even if it means processing some twice

The Coffee Shop Analogy (Revised)

You order coffee:
1. Barista makes your coffee ✓
2. Serves it to you ✓
3. 💥 Power outage before marking order complete!
4. Power returns
5. Order NOT marked complete → Makes coffee AGAIN
6. You get TWO coffees (one extra)

Result: Order processed TWICE (at least once)

How It Works: Manual Offset Commit

Consumer Configuration:

enable.auto.commit=false  # Manual control

Processing Flow:

1. Fetch message from Kafka
2. Process message successfully ✓
3. Write to database ✓
4. Commit offset manually ✓
5. If crash before step 4 → Message REPROCESSED

Timeline of Duplicate Processing

10:00:00 - Fetch message "order_12345: charge $100"
10:00:01 - Process order successfully ✓
10:00:02 - Charge customer $100 ✓
10:00:03 - Write to database ✓
10:00:04 - 💥 APPLICATION CRASHES (before committing offset)
10:00:05 - Restart
10:00:06 - Ask Kafka: "Where was I?"
10:00:06 - Kafka: "You were at offset 1000"
10:00:07 - Fetch message 1000 AGAIN
10:00:08 - Process order AGAIN
10:00:09 - Charge customer $100 AGAIN ⚠️ (DUPLICATE!)
10:00:10 - Write to database AGAIN
10:00:11 - Commit offset successfully

🎓 Decision Matrix: Which Guarantee to Use?

                    START HERE
                        ↓
            Can you afford to lose messages?
                    ↙       ↘
                  YES        NO
                   ↓          ↓
           AT MOST ONCE   AT LEAST ONCE
                   ↓          ↓
            Use cases:   Can you handle duplicates?
            • Sensors       ↙            ↘
            • Metrics     YES            NO
            • Logs         ↓              ↓
                    Implement      Add deduplication
                    idempotency    • Transaction IDs
                    • Txn IDs      • DB constraints
                    • DB keys      • Processing log

Use Case Decision Table

Scenario	Guarantee	Reasoning
Banking transactions	At Least Once	Cannot lose money transfers
User registrations	At Least Once	Cannot lose new users
E-commerce orders	At Least Once	Cannot lose customer orders
Stock trades	At Least Once	Cannot lose trade records
IoT sensor readings	At Most Once	Losing one reading is acceptable
Application logs	At Most Once	Missing one log entry is okay
Click analytics	At Most Once	Approximate counts are fine
System metrics	At Most Once	Slightly off counts acceptable

⚙️ Configuration Comparison

At Most Once Configuration:

# Consumer config
enable.auto.commit=true
auto.commit.interval.ms=5000

At Least Once Configuration:

# Consumer config
enable.auto.commit=false  # Manual control

Replication And ISR: Zero Downtime Architecture

🏗️ The Architecture: No Single Point of Failure

The Problem Visualized

Without Replication:

Topic: customer-orders
Partition 0 → Broker 1 ← Single copy
              💥 Crashes
              ↓
          🚨 DATA LOST FOREVER
          🚨 Service DOWN

With Replication:

Topic: customer-orders
Partition 0:
├── Broker 1 (Leader)    ← Primary copy
├── Broker 2 (Follower)  ← Backup copy
└── Broker 3 (Follower)  ← Another backup

Broker 1 💥 Crashes
         ↓
Broker 2 becomes Leader (3 seconds)
         ↓
✅ NO DATA LOST
✅ Service continues

🎭 Leaders and Followers

How Replication Works

Every partition has ONE leader and multiple followers:

Partition 0 (Replication Factor = 3):

                    Producers write here
                           ↓
        ┌──────────────────────────────┐
        │  Broker 1 (LEADER)           │
        │  [msg1][msg2][msg3][msg4]    │
        └──────────────────────────────┘
                 ↓           ↓
    ┌────────────┘           └────────────┐
    ↓                                     ↓
┌──────────────────┐           ┌──────────────────┐
│ Broker 2         │           │ Broker 3         │
│ (FOLLOWER)       │           │ (FOLLOWER)       │
│ [msg1][msg2]     │           │ [msg1][msg2]     │
│ [msg3][msg4]     │           │ [msg3][msg4]     │
└──────────────────┘           └──────────────────┘
    Actively                        Actively
    replicating                     replicating

Key Rules:

All writes go through the leader
All reads go through the leader
Followers continuously pull new data from leader
Followers do NOT serve client requests

🎯 In-Sync Replicas (ISR): The Safety Net

ISR = A follower that is fully caught up and healthy

What Makes a Replica "In-Sync"?

✅ IN-SYNC REPLICA:
   • Has all committed messages
   • Actively fetching from leader
   • Not fallen behind (< replica.lag.time.max.milliseconds)
   • Ready to become leader at any moment

❌ OUT-OF-SYNC REPLICA:
   • Missing recent messages
   • Stopped fetching (network issue, crashed)
   • Fallen too far behind
   • CANNOT become leader

Real-World Analogy: Emergency Contact List

You have 3 emergency contacts:

Contact 1 (Leader): Mom
  • Always answers immediately
  • Has all current information

Contact 2 (ISR): Dad
  • Answers within seconds
  • Fully updated on family matters
  • Can step in if Mom unavailable ✓

Contact 3 (Out-of-Sync): Cousin
  • Takes hours to respond
  • Out of the loop
  • Can't be relied on in emergency ✗

🔒 Message Commitment: The Two-Phase Process

Phase 1: Leader Writes

Producer sends: "Transfer $500 from Alice to Bob"
        ↓
Leader writes to local log
        ↓
Message is "uncommitted" (not safe yet)

Phase 2: ISRs Acknowledge

Leader notifies followers:
        ↓
Follower 1 (ISR) replicates ✓
        ↓
Follower 2 (ISR) replicates ✓
        ↓
Message now "committed" (safe!)
        ↓
Producer receives ACK

Producer Configuration Impact

acks=1 (Fast but Risky):

Producer → Leader writes → ACK immediately
           (Followers not yet replicated)

Leader crashes before replication?
→ Message LOST 💥

acks=all (Slow but Safe):

Producer → Leader writes → Wait for ALL ISRs → ACK

Takes longer, but:
→ Message GUARANTEED safe ✅
→ Zero data loss

🚨 Failure Scenario: Leader Election

Real-World Example: Diwali Sale at 2 AM

Initial State:

Topic: flash-sale-orders
Partition 0:
├── Broker 1 (Leader)    ← 50,000 orders/sec
│   ISR: [1, 2, 3]
├── Broker 2 (Follower)  ← In-Sync
└── Broker 3 (Follower)  ← In-Sync

Failure Timeline:

02:00:00 - Broker 1 crashes (power supply failure)
02:00:01 - Producer attempts write → Connection refused
02:00:02 - Zookeeper detects missing heartbeat
02:00:03 - Controller initiates leader election
           Candidates: Broker 2, Broker 3 (both ISR)
02:00:03 - Broker 2 elected as new leader
02:00:04 - Controller notifies all brokers
02:00:04 - Producer auto-discovers new leader
02:00:05 - Orders resume processing ✅

Total downtime: 5 seconds
Orders lost: ZERO (because acks=all)

Why Was Broker 2 Chosen?

✅ Was in ISR (fully caught up)
✅ Had all committed messages
✅ First in preferred replica list

What If Broker 2 Was Out-of-Sync?

02:00:03 - Controller checks ISR: [1, 3]
02:00:03 - Broker 2 NOT in ISR → SKIPPED
02:00:03 - Broker 3 elected instead

⚙️ Configuration Deep Dive

1. Replication Factor (Topic Level)

# Development: Fast, no safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic dev-logs \
  --replication-factor 1  # ❌ Never in production!

# Staging: Moderate safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic staging-events \
  --replication-factor 2

# Production: Standard safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic production-orders \
  --replication-factor 3  # ✅ Industry standard

# Mission-Critical: Maximum safety
/opt/kafka/bin/kafka-topics.sh --create \
  --topic financial-transactions \
  --replication-factor 5

2. Min In-Sync Replicas (Topic Level)

Works with acks=all on producer side:

# Weak durability (fast but risky)
--config min.insync.replicas=1
# Leader only needs to write
# If leader crashes before followers replicate → DATA LOST

# Strong durability (recommended)
--config min.insync.replicas=2
# Leader + at least 1 follower must acknowledge
# Can survive 1 broker failure

# Maximum durability (mission-critical)
--config min.insync.replicas=3
# Leader + at least 2 followers must acknowledge
# Can survive 2 broker failures

3. Replica Lag Time (Broker Level)

# In server.properties
replica.lag.time.max.milliseconds=10000  # 10 seconds

# If follower doesn't fetch within 10 seconds:
# → Removed from ISR
# → Cannot become leader
# → Logs warning

Understanding Kafka Storage Architecture: How It Handles Billions of Messages⚡

Ajinkya Singh — Mon, 24 Nov 2025 19:56:31 +0000

🎯 Introduction: Beyond the Message Queue

Imagine you're running a global streaming platform processing millions of viewer events every second—play, pause, skip, like, and watch history. How do you store this tsunami of data efficiently while keeping it instantly accessible? This is where Kafka's storage architecture becomes a masterpiece of engineering.

🏗️ The Three Core Responsibilities

Every Kafka broker juggles three critical tasks simultaneously:

1. 📥 Producer Gateway

Accepts incoming streams of events from applications across your network

2. 💾 Storage Engine

Writes messages to disk durably and efficiently—this is where the magic happens

3. 📤 Consumer Server

Rapidly locates and delivers data to consumers while replicating to other brokers

📊 The Storage Hierarchy

Topic: "viewer-activity"
│
├── Partition 0 (folder: /data/viewer-activity-0/)
│   ├── 00000000000000000000.log
│   ├── 00000000000000000000.index
│   ├── 00000000000000000000.timeindex
│   ├── 00000000000000850000.log
│   ├── 00000000000000850000.index
│   ├── 00000000000000850000.timeindex
│   └── 00000000000001700000.log (active segment)
│
├── Partition 1 (folder: /data/viewer-activity-1/)
│   └── [similar segment files...]
│
└── Partition 2 (folder: /data/viewer-activity-2/)
    └── [similar segment files...]

🧩 Why Segments? The Problem with Giant Files

Imagine storing all viewer events in one massive file:

❌ A 15TB file is impossible to manage

❌ Deleting old watch history requires rewriting everything

❌ Finding "what did user watch at 3 PM yesterday" means scanning terabytes

❌ File corruption destroys all your data

The Segment Solution:

Kafka divides each partition's log into segments—smaller, manageable chunks (typically 1GB each).

🔄 When Does Kafka Create a New Segment?

A new segment rolls when either condition is met:

Condition 1: Size Threshold

Current segment reaches 1GB (default)
→ Close it, start fresh segment

Condition 2: Time Threshold

7 days have passed (default)
→ Roll to new segment regardless of size

📅 Real-World Timeline Example:

Monday 9:00 AM  → Users start watching shows
                → Messages written to segment 0

Monday 2:30 PM  → Segment 0 hits 1GB
                → Kafka creates segment 1
                → New events go to segment 1

Tuesday 3:00 PM → Segment 1 hits 1GB
                → Kafka creates segment 2 (now active)

🗂️ Segment File Anatomy

For each segment, Kafka maintains three files:

Partition Folder: viewer-activity-0/
│
├── 00000000000000850000.log        (1GB - actual messages)
├── 00000000000000850000.index      (10MB - offset lookup map)
└── 00000000000000850000.timeindex  (10MB - timestamp lookup map)

File Naming Convention

The number 00000000000000850000 is the base offset—the first message's offset in this segment.

⚡ Index Files: Kafka's Speed Secret

🎯 The Two-Step Lookup Process

When Kafka needs to find a message, it uses TWO files in sequence:

First: Check the segment file name (base offset) to find the RIGHT segment
Second: Use the .index file to find the EXACT location within that segment

Let's see this in action!

🔍 1. Offset Index (.index)

Maps message offset → byte position in the log file

How It Works:

Consumer Request: "Give me messages starting from offset 850,125"

Step 1: FIND THE RIGHT SEGMENT (using base offsets)
        Available segments:
        • 00000000000000000000.log (base: 0)
        • 00000000000000850000.log (base: 850,000) ← THIS ONE!
        • 00000000000001700000.log (base: 1,700,000)

        Offset 850,125 falls between 850,000 and 1,700,000
        → Select segment: 00000000000000850000.log

Step 2: FIND EXACT POSITION (using .index file)
        Broker loads 00000000000000850000.index into memory
        Binary search finds:
        • offset 850,100 → byte 3072
        • offset 850,150 → byte 6144

        Message 850,125 must be between bytes 3072-6144

Step 3: READ THE MESSAGE
        Read from byte 3072 in 00000000000000850000.log
        Scan forward until offset 850,125 is found

Index Structure:

┌──────────┬─────────────┐
│  Offset  │  Byte Pos   │
├──────────┼─────────────┤
│  850000  │      0      │
│  850100  │   3072      │
│  850150  │   6144      │
│  850200  │   9216      │
│   ...    │    ...      │
└──────────┴─────────────┘

Result: ✨ No scanning millions of messages—instant lookup!

📋 Visual: The Two-File Lookup System

🎯 FINDING MESSAGE BY OFFSET (850,125)

File System View:
├── 00000000000000000000.log      (offsets: 0 - 849,999)
├── 00000000000000000000.index
├── 00000000000000850000.log      (offsets: 850,000 - 1,699,999) ⭐
├── 00000000000000850000.index    ⭐ USE THIS!
├── 00000000000001700000.log      (offsets: 1,700,000+)
└── 00000000000001700000.index

Step 1: Match offset to segment name (base offset)
        850,125 is >= 850,000 and < 1,700,000
        → Open segment: 00000000000000850000

Step 2: Use that segment's .index file
        → Open: 00000000000000850000.index
        → Find byte position: 3072
        → Read from .log file at byte 3072

🔑 Key Insight: Why Two Files?

Without segment files (base offset in filename):

❌ Would need to check EVERY index file
❌ Open thousands of files to find the right one
❌ Very slow!

With segment files (base offset in filename):

✅ Filename tells you the offset range instantly
✅ Only open ONE .index file
✅ Lightning fast!

The Two-Step Magic:

Segment filename (base offset) = Coarse filter (which file?)
Index file (.index) = Fine filter (which byte position?)

⏰ 2. Time Index (.timeindex)

Maps timestamp → corresponding offset (also uses two-step lookup!)

How It Works:

Consumer Request: "Show me all viewer activity from the last hour"
                  (timestamp: 2025-11-25 13:00:00)

Step 1: FIND THE RIGHT SEGMENT (using .timeindex files)
        Check all segments' time ranges:
        • Segment 0: timestamps 10:00 - 11:59
        • Segment 850000: timestamps 12:00 - 13:59 ← THIS ONE!
        • Segment 1700000: timestamps 14:00 onwards

        Timestamp 13:00:00 falls in segment 850000

Step 2: FIND THE OFFSET (using .timeindex)
        Broker checks 00000000000000850000.timeindex
        Binary search finds:
        2025-11-25 13:00:00 → offset 850,750

Step 3: USE OFFSET INDEX (now it's a regular offset lookup!)
        Uses 00000000000000850000.index
        Finds: offset 850,750 → byte position 225,280

Step 4: READ THE MESSAGES
        Reads from byte 225,280 in .log file
        Returns all messages from that point forward

Time Index Structure:

┌──────────────────────┬──────────┐
│     Timestamp        │  Offset  │
├──────────────────────┼──────────┤
│ 2025-11-25 10:00:00  │  850000  │
│ 2025-11-25 11:30:00  │  850300  │
│ 2025-11-25 13:00:00  │  850750  │
│ 2025-11-25 14:15:00  │  851000  │
│        ...           │   ...    │
└──────────────────────┴──────────┘

🧠 The Memory Advantage

Why are index lookups so blazing fast?

Disk Storage (Slow):
├── segment.log (1GB - rarely loaded fully)
├── segment.index (10MB - loaded into RAM!)
└── segment.timeindex (10MB - loaded into RAM!)

Memory (Lightning Fast):
└── Index files cached → Microsecond lookups

Because index files are tiny (10-20MB), they fit entirely in memory:

✅ No disk reads for lookups
✅ Binary search through in-memory structures
✅ Microseconds instead of seconds

🗑️ Retention Policies: Managing Disk Space

Without cleanup, your disks would eventually overflow. Kafka automatically deletes old segments based on policies.

1. Time-Based Retention

Configuration:
log.retention.hours = 168  (7 days)

Timeline:
Day 1  → Segment 0 created (Nov 18 viewer data)
Day 2  → Segment 1 created (Nov 19 viewer data)
...
Day 8  → Segment 0 is now 7 days old
       → Kafka deletes:
         - segment-0.log
         - segment-0.index
         - segment-0.timeindex

Visual:

Before Cleanup (Day 8):
[Seg0: Nov 18] [Seg1: Nov 19] [Seg2: Nov 20] ... [Seg7: Nov 25]
      ↓ 7 days old - DELETE

After Cleanup:
[Seg1: Nov 19] [Seg2: Nov 20] ... [Seg7: Nov 25]

2. Size-Based Retention

Configuration:
log.retention.bytes = 107374182400  (100GB per partition)

Current State:
Partition total size = 98GB ✓ OK

New Segment Added:
Partition total size = 103GB ✗ EXCEEDS LIMIT

Action:
→ Kafka deletes oldest segment
→ New total = 102GB ✓

3. Combined Policies

You can use both simultaneously—Kafka deletes when either condition is met:

log.retention.hours = 168     (7 days)
log.retention.bytes = 100GB

Segment deleted if:
✓ Age > 7 days  OR
✓ Total size > 100GB

💨 Why Deletion is Lightning Fast

Segment-Level Operations

Kafka never deletes individual messages. It operates on entire segment files.

❌ Inefficient (What Kafka DOESN'T do):
Read file → Skip old messages → Write remaining → Rebuild indexes
(Hours of processing)

✅ Efficient (What Kafka DOES):
Delete segment-000.log
Delete segment-000.index  
Delete segment-000.timeindex
(Milliseconds)

Benefits:

🚀 One filesystem operation
🚀 No data rewriting
🚀 No index rebuilding
🚀 Extremely low CPU usage

🎬 Real-World Message Journey

Let's trace a viewer event through the entire system:

Step 1: Producer Sends Event

Producer → Kafka Broker

Message: {
  user_id: "viewer-789",
  action: "started_watching",
  show_id: "breaking-code-s1e3",
  timestamp: 2025-11-25 14:45:00
}

Destination:
Topic: viewer-activity
Partition: 2 (based on user_id hash)
Offset: 1,700,082 (next available)

Step 2: Write to Active Segment

Active Segment: 00000000000001700000.log
Position: Append at byte 245,760

Step 3: Update Indexes

Offset Index:
1,700,082 → byte 245,760

Time Index:
2025-11-25 14:45:00 → offset 1,700,082

Step 4: Acknowledge Producer

Broker → Producer:
"✓ Message written at offset 1,700,082"

Step 5: Consumer Reads Message

Consumer Request: "Give me messages from offset 1,700,082"

Kafka Process:
1. Which segment? → 00000000000001700000.log
2. Load offset index into memory
3. Find: offset 1,700,082 → byte 245,760
4. Read from disk starting at byte 245,760
5. Return message to consumer

🧹 Log Compaction: Latest State Only

The Use Case

Scenario: Storing user profile updates

User ID: viewer-789

Messages Over Time:
1. viewer-789 → {email: "old@email.com", plan: "basic"}
2. viewer-789 → {email: "new@email.com", plan: "basic"}
3. viewer-789 → {email: "new@email.com", plan: "premium"}

Question: Do we need messages 1 and 2?
Answer: NO! Only the latest state matters.

How Compaction Works

Configuration:
log.cleanup.policy = compact

Result:
Kafka guarantees: For each key, retain at least the last known value

Background Process:
1. Log cleaner reads old segments
2. Identifies duplicate keys
3. Keeps only latest message per key
4. Creates cleaned segments
5. Replaces old segments

💀 Tombstones: Deleting Keys

To remove a key entirely:

Send message: {key: "viewer-789", value: null}

Effect:
→ Log cleaner removes the key
→ Includes its last valid value
→ After configurable grace period

Compaction vs. Retention Comparison

Aspect	Retention (Delete)	Compaction
What's Kept	Time/size window of all messages	Latest value per key
What's Deleted	Old segments entirely	Old values for same key
Use Case	Time-series events, logs	User profiles, state management
Configuration	`log.cleanup.policy=delete`	`log.cleanup.policy=compact`
History	Limited time/size window	No history, only latest

🚀 Performance Benefits Summary

Why Kafka is Blazing Fast

Sequential Writes → Always append to end (disk-friendly)
Memory-Mapped Indexes → Entire index files in RAM
Zero-Copy Transfer → Direct disk → network socket
Batch Processing → Multiple messages written together
Segment-Level Operations → No expensive per-message work

📊 Real-World Numbers

A single Kafka broker can handle:

✅ Millions of messages per second
✅ Hundreds of MB/s throughput
✅ Sub-millisecond latencies
✅ All while maintaining durability

Learn Kafka by Doing: Build a 3-Broker Kafka Cluster with Docker Compose

Ajinkya Singh — Sun, 23 Nov 2025 11:00:00 +0000

Overview

This configuration deploys a 3-broker Kafka cluster with automatic failover, data replication, and a management UI.

Docker Compose Configuration

services:
  kafka1:
    image: apache/kafka:latest
    container_name: kafka1
    ports:
      - "9093:9093"
      - "29092:29092"
    volumes:
      - ./kafka-data/kafka1:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 1
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9093
      KAFKA_LISTENERS: CONTROLLER://:9092,EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka2:
    image: apache/kafka:latest
    container_name: kafka2
    ports:
      - "9094:9093"
    volumes:
      - ./kafka-data/kafka2:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 2
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:29092,EXTERNAL://localhost:9094
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka3:
    image: apache/kafka:latest
    container_name: kafka3
    ports:
      - "9095:9093"
    volumes:
      - ./kafka-data/kafka3:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 3
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:29092,EXTERNAL://localhost:9095
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8000:8080"
    networks:
      - kafka-bridge
    environment:
      KAFKA_CLUSTERS_0_NAME: kafkamultinodecluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka1:29092,kafka2:29092,kafka3:29092"
    depends_on:
      - kafka1
      - kafka2
      - kafka3

networks:
  kafka-bridge:
    driver: bridge

Key Configuration Breakdown

Port Strategy

9093-9095: External access (your apps connect here)
29092: Internal broker communication
9092: Controller coordination
8000: Kafka UI web interface

Listeners Explained

CONTROLLER (9092) - Controllers coordinate cluster operations
INTERNAL (29092)  - Brokers communicate with each other
EXTERNAL (9093+)  - External clients connect

Critical Environment Variables

CLUSTER_ID: Must be identical across all brokers (forms the cluster)

BROKER_ID: Unique identifier per broker (1, 2, 3)

PROCESS_ROLES:

kafka1: broker,controller (both roles)
kafka2/3: broker (worker only)

CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092 (only kafka1 can be controller)

ADVERTISED_LISTENERS:

INTERNAL://kafka1:29092 - Used by other brokers
EXTERNAL://localhost:9093 - Used by external clients

Replication Factors:

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 - Consumer offsets replicated 3x
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 - Transaction logs replicated 3x
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 - Minimum in-sync replicas (trade-off between availability and durability)

Data Persistence

./kafka-data/kafka1 → /var/lib/kafka/data (in container)
./kafka-data/kafka2 → /var/lib/kafka/data
./kafka-data/kafka3 → /var/lib/kafka/data

Data survives container restarts.

Producer Script

producer-read.sh:

#!/bin/bash
BOOTSTRAP_SERVERS="kafka2:29092"
TOPIC_NAME=$1
CONTAINER_NAME="kafka1"

if [ -z "$TOPIC_NAME" ]; then
  echo "Usage: $0 <topic-name>"
  exit 1
fi

for i in {1..10}; do
  MESSAGE="Message $i from producer"
  docker exec -it $CONTAINER_NAME bash -c "echo \"$MESSAGE\" | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $BOOTSTRAP_SERVERS --topic $TOPIC_NAME"
  echo "Sent: $MESSAGE"
done

Usage: ./producer-read.sh customer-orders

Sends 10 messages to the specified topic through kafka2, running from kafka1 container.

Consumer Script

consumer-read.sh:

#!/bin/bash
BOOTSTRAP_SERVERS="kafka1:29092"
TOPIC_NAME=$1
CONTAINER_NAME="kafka2"

if [ -z "$TOPIC_NAME" ]; then
  echo "Usage: $0 <topic-name>"
  exit 1
fi

docker exec -it $CONTAINER_NAME bash -c "/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP_SERVERS --topic $TOPIC_NAME --from-beginning"

Usage: ./consumer-read.sh customer-orders

Reads all messages from the beginning, running from kafka2 container, connecting to kafka1.

Quick Start

Start cluster: docker-compose up -d
Access UI: http://localhost:8000
Send messages: ./producer-read.sh my-topic
Read messages: ./consumer-read.sh my-topic
Stop cluster: docker-compose down

Architecture Summary

┌─────────────────────────────────────────────┐
│  kafka1 (broker + controller)               │
│  Ports: 9093 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka2 (broker)                            │
│  Ports: 9094 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka3 (broker)                            │
│  Ports: 9095 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka-ui (management interface)            │
│  Port: 8000                                 │
└─────────────────────────────────────────────┘

All brokers share the same CLUSTER_ID and communicate on the kafka-bridge network.

🎯 Apache Kafka Single-Node Practice Guide

Ajinkya Singh — Fri, 21 Nov 2025 20:30:38 +0000

Your Personal Message Delivery System

Welcome to your hands-on Kafka learning journey! In this guide, we'll build a complete Kafka system on a single machine - perfect for learning, testing, and understanding how everything works together.

📦 What We're Building

Think of this as creating your own Digital Post Office on your computer:

┌─────────────────────────────────────────┐
│     YOUR LAPTOP (Single Machine)        │
│                                         │
│  ┌───────────────────────────────┐    │
│  │    KAFKA BROKER (Port 9092)   │    │
│  │         Your Post Office      │    │
│  ├───────────────────────────────┤    │
│  │                               │    │
│  │  📬 Topic: "customer-orders"  │    │
│  │  📬 Topic: "payment-alerts"   │    │
│  │  📬 Topic: "user-activity"    │    │
│  │                               │    │
│  └───────────────────────────────┘    │
│           ↑              ↓             │
│      PRODUCERS      CONSUMERS          │
│   (Send messages) (Receive messages)   │
└─────────────────────────────────────────┘

Important Note: Single-node setup is ONLY for:

✅ Learning and experimentation
✅ Development and testing
✅ Understanding Kafka concepts
❌ NOT for production use (no fault tolerance!)

🛠 Step 1: Installation & Setup

Option A: Using Docker (Recommended - Easiest!)

# 1. Generate a unique Cluster ID
CLUSTER_ID=$(docker run --rm apache/kafka:latest /opt/kafka/bin/kafka-storage.sh random-uuid)
echo "Generated Cluster ID: $CLUSTER_ID"

# 2. Create a docker-compose.yml file with the generated ID
cat > docker-compose.yml << EOF
services:
  kafka1:
    image: apache/kafka:latest
    container_name: kafka1
    ports:
      - "9093:9093"
    volumes:
      - ./kafka-data/kafka1:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 1
      CLUSTER_ID: ${CLUSTER_ID}
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9093
      KAFKA_LISTENERS: CONTROLLER://:9092,EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

networks:
  kafka-bridge:
    driver: bridge

volumes:
  kafka-data:
EOF

# 3. Start Kafka
docker-compose up -d

# 4. Check if it's running
docker ps

# Expected output:
# CONTAINER ID   IMAGE                  STATUS         PORTS
# abc123def456   apache/kafka:latest    Up 10 seconds  0.0.0.0:9093->9093/tcp

Alternative: Manual Cluster ID Generation

# If you prefer to set it manually, you can generate and copy the ID:
docker run --rm apache/kafka:latest /opt/kafka/bin/kafka-storage.sh random-uuid

# Output example: MkU3OEVBNTcwNTJENDM2Qk
# Then paste this ID into CLUSTER_ID in docker-compose.yml

Understanding the Configuration:

┌─────────────────────────────────────────────────┐
│           YOUR DOCKER KAFKA SETUP               │
├─────────────────────────────────────────────────┤
│                                                 │
│  Container: kafka1                              │
│  ┌───────────────────────────────────────────┐ │
│  │  Port 9093 (External) ← You connect here │ │
│  │  Port 29092 (Internal) ← Container-only  │ │
│  │  Port 9092 (Controller) ← Management     │ │
│  └───────────────────────────────────────────┘ │
│                     ↓                           │
│  Volume: ./kafka-data/kafka1                    │
│  (Your data persists here!)                     │
│                                                 │
└─────────────────────────────────────────────────┘

What Each Port Does:

9093 (EXTERNAL): Use this from your laptop (localhost:9093)
29092 (INTERNAL): Used by containers talking to each other
9092 (CONTROLLER): Kafka's internal coordination

Data Persistence:
Your messages are saved in ./kafka-data/kafka1 directory. Even if you stop the container, your data remains!

🎪 Step 2: Your First Kafka Topic

Let's create our coffee shop's order board!

Create a Topic

# Using Docker:
docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --partitions 3 \
  --replication-factor 1

What This Creates:

Topic: "coffee-orders" (3 sections on your order board)

┌─────────────────────────────────────────┐
│         COFFEE ORDERS BOARD             │
├─────────────────────────────────────────┤
│                                         │
│  Section 0    Section 1    Section 2   │
│  (Partition)  (Partition)  (Partition) │
│  ┌────────┐  ┌────────┐  ┌────────┐   │
│  │ Order 1│  │ Order 2│  │ Order 3│   │
│  │ Order 4│  │ Order 5│  │ Order 6│   │
│  │  ...   │  │  ...   │  │  ...   │   │
│  └────────┘  └────────┘  └────────┘   │
└─────────────────────────────────────────┘

List All Topics

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9093

View Topic Details

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic coffee-orders \
  --bootstrap-server localhost:9093

Output Explanation:

Topic: coffee-orders
PartitionCount: 3          ← Your order board has 3 sections
ReplicationFactor: 1       ← Only 1 copy (single node)
Partition: 0
  Leader: 1                ← Broker 1 manages this section
  Replicas: 1              ← Only one copy exists
  Isr: 1                   ← In-sync replicas

☕ Step 3: Sending Orders (Producer)

Now let's be the cashier and take some orders!

Start the Console Producer

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093

Type these orders (press Enter after each):

{"customer":"Alice","drink":"Latte","size":"Medium","price":4.50}
{"customer":"Bob","drink":"Espresso","size":"Small","price":3.00}
{"customer":"Charlie","drink":"Cappuccino","size":"Large","price":5.50}
{"customer":"Diana","drink":"Mocha","size":"Medium","price":4.75}
{"customer":"Eve","drink":"Americano","size":"Large","price":3.50}

What's Happening:

You (Cashier)                  Kafka                    Storage
     │                           │                         │
     │──Order: Alice's Latte────→│                         │
     │                           │──Save to Partition 0───→│
     │                           │                         │
     │──Order: Bob's Espresso───→│                         │
     │                           │──Save to Partition 1───→│
     │                           │                         │
     │──Order: Charlie's Capp.──→│                         │
     │                           │──Save to Partition 2───→│
     │                           │                         │

Press Ctrl+C to stop the producer

🧑‍🍳 Step 4: Processing Orders (Consumer)

Now let's be the barista and read the orders!

Start the Console Consumer (Read from Beginning)

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --from-beginning

You'll see all the orders scroll by:

{"customer":"Alice","drink":"Latte","size":"Medium","price":4.50}
{"customer":"Bob","drink":"Espresso","size":"Small","price":3.00}
{"customer":"Charlie","drink":"Cappuccino","size":"Large","price":5.50}
...

What's Happening:

Storage                    Kafka                    Barista (You)
   │                         │                          │
   │←──Request all orders────│                          │
   │                         │                          │
   │──Send order 1──────────→│──Alice's Latte─────────→│
   │──Send order 2──────────→│──Bob's Espresso────────→│
   │──Send order 3──────────→│──Charlie's Cappuccino──→│
   │                         │                          │

Start Another Consumer (Real-time Only)

Open a NEW terminal and run:

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093

This consumer only sees new orders (like a new barista who just arrived).

🎭 Step 5: Multiple Producers & Consumers

Let's simulate a busy coffee shop with multiple cashiers and baristas!

Experiment Setup

Terminal 1 (Cashier A - Producer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093

Terminal 2 (Cashier B - Producer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093

Terminal 3 (Barista A - Consumer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --from-beginning

Terminal 4 (Barista B - Consumer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --from-beginning

Now:

Type orders in Terminal 1 and 2 (both cashiers taking orders)
Watch them appear in Terminal 3 and 4 (both baristas see ALL orders)

Visual:

Cashier A ──┐
            ├──→ ORDER BOARD ──┬──→ Barista A (sees all orders)
Cashier B ──┘    (Kafka)       └──→ Barista B (sees all orders)

Understanding Kafka Complete Architecture Flow - Advanced🎯

Ajinkya Singh — Wed, 19 Nov 2025 19:07:56 +0000

Please read the last two articles in my Kafka series before this one — this part gets serious.

🌐 The Big Picture: Two Brains Working Together

Think of Kafka as a well-organized company with two main components:

┌─────────────────────────────────────────────┐
│           KAFKA CLUSTER                     │
│                                             │
│  ┌────────────────────────────────────┐    │
│  │  CONTROLLERS (The Brain 🧠)        │    │
│  │  • Manage who does what            │    │
│  │  • Track what's happening          │    │
│  │  • Make decisions                  │    │
│  └──────────────┬─────────────────────┘    │
│                 ↓ Commands & Updates       │
│  ┌────────────────────────────────────┐    │
│  │  BROKERS (The Workers 💪)          │    │
│  │  • Store the actual data           │    │
│  │  • Serve producers and consumers   │    │
│  │  • Follow controller's instructions│    │
│  └────────────────────────────────────┘    │
└─────────────────────────────────────────────┘

💡 Simple Analogy: Controllers are like managers who plan and coordinate, while Brokers are employees who do the actual work.

🚀 Part 1: SETUP - Controllers Organize Everything

Step 1: Controllers Start and Elect Leader

When Kafka starts, multiple controllers use the Raft Election algorithm to choose one leader:

Controller-1    Controller-2    Controller-3
     │               │               │
     └───────Raft Election───────────┘
                     ↓
          One becomes LEADER ⭐
                     │
            ┌────────────────┐
            │ Controller-1   │
            │   (LEADER) ⭐   │ ← Makes all decisions!
            └────────────────┘

💡 Simple: Like choosing a class monitor who manages everything.

Step 2: Controllers Create Metadata Registry

The Controller Leader creates a comprehensive "notebook" 📒 of everything happening in the cluster:

┌──────────────────────────────────────────┐
│        METADATA REGISTRY                 │
├──────────────────────────────────────────┤
│                                          │
│ TOPICS:                                  │
│  • "orders" → 3 partitions               │
│  • "payments" → 2 partitions             │
│                                          │
│ BROKERS:                                 │
│  • Broker-1 → Alive ✅ (IP: 192.168.1.10)│
│  • Broker-2 → Alive ✅ (IP: 192.168.1.11)│
│  • Broker-3 → Alive ✅ (IP: 192.168.1.12)│
│                                          │
│ WHO'S IN CHARGE (Leaders):               │
│  • orders-partition-0 → Broker-1         │
│  • orders-partition-1 → Broker-2         │
│  • orders-partition-2 → Broker-3         │
│                                          │
│ BACKUPS (Followers):                     │
│  • orders-partition-0 → Broker-2, Broker-3│
│  • orders-partition-1 → Broker-3, Broker-1│
│  • orders-partition-2 → Broker-1, Broker-2│
└──────────────────────────────────────────┘

💡 Simple: Like a school register showing which classes exist, which teachers are present, who teaches which subject, and who are the substitute teachers.

Step 3: Controller Tells Brokers Their Jobs

The controller assigns specific roles to each broker:

Controller ⭐ → Broker-1: "You are the LEADER for orders-partition-0"
                         "You are a BACKUP for orders-partition-2"

Controller ⭐ → Broker-2: "You are the LEADER for orders-partition-1"
                         "You are a BACKUP for orders-partition-0"

Controller ⭐ → Broker-3: "You are the LEADER for orders-partition-2"
                         "You are a BACKUP for orders-partition-1"

💡 Simple: Like a manager assigning tasks to employees.

📤 Part 2: PRODUCER SENDS DATA

Step 4: Producer Wants to Send Message

Your application (Producer) has data to send:

┌──────────────────────────┐
│  I have a message:       │
│  • Topic: "orders"       │
│  • Key: "user_123"       │
│  • Value: {order data}   │
│                          │
│  "Where do I send this?" │
└──────────────────────────┘

💡 Simple: Like having a letter but not knowing which post office to use.

Step 5: Producer Asks ANY Broker for Information

The producer can connect to any broker to get routing information:

Producer ──────────► Broker-1
"Where do I send     "Let me check
messages for         my copy of
'orders' topic?"     the registry..."

Key Point: Every broker has a copy of the controller's metadata!

┌───────────────────────────────────┐
│  Broker-1's Copy of Metadata:     │
│  • orders-partition-0 → Broker-1  │
│  • orders-partition-1 → Broker-2  │
│  • orders-partition-2 → Broker-3  │
└───────────────────────────────────┘

💡 Simple: Like asking a postman for directions - he has a map (copy of registry).

Step 6: Producer Calculates Which Partition

The producer's client library automatically determines the target partition:

1. Hash the key: hash("user_123") = 456789
2. Divide by partitions: 456789 % 3 = 0
3. Result: Goes to Partition 0

💡 Simple: Like a sorting machine that knows exactly which box to put each item in.

Step 7: Producer Sends to Correct Broker

Now the producer sends directly to the partition leader:

Producer ───────────► Broker-1 (Leader for P0)

Broker-1 receives:
1. ✅ Validates it's the leader for P0
2. 💾 Writes to disk: /kafka/data/orders-0/
3. 📝 Assigns offset: 1251
4. ✅ Sends "OK!" back to producer

💡 Simple: Like mailing a letter to the correct post office that handles your area.

Step 8: Broker Replicates to Followers (Background)

After writing, the leader broker automatically replicates to followers:

Broker-1 (Leader P0) ───COPY MESSAGE───► Broker-2 (Follower P0)
                                          ✅ Copied!
         │◄──────────ACK (Copied!)───────┤

Broker-1 (Leader P0) ───COPY MESSAGE───► Broker-3 (Follower P0)
                                          ✅ Copied!

Result: Data now exists on 3 brokers! 💪

💡 Simple: Like making photocopies of important documents and storing them in different safes.

📥 Part 3: CONSUMER READS DATA

Step 9: Consumer Wants to Read Messages

Your application (Consumer) wants to read data:

┌──────────────────────────┐
│  I want to read from:    │
│  • Topic: "orders"       │
│  • Group: "my-group"     │
│                          │
│  "How do I start?"       │
└──────────────────────────┘

💡 Simple: Like wanting to read a book but not knowing which library has it.

Step 10: Consumer Connects and Gets Metadata

The consumer connects to any broker to get cluster information:

Consumer ────────────► Broker-2
"Tell me everything    "Here's the full
about 'orders'"        cluster map!"

Broker returns:

┌───────────────────────────────────┐
│  Topic "orders" info:             │
│  • Partition 0 → Leader: Broker-1 │
│  • Partition 1 → Leader: Broker-2 │
│  • Partition 2 → Leader: Broker-3 │
└───────────────────────────────────┘

💡 Simple: Like getting a mall directory showing which stores are on which floor.

Step 11: Consumer Joins Group (Controller Coordinates)

The consumer joins a consumer group for coordinated reading:

Consumer → Broker → Controller ⭐
"I want to join      "Let me assign
group 'my-group'"    partitions..."

Controller decides:
┌───────────────────────────────────┐
│  Group "my-group" has 1 consumer  │
│  Topic "orders" has 3 partitions  │
│                                   │
│  Assignment:                      │
│  Consumer-A → [P0, P1, P2]        │
│  (gets all 3 partitions)          │
└───────────────────────────────────┘

💡 Simple: Like a teacher assigning homework to students.

Step 12: Consumer Reads and Tracks Progress

How Consumer Manages Reading Position:

STARTUP (Once):
Consumer → Broker: "Where did I leave off?"
Broker → Consumer: "P0: offset 1250, P1: offset 890, P2: offset 2100"
Consumer stores in memory ✅

CONTINUOUS READING:
Consumer fetches using local memory (no broker queries!)
• Fetch from P0: offset 1250 → 1300 → 1350 (tracked in memory)
• Fetch from P1: offset 890 → 940 → 990 (tracked in memory)
• Fetch from P2: offset 2100 → 2150 → 2200 (tracked in memory)

PERIODIC COMMIT (Every 5 seconds or after batch):
Consumer → Broker: "Save progress: P0=1350, P1=990, P2=2200"

🔥 KEY POINTS:

Consumer reads position from broker ONCE at startup
Tracks current position IN MEMORY while reading
Saves progress back to broker PERIODICALLY (every 5 seconds by default)

💡 Simple: Like checking your bookmark when opening a book, remembering your page while reading, and updating the bookmark occasionally.

Step 13: Consumer Pulls Data from Brokers

Consumer fetches data in parallel from all partition leaders:

Consumer ───────► Broker-1 (P0) → Returns 50 events
        ├────────► Broker-2 (P1) → Returns 50 events
        └────────► Broker-3 (P2) → Returns 50 events

Total: 150 events fetched!
         ↓
   Process all events
   (Your business logic)
         ↓
   All done! ✅

💡 Simple: Like reading from multiple books at the same time, keeping track of your progress in each.

🔧 Part 4: FAILURE HANDLING

SCENARIO A: Broker Fails (Controller Handles)

Broker-1 crashes! 💥

STEP 1: Controller Detects Failure
Controller ⭐: "Broker-1 stopped responding!"

STEP 2: Controller Checks Metadata
┌───────────────────────────────────┐
│  Partition 0 (orders):            │
│  • Leader: Broker-1 💀            │
│  • Followers: Broker-2 ✅, Broker-3 ✅│
│                                   │
│  Need new leader for P0!          │
└───────────────────────────────────┘

STEP 3: Controller Elects New Leader
Controller ⭐ → Broker-2: "You are now LEADER for P0!"
Broker-2: "OK! I'm the new leader!"

STEP 4: Controller Updates Metadata
┌───────────────────────────────────┐
│  Partition 0 (orders):            │
│  • Leader: Broker-2 ⭐ (NEW!)     │
│  • Followers: Broker-3 ✅         │
│  • Broker-1: 💀 (removed)         │
└───────────────────────────────────┘

STEP 5: Controller Notifies Everyone
Controller → All: "Metadata changed! P0 leader is now Broker-2!"

Total time: 2-3 seconds! ⚡

💡 Simple: Like when a teacher is absent, the principal quickly assigns a substitute.

SCENARIO B: Consumer Fails (Controller Rebalances)

Consumer-A crashes! 💥

STEP 1: Group Coordinator Detects
Coordinator: "Consumer-A missed 3 heartbeats!"

STEP 2: Coordinator Triggers Rebalance
Coordinator → Controller: "Group 'my-group' needs rebalancing!"
Controller → Other Consumers: "Stop! Rebalancing..."

STEP 3: Controller Reassigns Partitions
Before:
Consumer-A: [P0, P1] 💀 (dead)
Consumer-B: [P2] ✅

After:
Consumer-B: [P0, P1, P2] ✅ (takes over all!)

STEP 4: Controller Notifies Consumer-B
Controller → Consumer-B: "You now read P0, P1, P2"

Consumer-B resumes reading:
✅ Loads last saved positions
✅ Continues from where Consumer-A left off
✅ No messages lost!

💡 Simple: Like when one waiter is sick, another takes over their tables.

🎨 The Complete Picture

┌────────────────────────────────────────────────────────┐
│                   KAFKA CLUSTER                        │
│                                                        │
│  ┌───────────────────────────────────────────────┐    │
│  │     CONTROLLERS (The Managers 🧠)             │    │
│  │                                               │    │
│  │  Controller-1 ⭐  Controller-2  Controller-3  │    │
│  │    (Leader)      (Follower)    (Follower)    │    │
│  │                                               │    │
│  │  Maintains METADATA REGISTRY                  │    │
│  │  • Who's alive?                               │    │
│  │  • Who's the leader?                          │    │
│  │  • Who reads what?                            │    │
│  └──────────────────┬────────────────────────────┘    │
│                     ↓ Commands & Notifications        │
│  ┌───────────────────────────────────────────────┐    │
│  │     BROKERS (The Workers 💪)                  │    │
│  │                                               │    │
│  │  Broker-1      Broker-2      Broker-3        │    │
│  │  P0 (L)⭐      P1 (L)⭐      P2 (L)⭐         │    │
│  │  P1 (F)        P2 (F)        P0 (F)          │    │
│  │  P2 (F)        P0 (F)        P1 (F)          │    │
│  │                                               │    │
│  │  Stores & Serves Data                         │    │
│  └───────────────────────────────────────────────┘    │
│         ↑              ↑              ↑               │
└─────────┼──────────────┼──────────────┼───────────────┘
          │              │              │
          ↓              ↓              ↓
    Producer-1     Producer-2     Producer-3
          │              │              │
          └──────────────┴──────────────┘
                         │
         All query metadata from any broker
                         │
          ┌──────────────┴──────────────┐
          ↓                             ↓
    Consumer-A                    Consumer-B
    Group: g1                     Group: g1
    Reads: P0,P1                  Reads: P2

📋 Key Roles Summary

Controller (The Boss 👔)

MANAGES:

✅ Who's alive? (broker health)
✅ Who's in charge? (partition leaders)
✅ Who reads what? (consumer assignments)
✅ What exists? (topics, partitions)

DECIDES:

✅ New leader when broker fails
✅ Partition assignments for consumers
✅ Where new partitions go

NOTIFIES:

✅ Tells brokers their jobs
✅ Updates everyone on changes
✅ Coordinates rebalancing

Broker (The Worker 👷)

STORES:

✅ Actual data on disk
✅ Log files for partitions
✅ Copy of metadata (from controller)

SERVES:

✅ Producer write requests
✅ Consumer read requests
✅ Metadata queries

REPLICATES:

✅ Copies data to followers
✅ Syncs with leader
✅ Reports status to controller

Producer (The Sender 📤)

DOES:

✅ Creates messages
✅ Queries metadata (from any broker)
✅ Calculates partition (key hash)
✅ Sends to correct broker

DOESN'T CARE ABOUT:

❌ Controllers (transparent)
❌ Followers (writes only to leader)
❌ Other producers

Consumer (The Receiver 📥)

DOES:

✅ Joins consumer group
✅ Gets partition assignment
✅ Tracks reading position in memory
✅ Pulls from leader brokers
✅ Saves progress periodically

DOESN'T CARE ABOUT:

❌ How controller assigns partitions
❌ Follower replicas
❌ Other consumer groups

🎯 The Magic: Why This Works So Well

1. Separation of Concerns

CONTROLLERS think 🧠
BROKERS work 💪
Controllers don't handle data
Brokers don't make decisions
Like: Managers plan, workers execute

2. Everything Has a Backup

Controllers: 3 copies (1 leader + 2 followers)
Partitions: 3 copies (1 leader + 2 followers)
Metadata: All brokers have a copy
Result: If anything fails, backups take over!

3. Distributed = Fast + Reliable

Multiple brokers = Parallel processing
Multiple partitions = Load distribution
Multiple replicas = No data loss
Like: Many checkout lanes at a store

4. Automatic Recovery

Broker fails → Controller elects new leader (seconds)
Consumer fails → Controller reassigns partitions (seconds)
Controller fails → Another controller becomes leader (seconds)
All automatic! No human intervention needed!

I also took some help from Claude to understand and make a few concepts more visual.

🧠 Kafka Broker vs Controller - Complete Guide

Ajinkya Singh — Tue, 18 Nov 2025 19:05:53 +0000

Understanding the Two Critical Roles in Kafka's Architecture

The Big Picture

In Kafka 4.0 (with KRaft), servers can perform two distinct roles:

Role	Analogy	Primary Function
Broker 📦	Library Shelf Manager	Handles data storage and delivery
Controller 🎮	Library Head Librarian	Manages catalog and coordinates operations

Quick Tip: Think of Kafka as a digital library system. Brokers are the staff who shelve and retrieve books, while Controllers are the head librarians who maintain the catalog and coordinate everything.

Evolution: Before and After

❌ The Old Way (Before Kafka 4.0)

Problem: Two separate systems to manage!

┌─────────────────────┐
│  ZooKeeper Cluster  │ ← External dependency
│   (The Brain 🧠)    │    Must maintain separately
│                     │    Additional complexity
└──────────┬──────────┘
           │
           ↓ Manages metadata
┌─────────────────────┐
│   Kafka Brokers     │
│ (Data handlers only)│
│  • Store data       │
│  • Serve clients    │
└─────────────────────┘

Challenges:

Two systems to deploy, monitor, and maintain
ZooKeeper expertise required
Additional infrastructure costs
Complex failure scenarios

✅ The New Way (Kafka 4.0+ with KRaft)

Solution: Self-contained, all-in-one system!

┌───────────────────────────────────────┐
│      KAFKA CLUSTER (Self-Managed)     │
│                                       │
│  CONTROLLERS (Built-in Brain 🧠)      │
│  ┌──────┐  ┌──────┐  ┌──────┐       │
│  │Ctrl-1│  │Ctrl-2│  │Ctrl-3│       │
│  │Leader│  │Follow│  │Follow│       │
│  └──┬───┘  └──────┘  └──────┘       │
│     │                                │
│     ↓ Manages metadata               │
│  ┌──────┐  ┌──────┐  ┌──────┐       │
│  │Brkr-1│  │Brkr-2│  │Brkr-3│       │
│  │ 📦   │  │ 📦   │  │ 📦   │       │
│  └──────┘  └──────┘  └──────┘       │
└───────────────────────────────────────┘

Benefits:

✅ Single system to manage
✅ No external dependencies
✅ Faster metadata operations
✅ Simpler deployment

Role 1: The Broker (Library Shelf Manager 📦)

What It Does

The Broker is the data handler - it stores and serves data to producers and consumers.

Real-World Analogy

Imagine a library shelf manager who:

Receives new books from publishers (messages from producers)
Organizes them on specific shelves (partitions)
Retrieves books when patrons request them (serves consumers)
Maintains backup copies in storage rooms (replication)

Key Responsibilities

1️⃣ Storing Data 💾

Broker stores topic partitions on disk:

/var/kafka/data/
├── product-catalog-0/
│   ├── 00000000.log  ← Actual message data
│   ├── 00001000.log
│   └── offset: 1250
│
├── product-catalog-2/
│   └── Backup copy from Broker-3
│
└── customer-events-1/
    └── offset: 450

2️⃣ Handling Producer Requests 📤

Receives messages from producers
Appends to partition logs
Assigns unique offsets
Sends acknowledgments back

3️⃣ Handling Consumer Requests 📥

Serves read requests from consumers
Fetches data from partitions
Tracks consumer positions
Manages consumer offsets

4️⃣ Replication 🔄

Copies data between leader and follower partitions
Ensures data redundancy
Maintains in-sync replicas (ISR)
Handles failover scenarios

5️⃣ Providing Metadata 📋

Tells clients about cluster topology
Shares partition locations
Provides leader information
Responds to bootstrap requests

Visual: Broker in Action

        Producers                  Consumers
            │                          │
            ↓ Write                    ↑ Read
    ┌───────────────────────────────────┐
    │         BROKER-1 (Server)         │
    ├───────────────────────────────────┤
    │                                   │
    │  product-catalog-0/ (Leader)      │
    │  ├─ Messages: 1-1250              │
    │  └─ Actively serving clients      │
    │                                   │
    │  product-catalog-2/ (Follower)    │
    │  └─ Syncing from Broker-3         │
    │                                   │
    │  customer-events-1/ (Leader)      │
    │  └─ Messages: 1-450               │
    └───────────────────────────────────┘

Role 2: The Controller (Head Librarian 🎮)

What It Does

The Controller is the brain/orchestrator - it manages cluster state and coordinates operations.

Real-World Analogy

Imagine a head librarian who:

Doesn't shelve books personally (no data handling)
Maintains the master catalog (metadata)
Decides which staff manages which sections (partition assignment)
Tracks all library locations and staff availability (broker health)
Coordinates responses when staff call in sick (leader election)
If the head librarian is unavailable, an assistant takes over immediately

Key Responsibilities

1️⃣ Cluster State Management 🗺️

The Controller maintains the single source of truth:

Topic Registry:
  - Topic: "transaction-stream"
    Partitions: 6
    Replication Factor: 3
    Leaders:
      - Partition-0: Broker-1
      - Partition-1: Broker-2
      - Partition-2: Broker-3
      - Partition-3: Broker-1
      - Partition-4: Broker-2
      - Partition-5: Broker-3

Broker Registry:
  - Broker-1: ✅ Online, 15 partitions
  - Broker-2: ✅ Online, 18 partitions
  - Broker-3: ✅ Online, 17 partitions

Consumer Groups:
  - Group "data-analytics":
    Members: [Consumer-A, Consumer-B, Consumer-C]
    Coordinator: Broker-1

2️⃣ Leader Election ⭐

When a partition leader fails, the Controller:

Detects the failure immediately
Selects a new leader from in-sync replicas
Updates cluster metadata
Notifies all brokers
Clients automatically redirect to new leader

Example Scenario:

Before:  transaction-stream-0 Leader = Broker-1 ✅
         Broker-1 crashes! 💥
After:   transaction-stream-0 Leader = Broker-2 ⭐ (promoted!)
         Time taken: ~2-3 seconds

3️⃣ Cluster Change Notification 📢

The Controller broadcasts changes to all brokers:

🆕 New topic created → notify all brokers
⚠️ Broker goes down → redistribute partitions
⭐ New leader elected → update routing
🔧 Configuration changed → apply updates

4️⃣ Broker Lifecycle Management 🔄

Manages broker registration
Handles broker join/leave events
Smooth handoff during shutdowns
Updates cluster membership

5️⃣ Administrative Operations ⚙️

Topic creation/deletion
Partition reassignment
Configuration changes
Quota management

Visual: Controller Quorum

    CONTROLLER QUORUM (High Availability)

┌──────────┐  ┌──────────┐  ┌──────────┐
│  Ctrl-1  │  │  Ctrl-2  │  │  Ctrl-3  │
│ (LEADER) │◄─┤(Follower)│◄─┤(Follower)│
│    ⭐    │  │          │  │          │
├──────────┤  ├──────────┤  ├──────────┤
│ • Makes  │  │ • Standby│  │ • Standby│
│   all    │  │ • Ready  │  │ • Ready  │
│   decis- │  │   to     │  │   to     │
│   ions   │  │   take   │  │   take   │
│ • Notif- │  │   over   │  │   over   │
│   ies    │  │ • Syncs  │  │ • Syncs  │
│   brokers│  │   data   │  │   data   │
└────┬─────┘  └──────────┘  └──────────┘
     │
     ↓ Commands & notifications
┌────────────────────────────────────┐
│            BROKERS                 │
│  ┌────┐    ┌────┐    ┌────┐      │
│  │Br-1│    │Br-2│    │Br-3│      │
│  └────┘    └────┘    └────┘      │
└────────────────────────────────────┘

Important Notes:

Always use an odd number of controllers (3, 5, 7)
Uses Raft consensus algorithm
Requires majority to function (e.g., 2 out of 3)
If majority fails, cluster cannot make decisions

Combined vs Dedicated Roles

Option 1: Combined Role (Development/Testing)

Setup: Each node runs BOTH broker + controller

┌─────────────────────┐
│     NODE-1          │
│ ┌─────────────────┐ │
│ │   Controller    │ │
│ │   (Leader) ⭐   │ │
│ └─────────────────┘ │
│         +           │
│ ┌─────────────────┐ │
│ │     Broker      │ │
│ │   (Data 📦)     │ │
│ └─────────────────┘ │
└─────────────────────┘

Same for NODE-2 and NODE-3
(with follower controllers)

Pros:

✅ Simple setup
✅ Fewer machines (cost-effective)
✅ Good for development/testing
✅ Small-scale production

Cons:

❌ Resource contention (metadata + data compete)
❌ Less stable under high load
❌ Harder to scale independently
❌ "Noisy neighbor" problem

Best For:

Local development
Testing environments
Small production deployments (<10 brokers)
Low-traffic applications

Option 2: Dedicated Roles (Production)

Setup: Separate controller nodes from broker nodes

DEDICATED CONTROLLERS (Metadata Only)
┌──────────┐  ┌──────────┐  ┌──────────┐
│  Ctrl-1  │  │  Ctrl-2  │  │  Ctrl-3  │
│ (Leader) │  │(Follower)│  │(Follower)│
├──────────┤  ├──────────┤  ├──────────┤
│ 4GB RAM  │  │ 4GB RAM  │  │ 4GB RAM  │
│ 2 CPU    │  │ 2 CPU    │  │ 2 CPU    │
│ Small VM │  │ Small VM │  │ Small VM │
└────┬─────┘  └──────────┘  └──────────┘
     │
     ↓ Manages

DEDICATED BROKERS (Data Only)
┌──────────┐  ┌──────────┐  ┌──────────┐
│  Brkr-1  │  │  Brkr-2  │  │  Brkr-3  │
│   📦     │  │   📦     │  │   📦     │
├──────────┤  ├──────────┤  ├──────────┤
│ 64GB RAM │  │ 64GB RAM │  │ 64GB RAM │
│ 16 CPU   │  │ 16 CPU   │  │ 16 CPU   │
│ TB disk  │  │ TB disk  │  │ TB disk  │
└──────────┘  └──────────┘  └──────────┘

... scale to 100+ brokers as needed

Pros:

✅ Maximum stability (isolated operations)
✅ Independent scaling
✅ Optimized resources per role
✅ Better fault tolerance
✅ Industry standard for production
✅ Can upgrade independently

Cons:

❌ More machines (higher cost)
❌ More complex setup
❌ Overkill for small deployments

Best For:

Production environments
High-traffic applications
Enterprise deployments
Systems requiring 24/7 uptime

Real-World Examples

Example: Controller Leader Failover

Scenario: Main controller experiences hardware failure

BEFORE (Normal Operations):
Controller-1 (LEADER) ⭐ → Managing all metadata
Controller-2 (Follower) → Standby backup
Controller-3 (Follower) → Standby backup

━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Hardware failure on Controller-1! 💥
━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Detection (within seconds):
Controller-2: "Leader timeout detected!"
Controller-3: "Leader timeout detected!"
        │
        ↓ Raft Consensus Election

AFTER (2-3 seconds):
Controller-1 (OFFLINE) 💀
Controller-2 (LEADER) ⭐ → PROMOTED! Takes over all duties
Controller-3 (Follower) → Continues standby

✅ Service continues without interruption!
✅ No data lost!
✅ Brokers still serving all requests!

🔍 Question on Kraft’s Leader Election Algorithm

In Kraft’s leader election algorithm, correctness proofs often assume an odd number of nodes to avoid symmetry and tie-breaking issues.

But in real distributed systems, nodes can fail at any time.

👉 If a node fails mid-execution and the system is left with an even number of active nodes, how does the algorithm still guarantee that a unique leader is elected?

Would love to hear your views and interpretations on this!

Understanding Kafka Architecture - The Complete Mental Model 🧠

Ajinkya Singh — Mon, 17 Nov 2025 21:15:58 +0000

How all the pieces fit together to create a powerful streaming platform

The Goal

Understand the "Big Picture" - How events, topics, partitions, producers, consumers, brokers, and consumer groups all work together as one cohesive system.

Think of this as getting a bird's eye view of the entire Kafka ecosystem! 🦅

Building Block #1: The Event (Foundation)

What It Is

The fundamental unit - an immutable fact representing something that happened.

┌─────────────────────────────────────┐
│           EVENT/RECORD              │
├─────────────────────────────────────┤
│ Key: user_456                       │
│ Value: {"action": "purchase"}       │
│ Timestamp: 2025-11-18 14:30:00     │
└─────────────────────────────────────┘

Everything in Kafka revolves around these!

Building Block #2: The Kafka Cluster (Infrastructure)

What It Is

A collection of servers working together - NOT just one server!

        KAFKA CLUSTER
┌─────────────────────────────────┐
│                                 │
│  ┌────────┐  ┌────────┐        │
│  │Broker 1│  │Broker 2│  ...   │
│  │Server 1│  │Server 2│        │
│  └────────┘  └────────┘        │
│                                 │
│  ┌────────┐  ┌────────┐        │
│  │Broker 3│  │Broker 4│  ...   │
│  │Server 3│  │Server 4│        │
│  └────────┘  └────────┘        │
│                                 │
└─────────────────────────────────┘

Network of powerful servers!

What Brokers Do

Store your events
Handle requests from applications
Ensure the system stays available even if one fails

Why Multiple Brokers?

Scalability → Handle massive amounts of data
Fault Tolerance → Keep running even if servers fail

Modern Kafka (4.0+)

Brokers are self-managing using KRaft protocol
They coordinate with each other internally
No external ZooKeeper needed! 🎉

Visualize: A resilient network of powerful servers ready to handle your data streams.

Building Block #3: Topics (Organization)

What It Is

A logical name/category for a stream of related events.

KAFKA CLUSTER
├── Topic: "user-signups" 👤
├── Topic: "payment-transactions" 💰
├── Topic: "sensor-readings" 🌡️
└── Topic: "order-events" 📦

Key Characteristics

1. Distributed Across Brokers

Single topic doesn't live on just ONE broker:

Topic: "orders"
├── Partition 0 → Broker 1
├── Partition 1 → Broker 2
└── Partition 2 → Broker 3

This distribution = SCALE! 🚀

2. Durable Storage

Events stored for configurable retention period
Can be re-read multiple times
Not deleted after consumption

Building Block #4: Partitions (Parallelism)

What It Is

Each topic is divided into ordered lanes called partitions.

The Multi-Lane Highway Analogy 🛣️

Topic: "orders" (3 partitions)

┌────────────────────────────────────────────────────┐
│               MULTI-LANE HIGHWAY                   │
├────────────────────────────────────────────────────┤
│                                                    │
│  Lane 0 (Partition 0): Order1 → Order2 → Order3  │
│  ═════════════════════════════════════════════►   │
│                                                    │
│  Lane 1 (Partition 1): Order4 → Order5 → Order6  │
│  ═════════════════════════════════════════════►   │
│                                                    │
│  Lane 2 (Partition 2): Order7 → Order8 → Order9  │
│  ═════════════════════════════════════════════►   │
│                                                    │
└────────────────────────────────────────────────────┘

Each lane (partition) processes traffic (events) 
independently but IN ORDER within that lane!

Key Properties

1. Ordered Within Partition ✅

Partition 0:
Event A (offset 0) → Event B (offset 1) → Event C (offset 2)

Consumer always sees: A, then B, then C
ORDER GUARANTEED within the partition!

2. NO Order Across Partitions ❌

Partition 0: Event A (time: 10:00)
Partition 1: Event B (time: 09:59)

Consumer might see B before A
NO ORDER GUARANTEE across different partitions!

3. Each Partition Lives on a Broker

Topic: "payments" (3 partitions)

Partition 0 → Broker 1 (Server 1)
Partition 1 → Broker 2 (Server 2)
Partition 2 → Broker 3 (Server 3)

Load is DISTRIBUTED across servers! ⚖️

Why Partitions?

Enable parallelism → Multiple producers/consumers work simultaneously
Distribute load → Spread data across multiple servers
Scale horizontally → Add more partitions = more throughput

Building Block #5: Producers (Data Writers)

What It Is

Your application code that sends/publishes events to Kafka topics.

         PRODUCERS (Entry Ramps)

Mobile App 📱 ──┐
                │
Web Server 🌐 ──┼──► Kafka Topic: "events"
                │      ├─► Partition 0
IoT Device 🌡️ ──┘      ├─► Partition 1
                       └─► Partition 2

How Producers Work

Option 1: Automatic Partition Selection (No Key)

Producer sends events WITHOUT key:

Event 1 → Partition 0 (round-robin)
Event 2 → Partition 1 (round-robin)
Event 3 → Partition 2 (round-robin)
Event 4 → Partition 0 (round-robin)
...

Result: EVEN DISTRIBUTION across partitions

Option 2: Key-Based Routing (With Key)

Producer sends events WITH key:

Event (key: user_123) → Partition 1
Event (key: user_123) → Partition 1 (SAME!)
Event (key: user_456) → Partition 2
Event (key: user_456) → Partition 2 (SAME!)
Event (key: user_123) → Partition 1 (SAME!)

Result: ALL events with SAME KEY go to SAME PARTITION
        This maintains ORDER for related events! 🎯

Visual Example: Key-Based Routing

Producer: E-commerce Website

Order from user_123:
┌──────────────────────┐
│ Key: user_123        │
│ Value: Order details │
└──────────────────────┘
         ↓
    Kafka hashes key
         ↓
    Always → Partition 1

Another order from user_123:
┌──────────────────────┐
│ Key: user_123        │
│ Value: Order details │
└──────────────────────┘
         ↓
    Kafka hashes key
         ↓
    Always → Partition 1 (SAME!)

✅ All user_123 orders processed IN ORDER!

Producer Behavior

Asynchronous → Send and move on (don't wait for consumer)
High throughput → Can send thousands of events per second
Fire and forget → Ensures speed

Visualize: Entry ramps onto a highway, directing traffic into specific lanes.

Building Block #6: Consumers (Data Readers)

What It Is

Your application code that reads/subscribes to events from topics.

         Kafka Topic: "orders"
                 ↓
         ┌───────┴───────┐
         │               │
    Consumer A      Consumer B
         ↓               ↓
   Analytics App    Email Service

Each reads INDEPENDENTLY with its own position (offset)

Key Properties

1. Pull-Based Model

Traditional Systems:        Kafka:
Server → PUSHES → Client   Client ← PULLS ← Server

Benefits of Pull:
✅ Consumer controls pace
✅ Can process at own speed
✅ Can pause/resume

2. Independent Reading

Multiple consumers can read SAME topic:

Topic: "transactions"
     ↓
     ├──► Consumer A (reads everything)
     ├──► Consumer B (reads everything)
     └──► Consumer C (reads everything)

Each maintains its OWN offset (reading position)
Nobody affects anyone else! 🎭

3. Offset Tracking

Partition 0:
┌────┬────┬────┬────┬────┬────┐
│ 0  │ 1  │ 2  │ 3  │ 4  │ 5  │ ...
└────┴────┴────┴────┴────┴────┘
              ↑
         Consumer's
         current offset
         (remembers position)

If consumer stops and restarts:
✅ Resumes from last offset (position 2)
✅ No messages skipped
✅ No messages duplicated

Building Block #7: Consumer Groups (Team Work)

What It Is

A collection of consumer instances working together as a team to process events.

The Team Analogy 👥

Team A (Consumer Group "analytics"):
Worker 1, Worker 2, Worker 3

Team B (Consumer Group "email"):
Worker 4, Worker 5

Team C (Consumer Group "archiving"):
Worker 6, Worker 7, Worker 8

Each TEAM gets its own FULL COPY of the event stream!

How Consumer Groups Work

Rule: One Partition = One Consumer (within group)

Topic: "orders" (3 partitions)

Consumer Group "order-processors" (3 consumers):

Partition 0 ──► Consumer A ┐
Partition 1 ──► Consumer B ├─ Group "order-processors"
Partition 2 ──► Consumer C ┘

✅ Each partition assigned to EXACTLY ONE consumer
✅ Work is DIVIDED among team members
✅ Parallel processing! ⚡

Example: Load Distribution

Scenario 1: More partitions than consumers

Topic: 4 partitions
Group: 2 consumers

Partition 0 ──┐
Partition 1 ──┼──► Consumer A
               │
Partition 2 ──┤
Partition 3 ──┴──► Consumer B

Each consumer handles 2 partitions

Scenario 2: More consumers than partitions

Topic: 2 partitions
Group: 3 consumers

Partition 0 ──► Consumer A
Partition 1 ──► Consumer B
                Consumer C (IDLE - no partition assigned)

Extra consumers sit idle (but ready for failover!)

Scenario 3: Perfect match

Topic: 3 partitions
Group: 3 consumers

Partition 0 ──► Consumer A
Partition 1 ──► Consumer B
Partition 2 ──► Consumer C

Perfectly balanced! ⚖️

Multiple Consumer Groups (Independent Processing)

Topic: "news-feed"
     │
     ├──► Group A "website-updates"
     │    ├─ Consumer 1 → Partition 0
     │    ├─ Consumer 2 → Partition 1
     │    └─ Consumer 3 → Partition 2
     │
     ├──► Group B "archiving"
     │    ├─ Consumer 1 → Partition 0
     │    ├─ Consumer 2 → Partition 1
     │    └─ Consumer 3 → Partition 2
     │
     └──► Group C "sentiment-analysis"
          └─ Consumer 1 → All partitions

✅ Each group processes SAME data INDEPENDENTLY
✅ Each group maintains its OWN offsets
✅ Groups don't affect each other

Automatic Failover (Self-Healing)

Before failure:
Partition 0 ──► Consumer A ✅
Partition 1 ──► Consumer B ✅
Partition 2 ──► Consumer C ✅

Consumer B fails! 💥

After automatic rebalancing (seconds):
Partition 0 ──► Consumer A ✅
Partition 1 ──► Consumer A ✅ (took over!)
Partition 2 ──► Consumer C ✅

Or:
Partition 0 ──► Consumer A ✅
Partition 1 ──► Consumer C ✅ (took over!)
Partition 2 ──► Consumer C ✅

✅ No data loss!
✅ Processing continues!

Visualize: Teams of workers where each team processes the full stream, but within each team, workers divide up the lanes (partitions) to work in parallel.

THE GRAND PICTURE: How Everything Works Together 🎯

Complete Data Flow

STEP 1: PRODUCERS CREATE EVENTS
┌────────────────────────────────────────┐
│ Mobile App, Website, IoT Devices, etc. │
└────────────────┬───────────────────────┘
                 ↓
          Generate Events

STEP 2: EVENTS SENT TO TOPICS
┌─────────────────────────────────────────┐
│  Event with key "user_123"              │
│  → Kafka hashes key                     │
│  → Routes to specific partition         │
└────────────────┬────────────────────────┘
                 ↓
           Topic: "orders"

STEP 3: PARTITIONS STORE EVENTS
┌─────────────────────────────────────────┐
│ Partition 0 (Broker 1): [E1, E2, E3]   │
│ Partition 1 (Broker 2): [E4, E5, E6]   │
│ Partition 2 (Broker 3): [E7, E8, E9]   │
└────────────────┬────────────────────────┘
                 ↓
        Ordered, Immutable Log

STEP 4: CONSUMER GROUPS PULL EVENTS
┌─────────────────────────────────────────┐
│ Group "analytics":                      │
│   Consumer A reads Partition 0          │
│   Consumer B reads Partition 1          │
│   Consumer C reads Partition 2          │
│                                         │
│ Group "email":                          │
│   Consumer D reads Partition 0          │
│   Consumer E reads Partition 1, 2       │
└────────────────┬────────────────────────┘
                 ↓
         Process in parallel
         at their own pace

Visual: Complete System Architecture

┌─────────────────────────────────────────────────────────────┐
│                    KAFKA CLUSTER                            │
│                                                             │
│   ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐     │
│   │Broker 1 │  │Broker 2 │  │Broker 3 │  │Broker 4 │     │
│   ├─────────┤  ├─────────┤  ├─────────┤  ├─────────┤     │
│   │ P0 (L)  │  │ P1 (L)  │  │ P2 (L)  │  │ P3 (L)  │     │
│   │ P1 (F)  │  │ P2 (F)  │  │ P3 (F)  │  │ P0 (F)  │     │
│   │ P2 (F)  │  │ P3 (F)  │  │ P0 (F)  │  │ P1 (F)  │     │
│   └─────────┘  └─────────┘  └─────────┘  └─────────┘     │
│        ↑                           ↓                       │
│    WRITE                         READ                      │
└────────┼───────────────────────────┼──────────────────────┘
         │                           │
    ┌────┴────┐               ┌─────┴──────┐
    │PRODUCERS│               │CONSUMER    │
    │         │               │GROUPS      │
    │📱 App   │               │            │
    │🌐 Web   │               │Group A:    │
    │🌡️ IoT   │               │ C1, C2, C3 │
    │         │               │            │
    └─────────┘               │Group B:    │
                              │ C4, C5     │
                              └────────────┘

Legend:
P0 = Partition 0
(L) = Leader
(F) = Follower (replica)

Real-World Example: E-Commerce Order System

The Complete Flow

SCENARIO: Customer places an order on website

1️⃣ PRODUCER (Website) creates event:
┌──────────────────────────────────┐
│ Key: customer_789                │
│ Value: {                         │
│   order_id: "ORD-456",          │
│   items: ["laptop", "mouse"],   │
│   total: 1200                    │
│ }                                │
└──────────────────────────────────┘
         ↓

2️⃣ Kafka routes to TOPIC and PARTITION:
Topic: "orders"
Key "customer_789" → Partition 1 (always same partition!)
         ↓

3️⃣ BROKERS store in partition:
Broker 2 (Leader for Partition 1):
┌────────────────────────────────┐
│ Partition 1:                   │
│ Offset 100: ORD-453           │
│ Offset 101: ORD-454           │
│ Offset 102: ORD-456 ← NEW!    │
└────────────────────────────────┘

Broker 3 (Follower):           Broker 4 (Follower):
┌──────────────────────┐       ┌──────────────────────┐
│ Partition 1 (copy):  │       │ Partition 1 (copy):  │
│ Offset 102: ORD-456  │       │ Offset 102: ORD-456  │
└──────────────────────┘       └──────────────────────┘
         ↓                              ↓
    REPLICATED for durability!

4️⃣ MULTIPLE CONSUMER GROUPS process independently:

Group "payment-processing":
  Consumer A reads Partition 1 → Charges credit card

Group "inventory":
  Consumer B reads Partition 1 → Updates stock

Group "email":
  Consumer C reads Partition 1 → Sends confirmation

Group "analytics":
  Consumer D reads Partition 1 → Updates dashboard

✅ All process SAME order
✅ All work INDEPENDENTLY
✅ Each at their own pace

Key Principles That Make It All Work

1. Distribution

┌──────────────────────────────────┐
│ Work spread across many servers  │
│ ✅ Scalability                   │
│ ✅ Parallel processing           │
└──────────────────────────────────┘

2. Immutability

┌──────────────────────────────────┐
│ Events never change or deleted   │
│ ✅ Can be replayed               │
│ ✅ Multiple consumers can read   │
└──────────────────────────────────┘

3. Parallelism

┌──────────────────────────────────┐
│ Multiple partitions processed    │
│ simultaneously                    │
│ ✅ High throughput               │
│ ✅ Efficient resource use        │
└──────────────────────────────────┘

Fault Tolerance in Action

When Broker Fails

Before:
Broker 1 (P0-Leader) ✅
Broker 2 (P0-Follower) ✅
Broker 3 (P0-Follower) ✅

Broker 1 fails! 💥

After (2-3 seconds):
Broker 1 (P0-Leader) 💀
Broker 2 (P0-Leader) ⭐ Promoted!
Broker 3 (P0-Follower) ✅

✅ System keeps running
✅ No data loss

When Consumer Fails

Before:
Partition 0 → Consumer A ✅
Partition 1 → Consumer B ✅
Partition 2 → Consumer C ✅

Consumer B fails! 💥

After (seconds):
Partition 0 → Consumer A ✅
Partition 1 → Consumer A ✅ Took over!
Partition 2 → Consumer C ✅

✅ Processing continues
✅ No events missed

Summary: The Mental Model Checklist

The 7 Components

✅ Events - The data (immutable facts)
✅ Cluster - Network of servers
✅ Brokers - Individual servers in cluster
✅ Topics - Categories for events
✅ Partitions - Ordered lanes within topics
✅ Producers - Apps that write events
✅ Consumers - Apps that read events
✅ Consumer Groups - Teams that work together

The Flow

Producers → Topics → Partitions → Brokers
                                    ↓
                            Consumer Groups

The Guarantees

✅ Order within a partition
✅ Scalability through distribution
✅ Durability through replication
✅ Fault tolerance through automatic failover
✅ Parallel processing through partitions and consumer groups

Your Mental Model

Think of Kafka as:

🏭 A highly organized factory where:
   • Multiple assembly lines (partitions) run in parallel
   • Workers (producers) add items to lines
   • Quality checkers (consumers) inspect items
   • Teams (consumer groups) divide the work
   • Multiple facilities (brokers) ensure continuity
   • Everything is tracked and never lost

You now have a complete bird's eye view of Apache Kafka! 🦅

This mental model will be invaluable as you build applications and dive deeper into Kafka's capabilities. Every detail you learn will fit into this bigger picture! 🎯

🎮 Understanding Linux Cgroups

Ajinkya Singh — Sun, 16 Nov 2025 15:38:59 +0000

🚀 I'm Building My Own Container Runtime!

This is part of a complete series where I'm building Conti - a container runtime from scratch. Check it out on GitHub!

About This Series:

I'm sharing everything I learn while building my own container runtime

Most concepts come from videos, documentation, and LLM-assisted learning (for educational purposes)

Focus: Understanding through practice - raw Linux commands and practical implementation

Important: When building your own container, DON'T copy code from sources - it kills the fun! Write it yourself, break things, debug, and learn.

Why Build Your Own?

Deep understanding of how containers really work

Master low-level Linux concepts

Learn by doing, not just reading

It's incredibly fun when things finally click!

What are Cgroups?

Control Groups (Cgroups) are a Linux kernel feature that allows you to allocate, limit, and monitor system resources for processes and groups of processes.

The Core Formula

Cgroups = Resource Control + Resource Isolation + Resource Accounting

Why Do We Need Cgroups?

Imagine you're running a web server that hosts multiple applications:

E-commerce Store: Needs 8GB RAM, 40% CPU
Analytics Dashboard: Needs 4GB RAM, 30% CPU
Email Service: Needs 2GB RAM, 20% CPU

Without cgroups, these applications could:

❌ Steal resources from each other
❌ Cause system crashes
❌ Create unpredictable performance

With cgroups, you can:

✅ Guarantee each application gets its resources
✅ Prevent any single app from hogging the system
✅ Maintain stable, predictable performance

Real-World Analogy

Think of a shopping mall with limited parking spaces:

Without Cgroups (No Rules)

Early shoppers take all parking spaces
Late arrivals have nowhere to park
Chaos and complaints

With Cgroups (Managed Parking)

Restaurant customers: 40 spaces reserved
Movie theater guests: 30 spaces reserved
Retail shoppers: 30 spaces reserved

Each group is guaranteed their allocation, and no group can exceed their limit.

How Cgroups Work

Step-by-Step Process

1. Define Resource Groups
   └─> Create named groups for each resource type

2. Set Resource Limits
   └─> Assign specific limits to each group

3. Launch Applications
   └─> Start apps within their assigned groups

4. Kernel Enforces Limits
   └─> Automatic enforcement, no manual intervention needed

Cgroup Hierarchy

All cgroups live in the filesystem at /sys/fs/cgroup/:

/sys/fs/cgroup/
├── memory/          # RAM control
├── cpu/             # CPU time control
├── cpuset/          # CPU core assignment
├── blkio/           # Disk I/O control
├── net_cls/         # Network control
└── devices/         # Device access control

Resource Types

1. Memory Control

Maximum RAM allocation
Swap space limits
Out-of-memory (OOM) handling
Memory usage tracking

2. CPU Control

CPU percentage allocation
CPU core assignment (pinning)
Scheduling priorities
Multi-core management

3. Network Control

Bandwidth limitations
Priority queuing
Traffic shaping
Network class assignments

4. Disk I/O Control

Read/write speed limits
IOPS (operations per second)
Device-specific quotas
Priority levels

Memory Management

Creating a Memory Control Group

Scenario: You want to run a data processing script that shouldn't use more than 20MB of RAM.

Step 1: Create the Control Group

# Create a new control group called 'data_processor'
cgcreate -g memory:data_processor

Step 2: Set Memory Limit

# Set memory limit to 20MB (20 × 1024 × 1024 = 20,971,520 bytes)
cgset -r memory.limit_in_bytes=20971520 data_processor

# Also limit swap to prevent workarounds
cgset -r memory.memsw.limit_in_bytes=20971520 data_processor

Step 3: Run Your Application

# Execute the script within the control group
cgexec -g memory:data_processor python3 process_data.py

What Happens When Memory Limit Is Exceeded?

Application starts
    ↓
Allocates memory
    ↓
Reaches 20MB limit
    ↓
Tries to allocate more
    ↓
┌─────────────────┐
│ Kernel detects  │
│   violation     │
└────────┬────────┘
         ↓
┌─────────────────┐
│  OOM Killer     │
│  Activated      │
└────────┬────────┘
         ↓
   Process killed

Monitoring Memory Usage

# Check current memory usage
cat /sys/fs/cgroup/memory/data_processor/memory.usage_in_bytes

# Check if OOM killer was triggered
cat /sys/fs/cgroup/memory/data_processor/memory.oom_control

CPU Control

There are two methods to control CPU usage:

Method 1: Hard Limits (CFS Quota)

Use this when you need absolute limits that are always enforced.

The Formula

CPU Limit % = (cfs_quota_us / cfs_period_us) × 100

Where:

cfs_period_us: Time window (default: 100,000 μs = 100ms)
cfs_quota_us: CPU time allowed in that window

Example: Limit to 35% CPU

# Step 1: Create control group
cgcreate -g cpu:video_encoder

# Step 2: Set 35% limit
# 35% of 100,000 = 35,000 microseconds
cgset -r cpu.cfs_quota_us=35000 video_encoder

# Step 3: Run application
cgexec -g cpu:video_encoder ffmpeg -i input.mp4 output.mp4

Common CPU Percentages

Desired CPU	cfs_quota_us	Calculation
10%	10,000	10,000 / 100,000
25%	25,000	25,000 / 100,000
50%	50,000	50,000 / 100,000
75%	75,000	75,000 / 100,000

Method 2: Soft Limits (CPU Shares)

Use this when you want proportional sharing during resource contention.

Key Concept

Maximum value: 1024 (represents 100%)
Only enforced when multiple processes compete for CPU
If CPU is idle, processes can use more than their share

The Formula

Process CPU % = (process_shares / total_shares) × 100

Example: Three Applications Sharing CPU

Scenario: You have three services running:

Web Server: Should get 50% during contention
Background Jobs: Should get 30% during contention
Monitoring: Should get 20% during contention

# Create control groups
cgcreate -g cpu:web_server
cgcreate -g cpu:background_jobs
cgcreate -g cpu:monitoring

# Set CPU shares
cgset -r cpu.shares=512 web_server      # 512/1024 = 50%
cgset -r cpu.shares=307 background_jobs # 307/1024 ≈ 30%
cgset -r cpu.shares=205 monitoring      # 205/1024 ≈ 20%

# Launch applications
cgexec -g cpu:web_server nginx
cgexec -g cpu:background_jobs python worker.py
cgexec -g cpu:monitoring ./monitor.sh

Hard Limits vs Soft Limits

Aspect	Hard Limits (Quota)	Soft Limits (Shares)
Type	Absolute ceiling	Relative distribution
Enforcement	Always active	Only during contention
Idle CPU	Wasted if limit reached	Fully utilized
Use Case	Strict isolation	Flexible sharing
Analogy	Speed limiter in car	Highway lanes

Visualization

Hard Limit (30% quota):

CPU Available: 100%
Process usage: 30% ████████
Unused:        70% (wasted even if CPU is idle)

Soft Limit (30% shares, no contention):

CPU Available: 100%
Process usage: 100% ████████████████████████████████
(Can use full CPU when alone)

Soft Limit (30% shares, with contention):

CPU Available: 100%
Process 1:     30% ████████
Process 2:     40% ███████████
Process 3:     30% ████████
(Shares enforced when competing)

Practical Examples

Example 1: Development Environment

Scenario: You're running Docker containers for development:

Database: 2GB RAM, 40% CPU
API Server: 1GB RAM, 30% CPU
Redis Cache: 512MB RAM, 20% CPU

# Database container
cgcreate -g memory,cpu:dev_database
cgset -r memory.limit_in_bytes=2147483648 dev_database
cgset -r cpu.cfs_quota_us=40000 dev_database

# API Server container
cgcreate -g memory,cpu:dev_api
cgset -r memory.limit_in_bytes=1073741824 dev_api
cgset -r cpu.cfs_quota_us=30000 dev_api

# Redis Cache container
cgcreate -g memory,cpu:dev_redis
cgset -r memory.limit_in_bytes=536870912 dev_redis
cgset -r cpu.cfs_quota_us=20000 dev_redis

Example 2: Machine Learning Training

Scenario: Training ML model that needs lots of resources but shouldn't crash the system.

# Create control group with generous limits
cgcreate -g memory,cpu:ml_training
cgset -r memory.limit_in_bytes=16106127360 ml_training  # 15GB
cgset -r cpu.cfs_quota_us=300000 ml_training  # 300% (3 full cores)

# Run training script
cgexec -g memory,cpu:ml_training python train_model.py

Example 3: CI/CD Pipeline

Scenario: Running automated tests that shouldn't hog server resources.

# Create control group for CI jobs
cgcreate -g memory,cpu:ci_runner
cgset -r memory.limit_in_bytes=4294967296 ci_runner  # 4GB
cgset -r cpu.shares=256 ci_runner  # Low priority (25% share)

# Run tests
cgexec -g memory,cpu:ci_runner npm test

Example 4: Multi-Tenant Web Hosting

Scenario: Hosting multiple customer websites on one server.

# Customer A - Premium tier
cgcreate -g memory,cpu:customer_a
cgset -r memory.limit_in_bytes=8589934592 customer_a  # 8GB
cgset -r cpu.shares=768 customer_a  # 75% share

# Customer B - Standard tier
cgcreate -g memory,cpu:customer_b
cgset -r memory.limit_in_bytes=4294967296 customer_b  # 4GB
cgset -r cpu.shares=256 customer_b  # 25% share

Understanding Container Magic: The Overlay Filesystem Story

Ajinkya Singh — Thu, 13 Nov 2025 19:04:16 +0000

🚀 I'm Building My Own Container Runtime!

This is part of a complete series where I'm building Conti - a container runtime from scratch. Check it out on GitHub!

About This Series:

I'm sharing everything I learn while building my own container runtime

Most concepts come from videos, documentation, and LLM-assisted learning (for educational purposes)

Focus: Understanding through practice - raw Linux commands and practical implementation

Important: When building your own container, DON'T copy code from sources - it kills the fun! Write it yourself, break things, debug, and learn.

Why Build Your Own?

Deep understanding of how containers really work

Master low-level Linux concepts

Learn by doing, not just reading

It's incredibly fun when things finally click!

How modern containers achieve incredible efficiency and isolation

🎬 Act 1: The Party Problem

Imagine you're organizing a massive tech conference with three separate tracks happening simultaneously:

🎨 The Design Track - UI/UX designers discussing Figma and Sketch

💻 The DevOps Track - Engineers exploring Kubernetes and CI/CD

🤖 The AI Track - Researchers presenting machine learning models

Each track needs:

A presentation room (isolated space)
Audio/visual equipment (shared resources)
Custom materials (track-specific content)
Whiteboards for notes (temporary changes)

The Expensive Approach ❌

You could buy three complete sets of everything:

3 projectors @ $2,000 each = $6,000
3 sound systems @ $1,500 each = $4,500
3 furniture sets @ $3,000 each = $9,000

Total: $19,500 😱

The Smart Approach ✅

What if you could:

Share the expensive equipment (projectors, furniture)
Give each track private notebooks for their own notes
Let them customize only what they need

Total: $6,500 + 3 notebooks

This is exactly what overlay filesystems do for containers!

🏗️ Act 2: Understanding the Architecture

The Three-Layer Cake 🎂

Think of the overlay filesystem as a delicious three-layer cake:

┌─────────────────────────────────────┐
│     🍰 MERGED LAYER (What You See)   │ ← The complete cake you eat
│     Combines everything below        │
├─────────────────────────────────────┤
│     ✏️  UPPER LAYER (Your Changes)   │ ← Your frosting and decorations
│     Read/Write - Personal Edits      │
├─────────────────────────────────────┤
│     📦 LOWER LAYER (Base Recipe)     │ ← The original cake (untouched)
│     Read-Only - Shared Foundation    │
└─────────────────────────────────────┘

🎯 The Lower Layer (The Foundation)

Read-only - Like a published cookbook
Contains your base operating system (Ubuntu, Alpine Linux)
Shared across ALL containers
Never gets modified

✍️ The Upper Layer (Your Workspace)

Read/write - Your personal notepad
Stores all YOUR changes
Unique to each container
Where creativity happens!

👀 The Merged Layer (The Magic View)

What users actually see and interact with
Automatically combines Lower + Upper
Feels like one complete filesystem

🎪 Act 3: The Copy-on-Write Magic Show

Let me show you the most clever trick in the container world: Copy-on-Write (COW)

Scenario: The Recipe Book 📖

Imagine you have a master recipe book (lower layer) with 100 recipes.

Day 1: You read "Chocolate Cake" → No copying needed!

You're just reading from the shared book. Fast and efficient!

Day 2: You want to modify "Chocolate Cake" → NOW the magic happens!

STEP 1: System spots you editing
   ↓
STEP 2: "Hold on! Let me copy this page to YOUR notebook!"
   ↓
STEP 3: Copy "Chocolate Cake" to upper layer
   ↓
STEP 4: You modify YOUR copy
   ↓
STEP 5: Original recipe stays pristine ✨

The Result:

Master book: Still has original "Chocolate Cake" ✅
Your notebook: Has your modified "Spicy Chocolate Cake" ✅
Everyone else: Still sees the original recipe ✅

Real Container Example

# Container starts with nginx.conf from base image
$ cat /etc/nginx/nginx.conf
→ Reading from LOWER layer (shared base image)

# You edit the config file
$ echo "worker_processes 4;" >> /etc/nginx/nginx.conf
→ System COPIES file to UPPER layer
→ Your edit goes into YOUR copy
→ Base image remains unchanged!

# Your change is private to YOUR container
→ 1000 other nginx containers still see the original

🎯 Act 4: The Real-World Impact

Before Overlay FS: The Dark Ages 😱

Launching 100 web applications:

App 1:  Ubuntu (500MB) + Node.js (100MB) = 600MB
App 2:  Ubuntu (500MB) + Node.js (100MB) = 600MB
App 3:  Ubuntu (500MB) + Node.js (100MB) = 600MB
...
App 100: Ubuntu (500MB) + Node.js (100MB) = 600MB

TOTAL: 60,000 MB (60 GB!) 💀

After Overlay FS: The Renaissance ✨

Shared Ubuntu:     500MB (one time!)
Shared Node.js:    100MB (one time!)
App 1 changes:     2MB
App 2 changes:     3MB
App 3 changes:     1MB
...
App 100 changes:   2MB

TOTAL: 600MB + 200MB changes = 800MB 🎉

SAVED: 59.2 GB (98.7% reduction!)

🛠️ Act 5: Hands-On Workshop

Let's build our own overlay filesystem step-by-step!

Workshop Setup: Three Coffee Shops ☕

You're running three coffee shops, each needs menus and custom recipes.

Step 1: Create the Directory Structure

# Our shared base (the franchise template)
mkdir -p coffee-franchise/{base,shop-downtown,shop-uptown,shop-campus}
mkdir -p coffee-franchise/work-{downtown,uptown,campus}
mkdir -p coffee-franchise/merged-{downtown,uptown,campus}

cd coffee-franchise

# Create shared base menu (lower layer)
echo "1. Espresso - $3" > base/menu.txt
echo "2. Latte - $4" >> base/menu.txt
echo "3. Cappuccino - $4.50" >> base/menu.txt
echo "Standard Coffee Brewing Guide" > base/brewing.txt

Step 2: Mount Downtown Shop (First Container)

# Create the overlay filesystem
sudo mount -t overlay overlay \
  -o lowerdir=base,upperdir=shop-downtown,workdir=work-downtown \
  merged-downtown

# Check what's visible
ls merged-downtown/
# Output: menu.txt  brewing.txt (from base!)

Step 3: Make Shop-Specific Changes

# Downtown caters to business people - add premium options
echo "4. Cold Brew - $5" >> merged-downtown/menu.txt
echo "Downtown Specialty: Extra Strong!" > merged-downtown/special.txt

# Check what happened:
ls base/
# Output: menu.txt  brewing.txt (unchanged!)

ls shop-downtown/
# Output: menu.txt  special.txt (YOUR changes only!)

Step 4: Create Another Shop (Second Container)

# Mount campus shop
sudo mount -t overlay overlay \
  -o lowerdir=base,upperdir=shop-uptown,workdir=work-uptown \
  merged-uptown

# Campus caters to students - add budget options
echo "4. Small Coffee - $2" >> merged-uptown/menu.txt
echo "Campus Special: Student Discount 20%!" > merged-uptown/special.txt

# Each shop sees different menus!
cat merged-downtown/special.txt
# Output: Downtown Specialty: Extra Strong!

cat merged-uptown/special.txt
# Output: Campus Special: Student Discount 20%!

🎊 What We Achieved:

✅ One base menu shared across all shops

✅ Custom modifications per shop

✅ Complete isolation - changes don't affect others

✅ Massive space savings - one copy of common files

🎭 Act 6: The Delete Mystery

What happens when you delete a file from the lower layer?

# Try to delete the base menu from downtown
rm merged-downtown/menu.txt

# The file disappears from downtown view... but wait!
ls merged-downtown/
# Output: brewing.txt  special.txt (menu.txt is gone!)

# Check the other shop
ls merged-uptown/
# Output: menu.txt  brewing.txt  special.txt (still there!)

# The secret: a WHITEOUT file
ls -la shop-downtown/
# Output: c--------- 1 root root 0, 0 menu.txt (special marker!)

The Magic:

Instead of deleting from base (impossible - it's read-only!), the system creates a special "whiteout" marker that hides the file only in your view.

🚀 Act 7: Real Container Architecture

Here's how Docker/Kubernetes actually use this:

┌─────────────────────────────────────────┐
│   YOUR RUNNING CONTAINER (Merged)       │
│   What you see when you 'docker exec'   │
├─────────────────────────────────────────┤
│   CONTAINER LAYER (Upper) ✏️            │
│   - Your app logs                        │
│   - Temp files                           │
│   - Runtime changes                      │
├─────────────────────────────────────────┤
│   APP LAYER (Lower 3) 📱                │
│   - Your application code                │
│   - App dependencies                     │
├─────────────────────────────────────────┤
│   RUNTIME LAYER (Lower 2) ⚙️            │
│   - Python/Node.js/Java                  │
│   - Libraries                            │
├─────────────────────────────────────────┤
│   BASE OS LAYER (Lower 1) 🏗️            │
│   - Ubuntu/Alpine Linux                  │
│   - Core utilities                       │
└─────────────────────────────────────────┘
     ↑
     └─ All "Lower" layers are SHARED!

The Dockerfile Connection

FROM ubuntu:20.04           # ← Lower Layer 1 (shared!)
RUN apt-get install python3  # ← Lower Layer 2 (shared!)
COPY app.py /app/           # ← Lower Layer 3 (shared!)
CMD ["python3", "app.py"]    # ← Upper Layer (YOUR changes!)

Each line creates a layer. When you run 1000 containers from this image:

Ubuntu layer: Stored ONCE
Python layer: Stored ONCE
App layer: Stored ONCE
Runtime changes: 1000 unique upper layers (small!)