DEV Community

Yash Sonawane
Yash Sonawane

Posted on

I Deleted a Kubernetes Pod on Purpose — Here's What Actually Happened

#devops #kubernetes #docker #learning

If you're learning Kubernetes, here's a question you should be able to answer:

What happens when a production Pod suddenly disappears?

Not theoretically.

Let's actually think through it.

Imagine your application has:

3 Pods
 ↓
frontend
frontend
frontend
Enter fullscreen mode Exit fullscreen mode

Then someone runs:

kubectl delete pod frontend-7d8f9c6d-x82lm
Enter fullscreen mode Exit fullscreen mode

The Pod disappears.

Your first reaction might be:

"Oh no! The application is down!"

But if the application is managed correctly, something interesting happens.

Kubernetes creates another Pod.

And that's one of the most important ideas behind Kubernetes:

You don't manage individual containers. You declare the state you want, and Kubernetes works to maintain it.

🧠 Let's Break It Down

Suppose we have:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: frontend

spec:
  replicas: 3

  selector:
    matchLabels:
      app: frontend

  template:
    metadata:
      labels:
        app: frontend

    spec:
      containers:
        - name: frontend
          image: nginx:1.27
Enter fullscreen mode Exit fullscreen mode

We apply it:

kubectl apply -f deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Kubernetes creates:

Deployment
    ↓
ReplicaSet
    ↓
┌────────┬────────┬────────┐
│ Pod 1  │ Pod 2  │ Pod 3  │
└────────┴────────┴────────┘
Enter fullscreen mode Exit fullscreen mode

The important part is:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

We're not telling Kubernetes:

"Keep these exact three Pods alive forever."

We're saying:

"I want three healthy replicas of this workload."

That's a huge difference.

💥 Now Let's Delete One

Run:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

You might see:

NAME                         READY   STATUS
frontend-7d8f9c6d-x82lm      1/1     Running
frontend-7d8f9c6d-k92px      1/1     Running
frontend-7d8f9c6d-m45rt      1/1     Running
Enter fullscreen mode Exit fullscreen mode

Now:

kubectl delete pod frontend-7d8f9c6d-x82lm
Enter fullscreen mode Exit fullscreen mode

Run:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

You might briefly see:

frontend-7d8f9c6d-x82lm      Terminating
frontend-7d8f9c6d-k92px      Running
frontend-7d8f9c6d-m45rt      Running
frontend-7d8f9c6d-p91xz      ContainerCreating
Enter fullscreen mode Exit fullscreen mode

A new Pod appears.

Eventually:

frontend-7d8f9c6d-k92px      Running
frontend-7d8f9c6d-m45rt      Running
frontend-7d8f9c6d-p91xz      Running
Enter fullscreen mode Exit fullscreen mode

Back to three.

🤯 Who Created the New Pod?

This is where understanding Kubernetes becomes much more important than memorizing commands.

You deleted a Pod.

But the Pod wasn't responsible for maintaining itself.

The Deployment manages a ReplicaSet.

The ReplicaSet ensures the desired number of Pods exists.

So the chain looks like:

Deployment
     ↓
ReplicaSet
     ↓
Pods
Enter fullscreen mode Exit fullscreen mode

You delete:

Pod
Enter fullscreen mode Exit fullscreen mode

The ReplicaSet notices:

Desired: 3
Current: 2
Enter fullscreen mode Exit fullscreen mode

So it creates another Pod.

Now:

Desired: 3
Current: 3
Enter fullscreen mode Exit fullscreen mode

The system has reconciled itself.

🔄 This Is the Kubernetes Reconciliation Loop

This is one of the most important concepts in Kubernetes.

Think of it as:

Desired State
     │
     ▼
┌───────────────┐
│ Kubernetes    │
│ Controllers   │
└───────┬───────┘
        │
        ▼
Current State
        │
        ▼
   Difference?
        │
       Yes
        │
        ▼
    Take Action
        │
        ▼
Current State
        │
        └──────→ Repeat
Enter fullscreen mode Exit fullscreen mode

Kubernetes constantly works toward the desired state.

That's why Kubernetes isn't simply:

"A tool for running Docker containers."

It's a control system for managing desired state.

🔥 But What If a Node Dies?

Now let's make things more interesting.

Suppose:

Node 1
 ├── Pod A
 ├── Pod B
 └── Pod C
Enter fullscreen mode Exit fullscreen mode

Suddenly:

Node 1
    💥
Enter fullscreen mode Exit fullscreen mode

What happens?

It depends on your cluster configuration and workload, but Kubernetes can detect node failure and, where appropriate, reschedule workloads onto healthy nodes.

For example:

Before:

Node 1
 ├── Pod A
 └── Pod B

Node 2
 └── Pod C
Enter fullscreen mode Exit fullscreen mode

After failure:

Node 1
   ❌

Node 2
 ├── Pod C
 ├── Pod A
 └── Pod B
Enter fullscreen mode Exit fullscreen mode

assuming the cluster has sufficient resources and the workload is eligible for rescheduling.

This is where Kubernetes starts becoming really powerful.

🚨 But Kubernetes Isn't Magic

Here's something beginners often misunderstand.

Kubernetes doesn't automatically make your application highly available.

You need to design for it.

For example:

replicas: 1
Enter fullscreen mode Exit fullscreen mode

means you have only one replica.

If that Pod disappears:

Application
    ↓
1 Pod
    ↓
Pod fails
    ↓
No healthy replica
Enter fullscreen mode Exit fullscreen mode

Kubernetes can recreate the Pod, but there can still be a period where your application isn't serving traffic.

With:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

you have more redundancy.

        Service
           │
    ┌──────┼──────┐
    ▼      ▼      ▼
  Pod 1  Pod 2  Pod 3
Enter fullscreen mode Exit fullscreen mode

If one fails:

        Service
           │
      ┌────┴────┐
      ▼         ▼
    Pod 2     Pod 3

       Pod 1 ❌
Enter fullscreen mode Exit fullscreen mode

The workload can recover while other replicas continue serving traffic.

But even this isn't enough for every production scenario.

🌎 Spread Your Workloads

Imagine all three Pods are running on the same node:

Node 1
 ├── Pod 1
 ├── Pod 2
 └── Pod 3
Enter fullscreen mode Exit fullscreen mode

Node 1 fails.

Now all three Pods are affected.

A better design may distribute replicas:

Node 1
 └── Pod 1

Node 2
 └── Pod 2

Node 3
 └── Pod 3
Enter fullscreen mode Exit fullscreen mode

Now a single-node failure doesn't necessarily take down every replica.

This is where concepts such as:

  • Pod topology spread constraints
  • Pod anti-affinity
  • Node affinity
  • Taints and tolerations

become important.

❤️ Kubernetes Needs a Health Check

There's another problem.

What if the container is running but the application is broken?

For example:

Container:
Running ✅

Application:
Database connection broken ❌
Enter fullscreen mode Exit fullscreen mode

Kubernetes may still see a running container.

That's why health probes matter.

🩺 Readiness Probe

A readiness probe answers:

"Can this Pod receive traffic?"

For example:

readinessProbe:
  httpGet:
    path: /health
    port: 8080
Enter fullscreen mode Exit fullscreen mode

If the application isn't ready, Kubernetes can keep the Pod out of Service traffic.

So you can have:

Pod 1 → Ready
Pod 2 → Ready
Pod 3 → Not Ready
Enter fullscreen mode Exit fullscreen mode

Traffic can continue going to healthy replicas.

💓 Liveness Probe

A liveness probe answers a different question:

"Is this container still healthy enough to keep running?"

If it repeatedly fails, Kubernetes may restart the container.

So:

Readiness
   ↓
Should traffic be sent here?

Liveness
   ↓
Should this container keep running?
Enter fullscreen mode Exit fullscreen mode

Don't confuse them.

It's one of the most common Kubernetes interview questions.

📈 What Happens During an Update?

Now let's say you have:

Version 1
Enter fullscreen mode Exit fullscreen mode

running across three Pods.

You deploy:

Version 2
Enter fullscreen mode Exit fullscreen mode

A Deployment can perform a rolling update.

Conceptually:

v1   v1   v1
 │
 ▼
v2   v1   v1
 │
 ▼
v2   v2   v1
 │
 ▼
v2   v2   v2
Enter fullscreen mode Exit fullscreen mode

Instead of destroying everything at once, Kubernetes can gradually replace old replicas with new ones according to the Deployment strategy and configuration.

This is one reason Deployments are so useful.

🔙 What If Version 2 Is Broken?

This is where rollout management becomes important.

Check rollout status:

kubectl rollout status deployment/frontend
Enter fullscreen mode Exit fullscreen mode

View history:

kubectl rollout history deployment/frontend
Enter fullscreen mode Exit fullscreen mode

And, when appropriate:

kubectl rollout undo deployment/frontend
Enter fullscreen mode Exit fullscreen mode

Now you can move back toward a previous revision.

That's much better than:

"I hope someone remembers exactly what we changed."

🔥 Kubernetes Is Really About Desired State

Let's simplify everything.

You declare:

I want:
3 replicas
Enter fullscreen mode Exit fullscreen mode

Kubernetes observes:

I currently have:
2 replicas
Enter fullscreen mode Exit fullscreen mode

It acts.

2 → 3
Enter fullscreen mode Exit fullscreen mode

You declare:

I want:
version 2
Enter fullscreen mode Exit fullscreen mode

Kubernetes currently has:

version 1
Enter fullscreen mode Exit fullscreen mode

It acts.

v1 → v2
Enter fullscreen mode Exit fullscreen mode

You declare:

I want:
3 healthy replicas
Enter fullscreen mode Exit fullscreen mode

One Pod crashes.

Kubernetes observes:

2 healthy replicas
Enter fullscreen mode Exit fullscreen mode

It acts.

2 → 3
Enter fullscreen mode Exit fullscreen mode

That is the core idea.

🧪 Try This Yourself

If you're learning Kubernetes, don't just read this article.

Run the experiment.

Create a Deployment:

kubectl create deployment demo --image=nginx
Enter fullscreen mode Exit fullscreen mode

Scale it:

kubectl scale deployment demo --replicas=3
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

Delete one:

kubectl delete pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

Then immediately run:

kubectl get pods -o wide
Enter fullscreen mode Exit fullscreen mode

Watch what happens.

You can even watch continuously:

kubectl get pods -w
Enter fullscreen mode Exit fullscreen mode

Now you're not just reading about Kubernetes.

You're observing the control loop yourself.

🎯 The Question Every Kubernetes Beginner Should Ask

Don't ask:

"Which kubectl command should I memorize?"

Ask:

"What controller is responsible for this resource?"

For example:

Pod
 ↓
ReplicaSet?
 ↓
Deployment?
Enter fullscreen mode Exit fullscreen mode

Or:

Service
 ↓
Selector
 ↓
Endpoints
 ↓
Pods
Enter fullscreen mode Exit fullscreen mode

Or:

Ingress
 ↓
Ingress Controller
 ↓
Service
 ↓
Pods
Enter fullscreen mode Exit fullscreen mode

Once you understand these relationships, troubleshooting becomes dramatically easier.

🧠 My Kubernetes Learning Rule

Here's the rule I recommend:

Never memorize a Kubernetes command without understanding what state it is inspecting or changing.

For example:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Don't just remember the command.

Understand that you're asking Kubernetes:

"Show me the current state of the Pods."

And:

kubectl scale deployment demo --replicas=5
Enter fullscreen mode Exit fullscreen mode

isn't just a command.

You're changing the desired state:

Desired:
3

New Desired:
5
Enter fullscreen mode Exit fullscreen mode

That mental model is far more powerful.

🚀 Build This Project

If you want to turn this knowledge into a real DevOps project, build:

Self-Healing Kubernetes Application

Architecture:

                    Users
                      │
                      ▼
                   Ingress
                      │
                      ▼
                   Service
                      │
          ┌───────────┼───────────┐
          ▼           ▼           ▼
        Pod 1       Pod 2       Pod 3
          │           │           │
          └───────────┼───────────┘
                      ▼
                   Backend
                      │
                ┌─────┴─────┐
                ▼           ▼
             Redis       Database
Enter fullscreen mode Exit fullscreen mode

Then implement:

✓ 3 replicas
✓ Readiness probe
✓ Liveness probe
✓ Resource requests
✓ Resource limits
✓ Rolling updates
✓ Rollbacks
✓ Horizontal Pod Autoscaler
✓ Monitoring
✓ Centralized logging
✓ Helm
✓ CI/CD
✓ GitOps
Enter fullscreen mode Exit fullscreen mode

Now you have something you can explain in an interview:

"I built a Kubernetes application that automatically maintains replicas, performs rolling updates, exposes health checks, and is deployed through a GitOps workflow."

That's much stronger than:

"I know Kubernetes."

📚 Want to Learn Kubernetes From Beginner to CKA?

If you're preparing for the Certified Kubernetes Administrator (CKA), I've created a structured study guide:

CKA Complete Study Guide — Certified Kubernetes Administrator

📘 Get the book here:

CKA Complete Study Guide

The best way to use a book like this isn't:

Read → Finish → Forget
Enter fullscreen mode Exit fullscreen mode

Instead:

Read
 ↓
Understand
 ↓
Open a Kubernetes cluster
 ↓
Practice
 ↓
Break something
 ↓
Troubleshoot
 ↓
Repeat
Enter fullscreen mode Exit fullscreen mode

That's how Kubernetes becomes a real skill.

🛠️ More Resources for Your DevOps Journey

🐳 Docker Mastery

Docker Mastery

🏗️ Terraform Associate Crash Course

Terraform Associate Crash Course

🔀 Git Mastery

Git Mastery

⚙️ DevOps Complete Pack

DevOps Complete Pack

🐹 Mastering Go

Mastering Go

Final Thoughts

The most important Kubernetes lesson isn't:

kubectl delete pod
Enter fullscreen mode Exit fullscreen mode

It's understanding what happens after you run it.

You delete a Pod.

The controller notices the desired state isn't satisfied.

A new Pod is created.

The Service routes traffic to available replicas.

Health probes determine whether Pods are ready.

Deployments manage application versions.

Kubernetes continuously works toward the state you've declared.

That's the magic.

Not because Kubernetes is doing something mysterious.

But because you've moved from manually managing servers to declaring the state you want and letting controllers continuously reconcile the system.

So the next time you see a Kubernetes Pod disappear, don't panic.

Ask one question:

"Who is responsible for bringing the system back to the desired state?"

Once you can answer that question, you're no longer just memorizing Kubernetes.

You're starting to think like a Kubernetes engineer.

Top comments (0)