DEV Community

Yash Sonawane
Yash Sonawane

Posted on

☸️ Kubernetes Is Not Healing Your Application. It's Just Restarting It. Here's What You Need to Know.

Your Pod crashed 15 times. Kubernetes restarted it 15 times. And your application is still broken.

Welcome to Kubernetes. 😂

Imagine deploying an application:

kubectl apply -f deployment.yaml
Enter fullscreen mode Exit fullscreen mode

You check the Pods:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

And see:

NAME                       READY   STATUS             RESTARTS
payment-api-7d8f9c6b-x2k9   0/1     CrashLoopBackOff   15
Enter fullscreen mode Exit fullscreen mode

You delete the Pod.

Kubernetes creates another one.

It crashes again.

You restart the Deployment.

It crashes again.

At this point, you're wondering:

"Is Kubernetes actually intelligent, or is it just restarting things forever?" 💀

Here's the important lesson:

Kubernetes is designed to maintain the desired state of your infrastructure. It cannot automatically understand or fix every application failure.

Let's explore how Kubernetes actually works, why applications fail, and how to troubleshoot them like a real DevOps engineer.


🧠 1. Kubernetes Doesn't Manage Your Application the Way You Think

One of the biggest misunderstandings among Kubernetes beginners is believing that Kubernetes automatically makes applications healthy.

That's not how it works.

Kubernetes continuously observes the state of your cluster and reconciles the actual state toward the desired state.

For example, suppose you deploy an application with three replicas.

spec:
  replicas: 3
Enter fullscreen mode Exit fullscreen mode

You're telling Kubernetes:

"I want three instances of this application running."

Kubernetes attempts to maintain that requirement.

If one Pod disappears, its controller attempts to create a replacement.

             Desired State
             3 Replicas
                 |
                 v
          Kubernetes Controller
                 |
                 v
          Actual Cluster State
                 |
        +--------+--------+
        |        |        |
        v        v        v
      Pod 1    Pod 2    Pod 3
Enter fullscreen mode Exit fullscreen mode

This is called the reconciliation loop.

The controller repeatedly observes the current state, compares it with the desired state, and takes corrective action.

But remember:

If your application has a bug, Kubernetes may repeatedly recreate the broken workload.

It doesn't magically repair your source code.

🔥 2. Understand the Difference Between Pod and Container

This is an important Kubernetes concept.

A Pod is the smallest deployable unit in Kubernetes.

A Pod can contain one or more containers that share networking and can share storage volumes.

For example:

Kubernetes Cluster
       |
       v
      Pod
       |
   +---+---+
   |       |
   v       v
 App     Sidecar
Container Container
Enter fullscreen mode Exit fullscreen mode

If a container crashes, its restart behavior depends on the Pod's restart policy and workload controller.

If a Pod is deleted, a Deployment-managed ReplicaSet can create another Pod to maintain the desired replica count.

Important: A replacement Pod is not necessarily the same Pod. It receives a new identity and may be scheduled on a different node.

This distinction becomes very important when troubleshooting production workloads.

💣 3. Five Kubernetes Problems Every DevOps Engineer Should Understand

Let's look at common problems you'll encounter in real clusters.

Problem 1: CrashLoopBackOff

Your application repeatedly crashes.

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Possible causes:

  • Application startup error.
  • Incorrect environment variables.
  • Database connection failure.
  • Incorrect startup command.
  • Application exiting unexpectedly.

Start troubleshooting:

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

Read the previous container logs:

kubectl logs <pod-name> --previous
Enter fullscreen mode Exit fullscreen mode

Don't randomly delete Pods before understanding the failure.

Problem 2: ImagePullBackOff

Kubernetes cannot successfully pull the requested container image.

Example:

NAME       READY   STATUS
my-app     0/1     ImagePullBackOff
Enter fullscreen mode Exit fullscreen mode

Possible causes:

  • Incorrect image name.
  • Nonexistent image tag.
  • Private registry authentication problem.
  • Registry connectivity issue.

Inspect the Pod:

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

Check the image reference:

kubectl get pod <pod-name> \
  -o jsonpath='{.spec.containers[*].image}'
Enter fullscreen mode Exit fullscreen mode

Verify that the image exists and the cluster can access the registry.

Problem 3: Pending Pods

Your Pod is not getting scheduled.

NAME       READY   STATUS
my-app     0/1     Pending
Enter fullscreen mode Exit fullscreen mode

Possible causes:

  • Insufficient CPU or memory.
  • Node selectors don't match available nodes.
  • Untolerated taints.
  • PersistentVolumeClaim problems.

Investigate:

kubectl describe pod <pod-name>
kubectl get nodes
kubectl get pvc
Enter fullscreen mode Exit fullscreen mode

The Events section of describe often provides useful clues.

Problem 4: OOMKilled

Your container has been terminated because of an out-of-memory condition.

Check:

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

You might find:

Last State: Terminated
Reason: OOMKilled
Enter fullscreen mode Exit fullscreen mode

Inspect resource usage:

kubectl top pods
Enter fullscreen mode Exit fullscreen mode

Review resource requests and limits:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "500m"
Enter fullscreen mode Exit fullscreen mode

These are illustrative values. Set limits according to actual application requirements and observed usage.

Problem 5: Service Not Working

Your Pods are running, but your application isn't accessible.

Check your Services:

kubectl get svc
Enter fullscreen mode Exit fullscreen mode

Inspect the Service:

kubectl describe svc my-service
Enter fullscreen mode Exit fullscreen mode

Check its EndpointSlices:

kubectl get endpointslices
Enter fullscreen mode Exit fullscreen mode

Common problems include:

  • Incorrect Service selector.
  • Incorrect target port.
  • Pods not Ready.
  • Network policies blocking traffic.
  • Application listening on the wrong interface.

A running Pod does not automatically mean your Service can route traffic to it.


❤️ 4. Kubernetes Probes: The Difference Between Running and Ready

This is one of the most important topics in Kubernetes.

Imagine your application starts successfully, but needs 30 seconds to connect to its database and load configuration.

Should Kubernetes send traffic to it immediately?

Probably not.

That's why Kubernetes provides three types of probes.

Probe Purpose
Startup Probe Checks whether the application has started
Readiness Probe Determines whether the Pod should receive traffic
Liveness Probe Determines whether the container should be restarted

Let's configure them.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      containers:
        - name: payment-api
          image: nginx:1.27

          ports:
            - containerPort: 80

          startupProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
            failureThreshold: 30

          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 10

          livenessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 20
Enter fullscreen mode Exit fullscreen mode

This example uses Nginx and its root HTTP endpoint to demonstrate probe configuration.

For a real application, implement meaningful health endpoints and choose thresholds based on startup time and recovery behavior.

Here's the difference:

Application Starts
       |
       v
  Startup Probe
       |
       v
  Startup Complete?
       |
       v
  Readiness Probe
       |
       v
 Ready for Traffic?
       |
       v
  Receive Traffic
       |
       v
  Liveness Probe
       |
       v
  Still Healthy?
Enter fullscreen mode Exit fullscreen mode

A badly configured liveness probe can actually make an application less reliable by repeatedly restarting a process that could otherwise recover.

Choose probe behavior carefully.

🚀 5. The Real Power of Kubernetes: Self-Healing

Let's create a Deployment with three replicas.

Save this as deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app

spec:
  replicas: 3

  selector:
    matchLabels:
      app: web-app

  template:
    metadata:
      labels:
        app: web-app

    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
Enter fullscreen mode Exit fullscreen mode

Deploy it:

kubectl apply -f deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Check the Pods:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

You should see three Pods.

Now delete one:

kubectl delete pod <one-pod-name>
Enter fullscreen mode Exit fullscreen mode

Check again:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Kubernetes will create a replacement Pod to maintain the Deployment's desired replica count.

That's self-healing in action.

But remember:

Self-healing infrastructure is not the same as automatically fixing application bugs.

📈 6. Scaling: What Happens When Traffic Increases?

Imagine your application normally receives 100 requests per second.

Suddenly, traffic increases to 10,000 requests per second.

Your three replicas may no longer be sufficient.

Kubernetes provides Horizontal Pod Autoscaling (HPA).

Here's a basic example:

kubectl autoscale deployment web-app \
  --cpu-percent=60 \
  --min=3 \
  --max=10
Enter fullscreen mode Exit fullscreen mode

Check the autoscaler:

kubectl get hpa
Enter fullscreen mode Exit fullscreen mode

This configures a target average CPU utilization of 60% relative to requested CPU, with a minimum of three and maximum of ten replicas.

For CPU-based HPA to work, you generally need:

  • Metrics Server or another compatible metrics API.
  • CPU resource requests on your containers.
  • A workload that can actually scale horizontally.

For production workloads, CPU alone may not be enough. You might also scale using memory, request rate, queue length, or custom metrics.

And remember: adding Pods won't automatically fix a slow database or an application with shared bottlenecks.

🛠️ 7. The Kubernetes Troubleshooting Commands You Should Memorize

Here is a practical command cheat sheet.

Cluster information

kubectl cluster-info
kubectl get nodes -o wide
kubectl get namespaces
Enter fullscreen mode Exit fullscreen mode

Pod troubleshooting

kubectl get pods -A
kubectl describe pod <pod-name>
kubectl logs <pod-name>
kubectl logs <pod-name> --previous
Enter fullscreen mode Exit fullscreen mode

Deployment troubleshooting

kubectl get deployments
kubectl describe deployment <deployment-name>
kubectl rollout status deployment/<deployment-name>
kubectl rollout history deployment/<deployment-name>
Enter fullscreen mode Exit fullscreen mode

Service troubleshooting

kubectl get svc
kubectl describe svc <service-name>
kubectl get endpointslices
Enter fullscreen mode Exit fullscreen mode

Resource monitoring

kubectl top nodes
kubectl top pods
Enter fullscreen mode Exit fullscreen mode

Events

kubectl get events --sort-by=.metadata.creationTimestamp
Enter fullscreen mode Exit fullscreen mode

Save these commands. You'll use them frequently when working with Kubernetes.


🧪 8. Build a Real Kubernetes Project for Your GitHub

Reading Kubernetes tutorials is useful.

But building and troubleshooting your own cluster is where the real learning happens.

Here's a project idea:

Project: Kubernetes Application Reliability Lab

Build a small application and deliberately simulate common infrastructure failures.

Technology stack:

  • Docker
  • Kubernetes / Minikube
  • Nginx
  • Prometheus
  • Grafana
  • YAML manifests
  • GitHub Actions

Project features:

  1. Deploy an application with three replicas.
  2. Configure startup, readiness, and liveness probes.
  3. Configure CPU and memory requests and limits.
  4. Expose the application using a Service.
  5. Configure Horizontal Pod Autoscaling.
  6. Simulate a failing container.
  7. Simulate an unavailable dependency.
  8. Investigate failures using logs and events.
  9. Document root causes and solutions.

Architecture:

              Users
                |
                v
          Kubernetes Service
                |
        +-------+-------+
        |       |       |
        v       v       v
      Pod 1   Pod 2   Pod 3
        |       |       |
        +-------+-------+
                |
                v
          Application
                |
                v
         Prometheus
                |
                v
            Grafana
Enter fullscreen mode Exit fullscreen mode

What should you upload to GitHub?

kubernetes-reliability-lab/
│
├── manifests/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── hpa.yaml
│
├── monitoring/
│   └── prometheus.yaml
│
├── docs/
│   ├── crashloopbackoff.md
│   ├── imagepullbackoff.md
│   └── oomkilled.md
│
└── README.md
Enter fullscreen mode Exit fullscreen mode

Include screenshots, commands, failure scenarios, and troubleshooting explanations.

This project can help demonstrate practical Kubernetes knowledge during interviews.


🎯 Final Thoughts

Kubernetes is powerful, but it is not magic.

It can maintain replica counts, schedule workloads, restart containers, and reconcile desired state.

But it cannot automatically understand every application failure or make every architectural decision for you.

A strong Kubernetes engineer understands:

  • How Pods and Deployments work.
  • How Services route traffic.
  • How probes affect application availability.
  • How resource requests and limits work.
  • How autoscaling operates.
  • How to investigate failures using logs, events, and metrics.

Don't just memorize kubectl commands.

Understand what Kubernetes is doing behind those commands.

The difference between someone who knows Kubernetes and someone who can operate Kubernetes is the ability to troubleshoot when everything goes wrong.

👇 Let's discuss!

What's the most confusing Kubernetes problem you've faced?

  1. CrashLoopBackOff
  2. ImagePullBackOff
  3. Pending Pods
  4. Service networking
  5. PersistentVolume problems
  6. Kubernetes networking and DNS

Share your experience in the comments. Someone learning Kubernetes might find your solution incredibly helpful!


📚 Want to Prepare for CKA?

If you're serious about mastering Kubernetes, explore my learning resources:

☸️ CKA Complete Study Guide — Certified Kubernetes Administrator

🐳 Docker Mastery

🏗️ Terraform Associate (003) Exam Crash Course

🚀 DevOps Complete Pack

Learn Kubernetes. Practice in a real cluster. Break things safely. Become confident at troubleshooting.

Top comments (0)