janak0ff

Posted on Aug 15

Resolve Pod Deployment Issue

#kubernetes #devops #kodekloud

A junior DevOps team member encountered difficulties deploying a stack on the Kubernetes cluster. The pod fails to start, presenting errors. Let's troubleshoot and rectify the issue promptly.

There is a pod named webserver, and the container within it is named nginx-container, its utilizing the nginx:latest image.
Additionally, there's a sidecar container named sidecar-container using the ubuntu:latest image.

Identify and address the issue to ensure the pod is in the running state and the application is accessible.

Introduction

Have you ever deployed a Kubernetes pod only to see it stuck in a failing state? Don't worry—this happens to everyone! In fact, troubleshooting is one of the most valuable skills you can develop as a DevOps engineer.

In this beginner-friendly guide, we'll walk through a real-world scenario where a pod fails to start and learn how to identify and fix the issue step by step.

What You'll Learn

How to check pod status and diagnose issues
How to read and understand error messages
How to fix common Kubernetes deployment issues
How to verify your fixes work correctly
Essential troubleshooting commands every DevOps engineer should know

Understanding the Problem
Prerequisites
Step 1: Check Pod Status
Step 2: Get Detailed Information
Step 3: Analyze Error Messages
Step 4: Fix the Issue
Step 5: Verify Your Fix
Common Kubernetes Issues and Solutions
Essential Troubleshooting Commands
Best Practices
Conclusion

Understanding the Problem

The Scenario

A junior DevOps team member tried to deploy a pod but it's failing to start. The pod is named webserver and should have two containers:

Main Container: nginx-container using nginx:latest image
Sidecar Container: sidecar-container using ubuntu:latest image

The pod is currently showing errors and not starting properly.

What Are We Looking For?

NAME        READY   STATUS             RESTARTS   AGE
webserver   1/2     ImagePullBackOff   0          2m18s

This tells us:

1/2 READY: Only one of two containers is ready
ImagePullBackOff: Kubernetes is having trouble pulling an image
RESTARTS: 0: The pod hasn't restarted yet

Prerequisites

Before we start, make sure you have:

✅ A Kubernetes cluster (Minikube, kind, or cloud-based)
✅ kubectl configured and working
✅ Basic understanding of Kubernetes pods

Step 1: Check Pod Status

The first thing to do is check the status of your pods.

Command

kubectl get pods

Expected Output

NAME        READY   STATUS             RESTARTS   AGE
webserver   1/2     ImagePullBackOff   0          2m18s

Understanding the Output

Column	Meaning	Our Value
NAME	Pod name	`webserver`
READY	Ready containers / Total containers	`1/2` - One container is not ready
STATUS	Current pod status	`ImagePullBackOff` - Image can't be pulled
RESTARTS	Number of restarts	`0` - No restarts yet
AGE	Pod age	`2m18s` - Created 2 minutes ago

What Does This Tell Us?

The pod was created (AGE: 2m18s)
One container is running (READY: 1/2)
The other container is failing because it can't pull the image (ImagePullBackOff)

Step 2: Get Detailed Information

Now let's get more details about what's actually happening.

Command

kubectl describe pod webserver

Key Sections to Look For

Containers Section:

Containers:
  nginx-container:
    Container ID:   
    Image:          nginx:latests
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Mounts:
      /var/log/nginx from shared-logs (rw)

Sidecar Container (Running):

  sidecar-container:
    Container ID:  containerd://c6f35b248c85...
    Image:         ubuntu:latest
    Image ID:      docker.io/library/ubuntu@sha256:...
    State:          Running
    Ready:          True

Events Section:

Events:
  Type     Reason     Age                Message
  ----     ------     ----               -------
  Normal   Scheduled  2m18s              Successfully assigned default/webserver to jump-host
  Normal   Pulling    2m16s              Pulling image "ubuntu:latest"
  Normal   Pulled     2m14s              Successfully pulled image "ubuntu:latest"
  Warning  Failed     52s                Failed to pull image "nginx:latests": not found
  Normal   BackOff    13s                Back-off pulling image "nginx:latests"

What Did We Discover?

nix-container is failing with ImagePullBackOff
The image it's trying to pull is nginx:latests (notice the spelling)
The sidecar container is running fine with ubuntu:latest
The error message says: "not found"

Step 3: Analyze Error Messages

Let's look at the error messages more closely.

Error Messages from Events

Warning  Failed  52s  Failed to pull image "nginx:latests": 
rpc error: code = NotFound desc = failed to pull and unpack image 
"docker.io/library/nginx:latests": failed to resolve reference 
"docker.io/library/nginx:latests": docker.io/library/nginx:latests: not found

Breaking Down the Error

Part of Error	Meaning
`Failed to pull image "nginx:latests"`	The image couldn't be downloaded
`code = NotFound`	The image doesn't exist
`docker.io/library/nginx:latests`	The full image path being looked for
`not found`	The image tag doesn't exist

The Problem Identified! 🎯

Image: nginx:latests

The issue: The image tag is misspelled! It should be nginx:latest but is written as nginx:latests (with an extra 's' at the end).

Why this happens:

nginx:latest exists on Docker Hub ✓
nginx:latests does NOT exist on Docker Hub ✗
Kubernetes can't find the image, so it fails

Image Tags: A Quick Refresher

Image Tag	Meaning	Exists?
`nginx:latest`	The latest stable version	✅ Yes
`nginx:1.21`	Specific version	✅ Yes
`nginx:latests`	Misspelled tag	❌ No
`nginx:alpine`	Alpine-based version	✅ Yes

Step 4: Fix the Issue

Now that we know the problem, let's fix it!

Option A: Delete and Recreate (Recommended)

1. Delete the faulty pod:

kubectl delete pod webserver

2. Create a corrected YAML file:

Create webserver-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: webserver
  labels:
    app: web-app
spec:
  containers:
  # Main Nginx container
  - name: nginx-container
    image: nginx:latest              # ✅ Fixed from "latests"
    ports:
    - containerPort: 80
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx

  # Sidecar container
  - name: sidecar-container
    image: ubuntu:latest
    command: ["sh", "-c"]
    args: ["while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log 2>/dev/null; sleep 30; done"]
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx

  volumes:
  - name: shared-logs
    emptyDir: {}

  restartPolicy: Never

3. Apply the corrected pod:

kubectl apply -f webserver-pod.yaml

Option B: Edit the Existing Pod (Not Always Possible)

Sometimes you can edit a pod directly:

kubectl edit pod webserver

Find the image field and change:

image: nginx:latests

to:

image: nginx:latest

Save and exit. However, many pod properties can't be edited after creation, so deletion and recreation is often the only option.

Option C: One-Liner Quick Fix

If you want to create everything at once:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: webserver
  labels:
    app: web-app
spec:
  containers:
  - name: nginx-container
    image: nginx:latest
    ports:
    - containerPort: 80
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx
  - name: sidecar-container
    image: ubuntu:latest
    command: ["sh", "-c"]
    args: ["while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log 2>/dev/null; sleep 30; done"]
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx
  volumes:
  - name: shared-logs
    emptyDir: {}
  restartPolicy: Never
EOF

Step 5: Verify Your Fix

After applying the fix, let's verify everything is working.

1. Check Pod Status

kubectl get pods

Expected Output:

NAME        READY   STATUS    RESTARTS   AGE
webserver   2/2     Running   0          10s

What this tells us:

2/2 READY: Both containers are ready!
Running: The pod is running successfully!
RESTARTS: 0: No restarts occurred

2. Check Container Details

kubectl describe pod webserver | grep -A 10 "Containers:"

Expected Output:

Containers:
  nginx-container:
    Container ID:   containerd://...
    Image:          nginx:latest
    State:          Running
    Ready:          True
  sidecar-container:
    Container ID:   containerd://...
    Image:          ubuntu:latest
    State:          Running
    Ready:          True

3. Check Container Names

kubectl get pod webserver -o jsonpath='{.spec.containers[*].name}'

Expected Output:

nginx-container sidecar-container

4. Check Container Images

kubectl get pod webserver -o jsonpath='{.spec.containers[*].image}'

Expected Output:

nginx:latest ubuntu:latest

5. Verify the Website is Accessible

First, port forward to access the web server:

kubectl port-forward pod/webserver 8080:80

Expected Output:

Forwarding from 127.0.0.1:8080 -> 80

Then, in another terminal, test access:

curl http://localhost:8080

Expected Output (HTML from Nginx):

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...
</html>

Common Kubernetes Issues and Solutions

Issue 1: ImagePullBackOff

Error:

ImagePullBackOff

Solution:

Check if the image tag is spelled correctly
Verify the image exists on Docker Hub or your registry
Check network connectivity to the registry
Ensure you have permission to access private registries

Example Fix:

# ❌ Wrong
image: nginx:latests

# ✅ Correct
image: nginx:latest

Issue 2: CrashLoopBackOff

Error:

CrashLoopBackOff

Solution:

Check pod logs: kubectl logs <pod-name>
Check if container has a command
Ensure the command doesn't exit immediately
Check for application errors

Example Fix:

# ❌ Wrong - No command, exits immediately
- name: sidecar-container
  image: ubuntu:latest

# ✅ Correct - Keeps running
- name: sidecar-container
  image: ubuntu:latest
  command: ["sh", "-c"]
  args: ["while true; do sleep 30; done"]

Issue 3: Container Not Starting

Error:

Container is waiting to start

Solution:

Check if container name is correct
Check for resource constraints
Verify the image is pulling correctly

Issue 4: Pod Pending

Error:

Pending

Solution:

Check if there are enough resources on the node
Check for node affinity or taints
Verify the namespace exists

Issue 5: Container Name Mismatch

Error:

'nginx-container' doesn't exist

Solution:

Ensure container names match exactly
Check for spelling differences (case sensitive)
Verify the YAML has the correct name

# ❌ Wrong - Different name
- name: nginx

# ✅ Correct
- name: nginx-container

Essential Troubleshooting Commands

Status Commands

Command	Purpose
`kubectl get pods`	List all pods
`kubectl get pods -n <namespace>`	List pods in a specific namespace
`kubectl get deployments`	List all deployments
`kubectl get all`	List all resources
`kubectl get events`	View cluster events
`kubectl get events --sort-by='.lastTimestamp'`	Events sorted by time

Detailed Information Commands

Command	Purpose
`kubectl describe pod <pod-name>`	Detailed pod information
`kubectl describe deployment <name>`	Detailed deployment information
`kubectl logs <pod-name>`	View pod logs
`kubectl logs <pod-name> -c <container>`	View specific container logs
`kubectl logs <pod-name> --previous`	View previous container logs

Debugging Commands

Command	Purpose
`kubectl exec -it <pod-name> -- /bin/bash`	Shell into a pod
`kubectl exec -it <pod-name> -c <container> -- /bin/bash`	Shell into a specific container
`kubectl run test --image=busybox --rm -it -- /bin/sh`	Run a test pod
`kubectl port-forward pod/<pod-name> 8080:80`	Forward a port to a pod

YAML/JSON Commands

Command	Purpose
`kubectl get pod <pod-name> -o yaml`	Pod YAML output
`kubectl get pod <pod-name> -o json`	Pod JSON output
`kubectl get pod <pod-name> -o yaml > pod.yaml`	Save YAML to file

JSONPath Commands

Command	Purpose
`kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'`	Get container names
`kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].image}'`	Get container images
`kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].ready}'`	Check container readiness

Troubleshooting Flowchart

┌──────────────────────────────────────────────────┐
│                 kubectl get pods                │
└────────────────────┬─────────────────────────────┘
                     │
                     ▼
        ┌────────────────────────┐
        │   Pod Status?          │
        └───────────┬────────────┘
                    │
    ┌───────────────┼───────────────┐
    │               │               │
    ▼               ▼               ▼
┌─────────┐  ┌───────────┐  ┌──────────────┐
│Running  │  │ Pending   │  │ ImagePull    │
│         │  │           │  │ BackOff      │
└─────────┘  └───────────┘  └──────────────┘
    │              │               │
    │              │               │
    ▼              ▼               ▼
┌─────────┐  ┌───────────┐  ┌──────────────┐
│Success! │  │Check      │  │Check image   │
│         │  │Resources  │  │tag           │
│         │  │           │  │              │
└─────────┘  └───────────┘  └──────────────┘

Best Practices

✅ DO's

1. Always Check the Logs First

kubectl logs <pod-name>

2. Use describe to Get Detailed Information

kubectl describe pod <pod-name>

3. Test Image Locally Before Deploying

docker pull nginx:latest
docker run -p 8080:80 nginx:latest
curl http://localhost:8080

4. Use Specific Image Tags (Not :latest in Production)

# Development
image: nginx:latest

# Production
image: nginx:1.25.3

5. Check Container Names Carefully

# Container names are case-sensitive!
- name: nginx-container  # ✅
- name: Nginx-Container  # ❌ Different case

6. Use --dry-run to Validate YAML

kubectl apply -f pod.yaml --dry-run=client

7. Save YAML Files to Version Control

# Always save your configurations
git add pod.yaml
git commit -m "Add webserver pod"

❌ DON'Ts

1. Don't Delete Pods Without Checking First

# BAD - Delete without investigation
kubectl delete pod webserver

# GOOD - Describe first
kubectl describe pod webserver
# Then decide what to do

2. Don't Edit Running Pods

# Pods are generally immutable
# Delete and recreate instead

3. Don't Ignore Error Messages

# The error messages tell you what's wrong!
# Read them carefully

4. Don't Use Misspelled Image Tags

# BAD - Misspelled
image: nginx:latests

# GOOD - Correct
image: nginx:latest

5. Don't Forget to Verify Fixes

# Always verify after fixing
kubectl get pods
kubectl describe pod webserver

Real-World Example: Complete Troubleshooting Session

Here's a complete session showing the troubleshooting process:

# Step 1: Check pods
$ kubectl get pods
NAME        READY   STATUS             RESTARTS   AGE
webserver   1/2     ImagePullBackOff   0          2m18s

# Step 2: Get details
$ kubectl describe pod webserver
...
Containers:
  nginx-container:
    Image:          nginx:latests
    State:          Waiting
      Reason:       ImagePullBackOff
...
Events:
  Warning  Failed  Failed to pull image "nginx:latests": not found

# Step 3: Identify the issue
# Problem: "nginx:latests" is misspelled (should be "nginx:latest")

# Step 4: Fix the issue
$ kubectl delete pod webserver
pod "webserver" deleted

$ cat > webserver-pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: webserver
  labels:
    app: web-app
spec:
  containers:
  - name: nginx-container
    image: nginx:latest
    ports:
    - containerPort: 80
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx
  - name: sidecar-container
    image: ubuntu:latest
    command: ["sh", "-c"]
    args: ["while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log 2>/dev/null; sleep 30; done"]
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx
  volumes:
  - name: shared-logs
    emptyDir: {}
  restartPolicy: Never
EOF

$ kubectl apply -f webserver-pod.yaml
pod/webserver created

# Step 5: Verify
$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
webserver   2/2     Running   0          10s

$ kubectl describe pod webserver | grep -A 5 "Containers:"
Containers:
  nginx-container:
    Image:          nginx:latest
    State:          Running
    Ready:          True
  sidecar-container:
    Image:          ubuntu:latest
    State:          Running
    Ready:          True

# Step 6: Test the application
$ kubectl port-forward pod/webserver 8080:80 &
Forwarding from 127.0.0.1:8080 -> 80

$ curl http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</html>

# Success! 🎉

Summary: Key Takeaways

Troubleshooting Steps

Check pod status: kubectl get pods
Get details: kubectl describe pod <name>
Check logs: kubectl logs <name>
Check events: kubectl get events
Identify the issue: Read error messages carefully
Fix the issue: Edit YAML, delete/recreate
Verify: Check status, test accessibility

Common Issues to Look For

Issue	What to Check
ImagePullBackOff	Image tag spelling, image existence, registry access
CrashLoopBackOff	Command existence, application errors, logs
Pending	Resource availability, node constraints, namespace
Container mismatch	Container names, case sensitivity, YAML formatting

Essential Commands

Situation	Command
Check pod status	`kubectl get pods`
Get detailed info	`kubectl describe pod <name>`
View logs	`kubectl logs <name>`
Check events	`kubectl get events`
Delete pod	`kubectl delete pod <name>`
Apply YAML	`kubectl apply -f <file>`
Port forward	`kubectl port-forward pod/<name> <local>:<remote>`

Conclusion

Congratulations! You've successfully learned how to troubleshoot and fix Kubernetes pod deployment issues. This is a critical skill that every DevOps engineer needs.

What You Learned

✅ How to check pod status and identify issues
✅ How to read and understand Kubernetes error messages
✅ How to fix common issues like ImagePullBackOff
✅ How to verify your fixes work correctly
✅ Essential troubleshooting commands and best practices

Key Takeaway

The most important skill in troubleshooting is reading error messages carefully. In this case, the error message clearly told us that nginx:latests was not found. Always start by understanding what the error is telling you before making changes.

Next Steps

Now that you've mastered basic troubleshooting, consider exploring:

Advanced Pod Configuration - Resource limits, health checks
Deployments - Rolling updates, rollbacks
Services - Exposing applications
ConfigMaps and Secrets - Configuration management
StatefulSets - Stateful applications

Introduction

What You'll Learn

Table of Contents

Understanding the Problem

The Scenario

What Are We Looking For?

Prerequisites

Step 1: Check Pod Status

Command

Expected Output

Understanding the Output

What Does This Tell Us?

Step 2: Get Detailed Information

Command

Key Sections to Look For

What Did We Discover?

Step 3: Analyze Error Messages

Error Messages from Events

Breaking Down the Error

The Problem Identified! 🎯

Image Tags: A Quick Refresher

Step 4: Fix the Issue

Option A: Delete and Recreate (Recommended)

Option B: Edit the Existing Pod (Not Always Possible)

Option C: One-Liner Quick Fix

Step 5: Verify Your Fix

1. Check Pod Status

2. Check Container Details

3. Check Container Names

4. Check Container Images

5. Verify the Website is Accessible

Common Kubernetes Issues and Solutions

Issue 1: ImagePullBackOff

Issue 2: CrashLoopBackOff

Issue 3: Container Not Starting

Issue 4: Pod Pending

Issue 5: Container Name Mismatch

Essential Troubleshooting Commands

Status Commands

Detailed Information Commands

Debugging Commands

YAML/JSON Commands

JSONPath Commands

Troubleshooting Flowchart

Best Practices

✅ DO's

❌ DON'Ts

Real-World Example: Complete Troubleshooting Session

Summary: Key Takeaways

Troubleshooting Steps

Common Issues to Look For

Essential Commands

Conclusion

What You Learned

Key Takeaway

Next Steps

Resources