DEV Community

janak0ff
janak0ff

Posted on

Day 52: Revert Deployment to Previous Version in Kubernetes

#100daysofdevops #kodekloud #rollback #kubernetes

Earlier today, the Nautilus DevOps team deployed a new release for an application. However, a customer has reported a bug related to this recent release. Consequently, the team aims to revert to the previous version.

There exists a deployment named nginx-deployment; initiate a rollback to the previous revision.

Introduction

Welcome to Day 52 of my 100 Days of DevOps journey! Today, we're learning about a crucial Kubernetes skill: rolling back deployments.

Why Rollbacks Matter

Imagine you're at a restaurant. You order a new dish, but it's not what you expected. You want to go back to your usual order. Similarly, in Kubernetes, when a new deployment has issues:

  1. ✅ Quick recovery – Revert to a working version instantly
  2. ✅ Minimal downtime – Users barely notice the change
  3. ✅ No manual rebuilds – Kubernetes handles everything
  4. ✅ Audit trail – Every change is tracked

📋 What We'll Do Today

We have an nginx-deployment that was updated from nginx:1.16 to nginx:alpine-perl. A bug was reported, so we need to roll back to the previous version.

Our Task:

  • Deployment Name: nginx-deployment
  • Current Image: nginx:alpine-perl (Revision 2)
  • Previous Image: nginx:1.16 (Revision 1)
  • Goal: Rollback to Revision 1

📖 Understanding Rollbacks

What Happens During a Rollback? 

┌─────────────────────────────────────────────────────────────────────────────┐
│                        Rollback Process                                    │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────┐     │
│  │  BEFORE: Pods running nginx:alpine-perl (Revision 2)             │     │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐                         │     │
│  │  │  Pod 1   │ │  Pod 2   │ │  Pod 3   │                         │     │
│  │  │ alpine   │ │ alpine   │ │ alpine   │                         │     │
│  │  └──────────┘ └──────────┘ └──────────┘                         │     │
│  └────────────────────────────────────────────────────────────────────┘     │
│                                    │                                         │
│                                    ▼                                         │
│  ┌────────────────────────────────────────────────────────────────────┐     │
│  │  DURING: Rolling back to Revision 1                              │     │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐                         │     │
│  │  │  Pod 1   │ │  Pod 2   │ │  Pod 3   │                         │     │
│  │  │ alpine   │ │ alpine   │ │ alpine   │                         │     │
│  │  └──────────┘ └──────────┘ └──────────┘                         │     │
│  │  ┌──────────┐                                                    │     │
│  │  │  Pod 4   │  ← New pod with nginx:1.16                       │     │
│  │  │  1.16    │                                                    │     │
│  │  └──────────┘                                                    │     │
│  └────────────────────────────────────────────────────────────────────┘     │
│                                    │                                         │
│                                    ▼                                         │
│  ┌────────────────────────────────────────────────────────────────────┐     │
│  │  AFTER: All pods running nginx:1.16 (Revision 1)                │     │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐                         │     │
│  │  │  Pod 1   │ │  Pod 2   │ │  Pod 3   │                         │     │
│  │  │  1.16    │ │  1.16    │ │  1.16    │                         │     │
│  │  └──────────┘ └──────────┘ └──────────┘                         │     │
│  └────────────────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Deployment Revision History 

┌─────────────────────────────────────────────────────────────────────────────┐
│                    Revision History Example                                 │
│                                                                              │
│  Revision 1:  nginx:1.16                                                    │
│  └─── The original, stable version                                          │
│                                                                              │
│  Revision 2:  nginx:alpine-perl                                             │
│  └─── The new version with reported bug                                    │
│                                                                              │
│  Revision 3:  nginx:1.16                                                    │
│  └─── The rollback (re-applying Revision 1)                                │
│                                                                              │
│  KEY INSIGHT: The rollback creates a NEW revision (3)                      │
│  that replicates the content of revision 1!                                │
└─────────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

🔧 Step-by-Step Guide

Step 1: Check Current Deployment

First, let's see what we're working with:

# Check cluster status
kubectl cluster-info

# Get current deployments
kubectl get deployments

# Check current pods
kubectl get pods

# Get detailed deployment info
kubectl describe deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Current Status:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           5m7s

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-55658c8544-8zngx   1/1     Running   0          4m52s
nginx-deployment-55658c8544-j29mz   1/1     Running   0          4m57s
nginx-deployment-55658c8544-nc6zz   1/1     Running   0          4m53s

Containers:
   nginx-container:
    Image:         nginx:alpine-perl
Enter fullscreen mode Exit fullscreen mode

Key Observations:

  • ✅ 3 replicas running
  • ✅ Current image: nginx:alpine-perl
  • ✅ Container name: nginx-container
  • ✅ Revision: 2

Step 2: View Rollout History

Before rolling back, let's see the history:

# View rollout history
kubectl rollout history deployment nginx-deployment

# View specific revision details
kubectl rollout history deployment nginx-deployment --revision=1
kubectl rollout history deployment nginx-deployment --revision=2
Enter fullscreen mode Exit fullscreen mode

Rollout History Output:

REVISION  CHANGE-CAUSE
1         <none>
2         kubectl set image deployment nginx-deployment nginx-container=nginx:alpine-perl --record=true

Revision #1 Details:
  Image:      nginx:1.16

Revision #2 Details:
  Image:      nginx:alpine-perl
  Annotations: kubernetes.io/change-cause: kubectl set image deployment nginx-deployment nginx-container=nginx:alpine-perl --record=true
Enter fullscreen mode Exit fullscreen mode

Key Insights:

  • Revision 1: nginx:1.16 (Stable version)
  • Revision 2: nginx:alpine-perl (Buggy version)
  • The --record flag saved the change-cause

Step 3: Perform the Rollback

Now let's roll back to the previous version:

# Rollback to the previous revision
kubectl rollout undo deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

deployment.apps/nginx-deployment rolled back
Enter fullscreen mode Exit fullscreen mode

What this does:

  • Kubernetes reverts to Revision 1
  • Creates a new Revision (3) with the same content as Revision 1
  • No downtime during the process

Step 4: Monitor the Rollback 

# Check rollout status
kubectl rollout status deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

deployment "nginx-deployment" successfully rolled out
Enter fullscreen mode Exit fullscreen mode 
# Watch pods being updated
kubectl get pods -w
Enter fullscreen mode Exit fullscreen mode

During Rollback Output:

NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-fc677cbc9-68w6t   1/1     Running   0          15s
nginx-deployment-fc677cbc9-fxtj4   1/1     Running   0          18s
nginx-deployment-fc677cbc9-nklwq   1/1     Running   0          16s
Enter fullscreen mode Exit fullscreen mode

Notice: New pods with nginx:1.16 are created and the old ones are terminated.

Step 5: Verify the Rollback 

# Check deployment status
kubectl get deployments
Enter fullscreen mode Exit fullscreen mode

Output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           6m
Enter fullscreen mode Exit fullscreen mode 
# Check pods
kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-fc677cbc9-68w6t   1/1     Running   0          30s
nginx-deployment-fc677cbc9-fxtj4   1/1     Running   0          35s
nginx-deployment-fc677cbc9-nklwq   1/1     Running   0          32s
Enter fullscreen mode Exit fullscreen mode 
# Verify the reverted image
kubectl describe deployment nginx-deployment | grep -i image
Enter fullscreen mode Exit fullscreen mode

Output:

Image:         nginx:1.16
Enter fullscreen mode Exit fullscreen mode 
# Check updated rollout history
kubectl rollout history deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

REVISION  CHANGE-CAUSE
2         kubectl set image deployment nginx-deployment nginx-container=nginx:alpine-perl --record=true
3         <none>
Enter fullscreen mode Exit fullscreen mode

Note: Revision 3 is the rollback (replicating Revision 1).

📝 Complete Commands Summary 

# 1. Check current status
kubectl get deployments
kubectl get pods
kubectl describe deployment nginx-deployment | grep -i image

# 2. View rollout history
kubectl rollout history deployment nginx-deployment
kubectl rollout history deployment nginx-deployment --revision=1
kubectl rollout history deployment nginx-deployment --revision=2

# 3. Perform rollback
kubectl rollout undo deployment nginx-deployment

# 4. Monitor rollback
kubectl rollout status deployment nginx-deployment
kubectl get pods -w

# 5. Verify rollback
kubectl get deployments
kubectl get pods
kubectl describe deployment nginx-deployment | grep -i image
kubectl rollout history deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

📊 Before and After Comparison

Aspect Before Rollback After Rollback
Image Version nginx:alpine-perl nginx:1.16
Revision 2 3 (new revision)
Replicas 3 3
Pods 3 running 3 running
Deployment Status Available ✅ Available ✅
Change-Cause kubectl set image... <none>
Downtime - None ✅

🛠️ Troubleshooting

Rollback Failed or Stuck: 

# Check rollout status
kubectl rollout status deployment nginx-deployment

# Check pod details
kubectl describe pod <pod-name>

# Check pod logs
kubectl logs <pod-name>

# Check events
kubectl get events --sort-by='.lastTimestamp'
Enter fullscreen mode Exit fullscreen mode

Checking Rollback Details: 

# Check the rollout history with more details
kubectl rollout history deployment nginx-deployment --revision=3

# Check current image
kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}'

# Check replica sets
kubectl get replicasets

# Describe the replica set for the reverted version
kubectl describe replicaset <replica-set-name>
Enter fullscreen mode Exit fullscreen mode

🎯 Key Rollback Commands

Command Purpose
kubectl rollout undo deployment/nginx-deployment Rollback to previous revision
kubectl rollout undo deployment/nginx-deployment --to-revision=1 Rollback to specific revision
kubectl rollout status deployment/nginx-deployment Check rollback status
kubectl rollout history deployment/nginx-deployment View rollout history
kubectl rollout history deployment/nginx-deployment --revision=2 View specific revision details
kubectl rollout pause deployment/nginx-deployment Pause rollback
kubectl rollout resume deployment/nginx-deployment Resume rollback

📊 Revision History Example

Before Rollback: 

REVISION  CHANGE-CAUSE
1         <none>                          ← nginx:1.16 (Stable)
2         kubectl set image ... alpine    ← nginx:alpine-perl (Buggy)
Enter fullscreen mode Exit fullscreen mode

After Rollback: 

REVISION  CHANGE-CAUSE
2         kubectl set image ... alpine    ← nginx:alpine-perl (Buggy)
3         <none>                          ← nginx:1.16 (Rollback)
Enter fullscreen mode Exit fullscreen mode

🔧 Best Practices for Rollbacks

1. Always Check History First 

kubectl rollout history deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

2. Monitor the Rollback 

kubectl rollout status deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

3. Verify After Rollback 

kubectl describe deployment nginx-deployment | grep -i image
Enter fullscreen mode Exit fullscreen mode

4. Use Change-Cause for Tracking 

kubectl annotate deployment nginx-deployment kubernetes.io/change-cause="Rollback: Bug in nginx:alpine-perl"
Enter fullscreen mode Exit fullscreen mode

5. Test Rollbacks in Staging First

  • Always test rollbacks in non-production environments
  • Practice rollback scenarios regularly

📚 Quick Reference

Rollback Commands

Command Description
kubectl rollout undo deployment/<name> Rollback to previous version
kubectl rollout undo deployment/<name> --to-revision=1 Rollback to specific revision
kubectl rollout status deployment/<name> Check rollback status
kubectl rollout history deployment/<name> View all revisions
kubectl rollout history deployment/<name> --revision=2 View specific revision

Common Errors and Solutions

Error Solution
"rollout undo: no rollout history found" Deployment has no history
Rollback stuck Check pod logs and events
Pods not ready Check health checks
Image pull failed Verify image exists

Top comments (0)