DEV Community

janak0ff
janak0ff

Posted on

Revert Deployment to Previous Version in Kubernetes

#kodekloud #kubernetes

Earlier today, the Nautilus DevOps team deployed a new release for an application. However, a customer has reported a bug related to this recent release. Consequently, the team aims to revert to the previous version.
There exists a deployment named nginx-deployment; initiate a rollback to the previous revision.

Understanding Rollbacks

What is a Rollback?

A rollback is the process of reverting a deployment to a previous version. Think of it like the "Undo" button in a text editor, but for your entire application!

Why Rollbacks Matter

Scenario Without Rollback With Rollback
Buggy release Application broken, users unhappy Quick revert to stable version
Security issue Vulnerability exposed Immediate fix by reverting
Performance degradation Slow application Restore to fast version
Configuration error Application crashes Rollback to working config

How Rollbacks Work in Kubernetes

When you update a Deployment, Kubernetes:

  1. Creates a new ReplicaSet with the new image
  2. Gradually replaces old pods with new ones
  3. Keeps the old ReplicaSet as a backup

This means rolling back is as simple as telling Kubernetes to use the old ReplicaSet again!

┌─────────────────────────────────────────────────────┐
│                  Deployment                        │
│            Revision History: 1, 2, 3              │
└────────────────────┬────────────────────────────────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
        ▼            ▼            ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Revision 1   │ │ Revision 2   │ │ Revision 3   │
│ (Stable)     │ │ (Buggy)      │ │ (New)        │
│ ReplicaSet   │ │ ReplicaSet   │ │ ReplicaSet   │
│ 3 pods ✅    │ │ 0 pods ❌    │ │ 0 pods ⏳    │
└──────────────┘ └──────────────┘ └──────────────┘
       ▲                │                │
       │                │                │
       └────────────────┴────────────────┘
                Rollback goes here!
Enter fullscreen mode Exit fullscreen mode

The Scenario: When Things Go Wrong

Imagine this: You're the DevOps engineer at Nautilus. You've just deployed a new version of your application using nginx:alpine. Your users start reporting bugs. The team is panicking. What do you do?

You rollback to the previous stable version!

Our Setup

We have a deployment called nginx-deployment with two revisions:

REVISION  CHANGE-CAUSE
1         <none>                                    ← Previous stable version
2         kubectl set image ... nginx:alpine ...    ← Current buggy version
Enter fullscreen mode Exit fullscreen mode

Goal: Rollback from Revision 2 to Revision 1

Step-by-Step Rollback Guide

Step 1: Check Your Current Deployment

First, let's see what we're working with:

kubectl get deployments
Enter fullscreen mode Exit fullscreen mode

Output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           15m
Enter fullscreen mode Exit fullscreen mode

This shows we have a deployment called nginx-deployment with 3 running pods.

Step 2: View the Rollout History

Before rolling back, let's see the history of changes:

kubectl rollout history deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl set image deployment nginx-deployment nginx-container=nginx:alpine --record=true
Enter fullscreen mode Exit fullscreen mode

What this tells us:

  • Revision 1: The original deployment (stable)
  • Revision 2: The current deployment (buggy, using nginx:alpine)

Step 3: Examine the Previous Revision

Let's check what Revision 1 looked like:

kubectl rollout history deployment nginx-deployment --revision=1
Enter fullscreen mode Exit fullscreen mode

Output:

deployment.apps/nginx-deployment with revision #1
Pod Template:
  Labels:   app=nginx-deployment
  Containers:
   nginx-container:
    Image:  nginx:1.16
Enter fullscreen mode Exit fullscreen mode

Great! Revision 1 uses nginx:1.16 - this was our stable version.

Step 4: Check the Current Revision

For comparison, let's check Revision 2:

kubectl rollout history deployment nginx-deployment --revision=2
Enter fullscreen mode Exit fullscreen mode

Output:

deployment.apps/nginx-deployment with revision #2
Pod Template:
  Labels:   app=nginx-deployment
  Annotations:  kubernetes.io/change-cause: kubectl set image deployment nginx-deployment nginx-container=nginx:alpine --record=true
  Containers:
   nginx-container:
    Image:  nginx:alpine
Enter fullscreen mode Exit fullscreen mode

The problem: Revision 2 uses nginx:alpine which has a bug.

Step 5: Perform the Rollback

Now let's rollback to the previous revision:

kubectl rollout undo deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

deployment.apps/nginx-deployment rolled back
Enter fullscreen mode Exit fullscreen mode

What just happened? Kubernetes reverted to Revision 1!

Step 6: Monitor the Rollback

Watch the rollback in progress:

kubectl rollout status deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

deployment "nginx-deployment" successfully rolled out
Enter fullscreen mode Exit fullscreen mode

This confirms the rollback completed successfully.

Step 7: Verify the Rollback

Now let's verify everything is back to normal:

kubectl get deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           20m
Enter fullscreen mode Exit fullscreen mode

All 3 pods are ready!

Step 8: Check the Current Image

Let's confirm we're using the correct image:

kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}'
Enter fullscreen mode Exit fullscreen mode

Output:

nginx:1.16
Enter fullscreen mode Exit fullscreen mode

Success! We're back to the stable version.

Step 9: Check the New History

Look at the updated rollout history:

kubectl rollout history deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Output:

REVISION  CHANGE-CAUSE
2         kubectl set image deployment nginx-deployment nginx-container=nginx:alpine --record=true
3         <none>
Enter fullscreen mode Exit fullscreen mode

Important: A new revision (3) was created for the rollback. Revision 1 is still referenced internally.

Step 10: Verify Pods are Running

Check all pods are healthy:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-fc677cbc9-4tg9t   1/1     Running   0          46s
nginx-deployment-fc677cbc9-hxg9s   1/1     Running   0          50s
nginx-deployment-fc677cbc9-vg76c   1/1     Running   0          48s
Enter fullscreen mode Exit fullscreen mode

All pods are running with the correct ReplicaSet (fc677cbc9)!

Verification Commands Cheat Sheet

Here are all the commands you need to verify a successful rollback:

# Quick status overview
kubectl get deployment,rs,pods -l app=nginx-deployment

# Check image version
kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}'

# Detailed pod info
kubectl describe pod -l app=nginx-deployment | grep -E "Name:|Image:|Status:"

# Check events
kubectl get events --field-selector involvedObject.name=nginx-deployment

# Full deployment details
kubectl describe deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Common Rollback Scenarios

Scenario 1: Rollback to Previous Revision

This is the most common case - just undo the last change:

kubectl rollout undo deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Scenario 2: Rollback to a Specific Revision

If you want to go back to a specific version:

# First, check what revision you want
kubectl rollout history deployment nginx-deployment

# Then rollback to that revision
kubectl rollout undo deployment nginx-deployment --to-revision=1
Enter fullscreen mode Exit fullscreen mode

Scenario 3: Rollback with Annotations

Track why you rolled back:

kubectl annotate deployment nginx-deployment kubernetes.io/change-cause="Rollback to nginx:1.16 due to bug in alpine version"
kubectl rollout undo deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

Scenario 4: Rollback with Different Image

Manually set the image if you want to test a different version:

kubectl set image deployment nginx-deployment nginx-container=nginx:1.17 --record=true
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Rollback Issues

Issue 1: "No rollout history found"

Error:

Error: no rollout history found for deployment "nginx-deployment"
Enter fullscreen mode Exit fullscreen mode

Solution:

# Check if revisionHistoryLimit is set to 0
kubectl get deployment nginx-deployment -o yaml | grep revisionHistoryLimit

# If it's 0, set it to a higher number
kubectl patch deployment nginx-deployment -p '{"spec":{"revisionHistoryLimit":5}}'

# Then manually set the image
kubectl set image deployment nginx-deployment nginx-container=nginx:1.16
Enter fullscreen mode Exit fullscreen mode

Issue 2: Rollback Stuck

Symptoms:

Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Enter fullscreen mode Exit fullscreen mode

Solution:

# Check pod status
kubectl get pods

# Check pod events
kubectl describe pod -l app=nginx-deployment

# If stuck, force rollback
kubectl rollout undo deployment nginx-deployment --to-revision=1
Enter fullscreen mode Exit fullscreen mode

Issue 3: Container Not Found

Error:

error: unable to find container named "nginx"
Enter fullscreen mode Exit fullscreen mode

Solution:

# Find the correct container name
kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].name}'

# Use the correct name
kubectl set image deployment nginx-deployment [correct-name]=nginx:1.16
Enter fullscreen mode Exit fullscreen mode

Issue 4: Rollback Creates New Revision

Observation:

REVISION  CHANGE-CAUSE
2         Update to nginx:alpine
3         <none>  # Rollback created this
Enter fullscreen mode Exit fullscreen mode

Explanation: This is normal behavior! Rollbacks create new revisions so you can:

  • See exactly what was rolled back
  • Roll forward again if needed
  • Maintain a complete audit trail

Best Practices for Rollbacks

✅ DO's

1. Always Check History First

# Never rollback blindly
kubectl rollout history deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

2. Use Change Cause Annotations

kubectl annotate deployment nginx-deployment kubernetes.io/change-cause="Update to nginx:1.19"
Enter fullscreen mode Exit fullscreen mode

3. Monitor During Rollback

kubectl rollout status deployment nginx-deployment
Enter fullscreen mode Exit fullscreen mode

4. Keep Revision History

spec:
  revisionHistoryLimit: 5  # Keep last 5 versions
Enter fullscreen mode Exit fullscreen mode

5. Test in Staging First

# Test in staging
kubectl set image deployment/nginx-staging nginx=nginx:alpine
kubectl rollout status deployment/nginx-staging

# Then deploy to production
kubectl set image deployment/nginx-production nginx=nginx:alpine
Enter fullscreen mode Exit fullscreen mode

6. Have a Rollback Plan

  • Know who can perform rollbacks
  • Document the process
  • Practice in a test environment

❌ DON'Ts

1. Don't Delete Old ReplicaSets

# BAD - You lose rollback capability
kubectl delete rs nginx-deployment-xxxxxxxxxx

# GOOD - Let Kubernetes manage them
# Just leave them alone, they'll be cleaned up automatically
Enter fullscreen mode Exit fullscreen mode

2. Don't Rollback During Peak Hours

# Consider the impact on users
# Schedule during maintenance windows if possible
Enter fullscreen mode Exit fullscreen mode

3. Don't Forget to Communicate

# Notify your team about the rollback
# Document why it happened
Enter fullscreen mode Exit fullscreen mode

4. Don't Ignore the Root Cause

# Don't just rollback and move on
# Investigate what caused the bug
# Test the fix in staging
# Then try the update again
Enter fullscreen mode Exit fullscreen mode

Complete Rollback Script

Here's a complete rollback script you can use:

#!/bin/bash
# rollback.sh - Complete rollback procedure

echo "=== Kubernetes Rollback Procedure ==="

# 1. Get current state
echo -e "\n1. Current Deployment Status:"
kubectl get deployment nginx-deployment

# 2. Check rollout history
echo -e "\n2. Rollout History:"
kubectl rollout history deployment nginx-deployment

# 3. Check current image
CURRENT_IMAGE=$(kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}')
echo -e "\n3. Current Image: $CURRENT_IMAGE"

# 4. Ask for confirmation
echo -e "\n4. Are you sure you want to rollback? (y/n)"
read -r CONFIRM

if [ "$CONFIRM" != "y" ]; then
    echo "Rollback cancelled."
    exit 1
fi

# 5. Perform rollback
echo -e "\n5. Performing Rollback..."
kubectl rollout undo deployment nginx-deployment

# 6. Monitor rollback
echo -e "\n6. Monitoring Rollback Status:"
kubectl rollout status deployment nginx-deployment

# 7. Verify after rollback
echo -e "\n7. After Rollback:"
kubectl get deployment nginx-deployment

# 8. Check new image
NEW_IMAGE=$(kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}')
echo -e "\n8. New Image: $NEW_IMAGE"

# 9. Check pods
echo -e "\n9. Pod Status:"
kubectl get pods

echo -e "\n=== Rollback Completed ==="
Enter fullscreen mode Exit fullscreen mode

Real-World Example: Complete Rollback

Here's a complete example with all commands:

# 1. Check current state
kubectl get deployment nginx-deployment

# 2. Check history
kubectl rollout history deployment nginx-deployment

# 3. Check current image
kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}'
# Output: nginx:alpine

# 4. Annotate why we're rolling back
kubectl annotate deployment nginx-deployment kubernetes.io/change-cause="Rolling back due to bug in alpine version"

# 5. Perform rollback
kubectl rollout undo deployment nginx-deployment

# 6. Monitor rollback
kubectl rollout status deployment nginx-deployment

# 7. Verify image
kubectl get deployment nginx-deployment -o jsonpath='{.spec.template.spec.containers[0].image}'
# Output: nginx:1.16

# 8. Check history
kubectl rollout history deployment nginx-deployment

# 9. Check pods
kubectl get pods

# 10. Check events
kubectl get events --sort-by='.lastTimestamp' | tail -10
Enter fullscreen mode Exit fullscreen mode

Quick Reference Card

Rollback Commands

Command Description
kubectl rollout history deployment/NAME View all revisions
kubectl rollout history deployment/NAME --revision=N View specific revision
kubectl rollout undo deployment/NAME Rollback to previous
kubectl rollout undo deployment/NAME --to-revision=N Rollback to specific
kubectl rollout status deployment/NAME Check rollback status
kubectl rollout pause deployment/NAME Pause rollback
kubectl rollout resume deployment/NAME Resume rollback

Verification Commands

Command Purpose
kubectl get deployment NAME Check deployment status
kubectl get pods Check pod status
kubectl get rs Check ReplicaSets
kubectl describe deployment NAME Detailed info
kubectl get events View events

Conclusion

You've just learned how to safely rollback Kubernetes deployments! This is a critical skill that will save you when things go wrong.

Key Takeaways

  1. Rollbacks are easy: One command is all it takes
  2. Zero downtime: Rolling updates and rollbacks are gradual
  3. History matters: Kubernetes keeps track of changes
  4. Verify everything: Always check the rollback worked
  5. Communicate: Let your team know what happened

What You Learned

  • ✅ How to view deployment history
  • ✅ How to perform a rollback
  • ✅ How to verify a rollback
  • ✅ Common rollback scenarios
  • ✅ Troubleshooting techniques
  • ✅ Best practices

Next Steps

Now that you've mastered rollbacks, explore these related topics:

  1. Blue-Green Deployments - Advanced deployment strategy
  2. Canary Deployments - Gradual rollouts
  3. Feature Flags - Control features without redeploying
  4. GitOps - Declarative deployment management
  5. CI/CD Pipelines - Automate deployments and rollbacks

Final Words

Remember: It's not about avoiding failures, it's about being prepared to handle them gracefully. With Kubernetes rollbacks, you have a powerful tool to quickly recover from bad deployments.

Happy rolling back! 🚀

Top comments (0)