DEV Community: Mumtaz Jahan

DevOps Scenario Interview Question: Deployment Failed in Production

Mumtaz Jahan — Sat, 18 Apr 2026 14:11:02 +0000

Tags: devops kubernetes cicd career

Scenario: Your Deployment Failed in Production. What Steps Will You Take?

This is one of the most common real-world scenario questions asked in DevOps interviews. Interviewers don't want textbook answers — they want to know how you think under pressure.

Here's the complete answer framework.

Answer: Step-by-Step Approach

1. Check CI/CD Pipeline Logs

First thing — don't guess, read the logs.

# For Jenkins
cat /var/log/jenkins/jenkins.log

# For GitHub Actions — check the Actions tab in your repo

# For GitLab CI
gitlab-ci logs

The pipeline log tells you exactly where it broke.

2. Identify the Failed Stage (Build / Test / Deploy)

Every pipeline has stages. Narrow it down:

Build failed? → Dependency issue, Dockerfile error, compilation error
Test failed? → A test caught a regression before it hit production
Deploy failed? → Kubernetes issue, wrong image tag, resource limits, misconfigured secrets

Knowing the stage cuts your debugging time in half.

3. Verify Configuration Changes

Check what changed before the failure:

# Check recent git commits
git log --oneline -10

# Check Kubernetes config changes
kubectl describe deployment my-app

# Check if secrets/configmaps were updated
kubectl get configmap my-app-config -o yaml

Most production failures trace back to a config change someone forgot to mention.

4. Rollback to Previous Stable Version

Don't try to fix forward when production is down. Rollback first, fix later.

# Kubernetes rollback
kubectl rollout undo deployment/my-app

# Verify rollback status
kubectl rollout status deployment/my-app

# Check rollout history
kubectl rollout history deployment/my-app

This restores service immediately while you investigate the root cause safely.

5. Fix the Issue and Redeploy

Once production is stable:

Reproduce the issue in staging
Apply the fix
Test thoroughly
Redeploy with the corrected version

kubectl set image deployment/my-app my-app=my-image:v2.1-fixed
kubectl rollout status deployment/my-app

Pro Tip

Always maintain versioned Docker images — never use latest in production.

# Bad
image: my-app:latest

# Good
image: my-app:v2.0.1

Without versioned images, you can't rollback. Tag every release.

Bonus: What Interviewers Are Really Looking For

They want to see that you: don't panic, prioritize restoring service over finding blame, think in structured steps, and know the actual commands — not just theory.

*Preparing for a DevOps interview? Drop your toughest scenario question in the comments *

How to Fixed a Kubernetes CrashLoopBackOff in Production

Mumtaz Jahan — Fri, 17 Apr 2026 23:55:42 +0000

Tags: kubernetes devops debugging cloud

Problem: Application Was DOWN in Kubernetes

One of the most stressful moments in DevOps — you check your monitoring dashboard and your application is completely DOWN in Kubernetes. No graceful degradation. Just... down.

Here's exactly how I diagnosed and fixed it in under an hour.

Issue Found: Pod Was in CrashLoopBackOff

Running kubectl get pods revealed the culprit immediately:

NAME                        READY   STATUS             RESTARTS   AGE
my-app-7d9f8b6c4-xk2pq     0/1     CrashLoopBackOff   8          20m

CrashLoopBackOff means Kubernetes is repeatedly trying to start your container, it crashes, and Kubernetes backs off with increasing wait times before retrying. Something inside the container was causing it to exit immediately on startup.

Debug Steps

Step 1: Checked Logs (`kubectl logs`)

kubectl logs my-app-7d9f8b6c4-xk2pq --previous

The --previous flag is crucial here — it lets you see logs from the crashed container, not the current (possibly empty) one. The logs showed repeated connection errors on startup.

Step 2: Checked Config

kubectl describe pod my-app-7d9f8b6c4-xk2pq

I inspected the environment variables and ConfigMaps attached to the pod. The describe command is a goldmine — it shows events, resource limits, volume mounts, and more.

Step 3: Found DB Connection Issue

The logs made it clear: the app was trying to connect to the database using an incorrect connection string. The host value in the environment variable was pointing to a stale endpoint. The app would crash immediately on boot since it couldn't reach the DB.

Fix Applied

Corrected Environment Variables

Updated the Kubernetes secret/configmap with the correct database host:

kubectl edit secret my-app-db-secret
# or
kubectl set env deployment/my-app DB_HOST=correct-db-host.internal

Restarted the Deployment

kubectl rollout restart deployment/my-app

Then watched the rollout:

kubectl rollout status deployment/my-app

🎉 Result

 Application UP
 Issue resolved

The pods came up healthy, readiness probes passed, and traffic started flowing again.

Key Takeaways

Always check logs with --previous — the live container may have no logs if it crashes before writing any.

kubectl describe pod is your best friend for seeing the full picture: events, env vars, resource pressure.

CrashLoopBackOff is almost always one of: bad env vars/secrets, missing config, OOM kill, or a bug triggered at startup.

kubectl rollout restart is safer than deleting pods manually — it does a rolling restart with zero downtime.

*Hit a similar issue? Drop your debugging story in the comments *