DEV Community

Hive80-lab
Hive80-lab

Posted on

The Git Commands That Save You When Prod Is Broken at 3AM

#git #python #devops #productivity

The Git Commands That Save You When Prod Is Broken at 3AM

Your production server is crashing. Customers are emailing support. Your pager
is screaming.

You SSH into the server and stare at a terminal that has seen too many
midnight crisis calls.

Git is your safety net. These are the commands that will save you from
yourself when you're tired, stressed, and desperate.

The Golden Rule: Never Reset Without Checking 

git log --oneline -5
git reflog
Enter fullscreen mode Exit fullscreen mode

What this shows: Every HEAD pointer in the last 500 commits.
Reflog saves you from "oops I did git reset --hard HEAD~5 and my
feature branch is gone forever."

Always check reflog BEFORE ANY hard reset.

Scenario A: The Bad Merge Is Live Now

You merged a feature branch with a bug that's breaking prod.

Wrong way:

git reset --hard HEAD~1   # ❌ Deletes the bad branch
git checkout master
Enter fullscreen mode Exit fullscreen mode

Right way (safe revert):

git checkout master
git revert --no-commit HEAD~1
git commit -m "Revert bad merge: fix prod-breaker"
git push origin master
Enter fullscreen mode Exit fullscreen mode

Revert creates a new commit that undoes the changes. Your branch history
remains intact. Production is safe, developers are happy.

Scenario B: The Commit With Three Bugs Is Live

You need to cherry-pick the one good commit, but you're stuck with three
bad commits ahead of it.

The fix:

# List commits you care about
git log --oneline --graph

# Cherry-pick the one good commit, drop the bad ones
git checkout master
git cherry-pick 3a5b7c9  # The good commit hash
Enter fullscreen mode Exit fullscreen mode

If the cherry-pick fails, abort it:

git cherry-pick --abort
Enter fullscreen mode Exit fullscreen mode

Scenario C: You Need a Hotfix in a Long-Running Branch

You're in a 3-month-long feature branch and a hotfix is needed immediately.

The fix:

# Create a temp branch for the hotfix
git checkout -b hotfix-urgent
git cherry-pick <commit-hash>

# Push hotfix branch
git push origin hotfix-urgent

# Rebase feature branch on top of hotfix
git checkout feature-long-running
git rebase master  # Now master is updated with the hotfix
Enter fullscreen mode Exit fullscreen mode

Result: your feature branch is now on top of master, with no merge commits,
and you can finish it in peace.

Scenario D: You Need to Find the First Bad Commit

A regression appeared after the last release. You don't know which commit
introduced it.

The git bisect command:

git bisect start
git bisect bad master  # HEAD points to the known bad state
git bisect good v2.1.0  # Last known good version

# Git will automatically bisect and ask you to test each revision
# Run your test suite, then say:
#   git bisect good  # If revision is good
#   git bisect bad  # If revision is bad

# When done:
git bisect reset
Enter fullscreen mode Exit fullscreen mode

Git will point you to the exact commit that broke things. Save yourself
hours of manual digging.

Scenario E: You Stashed Changes That You Forgot

You have a feature halfway done but need to fix a production bug.

git stash list
git stash pop  # Apply the most recent stash
# OR
git stash apply stash@{1}  # Apply a specific stash by index
Enter fullscreen mode Exit fullscreen mode

To delete a stash after applying:

git stash drop stash@{1}
Enter fullscreen mode Exit fullscreen mode

Scenario F: You Are in Detached HEAD Limbo 

git checkout <commit-hash>
# Now you're in detached HEAD state — you're literally on a loose commit

# Return to master:
git checkout master

# OR, if you're attached to a branch but don't know which:
git branch -a
Enter fullscreen mode Exit fullscreen mode

Never work in detached HEAD. Always checkout a branch first.

Scenario G: The Reflog Isn't Showing What You Need

You accidentally deleted a branch:

git branch -D removed-feature  # Force delete
# Oops. Is the branch still recoverable?

git reflog
# Look for the HEAD reference that points to that branch
git checkout HEAD@{5}  # Go back in time to where the branch existed
git checkout -b removed-feature  # Recreate the branch
Enter fullscreen mode Exit fullscreen mode

The reflog is a time machine. It doesn't expire immediately. Use it.

The Mental Checklist for Midnight Crises

Step Command Why
1 git log --oneline -5 See recent commits
2 git reflog Check saved HEAD pointers
3 git checkout master Move to production branch
4 git status See uncommitted changes
5 git diff HEAD~3..HEAD Review what changed
6 Choose revert OR cherry-pick Safe recovery

Real Example: The 3AM CDN Cache Bug

The situation:
A CDN update caused our homepage to cache a 503 error page. Users saw
"Site Unavailable" for 45 minutes.

The fix:

git checkout main
git log --oneline -10
# Last commit: "Update CDN config v2"

# Confirm it's the bad one
git show HEAD --stat

# Revert just the bad change
git revert --no-commit HEAD
git commit -m "Revert CDN config v2: fixes 503 cache issue"
git push origin main

# The CDN picks up the fix in ~30 seconds
Enter fullscreen mode Exit fullscreen mode

Total time: 3 minutes. Git saved us 45 minutes of downtime.

The Cost of Not Knowing These Commands

  • 3 hours of downtime = $2,400-$10,000 per incident (varies by customer count)
  • Lost customers = $500-$10,000 CLV per customer
  • Support ticket backlog = $100/hour in overtime
  • Reputation damage = indefinite long-term revenue loss

A 15-minute git refresher today saves you potentially thousands in a single
night.

One Action You Can Take Right Now

  1. Run through these scenarios with a dry-run repository.
  2. Create a personal "Git Rescue Cheat Sheet" PDF.
  3. Share it with your team (or worse, force them to read it).

CTA:

Discussion Question

What's your most memorable 3AM git story? Comment below and share your

lesson learned.

Running a stack like this? Our Ops Starter Kit ships the runbooks, checklists, and monitoring patterns behind these posts — grab it and cut your next incident short.

Top comments (0)