DEV Community

Hive80-lab
Hive80-lab

Posted on

The Git Commands That Save You When Prod Is Broken at 3AM

#git #devops #sre #programming

Prod is broken. You're tired. And the worst possible thing you can do is start remembering git — memory fails at 3AM. Here are the six commands that actually matter in an incident, in the order you'll need them. Bookmark this; future-you at 3AM will thank present-you.

1. git reflog — the undo history that never lies

You reset the wrong branch? Rebased over your hotfix? Reflog is the flight recorder: it logs every move HEAD ever made, even "destructive" ones.

git reflog
# a1b2c3d HEAD@{0}: reset: moving to origin/main
# e4f5a6b HEAD@{1}: commit: hotfix: payment retry loop   <- this is what you lost
git checkout e4f5a6b
Enter fullscreen mode Exit fullscreen mode

Nothing is gone until the garbage collector eats it (default: ~90 days for reachable, ~30 for dangling).

2. git revert — the ONLY undo on shared branches

On main, never reset --force. Reset rewrites history; everyone's next pull becomes a minefield. Revert writes a new commit that undoes an old one:

git revert abc1234        # safe on shared branches
git revert --no-commit abc1234 def5678 && git commit   # batch several
Enter fullscreen mode Exit fullscreen mode

3. git cherry-pick — move ONE commit to safety

The hotfix pattern: the fix is sitting on a feature branch, prod needs it now, without the other 40 commits.

git checkout main && git pull
git cherry-pick abc1234
Enter fullscreen mode Exit fullscreen mode

If it conflicts, git cherry-pick --abort exits clean. No harm done.

4. git bisect — find the breaking commit in log(n) steps

500 commits since the last good deploy? Don't bisect by hand:

git bisect start
git bisect bad                    # current broken state
git bisect good v2.3.0            # last known-good tag
# git checks out the midpoint; run your test; then:
git bisect good   # or bad
# ~9 steps for 500 commits. Then:
git bisect reset
Enter fullscreen mode Exit fullscreen mode

5. git stash — the emergency drawer

Mid-investigation, everything half-edited, and now you must jump branches:

git stash push -m "wip investigating payment bug"
git checkout main
# ... later:
git stash pop
Enter fullscreen mode Exit fullscreen mode

git stash list shows everything; git stash show -p stash@{1} previews without applying.

6. The panic button: recover a "deleted" branch 

git branch -D experiment        # whoops
git reflog                      # find the last commit on it
git checkout -b experiment <sha>
Enter fullscreen mode Exit fullscreen mode

The rule that prevents all of this

Never debug by memory at 3AM — debug by artifacts. The teams that recover fastest keep a runbook with these exact commands pre-filled for THEIR repo (branch names, tag formats, the test to run during bisect). That's why every ops kit I ship includes the runbook layer, not just scripts: Agent Ops 24/7 runs the busywork loop, the Automation Starter Pack ($19) covers the deploy-and-watchdog layer, and the Ops Mega Bundle is everything at bundle pricing. Full catalog: hive80lab.gumroad.com.

Question for the comments: what's your worst 3AM git moment? Mine: reset --hard on a shared branch at 02:50, and the sprint's work existed only in a teammate's un-pushed working copy. Reflog saved us. What saved you?

Top comments (0)