Today’s incident wasn’t about advanced tools, Kubernetes magic, or complex debugging.
It was about understanding Git, branches, and deployment flow under pressure.
Just a few hours before the Live demo of a new feature in Production, A faulty commit broke the pipeline, and haste made it worse. In a hurry with pure intentions of making the system live. Developer A quickly reverted the change, but the pipeline still failed. Meanwhile, Developer B merged new code into main, adding more noise to an already messy history.
Read the story at my personal blog: https://wasiabdullah.com/live-on-call-troubleshooting-on-production-live-app/
While everyone is worried about the system and the upcoming session. I hold my nerves.
After chasing CI logs, I checked what the server was actually running versus what existed on remote main in GitHub. A force pull, manual rebuild, and redeploy brought the system back live in minutes - just by relying on Git basics and deployment clarity.
git fetch origin
git reset --hard origin/main
docker compose build --no-cache
docker compose up -d
Top comments (0)