What's the worst mistake you've ever pushed to production or deleted by accident? 🤔

Have you ever deleted live data from a server?
Pushed something that broke everything? 😬

How did you recover from it?
Let’s share some war stories — and lessons learned. 💬

Comments

[Ben Halpern]

I pushed some logic to Heroku which triggered a reboot on boot. I.e. infinite loop.

It put the application into a state which we could not get out of — and then our account became suspended and we triggered a system-wide "warning" on their status page. Not like I caused an outage or anything, but it my code was bad enough to cause a system-wide circumstance of some kind.

[Sebastian Van Rooyen]

Not my experience specifically but a colleague migrated to a new service provider without updating the API key in the production environment variables. Entire time they were trying to do "bug-fixes" instead of looking at the logs which would identify the 401 access issue specifically linked to the API key. Normal human error mistake. Could happen to anybody

[Cesar Aguirre]

• Took down a database server with a poorly written query
• Wrote a module that every time we deployed it into a custom CRM took down the website

:\