DEV Community

Discussion on: What have you crashed?

Collapse
 
johncip profile image
jmc • Edited

One of my early PRs at a shop doing mostly zero-downtime schema migrations turned out to... not be zero-downtime. Unexpectedly so. We got the first user email within seconds. Good times.

Once, while working a sysadmin for my family's business, I wiped an important application server and re-installed the OS only to find out that I couldn't restore the db from a backup tape. I spent all day and night in the server room, freaking out, on long phone calls with Veritas and MS, while my uncle had to do the software's job on paper, without the benefit of the now-missing historical data.

Phone support was no help, but I eventually realized that the restore job might be crashing because, in an unrelated move, I'd also re-organized the domain's admin accounts, and the one that made the backup wasn't the same one being used to restore. (Likely a bug -- the error message was completely unrelated the new account had sufficient privileges.)

In the end, I was thankfully able to restore everything, but I looked like an idiot.