DEV Community

Discussion on: Fire Drills for Software Teams

Collapse
 
rapidnerd profile image
George

I love the idea of fire drills. I worked for a company that was badly affected by the GitLab accident. We lost our entire network system and crucial data that kept us alive. We obviously had backups of it else were but the fact we had our current code pushed onto GitLab for everyone to work on and then it just suddenly goes poof it became a real pain the ass.

Remembering the moment it happened for me it was about 3 am getting a text from my boss saying that GitLab has gone down and we need to begin a procedure called Donkey Kong (idk why it was named that). Just remember thinking to myself what on earth is that?

Even though I've since left the company the sheer panic in every team members voices when this happened. We introduced a fire drill system so that every 6 hours a live backup would be sent from Github to a dedicated server. I really hope this never happens again even though I've switched personally and professionally to GitHub I know it can happen at any time.

If it costs a bit of money a month, or takes a while to setup a fire drill to protect yourself and/or a company it is 100% worth it, in the event something like this would to happen again at least you'll know that there is something in place to help you and others.

Collapse
 
mshmsh5000 profile image
Matt Holford

Absolutely—part of the value is having actual steps in place, and part of it is just flexing the "rapid response" muscle. The team gets better at coordinating in real time, regardless of what the actual disaster is.