DEV Community

Discussion on: "Crap, I broke production" - How do we ensure it never happens again?

Collapse
 
andreidascalu profile image
Andrei Dascalu

Well, no amount of best practices will guarantee production will never break again. Even not breaking in the exact same way again is quite a goal, though it's generally doable as long as adopted practices are respected.

In the face of disaster lessons are learned and strategies implemented. But in too many cases those strategies keep up only as long as people remember the size of the potential disaster and use it to keep off the pressure to sidestep safeguards. Otherwise, a few months from adoption someone will find a justification to silently bypass something.

In the happy case, all hell breaks loose and you have a chance to enhance discipline after hopefully averting disaster at the last minute like James Bond. Things do improve then.

In the sad case, it works, management praises the dodger for delivering something quickly simply introducing an incentive to do it more and get away with it. Eventually several hells break loose at the same time leaving the team unable to avert them all.