Our Morning Deployment Undid the Fix We Made at Three in the Morning

Our network rules for the cloud are held as code. Every firewall rule, route and load balancer setting lives in a repository, and a pipeline applies the repository to the live environment every morning at seven, so anything changed by hand is put back the way the code says. We built it that way on purpose, to stop the estate drifting.

In January a carrier we exchange consignment files with moved its servers to new addresses, with less notice than it should have given. At three in the morning our night engineer found the overnight transfers failing, worked out why, and added the new addresses to the firewall by hand in the console. The files went through. He wrote it up in the incident record, which was exactly what our process asked of him, and went home at six.

At seven the pipeline ran. It compared the live firewall with the repository, found two rules that were not in the code, and removed them. The transfers failed again. The day team found the incident record marked resolved, and spent the first hour of the morning diagnosing a fault that had already been diagnosed, while forty depots waited for their loading files.

The automation did what we had built it to do. Our incident process told engineers to fix the service and record what they did. Our pipeline treated every change outside the repository as damage to be repaired. Each was reasonable, and nobody had made them agree with each other.

An emergency change made by hand now has to be followed by a matching change to the repository before the incident can be closed, and the engineer on the next shift owns that if the person who made the fix has gone home. The pipeline no longer removes differences silently. It reports any rule it is about to remove, and if that rule was created in the last twenty four hours it stops and asks the on call engineer, rather than reverting it. Every morning run posts a summary of what it changed to the operations channel.

Since then the pipeline has stopped twice on overnight fixes, and both were committed within the hour.

Automation that restores the agreed state is only as good as the agreement. We had forgotten to tell it about the one at three in the morning.

– Serguey Shinder