We spent most of a morning investigating a staging failure in checkout, working through our own recent changes, before someone from another team mentioned they had run a migration the night before that renames a column we read. Their change, our failure, four engineers, one morning.
There was one staging environment. One namespace, one database, one set of partner stubs, shared by nine teams, booked first in a spreadsheet and later in a channel. A release soak took two hours in it, which meant a team got a slot roughly when the queue reached them.
The attribution problem was the expensive one and we had never measured it. When I went back through eight weeks of staging failures, thirty one out of seventy four had been caused by a change belonging to a different team than the one investigating. Every one of those was diagnosed eventually, by hand, by people reading a stack trace that pointed at their own service and was telling the truth about somebody else's change.
We already had the environment described in code, so building a per pull request environment was less work than the two years of process we had grown around the shared one. A namespace, a database restored from a sanitised nightly snapshot, partner stubs, a generated hostname. Six minutes to create, destroyed on merge or after twelve hours, and a little under three hundred pounds a month for all of them together.
The shared environment still exists, because a few things genuinely cannot be duplicated: one partner test account, one link to a mainframe. Those now sit behind a lock the pipeline takes and releases, rather than a person writing in a channel.
A failed soak names exactly one change now, which is what a soak was always supposed to do.
What I take from it is that we had spent two years making the automated part of our release faster, from twenty six minutes down to nineteen, and the automation had never been the constraint. The constraint was one scarce physical thing that everybody queued for, and it had been there so long that none of us counted it as part of the release process at all.
– Sergey Shinder
Top comments (1)
The 31/74 measurement makes the case stronger than the anecdote. Did you keep the shared environment for the two non-duplicable integrations behind a lease with a timeout, or is the lock held for the whole test job? That failure mode is usually the next queue.