Our collection booking site, where business customers arrange for us to pick up from their premises, is built and run by a team of nine developers. They release several times a week, and like most teams who release often they use feature flags, so new code can go out switched off and be turned on gradually.
In June the site slowed down during the afternoon peak. The engineer on call opened the flag console, found a flag called new_booking_flow, and decided that switching it off was the quickest way to take a recent change out of the path. The site recovered. Bookings kept arriving, customers kept receiving confirmations, and the incident was closed in twenty minutes.
The flag was not recent. It had been created in 2024 to move bookings from the old flow to the new one, and nobody had ever removed the old code behind it. With the flag off, the site went back to the old flow, which wrote bookings into a table the transport planning system had stopped reading when we migrated. For three hours, six hundred and twenty collections were accepted, confirmed by email and sent nowhere. We found out the next morning, when customers rang to ask where their drivers were.
The console held four hundred and thirty flags. About three hundred had been fully on or fully off for more than six months. Some were finished releases with dead code still attached. Others were genuine operational switches, such as turning off card payments if the payment provider fails. In the console they looked identical, and nothing told the engineer which kind he was holding.
Every flag now gets a type, an owner and a review date when it is created. A release flag must be removed, code and all, within six weeks of reaching every customer, and the build fails once that date has passed, so the clean up cannot be put off indefinitely. Operational switches are listed in the runbook with what they do and when they must not be used, and each one is exercised in staging every quarter. The console shows the type beside the name. We are down to sixty flags.
A flag that is never removed stops being a release tool and becomes an untested second version of the product. Switching one off during an incident is a deployment, even when it takes a single click.
– Serguey Shinder
Top comments (0)