Our Android app went out on a Tuesday at twenty percent staged rollout, and by Wednesday lunchtime the crash reporter showed one in six sessions on that version ending in a crash at the payment screen. Every instinct from our backend work said roll back. On a phone, there is no roll back.
A store release is not a deployment we control. We could halt the staged rollout, and did, within twenty minutes of seeing the graph. That stops new installs. It does nothing for the roughly forty thousand phones that already had the new version and would keep it until we shipped a newer one. The store does not let you offer an older build over a newer one. The fix was a two line change, and it still took a day and a half to reach users: build, internal testing, store review, a new staged rollout, and then people actually opening the store.
Everything in our release process had been designed for services, where the worst case is five minutes of bad traffic before the previous image comes back. The app pipeline was the same pipeline with a different last step, and the last step was the irreversible one. Our gate was a person reading a crash dashboard some time after the rollout began, and the crash came from a payment SDK update misbehaving on one manufacturer's older Android release, which none of our test devices ran.
Three things changed. Every new feature in the app, and every change to a third party SDK's code path, ships behind a remote flag that the server can turn off per version and per device model, so a bad path can be closed in minutes without anyone installing anything. The staged rollout is driven by a job rather than a person: one percent, then five, then twenty, each held for a fixed number of sessions, and halted automatically if crash free sessions fall below the previous version's by more than half a point. And the test matrix now uses a device cloud chosen from our real install base, the top models by share covering eighty percent of users, rather than the phones on our desks.
On a server, a release is reversible and so it can be fast. On a phone, it is permanent the moment it is installed, and the only undo button is the one you built into the app before you shipped it.
– Sergey Shinder
Top comments (0)