We push software to the counter PCs and tills in our forty six branches in waves. The first wave gets an update on Monday, and if nothing breaks the remaining waves follow through the week. It is a sensible design. The person who built it generated the waves by sorting the branch list by name and cutting it into groups.
That meant the first wave was Aberdeen, Ashford, Barnsley, Bedford and Bolton, every time, for two years.
I found out when the Aberdeen branch manager rang me to ask, politely, why his branch was always the one that broke. He had kept a list. Of nineteen faults we had shipped to the tills in that period, his counter had met fourteen first, and his staff had spent Monday mornings on the phone to our service desk while builders queued. Nobody had told him his branch was a test site. Nobody had asked.
The ordering also made the test weaker. The five first wave branches happened to be among our smallest, and none of them ran a timber cutting service or a large trade accounts desk. In April an update that slowed the till badly on any order with more than forty lines went through the first wave without a single complaint and reached our three busiest branches on the Wednesday.
We rebuilt the waves on purpose. The first wave is now chosen to cover the variety of the estate: one large branch, one small, one with a cutting service, one in Scotland and one in Wales, and the branches rotate every quarter so no single counter carries the risk for long. Each first wave manager is told on the Friday before, gets a direct number for that week, and has agreed to ring it rather than wait. The wave list lives in a file that a person reviews, not in a sort order.
In the two quarters since, first waves have caught four faults, including one that only appeared at branches with a cutting service, and none of them reached all forty six branches.
A pilot group only tells you about the sites in it. If you let a sort order choose them, you test on the same unrepresentative sites every time, and the people working there are the last to know they are part of the test.
– Serguey Shinder
Top comments (0)