The morning shift at most of our depots starts at six. For years that meant very little to my team, because the capacity plan for the warehouse system was built from hourly figures, and the busiest hour of an ordinary day ran at about forty percent of what the servers could carry.
Last November the night shift handed over at our largest depot and the day shift spent its first twenty minutes looking at a sign in screen. Then the same thing happened at four more depots. Scanners showed a spinning wheel, supervisors restarted them, which made it worse, and the first trailers left forty minutes late. By eight o'clock every graph we owned looked healthy, and the review very nearly concluded that it had been the network.
What actually happened lived inside about six minutes. Between five to six and one minute past, roughly fourteen hundred scanners and two hundred shared terminals across eighteen depots all signed in, each one opening a session on the warehouse system and requesting a token from our identity service. In peak season a third of those people were agency staff on their first or second shift, and a first sign in builds a profile, which takes about four times as long. The identity service queued, the session broker timed out and retried, and every restart on the floor added another request to the queue. Averaged over the hour, it had been a quiet morning.
We now measure shift changes by the minute, and the capacity plan is written against the first ten minutes of the busiest shift in the busiest week, not against the hour. The load test replays those ten minutes. Agency profiles are created the evening before a booked shift, from the agency's booking list. Sessions now survive breaks and a scanner going back into its cradle, so the second wave of sign ins after lunch has disappeared. And the three largest depots start their zones at ten minute intervals, which operations agreed to once they saw the graph, because it also shortened the queue at the equipment cage.
In our peak rehearsal in August, the slowest depot had its whole shift signed in within eleven seconds.
Our users work by the clock on the wall, and they all look at the same clock. A capacity figure averaged over an hour had hidden the one minute in which every one of them needed us at once.
– Serguey Shinder
Top comments (0)