We run our own till software on two thousand four hundred shop terminals. Version 4.2 rolled out over a fortnight without incident. Then 4.3 went out, carrying a fix we actually needed, and two days later the adoption dashboard still read zero percent. Not slow. Zero.
Nothing was wrong with 4.2 as a piece of software. The terminals ran it perfectly for eleven days. What 4.2 contained was a tidy up: the updater, which had been its own small service, was folded into the main application, because two processes where one would do had been on somebody's list for a year and the review agreed with them. Its check for a new version moved to application startup.
A terminal starts its application once, when the shop opens. On 4.2 the check ran about fifty seconds into startup, which is before the site's tunnel to us is established, so it got a connection error. Being a once per start check, it did not run again until the next morning, when it failed at the same point for the same reason. Nothing was flaky. Every terminal in the estate failed deterministically, every day, and reported nothing, because a failed update check was a debug line on a till in a shop.
Getting 4.3 out took five days through our remote support tool, which is licensed for sessions with an engineer on the other end and is not a distribution channel. Nineteen sites needed somebody to visit, because the terminal had been switched off during the window we were working through.
The updater is a separate package again with its own release train, changed rarely and never in the same release as the application it installs. It checks on a schedule through the day rather than once at start. Every terminal reports its version and last successful check hourly, and we alert on the proportion of the estate that has gone quiet, which is a fact about the channel rather than about any one rollout. And after every release the pipeline publishes a version that changes nothing and asserts that ninety five percent of terminals take it within the hour.
Everything you ship can be fixed by shipping again, except the part that does the shipping. Keep it apart from what it carries, and prove the road is still open before you need it.
– Sergey Shinder
Top comments (0)