Moving the warehouse system out of our data centre was planned as a weekend. The application had been rehearsed twice, database replication was running, and the cutover plan fitted on two pages. We expected to change one DNS record on Saturday night and go home.
Three weeks before the date, during a rehearsal at a single depot, we found out that a DNS record was not what most of our devices used. The scanners pointed at the server by its IP address, entered in a configuration profile in 2016. So did the label printers, which had the address typed into each one through its front panel. The weighbridge terminals at four depots kept it in a text file. Two conveyor controllers had it compiled into their software by the integrator who installed them, and the engineer who did it no longer worked there. A name can follow a server to the cloud. An address belongs to the network we were leaving.
Nobody had chosen this. Each device had been set up by whoever installed it, an address was what the installation guide showed, it worked, and it kept working for ten years because the server never moved. Our inventory recorded the model, serial number and site of every device. It did not record what any of them talked to.
The weekend became four months. The scanners were straightforward, one profile pushed from device management. The printers needed somebody at the panel of each one, a hundred and twelve times. The conveyor controllers needed the integrator back, at a day rate, in a maintenance window operations could only give us on a Sunday. For those four months we ran a forwarder from the old address to the new environment, which added a little delay to every transaction and a single point of failure we had not wanted.
Every device now refers to services by name, and those names carry a short time to live. The inventory has a field for what each device connects to, filled in at installation, and the standard we give contractors says a name or nothing. Before any future move, a report lists every device whose configuration contains a raw address. Today it lists nine.
The server took a weekend to move. The references to it had been written into places we did not know we owned, and finding them was the actual migration.
– Serguey Shinder
Top comments (0)