DEV Community

Serguey Shinder
Serguey Shinder

Posted on

We Deleted a Server in July, Restored It in November and the Domain No Longer Recognised It

Our lorries' digital tachographs record drivers' hours, and the law requires us to download each driver's card at least every twenty eight days. For six years the card readers at our depots sent those downloads to a small Windows server in our cloud subscription, where analysis software picked them up.

In the summer we moved the analysis to the supplier's own online service, and in July we deleted the server. Our decommissioning checklist said to confirm nothing was still using it. We checked the analysis software and missed that two depots' card readers had never been repointed. Their uploads had been failing ever since.

The transport compliance manager noticed in November. The readers at those two depots had kept some files locally, but we needed the server back for a few days to receive the rest while the readers were reconfigured. The final backup had been kept under our retention policy, so the restore took forty minutes. The server started, and nobody could sign in to it with a domain account. The service that received uploads, which ran under a domain service account, would not start either.

The backup was fine. The domain had moved on. A Windows machine joined to a domain changes its own password with the domain every thirty days, and the copy we restored remembered a password the domain had replaced months before. The checklist had removed the virtual machine and left its account in the directory, so we could repair the relationship rather than rejoin the server. One of my engineers signed in with the machine's local administrator account, ran the standard repair command with domain credentials, and the server was working an hour later.

It cost a day, not a disaster, but it showed two gaps. Servers are now switched off for sixty days before they are deleted, so anything still depending on them complains while bringing them back is just pressing start. And our restore runbook, which ended at the server starting, now ends at signing in with a domain account and checking that its services run.

A restore brings back the machine, not the relationships it had. Test the part that depends on everything else having stood still.

– Serguey Shinder

Top comments (0)