Since 2022 our file servers and document store have been backed up every night to a cloud provider. The backups cannot be altered for thirty days, they are checked every week, and we restore individual folders from them most weeks without any trouble. When the board asked last autumn whether we could recover from ransomware, I said yes.
In February we ran the exercise properly, against the whole estate rather than a folder. The data had grown to about forty six terabytes. Our head office internet line is one gigabit, which sounds like plenty. In practice the restore ran at about three hundred and fifty megabits a second, because the provider limits each restore stream and because the same line carries everything else the building does. At that rate the full restore would have taken a little over twelve days. The recovery objective in our continuity plan for those systems is forty eight hours.
Nobody had made a mistake. The arrangement had been designed when the estate was eleven terabytes and a full restore over the line took about three days. Every year since, the data grew and the line did not, and every test we ran was a folder, which proved that the backup existed and said nothing about how long it would take to bring all of it home. The time to restore depended on the volume and on the line, and nobody owned both.
We changed three things. A copy of the last fourteen days now sits on locked storage at our second site, so the restore we are most likely to need runs at the speed of our own network, which took nineteen hours in the last test. The data has been sorted into tiers with the business, and the four terabytes that operations, finance and customer services need first are restored first, so the company can work within a day while the archive follows behind it. For the archive, the provider will ship the data to us on disks, and that service is now in the contract with a written delivery time, rather than something we would have discovered during an incident. The annual test restores everything, and it is timed.
A backup in the cloud is protected from almost everything except the distance between it and us. I now ask how long the restore would take, not only whether the backup is there.
– Serguey Shinder
Top comments (0)