Recovery testing: restore order and verification, not just restore success

Restore success is not recovery

A backup that restores cleanly proves the backup is readable. It does not prove the organisation can return to service, because recovery also depends on the order services come back, the identity and network layers they need first, and whether the restored data is clean.
Teams often measure the wrong thing. Restore tests that check a file count or a database checksum pass on backups that contain an attacker's persistence alongside legitimate data.

Design the exercise around dependencies

Choose a service that matters and work backwards. What does it need to authenticate users. What does it need to resolve names. What does it need to reach its data store. What does it need to be reachable by users. Write the order down before the exercise, then test whether the documented order matches reality.
In practice the order is usually wrong in a specific way: an identity service is restored late, so nothing can authenticate; a DNS record points at the old address, so clients never reach the restored service; or a certificate required by the restored service was only stored on the system that was lost.
Finding these during an exercise is cheap. Finding them during a real recovery, with the clock running, is not.

Verify that restored means clean

A restore reproduces whatever was in the backup at that point in time. If the compromise predates the most recent backup, restoration reintroduces it. Establish how you will date the compromise, and how you will select a restore point that predates it without losing acceptable amounts of data.
Immutable or write-once storage is what makes this possible at all, because a backup an attacker can delete or encrypt is not a recovery option. Confirm that the immutability is enabled in the product rather than assumed from the vendor's documentation.

Exercise the decisions, not just the technology

Recovery involves choices that a restore script cannot make: which restore point, whether to rebuild or restore, whether to bring a service back before it is fully patched. These should be exercised by the people who would make them, with a written record of who decided what.
Time the exercise and compare against the documented objective. If the target is four hours and the first attempt takes two days, that difference is the finding, and it is more useful than a pass or fail result.

Keep the evidence

Record the restore point and its timestamp, the components restored, the verification performed, and the residual gaps. Keep the record with the backup configuration so the next exercise starts from what was actually tested rather than from assumptions.

References

• NIST SP 800-184, Guide for Cybersecurity Event Recovery: https://csrc.nist.gov/pubs/sp/800/184/final
• NIST SP 800-34 Rev. 1, Contingency Planning Guide: https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final
• CISA, Ransomware Guide: https://www.cisa.gov/stopransomware/ransomware-guide