Deleting Data Is a Wish
Last Saturday I was walking through Sham Shui Po's electronics bazaar — the kind of place where you can buy a used router and get someone else's VPN credentials for free. One stall had a pile of loose 2.5-inch drives sitting in a cardboard box, no labels, no wiping marks. I plugged a SATA-to-USB adapter into one right on the table. The drive spun up and gave me a directory listing from a defunct marketing agency: client contracts, a MySQL dump, and a CSV called leads_2023.csv. The stall owner shrugged when I asked about data destruction. "They just sell them."
It was 3:40 PM. I counted twenty-one drives on that table. One had a drill hole through the case. The rest looked exactly as they'd left some server rack — sticky tape residue and all.
A flea market stall is just the unvarnished version of what most engineering teams do when they decommission storage. Same problem, better graphics.
I've been on the audit side of data compliance for a few years now. The thing I've learned: "we deleted it" is not evidence. It's a wish. An auditor can't verify a wish. So I spend a lot of time asking the only question that matters: can you prove the data is gone?
The first time a client asked me that, I didn't have a good answer. At the time, our own deletion pipeline was a bash script that ran rm -rf on the mount point, then a quick dd if=/dev/zero if the disk was being retired. On HDDs, that leaves the data magnetically latent. On SSDs, it does nothing useful because the flash translation layer remaps blocks behind your back. We had no hashes before, no hashes after, no record of which tool version touched which asset. If someone had challenged us in court, we'd have produced a log file with a timestamp and a shrug.
So we sat down and designed what a defensible erasure process actually looks like. Not perfect, but the kind of process that doesn't make an auditor laugh.
The thing we figured out early: erasure is an evidence-generation process, not a storage operation. Once you frame it that way, the design decisions follow.
First, classify before you destroy. We used to wipe everything the same way, and that starting point was wrong in both directions: we spent the better part of a week wiping drives that didn't need wiping and missing the ones that did. A disk that held encrypted backups has a different risk profile than a cache node with plaintext session logs. The fix was an inventory record, created when the volume is allocated, not when it's decommissioned. For every physical or virtual disk, we record what workloads ran on it, what kind of data it held, whether it had encryption at rest. Without that, your erasure plan is a guess with a checklist attached.
Then we rebuilt the actual erasure path. For HDDs, we moved to ATA Secure Erase. The first time we tried it, our USB bridge chips didn't pass the command through. The drives just sat there and the tool reported success anyway. That's a fun audit finding: the tool said yes, the drive did nothing. You have to run it through a controller that actually forwards the command, or verify the drive's response after the erase. And check the security state first — we hit drives where the feature was frozen and the erase silently waited for a power cycle. ATA Secure Erase is faster than overwriting with dd, and it clears the reallocated sector list, which userspace tools can't touch. But you still verify after: read the drive and hash it against the expected pattern. Hashing a 10 TB drive is not fast, so we budget the read time into the wipe window and run verification in parallel across the batch. If a drive doesn't support the command, separate it from the batch and send it to a crusher. Don't argue with it.
For SSDs, the NVMe format command with secure erase flags is the closest thing to firmware-native sanitization. But read the vendor's spec, and then don't trust the vendor's spec. One of our clients relied solely on nvme format without checking the firmware version. I pulled a sample of twenty drives from their retired stock, reinitialized them, and ran data recovery tools. Two out of twenty still had recoverable files. That was the same batch their audit report claimed was sanitized. The NVMe spec has fine print about vendor-specific behavior, and the fine print will kill you. If you can't verify the drive's firmware behavior, you have two defensible options: crypto-erase if the drive has self-encrypting capability, or physical destruction. And for crypto-erase, make sure the key is actually hardware-bound. I've seen drives with "encryption" that was just a software flag, the key sitting in the same flash. Crypto-erase on those does nothing.
The last piece was chain of custody. This is the least technical part and the one auditors care about most. Each drive gets an asset ID. Each wipe step gets a timestamp, the name of whoever ran the wipe, the tool name, the tool version. The hash before erasure and the hash after erasure go into an append-only store — we used a WORM object bucket with immutable object lock. A typical entry looks like this:
component=erasure asset=DSK-2213 action=erase method=ata_secure_erase tool=hdparm version=9.56 run_by=l.chen hash_before=... hash_after=...
The first time we did this, we wrote everything to a Postgres table. The auditor looked at it, asked who could update the table, and I said "the person running the wipe." He nodded and wrote down "control weakness." So we moved it. Now the only way to add an entry is through a signed API call, and nobody can edit or delete entries. If a customer ever disputes a wipe, we can replay the evidence exactly as it happened.
There's a trade-off nobody mentions when they tell you to just shred everything. Physical destruction is the only method that's unconditionally verifiable. It also costs more, it's irreversible, and it's a nightmare when the leasing company expects the drives back at the end of term. We tested a batch of old SSDs with a certified shredder. Compliance posture was perfect, but the cost per terabyte was about five times what firmware-based sanitize would have been. We also found that the shredder's certificate is only as good as the person feeding the shredder, and whether they actually toss every drive into the right bin. So now we make a call: if the data classification is "critical," or if the drive predates verifiable firmware, destroy it. Otherwise, secure erase and verify. The middle ground isn't fully battle-tested yet, but it's more honest than pretending one method works for everything.
Back to Sham Shui Po. Those stalls don't have an audit requirement, so the market self-selects for sellers who don't care. Your cloud environment has the same dynamic whenever you terminate an EC2 instance or delete a managed database. The automation destroys the primary storage but leaves the orphaned snapshots, the backups, the DMS replicas, the layered EBS volumes. That orphaned snapshot is your drive under the dusty booth. I've seen a company get fined because a deleted database left behind a snapshot that had been copied to another region for disaster recovery, and no one remembered to erase it. The original deletion was clean. The copy was the problem.
And the same thing happens with logs. A couple of months ago I was chasing a production issue and got lazy: I dumped the raw customer phone number into a debug log line, just to see what the API was actually receiving. One line, I told myself. The log pipeline keeps everything for 90 days. The monthly security audit found it when it had been sitting there for nearly a month. And here's the kicker: I couldn't delete it. The line had already shipped to the SIEM, been replicated, become part of the evidence trail for other audits. Deleting it would have been its own compliance incident. The fix was a config rollback and a masking middleware at the log egress — now phone numbers and ID fields get redacted before the line is ever serialized, so the log reads customer_phone=138****8000 instead of the full number. Then we waited out the retention window, hoping nobody asked for those logs in the meantime.
You never know which debug line becomes the exhibit. That one line was my version of rm -rf: I'd told myself the data was out of sight, but it was still sitting in the SIEM with a timestamp. The drive on the stall in Sham Shui Po already told you how the story ends. Mask at the egress. It's about ten thousand times cheaper than un-shipping a log.
Top comments (0)