A chained PHP2Shell exploit reportedly rooted 25 servers this week, and the detail that should stop you is not the count. It is that at least one of them belonged to a government. Think about the first hour of that incident from the inside: somebody has to answer, in writing, what happened, in what order, and what the attacker touched. And every artifact that would answer those questions — access logs, shell history, the web server's own records, the deployment history — was sitting on the machines that were rooted.
The attacker did not need to delete the evidence. The evidence was stored inside the blast radius. That is the ordinary condition of almost every company, and it is not a backup problem. Backups protect against loss. This is a different question: after the incident, is there any copy of the record that the incident itself could not reach?
Two sentences that sound the same
"We have the logs." "Someone else has the logs."
Those look like the same claim with extra words. They are different categories of claim.
The first one is a statement about your own storage. It is true until it is inconvenient: until the disk is encrypted by ransomware, until the retention window rolls over, until the bucket policy changes, until the person who knew the query syntax leaves, until the account is suspended, until legal tells you to stop touching it. It is also unverifiable by anyone who does not already trust you, which is exactly the set of people who will be asking.
The second one is a statement about the world. It says: there exists an entity, outside my control, with its own reasons to keep this, who can produce it without my cooperation. That sentence survives your account being deleted. It survives your company being sold. It survives you being wrong about what happened.
The distance between the two sentences is roughly one subpoena: the moment when someone with authority asks a third party for the copy, and you find out whether your record was a record or a habit.
What a witness actually witnesses
A witness does not attest that you are honest. It attests that a specific string existed at a specific time, and that both of you can prove it later without trusting each other. That is a much smaller claim, and it is the only one that holds up.
Three kinds of witnesses are worth knowing, in ascending order of how much they cost you.
Protocol witnesses. Certificate Transparency logs, transparency logs like Rekor, public blockchains. You submit a hash, the log returns an inclusion proof, and the log's own append-only structure makes retroactive edits detectable by strangers. The cryptography is nice; the important part is that the log is not yours. A dozen independent parties mirror it. Note the limit: a transparency log proves a hash was logged at a time. It does not prove the artifact was any good, or that the thing you logged is the thing you shipped. Witnesses testify to existence, not to quality — and every incident review eventually discovers this distinction the hard way.
Commercial counterparties. The most underrated category, because it is free and you are already doing it. The email you sent a customer describing what you changed. The invoice. The shipping scan. The bank statement. The support ticket they opened, which lives in their helpdesk under their retention policy. Each of these is a copy of a fact about you, held by somebody whose incentives are not yours: a customer keeps the email because they might need it, a bank keeps the statement because regulators require it, a carrier keeps the scan because they get charged for disputes.
This is why "the most durable backup in your company is probably an email you sent to a customer" is not a joke. That copy is on their vendor's servers, governed by their retention policy, indexed by their search, and — critically — not deletable by you. When the question is "did you tell anyone before you knew," a customer's inbox answers it, and your own Sent folder does not, because your Sent folder is a file you can edit.
Institutions. Notaries, auditors, regulators, customs stamps, court filings. Slow, expensive, occasionally bureaucratic to the point of comedy — and that is the product. An institution's whole value is that it has no incentive to help you look good, and no ability to quietly fix your record without leaving a trace.
Witnessing is a transaction
Nobody tells you this part: a witness is not a service you consume, it is a trade you make.
To get a copy into someone else's hands you have to give them something — a document, a notification, a signature, a record of your own mistake. The price is usually not money. It is exposure. Which is why companies that badly need witnesses skip them, and describe the skipping as a security decision.
There is a second cost that people discover later: a witness also holds the gap. If you told a customer on the 3rd that the migration was complete, and your own records show it finished on the 9th, the customer's copy of that email is now evidence of the gap. The witness is neutral. It will testify for you or against you with the same enthusiasm.
So the honest version of the trade is this: witnesses make you provable, and being provable means being wrong in public with a timestamp. Most organizations would rather be unprovable. That preference is rarely written down anywhere, and it is the single most common reason post-incident timelines are reconstructed from memory.
The witness test
Four questions. They take about ten minutes and they will make you uncomfortable, which is the point.
- Who, outside this company, holds a copy of this claim? Not "who could be told." Who already has it.
- Do they have a reason to keep it that has nothing to do with me? A customer's own records count. A favor does not.
- Can they produce it without my cooperation? If the answer requires your admin to export something, it is your record wearing a disguise.
- Would their copy survive my account, my vendor, or my company disappearing? The typical answer is: the copy survives, the ability to find it does not. Put the location of the receipt in the receipt's metadata, not in someone's head.
A single "no" is not fatal. Four "no"s means you do not have a record. You have a memory with a filename.
What to do with this on a Friday afternoon
Pick the five claims you would have to prove after a bad week. Not five policies — five facts:
- The credential was rotated before the breach window.
- The customer was notified before the story broke.
- The patch was deployed to production, not just merged.
- The data left the region on a date we can name.
- The build that shipped is the build we tested.
For each one, write down the external holder of the proof. If the column is empty, you have found a claim that only you can vouch for, which is another way of saying you have found a claim you cannot make.
Then do the cheapest fix available: forward the receipt to a counterparty who has their own reason to keep it, and write one line in the ticket saying where the copy now lives. It is unglamorous, it takes an afternoon, and it is the difference between an incident review and a deposition.
The uncomfortable part
Witnesses are a constraint, not a feature. Once a copy is outside your walls, you cannot revise the story — not because anyone is watching, but because the record is now in more than one place and the places do not agree with each other by default. That is what makes it worth something.
Every company says it wants accountability right up to the moment accountability has a paper trail. The ones that actually have one tend to be the ones that were forced into it: a regulated industry, a litigious customer, a partner who demanded evidence before signing. Nobody volunteers for a witness. Which is why the practical advice is not "be braver." It is: pick the one claim that would hurt most to be unable to prove, and put a copy of it in someone else's hands this month. Just one. The habit generalizes; the intention does not.
A record you control is a draft. A record someone else controls is evidence. Everything in this series — the exposure snapshot, the trust-anchor inventory, the drill card, the drift layer, the timestamp — assumes that at some point the two will be compared. The witness layer is where that comparison becomes possible without asking anyone to take your word for it.
Next: the retention layer. A witness that keeps the copy for thirty days is a witness for thirty days — and most of the "permanent" records you are counting on have a quiet expiry date written by someone in procurement.
If you are sitting on a similar deadline right now — a log retention window that quietly expires in 30 days, a revocation nobody confirmed ever propagated, a claim you could only prove with your own copy — say so in the comments. We keep a one-page worksheet for exactly that situation (what to prove, when it happened, who else holds the copy), and I will send it over. No form, no email list.
Top comments (0)