We Need Fair Witnesses Again

In 1961, Robert Heinlein invented a profession. The Fair Witness. Someone trained to report only what they directly observed. No inference. No assumptions. Ask one what color a house is and they'll tell you the color of the side they can see. The other sides? Not their department.

Heinlein intended this as satire about lawyers and their tendency to make inferences beyond direct observation. It's starting to look like a job description.

Here's the problem. We're entering an era where seeing is no longer believing. A video of a CEO announcing layoffs. A voice memo from your kid asking for money. A photo of a protest. All of it can be generated in minutes, and most of it is convincing.

The obvious fallback is human testimony. Just ask someone who was there.

Except humans are terrible witnesses. Decades of research by Elizabeth Loftus and others show that memory isn't a recording. It's a reconstruction. We fill gaps. We absorb details from later conversations. We become confident about things that never happened. Eyewitness misidentification is one of the leading factors in wrongful convictions.

Here's the uncomfortable part. Memory isn't a file you open. It's a file you open, edit, and save again. Every time you recall something, the brain briefly makes that memory pliable and then stores it back. Neuroscientists call this reconsolidation. Whatever you're feeling, whatever you've heard since, whatever someone suggested in the hallway, all of these can quietly get folded into the new copy. Then that copy becomes the original. Do this enough times and you can end up vividly, confidently remembering something that drifted a long way from what happened.

This means the memories we've told most often, the ones we're surest about, may be the least reliable of all. I find that a little unsettling. I also find it explains a lot of family holidays.

So we have fake evidence on one side and unreliable memory on the other. That's not a great sandwich.

This is where the Fair Witness stops being science fiction. The skill Heinlein described isn't superhuman perception. It's discipline. Separating what you saw from what you concluded. Saying "I observed X" instead of "X happened." Being boring on purpose.

Which raises the real question. Who do you trust, and why?

For most of history, the answer was proximity. You trusted your neighbor, your doctor, the guy at the hardware store. You knew their track record. You knew where they lived.

Then we outsourced trust to institutions. Newspapers. Courts. Universities. We didn't know the reporter, but we trusted the masthead.

Now both are wobbling. Institutions have lost credibility, sometimes deservedly. And proximity doesn't scale to a feed of strangers and synthetic content.

So what's left? I'd argue trust has to shift from who to how. Not "do I like this source" but "can this source show me how it knows?" What did they observe directly? What did they infer? What are they guessing? A trustworthy witness, human or machine, is one that makes its reasoning inspectable and admits the limits of its view.

That's a higher bar than charisma or credentials. It's also the only one that holds up when anyone can fake a face.

Trust shouldn't be a feeling. It should be verifiable through an audit trail.

Deepfakes don’t attack your honesty. They attack attribution: the link between you and a piece of evidence. So the answer mostly involves rebuilding that link through means other than “the video looks real.” The uncomfortable conclusion: you can’t establish authenticity at the level of individual artifacts anymore. You establish it at the level of a person embedded in relationships, channels, and a history of telling the truth when it cost something. Which is roughly how reputation worked for most of human history. The century or so when a photograph settled arguments may turn out to be the anomaly.

That's why I think the Fair Witness discipline is about to become valuable. Maybe even a credential. People and systems whose primary value is a verified chain from observation to report. Journalists who show their work. Provenance standards that track where an image came from and what touched it. And yes, AI agents built to cite what they actually retrieved instead of what sounds right. (I've spent a lot of time on that last one. It's harder than it sounds. So is getting humans to do it.)

Here's the twist. AI agents can suffer from reconsolidation too.

Many agent memory systems retrieve a memory, use it, and then have a model rewrite it before saving. That rewrite gets colored by the current conversation and whatever the model inferred along the way. Summarize the history, then summarize the summary, and after a few rounds you get fluent, confident memory that has drifted from what was actually said. Reconsolidation with a token budget.

And hedges are usually the first casualty: qualifications and uncertainties get deprioritized during summarization. "The user was unsure about X" quietly becomes "The user prefers X."

The difference is that agents don't have to work this way. Humans can't preserve the original memory unchanged. Agents can.

Keep an immutable record of the original observation and treat summaries as views, not replacements. Tag what was stated versus what was inferred. Link every memory back to its evidence. Regenerate from the source, never from the last rewrite. Those aren't exotic techniques. They're Fair Witness rules, written for machines.

The irony is nice. The technology that revealed how unreliable our trust in evidence actually is might force us to get serious about what witnessing actually means.

Humans can't stop reconsolidating. But we can build witnesses that don't. Heinlein's Fair Witnesses wore white robes so everyone knew when they were on duty.

We may need something like that again. Probably with better metadata.