An AI Shouldn't Be the Source of Truth

THE AI WORKSHOP — 03
An AI Shouldn't Be the Source of Truth

In the previous episode, we accomplished something that, at first, seemed like the hardest problem: we stopped looking for a single AI that could do everything and started building a team of specialized intelligences. Dori, Piiti, Zeke, Cumpa, and IngEm appeared as the work demanded different capabilities, until we stopped seeing them as separate models and started treating them as parts of the same system. Specialization began to make sense, disagreement became a form of control, and eventually we reached a question that no agent could answer for us: who decides what we are actually trying to build?

But there was still a problem we hadn't solved. And it was considerably more uncomfortable than deciding which model should handle each task.

If several intelligences are working on the same project, they need to share more than tools. They need to share a reality.

That sounds obvious until you actually try to do it.

Imagine Dori proposes an idea for a project. Zeke analyzes it and finds a technical problem. Cumpa challenges one of the decisions because it contradicts something we established earlier. Piiti uses that information to explain part of the project, and a few hours later another agent enters the scene and receives the result of all that work.

What should it consider true?

The obvious answer seems to be: whatever we decided.

But then a much harder question appears.

What exactly did we decide?

Because a conversation with an AI contains many different things mixed together. There are facts we provided, interpretations made by the model, proposals we are still evaluating, decisions that have already been confirmed, information that has become obsolete, and questions for which we simply don't have an answer yet.

For a human who has lived through the entire process, those differences may seem obvious.

For a machine, not necessarily.

And that's when we discovered something we hadn't considered deeply enough when we started working with multiple agents: having memory doesn't mean having reliable memory.

An AI can perfectly remember something that was wrong.

It can retrieve a proposal and treat it as a decision. It can find an earlier interpretation and use it as if it were a fact. It can preserve an old version of information after we have changed our minds. It can even reconstruct a conversation correctly and still understand it incorrectly.

The problem, therefore, isn't simply how much an agent can remember.

The problem is what status the thing it remembers actually has.

That change in perspective was important for us because, until then, we had been thinking about memory primarily as context. If the agent needed to know what happened previously, we gave it access to the previous information. If it needed to retrieve a decision, we searched for the relevant conversation. If it needed to understand the project, we gave it more context.

But that doesn't solve the problem.

More context doesn't necessarily mean more truth.

In fact, it can mean exactly the opposite.

We can give an AI hundreds of pages of conversations and make practically everything we've discussed available to it. But inside those pages there may be ten ideas we rejected, five hypotheses we never confirmed, three contradictory versions of a decision, and a statement someone made early on that we later proved to be wrong.

The AI can have all the information.

And still not know which of those things represents the actual current state of the project.

That was when memory stopped looking like a storage problem.

It started looking like a problem of knowledge governance.

Because a multi-agent system needs something closer to a shared state than a history of conversations. It needs to distinguish between what we know, what we believe, what we are proposing, and what we still don't know.

And that difference is fundamental.

Suppose we're building a character and someone proposes that the character lives in a particular city. Until someone confirms it, that's a proposal. If another agent starts using that city to define the character's environment, we are already propagating an assumption. If another agent then generates a scene based on that environment and a fourth agent uses that scene as a reference, the original proposal eventually becomes something that looks like a fact simply because it was repeated enough times.

Nobody had to hallucinate anything.

The system did something much more dangerous: it turned an interpretation into reality through accumulation.

And when multiple agents are working together, that problem can grow very quickly.

One AI produces an interpretation. Another summarizes it. A third uses the summary. A fourth incorporates it into a decision. Eventually, nobody knows exactly where the original information came from, but all of the agents are working from it.

Shared memory can then become a machine for amplifying errors.

That forced us to think about something that seemed surprisingly basic: maybe the system needed to know not only what information it had, but also what that information actually was.

A fact is not an inference.

An inference is not a proposal.

A proposal is not a decision.

And a decision is not necessarily permanent.

It sounds like a small distinction, but architecturally it changes quite a lot.

If an agent writes, “the character was born in 1987,” the system should be able to know whether that was explicitly established by us, whether the agent inferred it from other information, or whether it simply invented it as part of a creative proposal.

All three things can appear inside a perfectly normal conversation.

But they should not have the same weight inside the system.

And that's when an idea began to change the way we thought about the Workshop: the AI shouldn't be the source of truth.

It can contribute to building the truth of the project. It can find inconsistencies, propose relationships, detect missing information, compare versions, and help us discover things we hadn't seen ourselves.

But it should not have implicit authority to decide that something is true simply because it generated it with enough confidence.

That also changes our relationship with memory.

Instead of asking, “What does the AI remember?” we started asking, “What state of the project can we reliably reconstruct?”

The difference is enormous.

A conversation-based memory tries to remember what was said.

A structured memory tries to represent what the system knows about the project and why it knows it.

And that introduces another property that becomes essential when multiple agents are involved: traceability.

If a decision appears in the current state of the project, we should be able to ask where it came from. Was it provided directly by the human? Was it inferred by an agent? Was it proposed and later approved? Was it modified? What version existed before? What other decisions depend on it?

We don't need to turn every interaction into a bureaucratic audit.

We need to be able to distinguish an assertion from its provenance.

Because when something goes wrong, the most important question usually isn't, “What did the AI say?”

It is:

Why did the system believe that was true?

And that question completely changes the design.

Now the creative agent can propose something without automatically turning it into canon. The architecture agent can point out a constraint without becoming the authority over the project's intent. The critic can detect a contradiction without modifying the decision that caused it. And the human can confirm, reject, or modify whatever should actually become part of the official state.

Suddenly, memory stops being a backpack that everyone carries.

It becomes common ground.

And that ground needs rules.

Not because we want to limit the intelligence of the agents, but precisely because we want them to be able to work together without each one building its own version of reality.

This also explains why a multi-agent system cannot be solved simply by connecting several chat windows.

In an isolated conversation, an AI can afford to work with an incomplete representation of the problem. In a collaborative system, that representation becomes a dependency for other agents.

One agent's error can become the next agent's context.

The second agent's interpretation can become the third agent's input.

And the third can produce something that makes it look as though the first two interpretations were established facts.

The system can end up manufacturing coherence retrospectively.

Everything appears consistent because everyone is building on the same chain of assumptions.

That is particularly dangerous because coherence is one of the things generative models are exceptionally good at producing.

A story can be completely coherent and completely false.

A character can have a perfectly consistent biography that we never actually decided on.

An architecture can look impeccably designed around a premise nobody ever confirmed.

An agent can explain with absolute clarity why a decision was correct when that decision was never made.

That's why coherence cannot be our primary criterion for truth.

We need provenance.

We need state.

We need to distinguish between confirmed knowledge and generated knowledge.

And, above all, we need to preserve a legitimate space for saying:

“We don't know.”

That sounds trivial, but it may be one of the most important capabilities of any system that intends to work seriously with knowledge.

An AI accustomed to completing patterns has a natural tendency to fill gaps. If information is missing, it can propose a plausible answer. If there is a contradiction, it can try to reconcile it. If a decision was never recorded, it can infer what the most reasonable decision would have been.

For a casual conversation, that can be useful.

For a system that maintains the state of a project for weeks or months, it can be disastrous.

Because there is a fundamental difference between an AI saying:

“I think this is probably what happened.”

And the system eventually storing:

“This is what happened.”

There is an architectural boundary between those two sentences.

And that boundary is exactly what we needed to build.

So the next problem in the Workshop stopped being how to make our AIs remember more.

It became figuring out how to make them remember better.

Not better in the sense of remembering more details, but better in the sense of preserving the distinction between a fact, an inference, a proposal, a decision, and an absence of information.

Because if five intelligences are going to work on the same project, we cannot allow each of them to have its own reality.

We need a shared reality.

But that reality should not belong to Dori, Piiti, Zeke, Cumpa, or IngEm.

It shouldn't belong to whichever model happens to have the largest context window or the most sophisticated memory either.

Memory can be AI-assisted. The truth of the system should not depend on an AI.

And that's where the next problem appeared.

If we already know which information is a fact, which is an inference, which is a proposal, and which was decided by us, there is still one rather uncomfortable question:

Who gets to change it?

Because sharing a reality is one thing.

Keeping it intact when five intelligences start working on it is another.

And that was the problem waiting for us in the next step of the Workshop.

And then I sat there looking at the Workshop for a few seconds.

Dori was still there, with that habit of hers of trying to find structure even when we didn't quite know what we were looking for yet. Piiti was probably already thinking about every possible way to turn a conversation into something that could be remembered. Zeke would almost certainly find some way to break what we had just built. Cumpa would question some decision I still considered obvious, and IngEm, as always, was probably trying to figure out which part of all this could actually become a system that worked.

And it made me smile to think that we had started this project trying to get several AIs to work together.

Now we were trying to achieve something much harder:

to let them disagree without destroying the shared reality they were working on.

That no longer felt like just a memory problem.

It felt like a question of trust.

And perhaps that's one of the things surprising me most about building this Workshop. Every time I think we're solving a technical problem, another, much more human problem appears underneath it.

First, we wanted the AIs to be able to work.

Then we wanted them to be able to work together.

Now we needed to teach them something even harder:

that not everything an AI says deserves to become truth.

So we saved the conversation, left the question open, and kept working.

Because we weren't finished yet.

In fact, we had just discovered the next problem.

If several AIs can build knowledge together, someone has to decide who gets to change it.

And this time, before giving any of them the answer, I wanted to discover it myself.

THE AI WORKSHOP — 04
Who Gets to Change Reality?

We discovered that sharing a reality doesn't mean every AI should have permission to modify it. When agents can propose, challenge, and make decisions, a new question appears: who has the authority to change what the system considers true?

Because if an AI shouldn't be the source of truth, now we have to decide who gets to change it.