AI Explainability and Hallucination

#ai #opensource #programming #machinelearning

Two names for the same root cause

"Hallucination" and "explainability" get treated as separate problems in most AI discourse — one is about models generating false facts, the other is about models failing to justify their outputs. They're not separate. They're the same failure, pointed in two different directions.

Hallucination is a model generating false information about the world with the same fluency and confidence it uses for true information.

Unfaithful self-explanation is a model generating a false account of its own reasoning process with that same fluency and confidence.

Both come from the same structural fact: a language model has no privileged, verified channel to ground truth — not about the external world, and not about its own internal computation. It produces plausible tokens either way. Whether those tokens happen to be accurate is not something the generation process itself can guarantee, in either direction.

The self-explanation case, specifically

This gets less attention than classic hallucination, so it's worth being precise about it. When you ask a model "why did you say that" or "where did that come from," you are not querying a log. You are prompting the model to generate more text — text about its own prior generation, produced by the same mechanism, with no more privileged access to the truth of the matter than the original output had.

Anthropic's own research on chain-of-thought faithfulness (Lanham et al., 2023) found exactly this: models frequently produce reasoning traces that read as coherent justifications but don't reliably reflect the computation that actually produced the answer. The explanation is a second output, not a window into the first.

A concrete example: an AI system with live web search was asked to justify a suggestion, and answered that it had "mapped the knowledge graph" behind the user's public work. The cited sources turned out to be real — independently verified. The claim about how it arrived at them was not verifiable at all. It was generated text, describing a process, produced by the same system whose process was in question. That's not a lie, necessarily. It's structurally unfalsifiable either way, which is close to the same problem.

Why "just ask it to be more careful" doesn't fix this

Prompting a model to "only state what you're confident about" or "explain your reasoning honestly" doesn't change the mechanism generating the explanation — it changes the training-time incentive shaping the surface form of the output. Both the honest-sounding explanation and the confabulated one come from the same sampling process. Confidence and fluency are not correlated with accuracy in either the factual-claim case or the self-report case, and no amount of prompting reaches into the model and adds a verified channel that wasn't architecturally there.

What actually resolves it — and where that resolution stops applying

There's a real, working answer to this — but it's narrower in scope than "fixing hallucination" in general, and the scope boundary matters.

.me, a declarative semantic kernel, exposes explain(path). For any derived value, it doesn't generate a description of how that value was computed. It returns the literal expression that was evaluated and the literal input values that were read to evaluate it:

me.explain("robots.surgeon.canProceed") →
{
value: true,
expr: "canLift && softGripReady && !needsHumanReview && contextAllowsMotion",
derivation: {
inputs: [
{ path: "robots.surgeon.canLift", value: true },
{ path: "robots.surgeon.softGripReady", value: true },
// ...
]
}
}

This works because .me computes over an explicit, deterministic dependency graph — every derived value has a tracked expression and tracked inputs, by construction. There is no generation step between "what happened" and "what gets reported" — the explanation is the computation's own record, not a description produced after the fact. That's why it's checkable: recompute the expression over the disclosed inputs, and it either matches or it doesn't.

This does not generalize directly to open-ended LLM generation. A free-form language model isn't evaluating a fixed expression over tracked inputs — it's sampling from a learned distribution over token sequences, with no equivalent "expression" to disclose. .me's explain() doesn't make GPT-4's chain-of-thought faithful. It solves the problem for a different, narrower class of system: deterministic, declarative computation with trackable dependencies.

What does transfer

The transferable part isn't the specific mechanism — it's the architectural principle underneath it: don't ask a generative process to narrate itself when verifiability matters. Architect the system so the record of what happened is produced by the computation, not described by it afterward.

For agentic LLM systems, this cashes out as concrete, applicable patterns already in partial use across the industry:

Log actual tool calls and actual returned values, not the model's summary of them.
Surface actual retrieved documents in RAG systems, not just the model's paraphrase of what it found.
Treat a model's self-report about its own reasoning as a hypothesis to check against an external log, never as the log itself.

None of that makes a language model's internals inspectable the way .me's dependency graph is. But it moves the ground-truth record outside the part of the system that was never going to be able to describe itself reliably — which is the actual lesson, independent of whether the underlying computation is a semantic kernel or a transformer.