DEV Community

Sui Gn
Sui Gn

Posted on

Stop Asking AI for Post-Hoc Explanations — Building Deterministic Receipts with me.explain(path)

#ai #opensource #programming #machinelearning

The anti-pattern

If your system's audit trail is "ask the model why it did that," you don't have an audit trail. You have a second model output, generated by the same process as the first, with no architectural guarantee it reflects what actually happened.

This is easy to miss because the answer you get back is usually fluent, specific, and plausible. Anthropic's research on chain-of-thought faithfulness (Lanham et al., 2023) found that these self-reports frequently don't reflect the computation that produced the original answer — the model produces a coherent-sounding story, not a trace. Fluency and faithfulness are uncorrelated. Nothing in the generation process ties them together.

Concretely: I watched an AI system get asked "where did that come from," and answer that it had "mapped the knowledge graph" behind a user's public work. The cited sources were real — verified independently. The claim about how it found them was pure narration, generated by the same system whose process was being questioned, with no way to check it from outside.

An AI's account of its own reasoning is generated by the same untrusted process being questioned. .me already solves this — not by making the account more honest, but by returning the computation itself instead of a description of it.

The pattern: deterministic receipts

A receipt, in this sense, is data produced as a byproduct of the actual computation, not generated afterward to describe it. Three properties define one:

  1. It's a side effect, not a separate generation call. The computation itself writes it down. There's no step where a language model is asked "so, what did you just do?"
  2. It contains literal values, not descriptions. The actual expression evaluated, the actual inputs read, the actual data returned — not a paraphrase of any of those things.
  3. It's independently recomputable. Given the receipt, a third party (human or system) can redo the computation and check whether the output matches, without trusting anything the original process said about itself.

A self-report can satisfy none of these by construction. A receipt is data with a causal relationship to the computation. A self-report is text with, at best, a correlational relationship to it — and per the faithfulness research, often not even that.

A working reference implementation

.me, a declarative semantic kernel, ships exactly this pattern for derived values, via explain(path):

me.explain("robots.surgeon.canProceed")
// →
{
  value: true,
  expr: "canLift && softGripReady && !needsHumanReview && contextAllowsMotion",
  derivation: {
    inputs: [
      { path: "robots.surgeon.canLift", value: true },
      { path: "robots.surgeon.softGripReady", value: true },
      // ...
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

No generation happens here. explain() looks up the expression that was literally evaluated and the literal input values that were literally read at evaluation time, and hands them back unmodified. If the path isn't derived from anything — a plain stored value — it says so directly (derivation: null) instead of inventing a causal story to fill the gap.

It's demonstrated against real assertions, not just documented as intent — Robots_Contexts.ts has four robots resolving the same decision differently by context, and the test asserts the returned dependency list contains the true upstream paths, checked in CI. The overhead is benchmarked, not assumed: ~0.007ms added at p95 for a fully disclosed trace.

Applying the pattern without .me
You don't need this specific kernel to use the pattern. The receipt property is architectural, not tool-specific. For agentic systems built on top of an LLM:

Tool calls: log the actual arguments sent and the actual raw response received. Not the model's summary of the response — the response.

RAG: surface the actual retrieved chunks and their actual similarity scores. Not the model's paraphrase of what it found.

Multi-step agents: maintain a state log written by the orchestration code as each step executes, independent of whatever the model says it's doing. The log is the receipt. The model's narration is commentary on the log, not a substitute for it.

When you do ask the model to explain itself: treat the answer as a hypothesis to check against the receipt, never as the receipt itself.
Scope, honestly
This does not make a general-purpose LLM's free-text chain-of-thought faithful — there's no fixed expression graph for open-ended generation to disclose the way .me can for a derived value. What it does is remove the cases where you were relying on self-report when a real computational trace was available instead — which, in most production agent systems, is most of the cases that actually matter for debugging and audit.

Full essay with the underlying argument: me.explain(Why Did You Say That)

Top comments (0)