A question that used to be answered correctly stopped being answered.
The question was about a returns policy: how long do customers have to return an item? Last week the pipeline said ninety days. This week it said "not stated in the retrieved evidence." Nobody touched the prompt. Nobody changed the model. The documents were the same.
What did change was a library upgrade in the chunker, from version 3.4 to 3.5, with three default settings different. That is the kind of change that ships every week without anyone thinking about it.
The part that matters is this. The passage with the answer was still in the index, with the same digest. It had moved to position nine out of ten, and the search never reached it. Nothing failed and nothing logged an error. The answer simply changed.
If you build RAG or agent pipelines, you've had this. The hard part isn't noticing. It's finding where.
Why logs and traces don't answer it
Logs are what your system says about itself. Traces add timing and structure. Both are good at "is it slow?" and "did it throw?". Neither is good at "what is different between yesterday's run and today's?", because nobody kept the exact bytes each step read and wrote.
So the investigation becomes archaeology. You diff config files, check package versions and re-run things by hand, then try to remember what the index looked like last Tuesday. With five or six steps between the question and the answer (load, split, embed, index, retrieve, answer), any of them can be the cause, and each one can be cheap or expensive to check.
A deterministic record around a non-deterministic model
The model can stay non-deterministic. The record around it is deterministic.
The idea behind onetrace is small. Every step of the pipeline leaves a receipt as it runs:
- digests of what went in and what came out;
- the tool and version that did the work;
- the settings it ran with.
The receipts are hash-chained into a run folder on your own disk, so a record that was edited afterwards no longer checks.
Recording a step is one decorator. Recording a run is one more:
import onetrace as ot
@ot.stage("retrieve", instrument=ot.pkg("word-overlap", "onetrace",
kind="retriever", config={"top_k": 2}))
def retrieve(question):
...
@ot.stage("answer", instrument=ot.pkg("first-passage", "onetrace", kind="model-call"))
def answer(question, passages):
...
@ot.run(stages=["retrieve", "answer"])
def main():
...
Your code runs as before. ONETRACE_DISABLE=1 turns it off without touching the code.
The model doesn't have to become deterministic. That's the point: variation is allowed, every change is caught as a different digest, and the change is located.
Comparing two runs: same, same, same… first difference
With two recorded runs, the comparison is mechanical. Walk the steps in order and compare each step's recorded output:
- Same: the output digest matches.
- First difference: the first step, in order, whose output differs.
- Changed after it: still different further down.
- Reconverged: different work upstream, the same output here. This is the case evals never see, because evals only look at the final answer.
For the returns-policy incident, the comparison named split as the first difference, with the three chunker settings that changed. Everything after it was marked. Then the text diff showed exactly how the returns passage had been cut into two chunks.
Not just which step. The exact text that changed, with each side checked against its receipt before it's shown.
Does it scale? The cost is in steps, not documents
The obvious worry is that this works on a toy and falls over on real data. So we ran the same chunker change on a much bigger corpus: 47,430 public SEC filings.
The comparison doesn't read the data again. Each step's outputs were hashed while the runs were recorded, so comparing two runs means comparing one digest per step. Our pipeline had nine steps. The first difference was found in the same nine comparisons at 13 documents, at 20,000 and at 47,430. A five-step pipeline would take five. The cost grows with the number of steps, not the size of the data.
It also gave the blast radius from index digests alone, without opening a single document:
Two chunker changes on 47,430 filings, both found in the same nine comparisons, with documents changed read from index digests alone.
Two honest notes:
- The final answer was identical in both large runs, but that isn't evidence of robustness. It was a refusal both times ("not stated in the retrieved evidence"), because the retrieved evidence was useless in both. A stable refusal tells you nothing about how robust a system is.
- These large runs used an earlier research build of the same method, and the figures are read from its own output. The comparison that ships in onetrace today walks steps the same way.
A surprise: batch size alone changed what was retrieved
While testing which steps can be re-run to the same bytes, we found something we hadn't expected. With the same code, data and model, changing only the embedding batch size changed which documents were retrieved. Floating-point differences of around 10⁻⁷ were enough to reorder near-ties. Every log was green.
That's why onetrace doesn't take your word for which steps are reproducible. A step that calls a hosted model is marked as not reproducible, and the comparison treats it that way. When such a step's output differs and nothing recorded about it changed, the console looks past it and names the first later step that does have a recorded change.
The console: the same comparison, in your browser
After setup, one command opens your own runs in the browser. New runs appear on their own.
onetrace-console 0.1.1 is out. It reads your run folders and shows:
- an overview of every run and what the evidence for each one establishes;
- one page per run, with each check spelled out;
- comparisons, with the first difference marked, the settings that changed there, and the exact text that changed.
It runs on your machine, listens on 127.0.0.1 only, and makes no network calls. Every page gives the command that repeats its check, so the same comparison can gate a CI job.
pip install onetrace onetrace-console
onetrace-console demo --open # the demo's 30 runs, locally
onetrace-console runs --open # your own runs folder
Or try it without installing: demo.oneproof.dev is a read-only copy with 30 real runs from four small example pipelines. Start with Comparisons.
What it doesn't do
- It doesn't judge truth. A faithfully recorded wrong answer still verifies. It shows what ran, not whether the answer was right.
- It never says why. It shows where two runs differ and what was recorded there. The why is still your job, but a much smaller one.
- It only sees the steps you decorate, and it tells you which those are.
- "Not checked" is never "passed". Every gap is named, in the product and in the docs.
What's next: who allowed it
Coming next, target design: a policy decision, and a bound permit refused because the request changed on the way.
For agents that act, not just answer, the next question isn't only "what changed?" but "who allowed it?". The target design adds a decision point, onedoor. It checks each action an agent proposes against policy, condition by condition. A bound permit then carries that decision to the system that acts, tied to one exact request. If the request changes on the way, the recipient refuses it and nothing happens, and the refusal is recorded with the run. This isn't in the current release, and the image above is labelled as such.
Try it
- Live demo: demo.oneproof.dev
- Docs and a 15-minute get started: oneproof.dev
- Source (Apache-2.0): github.com/shamiksaharcciit-oss/onetrace-console
There are guides for plain Python, LangChain, LlamaIndex and Langflow, and a recipe your coding agent can follow to add onetrace for you.
If you debug RAG or agent pipelines, I'd like to hear what you'd need it to record that it doesn't, and where the decorators get in the way.
Originally published on Medium.
Top comments (0)