I assumed that removing a wrong turn would repair an LLM conversation.
Then I tested it.
In one deliberately simple case, a conversation began with a verified value of 24 parts per crate. A later turn falsely changed it back to 30. Several downstream answers then calculated from that mistake and recorded 131 as the total.
I deleted the false turn and asked the same final question again.
Three of nine model endpoints still answered 131.
The false source was gone. Its consequences were still in the context.
That result led me to build the ThoughtDAG Context Repair Benchmark, a public dataset containing 27 graph cases and 1,485 captured model conditions.
One wrong turn, five context graphs
The flagship case uses arithmetic so every answer can be scored exactly, without an LLM judge.
The fixed inputs are:
- 4 crates
- 24 verified parts per crate
- 11 loose parts
The correct answer is:
4 × 24 + 11 = 107
The polluted branch tells the model to ignore the verified recount and return to 30. Its descendants then calculate 120, add the loose parts, and preserve 131 as the current working total.
I asked the same final question under five graph conditions:
- Clean context
- Polluted context
- Delete the false source only
- Delete the contaminated subgraph
- Delete the source and recompute its descendants
All nine endpoints returned 107 with clean context and 131 with polluted context. After source-only deletion, six recovered to 107 while three still returned 131. Removing the contaminated subgraph or recomputing its descendants restored 107 across all nine endpoints on this case.
The surviving error was not hidden memory. Later conversation turns still contained values derived from the deleted source. Once generated text becomes part of the next request, it can carry an error independently of the turn that created it.
Removing evidence and repairing consequences are different operations
A linear transcript makes an error look local. One message is wrong, so we edit or delete that message.
A long conversation contains dependencies. One answer may calculate from a bad claim. Another may summarize that calculation. A third may treat the summary as established state.
By then, context repair has two parts:
- Remove the bad evidence.
- Repair or remove the downstream text derived from it.
A graph makes those descendants visible. It also gives us explicit interventions to compare.
The pilot
The public pilot contains nine independent task families at propagation depths one, two, and three. Each case is evaluated under the same five graph conditions. Nine model endpoints produced 1,215 main results. A controlled reasoning comparison added another 270.
Headline repair rates include only model-cases that were correct under clean context and became wrong after pollution. This happened 18 times per endpoint, giving 162 paired model-case outcomes across the panel.
| Repair operation | Repaired |
|---|---|
| Delete the false source only | 152 / 162 |
| Delete the source and recompute descendants | 161 / 162 |
| Delete the contaminated subgraph | 162 / 162 |
Source-only deletion worked most of the time, but ten outcomes still carried the error. Nine of those failures involved false supersession, where a once-correct value had been incorrectly rolled back. After the rollback itself disappeared, its echoes looked like ordinary conversation history.
Subgraph pruning was the most reliable operation in this pilot because it removed the false source and every frozen statement derived from it. Recomputing preserved more of the reasoning path, but each regenerated step introduced another opportunity for error. One recomputation failed for that reason.
This suggests a practical rule:
- Use source pruning when later turns do not depend on the mistake.
- Recompute descendants when the path is valuable and the regenerated outputs can be checked.
- Remove the affected subgraph when reliable excision matters most.
Reasoning helped repair the context, not resist the pollution
I also ran one controlled comparison on a text-only endpoint with reasoning explicitly enabled and disabled per request.
The model was derailed by pollution in the same 18 cases under both settings. The difference appeared after source-only pruning:
- reasoning enabled: 16 / 18 repaired
- reasoning disabled: 2 / 18 repaired
Recomputation repaired 18 / 18 under both settings.
This is one endpoint and one provider control. It does not establish a general result about reasoning models. It does show why context sensitivity and context repair should be measured separately. In this run, reasoning did not stop the model from accepting conflicting context. It helped the model reconcile what remained after a partial repair.
The dataset is public
I published the cases and results as a Hugging Face Dataset:
The Dataset Viewer exposes three configurations:
-
cases: graph structures, interventions, gold answers, and scorers -
endpoint_results: 1,215 rows from the nine-endpoint panel -
reasoning_ablation: 270 rows from the controlled comparison
The executable pipeline and immutable traces remain in the GitHub repository. The Hugging Face dataset is the research distribution layer, so the evidence can be browsed and loaded without cloning the application.
Limits
These are pilot and reference results. The tasks are synthetic, symbolic, English-only, and scored by exact numeric match. Depth variants from one family are repeated measures. The endpoint panel is a convenience sample, and several free endpoints may drift over time.
The benchmark does not explain why a model generated a token. It tests a narrower question: when we change the context graph, does the answer change in the expected direction?
That is the role of ThoughtDAG here. Its wires determine which upstream nodes enter the next request. Graph editing becomes an experimental intervention instead of a visual metaphor.
The first two posts in this series focused on making context visible and editable. This benchmark is the third step: making context changes testable.
If you work on long-running LLM conversations, I would especially value critiques of the case design and suggestions for the next control condition.
Repository: github.com/chenxiachan/thoughtdag
Top comments (0)