There are two different things a model forgets over a long document, and conflating them is why the problem never gets solved.
Forgetting the text it wrote. RAG fixes this. The text is in the index; retrieve it.
Forgetting derived state. RAG cannot fix this. After a key changes hands five times across fifty chapters, "who holds it now" is in no passage. Retrieve "black key" and you get five mutually contradictory excerpts, none of which is the current value — because the current value was never written down anywhere.
The second one is what actually breaks long-form work, and it is structurally not a retrieval problem.
The benchmark
Ground truth by construction — the world spec comes first, prose is generated from it, so nothing is hand-annotated. 95k-character baseline, the model continues five chapters and writes state back as explicit fields. Graded field by field.
| Arm (qwen-plus, 11 runs each) | State accuracy | sd |
|---|---|---|
| bare model | 75.0% | 0 |
| vector RAG | 75.0% | 0 |
| state machine | 98.9% | — |
The zero is the interesting part. Across 33 runs at two baseline lengths, neither control arm ever moved off 75.0%. Same two fields wrong every time. That is a wall, not a gradient.
And stretching the baseline from 20k to 95k characters made the gap wider (92.5% → 98.9%) — RAG is precisely the technique that should improve with more context.
How it works
The model never emits a finished artifact. It emits a semantic transaction:
{
"operation": "append_scene",
"state_changes": [
{ "object": "obj:black-key", "field": "holder",
"from": "char:lin-zheng", "to": "char:shen-yan" }
],
"assertions": ["zhao-qi-alive"]
}
A deterministic compiler validates it and bounces failures back with evidence. Constraints are data, not code — six general predicates (equals, not_equals, contains, not_contains, range, unchanged), which is why the same compiler drives a narrative world and a sales dataset unmodified.
Trade-offs
Prose consistency is statistically indistinguishable from RAG (p=0.99, p=0.34). It costs 25% more tokens. Cross-model validation is still running — I pulled the single-run numbers off the README after ten runs revealed the control arms are bimodal on that model.
node compiler/selftest.mjs # 27 checks across two domains, no deps, no build
Top comments (0)