Vector RAG can't fix long-context state tracking (33 runs, zero variance)

he fangsheng — Wed, 05 Aug 2026 06:56:54 +0000

There are two different things a model forgets over a long document, and conflating them is why the problem never gets solved.

Forgetting the text it wrote. RAG fixes this. The text is in the index; retrieve it.

Forgetting derived state. RAG cannot fix this. After a key changes hands five times across fifty chapters, "who holds it now" is in no passage. Retrieve "black key" and you get five mutually contradictory excerpts, none of which is the current value — because the current value was never written down anywhere.

The second one is what actually breaks long-form work, and it is structurally not a retrieval problem.

The benchmark

Ground truth by construction — the world spec comes first, prose is generated from it, so nothing is hand-annotated. 95k-character baseline, the model continues five chapters and writes state back as explicit fields. Graded field by field.

Arm (qwen-plus, 11 runs each)	State accuracy	sd
bare model	75.0%	0
vector RAG	75.0%	0
state machine	98.9%	—

The zero is the interesting part. Across 33 runs at two baseline lengths, neither control arm ever moved off 75.0%. Same two fields wrong every time. That is a wall, not a gradient.

And stretching the baseline from 20k to 95k characters made the gap wider (92.5% → 98.9%) — RAG is precisely the technique that should improve with more context.

How it works

The model never emits a finished artifact. It emits a semantic transaction:

{
  "operation": "append_scene",
  "state_changes": [
    { "object": "obj:black-key", "field": "holder",
      "from": "char:lin-zheng", "to": "char:shen-yan" }
  ],
  "assertions": ["zhao-qi-alive"]
}

A deterministic compiler validates it and bounces failures back with evidence. Constraints are data, not code — six general predicates (equals, not_equals, contains, not_contains, range, unchanged), which is why the same compiler drives a narrative world and a sales dataset unmodified.

Trade-offs

Prose consistency is statistically indistinguishable from RAG (p=0.99, p=0.34). It costs 25% more tokens. Cross-model validation is still running — I pulled the single-run numbers off the README after ten runs revealed the control arms are bimodal on that model.

node compiler/selftest.mjs   # 27 checks across two domains, no deps, no build

https://github.com/dongsheng123132/2origin

Four Windows bugs that only exist on your customers’ machines

he fangsheng — Wed, 05 Aug 2026 06:52:43 +0000

I ship a Windows installer that sets up Claude Code / Codex CLI / ClawX / Hermes for non-technical users in China. Over ~14 months of real customer machines I collected a pile of failure modes that basically never show up on a developer's laptop, and open-sourced them as a case library.

A few that cost me the most hours:

Self-update silently did nothing on any machine whose install path contained Chinese characters. The updater wrote a .bat in UTF-8; cmd.exe parses it as GBK. No error, no log, exit code useless (a self-deleting .bat always returns 1).
An app-wide crash from a one-line helper: tail() sliced a String by byte offset, hit a multi-byte character, panicked — and the release profile is panic=abort, so the whole GUI vanished with no dialog.
"The model is listed in /v1/models" is not the same as "the model answers". Two model ids that enumerated fine returned 0/4 and 503 respectively.
taskkill /IM Claude.exe, intended to close the Claude desktop app, also kills every Claude Code CLI session on the box — same image name.

Repo (MIT, no telemetry, nothing to install): https://github.com/dongsheng123132/u-rescue

The product itself is closed-source and Windows-only, so I'm not pitching it here — the cases are the useful part. Corrections welcome, especially if you've hit the GBK/UTF-8 one on a different toolchain.

DEV Community: he fangsheng

Vector RAG can't fix long-context state tracking (33 runs, zero variance)

The benchmark

How it works

Trade-offs

Four Windows bugs that only exist on your customers’ machines