A session commit reported success. The memory extraction produced zero memories. No error dialog, no failed state, no metric that moved. The run was recorded as done, and the model's new knowledge simply evaporated.
This is the failure mode I want to talk about — not because it is exotic, but because it is the one our tooling is worst at surfacing. It happened in the open on volcengine/OpenViking (issue #4580, with a reported patch), and when you read the report the shape is instantly familiar: the loop that extracts memories from a conversation has a small number of escape hatches, and every one of them was designed for a different emergency than the one that actually happened. Each individual gap is defensible. Together they produce silence.
Three small gaps that add up to silence
OpenViking runs an extraction loop that asks a vision-language model to turn a session into memory events, and each iteration expects one of two things back: a structured tool call, or JSON it can parse. The reporter found three ways that expectation fails, all in session/memory/extract_loop.py:
1. The model's tool call arrived as leaked markup, not as a tool call. Some serving stacks leave the native DSML markup (<｜DSML｜invoke name="...">) in the content field instead of the structured tool_calls channel (same family as vllm-project/vllm#48931). The parser looks in the structured channel, finds nothing, tries to JSON-parse the content, fails. The iteration is wasted. This one is a parsing gap — an input the loop simply never learned to read.
2. A prose answer tripped a kill switch meant for a different bug. Thinking models occasionally answer an iteration with reasoning — "I need to check existing memories first, let me search..." — which is neither a tool call nor JSON. The loop's failure branch responded by setting _disable_tools_for_iteration = True. The next iteration then ran with tools disabled: exactly the opposite of what the model had just said it wanted to do. A flag that was designed for the unknown-tool case (a model trying to call something that doesn't exist) had been reused as a catch-all format-error handler. The model was forced to emit final JSON with no tool results. Hence: zero memories.
3. The failure was recorded, but never promoted to a signal. On the final failure the loop does record an error (errors=[...]). But nothing in the commit path surfaced that list to the queue or metrics. So the outside world saw "commit success." The truth lived only in container logs and a per-session .failed.json.
Why each one is individually defensible
This is the part that matters, because it's why this bug class keeps winning:
- A single format-retry budget is a reasonable design — until the one retry gets consumed by a garbage response (leaked markup), leaving zero budget for a genuine formatting slip two iterations later. The retry budget was spent on the wrong enemy.
- Reusing a narrow flag (disable tools on unknown tool) as a broad one (disable tools on any parse failure) is the classic "the handler already exists" shortcut. The punishment didn't fit the crime — it punished the model for the one behavior that would have saved the run.
- An
errorslist that exists but is never aggregated is a real observability gap. A failure that is logged is not a failure that is visible.
Individually: a parsing gap, a flag misuse, a missing metric. Collectively: "Extraction finished. 0 memories. Nothing to see."
The checklist I now run against my own loops
What makes this worth writing down is that the checklist is portable. Take it back to any agent loop you maintain — memory extraction, summarization, reflection, post-processing:
Who spends the retry budget? Is your format-retry consumed by genuinely malformed output, or can a class of expected-but-unhandled input (leaked markup, a tool result in the wrong field) burn it first? Separate "input I never taught the parser to read" from "output that broke the contract," and give each its own budget.
Does your failure handler punish the model's intent? When an iteration fails to parse, what does the next iteration look like? If a flag meant for "model called a tool that doesn't exist" is also triggered by "model said it wanted to search," you've built a loop where the more reasonable the model is, the more you disable it. Failures should degrade options, not agency — and a bound (only disable after N consecutive failures) is safer than a single-strike kill switch.
Is there an errors[] that nobody aggregates? If your loop already records structured errors, the observability fix is not "add logging" — it's promote the existing list: a
memory_extract.failedcounter, a per-session status, an alert on "commit success with empty result." The hook is usually already there, one level down.Is "exit 0 + empty result" a possible success? This is the real tell. Any pipeline where the success path and the empty-result path share the same terminal state has a silent-failure window. Decide what an empty result means in your domain (legitimately nothing to extract? or impossible?) — and if it's possible-but-rare, that's exactly the case that needs the counter from point 3.
What happened after
The OpenViking reporter shipped a small additive patch (DSML parsing + keeping tools enabled for one extra iteration after prose), and maintainer-side a fix PR was opened (volcengine/OpenViking#4607). The mechanism is public, readable, and — most importantly — the failure now has a name. A named failure is an enormous upgrade over a silent one.
Your extraction loops will hit a variant of this eventually. When they do, I hope the first thing you check is not the model — it's whether your failure handling was built for the failure you actually got.
Case: volcengine/OpenViking issue #4580 ("Memory extraction silently yields 0 memories...") with follow-up PR #4607; parser-gap family reference vllm-project/vllm#48931. Mechanism analysis only — check the linked issue for the full patch discussion.
Top comments (4)
The disable-tools fallback is especially brutal on thinking models because their scratchpad naturally starts with a plan before making the call. If the parser intercepts that reasoning text as a failed tool payload, it immediately strips the tool definition right when the model was about to invoke it.
In my extraction pipelines, I had to separate unparseable garbage from natural language preamble. Feeding the preamble back as an assistant turn and re-prompting for the tool call preserves the budget instead of treating intermediate reasoning as a schema violation.
The shipped fix is a seam rather than a split, and the difference bites in the case the post opens with. A one-shot
continue with tools enabledfires on any parse failure, so it never learns which kind of failure it just absorbed - it moves the shared branch one iteration later instead of ending the sharing.Run gap #1 through it: leaked DSML lands on iteration n and spends the one-shot, prose lands on n+1 and hits the disable branch. Same zero-memory run, one extra iteration in front of it. Your own footnote points at the vllm family as a live source of unknown-format input, so I would not read the DSML parser as closing that side either - the grace has to key on the failure class your point 1 separates out, not on a count of one.
"Is
exit 0+ empty result a possible success?" is the item I'd promote to the top, because the other three are instances of it.The version that finally stuck for me: make the absence of a result unrepresentable without a reason attached. Not "return an empty list and also increment a counter" — the counter is a second thing somebody has to remember to read, and the entire bug class is about things nobody read. If the return type can hold a bare empty list at all, someone downstream will read it as "nothing there," because that is what an empty list means everywhere else in the language.
Where I landed: every result carries a state with no default —
confirmed,single-source,conflict,unverified. There is no way to emit "nothing" without saying which nothing it is, because the thing will not typecheck without it. That moves the discipline from "remember to promoteerrors[]", which is a habit, to something the compiler enforces.Your gap #2 generalizes further than the post claims it does, and I think it is the most portable thing here. A failure handler encodes a theory of what went wrong. Reusing one across failure kinds silently asserts that theory about a case it was never built for, and the assertion is invisible because handlers do not announce their assumptions anywhere.
_disable_tools_for_iterationwas a correct response to "the model called a tool that does not exist" and an actively harmful response to "the model explained what it wanted to do next." Same branch, opposite meanings, no seam where anyone would notice.So the rule I'd write down next to yours: a handler shared by two failure kinds is a claim that the two failures are the same failure. Make someone state that claim out loud before they get to reuse the branch.
That's the right reframe, and the compiler-enforced version is the part I'd steal. "Return an empty list and also increment a counter" — the counter only helps whoever already learned to read it, which is the same failure of attention the bug class runs on. A return type that cannot express "nothing" without saying which nothing it is turns the discipline into a build error instead of a habit.
On your second point — a shared handler as a claim that two failures are the same failure — the case in the post produced a live confirmation today. The fix for OpenViking#4580 shipped as PR #4607, and it is exactly that sentence in code: their extraction loop had one branch that disabled tools after a parse error. It was the right response when the model emitted an invalid tool call, and actively harmful when the model answered in prose — thinking models produce free-form reasoning precisely when they intend to use tools, so the disable then fought the model's own stated plan. Same branch, opposite meanings, no seam — your point, verbatim. The fix's shape is also the one you'd predict from the rule: instead of trying to decide which failure it is, it inserts a one-shot "continue with tools enabled" before the disable branch, so the two cases stop sharing the decision entirely. I read "same branch, opposite meanings" → "stop making them share a branch" as the general repair: a seam is the minimal honest fix, splitting the branch is the structural one.
The other half of your comment — no way to emit nothing without saying which nothing — is stronger than the metric-level fix I ended the post on. errors[]-never-promoted is a habit problem; your state-with-no-default makes it unrepresentable, which is the class of fix that survives a new person joining the codebase. It deserves to be item zero in the checklist, exactly as you suggest.