What refining one small feature taught us about the gap between how models look up tool calls
Most agent products give a model access to a search or memory tool and assume the hard part is over. The tool exists, it’s described clearly, the system prompt tells the model when to use it, and from there the assumption is that a capable model will reach for it whenever the question calls for real data instead of guessing.
That assumption is doing more load-bearing work than most teams realize, and it’s worth examining closely, because it quietly determines whether an agent feature is trustworthy or just plausible.
We found this over the last week by slowly refining a small feature until it actually held up under scrutiny: a catch-up brief in VEKTOR Slipstream that reads a user’s own stored memory and summarizes what they’ve been working on, what’s been decided, and what’s still open.
Simple in concept. The kind of feature you’d expect to be a thin wrapper around a memory search. What we learned building it properly is that the wrapper being thin is exactly the problem.
The assumption baked into most tool-calling features
A tool-calling system prompt typically reads something like “use the search tool when you need real-time or specific data, otherwise answer from your own knowledge.” That instruction is a request, not a guarantee. The model reads it, weighs it against everything else it has learned about when a tool call is worth the added latency and complexity, and makes a judgment call.
Strong, heavily RLHF’d frontier models tend to make that judgment call well, most of the time, on straightforward prompts. Smaller models, local models, and even strong models under certain phrasing pressure make it inconsistent.
That inconsistency doesn’t usually look like failure. It looks like a normal, well-formatted, confident answer that happens to contain a detail nobody actually retrieved. We watched this happen directly: a summary that read cleanly end to end, with one line describing a purchase decision that didn’t exist anywhere in the underlying memory.
Nothing about the output signaled uncertainty. It was indistinguishable in tone and formatting from the parts that were completely accurate, which is what makes this failure mode genuinely difficult to catch in normal use. A user has no visual cue telling them which sentence was grounded and which one was filled in.
Once you see it, the pattern is obvious in retrospect: asking a model to decide, on its own, whether to verify itself before answering is asking it to grade its own homework in real time, under a latency incentive to skip the check.
Tool-calling diagram
Refining the feature meant moving the decision out of the model’s hands
The instinct when this shows up is to improve the prompt. Say it more forcefully. Add “you must call the search tool before answering.” That helps a little with models that are already reasonably good at following instructions and does close to nothing for models that aren’t, because the underlying problem was never about phrasing. It was about where the decision lived.
The actual fix was architectural rather than linguistic: run the retrieval before the model is ever called, every time, as a deterministic step in the code rather than an optional step in the prompt.
That meant building a dedicated path for this specific feature that always executes a fixed set of memory queries first, covering the shape of what a catch-up brief actually needs: current focus, recent decisions, open questions, and recently stored notes. Not one broad query left open to interpretation. Four targeted ones, run every time, regardless of which model is about to generate the summary.
Those results get merged and ranked using the same retrieval infrastructure the tool-calling path already relied on (keyword search fused with semantic search), so there’s no quality loss compared to what a well-behaved tool call would have produced. Everything gets assembled into a single context block, and the model’s task changes shape entirely. It’s no longer “answer this question, and optionally look something up first.” It’s “summarize this specific evidence, in this specific structure, using nothing else.”
Two more pieces made the difference stick. First, every model now receives the same fixed section template, so the output’s structure stops varying by provider. That alone removed a surprising amount of inconsistency, since even two well-grounded models will organize the same information differently if left to choose their own format.
Second, we added an explicit instruction stating that any section without supporting evidence in the context block should say so plainly rather than being filled in. That sentence only works because the retrieval step guarantees the context block is real and current. Telling a model to “only state what’s in the evidence” is meaningless if the evidence wasn’t reliably gathered in the first place.
What the model is being asked to do now is something closer to reading comprehension than open-ended reasoning: compress this specific evidence faithfully into this specific shape. That’s a task even a smaller model handles well, because it no longer requires the model to make a judgment call about its own epistemic state. It only has to stay close to text that’s directly in front of it.
A side benefit that came from doing this properly
Because the retrieval step queries live memory fresh on every request rather than reading from a cached snapshot, the brief stays current automatically. There’s no invalidation logic to write and no staleness window to think about. That property wasn’t a separate feature we built. It fell out naturally from choosing to do retrieval synchronously, as part of serving each request, instead of treating it as something a background job could populate ahead of time.
A related fix that reinforced the same lesson
While we were tightening this path, a second, smaller issue surfaced. Reasoning-family models reject any request that combines function tools with an unset reasoning effort parameter on the standard chat completions endpoint. The API is explicit about this in its error message. Our first fix simply stopped sending the parameter, on the assumption that omitting it would let the model fall back to something safe by default.
It didn't, as the model still applies its own internal default, and that default still conflicts with the presence of tools regardless of whether the parameter was explicitly sent or just absent. The actual fix required treating “don’t send it” and “send an explicit safe value” as two different things, and sending reasoning effort as none whenever a reasoning-family model was in play.
It’s a small issue on its own, but it’s worth mentioning because the shape of the mistake matches the larger one exactly: assuming that silence gets interpreted as a sensible default, when in practice the system fills that silence with its own assumption, and that assumption is rarely the one you’d have chosen if you’d been asked directly.
What this changes about how we think about shipping agent features
None of this required a bigger model or a cleverer prompt. It required micro refinements about which parts of a feature’s correctness we were willing to leave up to a model’s judgment and which parts needed to be guaranteed in code. Grounding turned out to belong firmly in the second category the moment the output needed to be trustworthy rather than merely plausible.
The useful test we now apply before shipping anything that touches memory or retrieval is simple: if this feature’s correctness depends on the model choosing to do something first, would we be comfortable if it chose not to, on any model a user might select. If the honest answer is no, that choice doesn’t belong to the model. It belongs in the code that calls the model.
We also stopped treating our strongest available model as the bar for “does this feature work.” It’s the wrong baseline. A feature that only behaves correctly on the smartest model available doesn’t have a grounding architecture. It has a habit that happens to hold up under favorable conditions, and favorable conditions aren’t something a real product gets to assume its users are running.
Changelog for 1.7.8:
16 Jul 2026 — Catch-up Brief Deterministic Grounding · Reasoning-Model Tool-Call Fix · Floating Desk Toolbar · Cross-Theme Colour Consistency
Catch-up Brief — Deterministic Memory Grounding
The catch-up brief previously left it up to whichever model was selected to decide whether to search memory before answering — strong tool-callers mostly stayed grounded, but weaker/local models frequently skipped retrieval and padded the answer with plausible-sounding invention.
Retrieval now runs server-side first, always, via a fixed set of memory queries covering focus/decisions/open-questions/recent-notes, merged into one context block.
The model receives a strict section template plus an explicit instruction to only state what’s in that context, writing “Nothing new this week” for empty sections instead of inventing content. Output is now consistent across providers, including smaller local models, and is inherently self-updating since it re-queries live memory on every request.
Reasoning-Model Tool Calls (Luna, Terra, Sol & o-series)
The new gpt-5.6 family models added in v1.7.7 (Luna, Terra, Sol) plus other o-series/gpt-5-family models failed every DESK tool-calling request with Function tools with reasoning_effort are not supported.
Simply omitting reasoning_effort wasn’t enough — these models still apply their own default server-side, which conflicts with function tools on /v1/chat/completions. Fixed by explicitly sending reasoning_effort: 'none' whenever a reasoning-family model is in play, since this code path always sends tools.
Desk Toolbar — Floating Frosted Panel
The bottom input bar (formatting row, model picker, THINK/COLLAB/JOT) is now a floating translucent panel with backdrop blur and rounded corners on all sides, inset from the window edge, instead of a flat opaque bar flush to the bottom.
Cross-Theme Colour Consistency
Fixed the silver theme’s background layering, where the card surface colour was identical to the page background (no visible depth) and the next step jumped straight to a harshly dark hover state.
Standardised the quick-action toolbar, send buttons, and sidebar navigation highlighting to draw from the same theme accent variables instead of one-off hardcoded colours, and gave graph “Semantic” nodes a fixed, theme-independent colour so they stay visible against every theme instead of fading to near-white.
Desk 1.7.8 weekly brief
VEKTOR Memory builds local-first, privacy-preserving persistent memory infrastructure for AI agents. Full technical documentation and changelog at vektormemory.com/docs.
Agentic Ai
AI
AI Agent
LLM
Llm Applications
Top comments (0)