The Cost of Asking Users to Repeat Themselves
You ask a coding assistant to "add validation to the login function," and it comes back with a version that checks for empty strings, because that's the reasonable default. Except you meant something else. You always mean something else, because you never write down the thing you want, and now you're three turns deep explaining, again, that your team's validation means checking against the auth service, not the input box. Every developer who's used an AI coding assistant knows this loop. You under-specify, the model guesses, you correct, it re-guesses, and eventually you get code that works — after paying a tax in back-and-forth that a colleague who already knows your habits would never charge you.
That gap between "a stranger" and "someone who remembers how you talk" is what a new benchmark set out to measure. Called CAPA (Cross-Session Adaptation to Personalized Ambiguity), it's built to test something coding assistants have quietly assumed rather than proven: that if a model remembers your past sessions, it should need less clarification this time. The paper behind it, Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants, takes twelve models, gives them five of a user's prior resolved sessions as history, and checks whether that history changes how well they handle a new ambiguous request on the first try.
Start with the biggest swing. With same-user history, Claude Opus 4.8 hits 90.0% executable success, and its first-turn success (getting it right without any clarifying back-and-forth) jumps from 24.3% to 60.3%, a 36.0 percentage point swing. GPT-5.5 goes from a nearly nonexistent 2.3% first-turn success rate to 31.0%. Those look impressive until you set them beside the blank-control baseline on the original, unambiguous HumanEval tasks, where GPT-5.5 and DeepSeek V4 Pro both hit 100% executable success with no ambiguity and no history at all. History closes a real gap, but it doesn't close all of it, and the interesting part is what decides which side of that line a request lands on.
Six Ways Developers Leave Things Unsaid
Developers leave things unsaid in at least six distinct ways, according to the taxonomy CAPA builds its tasks around, and each one breaks differently. The list: domain-cognitive polysemy (a word that means one thing to you and another to the model), structural logic misalignment (a request whose scope is unclear), habitual context omission (the stuff you never say because you always mean it), system-boundary misconception (assuming the model knows where your system starts and stops), conversational context misalignment (losing track of what "it" refers to across turns), and implicit constraint under-specification (a hard requirement that never got written down at all).
Two of these sit at opposite ends of how forgiving they are. Habitual context omission is the easy case: it's just you, being consistent. If you always want type hints, or always reach for pytest instead of unittest, that's a pattern a model can pick up from five old sessions and apply without much drama. With GPT-5.5 running on same-user history, this mechanism hits 88.0% executable success and closes out in an average of 2.630 turns, the best result of the six. The model is simply remembering your habits.
Implicit constraint under-specification doesn't offer that mercy. Here the missing piece is a hard requirement the code has to satisfy, with no safe default to fall back on. If the model guesses wrong about an unstated requirement, say it skips the retry logic you always expect around network calls, the result is wrong code that has to be unwound and redone, not a cosmetic mismatch. For GPT-5.5 with history, this mechanism carries the highest cost of the six at 3.410 turns to completion, alongside an executable success rate of 81.0% that ties for the lowest.
This mechanism-level variance mostly disappears once a task gets solved, though. Executable success only spans 81.0% to 88.0% across all six mechanisms for GPT-5.5, a fairly tight band. First-turn success, though, swings from 24.0% to 38.0% depending on which of the six mechanisms is in play. Eventual success is comparatively mechanism-agnostic; the model usually gets there. Getting it right on the first guess depends heavily on what kind of thing you didn't say. A caveat the paper itself flags: every CAPA task pairs two mechanisms, so these per-mechanism numbers are marginal associations, not the clean effect of one mechanism in isolation.
Why Task-Specific Gating Beats Generic Memory Systems
Once you accept that history helps, the next question is how you hand that history to the model. The lazy answer is: dump it all in. Take five past resolved sessions, concatenate them into the prompt, and let the model sort it out.
That's what "same-user history" means in its raw form, and it's better than nothing. But raw concatenation has an obvious problem: it treats every past session as equally relevant, even when four of the five have nothing to do with the ambiguity sitting in front of the model right now. More context isn't the same as the right context.
The CAPA benchmark tests a fix for this: a gating method where a separate gate LLM reads through the resolved history looking for a consistent pattern of how a particular kind of ambiguity got resolved before. When it finds solid evidence, it surfaces the single most informative prior session instead of all five. When it doesn't, it produces clarification guidance flagging what's still unresolved rather than forcing the model to infer from noise. It's a parameter-free workflow, no retraining, just a smarter filter sitting between the history log and the prompt. (These deeper experiments run on GPT-5.5, DeepSeek V4 Pro, and GLM-5.2; the Claude Opus 4.8 numbers above come from the main twelve-model comparison.)
The numbers back this up cleanly. Against raw same-user history, gating lifts first-turn executable success (FT-ES) by 0.66 to 13.33 percentage points depending on the model. GPT-5.5 goes from 31.0% to 44.33%. GLM-5.2 goes from 46.67% to 51.33%. Executable success (ES) barely moves, staying within ±1.0 percentage point of the raw-history baseline, so the first-turn gains come without giving up eventual success, and turns-to-completion (TTC) drops for GPT-5.5 and GLM-5.2 as well. Same accuracy, fewer clarifying round-trips, on the first try more often.
Now compare that to the other obvious move: reach for an off-the-shelf memory system instead of building something task-specific. Mem0 and A-mem are general-purpose personalization architectures, designed to retrieve facts and preferences about a user across arbitrary conversations, not to isolate a single coding-ambiguity resolution pattern. Try them on this benchmark and something counterintuitive shows up: for DeepSeek and GLM-5.2, both mem0 and A-mem underperform raw same-user history across all three metrics (ES, FT-ES, and TTC). Losing to gating would be one thing. They lose to plain concatenation.
That's a strange result on paper. Mem0 and A-mem are more sophisticated retrieval engineering than "paste in five old sessions," with indexing, embedding-based retrieval, and memory consolidation behind them, and the brute-force approach still wins. Task-specific gating, meanwhile, posts the best first-turn success of the approaches compared on every model tested, while eventual success stays within a point of raw history and turns drop on two of the three models.
The likely explanation is a mismatch between what these systems are built to find and what a coding assistant needs in the moment. Mem0 and A-mem are tuned to surface durable facts and preferences: your name, your stack, your tone. Solving a coding task with an implicit constraint requires something else: the specific resolution pattern from a past session where the same category of ambiguity showed up and got settled correctly. General memory retrieval optimizes for the wrong axis. It's very good at remembering that you like FastAPI and bad at remembering that you resolved a scope ambiguity by asking about the auth boundary last time.
Architecture beats volume. Having more history in the prompt doesn't help if the retrieval mechanism can't tell which session is the one that matters for this ambiguity type. Here's how the three approaches stack up on the same request:
Same request, three ways of feeding it history: the task-specific gate beats raw concatenation, while generic memory layers fall below it.
Complex Requests Expose the Real Gap
Ask a coding assistant to fix something small and self-contained, and there's little headroom for memory to matter. Across GPT-5.5, DeepSeek V4 Pro, and GLM-5.2, simple tasks (the ones that resolved in a turn or two) already land between 88.66% and 97.94% executable success once same-user history is in the prompt. There's not much room left to improve on a test most models are already acing.
Complex tasks break that pattern. Push the same three models into five-to-eight-turn territory and executable success drops to a range of 62.28% to 75.44%. The drop is consistent across all three models, and it amounts to the difference between a task the model basically has locked in and one it fails roughly a quarter to nearly two-fifths of the time.
Turns-to-completion points the same direction. Feeding a model its own resolved history cuts the number of turns needed to finish, across all twelve models tested. GPT-5.5 drops from 4.417 turns down to 2.970, a reduction of 1.447 turns. A model that used to need four-plus rounds of back-and-forth to nail down what you wanted now gets there in three. Claude Opus 4.8 needs only 2.113 turns with history. In practice that's a meaningfully shorter conversation.
Put the two findings together: personalization pays for itself on the requests that were already going to take multiple turns to untangle, not on the easy stuff. On a one-shot task, there's rarely enough ambiguity for history to correct. On a complex, multi-turn request, every unclarified assumption from turn one compounds into turn two, and by turn five the model is often working from a request that's drifted pretty far from what you meant. That's where a record of how you've resolved this kind of ambiguity before does its work, short-circuiting a slow, expensive negotiation that would otherwise eat several turns before landing on the right implementation.
If you're evaluating whether a coding assistant's memory features are worth the engineering effort, the simple-task numbers won't tell you much either way. The complex, multi-turn cases are where you'll see whether personalization is doing anything.
What Shuffled History Reveals About Genuine Personalization
Here's a test that cuts through a lot of the noise around "memory" features: take a model, hand it a batch of prior sessions, but swap in the wrong user's history. Not someone's garbage output or noise, just someone else's resolved conversations. Then measure what happens.
According to the Fewer Clarifications, Better Code benchmark, that mismatched history still helps. A lot, in fact: giving a model shuffled, cross-user history improves executable success by up to 10.67 percentage points over giving it no history at all. That's just the model getting to see some example of how an ambiguous request gets resolved, from anyone, at all. Structure and precedent, even borrowed from a stranger, tell the model roughly what a good clarifying question or a sane default looks like.
Then compare shuffled history against the correctly matched, same-user version. Swap in the right person's history instead of a random one, and first-turn executable success climbs a further 2.0 to 12.0 percentage points, with turns-to-completion improving another 0.04 to 0.19 turns on top of the shuffled-history baseline. For most models that gain is smaller than the jump from no history to any history, but it's the part that's about you specifically. Two layers, stacked: a broad "context exists" effect that any old history supplies, and a narrower "this is how you specifically settle this ambiguity" effect that only the right history can produce. The first layer is table stakes. The second is the one to build for.
Try Benchmarking Your Own Assistant's Memory
If you're building or evaluating a coding assistant that claims to remember users, don't take that claim on faith. Test it the way the CAPA benchmark does: measure whether history actually improves outcomes, or just sits as dead weight in the prompt.
Measure first-turn executable success before and after you add session history. In the CAPA evaluation, matched history lifted first-turn success by 15.6 percentage points on average across twelve models; Claude Opus 4.8 went from 24.3% to 60.3%. If adding history to your own system doesn't move that number, suspect your retrieval or context assembly before you blame the concept of memory.
Then check what kind of history you're feeding the model. Raw concatenation of past sessions is the easy path, but the benchmark found that a task-specific gating filter (an LLM that reviews prior sessions and surfaces the single one with real ambiguity-resolution evidence) beat raw concatenation on first-turn success for every model it was tested on, with turns-to-completion improving on two of the three. If you're just dumping every past conversation into the context window, you're likely paying token cost for noise a gate would filter out.
And if you've already bolted on a general-purpose memory layer like mem0 or A-mem, don't assume it's helping. In the CAPA tests, both underperformed plain same-user history on DeepSeek and GLM-5.2 across every metric measured. Generic memory built for chat assistants doesn't automatically know what matters in a coding context, and it can actively drag your numbers down.
You don't have to build any of this from scratch. The CAPA benchmark publishes its ambiguity taxonomy, its session-construction pipeline, and its metrics (ES, FT-ES, TTC) precisely so other teams can run the same tests against their own systems. If you want to know whether your assistant's memory is doing real work, that's the methodology to replicate.
Top comments (0)