DEV Community

kgaidev
kgaidev

Posted on

We ran a decision-memory tool through a 61-task coding benchmark. Here's what changed, and what it did to the token bill

Most memory benchmarks for AI agents ask whether a system can find a sentence in a long chat. Coding agents rarely fail that way. They fail by making the obvious fix when the team already decided on a different one, and nothing in the code says why. We wanted a test of that, and SDE-bench is one.

We build kgai, a decision memory for coding agents, so read what follows as a vendor's measurement. The benchmark, its harness and the comparison runs are someone else's. Our scripts, the raw results and the exact memory each agent saw are public.

What SDE-bench tests

SDE-bench is the coding-agent part of the open Agent Memory Benchmark. It has 61 bug-fix tasks in a Python codebase. Each task is built so that the obvious fix fails a hidden test, because the test encodes a decision the team made earlier: retry server errors plus exactly 429 and 408, not every client error; cap the attempts at 7 because that's what the rate-limit window measured; apply percent discounts before fixed ones. The decision lives in an old developer chat or a commit message, never in the code.

The agent gets the repository and the bug report and writes a fix. The harness grades it against the failing test and the hidden one. If the fix fails, the agent is told so and tries again, and each of those rounds counts as an intervention. The headline score is how many tasks are fixed on the first try.

Each task also comes with a corpus of past project knowledge: the chat or commit that holds the decision, 140 decoy developer chats that look relevant and aren't, and 100 real commits from the host repository as noise. A memory system's job is to turn that corpus into what the agent needs before it starts.

How we set it up

Every arm runs the same agent and model: Claude Code with Claude Sonnet 5, the harness's defaults.

kgai doesn't store documents. In normal use an agent records decisions into it while it works, each one with its reason, and later agents read them back. So we ran the same order. First, the same model, with kgai's stock skill and no benchmark-specific instructions, read every corpus document as if it were a past session and recorded whatever that session decided. It never saw the hidden tests or the task descriptions. From the 240 shared documents it recorded 21 decisions and let the rest go, most of the decoys among them. From the 66 documents tied to individual tasks it recorded 84. For 6 tasks it recorded nothing.

Then the runs. Before each task, the harness handed the agent the answer to one lookup with the bug report as the question, through the generic memory slot any memory system can use. A typical answer was one to six records, each a decision and its reason, a few hundred words in all.

Results

61 tasks, Claude Sonnet 5 Fixed on the first try Interventions per task Cost per run
No memory (published, 5 runs) 21–25 0.80–0.92 $26–28
Hindsight (published, 5 runs) 40–46 0.28–0.41 $19–22
kgai (our 3 runs) 47, 48, 51 0.23–0.25 $12.5–17.4

The first two rows are the benchmark authors' own runs, published in the harness repository. Across our 183 task runs, kgai solved 182 in the end and 146 on the first attempt. Ten tasks never went right the first time in any of our runs; one of them is a MIME-type table with half a dozen exceptions.

Cost is the API-equivalent figure Claude Code reports for each run. Ours leaves out the one-time capture, $33 for all 306 documents, reused by all three runs; as far as their result files show, the published runs count the agent only as well.

What it did to the token bill

Nearly every setup ends with nearly every bug fixed. The difference is how many rounds it took, and each round is another pass through the code, the tests and the conversation so far. Without memory the agent averaged close to one extra round per task and a run cost around $27. With a memory of earlier decisions, both memory setups cut the extra rounds by more than half, and the same 61 fixes cost less: $19–22 a run with Hindsight, $12.5–17.4 with kgai.

The memory that helped was small and specific, the decision and its reason rather than the conversation it came from. The tests here check exact values, 7 attempts and not 8, so a record that names the value and why it's that value is what lets an agent pass on the first try.

Limits

These are three runs, and results move between runs; the published Hindsight runs alone span 40 to 46. It's one codebase in one language. Our capture step used the same model the agent used. And where a decision lived only in a commit titled "fix: …", kgai's skill deliberately didn't record it, which cost a few tasks.

The scripts, the raw results and the memory each agent got are at https://github.com/kgaidev/kgai/tree/main/bench/sde-bench. If you run a different memory system through the same harness, we'd like to see the numbers.

Top comments (6)

Collapse
 
mattbaconz profile image
Matthew •

did you count runs where the agent said it fixed the bug but the hidden test still failed? when i ran qwen3.5:4b on 51 aider polyglot exercises it claimed done 43 times and 37 of those failed, so i ended up counting claims separately from passes

Collapse
 
kgaidev profile image
kgaidev •

Yes, that's actually the only thing it counts. The harness never takes the agent's word for it. Every time the agent stops, it runs the failing test plus the hidden one against the diff, and if that fails the agent gets the pytest output and goes again. Each of those rounds is an intervention. So "fixed on the first try" means the first time it said done, it really was.

Across our 183 task runs that was 146. The other 37 stopped at least once with a fix that still failed, 44 failed stops in total, and one task hit the cap of 5 and never passed. The published runs without memory get the first stop right on 21 to 25 of 61.

Did your 37 failed claims cluster on certain exercises or were they spread out?

Collapse
 
pomereneio profile image
Kyle Ma •

The small-and-specific finding is the one that should generalize. Storing the decision and its reason, rather than the transcript, gives the next run something it can act on without re-deriving it — and without paying to re-read everything that led to it.

The other thing worth keeping is the reporting discipline here: the no-memory baseline is shown alongside the memory numbers ($27 without memory, $19–22 for Hindsight, $12.5–17.4 for kgai). Without that baseline, a memory number on its own tells you nothing about whether the memory helped or by how much. With it, the comparison is checkable, even when the tool itself is the author's own.

Collapse
 
hacksicilian profile image
hacksicilian •

Your instinct that the turn count is more trustworthy than per-turn dollars is right, and there's a mechanism worth naming. Per-turn cost folds in two things that move independently: how much accumulated context the turn re-sends, and the cache state when it runs. A retry late in a task re-sends a much larger conversation than an early one, so two runs with the same turn count can differ in dollars purely from where the retries landed. And if injecting the memory record shifts the serialized prefix between turns, a cache write (billed several times a read) can spike a single turn without changing the count at all.

If you want a cost figure as stable as the turn count, log cache-read vs cache-write tokens per task - it separates re-reading cheap cached context from rewriting it, which is where the variance actually lives. The fewer-retries finding then reads cleanly: each avoided retry removes a full re-pass over code plus tests plus history, and that re-pass is mostly cache-write and output, the expensive end. Keeping the no-memory baseline next to the numbers is the part most of these writeups drop, so good on you for leaving it in - it's the only thing that makes the delta checkable.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Did the memory cut the number of steps, or only shrink each step?

Collapse
 
kgaidev profile image
kgaidev •

Mostly the number of steps. Turns per task went from about 16 without memory to about 14 with it, and Hindsight's runs land in the same place. But a task fixed on the first try takes 12 or 13 turns in every setup, memory or not. So the attempt itself doesn't get shorter, there are just fewer retries, and a retried task runs 18 or 19 turns.

Each turn also got cheaper in our runs, roughly $0.015 to $0.020 against $0.027 without memory. That figure moved quite a bit between our own three runs though, so I'd trust the turn count more.