Most memory benchmarks for AI agents ask whether a system can find a sentence in a long chat. Coding agents rarely fail that way. They fail by making the obvious fix when the team already decided on a different one, and nothing in the code says why. We wanted a test of that, and SDE-bench is one.
We build kgai, a decision memory for coding agents, so read what follows as a vendor's measurement. The benchmark, its harness and the comparison runs are someone else's. Our scripts, the raw results and the exact memory each agent saw are public.
What SDE-bench tests
SDE-bench is the coding-agent part of the open Agent Memory Benchmark. It has 61 bug-fix tasks in a Python codebase. Each task is built so that the obvious fix fails a hidden test, because the test encodes a decision the team made earlier: retry server errors plus exactly 429 and 408, not every client error; cap the attempts at 7 because that's what the rate-limit window measured; apply percent discounts before fixed ones. The decision lives in an old developer chat or a commit message, never in the code.
The agent gets the repository and the bug report and writes a fix. The harness grades it against the failing test and the hidden one. If the fix fails, the agent is told so and tries again, and each of those rounds counts as an intervention. The headline score is how many tasks are fixed on the first try.
Each task also comes with a corpus of past project knowledge: the chat or commit that holds the decision, 140 decoy developer chats that look relevant and aren't, and 100 real commits from the host repository as noise. A memory system's job is to turn that corpus into what the agent needs before it starts.
How we set it up
Every arm runs the same agent and model: Claude Code with Claude Sonnet 5, the harness's defaults.
kgai doesn't store documents. In normal use an agent records decisions into it while it works, each one with its reason, and later agents read them back. So we ran the same order. First, the same model, with kgai's stock skill and no benchmark-specific instructions, read every corpus document as if it were a past session and recorded whatever that session decided. It never saw the hidden tests or the task descriptions. From the 240 shared documents it recorded 21 decisions and let the rest go, most of the decoys among them. From the 66 documents tied to individual tasks it recorded 84. For 6 tasks it recorded nothing.
Then the runs. Before each task, the harness handed the agent the answer to one lookup with the bug report as the question, through the generic memory slot any memory system can use. A typical answer was one to six records, each a decision and its reason, a few hundred words in all.
Results
| 61 tasks, Claude Sonnet 5 | Fixed on the first try | Interventions per task | Cost per run |
|---|---|---|---|
| No memory (published, 5 runs) | 21–25 | 0.80–0.92 | $26–28 |
| Hindsight (published, 5 runs) | 40–46 | 0.28–0.41 | $19–22 |
| kgai (our 3 runs) | 47, 48, 51 | 0.23–0.25 | $12.5–17.4 |
The first two rows are the benchmark authors' own runs, published in the harness repository. Across our 183 task runs, kgai solved 182 in the end and 146 on the first attempt. Ten tasks never went right the first time in any of our runs; one of them is a MIME-type table with half a dozen exceptions.
Cost is the API-equivalent figure Claude Code reports for each run. Ours leaves out the one-time capture, $33 for all 306 documents, reused by all three runs; as far as their result files show, the published runs count the agent only as well.
What it did to the token bill
Nearly every setup ends with nearly every bug fixed. The difference is how many rounds it took, and each round is another pass through the code, the tests and the conversation so far. Without memory the agent averaged close to one extra round per task and a run cost around $27. With a memory of earlier decisions, both memory setups cut the extra rounds by more than half, and the same 61 fixes cost less: $19–22 a run with Hindsight, $12.5–17.4 with kgai.
The memory that helped was small and specific, the decision and its reason rather than the conversation it came from. The tests here check exact values, 7 attempts and not 8, so a record that names the value and why it's that value is what lets an agent pass on the first try.
Limits
These are three runs, and results move between runs; the published Hindsight runs alone span 40 to 46. It's one codebase in one language. Our capture step used the same model the agent used. And where a decision lived only in a commit titled "fix: …", kgai's skill deliberately didn't record it, which cost a few tasks.
The scripts, the raw results and the memory each agent got are at https://github.com/kgaidev/kgai/tree/main/bench/sde-bench. If you run a different memory system through the same harness, we'd like to see the numbers.
Top comments (0)