The latest articles on DEV Community by wontopos (@wontopos). https://dev.to/wontopos https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3839754%2F4ab49b52-2f26-4c93-959b-2afaaa155e20.png https://dev.to/wontopos en wontopos Mon, 23 Mar 2026 10:26:48 +0000 https://dev.to/wontopos/wmb-100k-we-built-the-first-100000-turn-benchmark-for-ai-memory-systems-2doi https://dev.to/wontopos/wmb-100k-we-built-the-first-100000-turn-benchmark-for-ai-memory-systems-2doi <p>Most AI memory benchmarks are surprisingly small.</p> <p>LOCOMO tests 600 turns. LongMemEval tests around 1,000. That's roughly one week of casual usage.</p> <p>But real AI companions, assistants, and memory systems don't get used for a week — they get used for months. Years. What happens to memory accuracy at that scale? Nobody had tested it.</p> <p>So we built WMB-100K.<br> <a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4covg2lbl9mj7gvvkh71.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4covg2lbl9mj7gvvkh71.png" alt=" " width="543" height="231"></a><br> What it is</p> <p>WMB-100K is an open-source benchmark that tests AI memory systems at 100,000 turns — roughly a year of heavy usage. It measures one thing: can your memory system find the right information when it matters?</p> <p>Not LLM reasoning. Not response quality. Just memory.</p> <p>What makes it different</p> <p>Three things set WMB-100K apart from existing benchmarks:</p> <p>Scale — 100,000 turns across 10 life categories (daily life, relationships, health, career, finances, and more)<br> Difficulty levels — 5 levels from simple fact lookup to multi-hop reasoning across 3,134 questions<br> False memory probes — 430+ questions about things that were never mentioned. "I don't know" is the correct answer. Confidently giving wrong information = -0.25 pts penalty<br> Why false memory matters</p> <p>Every other benchmark rewards correct answers. WMB-100K also punishes hallucination.</p> <p>An AI that forgets something is annoying. An AI that remembers something that never happened is dangerous.</p> <p>How to run it</p> <p>Dataset is already included. You just need an OpenAI API key for scoring.</p> <p>Total cost: ~$0.07</p> <p>Results so far<br> <a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fcq1nd7u6b44igm5jle30.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fcq1nd7u6b44igm5jle30.png" alt=" " width="800" height="386"></a><br> We tested memory systems at 100,000 turns. Accuracy drops to near zero at this scale — in ways smaller benchmarks never catch. Systems that score 66% at 600 turns flatline at 100K.</p> <p>We're testing more systems and will be updating the leaderboard. If you run it against your own system, drop your results in the GitHub issues — or leave a comment below. Would love to see how different systems hold up.</p> <p>If you find it useful, a ⭐ on GitHub goes a long way.</p> <p>GitHub: <a href="https://github.com/Irina1920/WMB-100K" rel="noopener noreferrer">https://github.com/Irina1920/WMB-100K</a></p> ai machinelearning opensource python