The latest articles on DEV Community by woochan (@woochan). https://dev.to/woochan https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4040938%2F28e5d058-a4bf-4036-aca4-c0dd8d5bdfb7.png https://dev.to/woochan en woochan Mon, 07 Sep 2026 12:38:27 +0000 https://dev.to/woochan/why-im-building-a-new-ai-memory-benchmark-and-why-the-existing-ones-fall-short-4b37 https://dev.to/woochan/why-im-building-a-new-ai-memory-benchmark-and-why-the-existing-ones-fall-short-4b37 <p>Hi everyone! As this is my first post on DEV, I wanted to take a moment to introduce myself, share my journey, and talk about what I'll be writing about moving forward. While I know my first few posts might not catch a huge wave right away, I wanted to start documenting this journey out in the open.</p> <p>I'm a developer currently working at <strong>wontopos</strong>, a startup building memory APIs for AI applications (similar to platforms like Mem0 and Zep). Right now, my primary focus isn't just building the API itself—I am deep into researching and building a brand-new <strong>AI memory benchmark</strong>. Until this benchmark is fully completed, most of my upcoming posts will be focused on this building process.</p> <h2> The Problem: Why Another Benchmark? </h2> <p>As I've been browsing various developer communities and reading discussions on AI memory, I noticed a lot of valid criticism directed toward existing benchmarks like <em>LongMemEval</em> or <em>Locomo</em>. </p> <p>Developers and engineers often point out several frustrating issues:</p> <ul> <li> <strong>Vendor Bias &amp; Discrepancies:</strong> There's often a noticeable gap between benchmarks published by memory API companies themselves and the results others get independently.</li> <li> <strong>Model Dependency:</strong> Testing results can swing wildly depending on which underlying model is used during the evaluation.</li> <li> <strong>Flawed Design:</strong> Many current benchmarks simply contain structural errors or edge-case oversights that don't reflect real-world production environments.</li> </ul> <p>Seeing these pain points firsthand, I decided to take matters into my own hands and build a truly fair, reliable memory benchmark from scratch.</p> <h2> The Elephant in the Room: "Isn't This Just a Marketing Trick?" </h2> <p>I know what you're probably thinking: <em>“You work for a memory API company. Won't you just rig this benchmark to favor your own product?”</em></p> <p>It’s a completely fair and natural skepticism. That is exactly why I want to share the creation process step-by-step, discuss the design trade-offs openly, and—most importantly—<strong>invite the community's feedback and critique</strong> to keep me honest.</p> <h2> What's Next? </h2> <p>I’ve already laid down the basic foundations for the benchmark design, and I'll be sharing updates, technical hurdles, and progress reports here until it's fully realized. </p> <p>I’d love to hear from you all:</p> <ul> <li>What are your biggest frustrations with current AI memory benchmarks?</li> <li>Have you tested tools like Mem0, Zep, or others? What did you wish their evaluation metrics measured better?</li> </ul> <p>Let me know in the comments below, and thanks for reading along!</p> ai benchmark startup opensource