DEV Community: woochan

A memory benchmark draft with 17 axes and no headline number: is it worth running?

woochan — Fri, 16 Oct 2026 10:31:41 +0000

I work at a company that builds memory infrastructure, so I'm biased: we compete in the thing this benchmark measures. We wrote a long-term memory benchmark called glasshouse, and I'd like to know if you think it is worth running on your own system.

Why we made it

Memory systems report their own numbers, and the numbers don't agree. The case that pushed us to build this: Zep reported 84% on LoCoMo, Mem0 recomputed it and got 58.44%, and Zep came back with 75.14%. Same system, same benchmark, three numbers. LongMemEval and BEAM ship a dataset and a script, but neither has a place to submit results, so everyone reports their own.

What is in it

2,847 questions over one conversation of 1.97 million tokens, in 10 languages, with 50 photographs.
17 axes, each reported separately. There is no single headline number, because a system can be strong at one axis and useless at another.
The standard run is full (103,572 turns, 1,547 questions). A smaller core tier (1,882 turns) is the control: if a system drops from core to full, it knows the answer but cannot retrieve it under volume.

Axes I think are unusual

STALE: a fact changed. Finding the new value is full credit, "I don't know" is half, and confidently repeating the old value is zero.
CONTRADICT: the speaker says two incompatible things weeks apart. Saying they disagree is full credit, and confidently picking one is zero.
RECONCILE: two statements look incompatible but hold under different conditions. Explain both, or pick one and call the other wrong?
FALSE_MEMORY: you assert something that was never said. Does it agree?
Cross-language: 800 questions that store in one language and ask in another, across all 10.

The rule for partial credit is one question: is an incomplete answer dangerous? A visible gap earns half, because the user can ask again. A wrong value stated as current earns zero. The build also fails if answering "I don't know" to everything scores more than 20%.

What is not ready

It is a draft, not released. The submissions folder is empty and we have not submitted our own system either.
You have to write the ingest step yourself. We ship no runner, because the only one we could write is for the product we sell.
Scoring calls a judge LLM through OpenRouter on your own key, so it costs money. The judge has been run in answer mode, but memory mode has not been run against it yet.
A synthetic conversation is not a real user's life.

What I'd like to know

Would you run this on your own memory system? If not, what is missing?
Which axis looks badly designed or unfair?
Would you trust a benchmark run by a company that also sells a memory product?

Details

You don't need an account to read any of it, so please take a look if you want the details: github.com/wontopos/glasshouse.

I wrote the first draft with an AI assistant and edited it myself.

Honest question: at this price and performance, would you use this memory API?

woochan — Fri, 16 Oct 2026 07:31:38 +0000

I work at Wontopos, so I'm biased. About three months ago we launched a long-term memory API for AI agents, and honestly almost nobody has looked at it. I'm not asking you to sign up. I'd like a blunt answer: at this price and performance, would you use it?

What it is

You store a user's memories once. For each query, recall returns only the relevant ones, and you pass them into your model's prompt. Memory lives per user, so each end-user or agent gets an isolated space.

The core calls are store, recall, search, supersede (replace an outdated memory) and forget. SDKs for Python, TypeScript and Rust, plain HTTP works too, and there is an MCP server (beta) for Claude Code and Cursor.

What I think is different

Same recall across languages. We measure it as a grid of 70 store-language by query-language pairs, and it averages 95.2% recall@5. A memory saved in Korean should be found by a query in Japanese.
Bounded context. Recall works over histories of 1.4M tokens and still hands back a slice of roughly 1,470 tokens, so the prompt does not grow with the history.
Images. A memory can carry an image with no caption or alt text, and it is found from a sentence in any language.
Ask again. If one pass does not carry the answer, a second pass returns memories the first one did not.
Deletion in one call. One call erases a single memory or an entire user, including images and revisions (GDPR).
Your LLM key is yours. Keys are sent per request and never stored, and nothing is trained on your data.

Price

	Input / 1M tokens	Output / 1M tokens
Tablet	$2	$3
Scroll	$4	$8

Each request adds a flat $0.0001. No subscription, no storage rent, no memory caps. You add a card and top up at least $5, and there is a hard spend cap. If you bring your own LLM keys, the provider bills those.

Numbers (measured by us)

Tablet 2: 95.7% on LongMemEval-S, 67.5% on BEAM-1M. The BEAM-1M number is averaged over 5 independent runs (σ 0.22%, none cherry-picked).
Scroll 1.2: 92.3% on LongMemEval-S, about 2,800 tokens and about 450 ms per query. BEAM-1M is 58.6% with re-asking turned off.
Where it is weak: low-resource languages. Swahili is 53.0% recall@5.

The harness and per-question records behind the BEAM-1M number are public, and the Tablet 2 paper reports three findings that run against us.

What I'd like to know

Would you pay this for memory in your own app?
What would make you pick this over Mem0, Zep, or just keeping context in a file?
Which number looks wrong or unbelievable?

Details

You don't need to sign up, you can just read the information. Knowing what I'm talking about should let you give a more accurate answer than my summary, so please take a look if you want the details: wontopos.com and the research page.

You can answer here in the comments. Even better, there is a short survey on the site: click the Survey button in the top menu of the main page. No sign-up is needed for that either.

I wrote the first draft with an AI assistant and edited it myself.

I Posted in 5 Developer Communities for 5 Weeks. Here's What Got Replies and What Got Killed.

woochan — Tue, 13 Oct 2026 03:53:59 +0000

I do marketing for a memory API for AI agents. For about five weeks I've been posting in developer communities: dev.to, Hacker News, Reddit, Velog, and Indie Hackers. I'm writing this down because some of the numbers surprised me, and I'd like to hear how you do it.

dev.to: what got comments

I've published 7 posts here so far. The announcement posts got almost nothing. "Glasshouse v0.1 Is Out" got 1 comment, my intro post got 1, and my first post about why I'm building a benchmark got 0.

The posts that got comments were different. A post about how I'm trying not to rig my own benchmark got 19. A question to developers got 8. A post asking what's wrong with my benchmark (15 stars, 0 runs) got 7. One narrow question about memory API features got 0.

My takeaway so far: people reply when I ask them something they can answer, or when I show them where I might be wrong. They don't reply to "I shipped something."

Hacker News: dead posts

Several of my posts and comments on my first accounts were marked dead. Two Show HNs I made in April got 1 and 2 points, with no comments. I don't know exactly why. I started a new account, and I'm commenting on other people's threads before I post anything with a link.

I posted 8 times in r/AI_Agents, mostly questions about how to trust a memory benchmark and what to add to mine. I said I work at Wontopos, and I left 25+ comments answering people under those threads. Most posts got 1 to 2 comments each, and the best one got 9.

What I take from it: Reddit rewards answering people directly more than posting.

Indie Hackers

New accounts can't create posts yet, so I'm commenting until that opens up.

Velog (Korean)

I've been writing a Korean series there. Each post gets between 11 and 57 views, and the first episode did best.

Questions for you

Did your first posts get killed somewhere? What did you change?
How long did Indie Hackers take to unlock posting for you?
What got you your first real replies?

What Memory API Feature Would Make You Think "Okay, I'd Actually Try That"?

woochan — Fri, 02 Oct 2026 08:16:06 +0000

I work at Wontopos, which builds a memory API for AI agents in the same space as Mem0 and Zep, and I keep getting stuck on one question. What would make a developer trust a memory API enough to actually pay for one.

What it does

You store what a user says once, and on each query the API returns only the memories that are relevant, ready to put into an LLM prompt. It is a hosted API with a closed engine, and there is no self-hosting yet. The details are here: https://wontopos.com/en/why

Where I'm stuck

I know a lot of you want to manage memory yourselves, and I get why, but I don't know how a paid product is supposed to earn that trust. Retrieval performance keeps improving, so I don't think a better score is what decides it for long. And memory is slow to notice anyway. You only feel it working weeks into using something, so a better number on a benchmark doesn't land the way a visible feature does.

What I'm asking

What I'm trying to find is an idea for a memory API that people could feel right away. If you've built your own memory or tried a few products, what's the thing you wish existed. And is there any feature you can think of where you'd go, if a memory API had that, I'd actually try it.

Devs, Would You Mind Answering One Question?

woochan — Wed, 30 Sep 2026 11:07:09 +0000

It's been about two months since I started actually running this, not just building it, and I keep coming back to the same question instead of moving past it.

What I'm building

I work at Wontopos, a memory API in the same space as Mem0 and Zep. Store what a user says once, retrieve only what's relevant on each query, hand that to an LLM prompt. The long term goal, and I'll admit this out loud, is to become something close to standard infrastructure for AI memory at the B2B level. Not a feature bolted onto someone else's product, but the layer everyone building on AI just reaches for.

That's a long way off, and I'm aware of how that sounds coming from two months in. Right now what actually matters is much smaller. Get real developers to use it, and find out where we're wrong before we've committed to being wrong at scale.

Where the confidence went

This is the part I didn't expect. Watching actual usage come in has made me less confident, not more. I thought two months of real traffic would sharpen my sense of what people want from something like this. Instead it mostly sharpened my sense of what I don't know. People reach for memory for reasons I didn't anticipate, and skip it for reasons I also didn't anticipate. Neither of those is something I can fix by thinking about it harder alone.

What I'm actually asking

So here's the honest question, and I mean it plainly rather than as a lead-in to a pitch.

Where would you actually reach for a memory API. And what would you need it to do that you're not getting from whatever you use now, whether that's another memory product, rolling your own with a vector store, or just stuffing more into the context window and hoping.

Even a short answer helps. I'd rather hear "I wouldn't use this and here's why" than nothing at all, because silence doesn't tell me whether we're solving a real problem or a problem we invented.

I Made a Memory Benchmark as Fair as I Could. 15 Stars, 0 Runs. What's Wrong With It?

woochan — Mon, 28 Sep 2026 05:08:58 +0000

I published a memory benchmark six days ago. I spent most of the build trying to make it fair rather than trying to make it look good, and now I have no idea whether that worked.

Here is the whole state of it:

stars        15
forks        0
issues       0
submissions  0

People star it. Nobody runs it. Which means the one thing I actually wanted, someone telling me a question is broken or a rule is wrong, has not happened once.

What it is, quickly

Glasshouse is a long-term memory benchmark. 2,847 questions over a conversation that runs to 1.97 million tokens, in 10 languages, with 50 photographs. Every axis is reported on its own and there is no single headline number, because a system can be great at one and useless at another. It goes past plain recall too. If a fact changed and the system cannot find the new value, "I don't know" scores better than confidently giving the old one. If two stored facts disagree and nothing settles it, saying they do not agree is the right answer.

The reader and the judge are fixed by the version, so a submitter does not get to pick the model that reads the answers. Every submission carries its per-question record, so anyone can recompute the number instead of trusting it.

What I think might be in the way

I can guess at some of it, and none of these are accidents, which is exactly why I want to know whether I got the tradeoff wrong.

You have to write the ingest yourself. I deliberately ship no runner for storing the corpus, because the only client I could write is for the memory system my company sells, and that does not belong in a benchmark anyone is meant to be able to win. It does mean step one is work.
Scoring costs your own money. The judge runs through OpenRouter on your key.
The standard run is 1.97 million tokens. That is not something you try on a whim.
The official reader is claude-opus-5.5. I fixed it because changing the reader changes every number, but it is not the cheap option.

What I'm asking

If you opened it and closed the tab, I would genuinely like to know at which line. And if you are the kind of person who would run something like this, what would have to be different for you to actually do it?

Repo: https://github.com/wontopos/glasshouse

Glasshouse v0.1 Is Out: A Memory Benchmark for AI Systems

woochan — Tue, 22 Sep 2026 07:34:24 +0000

Glasshouse v0.1 is out. It's a long-term memory benchmark for AI systems, and it's what my last two benchmark posts here were about. If you haven't read those, here's the short version.

Why I built it

Going through developer communities, I kept running into people raising the same problems with memory benchmarks. The numbers a vendor publishes don't match the numbers someone else measures, and swapping the model that does the grading moves the results more than the gap between the systems being compared.

What's in v0.1

2,847 questions over a conversation that runs to 1.97 million tokens, in 10 languages, with 50 photographs.
The conversation comes in four sizes, from 1,882 turns to 103,572, and the part that holds the answers is identical in all of them. Where the score falls apart tells you whether a system holds up as the history accumulates.
Each axis is reported separately. There is no single headline number, because a system can be strong at one and useless at another.
It goes past plain recall. When a fact changed and the system cannot find the new value, saying "I don't know" scores better than confidently repeating the old one. When two stored facts disagree and nothing settles it, saying they don't agree is right and picking one is wrong. When the answer was never stated, it checks whether the system says so.

If you read my earlier post, some of the numbers have moved since. That post said they would.

What I'm asking for

submissions/ is empty. We haven't submitted either.

If you're an individual, run it, and open a pull request or an issue when something is wrong. There's no threshold for individuals, on purpose, and an objection that names a specific error gets answered in public. There's a file in the repo listing what people suggested on Reddit while v0.1 was being built, and what each suggestion became. The stale fact axis, the contradiction axis and the false memory probes all started as someone's comment. One suggestion wasn't used, and it's listed anyway, because a record that only shows what was taken can't be checked.

If you're a company, you can submit a result or add your company, and how that works is in the repo.

Everything, including how to run it, is here: github.com/wontopos/glasshouse

If you run it and a number looks wrong to you, that's exactly what I want to hear.

An Introduction to Wontopos, the Memory API I Work On

woochan — Fri, 18 Sep 2026 09:30:14 +0000

My posts here so far have only been about the benchmark I'm working on. This time I want to introduce the company I work at, Wontopos.

The company

Wontopos builds WOS: long-term memory for AI agents. It stores an end user's memories once, then recalls only the relevant ones per query so you can feed them into an LLM prompt.

It is not only for teams. If you are building a product, you use the SDK and your code decides exactly when to store and what to recall. If you just want a finished tool to remember (Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI), you use MCP and write zero integration code. Same account underneath, so what one surface stores, the other recalls.

What it does

Each query returns a small, bounded set no matter how much is stored, so input cost does not grow with the store. A store that has been filling up for a year does not cost more to read from than a new one.

No LLM ever runs over your stored memories, on any model. Tablet runs no LLM at all. Scroll may use one to reformulate the query only, never your stored data.

No language is privileged: you can store and query in any language, mixed freely.

The models

Four are live. Tablet is lean and returns less, Scroll returns more context. They read the same stored memories.

Tablet 1 (live)
The original. About 1,200 tokens per query. Lean and fast, with no LLM anywhere in it.

Tablet 2 (live, and the default)
About 1,000 tokens per query. It does three things Tablet 1 could not. The engine knows who said what, so you can register speakers and recall one person's words. It stores images as memories, and the caption may be empty, in which case the image is the memory and is searchable on its own. And it can be asked again: if one pass does not carry the answer, verify sends the engine back for memories it has not already returned, up to three times, with no LLM at any value.

Scroll 1 (live)
About 3,700 tokens per query. Adds an LLM query-understanding layer over the same stored memories. That LLM reformulates the query only.

Scroll 1.2 (live, and the current Scroll)
Sentence-level recall over the same memories, so a fact said once, in passing, still surfaces. About 2,800 tokens per query, which is fewer than Scroll 1.

Book (in development, not priced)
A fundamentally different design, not the context-for-accuracy trade. Built for one goal: never repeat the same mistake.

The research

Every measurement we have published, on public benchmarks, with every run reported:
https://wontopos.com/research

I Sell Memory APIs. I'm Also Building the Benchmark. Here's How I'm Trying Not to Rig It.

woochan — Sun, 13 Sep 2026 05:09:24 +0000

Hey everyone. This time I'll go through what got me started on this benchmark, and the core of how it's actually built.

It started from reading complaints, not from an idea. The same ones kept coming up: numbers a vendor publishes don't match numbers someone else measures, swapping the model that does the grading moves the results more than the gap between the systems being compared, and because of that nobody really uses published numbers anyway. They test two options on their own data and keep whichever annoys them less. I work at Wontopos, and we sell a memory API, so I can't point at that and shrug.

All numbers below are from the current build. Nothing is final yet, so some of them will have moved by the time you read this.

Fairness first, because you have no reason to trust me

I know how "I built a fair benchmark" sounds coming from someone at a memory company. So instead of promising anything, here's what's written in the repo.

Wontopos hosts it and keeps it running. That is the whole role. We also build memory infrastructure, which means we compete in the thing we administer, so the limits are written down rather than promised:

Our submissions go through the same approval as everyone else's. We do not merge our own.
We do not decide who is admitted. The rules do.
Our numbers are verified the same way as everyone else's.
Wontopos publishes nothing on a new version for fourteen days.
If other memory companies want to co-administer, that's better than us alone, and the offer is open.

A few of the submission rules point the same way:

Publish the per-question record. Anyone can recompute the number from it. The aggregate is a claim, the record is the evidence.
The reader, the judge and the prompts are set by the version. A submitter does not choose them.
The harness has to be one a customer could use. A number produced through a path only its author can reach is not a number anyone else can get.

Everything there is Apache 2.0, so if we ever become the problem, the whole thing can be taken and run elsewhere without asking us.

It's all here: github.com/wontopos/glasshouse. Fair warning about what you'll find. The benchmark itself isn't in there yet and submissions/ is empty. The rules went up first, and we haven't submitted either.

The corpus

One person's life, told across about 17 months of conversation, with the facts you're supposed to remember buried inside it. 103,572 turns, 1,991 sessions, roughly 1.9M tokens.

It runs at four haystack sizes, from a small core up to the whole thing. The 1,882 turns that actually contain the answers are identical in all four, character for character. What changes is how much unrelated conversation is packed around them, so you can watch a system degrade as the haystack grows instead of getting one score and no idea what it means.

The axes

1,547 questions across 14 axes at the full size. Most are ordinary recall: it was said, can you get it back, can you get it when the question uses different words, can you say when it happened, can you combine two facts.

These four are why I built this.

Stale facts. A value changed. The new one exists, but I made it hard to find on purpose. Three outcomes instead of two:

Answer	Score
New value	1.0
"I don't know"	0.5
Old value, stated as current	0.0

The reason for the middle row is the whole point. Put "I don't know" and "confidently out of date" in the same bucket and you've hidden the thing that actually hurts in production, because those are very different things to be paying for.

Contradictions. The conversation states two different values for the same fact, nobody corrects it, and nothing tells you which is right. There is no correct answer. Confidently picking one is wrong. Saying "these don't match" is right.

Apparent contradictions. The mirror image. Two statements look like they clash but hold under different conditions, so both are true. Telling this apart from a real contradiction is the point.

Abstaining. Questions about things never mentioned at all. An empty answer scores full marks, anything invented scores zero. On top of that there are 500 false-memory probes, which plant something that was never said inside the question itself and check whether the system plays along.

Images

50 photos are shared inside the conversation, with 150 questions about them. Every one of those questions is pinned to a date: "In the photo from 14 April 2026, what was on the sofa?"

That's there because with 50 photos, a question like "was the laptop open?" points at two of them at once. Inside the flow of the conversation that's fine, but a question lifted out on its own, or translated into another language, becomes impossible to answer correctly. The date narrows it to exactly one photo, and it doesn't leak anything, since nothing is asking when it happened.

Languages

The corpus carries 100 sessions in 10 languages, 10 sessions per language, mixed in with everything else. 800 questions run on this, in both directions:

A fact stored only in another language, asked in English.
A fact stored in English, asked in another language.

Both directions matter because they break differently. One tests whether anything crosses the language boundary at all, the other tests whether the query side does.

Speed, and why it's measured this way

Speed matters, but measuring it fairly is harder than it looks, and this is the part I rewrote the most.

The problem is geography. Raw latency includes the speed of light. Measure from Seoul against a server in the US and you're 190ms in before any work has happened. Report that as-is and you're ranking where the server sits, not how good it is.

So the network floor gets measured separately and subtracted. What's left I call "latency minus round trip", not "pure compute", because response transfer doesn't fully subtract and calling it pure compute would be overstating it.

Two conditions reduce the leftover error, and both are part of the procedure:

Warm the connection first. TCP starts slow and ramps up. Without warming, a response crossing 14KB picks up an extra round trip, and a 3,700 token response sits right on that boundary.
Record the response size. The remaining error scales with it, so writing it down lets a reader judge how much slack is in the number.

The budget. People stop feeling like a conversation is flowing at around 1,000ms. The LLM's first token eats about 500ms of that on its own. So memory gets the remaining 500ms, and that's the target. Twice as fast as the target scores +1, four times slower scores -1, and it's a log scale in between, because the difference between 200ms and 400ms matters more than the difference between 3s and 3.2s.

Per question, and capped. Each question scores between -1 and +1, and the results are averaged rather than summed. Sum penalties instead and the total sinks on its own as you add questions, which means the score stops describing the system at all.

Speed is reported next to accuracy, not folded into it silently, and the weight is stated. And if speed can't be measured on a given system, the term is removed rather than zeroed. Being unmeasurable shouldn't be a penalty.

Nothing has been scored yet, and I'm not quoting numbers before there are numbers.

One question for you

If you were going to run something like this, what would have to be in it before you believed the result? And if you've ever looked at a published memory benchmark score and thought "no", what tipped you off?

Why I’m Building a New AI Memory Benchmark (And Why the Existing Ones Fall Short)

woochan — Mon, 07 Sep 2026 12:38:27 +0000

Hi everyone! As this is my first post on DEV, I wanted to take a moment to introduce myself, share my journey, and talk about what I'll be writing about moving forward. While I know my first few posts might not catch a huge wave right away, I wanted to start documenting this journey out in the open.

I'm a developer currently working at wontopos, a startup building memory APIs for AI applications (similar to platforms like Mem0 and Zep). Right now, my primary focus isn't just building the API itself—I am deep into researching and building a brand-new AI memory benchmark. Until this benchmark is fully completed, most of my upcoming posts will be focused on this building process.

The Problem: Why Another Benchmark?

As I've been browsing various developer communities and reading discussions on AI memory, I noticed a lot of valid criticism directed toward existing benchmarks like LongMemEval or Locomo.

Developers and engineers often point out several frustrating issues:

Vendor Bias & Discrepancies: There's often a noticeable gap between benchmarks published by memory API companies themselves and the results others get independently.
Model Dependency: Testing results can swing wildly depending on which underlying model is used during the evaluation.
Flawed Design: Many current benchmarks simply contain structural errors or edge-case oversights that don't reflect real-world production environments.

Seeing these pain points firsthand, I decided to take matters into my own hands and build a truly fair, reliable memory benchmark from scratch.

The Elephant in the Room: "Isn't This Just a Marketing Trick?"

I know what you're probably thinking: “You work for a memory API company. Won't you just rig this benchmark to favor your own product?”

It’s a completely fair and natural skepticism. That is exactly why I want to share the creation process step-by-step, discuss the design trade-offs openly, and—most importantly—invite the community's feedback and critique to keep me honest.

What's Next?

I’ve already laid down the basic foundations for the benchmark design, and I'll be sharing updates, technical hurdles, and progress reports here until it's fully realized.

I’d love to hear from you all:

What are your biggest frustrations with current AI memory benchmarks?
Have you tested tools like Mem0, Zep, or others? What did you wish their evaluation metrics measured better?

Let me know in the comments below, and thanks for reading along!