Glasshouse v0.1 Is Out: A Memory Benchmark for AI Systems

Glasshouse v0.1 is out. It's a long-term memory benchmark for AI systems, and it's what my last two benchmark posts here were about. If you haven't read those, here's the short version.

Why I built it

Going through developer communities, I kept running into people raising the same problems with memory benchmarks. The numbers a vendor publishes don't match the numbers someone else measures, and swapping the model that does the grading moves the results more than the gap between the systems being compared.

What's in v0.1

• 2,847 questions over a conversation that runs to 1.97 million tokens, in 10 languages, with 50 photographs.
• The conversation comes in four sizes, from 1,882 turns to 103,572, and the part that holds the answers is identical in all of them. Where the score falls apart tells you whether a system holds up as the history accumulates.
• Each axis is reported separately. There is no single headline number, because a system can be strong at one and useless at another.
• It goes past plain recall. When a fact changed and the system cannot find the new value, saying "I don't know" scores better than confidently repeating the old one. When two stored facts disagree and nothing settles it, saying they don't agree is right and picking one is wrong. When the answer was never stated, it checks whether the system says so.

If you read my earlier post, some of the numbers have moved since. That post said they would.

What I'm asking for

submissions/ is empty. We haven't submitted either.

If you're an individual, run it, and open a pull request or an issue when something is wrong. There's no threshold for individuals, on purpose, and an objection that names a specific error gets answered in public. There's a file in the repo listing what people suggested on Reddit while v0.1 was being built, and what each suggestion became. The stale fact axis, the contradiction axis and the false memory probes all started as someone's comment. One suggestion wasn't used, and it's listed anyway, because a record that only shows what was taken can't be checked.

If you're a company, you can submit a result or add your company, and how that works is in the repo.

Everything, including how to run it, is here: github.com/wontopos/glasshouse

If you run it and a number looks wrong to you, that's exactly what I want to hear.