A short update to the LongMemEval-V2 part of my last benchmark post. Part of The Organism Files.
In August we kept checking the LongMemEval-V2 leaderboard for our submission. The checks came back “coming soon.” After the last one, I stopped following it and we moved on to other work.
On October 18, while working on a different benchmark submission, I decided to take another look.
There we were: Perpetual Recall V3, second on the small-tier board.
The Git history dates our entry to August 9. We’d made the board more than two months before we found out. Here’s the commit.
That brings one correction to my August post: I said we had no confirmed rank. Our checks hadn’t picked up the entry. The result was there; we just didn’t know it yet.
So this isn’t a new benchmark run. It’s an update on where the work we already did landed.
What the board says
LongMemEval-V2 tests whether a memory system can retrieve useful evidence from an agent’s history for a separate reader to answer questions. Its leaderboard measures accuracy and memory-query latency.
These were the three scored small-tier entries when we checked on October 18, ordered by LAFS Gain:
| Rank | System | LAFS Gain | Memory-query latency | Accuracy |
|---|---|---|---|---|
| 1 | AgentRunbook-C v2 Online | +2.8231 | 130.5s | 75.6% |
| 2 | Perpetual Recall V3 — ours | +1.0955 | 1.96s | 53.2% |
| 3 | OPS-Agentic-Memory-v1 | +0.3223 | 153.6s | 81.4% |
Second place is on LAFS Gain, the benchmark’s measure of improvement over its reference accuracy–latency frontier. It isn’t second place on accuracy. Scoring definition
Both other entries answer more questions correctly. Their memory queries take more than two minutes; ours take just under two seconds. Those timings cover gathering evidence, not generating the final answer.
That’s the bet we’ve been working on: improve what the memory brings back without making the agent wait minutes for it.
The accuracy gap is real. So is the speed advantage.
The benchmark’s combined metric gives that trade-off a place on the board, and I’m proud our work earned one.
Keeping the runs straight
The listed result is our August 8 run: 53.2%. It isn’t the 56.98% update I quoted last time. That update never appeared on the board.
I also need to tighten one description from that post. I called the 54.32% version “zero text aimed at the reader.”
More precisely, it runs a premise check and adds a short evidence note when it cannot find support for what the question assumes. That happens on 44 of 451 questions.
Everything else it returns is retrieved record content, code-computed summaries, and verified quotes.
That distinction belongs in the record.
The higher score is a separate question
Last time I also described a 65.41% run with a fixed grounding header. I held it back from a leaderboard claim because the header tells the reader how to use the evidence, and I wanted the benchmark maintainers to decide whether that belonged in scope.
In August we emailed them both versions and asked.
On October 18 we also submitted both through the official form:
- 65.41%: the version with the grounding header, disclosed word for word.
- 54.32%: the version without that header.
Each submission describes exactly what reaches the reader. We asked them to post whichever they rule in scope.
We haven’t seen a ruling yet. When one arrives, I’ll publish it either way.
And Muninn?
The Muninn story continues too. I was actually preparing our entry for it when I decided to check LongMemEval-V2 again.
That’s the separate leaderboard whose deadline we missed. We’ve now applied for the next cycle and are waiting for the keys so we can run a smoke test before submitting the entry.
The same promise stands: when a score exists, whatever it is, it goes here.
For now, this update is simpler.
We made the board. It took us a while to find out.
I’m glad we looked again.
Top comments (0)