Every AI memory provider quotes their LongMemEval temporal-reasoning score.
The leaders sit around 90%. I spent an afternoon reading the actual dataset
instead of running it, and found that "temporal reasoning" is really at least
three different capabilities stapled together under one accuracy number.
Here's what's actually in there with the JSON and terminal output to back
it up.
What I did, and didn't do
I did not run the benchmark. This is an analysis of the dataset itself:
question types, answer formats, and how the traps are constructed.
That's a real limitation and I want it stated up front. I can tell you what
LongMemEval tests. I can't tell you which questions any particular provider
fails on, because I didn't run it.
Everything below comes from the 500 question files that ship with the dataset,
readable with MemoryBench's list-questions command and a text editor. No API
keys, no cost.
42% of the benchmark is time-dependent
Total questions: 500
temporal-reasoning: 133 (27%)
knowledge-update: 78 (16%)
Two categories, both about how facts change over time. Together they're a little
over two-fifths of the whole benchmark.
That's a lot of weight on one capability. It also means a provider's headline
number is heavily influenced by how it handles time which makes it worth
knowing what "handling time" actually requires here.
Trap one: state-change pairs
Some questions come in pairs that share a scenario and ask about different
moments in it.
07741c44 "Where do I initially keep my old sneakers?"
answer: under my bed
07741c45 "Where do I currently keep my old sneakers?"
answer: in a shoe rack in my closet
Same object. Same conversation history. Both questions point at the same two
answer sessions. The only difference is initially versus currently.
(Worth noting: both of these are tagged knowledge-update in the dataset, not
temporal-reasoning the state-change pattern shows up across both
categories, which is part of why treating them as one bucket loses
information.)
This is a clean test, and it's brutal for a common design. A memory system that resolves conflicts by recency - newest fact wins, older one is superseded answers the second question correctly and the first one wrong, by construction.
A system that keeps only the first mention fails the reverse.
Passing both requires storing both states with their timestamps and resolving which one the question is asking for.
There are 44 questions in consecutive-ID pairs across the dataset. Not all are state-changes, but the pattern is deliberate.
Trap two: time-filtered disambiguation
Not every temporal question is about a fact changing. Some are about picking the right event out of several similar ones.
0bc8ad92 "How many months have passed since I last visited a museum
with a friend?"
asked 2023/03/25 → answer: 5
0bc8ad93 "I mentioned visiting a museum two months ago. Did I visit
with a friend or not?"
asked 2023/03/11 → answer: "No, you did not visit with a friend."
There are multiple museum visits in the history. Retrieving "museum visit" by semantic similarity isn't enough you have to filter by time window and check an attribute (was a friend present) on the specific event that matches.
Note these two have different answer sessions and question dates a fortnight
apart. They aren't the same haystack asked twice; they're two instances of the
same trap design.
Trap three: abstention decoys
30 questions carry an _abs suffix. They're near-identical to a real question,
with one entity swapped for something that was never mentioned.
2698e78f "How often do I see my therapist, Dr. Smith?"
2698e78f_abs "How often do I see Dr. Johnson?"
Dr. Johnson doesn't exist in the conversation history. The ground truth:
The information provided is not enough. You mentioned seeing Dr. Smith but
not Dr. Johnson.
That's not scoring whether a system returns null. It's scoring whether it can refuse and explain the near-miss that probably caused the confusion. A system that pattern-matches "therapist" and confabulates a frequency fails. So does one that says only "I don't know."
6% of the benchmark is testing reasoned refusal. That's a design decision worth noticing - abstention is treated as a first-class capability, not an edge case.
In a 10-question sample, arithmetic outnumbers retrieval about 2:1
I pulled ten temporal-reasoning questions and checked their answer formats
directly against the source JSON:
08f4fc43 30 days. 31 days (including the last day) is also acceptable.
0bb5a684 7 days. 8 days (including the last day) is also acceptable.
0db4c65d 18 days. 19 days (including the last day) is also acceptable.
2a1811e2 21 days. 22 days (including the last day) is also acceptable.
2c63a862 14 days. 15 days (including the last day) is also acceptable.
5e1b23de 3
d01c6aa8 27
2ebe6c92 'The Nightingale' by Kristin Hannah
4dfccbf8 I started taking ukulele lessons with Rachel.
71017277 my aunt
Seven are computed date arithmetic. Three are retrieved facts filtered by a time
window a 70/30 split in this sample. That's two genuinely different
capabilities:
Date arithmetic: find an event, extract its timestamp, subtract from the question date, express the difference in the right unit.
Time-filtered retrieval: find the specific event that falls in a stated window, then answer a question about it.
A system could be excellent at one and poor at the other, and the category
average would hide it entirely.
I want to be honest about what this number is and isn't: it's a 10-question sample out of 133 temporal-reasoning questions, read by hand rather than pulled programmatically at scale running the full category costs API budget I didn't have for this pass.
Take 70/30 as directional, not as a precise benchmark-wide
ratio. The qualitative point that "temporal reasoning" bundles two different skills holds regardless of the exact split.
The tolerance in those answers is worth noting too "31 days (including the last day) is also acceptable." The dataset authors built in slack for off-by-one errors, which tells you they expected date arithmetic to be genuinely error-prone.
The retrieval problem underneath everything
One more number that reframes all of this.
0bc8ad92
haystack sessions: 49
answer sessions: 3
Every question ships with roughly fifty conversation sessions. The answer lives in two or three of them. The rest are deliberate distractors unrelated chats about pitch decks, recipes, travel plans.
That's a 6% signal-to-noise ratio before any temporal reasoning happens. A system has to find three needles in fifty haystacks, and only then start
reasoning about dates.
So a failed temporal question could be a failure at any of three stages:
- The right fact was never extracted from the conversation
- It was extracted but not retrieved
- It was retrieved but the temporal reasoning was wrong
A single accuracy number can't distinguish them.
Why this matters
If you're building a memory system, "we score 91% on temporal reasoning" isn't actionable. The 9% you're failing could be date arithmetic, disambiguation, state-change pairs, or retrieval and the fixes for those are completely different.
If you're evaluating one, the same applies in reverse. Two providers with
identical temporal scores could have opposite strength profiles.
The obvious improvement is sub-scores: report arithmetic, retrieval, and
state-change separately. The dataset already supports it the question types and ID patterns are right there.
What I'd want to know next
The question I can't answer from the dataset alone: when a system fails a temporal question, where in the pipeline did it break?
Answering that means running a benchmark and then inspecting what was actually stored versus what the source said separating extraction failures from retrieval failures. That's a bigger piece of work and it needs real API budget.
If someone's already done it, I'd like to read it.
Reproducing this
Everything here is free to verify:
git clone https://github.com/supermemoryai/memorybench
cd memorybench
bun install
bun run src/index.ts list-questions -b longmemeval -t temporal-reasoning
bun run src/index.ts list-questions -b longmemeval -t knowledge-update
The dataset downloads on first use (265 MB) and splits into 500 individual
question files under data/benchmarks/longmemeval/datasets/questions/.
Read them directly that's where everything above came from.
I build MiniEval, a faithfulness gate for AI memory. It works on the extraction step -checking
whether a stored fact is faithful to the message it came from which is one
layer beneath what this benchmark measures. I went looking at LongMemEval to understand how the field evaluates memory, and found the category structure more interesting than the scores.
Top comments (0)