Earlier this week I tested whether Jev remembers how 2023 earnings turned out. It doesn't. So I ran the same pre-registered test on twelve current models through OpenRouter and on von, an open Jev-style model. Ten of the twelve remember. Shown two reports from the same company and the same season, one from 2023 and one from 2024, GPT-6 Astra picked the one that beat consensus 91% of the time. It had the company name and the date, nothing else.
The test
- 3,708 earnings announcements from 250 S&P 500 companies, January 2023 to August 2026.
- Each model sees the company, ticker, exchange, sector and announcement date, and answers three questions with a probability: did EPS beat consensus, did the stock beat the S&P 500 over the 2-day reaction, and did it over the next 20 trading days.
- For each company I take its 2023 and 2024 reports from the same quarter where the outcome differs, and count how often the model scores the right one higher. That's a within-company AUC. 0.5 means it can't tell.
- The protocol, the model list and every request were hashed and timestamped with OpenTimestamps before the first call of each round.
Results
| Model | Within-company AUC (95% CI) |
|---|---|
| GPT-6 Astra | 0.904 (0.886 to 0.921) |
| Gemini 3.8 Flash | 0.799 (0.776 to 0.822) |
| Claude Opus 5.5 | 0.766 (0.742 to 0.792) |
| GPT-6 Sol | 0.691 (0.665 to 0.717) |
| Grok 4.7 | 0.603 (0.573 to 0.634) |
| Claude Sonnet 5 | 0.587 (0.561 to 0.613) |
| Qwen3.8 Max | 0.578 (0.550 to 0.605) |
| GPT-6 Luna | 0.573 (0.547 to 0.601) |
| DeepSeek V4.1 Flash | 0.551 (0.521 to 0.578) |
| von 1.2 (open Jev-style) | 0.541 (0.512 to 0.569) |
| Kimi K3 | 0.538 (0.511 to 0.567) |
| GLM-5.3 | 0.531 (0.501 to 0.560) |
| Jev 1.13 | 0.504 (0.478 to 0.530) |
| DeepSeek V4 Pro | 0.504 (0.476 to 0.533) |
| Qwen3.8 Flash | 0.498 (0.470 to 0.527) |
| Llama 3.1 70B | 0.490 (0.464 to 0.517) |
After Holm correction, every new API model except DeepSeek V4 Pro and Qwen3.8 Flash shows memory, and GLM-5.3 only just clears the bar. von shows some too. The Sonnet 5, Jev and Llama rows come from the first study.
What it means if you backtest with an LLM
If your pipeline passes a ticker and a date to one of the top models, part of the backtest is the model's memory. GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5.5 are the clearest cases. Two cheap defences: strip names and dates when the task allows, and test your own setup with a within-company comparison like this one before you trust the numbers.
Things I didn't expect
- Price and size don't predict it inside a family. DeepSeek V4 Pro showed nothing while V4.1 Flash did, and Qwen3.8 Max remembers while Qwen3.8 Flash doesn't.
- von, a small open model built to stand in for Jev, shows a little memory (0.541) where Jev shows none (0.504). Running a model locally doesn't make it clean.
- The cheap tier remembers too. GPT-6 Luna scored 0.573, and all 3,708 of its calls cost US$0.15.
- Six endpoints won't switch reasoning off, including GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5.5. They ran at the lowest effort, so I can't separate their memory from what a little reasoning adds.
Limits
- One prompt, 250 companies, one window. A model that shows nothing here may still remember in another setting.
- Each row is the model OpenRouter served on 24 September 2026. Versions change.
- With about 1,200 pairs per model, the test can detect a pooled AUC of about 0.546. Anything smaller is invisible.
- GPT-6 Astra stopped at 1,464 of its 3,708 calls when the account ran out of credit. I finished it after seeing that partial result (0.889), so that choice wasn't blind. The requests didn't change, and every call ran on 24 September.
- The timestamps prove less than they sound. Bitcoin block 968402, which anchors the first-round freeze, was mined at 14:36 UTC, four minutes after that round's last call. On its own the proof shows the protocol existed by then, not that it came first. The second round's block is stamped two minutes after its first call, by which time 5,486 of its 14,832 requests had gone out.
- API cost: US$55 for the first eight models and US$73 for the second four, where GPT-6 Astra alone was US$48 and Claude Opus 5.5 US$24.
Reproduce
Protocols, deviations, code, every answer and all three freeze proofs: github.com/lizhuojunx86/llm-memory-audit. The earnings data comes from FMP and can't be redistributed; the scripts rebuild it with your own key.
If you've caught this in your own backtests, I'd like to hear how.
Top comments (0)