Try to Break Our AI Memory Benchmark

Facts change. An earnings forecast is revised. A policy is amended. A medication dose is corrected. An entity record is updated.

Many AI memory systems retain both the old and new versions. When retrieval ranks both highly, stale information can silently enter the model context.

We built Lians to prevent that failure while preserving the historical record. That is a meaningful technical claim, especially for regulated AI, so it should be independently tested rather than accepted as marketing.

The challenge

We have opened a public challenge asking developers to try to break Lians temporal recall.

The included benchmark currently reports:

• 0 stale facts in top-5 recall
• 100 percent supersession accuracy on 22 fact pairs
• Point-in-time reconstruction through recall_at and snapshot

We want adversarial tests, not applause.

Run the benchmark

git clone https://github.com/Lians-ai/Lians.git
cd Lians/agentmem
pip install -e ".[dev]"
pytest tests/test_supersession_benchmark.py tests/test_recall_quality.py -v

No API key is required.

Try to break it

The most useful tests will go beyond the included examples:

1. Add a fact with an event timestamp.
2. Add a corrected or superseding fact.
3. Confirm present-time recall excludes the stale version.
4. Reconstruct what was known before the correction.
5. Change the wording while keeping the same underlying entity and metric.
6. Test ambiguous names, partial corrections, conflicting sources, or delayed updates.

We are specifically looking for:

• Stale facts appearing in present-time recall
• A valid update failing to supersede the old version
• Two distinct facts being incorrectly merged
• Historical reconstruction returning information that was not yet known
• Differences across environments or embedding configurations

Report evidence

The canonical challenge is GitHub issue #60.

Please include the fact pair or dataset, your environment, the command or script, the expected result, and the actual result.

If a failure is reproducible, we will turn it into a regression test and credit the contributor. People who find meaningful edge cases will also be invited to a technical pairing session with the maintainers.

Why this matters

A current answer and a historical reconstruction are different products.

A system reviewing a past financial decision, clinical recommendation, legal analysis, or policy action must preserve what was knowable at the time. Later corrections should improve current answers without rewriting the original decision context.

That is the standard we want Lians to meet. The fastest way to improve the product is to expose the benchmark, make the claims falsifiable, and welcome critical results.

If your team is deploying an agent that depends on changing facts, you can also request a free temporal-memory audit at lians.ai. We will examine one sanitized workflow and identify where stale facts or missing evidence could affect reliability.