We Tested Three Embedding Models for AI Memory. The Wrong Answer Was Already in the Top Five.

owenlsatwinner — Tue, 27 Oct 2026 07:08:48 +0000

The meeting had moved. The search results had not caught up.

In an early memory-retrieval test for Sipplay, the query asked when the product review had finally been scheduled and which room it would use. Two cards looked relevant: the original Thursday morning booking, and the updated Friday afternoon arrangement.

Two of the three embedding models put the old booking first. Both found the new one, too. It was sitting in second place.

That is a useful failure to keep around. A dashboard watching only whether the correct memory appeared in the top five would have called the query a success.

I work on Sipplay, an AI character chat and interactive storytelling product. This article draws on our internal memory evaluation report. These are development-set results, not an independent benchmark or a claim that the final production system has passed validation.

Why we tested retrieval separately

When an AI character appears to forget something, several things may have gone wrong. The system may never have stored the fact. It may have decided not to search. It may have written a poor search query. Or it may have retrieved the right card and then used it badly.

Changing the embedding model only addresses part of that chain.

We separated the evaluation into three layers. First, test embeddings with human-written reference queries. Second, test the model that decides whether to retrieve and prepares the search plan. Third, run those components together with the same filters and retrieval settings.

The first layer answers a deliberately narrow question: given a suitable query, can the retriever put the right memory near the top?

The first result was encouraging—and too small

Our initial test used 40 Chinese memory cards and 24 queries. All three candidates used 1,024-dimensional vectors and a top-five result set through the same gateway. We repeated each model three times.

Qwen3 Embedding 8B achieved 100% Hit@1. Voyage 4 and OpenAI text-embedding-3-small each achieved 95.8%. All three achieved 100% Recall@5.

Hit@1 asks whether the first result is relevant. Recall@5 asks what share of the required memories appeared within the first five results, averaged across queries.

The outdated meeting was the instructive miss: Voyage and OpenAI ranked the old arrangement ahead of the update; Qwen ranked them correctly. But 24 queries could not settle model selection. One query alone moves Hit@1 by roughly four percentage points.

A larger development set changed the picture

The next round used 200 cards and 80 queries, again at 1,024 dimensions and top five, with three complete runs per model. The report records the following results:

Model	`Hit@1`	`Recall@5`	Mean reciprocal rank	Forbidden card ranked first
Voyage 4	95.0%	100.0%	0.973	1.3%
Qwen3 Embedding 8B	95.0%	99.4%	0.975	2.5%
OpenAI text-embedding-3-small	87.5%	99.4%	0.933	8.8%

Here, a forbidden card means an annotated distractor such as an outdated state or a fact attached to the wrong person. It does not mean a confirmed leak from a production account.

Voyage and Qwen tied on Hit@1. Qwen had a slightly higher reciprocal-rank score; Voyage had better Recall@5 and fewer forbidden cards in first place. The team selected Voyage for the next evaluation stage based on the combined quality, observed latency, and cost results.

That is a candidate decision. The report still calls for independent blind testing. Repeating the same queries three times also does not turn 80 questions into 240 independent examples.

Similarity cannot tell you which fact is still valid

The Thursday and Friday cards were both about the right meeting. A similarity model had good reason to consider both relevant.

The application needs an additional rule: the rescheduled event supersedes the original one.

For a memory system, that means keeping enough metadata to distinguish the owner, character, scope, and status of each record. The planned production path filters by user, character, namespace, and active/stale/deleted status before ranking eligible memories. Time-aware reranking can help when several valid candidates remain.

The unfiltered test deliberately kept confusing records in the pool. That makes it useful for diagnosing ranking behavior. It does not replace a separate test of the filtered production path.

Both tests matter. Otherwise, a database filter can hide a weak ranker, or a broken filter can make a good ranker look responsible for a problem it cannot solve.

Give wrong answers their own labels

Our test cases identify relevant memory IDs and forbidden memory IDs separately. That extra annotation makes the results much easier to interpret.

For example, imagine a synthetic fixture in which the user has a peanut allergy and a coworker has a mango allergy. Retrieving an allergy-related sentence is insufficient. It must belong to the right person.

We track failures such as a missing memory, a correct result ranked too low, an outdated state ranked first, and a swapped person or relationship. Cross-user or cross-character retrieval belongs in a separate isolation check, with zero tolerated occurrences in the evaluated cases.

Those labels point to different repairs. A missed result might call for better retrieval instructions. A stale result may require state handling. An isolation error requires fixing the data boundary.

Measure the failure your product actually notices

Precision@5 can be misleading when a query has only one relevant card. Even if that card ranks first, one relevant result among five produces 20% precision. Setting a 90% target would create an impossible gate for that case.

For this dataset, Hit@1, Recall@5, and ranking metrics give a more useful picture. We also need the forbidden-result rate, because a plausible wrong memory can be more damaging than an empty result.

The next step is to evaluate the actual query produced by the intent model, then check whether the response uses the retrieved material correctly. Retrieval quality and answer quality need separate scores.

For AI roleplay, that distinction becomes very concrete. A character can have a promise somewhere in its memory store and still act as though the promise never happened.

Our meeting test offered a smaller version of the same problem. The correct answer was present. The system still needed a reason to choose it.

Source note: figures are transcribed from Sipplay's internal memory evaluation report. The embedding rounds are labeled August 16, 2026. The corpora are Chinese; these results should not be treated as an English-roleplay benchmark. The raw fixtures and execution logs are not included here, so this article describes findings and methodology rather than a fully reproducible public benchmark.