Every speed benchmark I had would stay green while search quietly returned the wrong entries. So I built a small labelled test set, and found that my default model ranked the wrong entry first more than half the time.
MindSage finds journal entries by meaning, not keywords. Each entry is turned into a vector by an embedding model, and a question is matched against those vectors. The chat feature (post 02) feeds its top five results to a language model, so if search is wrong, the answer is confidently wrong too.
Speed benchmarks can't see any of that. You can swap a model, change a threshold or break the indexing, and every latency number looks fine while the feature stops working. I needed a test that measures whether the right entry comes back.
The test set
I wrote 18 short journal entries , each on a clearly different subject: a running injury, a call with a sister, a failed deployment at work, insomnia, a mum's new pottery class, a dal recipe, a dentist visit, and so on. Then I wrote 15 questions and labelled by hand which entries answer each one.
The subjects are deliberately distinct. With near-duplicate entries a low score would say nothing about the code. With distinct ones, a working system should get the right entry first almost every time. This is not a research benchmark. It's a regression guard that answers "did this change break search?"
The labels are committed to the repo, so the judgement is made once and every run is scored against the same answers.
Three numbers
- precision@1: how often the top result is a right answer. This is what the user sees first.
- recall@5: how much of the right material is in the top five. This is what the chat model gets to read.
- MRR (mean reciprocal rank): rewards getting the right entry near the top, not just somewhere in the list.
The baseline was bad
With nomic-embed-text:v1.5, the model the app ships with, recall@5 was 0.767, MRR 0.644 and precision@1 0.467. The right entry was usually somewhere in the top five, but the top hit was wrong in 8 of 15 questions. These weren't judgement calls:
| Question | Top result | Should have been |
|---|---|---|
| "What have I written about my knee?" | the dentist entry | the running injury |
| "What is my mum up to these days?" | a friend moving away | the pottery class |
| "Anything about cooking or food I made" | the pottery class | the dal recipe |
| "Have I been to any medical appointments?" | a river walk | the dentist |
Hypothesis 1: missing task prefixes. Wrong.
The nomic model's documentation says to prefix stored text with search_document: and questions with search_query:. The app didn't do either. That looked like the obvious bug.
Adding the prefixes made every number worse : precision@1 dropped from 0.467 to 0.333 and MRR from 0.644 to 0.544. My best guess is that the way the model is packaged already handles this, but I didn't dig further. What matters is that the "obvious fix" from the docs would have shipped a regression without this test. It's written down in the repo so nobody spends an afternoon rediscovering it.
Hypothesis 2: a different model
Next I ran Google's embeddinggemma on the same entries, questions and labels. Only the model changed.
18 entries, 15 questions, top 5. Higher is better. Runs of 27 August 2026.
Wrong top hits went from 8 of 15 to 4 of 15, and the clearly wrong ones were among those fixed. "My knee" now returns the running injury, and "my mum" returns the pottery class.
It's also nearly a drop-in. Both models produce 768-number vectors, so the vector table doesn't change. Every existing entry would still need re-embedding, because vectors from two models can't be compared.
What it costs
| nomic | embeddinggemma | |
|---|---|---|
| Model on disk | 274 MB | 621 MB |
| Embeddings per second | 9.44 | 7.53 |
| Re-index 5,000 entries | 8.8 min | 11.1 min |
| precision@1 | 0.467 | 0.733 |
So the trade is 347 MB of the user's disk and about 2 extra minutes of background indexing at 5,000 entries, for a top result that's right 73% of the time instead of 47%. I think that's a good trade. It's a product decision, though, not a benchmark result, and I haven't shipped it yet.
How much to trust this
Not as much as the percentages suggest. precision@1 going from 0.467 to 0.733 means 7 correct top hits became 11. That's four questions. The direction is consistent across all three metrics, and the misses that disappeared were unambiguous, so I trust the conclusion. I wouldn't quote "+57%" as if it came from a large evaluation.
One thing I do trust fully: the test is deterministic. Separate runs on different days reproduced the baseline exactly, 0.767 / 0.644 / 0.467 to three decimals. That's what makes it useful as a guard. If those numbers move, something changed.
Next
- Embed a title plus summary instead of the whole entry. Long entries dilute the vector, and questions are short.
- Hybrid search. A plain keyword match would get "my knee" right instantly. SQLite's FTS5 can sit next to the vector table in the same file.
- Grow the test set with harder, overlapping entries, so it can tell good from very good.
Takeaways
- If a feature depends on search, test search quality, not just speed. Speed tests stay green while quality breaks.
- Eighteen hand-written entries were enough to catch a bad default, a wrong fix from the docs, and a better model.
- Say how small your sample is. It makes the conclusion more believable, not less.
Originally published at https://mindsageapp.pages.dev/blog/embedding-models-journal.
Top comments (2)
"Every speed benchmark I had would stay green while search quietly returned the wrong entries" — that sentence should be printed on a sticker. Latency is the metric that's easy to measure, so it's the one that gets a dashboard, and retrieval quality silently rots behind it until a user asks about their knee and gets the dentist entry.
What I like about the setup is that it's a regression guard, not a research benchmark, and you were honest about that distinction. 18 deliberately-distinct entries is the right call — with near-duplicate entries a low precision@1 tells you nothing about your code, only about the inherent ambiguity. Committing the labels so the judgement is made once is the detail that makes this actually runnable in CI.
precision@1 of 0.467 on the shipped default is brutal and exactly the kind of thing nobody catches without a labelled set. One thing I'd add to the three numbers: a per-question failure log, because MRR going up on average can hide a few questions getting worse. Did any question regress when you swapped models, or did the better model dominate across the board? The dominated-everywhere case is rarer than people hope.
Committing the labels so every run is scored against the same answers is the part most people skip, and it's what turns this into a regression guard. We run something similar for docs search: 30 labelled questions in CI, and when we injected four deliberate retrieval regressions it failed the build on three of them.
One thing that made our set more honest: a few questions the corpus does not answer, plus a few near-domain ones that sound close to an entry but aren't covered. With 18 deliberately distinct entries, precision@1 can look great while the system would still happily return a 'best' entry for a question nothing answers. Curious whether you've tried adding unanswerable questions to see where the top score lands for those.