I shipped an iPhone notes app called Reteum this week. The one hard rule was that there's no server of mine in the loop. No account, no analytics, and no cloud model reading anyone's notes. So search had to run on the phone, for text and for photos, and it had to be good enough that people would actually use it.
This is what that looked like, and why a benchmark stopped me from making an easy upgrade.
What search has to do
People don't remember the words they used. They remember roughly where they were, what was in a photo, or what language they were thinking in. So the search I wanted had to handle queries like:
- "Seattle last month", which should find what I saved there around then
- "beach", which should find beach photos with no caption
- an English query for a note I wrote in Japanese
Keyword search can't do any of that. Embeddings can, if the model is small enough to live on a phone and still handles many languages.
The models
Text: Google's EmbeddingGemma, the 300M version. I fine-tuned it for the kind of writing people put in a notes app: short, messy, often with typos, sometimes mixing languages, often using a broad word ("cooking") for something specific (a soup recipe).
Photos and video: SigLIP 2 (base, patch 16, 224). Video is handled as a few still frames.
Voice: Apple's on-device speech recognition turns voice notes into text, and that text then goes through the same embedding path.
All of this runs on the iPhone, including in airplane mode. The same text embeddings drive the Atlas, which groups notes into subjects without folders or tags.
The tempting upgrade
Then Google released EmbeddingGemma 2, which reads text, photos and video. One model for everything sounded like an obvious win: less to ship, one embedding space. Before swapping anything I tested it against what I already had.
How I tested
- Notes: 503 short notes and 3,018 searches in 15 languages, written for the test and not taken from anyone's real notes. Their topics and people were kept out of the training data. A search only counts if the right note comes first.
- Grouping: 600 notes on 60 topics never seen in training, clustered automatically and scored with the adjusted Rand index (1 is perfect).
- Photos: XM3600, 3,600 photos with human descriptions in many languages. The description is the query.
- Video: 200 MSR-VTT clips with English descriptions.
I compared three text setups: my fine-tuned model, EG2 as released, and EG2 after the same fine-tuning, including averaging several fine-tuned versions ("model soups"). I trained nine EG2 versions and kept the best one.
Results: notes
Top-1 across all 3,018 searches:
| Model | Right note first |
|---|---|
| My fine-tuned EmbeddingGemma | 67.9% |
| EG2 as released | 63.0% |
| EG2 fine-tuned (best of 9) | 66.6% |
English was 76.4% vs 71.3% vs 73.5%. The biggest gap was Swahili, 60.9% vs 37.0% for EG2 as released. EG2 tuned did win in a few languages, including Arabic, Korean and Portuguese, so it isn't a clean sweep.
Broken down by kind of query, cross-language search was the strongest: 83.8% for my model when the query language differs from the note's. Searching by feeling ("when I was anxious") was the weakest for every model, 14.5% at best, because a note rarely names the feeling it was written in. That's also why I don't use feeling queries as a demo example.
Grouping settled it. Adjusted Rand index was 0.499 for my model, 0.399 for EG2 as released, and 0.442 for the best tuned EG2. The Atlas depends on that, so notes stay on the current model.
Results: photos and video
Here it went the other way.
| Photos (XM3600, top-1) | SigLIP 2 | EG2 |
|---|---|---|
| Average | 54.1% | 68.7% |
| Japanese | 37.8% | 81.1% |
| Chinese | 43.1% | 70.0% |
EG2 lets you choose how many visual tokens it takes from an image. 140 tokens was as accurate as 280 at half the work. Fine-tuning EG2 for notes barely moved photo search (68.4% to 68.6%), so photos can use the released model. One caveat: XM3600 is public and might be in EG2's training data, so I'm also checking with my own photos.
For video, reading the whole clip beat still frames. SigLIP 2 on 4 frames got 65.5% right first, EG2 on the whole video at 140 tokens a frame got 71.5%, and at 70 tokens a frame 70.5% while running more than twice as fast. I'll ship the 70-token setting.
The decision
A split: my fine-tuned EmbeddingGemma stays for notes and the Atlas, and photo and video search moves to EmbeddingGemma 2 in an update. Both still run on the phone.
Two takeaways for anyone doing on-device search:
- Test on your own task. A newer, more general model lost on short personal notes and on clustering, and won by a mile on photos. A single "it's better" number would have hidden both.
- Measure the weak query types separately. The overall top-1 looked fine, while feeling queries sat at 14.5%. That changed what I promise in marketing copy as much as what I ship.
Full per-language tables are here: https://reteum.com/on-device-ai-benchmark
The app is iPhone only (iOS 26+), with an English UI and notes in any of many languages: https://reteum.com. If you've shipped on-device embeddings on iOS, I'd like to hear how you handled model size and indexing.
Top comments (0)