DEV Community: jack

Picking an on-device embedding model for an iPhone notes app: what the benchmark changed

jack — Tue, 27 Oct 2026 13:06:40 +0000

I shipped an iPhone notes app called Reteum this week. The one hard rule was that there's no server of mine in the loop. No account, no analytics, and no cloud model reading anyone's notes. So search had to run on the phone, for text and for photos, and it had to be good enough that people would actually use it.

This is what that looked like, and why a benchmark stopped me from making an easy upgrade.

What search has to do

People don't remember the words they used. They remember roughly where they were, what was in a photo, or what language they were thinking in. So the search I wanted had to handle queries like:

"Seattle last month", which should find what I saved there around then
"beach", which should find beach photos with no caption
an English query for a note I wrote in Japanese

Keyword search can't do any of that. Embeddings can, if the model is small enough to live on a phone and still handles many languages.

The models

Text: Google's EmbeddingGemma, the 300M version. I fine-tuned it for the kind of writing people put in a notes app: short, messy, often with typos, sometimes mixing languages, often using a broad word ("cooking") for something specific (a soup recipe).

Photos and video: SigLIP 2 (base, patch 16, 224). Video is handled as a few still frames.

Voice: Apple's on-device speech recognition turns voice notes into text, and that text then goes through the same embedding path.

All of this runs on the iPhone, including in airplane mode. The same text embeddings drive the Atlas, which groups notes into subjects without folders or tags.

The tempting upgrade

Then Google released EmbeddingGemma 2, which reads text, photos and video. One model for everything sounded like an obvious win: less to ship, one embedding space. Before swapping anything I tested it against what I already had.

How I tested

Notes: 503 short notes and 3,018 searches in 15 languages, written for the test and not taken from anyone's real notes. Their topics and people were kept out of the training data. A search only counts if the right note comes first.
Grouping: 600 notes on 60 topics never seen in training, clustered automatically and scored with the adjusted Rand index (1 is perfect).
Photos: XM3600, 3,600 photos with human descriptions in many languages. The description is the query.
Video: 200 MSR-VTT clips with English descriptions.

I compared three text setups: my fine-tuned model, EG2 as released, and EG2 after the same fine-tuning, including averaging several fine-tuned versions ("model soups"). I trained nine EG2 versions and kept the best one.

Results: notes

Top-1 across all 3,018 searches:

Model	Right note first
My fine-tuned EmbeddingGemma	67.9%
EG2 as released	63.0%
EG2 fine-tuned (best of 9)	66.6%

English was 76.4% vs 71.3% vs 73.5%. The biggest gap was Swahili, 60.9% vs 37.0% for EG2 as released. EG2 tuned did win in a few languages, including Arabic, Korean and Portuguese, so it isn't a clean sweep.

Broken down by kind of query, cross-language search was the strongest: 83.8% for my model when the query language differs from the note's. Searching by feeling ("when I was anxious") was the weakest for every model, 14.5% at best, because a note rarely names the feeling it was written in. That's also why I don't use feeling queries as a demo example.

Grouping settled it. Adjusted Rand index was 0.499 for my model, 0.399 for EG2 as released, and 0.442 for the best tuned EG2. The Atlas depends on that, so notes stay on the current model.

Results: photos and video

Here it went the other way.

Photos (XM3600, top-1)	SigLIP 2	EG2
Average	54.1%	68.7%
Japanese	37.8%	81.1%
Chinese	43.1%	70.0%

EG2 lets you choose how many visual tokens it takes from an image. 140 tokens was as accurate as 280 at half the work. Fine-tuning EG2 for notes barely moved photo search (68.4% to 68.6%), so photos can use the released model. One caveat: XM3600 is public and might be in EG2's training data, so I'm also checking with my own photos.

For video, reading the whole clip beat still frames. SigLIP 2 on 4 frames got 65.5% right first, EG2 on the whole video at 140 tokens a frame got 71.5%, and at 70 tokens a frame 70.5% while running more than twice as fast. I'll ship the 70-token setting.

The decision

A split: my fine-tuned EmbeddingGemma stays for notes and the Atlas, and photo and video search moves to EmbeddingGemma 2 in an update. Both still run on the phone.

Two takeaways for anyone doing on-device search:

Test on your own task. A newer, more general model lost on short personal notes and on clustering, and won by a mile on photos. A single "it's better" number would have hidden both.
Measure the weak query types separately. The overall top-1 looked fine, while feeling queries sat at 14.5%. That changed what I promise in marketing copy as much as what I ship.

Full per-language tables are here: https://reteum.com/on-device-ai-benchmark

The app is iPhone only (iOS 26+), with an English UI and notes in any of many languages: https://reteum.com. If you've shipped on-device embeddings on iOS, I'd like to hear how you handled model size and indexing.

How to Turn Any Concept Into an Animated Explainer Video in Minutes (With AI)

jack — Thu, 22 Oct 2026 04:19:41 +0000

Explaining technical ideas — how an API works, what a VPN does, why a math formula holds — is far easier with visuals. But making animated explainer videos used to mean storyboards, voice-over sessions and hours in After Effects.

Here's a faster workflow I use, built around AI.

1. Start with one clear question

Good explainers answer a single question: "What happens when I call an API?" Write it down. Everything in the video should serve that question.

2. Draft a short script (or let AI do it)

Aim for 60–120 seconds, roughly 150–300 words. Structure: hook → problem → simple analogy → how it works step by step → recap.

3. Generate the visuals and narration

Tools like Easyimate take a topic or script and produce a narrated, animated explainer automatically — scenes, motion and voice included. Other options worth knowing: Vyond and Animaker (template-based, more manual control) and Canva (good for simple motion slides).

4. Edit for clarity, not polish

Watch it once with sound off. If the visuals alone don't tell the story, simplify the scene. Cut any sentence that doesn't move the explanation forward.

5. Pick the right format

16:9 for YouTube, courses and docs
9:16 vertical for Shorts, Reels and TikTok

Tips for developers and teachers

Embed short explainers in README files or docs to cut support questions.
Teachers: one 90-second video per concept beats a 20-minute lecture recording.
Reuse: one script can become a long video plus several vertical clips.

Wrap-up

You don't need animation skills anymore — just a clear question and a good script. Try it on the concept your users or students ask about most.

Disclosure: I work on Easyimate.

How we use Jev to answer quiz questions (a 90-question benchmark)

jack — Tue, 06 Oct 2026 13:59:50 +0000

I build QuizPilot, an open-source Chrome extension that reads practice questions on a web page and suggests answers. Since TypeSafe released Jev, our code sends every multiple-choice and true/false question to it. This post covers how we map quiz questions onto Jev, and what a small benchmark showed.

The short version

Accuracy: Jev answered 85–86 of 90 questions correctly across two runs. A lightweight LLM got 89, a frontier reasoning LLM got all 90.
Speed: one Jev request answered 60 questions in 0.17–0.44 s. The lightweight LLM took 2.6 s, the reasoning LLM 8.5 s.
Where it slips: every true/false and multi-select answer was right. All of its mistakes were single-choice questions that need several steps of arithmetic.
Confidence is useful: 7 of its 9 wrong answers came back with a confidence below 0.6, against an average of 0.93 for its right answers.

Why a decision model for quiz questions

A multiple-choice question already lists every possible answer. An LLM still writes its reply as text, and we parse the letter out. Jev takes a state and typed questions (pick one of these options, or yes/no), and returns the choice with a probability. No prose to parse, no answer outside the options.

How a quiz question becomes a Jev question

Single choice → one Jev categorical question; the choices are the options.
True/false → one yes/no (noul) question: "Is this statement true?"
Multiple select → one yes/no question per option ("Is option C one of the correct answers?"). Every option at 0.5 or above is ticked; if none reaches 0.5, the likeliest one is.
Fill-in, short answer, picture questions → not sent to Jev; it doesn't write text or read images.

A trimmed request for one true/false question:

{
  "model": "typesafe/jev-1.13",
  "state": "Questions from a quiz page. Each question is independent. Judge by factual correctness, not by wording or position of the options.",
  "questions": {
    "q0": {
      "type": "noul",
      "instructions": {
        "statement": "The harmonic series 1 + 1/2 + 1/3 + … converges.",
        "task": "Is this statement true?",
        "criteria": { "true": "The statement is correct", "false": "The statement is incorrect" }
      }
    }
  }
}

A yes/no answer is a single probability, so we turn it into a confidence by its distance from 0.5: |2p − 1|. A multiple-select question's confidence is the lowest confidence among its options. A whole page goes out in one request, split only when it nears Jev's input limit.

The adapter is open source: github.com/jelly-ham/quizpilot-extension.

The benchmark

90 questions with known answers, written by me, none from a real exam:

Basic set (60): 20 single choice, 20 multi-select, 20 true/false; half easy, half harder.
Hard set (30): mostly multi-step arithmetic — trailing zeros of 100!, 2^100 mod 7, inclusion–exclusion counting.

Comparison models: a lightweight general LLM with thinking off, and a frontier reasoning LLM at medium effort. Jev and the lightweight LLM ran each set twice.

| Model | Basic (60) | Hard (30) | Total (90) | Latency |
|---|---|---|---|
| Jev | 59, 59 | 26, 27 | 85–86 | 0.17–0.44 s |
| Lightweight LLM | 60 | 29, 29 | 2.6 s |
| Reasoning LLM | 60 | 30 | 8.5 s |

Jev by question type, both runs combined: true/false 40/40, multi-select 50/50, single choice 61/70.

Where Jev gets it wrong

All nine mistakes were single-choice questions whose answer has to be computed, not recognised: trailing zeros of 100!, 2^100 mod 7, the sum of primes from 1 to 100, the number of real roots of x³ − 3x = 1, and (once) the interior angles of a 12-gon. Fact-recall questions, including harder ones, were all right.

Confidence tells you when to re-check

Across 180 Jev answers, 9 were wrong. Seven of the nine came back with confidence between 0.26 and 0.49; the two exceptions were the same question (trailing zeros of 100!) at 0.69 and 0.71. Right answers averaged 0.93.

Re-checking everything below 0.6 would have touched 14 of 180 answers (8%). Using the lightweight LLM's answers from the same benchmark as the fallback, the combination scores 60/60 on the basic set and 28/30 on the hard set. (Combined from separate runs, not a live run with automatic review.)

Limits

90 questions we wrote ourselves show large gaps, not small ones.
Text only; picture questions aren't covered.
Jev 1.13 via OpenRouter, October 6, 2026.

Full post with more detail: quizpilot.link/en/blog/jev. Happy to answer questions about the adapter or the question set.