I wanted a Flutter app to answer questions over a user's own notes without any of it leaving the phone. No server, no API key, no document upload. On a plane, on a locked-down work laptop, it should still work.
The full-stack packages get you there. flutter_gemma runs a small model like Gemma on-device, and ai_edge_rag builds on Google's MediaPipe on-device stack. If you want on-device retrieval working this afternoon, use one of those and stop reading.
I kept going because I wanted to know what "RAG" actually is once you take the abstraction away. On a phone you account for every millisecond and every megabyte yourself, and a black box that hides both is exactly the thing you cannot afford there. So I built the pipeline out of small Dart pieces and watched each stage.
Full disclosure before we start: I wrote the three packages I lean on below. None of the ideas depend on them, and I point back at the full-stack alternatives at the end.
What RAG actually is
Retrieval-augmented generation is two verbs. Retrieve the few pieces of the user's own text that are most relevant to their question, then generate an answer with those pieces pasted into the prompt. The model stays general, and the knowledge is whatever you retrieved.
The "vector database" that every tutorial reaches for is, at the scale one app holds on one device, a single matrix and a dot product. A thousand notes is not a database problem. It is arithmetic, and Dart is good at arithmetic.
One split matters before the stages. You build the index once, and you answer every question against it. Building means you chunk the documents and embed them. Answering means you embed the question and search. The answering path is the one that has to run on the phone with nothing leaving it, and it is pure Dart. Building can run wherever is convenient: on the phone at first launch, or on your own machine when the corpus is fixed and you ship the index with the app.
Stage one: chunk by meaning, budget by tokens
You cannot embed a whole document as one vector and hope for a useful match, and you cannot fit a whole document into the prompt. So you split. The naive split is by character count, which cheerfully cuts words and sentences in half. Splitting on paragraph boundaries keeps each chunk about one idea.
final chunker = Chunker.paragraphs(maxChars: 800);
Character count is a rough ruler, though. The budget the model enforces is tokens, and only the model's own tokenizer knows the real count. "The quick brown fox" is four words and a different number of tokens for every tokenizer. So the honest version splits on paragraphs first, then re-splits any chunk whose real token count is over budget:
final tok = Tokenizer.fromFile('assets/tokenizer.json');
for (final c in chunker.chunk(doc)) {
if (tok.encode(c.text).length > maxTokens) {
// this chunk overflows the model's real budget: split it further
}
}
That exact count comes from hf_tokenizers, which wraps the Rust tokenizers crate over FFI and loads the same tokenizer.json the model ships with. One caveat, stated plainly: it is a native package that runs on desktop and server Dart today, not yet on Android or iOS. That lands where index-building usually lives, since you often prepare the index off the phone anyway. If you must build the index on the phone itself, chunk by paragraph and let the embedder's own tokenizer enforce the budget instead.
Stage two: embeddings turn text into geometry
An embedding is a function that turns a piece of text into a vector of numbers, arranged so that text with similar meaning lands close together in space. "annual leave" and "vacation days" share almost no letters and come out as near-identical vectors. That closeness is the whole trick behind retrieval.
Nothing in my pipeline generates embeddings. That is the one heavy piece, and it belongs to a model: MediaPipe Text Embedder on device, or whatever you already run. On a phone the embedder is the largest thing resident besides the model itself, so its size and speed set your real budget more than any of the code here. rag_kit takes a single function that turns a batch of strings into vectors and stays out of the way of how you produce them.
final retriever = Retriever(
embedder: myEmbedder, // batch of texts in, vectors out
store: InMemoryVectorStore(),
chunker: Chunker.paragraphs(),
);
Keeping the embedder as a plain function you pass in lets you hand it a fake in a test and exercise the whole pipeline with no model at all.
Stage three: the search is a dot product
To find the chunks nearest to a question, you compare the question's vector against every stored vector and keep the closest few. The standard measure is cosine similarity, the cosine of the angle between two vectors. Same direction scores near one, unrelated scores near zero. It ignores length and looks only at orientation, which is right when the direction is the meaning.
Cosine between two normalized vectors is just their dot product, and a dot product is a multiply-and-add down two lists. The obvious implementation is a double loop. It works, and it is also the slowest thing you will write, because it fights the machine on every row.
The fix is not a cleverer algorithm. It is memory layout. If you store each vector as its own List<double>, the CPU chases a pointer to a fresh heap object for every row and reads eight bytes per component. If instead you pack every vector end to end into one contiguous Float32List, the whole corpus is one flat buffer the CPU can stream, four bytes per component, and read four lanes at a time through Dart's SIMD Float32x4 types.
That is the difference between the slow version and the fast one, and it is what vector_kit does under VectorMatrix. It caches each row's length when the row is added, so topKCosine never has to normalize by hand at query time.
final index = VectorMatrix(384);
for (final v in chunkVectors) {
index.add(v); // packed in, L2 norm cached
}
final hits = index.topKCosine(questionVector, 5);
On the demo corpus, twenty thousand chunks of 384 dimensions, a top-5 query lands in 1.4 ms against 15.9 ms for the honest hand-written cosine loop over the same data. Eleven times, from nothing but layout and a bounded min-heap instead of sorting all twenty thousand scores. Measure it on your own hardware before you trust the number; the absolute figure moves with the device.
Stage four: the megabytes you forgot to count
Speed is what people benchmark. Memory is what actually kills the app on a phone. Twenty thousand vectors of 384 dimensions at four bytes each is 29.3 MB sitting resident, and it sits there next to an on-device language model that already wants most of the RAM the phone will give you. Two hungry things in one address space is how you get the low-memory killer.
The lever is precision. Each embedding component is a small number that does not need a full 32-bit float. Store one per-vector scale, the largest absolute component divided by 127, and quantize every component to a signed byte against it. Rank order survives, because the 255 levels of a signed byte are plenty to preserve the ordering. The same index drops from 29.3 MB to 7.6 MB, close to a quarter, with full recall on the demo's data.
final quantized = QuantizedMatrix.from(index);
final hits = quantized.topKCosine(questionVector, 5);
Nothing is free. Quantization introduces rounding error, and on some data it will nudge the ranking. Full recall on one demo corpus is not a promise about yours, so you measure recall against the float version before you ship it. But a near four-times cut in resident memory is often the difference between a feature that runs beside the model and one that does not.
Putting it together
Once the pieces exist, the pipeline is short. rag_kit wires chunking, your embedder, the store, and context building into one object.
await retriever.addText(handbook, sourceId: 'handbook');
final context = await retriever.buildContext(
'how do I request leave?',
maxChars: 4000,
);
// context is now ready to paste into the model prompt
Chunk and embed on the way in, search and paste on the way out. At query time no network call happens, and the user's notes never leave the device.
When to build from primitives, and when not to
If you want on-device RAG working today and you are happy to take the whole stack as given, use flutter_gemma or ai_edge_rag. They carry the model, the embedder, and the store together, and that is the right choice more often than not.
Build from the primitives when you need to own a stage: your own embedding model, your own memory budget, your own chunking rules for the shape of your documents. Or when you want to understand what you shipped, because on a device you are accountable for what it costs in time and RAM, and you cannot account for a box you cannot see into.
Know the ceiling too. Brute force over a packed matrix is the right answer up to roughly a hundred thousand vectors. Past that you want a real approximate-nearest-neighbor index like HNSW, and a different set of tradeoffs. On a phone, holding one user's own data, you are almost never past that ceiling.
So there was no vector database to install. At one device's scale the retrieval everyone treats as infrastructure is a matrix and a dot product, running in pure Dart with the user's notes never leaving the phone. The chunker and your existing embedder are the rest of it.
The Dart pieces above: hf_tokenizers for exact token counts (desktop and server today), vector_kit for the packed-matrix search and int8 quantization, and rag_kit for the pipeline wiring. vector_kit and rag_kit are pure Dart with no runtime dependencies; hf_tokenizers wraps the Rust tokenizers crate over FFI and ships a native library.
Top comments (0)