Wiring Android's Room Database to a Local LLM for Semantic Query Rewriting

---
title: "Android Room + On-Device Embeddings: The Hybrid Search Pipeline That Hits Sub-100ms"
published: true
description: "Wire Android Room to on-device embeddings with FTS5 for a hybrid retrieval pipeline hitting sub-100ms on mid-range hardware. Stop letting keyword search fail your users."
tags: kotlin, android, architecture, mobile
canonical_url: https://blog.mvpfactory.co/android-room-fts5-on-device-embeddings-hybrid-search
---

What We Are Building

By the end of this tutorial you will have a working hybrid retrieval pipeline for Android: FTS5 for high-recall candidate selection, a quantized on-device embedding model for intent rewriting, and a cosine similarity re-ranker that keeps you in the 50ms tier instead of the 300ms tier. This runs on mid-range hardware, fits a 30MB memory budget, and delivers semantic accuracy that pure SQL cannot touch.

Let me show you a pattern I use in every project where local search matters.

---

Prerequisites

• Android project with Room 2.5+
• ONNX Runtime for Android dependency
• A quantized INT8 embedding model (MiniLM-style, ~23MB after quantization)
• Kotlin coroutines

---

Step 1 — The Problem With Pure SQL Search

Most teams treat Room like a key-value store with a WHERE clause. That is the mistake.

A user typing "meeting notes from last Tuesday about the budget" returns zero results when your stored record says "Q3 planning session - finance review." Same semantic content, zero lexical overlap. FTS5 improves recall with BM25 ranking, but it still operates on surface tokens. The moment you introduce synonyms or natural-language phrasing, you are back to zero.

The fix is semantic query rewriting: intercept the raw query, project it into an embedding space, and use that vector to rerank FTS5 candidates.

---

Step 2 — Define Your FTS5 Virtual Table

FTS5 is your high-recall, low-latency first pass. Note the annotation — @Fts4 without ftsVersion defaults to FTS4, not FTS5.

@Entity(tableName = "notes")
data class NoteEntity(
    @PrimaryKey val id: Long,
    val title: String,
    val body: String,
    val createdAt: Long
)

// ftsVersion = FTS_VERSION_5 required — @Fts4 defaults to FTS4 without it
@Fts4(contentEntity = NoteEntity::class, ftsVersion = FtsOptions.FTS_VERSION_5)
@Entity(tableName = "notes_fts")
data class NoteFts(
    val title: String,
    val body: String
)

FTS5 with BM25 on a 100K-row corpus on a mid-range Snapdragon 6-series device completes in the 15–40ms window. That is your floor.

---

Step 3 — Wire the On-Device Embedding Model

Quantize your model to INT8 before you even think about shipping. A typical 90MB MiniLM-style model drops to ~23MB and inference falls from ~180ms to ~55ms on CPU.

class EmbeddingEngine(context: Context) {
    private val session: OrtSession = OrtEnvironment
        .getEnvironment()
        .createSession(loadModel(context, "minilm_int8.onnx"))

    suspend fun embed(text: String): FloatArray = withContext(Dispatchers.Default) {
        val tokens = tokenizer.encode(text, maxLength = 128)
        val inputTensor = OnnxTensor.createTensor(env, tokens)
        session.run(mapOf("input_ids" to inputTensor))
            .get("last_hidden_state")
            .meanPooling()
            .l2Normalize()
    }
}

Store embeddings as BLOB columns in Room, quantized to INT8 at write time. At 128 dimensions, each vector costs 128 bytes — negligible at scale.

---

Step 4 — The Hybrid Ranker

Run FTS5 for the top-50 candidates, then re-rank with cosine similarity against the query embedding. The cosine pass only touches the candidate set, not the full corpus.

suspend fun hybridSearch(query: String): List<NoteEntity> {
    val queryVec = embeddingEngine.embedWithCache(query)
    val ftsResults = noteDao.ftsSearch(query, limit = 50)

    return ftsResults
        .map { note ->
            val storedVec = note.embedding.dequantize()
            val score = cosineSimilarity(queryVec, storedVec)
            note to score
        }
        .sortedByDescending { it.second }
        .take(10)
        .map { it.first }
}

Here is the minimal LRU cache that eliminates inference on repeated queries — five lines that save you 55ms per hit:

private val queryCache = object : LinkedHashMap<String, FloatArray>(16, 0.75f, true) {
    override fun removeEldestEntry(eldest: Map.Entry<String, FloatArray>) = size > 100
}

suspend fun embedWithCache(query: String): FloatArray {
    val key = query.trim().lowercase()
    return queryCache.getOrPut(key) { embed(key) }
}

A 100-entry INT8 cache adds ~8MB overhead and collapses cache-hit latency to single-digit milliseconds.

---

Latency Reality Check

Strategy	P50 Latency (Snapdragon 6xx)	Memory	Recall
Pure FTS5 BM25	20–40ms	~2MB	Low–Medium
Pure Vector Scan (full corpus)	280–400ms	50–200MB	High
Hybrid (FTS5 + re-rank top-50)	55–95ms	~25MB	High
Hybrid + result cache	5–15ms (hit)	+8MB LRU	High

---

Gotchas

Skip the FTS5 recall phase and the query planner destroys you. A full in-memory cosine scan over 100K vectors runs 280ms before you add model inference. Keep FTS5 as the gating layer.

Never run embedding inference on the main thread. Always dispatch to Dispatchers.Default. Debounce your query input with a 200ms delay — this alone cuts unnecessary embedding calls by ~60% on incremental search UIs.

INT8 quality drop is marginal. Typically less than 2% on MTEB benchmarks. The 4x memory savings and 60–70% inference speedup are not optional — they are the difference between shipping and not shipping on mid-range devices.

---

Conclusion

The hybrid FTS5 + on-device embedding pipeline runs on mid-range Android hardware today, fits within a 30MB memory budget, and delivers retrieval quality that pure SQL cannot match.

Three things to internalize before you ship: use FTS5 as your recall layer (top-50), not your final ranking signal. Quantize to INT8 before shipping. Cache query embeddings aggressively — the docs do not stress this enough, but it is the single highest-leverage optimization in the pipeline.