Wiring Android's MediaPipe Image Embedding API to a Vector Store for Real-Time On-Device Visual Search

---
title: "On-Device Visual Search: MediaPipe + FAISS on Android"
published: true
description: "Build a fully offline visual similarity search pipeline on Android using MediaPipe ImageEmbedder, FAISS Product Quantization, and keep RAM under 500MB with >90% recall."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/on-device-visual-search-mediapipe-faiss-android
---

What You Will Build

A fully offline visual similarity search pipeline on Android — no network, no server, no exfiltration surface. By the end of this tutorial you will have MediaPipe's ImageEmbedder generating float embeddings from camera frames, a JNI-bridged FAISS index answering top-K queries in under 10ms, and an mmap-backed serialization strategy that survives process death in ~35ms on cold start.

Here is the target architecture:

CameraFrame / Bitmap
       ↓
MediaPipe ImageEmbedder  →  Float[1024] embedding
       ↓
FAISS IVFPQIndex (JNI)   →  Top-K candidate IDs
       ↓
Local SQLite / Room      →  Hydrated result objects

No cloud required. This is the pattern I use in every production visual search project.

Prerequisites

• Android project targeting API 24+
• MediaPipe Tasks Vision dependency added to your Gradle file
• FAISS compiled for Android via JNI (prebuilt .so or compile from source)
• Familiarity with coroutines and Room

---

Step 1 — Generate Embeddings with MediaPipe

MediaPipe's ImageEmbedder with EfficientNet-Lite0 produces 1,280-dimensional float embeddings. On a mid-tier Snapdragon 778G, the CPU delegate runs in 14–22ms. The GPU delegate cuts that to 8–12ms but adds ~400ms of first-run JIT overhead — fine for batch work, not for interactive search.

Use the CPU delegate with setQuantize(false). FAISS needs raw floats for cosine distance.

val embedder = ImageEmbedder.createFromOptions(
    context,
    ImageEmbedderOptions.builder()
        .setBaseOptions(BaseOptions.builder().build())
        .setQuantize(false) // keep float for FAISS cosine distance
        .build()
)
val result = embedder.embed(mpImage)
val embedding: FloatArray = result.embeddingResult()
    .embeddings()[0].floatEmbedding().values()

The model weighs ~6MB on disk. Allocate 20ms to this layer in your latency budget.

---

Step 2 — Pick Your PQ Tier

Product Quantization splits each vector into M sub-vectors and quantizes each to one of 2^nbits centroids. Here is the memory math for 1M vectors at 1,280 dimensions:

Configuration	Bytes/Vector	1M Vectors	Est. Recall@10
Flat (float32)	5,120	4.9 GB	100%
IVFPQ — PQ64×8	64	61 MB	~96%
IVFPQ — PQ32×8	32	31 MB	~91%
IVFPQ — PQ16×8	16	16 MB	~84%

Start with PQ32×8. It compresses 4.9GB down to 31MB with >90% recall@10 — comfortably inside a 500MB process budget once you factor in the model, JNI overhead, and app heap.

---

Step 3 — Serialize the Index for Process Death

Android kills processes. Your index must reload in under 500ms or users feel it. Here is the minimal setup to get this working with mmap:

// Write — call from a background thread after index build
fun persistIndex(index: FaissIndex, file: File) {
    file.outputStream().use { out ->
        index.serialize(out) // JNI → faiss::write_index
    }
}

// Read — zero-copy via MappedByteBuffer on cold start
fun loadIndex(file: File): FaissIndex {
    val channel = FileInputStream(file).channel
    val buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, file.length())
    return FaissIndex.deserialize(buffer)
}

A 31MB PQ32×8 index mmap-loads in ~35ms. For incremental updates, keep a small write-ahead delta index (flat), merge nightly in a background worker, and atomic file-swap to prevent torn reads.

---

Step 4 — Tune Recall vs. Latency

The IVF coarse quantizer introduces nlist Voronoi cells. At query time you search nprobe of them:

• nlist = 1024, nprobe = 64 → ~91% recall@10, ~4ms query latency
• nlist = 1024, nprobe = 128 → ~95% recall@10, ~7ms query latency

Start at nprobe = nlist / 16 and tune upward until recall meets your SLA. Allocate 10ms to FAISS in your query budget. If it blows that, optimize this layer independently — do not collapse the pipeline into a black box.

---

Gotchas

GPU delegate first-run JIT. The ~400ms warmup cost makes GPU a trap for interactive search. CPU delegate is the right default.

Flat index for small catalogs. Go flat only if you have fewer than 50K vectors and can afford the RAM. At scale, PQ32×8 is non-negotiable.

Torn reads on index swap. Always atomic file-swap when replacing a serialized index. A partial write corrupts deserialization silently.

setQuantize(true) breaks cosine distance. The docs do not call this out loudly, but quantized embeddings are incompatible with FAISS cosine distance computation. Keep floats.

---

Conclusion

Fully offline visual search on Android is a solved problem if you respect the memory budget from day one. Lock in PQ32×8 for 1M vectors, mmap your serialized index for 35ms cold-start loads, and budget your 35ms query window as 20ms (embedding) + 10ms (FAISS) + 5ms (Room hydration). Optimize each layer independently when it misbehaves.

The privacy and latency wins over network-dependent pipelines — 80–300ms eliminated, zero exfiltration surface — make this the right default architecture for any visual search feature on Android.