DEV Community

ahwai zhang
ahwai zhang

Posted on

Building a Low-Latency iOS Voice-to-Memory Engine with Local Vector Retrieval

#architecture #ios #mobile #performance

When designing Lumo (Memory Copilot), our engineering team faced a strict constraint: Capture must take less than 3 seconds, and retrieval must understand semantic intent rather than fuzzy strings.

Here is how we designed the pipeline from audio stream to vector synthesis.

Architecture Overview 

[User Speech] 
      │ (AVAudioEngine 16kHz PCM)
      ▼
[On-Device / Stream ASR] ──> [Text Normalization]
                                    │
                    ┌───────────────┴───────────────┐
                    ▼                               ▼
          [Local SQLite / JSON]          [Vector Embeddings]
          (Cold Encrypted Store)         (HNSW Vector Index)
                                                    │
                                                    ▼
                                         [Semantic Query Agent]
Enter fullscreen mode Exit fullscreen mode

1. Zero-Friction Capture Flow

Developers don't have time to categorize notes into trees. In Lumo:

  • Audio capture starts within 120ms of the action via pre-warmed audio hardware sessions.
  • Voice transcriptions are normalized and stripped of filler words.
  • An asynchronous background worker extracts entities, timestamps, and action items without blocking the main UI thread.

2. Overcoming the Semantic Gap

Traditional full-text search (FTS5) fails on questions like:

  • "How did we resolve that CORS issue on the reverse proxy?"

If your note only says "Nginx added Access-Control-Allow-Origin header", FTS returns zero results because the word "CORS" was never typed!

By indexing chunks through dense vector representations, Lumo computes cosine similarity across concepts, bridging the gap between how you ask questions and how you took notes.

3. Privacy & Offline-First Guarantees

We believe an AI memory agent must be confidential by default. User data is partitioned and protected with hardware-backed encryption keys on iOS.

Experience the fluid capture in action:

Top comments (0)