How I Built a Zero-Cloud AI Memory OS for Android 📱🧠

#ai #productivity #ocr #privacy

We’ve all been there: you saw a link, a funny quote, or an important address on your screen a few days ago, but you can’t remember which app it was in. Was it WhatsApp? A random website? A fleeting notification?

Big tech’s answer to this is usually cloud-based memory or "recall" features that stream your screen context to a server. That solves the memory problem, but it creates a massive privacy nightmare.

When I previously built Pribado—a general-purpose, non-crypto key management platform for everyone—the core constraint was making it function completely without a backend. I wanted to apply that exact same strict "zero-knowledge" constraint to a digital memory assistant.

The result is Aye-Aye, a private AI memory OS for Android. It runs 100% locally on your device to passively index your digital footprint and turn it into an instantly searchable knowledge base. Zero cloud, zero telemetry, and absolute privacy.

Here is a breakdown of how I made local RAG and on-device embeddings work on Android without melting the phone.

🏗️ The Architecture: Capturing Context Locally
To capture data without friction, I had to avoid making the user manually copy-paste everything.

Accessibility Service Hooks: Aye-Aye uses Android's native accessibility features to ingest structured on-screen text in real-time. The challenge here is UI noise, so the app filters and automatically de-duplicates the text before processing.

Notification Listener: A background service logs incoming alerts and correspondence from permitted apps, mapping them straight to a local timeline.

Offline OCR: For images or apps that block accessibility text reading, I implemented a floating bubble that triggers manual screen capture. If no structural text is exposed, it runs an offline OCR engine to pull text straight from the pixels.

🧠 Entity Extraction & Hybrid Search
Once the text is captured, it needs to be searchable. Dumping raw strings into a database isn't enough for a true "memory" OS.

Local Parsing: The app automatically extracts people, places, organizations, dates, and assets. I also spent time specifically optimizing the extraction for Filipino/Tagalog language nuances, alongside standard English.

Dual-Engine Search:

I use SQLCipher FTS5 for encrypted, lightning-fast keyword searches with stemming and fuzzy matching.

For semantic search, the app downloads a highly optimized 37 MB bge-small embedding model. This runs locally to vectorize your vault, letting you query your history by abstract concept ("that article about mechanical keyboards") rather than strict keywords.

I combined both engines using Reciprocal Rank Fusion (RRF) for the highest accuracy.

💬 Grounded RAG on Android
The final piece is the chat interface. You can access your data via a native floating chat overlay. Aye-Aye runs a local Large Language Model (LLM) using Retrieval-Augmented Generation (RAG). The LLM is strictly constrained by your captured history—it can only answer based on what has actually crossed your screen.

Running the ingestion, OCR, vector generation, database storage, and LLM inference fully sandboxed on physical Android hardware was an incredible optimization challenge. Memory management and battery efficiency are ongoing battles when you refuse to offload compute to the cloud.

🧪 Looking for Beta Testers
I’m currently in the beta testing phase and need developers and power users to stress-test the local embedding models and accessibility hooks across different Android hardware profiles.

If you're interested in testing a truly private, offline-first AI, I'd love your feedback on the architecture and performance!

👉 Join the Beta Testing Form