DEV Community

GhostOS
GhostOS

Posted on

How We Built an Ultra-Lightweight Desktop AI Assistant (<35MB RAM) with Zero-DB RAG and Zero Bloat

#python #ai #opensource #machinelearning

If you’ve tried running desktop AI assistants recently (LM Studio, Jan, AnythingLLM, Open-WebUI), you’ve probably noticed an infuriating trend:

Most desktop AI tools consume 800MB to 1.4GB of RAM before you even load a single neural weight into memory.

Why? Because almost every modern AI assistant is packaged as a bloated Electron container. When you finally load a 7B LLM (3GB VRAM), a vision model (2GB), and Whisper (1GB) for voice interaction, standard laptops and base 8GB Apple Silicon Macs instantly freeze or crash with brutal Out-Of-Memory (OOM) errors.

We asked a fundamental question:

Can we build an autonomous, feature-packed desktop AI companion that idles at <35MB RAM, runs 100% offline, and never crashes an 8GB machine?

That was the birth of GhostOS (GitHub: GhostOS-ai/GhostOS). Here is the engineering story of how we pulled it off.

1. Ditching Electron: The ~35MB RAM Native Architecture

Electron bundles an entire Chromium browser and Node.js runtime for every single window.

Instead of Electron, GhostOS uses OS-native web rendering engines:

  • macOS: Apple WebKit
  • Windows: Microsoft Edge WebView2
  • Linux: WebKitGTK

The entire floating Casper mascot and Spotlight command bar (HUD) are rendered as transparent, frameless native webviews bridged directly to Python.

Baseline Idle RAM Comparison:
┌─────────────────────────┬──────────────────┐
│ App                     │ Idle RAM (No LLM)│
├─────────────────────────┼──────────────────┤
│ Jan / AnythingLLM       │ ~850 MB – 1.2 GB │
│ LM Studio               │ ~700 MB – 950 MB │
│ GhostOS (Native Engine) │ ~35 MB           │
└─────────────────────────┴──────────────────┘
Enter fullscreen mode Exit fullscreen mode

The difference is a 25x reduction in RAM footprint. Your CPU fans stay silent.

2. Dynamic VRAM Juggler: Solving the Multi-Model OOM Trap

In an ideal workflow, you want:

  1. An LLM for reasoning and coding.
  2. A vision model for screen understanding.
  3. A speech model for voice input.

If you keep all three resident in GPU VRAM, you need at least 6GB–8GB of pure VRAM, leaving nothing for macOS or Windows OS compositor.

We implemented a Dynamic VRAM Juggler:

  • The engine maintains a hot-swap memory scheduler.
  • Models are unloaded to system cache and swapped into active GPU memory on-demand in <150ms.
  • Peak VRAM is strictly capped at ≤3.5GB, making it rock-solid on budget PCs and 8GB MacBooks.

3. Zero-DB RAG: Document Chat Without Docker or Chroma

Almost every RAG (Retrieval-Augmented Generation) tutorial tells you to spin up Docker, install heavy vector databases (Chroma, Qdrant, Milvus), and download hundreds of megabytes of Python dependencies.

For a desktop tool, that’s crazy overkill.

We replaced the entire heavy vector stack with pure SQLite FTS5 (Full-Text Search 5) + BM25 ranking:

  • Zero external database daemon.
  • Zero network ports opened.
  • Drag-and-drop any PDF, code file, or markdown note directly onto Casper.
  • Document indexing happens in milliseconds with sub-millisecond retrieval latency.

4. 5 Killer Features That Elevate Productivity

  1. Ghostwriter (Global Clipboard AI): Select text anywhere — Slack, VS Code, Notes, Browser — hit your shortcut, and GhostOS rewrites, fixes grammar, or translates inline.
  2. Memory Vault: Cross-session persistent context graph stored in SQLite with zero VRAM overhead. GhostOS remembers your preferences and projects without wasting context tokens.
  3. Terminal Sandbox Lens: A proactive safety layer that intercepts dangerous commands (rm -rf, DROP DATABASE, raw dd, format C:) and explains the risks in plain English.
  4. Daily Standup & Commit Crafter: Analyzes your git diffs and terminal sessions to craft clean conventional commits and ready-to-share daily standup summaries.
  5. Auto-Minutes: Transcribes audio meetings and outputs an executive summary with actionable checklists.

5. 100% Paranoid Privacy Pledge

No telemetry. No tracking. No phone-home pings. GhostOS operates completely air-gapped with Wi-Fi switched off.

🚀 Quickstart: Try It in 30 Seconds

GhostOS is 100% open-source under the MIT license and published on PyPI.

# Install globally via pip
pip install ghostos-ai

# Launch your floating Casper desktop companion
ghost pet
Enter fullscreen mode Exit fullscreen mode

Or clone the repo:

git clone https://github.com/GhostOS-ai/GhostOS.git
cd GhostOS
pip install -e .
ghost pet
Enter fullscreen mode Exit fullscreen mode

⭐️ If you believe desktop software should be fast, lightweight, and private, give us a Star on GitHub! We'd love to hear your thoughts and benchmarks in the comments.

Top comments (0)