DEV Community

Zane
Zane

Posted on

Why Chatboxes Fail at Multimodal AI (And How We Built a 2D Spatial Canvas in Tauri 2 + Rust)

#architecture

When you're building an AI tool for real research or creative work, almost every tutorial tells you to build a chat interface.

You render an input box at the bottom, stream markdown chunks into a scrolling feed, and call it a day.

For simple Q&A, chat is fine. But the second you start working with multimodal assets — voice memos, high-res reference images, scanned PDF pages, and sprawling drafts — the 1D chat stream completely falls apart.

Earlier context scrolls off-screen. Comparing two reference quotes requires scrolling up and down endlessly. And the AI's internal context window collapses everything into flat tokens, stripping away all spatial relationships.

To fix this, we spent the last several months building Knovum: a local-first, on-device AI workspace built on Tauri 2, Rust, and Vue 3 that replaces the chat box with an infinite 2D spatial canvas.

Here is an architectural breakdown of how we wired local audio transcription, on-device OCR, and a 2D packing layout engine in Rust without touching the cloud.

1. The Core Insight: Spatial Memory vs. Linear History

Human cognition is inherently spatial. When you organize notes on a physical desk, you place primary sources on the left, active notes in the center, and reference images on the right. Your brain remembers where things are.

A chat box destroys this spatial topography:

  • In a 1D queue, the temporal order of prompts dictates position, not logical relationships.
  • In a 2D canvas (built on the open JSON Canvas specification), every card (audio clip, image, document, or AI response) has an explicit coordinate (x, y, width, height).
  • Cards can be grouped into visual Sections, connected with directed edges, or nested.

Knovum 2D Spatial Canvas

2. On-Device Voice Transcription with Whisper.cpp

When brainstorming or capturing field interviews, typing is often too slow. We wanted users to record quick voice notes directly onto the canvas and have them transcribed into editable text cards.

Instead of sending audio recordings to cloud speech APIs (which violates privacy and costs money), we bundled a compiled whisper.cpp sidecar:

  • Audio pipeline: Captured audio is buffered into 16kHz PCM chunks.
  • Background execution: The transcription worker runs on a background OS thread with lower priority to prevent frame drops in the UI canvas.
  • Direct Card Attachment: Transcribed text generates a linked text node anchored directly to the parent audio node on the canvas.

Because whisper.cpp runs locally on CPU or Metal (on Apple Silicon), transcription happens with zero network requests and zero privacy leakage.

3. CPU-Only ONNX OCR Without Python Dependencies

If a user drops a screenshot, whiteboard photo, or book scan onto the canvas, the AI must be able to understand and quote it.

The typical developer impulse is to install Python, PyTorch, and a multi-gigabyte virtual environment. But in a desktop product, bundling Python runtimes leads to bloated 2GB+ installers and endless environment breakage.

We solved this by running PP-OCRv6 directly via the ONNX Runtime CPU worker in Rust:

  1. We download and SHA-256 verify the ONNX weights on initial run (~15MB).
  2. The Rust backend handles image decoding and preprocessing.
  3. ONNX Runtime evaluates the model on standard CPU instructions (AVX2/NEON), taking under 250ms per scan on modern hardware.
  4. Bounding boxes are preserved: when an AI card quotes a phrase from an image on the canvas, the source quote can be visually highlighted on the original image.

4. 2D Layout Automation: The Rust Packing Pass

When you drag five images and generate three AI analysis cards simultaneously, manual layout becomes tedious. But standard CSS Flexbox or Grid doesn't work on an infinite freeform 2D canvas.

We wrote a dedicated geometric layout pass in Rust (planner::packing):

  • Collision detection: Treats each canvas node as a 2D axis-aligned bounding box (AABB).
  • Cluster affinity: Nodes with shared semantic parentage (e.g. an AI summary card and the three documents it cites) are packed into visual clusters using a modified 2D bin-packing algorithm.
  • Deterministic ordering: The algorithm guarantees that existing user-placed nodes are never shifted unexpectedly, preserving the user's spatial memory.

Running this in compiled Rust means calculating complex layouts for hundreds of spatial nodes takes less than 2 milliseconds.

5. The Local-First Storage Stack

To ensure that user research remains 100% durable and private:

  • Canvas Files: Stored directly as .canvas JSON Canvas files, fully readable by Obsidian or other open tools.
  • Relational Metadata: Managed in a local SQLite database with strictly numbered, forward-only migrations.
  • Semantic Vectors: Indexed in embedded LanceDB, allowing hybrid full-text and vector search across notes, transcripts, and OCR'd text without running an external database daemon.

The entire desktop app idles at ~45MB of RAM, leaving almost all system memory available for local LLM weights (via Ollama or LM Studio).

Getting the Code & Trying It Out

Knovum is source-available and free for personal & research use.

If you're building spatial interfaces, local AI tools, or desktop apps on Tauri 2 and Rust, we'd love to hear your feedback and architectural thoughts!

Top comments (0)