<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ayush Sharma</title>
    <description>The latest articles on DEV Community by Ayush Sharma (@ayumuml).</description>
    <link>https://dev.to/ayumuml</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4250505%2Fa72ef046-0cc9-4d4d-854f-d9fcc4e55e7f.png</url>
      <title>DEV Community: Ayush Sharma</title>
      <link>https://dev.to/ayumuml</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ayumuml"/>
    <language>en</language>
    <item>
      <title>Taming a 14.6 GB RAM Cliff and 4 GB GPU VRAM Limits in a Local-First AI Desktop App (Tauri v2 + Node.js + PyTorch)</title>
      <dc:creator>Ayush Sharma</dc:creator>
      <pubDate>Tue, 17 Nov 2026 19:54:21 +0000</pubDate>
      <link>https://dev.to/ayumuml/taming-a-146-gb-ram-cliff-and-4-gb-gpu-vram-limits-in-a-local-first-ai-desktop-app-tauri-v2--5hk2</link>
      <guid>https://dev.to/ayumuml/taming-a-146-gb-ram-cliff-and-4-gb-gpu-vram-limits-in-a-local-first-ai-desktop-app-tauri-v2--5hk2</guid>
      <description>&lt;p&gt;Whenever I tried using cloud LLMs to get through 500-page technical textbooks or multi-thousand-chapter web serials, I ran into the same two failure modes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Lossy summarization flattens nuance:&lt;/strong&gt; Standard AI summaries strip out equations, drop edge cases, and hallucinate connective tissue.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zero source traceability:&lt;/strong&gt; When a summary makes a surprising claim, there is no deterministic link back to the exact paragraph in the original PDF or EPUB.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Over the past 9 months, I built &lt;strong&gt;&lt;a href="https://github.com/ayumu-ml/Omnitome" rel="noopener noreferrer"&gt;Omnitome&lt;/a&gt;&lt;/strong&gt;—an open-source (AGPL-3.0), local-first Windows desktop application that performs &lt;strong&gt;loss-bounded semantic compression&lt;/strong&gt; (with a documented target of ~7:1) where every synthesized sentence carries a clickable citation back to the immutable source paragraph.&lt;/p&gt;

&lt;p&gt;🎥 &lt;strong&gt;3-Minute Architecture &amp;amp; UI Walkthrough:&lt;/strong&gt; &lt;a href="https://youtu.be/Nmb-qiZpnrk" rel="noopener noreferrer"&gt;https://youtu.be/Nmb-qiZpnrk&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Here is how the multi-process architecture works under the hood, and the four hardest systems bugs I had to solve to make the entire pipeline run offline on a &lt;strong&gt;4 GB NVIDIA RTX 3050 laptop GPU&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Multi-Process Isolation &amp;amp; The Sole SQLite Writer Law
&lt;/h2&gt;

&lt;p&gt;Packaging a modern React UI, an asynchronous orchestration layer, and heavy CUDA PyTorch models into a single desktop app without freezing the UI required strict process boundaries:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tauri v2 Desktop Shell (Rust):&lt;/strong&gt; Native window supervision, OS lifecycle management, and child process teardown.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Node.js 22 LTS Orchestrator (&lt;code&gt;better-sqlite3&lt;/code&gt;):&lt;/strong&gt; Pipeline coordinator and &lt;strong&gt;sole writer&lt;/strong&gt; to &lt;code&gt;data.db&lt;/code&gt; in SQLite WAL mode.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Python FastAPI Sidecar (PyTorch FP16 CUDA):&lt;/strong&gt; Runs BGE-M3 (1024-dim embeddings), BGE Reranker v2, and Kokoro-82M neural TTS.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Why Node.js Is the Sole SQLite Writer
&lt;/h3&gt;

&lt;p&gt;While SQLite in WAL (&lt;code&gt;Write-Ahead Logging&lt;/code&gt;) mode supports concurrent readers, allowing Rust, Node.js, and Python processes to write directly to &lt;code&gt;data.db&lt;/code&gt; triggers &lt;code&gt;SQLITE_BUSY&lt;/code&gt; lock contention during heavy ingestion. In Omnitome, Python sidecars and the React frontend have &lt;strong&gt;zero direct database write access&lt;/strong&gt;—every mutation flows strictly through Node.js REST boundaries.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Crushing the 14.6 GB Node.js RAM Cliff (F65 Sequential Await Barrier)
&lt;/h2&gt;

&lt;p&gt;In early builds, ingesting a 10-book, ~508,000-word multi-format test corpus caused Node.js heap memory to spike to &lt;strong&gt;14.6 GB&lt;/strong&gt;, crashing consumer laptops.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Root Cause:&lt;/strong&gt; Unbounded &lt;code&gt;Promise.all&lt;/code&gt; concurrency queued hundreds of document chunks, embedding payloads, and DOM parsing buffers in memory simultaneously while waiting on the GPU sidecar.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Fix (F65 Sequential Await Barrier):&lt;/strong&gt; I refactored the 5-stage ingestion pipeline (&lt;code&gt;Parse -&amp;gt; Vectorize -&amp;gt; Classify -&amp;gt; Plan -&amp;gt; Compress &amp;amp; Attribute&lt;/code&gt;) behind a strict chunk-by-chunk asynchronous await gate with explicit buffer release.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verified Result:&lt;/strong&gt; Peak Node.js RSS dropped by &lt;strong&gt;98%&lt;/strong&gt;—from &lt;strong&gt;14.6 GB down to 296 MB&lt;/strong&gt;—processing all &lt;strong&gt;508,000 words in 5.9 minutes&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  3. Running 3 Neural Models on a 4 GB GPU (&lt;code&gt;LifecycleManager&lt;/code&gt; VRAM Eviction)
&lt;/h2&gt;

&lt;p&gt;Omnitome's local neural stack runs three models:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BGE-M3 (FP16):&lt;/strong&gt; ~2.1 GB VRAM&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BGE Reranker v2:&lt;/strong&gt; ~2.2 GB VRAM&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kokoro-82M Neural TTS:&lt;/strong&gt; ~0.6 GB VRAM&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Keeping all three resident requires &lt;strong&gt;~4.9 GB of VRAM&lt;/strong&gt;, immediately crashing a 4 GB RTX 3050 laptop GPU.&lt;/p&gt;

&lt;p&gt;To solve this without tearing down the Python sidecar process between tasks, I implemented an asynchronous &lt;strong&gt;&lt;code&gt;LifecycleManager&lt;/code&gt;&lt;/strong&gt; state machine (&lt;code&gt;UNLOADED -&amp;gt; READY -&amp;gt; EVICTED&lt;/code&gt;) with a sliding-window TTL governor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Models load in &lt;strong&gt;FP16 precision&lt;/strong&gt; on demand.&lt;/li&gt;
&lt;li&gt;When a stage completes or sits idle past its TTL window, the governor evicts the model weights and flushes the CUDA cache, reclaiming &lt;strong&gt;3.2 GB of VRAM&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  4. Calibrating BGE-M3 Retrieval &amp;amp; Fixing the CJK Token Explosion
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CLS Pooling Calibration:&lt;/strong&gt; Switching BGE-M3 vector extraction to calibrated CLS pooling improved our retrieval-separation metric &lt;strong&gt;1.68x&lt;/strong&gt; (from &lt;code&gt;0.2520&lt;/code&gt; to &lt;code&gt;0.4225&lt;/code&gt;) across our 19-suite regression harness.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CJK Multilingual Token Safety:&lt;/strong&gt; When testing unspaced Japanese texts, naive whitespace word counters treated entire paragraphs as a single "word," bypassing chunk guards and causing context-window overflows. Adding script-aware regex weighting (&lt;code&gt;~0.7&lt;/code&gt; words per CJK glyph) and sentence-boundary splitting (&lt;code&gt;。！？\n&lt;/code&gt;) eliminated the overflow completely.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Try Omnitome v1.1.1 (Open Source)
&lt;/h2&gt;

&lt;p&gt;Omnitome &lt;code&gt;v1.1.1&lt;/code&gt; is live today as a single 1.95 GB Windows x64 installer (bundling the Python + CUDA runtime so no local Python setup is needed):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;📂 &lt;strong&gt;GitHub Repository &amp;amp; Installer:&lt;/strong&gt; &lt;a href="https://github.com/ayumu-ml/Omnitome" rel="noopener noreferrer"&gt;https://github.com/ayumu-ml/Omnitome&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🚀 &lt;strong&gt;Product Hunt Launch Page:&lt;/strong&gt; &lt;a href="https://www.producthunt.com/products/omnitome" rel="noopener noreferrer"&gt;https://www.producthunt.com/products/omnitome&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;🎬 &lt;strong&gt;3:44 Video Walkthrough:&lt;/strong&gt; &lt;a href="https://youtu.be/Nmb-qiZpnrk" rel="noopener noreferrer"&gt;https://youtu.be/Nmb-qiZpnrk&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you're building local-first AI tools with Tauri, Rust, or PyTorch on constrained hardware, I'd love to hear how you handle sidecar supervision and VRAM residency!&lt;/p&gt;

</description>
      <category>ai</category>
      <category>rust</category>
      <category>python</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
