A developer's guide to building secure, on-device Retrieval-Augmented Generation (RAG) applications for desktop.
If you look at the current wave of AI applications, almost all of them rely on cloud APIs like OpenAI or Anthropic.
While cloud APIs are easy to set up, they introduce three main problems for business and enterprise software:
Data Leakage: Uploading sensitive business metrics, team structures, or financial strategy to third-party APIs.
API Overhead: Paying recurring monthly usage costs.
Connectivity Dependency: The app breaks if the user goes offline.
To solve this for our users at the Founder Frameworks Lab, we built the Founder AI Desktop App to run 100% offline.
Here is a step-by-step developer's guide to how we wired up a local Python GUI (PyQt6) with a local LLM engine (Llama.cpp) and a local vector database (ChromaDB) to create an offline strategy consultant.
The Architecture
The application runs entirely on-device, implementing a local Retrieval-Augmented Generation (RAG) pipeline:
┌──────────────────────┐
│ PyQt6 GUI │
└──────────┬───────────┘
│
▼
┌──────────────┐ ┌───────────────────┐ ┌──────────────────┐
│ ChromaDB │ ◄─┤ Query Engine ├─► │ Llama.cpp │
│ (Local Embed)│ │ (LangChain Orche) │ │ (Local GGUF LLM) │
└──────────────┘ └───────────────────┘ └──────────────────┘
Step 1: Building a Native Desktop GUI with PyQt6
Rather than wrapping our app in Electron (which has high RAM and file size overhead), we opted for a native Python GUI using PyQt6.
Here is a basic structure for setting up the main window and a thread to run local inference asynchronously (preventing the GUI from freezing during generation):
python
import sys
from PyQt6.QtWidgets import QApplication, QMainWindow, QTextEdit, QPushButton, QVBoxLayout, QWidget
from PyQt6.QtCore import QThread, pyqtSignal
class InferenceWorker(QThread):
finished = pyqtSignal(str)
def init(self, prompt):
super().init()
self.prompt = prompt
def run(self):
# Local inference logic goes here
# response = local_llm_query(self.prompt)
response = "Mocked offline response."
self.finished.emit(response)
class MainWindow(QMainWindow):
def init(self):
super().init()
self.setWindowTitle("Founder AI (Offline)")
self.setGeometry(100, 100, 600, 400)
self.text_area = QTextEdit(self)
self.text_area.setReadOnly(True)
self.btn = QPushButton("Query System", self)
self.btn.clicked.connect(self.start_inference)
layout = QVBoxLayout()
layout.addWidget(self.text_area)
layout.addWidget(self.btn)
container = QWidget()
container.setLayout(layout)
self.setCentralWidget(container)
def start_inference(self):
self.worker = InferenceWorker("How do I fix operations?")
self.worker.finished.connect(self.display_result)
self.worker.start()
def display_result(self, result):
self.text_area.append(f"\nAI: {result}")
if name == "main":
app = QApplication(sys.argv)
window = MainWindow()
window.show()
sys.exit(app.exec())
Step 2: Setting Up Local Vector Storage with ChromaDB
To ground our AI in our proprietary startup playbooks and prevent it from hallucinating, we vectorize our text manuals and query them locally using ChromaDB:
python
import chromadb
from chromadb.utils import embedding_functions
Initialize local persistent client
client = chromadb.PersistentClient(path="./chroma_db")
Use a local sentence-transformer embedding model
emb_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
model_name="all-MiniLM-L6-v2"
)
collection = client.get_or_create_collection(
name="founder_frameworks",
embedding_function=emb_fn
)
Insert playbooks (run once during prep)
collection.add(
documents=["Overall Business Diagnostic (ECG KISS)...", "Weekly Sprint (PS ERP)..."],
ids=["doc1", "doc2"]
)
Query the vector database locally
query_results = collection.query(
query_texts=["How do I organize quarterly planning?"],
n_results=1
)
context = query_results['documents'][0]
Step 3: Local LLM Inference via Llama.cpp
To process the query and generated context, we load a quantized GGUF model (like Llama-3-8B-Instruct) on-device using a Python binding for Llama.cpp:
python
from llama_cpp import Llama
Load model locally, enabling GPU/Metal acceleration
llm = Llama(
model_path="./models/llama-3-8b-instruct.Q4_K_M.gguf",
n_ctx=2048, # Context window
n_gpu_layers=-1 # Offload all layers to GPU
)
Format prompt with retrieved context
prompt = f"Context: {context}\n\nQuestion: How do I organize quarterly planning? Answer strictly using the context."
response = llm(
f"<|user|>\n{prompt}<|end|>\n<|assistant|>\n",
max_tokens=512,
stop=["<|end|>"]
)
print(response["choices"][0]["text"])
Grounding the AI in Structured Business Logic
By combining PyQt6, ChromaDB, and Llama.cpp, we built a fully private AI agent. To ensure the generated answers are structured, we enforce a strict 4-Step Consultant Logic based on our 13 scaling frameworks:
Scenario Translation: Contextualize the developer/founder issue.
Framework Alignment: Run queries through the lens of a specific system (e.g. SLR CAMERAS for yearly planning).
Component Mapping: Isolate the steps required.
Three-Mind View: Output perspectives from a Dreamer (Growth), Guardian (Risk), and Doer (Execution).
Releasing the App
To package this app into standalone installers for users, we automate builds with PyInstaller and GitHub Actions:
Check out the compiled macOS and Windows binaries on our official GitHub Release Page.
Learn more about our underlying operational frameworks on the Founder Frameworks Lab Homepage.
Get the physical playbook on Amazon or Notion Press via the Founder Frameworks Book Page.
Find your operational drag first using our free Execution Velocity Grader.
Top comments (1)
We switched to cloud desktops last year and the biggest benefit was consistent performance regardless of where team members work from. No more worrying about hardware specs or security on personal devices.