This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
What I Built
I built NOVA (Neural Operational Virtual Assistant), a local-first, JARVIS-inspired AI desktop assistant designed to take care of repetitive computer tasks and make everyday digital work easier.
Think about how often we switch between applications, search for files, repeat the same instructions, check system information, or explain the same project context to an AI assistant.
NOVA aims to bring these interactions together in one place.
It supports:
- Local AI conversations: Ask questions and get responses from locally running language models.
- Offline voice interaction: Talk to NOVA using local speech recognition and speech synthesis.
- Persistent memory: Remember user preferences, projects, conversations, tasks, and useful knowledge.
- Desktop automation: Perform supported application, file, and system operations through registered tools.
- Visual intelligence: Understand relevant desktop context using local screen capture and OCR.
- Multi-agent orchestration: Coordinate research, development, system, vision, and memory tasks.
- Proactive automation: Schedule reminders, monitor configured conditions, and prepare useful briefings.
- Personalization: Adapt recommendations and context based on user preferences.
The idea is simple: instead of manually handling every small digital task, I want to interact with one assistant that can understand context and help execute the work.
NOVA is built to make the user a little lazier, without taking control away from them.
Demo
Project: NOVA — Neural Operational Virtual Assistant
Platform: Windows Desktop
Version: v1.0 Release Candidate
The release will showcase:
- NOVA's desktop command center.
- Local AI conversation using Ollama.
- Push-to-talk voice interaction and speech recognition.
- Persistent project and conversation memory.
- Permission-controlled desktop operations.
- Visual intelligence and system monitoring.
Code
GitHub: https://github.com/kusalar
NOVA Repository: https://github.com/kusalar/nova-local
The project is developed using a modular architecture, with separate desktop, backend, AI, memory, voice, automation, and orchestration components.
How I Built It
NOVA is built around open-source technologies and open-weight AI models. My goal was to avoid making the assistant dependent on a single closed AI provider.
Local AI and Open-Weight Models
I use Ollama as the local model runtime, with Qwen 2.5 models for language generation.
This allows NOVA to run inference directly on the user's machine. Models can be changed, tested, and optimized depending on available hardware.
Offline Voice Pipeline
NOVA's voice architecture combines:
- sounddevice for microphone audio capture.
- openWakeWord for wake-word detection.
- Vosk for offline speech recognition.
- Ollama for local language generation.
- Piper TTS for offline speech synthesis.
This creates a local voice pipeline without depending on cloud speech APIs.
Persistent Semantic Memory
I implemented a SQLite-based memory system with five categories:
- User profile memory
- Project memory
- Conversation memory
- Task memory
- Knowledge memory
For semantic retrieval, NOVA uses FastEmbed with BAAI/bge-small-en-v1.5, with a lightweight fallback mechanism.
This helps NOVA retain useful context between conversations instead of treating every interaction as a completely new session.
Desktop Automation and Permissions
NOVA includes a registered desktop tool system with 23 tools covering application, file, and system operations.
The execution architecture includes:
- Permission-based tool execution.
- Operation previews and user approval.
- Execution logging.
- Cancellation and rollback support.
- Restricted access to sensitive locations and operations.
I wanted NOVA to be useful without allowing an AI model to execute arbitrary actions without authorization.
Multi-Agent Orchestration
NOVA uses five logical agents:
- Research Agent
- Developer Agent
- System Agent
- Vision Agent
- Memory Agent
These agents share a local model runtime. A workflow orchestrator coordinates tasks, manages dependencies, and applies execution permissions.
Visual Intelligence
I added local screen awareness using Windows screen capture and OCR.
It can extract relevant desktop context while applying privacy filters to sensitive information. The subsystem is designed to be read-only.
Technology Stack
| Layer | Technologies |
|---|---|
| Desktop | Tauri, React, TypeScript |
| Backend | Python, FastAPI |
| Local AI | Ollama, Qwen 2.5 |
| Speech recognition | Vosk |
| Wake-word detection | openWakeWord |
| Speech synthesis | Piper TTS |
| Memory | SQLite, FastEmbed |
| System monitoring | psutil |
| Visual intelligence | Windows OCR |
Why Does Open Innovation Matter?
For NOVA, open innovation is not just about choosing free technologies. It is about having control over the entire AI experience.
1. Privacy Comes First
A desktop assistant may interact with personal files, conversations, application information, and project data.
I wanted NOVA to process its core AI interactions locally rather than requiring every request to be sent to an external server.
Local inference, offline speech recognition, offline speech synthesis, and local persistent memory make this possible.
2. No Mandatory Cloud AI Subscription
Running open-weight models through Ollama means NOVA does not require a paid hosted AI API for its core language-model functionality.
Once the required models and dependencies are installed, its core AI pipeline can operate without continuous internet access.
3. Freedom to Change Models
One of the biggest advantages of open-weight models is flexibility.
I can experiment with different models, compare their performance, adjust context sizes, change generation settings, and select models that work better with my hardware.
I am not locked into a single provider's API, pricing, or model limitations.
4. Building AI for Ordinary Hardware
My development laptop is not a dedicated AI workstation.
It runs on an Intel Core i3-1115G4 processor, 12 GB RAM, and integrated graphics.
This limitation became an important engineering challenge. I had to consider CPU utilization, memory consumption, model size, inference settings, and background workloads.
It made me realize that building useful AI isn't always about using the biggest model. Architecture and optimization matter just as much.
5. Open Components Make Experimentation Possible
Because NOVA uses modular open-source components, I can improve individual subsystems without rebuilding the entire assistant.
I can replace a speech recognition engine, switch language models, modify the memory architecture, or improve the automation system independently.
That freedom is one of the biggest advantages of building on open innovation rather than depending entirely on a closed API.
My Agent Session
NOVA was developed incrementally with AI-assisted coding and engineering workflows.
The development progressed from a basic local assistant into a system with offline voice, persistent semantic memory, desktop automation, visual intelligence, multi-agent orchestration, proactive automation, personalization, and reliability monitoring.
The Lazy Payoff
The real payoff is reducing repeated effort.
Instead of opening multiple applications, repeating instructions, and manually maintaining context, I can use NOVA as a central interface for supported tasks.
For example:
- Persistent memory reduces the need to explain projects repeatedly.
- Voice interaction provides an alternative to typing every instruction.
- Desktop tools reduce repetitive application and file operations.
- Multi-agent orchestration helps coordinate more complex workflows.
- Proactive automation can prepare reminders and briefings without waiting for me to request them.
I am also testing and optimizing NOVA on my actual laptop rather than assuming that a local AI assistant will automatically be fast.
One important lesson from this project is that local AI requires careful resource management. Model inference, speech recognition, speech synthesis, and background tasks all compete for limited CPU and memory.
I would rather have a smaller assistant that works reliably than a feature-heavy assistant that struggles on ordinary hardware.
The goal isn't to make the computer work harder. It's to make the user do less repetitive work.
Stay slow. Stay steady. Let NOVA handle the busywork. 🦥
Top comments (0)