Most AI assistants today are impressive at one thing:
conversation.
You ask a question → the model generates an answer → you're done.
But I wanted something different.
I wanted an AI assistant that could actually operate my computer.
Not just tell me how to open an application.
Actually open it.
Not just tell me what's on my screen.
Actually inspect the screen.
Not just claim that an action succeeded.
Actually verify that it happened.
That project became Diego.
And after building it, I've realized that the interesting part of an AI desktop agent isn't the LLM.
The interesting part is everything that has to happen around the LLM.
What is Diego?
Diego is a desktop AI agent built around four major capabilities:
Talk → Perceive → Know → Act
The goal is to connect these capabilities into one system rather than treating them as independent features.
At a high level:
┌───────────────┐
│ DIEGO │
└───────┬───────┘
│
┌──────────────────┼──────────────────┐
│ │ │
TALK PERCEIVE KNOW
│ │ │
Voice / STT / TTS Screen / OCR Local documents
Wake word Vision PC facts
Live state RAG
│ │ │
└──────────────────┼──────────────────┘
│
▼
Reasoning / Planning
│
▼
ACT
│
┌─────────────┼─────────────┐
│ │ │
Desktop Browser Media
Windows Search YouTube
│ │ │
└─────────────┼─────────────┘
│
▼
VERIFY
The important design principle is that the LLM isn't the source of truth for everything.
1. Voice Isn't Just Speech-to-Text
Diego has a complete local voice interaction pipeline.
The flow is roughly:
Microphone
↓
VAD
↓
Speech Recognition
↓
Transcript Validation
↓
Intent / Command Routing
↓
Execution
↓
Verification
↓
Response
↓
TTS
↓
Listen Again
That distinction matters.
A simple voice assistant might do:
audio → speech-to-text → LLM → response
But real-world audio isn't clean.
There can be:
- silence
- background noise
- incomplete speech
- low-confidence transcription
- accidental activations
- garbage transcripts
So Diego uses confidence/evidence gating before low-quality transcripts are allowed to reach the agent.
This is one of those things that isn't very exciting in a demo.
But it makes a huge difference in an actual continuously running assistant.
2. Wake Word Detection
Diego also has a local wake-word subsystem.
The architecture includes:
- local wake-word detection
- wake verification
- audio backend integration
- normal wake-enabled runtime
- development mode without wake detection
The wake subsystem is intentionally treated as a relatively stable component while other parts of the system evolve.
That's another lesson I've learned:
Not everything needs to be constantly rewritten.
Sometimes the best optimization is knowing what not to touch.
3. Diego Can Authenticate Who Is Using It
Diego includes a local face-authentication subsystem.
The basic pipeline includes:
Camera
↓
Face Detection
↓
Known Face Encoding
↓
Authentication
↓
Agent Access
There's also a development mode that allows authentication to be bypassed when working on the rest of the system.
This becomes especially interesting once an assistant can perform real computer actions.
An assistant that can open applications, interact with your browser, manipulate windows, and access local knowledge needs a different security model from a chatbot.
4. Diego Can See My Screen
One of the biggest differences between a chatbot and a desktop agent is context.
Diego can work with:
- screen captures
- OCR
- visual analysis
- screen context
- live visual requests
For example:
"What's on my screen?"
should not be answered using something Diego learned about my computer yesterday.
It needs the current state of the screen.
That's why Diego explicitly separates:
LIVE STATE
from:
HISTORICAL / LOCAL KNOWLEDGE
This distinction becomes extremely important as the system grows.
5. Diego Actually Controls the Desktop
This is where things start getting interesting.
Diego can perform real desktop operations, including:
- opening applications
- focusing applications
- switching windows
- closing windows
- minimizing windows
- maximizing windows
- listing running applications/windows
- interacting with desktop tools
But there is an important difference between execution and successful execution.
A naïve agent might do:
open_app()
↓
"Done!"
Diego's architecture is moving toward:
execute()
↓
observe state
↓
verify expected result
↓
success / failure
Because computers fail.
Applications don't always open.
Windows aren't always where you expect them.
Browser actions can behave differently.
And an AI saying "done" doesn't magically make something true.
6. Diego Knows About Its Own Computer
One of the surprisingly useful capabilities is deterministic system information.
Diego can retrieve information about:
- operating system
- kernel and architecture
- CPU
- CPU cores
- RAM
- GPU
- GPU memory
- storage
- disks
- network interfaces
- Python environment
- project environment
For example:
"How much RAM do I have?"
doesn't need an LLM.
The system snapshot already knows.
That means the request can be answered directly.
This led to one of the architectural principles I'm increasingly using:
Don't use an LLM to answer something the computer already knows.
7. Local Knowledge: 1,618 Documents and 33,051 Chunks
This is one of the parts of Diego I'm most interested in.
The current validated knowledge base contains:
Documents: 1,618
Chunks: 33,051
It can index different types of local content including:
- TXT
- Markdown
- DOCX
- XLSX
- CSV
- JSON
- XML
- HTML
- PPTX
- source code
- configuration files
- logs
- other supported textual formats
The knowledge system supports:
Incremental indexing
Hash-based change detection
Duplicate prevention
Deleted-file cleanup
Background scanning
Cancellation / resume behavior
Local embeddings
Keyword search
Hybrid retrieval
Source citations
Bounded LLM context
This isn't just a folder of documents.
It's becoming a local knowledge layer for the agent.
8. Diego Can Learn About the PC Without Treating Everything as Knowledge
One thing I was particularly careful about was security.
If an AI can scan a computer, you don't want it casually indexing:
.env
.ssh
.gnupg
credentials
API keys
session data
private keys
So the read-only learning pipeline uses:
Approved Roots
↓
Read-only Scanner
↓
Security Policy
↓
Extraction
↓
Chunking
↓
Embeddings / Keyword Index
↓
DuckDB
There are protections around sensitive paths, symlinks, credentials, sessions, keys, .env, .ssh, .gnupg, .git, .venv, node_modules, and other excluded locations.
The goal is simple:
Learn about the machine without turning secrets into searchable knowledge.
9. The LLM Doesn't Need to Be Involved in Everything
This is probably one of the biggest architectural improvements I've made.
Instead of:
User
↓
LLM
↓
Answer
Diego increasingly works like:
User Request
↓
Can this be answered
deterministically?
│
┌──────────────┼──────────────┐
│ │ │
System Fact Live State Local Knowledge
│ │ │
System Snapshot Live Tools Retrieval
│ │ │
└──────────────┼──────────────┘
│
▼
Reasoning Required?
│
▼
LLM
For strong local knowledge matches, Diego can answer without an LLM call.
For partial matches:
Local Evidence
↓
Small Bounded Context
↓
LLM Reasoning
And only when necessary does the request enter the normal LLM/tool pipeline.
This reduces unnecessary inference and keeps the system more deterministic.
10. The Problem With Raw RAG
Another problem appeared once the knowledge system became useful.
Retrieval systems naturally produce things like:
chunk metadata
embedding scores
file paths
document identifiers
duplicate evidence
That's useful internally.
It's terrible as a voice response.
Imagine asking:
"What do you know about Diego?"
and hearing:
/home/user/project/...
score=0.823...
chunk_id=...
/home/user/project/...
That's not an assistant.
That's a database having a nervous breakdown.
So Diego now has a presentation layer that transforms retrieval output into human-friendly responses.
It:
- removes raw retrieval dumps
- removes embedding scores
- removes chunk metadata
- deduplicates evidence
- limits response length
- preserves filenames when explicitly requested
- preserves paths when explicitly requested
- keeps action instructions intact
The retrieval system and the conversation system therefore have different responsibilities.
11. Web Search Is Another Source of Knowledge
Diego also has browser/search functionality.
It can:
- perform web searches
- extract search results
- open relevant results
- read extracted web content
- interact with the browser
The idea is not that the web replaces local knowledge.
Instead:
Local Knowledge
↓
Enough information?
↙ ↘
YES NO
↓ ↓
Answer Search Web
This gives Diego multiple knowledge sources rather than forcing everything through a single pipeline.
12. YouTube and Media Control
Diego also has explicit media behavior.
For example:
"Play <song> on YouTube"
is treated differently from:
"Search YouTube for <song>"
That sounds obvious, but distinguishing searching for something from performing an action is important for agent design.
Diego also has media controls such as:
- play
- pause
- resume
- media-state verification
Again, the important part is verification.
The assistant shouldn't claim:
"It's playing."
unless it has evidence that playback actually started.
13. Agentic Execution: Plan → Execute → Observe → Verify
This is probably the direction I'm most excited about.
The execution model is:
Goal
↓
Plan
↓
Execute Step
↓
Observe
↓
Verify
↓
Replan if Necessary
↓
Continue
↓
Verified Completion
There are safeguards around:
- maximum steps
- retries
- replans
- timeouts
- loop detection
- idempotency
- task state
Why?
Because giving an LLM access to tools without boundaries is basically saying:
"Here are some buttons. Good luck."
That's not an agent architecture I'd want running continuously on my machine.
14. Local Models Instead of Permanent Cloud Dependency
This is probably the most controversial part of the project.
I don't want Diego's core existence to depend on:
API key
↓
Internet
↓
Cloud provider
↓
Model
Diego supports local LLM usage through Ollama.
Cloud/provider paths can exist where configured, but local functionality isn't inherently dependent on cloud models.
For me, local inference isn't about saying:
"Cloud AI is bad."
Cloud models are incredibly useful.
It's about having another option:
What if your personal AI could actually belong to your machine?
Running locally also forces you to confront a problem that disappears when you throw unlimited cloud compute at everything:
Optimization.
15. The Real Challenge: Making AI Fit on a PC
A desktop AI agent isn't just an LLM.
You're potentially running:
Voice processing
+
Wake word
+
Speech recognition
+
LLM inference
+
Embeddings
+
Vector / keyword retrieval
+
OCR
+
Screen capture
+
Browser automation
+
Background indexing
+
Desktop automation
+
TTS
All on one machine.
That means you start caring about:
- RAM
- VRAM
- CPU utilization
- GPU utilization
- model loading time
- context size
- background concurrency
- disk I/O
- inference latency
- caching
- startup time
This completely changes how I think about AI engineering.
Instead of:
"Can I run this model?"
the better question becomes:
"Can I run the entire system reliably?"
16. Model Warm-Up Made a Huge Difference
One optimization that produced a measurable result was model warm-up.
The current measurements were approximately:
Cold: ~2459 ms
Warm: ~306 ms
That's a dramatic difference.
The idea is simple:
Instead of repeatedly paying the cost of loading a model, Diego can keep the model warm for a configurable period.
This is a small example of a broader principle:
AI performance isn't only about model inference speed.
The surrounding system matters just as much.
17. Background Work Shouldn't Block the Assistant
Another major focus has been making expensive operations happen in the background.
Things like:
- knowledge indexing
- scanning
- embeddings
- model warm-up
- diagnostics
shouldn't unnecessarily block the interactive voice loop.
Diego therefore uses things like:
Background processing
Bounded concurrency
Timeouts
Cancellation
Caching
Incremental indexing
Graceful fallback
This is where an AI project starts looking less like a prompt-engineering project and more like a distributed systems problem running inside a desktop application.
18. Testing the Agent, Not Just the Model
Another thing I care about is validation.
The latest full validation currently reports:
557 passed
5 skipped
That's important to me because an AI agent isn't useful if its demo works once.
It needs to survive changes.
Especially when you have interconnected components like:
Voice
↓
Routing
↓
Knowledge
↓
Tools
↓
Desktop
↓
Verification
A change in one component can break something somewhere else.
So automated testing becomes part of the AI architecture—not an afterthought.
19. What Diego Is Today
I don't describe Diego as an AGI.
And I don't call it a fully autonomous self-improving AI.
There are still major things missing.
For example:
❌ True autonomous long-term self-improvement
❌ Reliable self-diagnosis + repair
❌ Deep project relationship graphs
❌ High-quality continuous personal learning
❌ Fully autonomous background task execution
❌ Strong long-term episodic memory
❌ Comprehensive multimodal personal knowledge
❌ Automatic safe remediation
Those are future engineering problems.
And I'd rather be precise about what exists than inflate a project with AI buzzwords.
20. Where I Want to Take Diego
The direction I'm exploring is a much more capable personal AI architecture.
Something that can:
┌───────────────┐
│ DIEGO │
└───────┬───────┘
│
┌──────────────┼──────────────┐
│ │ │
PERCEIVE REMEMBER ACT
│ │ │
Screen Knowledge Desktop
Voice Memory Browser
Vision Context Tools
│ │ │
└──────────────┼──────────────┘
│
REASON
│
VERIFY
│
IMPROVE
The interesting part isn't adding another chatbot feature.
It's making these systems work together reliably.
The Bigger Question
The more I build Diego, the more I think the interesting AI question isn't:
"How smart is your model?"
It's:
"How much can your AI actually do?"
A model can have an enormous context window.
It can score incredibly well on benchmarks.
It can write beautiful code.
But a personal AI becomes fundamentally different when it can:
listen → see → understand → retrieve → reason → act → observe → verify
And it can do those things on your own machine.
That's what I'm trying to build with Diego.
Not another chatbot.
Not another wrapper around an API.
A local desktop agent that can actually interact with the environment it lives in.
And the most difficult part?
Making all of it work together without needing a datacenter under my desk.
That's the fun part.
Diego is still being built.
And I'm just getting started.
Technical Stack & Concepts I'm Exploring
The project currently involves concepts across:
- Local LLM inference
- Ollama
- RAG
- Local embeddings
- Hybrid retrieval
- DuckDB
- Speech recognition
- VAD
- Wake-word detection
- TTS
- OCR
- Computer vision
- Browser automation
- Desktop automation
- Agentic planning
- Action verification
- Background processing
- Caching
- Incremental indexing
- Security-aware filesystem scanning
- Runtime diagnostics
- Automated testing
The goal isn't to use every piece of technology possible.
It's to figure out which pieces actually make a personal AI reliable.
One final thought
We're currently obsessed with making AI models bigger.
I'm increasingly interested in making AI systems better integrated with the environments they operate in.
Because maybe the future of personal AI isn't:
"Open a chat and ask."
Maybe it's:
"It's already there. It knows what's happening, understands the context, and can actually do something about it."
_That's Diego.
_
Let's connect
I'm sharing the development of Diego, the problems I'm solving, and the experiments I'm running along the way.
🔗 GitHub: GitHub
🌐 Portfolio: NASHEDIxCODER
💼 LinkedIn: me
If you're interested in local AI, AI agents, RAG, voice interfaces, desktop automation, or building AI systems from scratch, I'd love to connect and hear what you're working on.
Top comments (0)