I Built a Desktop AI Assistant That Controls Your Computer — Here's How

Mohamed Heni — Mon, 25 May 2026 09:30:57 +0000

TL;DR: I built Yaldabaoth — a desktop AI assistant that doesn't just answer questions. It reads your screen, runs PowerShell commands, clicks buttons, types text, and automates your entire workflow. No cloud dependency. No API calls for automation. Just Python, Rust, and raw OS control.

The Problem

In 2025, "AI assistants" meant one of two things:

Chatbots — A text box where you type and an LLM talks back
API wrappers — Tools that chain API calls together but have zero ability to touch the actual operating system

Neither of these actually helps you do things on your computer. Want to open an application, take a screenshot, parse a PDF, run a PowerShell script, and compile a report? Good luck chaining that through a chat interface.

I wanted something different. An assistant that sits on your desktop, sees what you see, and acts on your behalf — like having an engineer sitting next to you who can operate any part of the system.

So I built Yaldabaoth.

The Architecture

Yaldabaoth is a four-layer system:

Layer 1: The Shell — Tauri + React

Instead of Electron (which would have added 150+ MB to the binary), I used Tauri — a Rust-based framework that wraps a webview frontend in a native shell. The UI is React with a glassmorphism design.

Why Tauri:

Binary size: ~5 MB vs Electron's ~150 MB
Native performance for intensive operations
Direct Rust system access when needed

Layer 2: The Orchestrator — Python Backend

The Rust shell communicates with a Python backend that handles all the heavy lifting. Communication is via stdin/stdout JSON protocol — lightweight, no HTTP server needed.

The orchestrator manages:

Command routing (voice/text → appropriate handler)
State persistence between commands
Multi-step task chaining
Personality profile switching (Professional vs. creative modes)

Layer 3: The Automation Engine — Win32 API + PowerShell

This is where the magic happens. The Python backend has direct access to the Windows OS through:

pywinauto — Native Win32 API control for clicking, typing, window management
PowerShell subprocess — OS-level commands (service control, registry edits, file operations)
WMI — System information queries (processes, hardware, network)

Layer 4: The Perception System — OCR + Screen Parsing

Screen parsing runs in a separate thread to keep the UI responsive. It uses:

OCR-based text extraction — Screenshots → text → action decisions
Multi-threaded processing — One thread for screen capture + OCR, another for command execution, a third for UI responsiveness
Chained automation — Click → wait for UI update → re-scan screen → next action

The Hard Parts

Threading Nightmares

Screen parsing is slow. OCR a screenshot, parse the text, decide what to do — you're looking at 500ms to 2 seconds per cycle. If you do this on the main thread, your entire app freezes.

The solution was a producer-consumer architecture:

Thread 1: Screen capture → OCR → queue the parsed text
Thread 2: Command executor — reads from queue, takes action
Thread 3: Main UI thread — stays responsive

┌─────────┐    ┌──────────┐    ┌──────────┐
│ Screen  │───>│ Queue     │───>│ Command  │
│ Capture │    │ (JSON)   │    │ Executor │
└─────────┘    └──────────┘    └──────────┘
      │                              │
      v                              v
 ┌─────────┐                   ┌──────────┐
 │ OCR     │                   │ Win32 API│
 │ Parser  │                   │/PowerShell│
 └─────────┘                   └──────────┘

Python's threading.Queue with daemon=True threads was sufficient — no need for multiprocessing or async for this use case.

Voice-First vs. Text-First UX

I wanted a Push-to-Talk interface (F10 key) so you could speak commands naturally. But speech recognition introduces latency and errors. The compromise:

Voice input preferred for simple commands ("Open Chrome", "Check CPU usage")
Text fallback for complex multi-step sequences
The orchestrator normalizes both into the same command pipeline

Rust ↔ Python Bridge

Tauri apps expect Rust backends. Yaldabaoth needs Python. Bridging them without adding a web server was tricky.

The solution: stdin/stdout JSON-RPC. The Rust shell spawns the Python process and communicates via JSON messages on stdin/stdout. No sockets, no HTTP, no dependency on a running server. The Python process lives as long as the app is open.

// Rust side — minimal example
let python = Command::new("python")
    .arg("backend/main.py")
    .stdin(Stdio::piped())
    .stdout(Stdio::piped())
    .spawn()?;

// Send command
let cmd = r#"{"action": "click", "target": "Chrome"}"#;
python.stdin.as_ref().unwrap().write_all(cmd.as_bytes())?;

// Read response
let mut response = String::new();
python.stdout.as_ref().unwrap().read_to_string(&mut response)?;

What I Learned

1. Desktop automation is harder than cloud automation. Cloud APIs are designed to be called programmatically. Desktop UIs are designed for humans. Parsing a rendered UI and making decisions from it is fundamentally different from calling an API endpoint.

2. Threading early, threading often. I rebuilt the threading model three times. The first version was single-threaded and froze constantly. The second over-engineered with multiprocessing. The third — simple Queue-based threading — was just right.

3. Computer use was already possible in 2025. Before "computer use" became a buzzword in 2026, a Python script + OCR + Win32 API was all you needed. The novelty isn't the technology — it's wiring it together with a voice-first, responsive UI that feels like an assistant, not a script.

The Tech Stack

Component	Technology
Shell	Tauri (Rust + WebView2)
Frontend	React
Backend	Python
Automation	pywinauto, WMI, PowerShell
Voice	Push-to-Talk (F10)
Screen parsing	OCR + multi-threaded pipeline
Binary size	~5 MB

What's Next

Cross-platform support — Currently Windows-only due to Win32 API dependency. Linux adaptation via X11/Wayland is on the roadmap.
Better screen parsing — Using vision models directly instead of OCR for richer UI understanding.
Plugin system — Let users write custom automation modules.

Repo

The full source code is on GitHub: github.com/HENI-MOHAMED/Yaldabaoth

Built with Tauri, React, Python, Rust, and more coffee than I'd like to admit.

Building a Multi-Agent Tax Audit System with LangGraph and Odoo

Mohamed Heni — Fri, 22 May 2026 00:34:31 +0000

TL;DR: I built a multi-agent system that audits invoices, detects fiscal inconsistencies, and generates compliance reports — integrated with Odoo ERP and standalone databases.

The Problem

Tax auditing for small and medium businesses in Tunisia is manual, error-prone, and slow. Most accounting firms rely on Excel and manual checks. Regulations change frequently, and keeping up is a full-time job.

I wanted to build something that could ingest financial data, run audit rules automatically, and produce compliance reports — without needing a team of accountants.

The Architecture

The system uses a LangGraph-based multi-agent architecture:

Agent 1: Data Ingestion Agent

Connects to Odoo ERP or standalone SQL databases
Extracts invoices, ledgers, and financial statements
Normalizes data into a unified schema

Agent 2: Audit Agent

Applies tax rules against the data
Detects anomalies: missing invoices, misclassified expenses, VAT discrepancies
Flags items for human review

Agent 3: Reporting Agent

Generates compliance reports
Produces a summary of findings with risk levels
Suggests corrective actions

Orchestrator

LangGraph manages the flow between agents
Handles state, retries, and error recovery

Technical Challenges

Schema Mismatch: Every Odoo instance is customized differently. The ingestion agent had to handle dynamic schemas — detecting table structures at runtime and mapping them to a canonical audit model.

Multi-Agent Coordination: Getting three agents to work together without stepping on each other's state was the hardest part. LangGraph's checkpointing was essential here.

Regional Tax Rules: Tunisian tax law isn't well-documented in English. Building the rules engine meant working directly with Arabic and French regulatory texts.

What's Next

Real-time invoice validation
Multi-company support
A dashboard for non-technical accountants

The repo is at github.com/HENI-MOHAMED/Audit-Agent.

Built with Python, FastAPI, LangGraph, Odoo, and a lot of coffee.

DEV Community: Mohamed Heni

I Built a Desktop AI Assistant That Controls Your Computer — Here's How

The Problem

The Architecture

Layer 1: The Shell — Tauri + React

Layer 2: The Orchestrator — Python Backend

Layer 3: The Automation Engine — Win32 API + PowerShell

Layer 4: The Perception System — OCR + Screen Parsing

The Hard Parts

Threading Nightmares

Voice-First vs. Text-First UX

Rust ↔ Python Bridge

What I Learned

The Tech Stack

What's Next

Repo

Building a Multi-Agent Tax Audit System with LangGraph and Odoo

The Problem

The Architecture

Agent 1: Data Ingestion Agent

Agent 2: Audit Agent

Agent 3: Reporting Agent

Orchestrator

Technical Challenges

What's Next