Magnus AI: a real-time voice assistant that controls my Windows PC

What I Built

Magnus AI is a voice-first desktop assistant for Windows. You talk to it, and it can:

• Open apps and bring windows to the front
• Set volume and brightness
• Run one-phrase routines like "Work Mode", "Study Mode" and "Night Mode"
• Search the web, play YouTube, and set reminders
• Read your clipboard
• Look at your screen or webcam and help debug code

Demo

https://drive.google.com/file/d/137xF8eviOjMR5DFXnfyZKzUN_IJyS8Br/view?usp=sharing

Code

https://github.com/RIDDHIDEV-OPS/Magnus-AI

How It Works

• Voice loop:microphone audio is resampled to 16 kHz mono and streamed over a WebSocket to the Gemini Live API. Replies return as 24 kHz audio, with about 250 ms turn detection and barge-in so you can interrupt.
• Self-describing tools:each file in actions/ exposes one typed function, and its docstrings become function declarations at startup. A new skill is one new file.
• Windows control:Win32 APIs, psutil and PyAutoGUI.
• Interface:a PyQt6 HUD with an animated orb, waveforms and live CPU/RAM/GPU telemetry.
• Memory:preferences are stored in a local JSON file, git-ignored.
• Resilience:a fallback ladder across Gemini models when quota is hit.

What's Next

Magnus currently relies on Google's hosted Gemini model. My next step is an offline mode built on open-weight models (Gemma via Ollama plus faster-whisper), so voice and screen data never leave the machine.