SINK

I built SINK, a voice assistant for Windows that lets you control your laptop without using your hands.

I built it for my friend mahima. she can't use her hands properly because she applies henna (mehdi in indian language), so basic things like opening a browser, clicking a button, scrolling or searching a homework answer became very hard. Windows has voice control, but it only follows commands. It can't see what's on the screen or help with the actual work.

With SINK, you say "SINK" and then a command:

• Control the laptop: "open Chrome", "scroll down", "type hello", "search Python tutorials". It starts listening on launch, so no mouse is needed. It understands some Hinglish commands too.
• Click by name: "SINK click Send" finds the button through Windows UI Automation and clicks it, with no pixel guessing.
• See the screen: "SINK what's on my screen" and "SINK explain this error" use a local vision model.
• Solve copied questions: "SINK solve what I copied" opens a page with properly rendered maths and a Final Answers box at the top. Or say "in Notepad" for plain text. It always warns: "Please check it once, I can make mistakes."
• Show it's working: a small chibi robot shows "SOLVING..." with moving dots, so [friend's name] knows it hasn't frozen.
• Dictation, reminders, to-do notes and a small memory, all by voice.

screenshot of icon

Code

still4shrishti / SINK-

SINK | A Python-based AI desktop assistant that listens, talks, automates tasks, and reacts through an interactive virtual character.

SINK: Local Voice AI Desktop Assistant

SINK is a Jarvis-style voice assistant that runs fully on your own PC Say "SINK" and give a command, and it will open apps, click, scroll, type, set reminders, play games and chat with you It has a small animated robot face that reacts to what you say.

• Brain: Qwen 3, running locally through LM Studio
• Ears: faster-whisper (speech to text, on CPU)
• Voice: pyttsx3 (text to speech)
• Face and window: PySide6 (Qt)
• Hands: pyautogui (mouse and keyboard control)

Your voice and notes are not sent to any cloud service. Everything stays on your computer.

---

Features

Talk to it

• Wake word: say "SINK" (also hears "sync", "think", "zinc"). Example: "SINK open Chrome"
• Hands-free listening toggle, or type commands in the text box
• Answers general questions using Qwen 3 from LM Studio
• Chain commands: "press click and then scroll down"

Control the PC

Say

…

View on GitHub

Setup: pip install reqirements, start LM Studio's local server with a vision model, then run python gui.py.

How I Built It

Everything runs on the laptop. Nothing is sent to a cloud service.

• Speech to text: Whisper through faster-whisper, running locally on CPU.
• Language and vision model: open-weight models served locally through LM Studio's OpenAI-compatible API ([model names, e.g. Qwen3-VL for screen reading and solving]).
• Interface: Python with PySide6: the main window, particle "core", and the chibi companion.
• Computer control: pyautogui for mouse and keyboard, pywinauto (Windows UI Automation) to click buttons by name.
• Voice output: pyttsx3.

Parts I had to solve myself:

• Mishearing: Whisper often mishears short commands ("click" vs "tap"), so I wrote a layer that maps many ways of saying something to one command.
• Maths display: small models write maths in LaTeX, which looks broken in Notepad. I added a converter to plain symbols and a browser page that renders the maths properly.
• Safety: a blocked-terms list for protected system areas, and a mistake warning on every solution.

Why Does Open Innovation Matter?

My friend's screen can show private things like messages, school work and logins. Because the models are open-weight and run locally, that screen content never leaves the laptop. With a closed cloud API I would have been sending screenshots of a real person's screen to a third party, which I wouldn't be comfortable asking my friend to accept.

It also means no API bills, no rate limits, and no dependence on a company's pricing or policy. A student with a normal laptop can build an accessibility tool for someone they know. And because the tools (Whisper, local models, PySide6, pywinauto) are open, I could combine them in ways no single product offered.