LocalPilot: Building a Voice-Controlled Desktop Assistant with Cactus AI

Anas Dharar — Mon, 05 Oct 2026 06:51:54 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built LocalPilot, a local-first, voice-controlled desktop assistant for Windows.

The idea came from a friend who often had to interrupt his workflow to switch between applications and navigate system settings for simple tasks. I wanted to explore whether he could get those things done just by speaking naturally.

With LocalPilot, you can say things like "Open Calculator", "Open leetcode.com", "Set a timer for one minute", or "Which applications are running?" The assistant transcribes your speech, interprets your request, and selects an appropriate tool to execute the action.

The project combines lightweight speech recognition with AI-powered tool calling to make everyday computer interactions more natural.

Demo

GitHub repository: AnasDharar/localpilot

The repository contains the project code and implementation details.

Code

Check out the source code here:

github.com/AnasDharar/localpilot

The project is built in Python, with a PySide6 desktop interface, Whistle for speech-to-text, and Needle for tool calling.

How I Built It

The core idea was to separate speech recognition, intent interpretation, and action execution rather than build a large collection of keyword-matching rules.

Cactus: The ecosystem behind the lightweight AI components used in the project.
Whistle: Converts microphone input into text, giving LocalPilot a transcript of the user's spoken command.
Needle: Handles tool calling, helping map natural-language requests to registered Python functions and their arguments.
Python and PySide6: Power the desktop application, interface, and Windows actions.

The intended flow is:

Voice → Whistle transcription → Needle tool selection → Validated Python function → Windows action → UI feedback

For example, a request such as "Open leetcode.com" can be interpreted as a website-opening action, with the URL validated before LocalPilot asks the default browser to open it.

I used GitHub Copilot Agent throughout development to help implement features, work through integration issues, and iterate on the application. I still had to make the architectural decisions, test the behavior, and refine the implementation around the project's goals.

Why Does Open Innovation Matter?

LocalPilot is an experiment in what becomes possible when developers can build on open-source tools and inspect how their systems work.

Instead of depending entirely on a closed, cloud-hosted assistant API, I could combine lightweight speech recognition with a tool-calling framework and write the operating-system integration myself.

That gives me more control over how commands are interpreted, which actions are available, and what validation happens before an action executes. It also makes the project easier to extend: adding a new capability can mean registering another well-defined Python tool instead of redesigning the entire assistant.

Open innovation also lowers the barrier to experimentation. A student developer can take existing AI building blocks and turn them into a practical desktop application without having to train every model from scratch.

My Agent Session

I used GitHub Copilot Agent to help develop LocalPilot, from implementing application features to refining the integration between its components.

Prize Categories

GitHub Copilot: I used GitHub Copilot Agent as a development tool throughout the project to help implement features, troubleshoot issues, and iterate on the codebase.

The Next Steps

There’s still a lot I want to improve in LocalPilot. My next priorities are making tool calling more robust, enabling chained tool calls so the assistant can execute multiple steps to complete a single request, adding a global hotkey to activate LocalPilot from anywhere on the desktop, and improving speech-to-text accuracy to handle different accents, background noise, and natural speech more reliably. The goal is to make LocalPilot feel less like a collection of individual commands and more like a seamless, dependable desktop assistant.

DEV Community: Anas Dharar