Building a Voice-Controlled Local AI Agent: A Journey into Speech-to-Text and Tool-Use

Amartya — Fri, 10 Apr 2026 15:23:19 +0000

In the era of Large Language Models (LLMs), the gap between "chatting with an AI" and "controlling your computer" is rapidly closing. Recently, I embarked on a project to build a Voice-Controlled Local AI Agent that allows users to manage their filesystem, generate code, and summarize text—all through natural speech.

In this article, I'll walk you through the architecture, the high-performance models I chose, and the unique challenges I faced along the way.

The Vision

The goal was simple but ambitious: create a specialized agent that accepts audio input (via mic or file upload), understands the user's intent, and executes the appropriate local tool (like creating a file or writing a Python script).

The Architecture

The agent is built on a "Three-Step Pipeline" designed for speed and reliability:

Speech-to-Text (STT): Converting raw audio into clean, actionable text.
Intent Classification: Using an LLM to "parse" the text into a structured JSON object (Intent + Arguments).
Tool Execution: mapping the classified intent to local Python functions that interact with the filesystem.

For the frontend, I chose Streamlit. It provided a clean, dark-themed UI that allowed for rapid prototyping of audio input widgets and real-time status updates for the user.

Model Selection: Choosing Speed Over Bulk

Because local hardware can often be a bottleneck for heavy models like Whisper or Llama-3, I opted for a hybrid API-first approach to ensure the agent felt "instant."

1. Speech-to-Text: OpenAI Whisper-large-v3 (via Groq)

I chose Groq's implementation of Whisper-large-v3. It is arguably the fastest STT API available today, transcribing audio in sub-second times. This is crucial for a voice agent; any lag in transcription makes the interaction feel clunky.

2. The Brain: GPT-4o-mini & Llama-3.1-8b

For the logical "brain," I supported two providers:

GPT-4o-mini: Exceptional at "Structured Outputs" and following strict JSON schemas.
Llama-3.1-8b-instant: Preferred for its sheer speed and efficiency on Groq, making the "thinking" process almost invisible to the user.

Challenges Faced (and Solved)

1. The "Strict Schema" Struggle

One of the biggest hurdles was implementing Strict JSON schemas with OpenAI. OpenAI's newest structured output features require every object in the schema to explicitly forbid additional properties (additionalProperties: false).

Solution: I leveraged Pydantic's ConfigDict(extra='forbid') and redesigned the models to move away from generic "argument" dictionaries toward explicit, typed fields. This resolved 400-level API errors and made the tool calling much more robust.

2. Multi-Provider Orchestration

Handling different APIs (OpenAI vs. Groq) meant dealing with different SDKs and parsing logic. While OpenAI supports a convenient .parse() method for Pydantic, Groq requires a manual fallback using JSON mode.

Solution: I built a unified VoiceAgent class that abstracts these differences, allowing the user to toggle between "speed" (Groq) and "standard" (OpenAI) with a single click in the UI.

3. Local Safety

Allowing an AI to write files to your drive is inherently risky.

Solution: I implemented a strict "Clamping" policy. All tool executions are restricted to a dedicated output/ folder, ensuring the agent can't accidentally overwrite system files or escape its sandbox.

The Result

What started as a simple voice-to-text project evolved into a sophisticated local assistant. By combining Streamlit for the UI, Groq/OpenAI for the heavy lifting, and Pydantic for structured logic, the agent can now turn a spoken sentence like "Create a Python file for a Fibonacci sequence" into a saved script in less than two seconds.

Future Work

The next step is to add Function Calling capabilities allowing the agent to browse the web or interact with local databases. The future of the local agent isn't just about hearing; it's about doing.

Found this interesting? You can check out the full source code and documentation in my repository. REPO LINK

DEV Community: Amartya