Voice Agent

Building a Local-First Voice AI Agent: Architecture, Models, and Constraints

The demand for capable, privacy-preserving AI agents is growing, but developing these systems to run entirely on local consumer hardware presents a strict set of engineering constraints. Cloud-based agents can afford to use massive, generalized models in complex cyclic loops. Local agents, constrained by limited memory (6GB to 8GB of VRAM), require a much more deliberate approach.

To explore these constraints, I developed VoiceAgent, a voice-controlled, completely offline AI assistant capable of processing speech, detecting user intents, writing code, and executing sandboxed file system operations.

This article outlines the architecture of the system, the rationale behind the selected models, and the engineering challenges overcome during the build.

System Architecture: The Deterministic Pipeline

When designing autonomous agents, the industry standard is often a ReAct (Reasoning and Acting) loop, typically orchestrated by frameworks like LangGraph. In these architectures, the LLM determines when to call a tool, evaluates the output, and decides internally when to finish.

While powerful for frontier models (like GPT-4), cyclic architectures are highly unstable for local models under 10 billion parameters. Small models frequently hallucinate tool parameters, invent nonexistent files, or fall into infinite execution loops.

To solve this, VoiceAgent abandons the cyclic loop in favor of a deterministic pipeline powered by structured outputs:

1. Input and Transcription

Audio is captured via the user interface and transcribed entirely offline using Hugging Face's Whisper models.

2. Intent Routing via Structured Outputs

The transcribed text is passed to a Router LLM. Instead of generating free-form text, the router model is strictly constrained to a Pydantic schema (JSON). Its sole purpose is to map the user's natural language to a predefined action plan containing an intent (e.g., create_file, write_code, summarize_text) and the extracted arguments.

3. Plan Normalization and Selective HITL

A Python middleware validates the model's output. If the requested intent modifies the file system (creating directories or writing code), the pipeline pauses and requires explicit Human-In-The-Loop (HITL) approval before proceeding. Safe operations, such as reading or summarizing an existing file, bypass this check and execute immediately.

4. Deterministic Execution

Once approved, a Python executor runs the determined tools. The LLM does not interact directly with the file system; it only provides the validated arguments to the deterministic Python functions.

This linear approach effectively eliminates tool-calling hallucinations, ensuring that the system either executes exactly what was structured or fails gracefully.

Engineering Challenges and Model Selection

Running an end-to-end agent on limited local hardware requires ruthless optimization. You cannot simply load a monolithic 70B parameter model. Instead, VoiceAgent relies on a split-model architecture, delegating specific tasks to specialized, smaller models.

Challenge 1: The Schema Adherence Problem

Smaller models famously struggle to output valid, parseable JSON. They often inject conversational filler, which breaks parsing logic.

During development, I benchmarked several models for the intent routing task. While llama3.1:8b was slightly faster in pure generation speed (averaging 4.8 seconds for a complex routing request), qwen2.5:7b-instruct-q4 was selected as the designated Router LLM. Despite a slightly slower inference time (5.4 seconds), Qwen 2.5 demonstrated vastly superior reliability in adhering to strict JSON schemas without hallucinating extraneous text.

Challenge 2: Context Dilution

Providing an AI agent with access to a file system often involves adding the entire directory tree to the system prompt. On models with smaller context windows and limited reasoning capabilities, this rapidly degrades performance and leads to confused outputs.

To mitigate this, VoiceAgent utilizes targeted context injection. The full directory tree is never blindly passed to the router. Instead, the system relies on specialized generation models. When the router identifies a code generation task, the pipeline hands the task off to qwen2.5-coder:7b. When text summarization is required, it utilizes llama3.1:8b. This compartmentalization keeps prompts lean and generation high-quality.

Challenge 3: System Security and Path Traversal

Autonomous file writing is inherently dangerous. A naive implementation might allow a model to generate arguments like ../../etc/passwd or overwrite critical project files.

To secure the agent, all file operations are strictly jailed within an isolated /output directory. The Python execution layer actively resolves absolute paths and programmatically blocks path traversal attempts. Furthermore, the root sandbox directory is protected against programmatic deletion, regardless of what the LLM requests.

Challenge 4: UI State vs. Agent Memory

A significant challenge arose when integrating the agent pipeline with the frontend interface. Because Streamlit operates on a continuous rerun cycle, uploaded or recorded audio blobs would persist in the widget state. This resulted in "ghost prompts," where the application would transcribe and execute the same audio file in an infinite loop upon every UI refresh.

This was resolved by implementing stateful cryptographic hashing. The system now hashes the payload of the audio blob and tracks it within the session state. This decouples the UI render cycle from the agent execution logic, ensuring that an audio command is only ever transcribed and processed once.

Conclusion

Building VoiceAgent demonstrated that highly capable, offline AI assistants do not require massive parameter counts or complex, opaque cloud frameworks.

By enforcing strict structured outputs, isolating high-risk logic behind a deterministic execution layer, and selectively applying Human-in-the-Loop oversight, it is entirely possible to build a safe, fast, and reliable agent that operates comfortably within the constraints of consumer hardware.