NamiAgent Ships a From-Scratch Local Agent Tutorial — Voice Still Dies Before the First MCP Tool Call

On 7 November 2026 the NamiAgent repository (aa1143/NamiAgent) crossed a few hundred stars as a runnable local agent tutorial. The README is blunt: start from a ~100-line while loop that calls an OpenAI-compatible model, executes tools, and feeds results back into context. Lessons 01–11 add tools, permissions, context management, skills, a scheduler, a stdio MCP client, sub-agents, planning, and long-term memory. Lesson 11 ships 49 tests that run against a fake model so you never need an API key. A local web UI (python web/app.py) shows each model turn and the full context on the right. Dependencies stay at the openai package; everything else is plain Python 3.10+.

That is exactly the shape a lot of practitioners want when they are tired of framework soup. The moment the input is speech instead of a typed prompt—an interview answer, a stand-up, a customer call, or a multilingual review—the pipeline lengthens by everything that sits between the microphone and a clean string that the agent loop can consume.

The 100 ms that never appear in the tutorial

A typical Windows desktop capture stack for a call or interview looks like this:

1. Endpoint (USB headset, Bluetooth HFP/A2DP, built-in mic, or WASAPI loopback).
2. Windows Audio Engine shared-mode mix plus any APO (communications enhancement, spatial, vendor noise suppression).
3. Your client’s period and buffer (practical range 10–40 ms before you see a stable frame).
4. Local VAD (Silero, a tiny ONNX model, or energy + hangover).
5. Streaming or chunked STT (GGUF, ONNX, or a quantized Whisper/Parakeet/Qwen-ASR variant).

The numbers that never appear in agent-framework benchmarks are the first three steps. Recent measurements of small local ASR models put time-to-final segment in the 35–50 ms range once samples have already left the capture device. On a modern laptop the engine + client contribution is commonly 30–90 ms before the first sample reaches your process. Exclusive mode can shrink the engine part by bypassing the shared mixer and APOs, but then you fight device exclusivity and the rest of the desktop.

If your local model finishes inside the remaining budget, end-to-end from speech end to on-screen text can stay under a couple of hundred milliseconds without shipping the audio off the machine. That transcript is what you can feed into the same style of loop NamiAgent demonstrates, or into an MCP tool that the agent already knows how to call.

Wiring a transcript into the agent loop without another round-trip

Once you have clean text, the useful work is deterministic extraction or a direct tool call, not another free-form prompt. A minimal Python sketch that turns a short meeting snippet into something an agent can act on looks like this:

import re
from typing import Dict, List

def extract_actionable(text: str) -> Dict:
    # crude but deterministic; replace with a small local classifier or regex set
    bugs = re.findall(r"(bug|issue|fails|broken|crash)[^.!?]*", text, re.I)
    steps = re.findall(r"(step|when|if|then)[^.!?]*", text, re.I)
    return {
        "title": (bugs[0] if bugs else text[:60]).strip(),
        "steps_to_reproduce": steps[:3],
        "raw": text,
        "language": "en"  # or detect from the STT language id
    }

# feed the dict to the agent loop or POST to an MCP tool
payload = extract_actionable(transcript)

NamiAgent’s lesson-08 already shows a stdio MCP client. You can drop the same payload into a custom tool that writes a structured note, opens a local file, or calls a remote endpoint—without the model having to re-parse the raw speech. The web UI that ships with the project makes the context growth visible; watching the token count climb while you are still waiting for the next utterance is a useful reminder that the capture path owns the critical path.

Tradeoffs that actually matter on Windows

Shared-mode WASAPI is convenient and survives device changes, but every APO and the shared mixer add latency and can inject communications enhancements that distort the signal before VAD even sees it. Exclusive mode drops most of that path and can get you closer to the device period, yet it takes the endpoint exclusively—system sounds and other apps go silent. Loopback capture (for system audio or a virtual cable) has its own buffering quirks.

Local STT keeps the audio on the machine and removes the network RTT, but you pay in CPU/NPU and in model size. A 2B-class quantized model that finishes a short utterance in under 100 ms is usable; a larger model that needs a full second is not, no matter how accurate the final transcript. Memory management inside the agent loop (NamiAgent’s lesson-11 style confirmation before writes) is orthogonal: you still need the text to arrive before you can decide whether to confirm.

The practical architecture that keeps working is therefore:

• Capture and VAD in the same process (or a tightly coupled helper) so you control the buffer sizes.
• Stream or chunk the audio to a local STT runtime that returns partials quickly.
• Emit structured text (or a small JSON payload) into the agent’s context or an MCP tool.
• Let the agent loop do the tool calling and permission checks exactly as the tutorial demonstrates.

Cloud agents and MCP servers are happy to consume text. They are not responsible for the 30–90 ms of shared-mode WASAPI buffering, the communications enhancements, or the VAD that decides when an utterance actually ended.

What ships today

NamiAgent gives you a clean, inspectable local agent loop with MCP and memory and a web UI that shows every step. Pairing it with a local capture + STT pipeline turns spoken input into something the loop can act on without leaving the machine for the first hop. The remaining latency budget is still owned by the Windows audio path and the VAD/STT choices you make before the first tool call.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.