Building a .NET Dev Agent, Part 2: Give It a Voice

The first time my dev agent answered out loud, it took sixty-eight seconds to tell me that four tests had failed.

It was correct, and it was thorough: four test names read out in full, AddedLines_reports_new_file_numbers_across_hunks included, each with its line number and its expected value, and a closing paragraph offering to look at the parser. On screen that answer is a glance. Through headphones it is a minute of standing still, waiting to hear the one sentence that mattered, which came last: every failure was off by one.

Wiring speech-to-text and text-to-speech onto an agent takes an afternoon. This post is about what happens after that afternoon, when you discover that an answer written to be read is the wrong shape to be heard, and that the transcript your agent receives is wrong roughly every other word. Both problems turn out to be design problems, not audio problems, and everything below is measured on the agent from Part 1.

Where Part 1 left the agent

Building a .NET Dev Agent, Part 1: Teach It Your Actual Workflow built three typed tools on top of a Claude tool_use loop: git_status, dotnet_build and dotnet_test, each one filtering its command's output down to what a model needs. Those three tools come over unchanged, with one exception: Part 1's RunAsync helper had a bug that only shows up on a machine with two .NET SDKs, and building this post found it. Part 1 now carries the fix; the details are near the end of this post.

Here is the whole loop:

1. Hold a key, talk, release.
2. A local Whisper turns the audio into text.
3. The agent runs, with Part 1's tools plus one new one.
4. A local Piper voice reads the reply back, and anything that should not be read goes to the screen.

Everything that touches audio runs locally, from open-source models, with no account needed for anything but Claude itself.

Two containers

Speech-to-text is whisper.cpp's whisper-server, which takes a WAV file over HTTP and returns JSON. Text-to-speech is Piper, whose HTTP server takes JSON and returns a WAV file. Both run on CPU.

services:
  whisper:
    # v1.9.5. The image is tagged by commit; ":main" moves every day.
    image: ghcr.io/ggml-org/whisper.cpp:main-d1be6fde11ac6e0407606b4e42fe72d34add8037
    # The image's entrypoint is `bash -c`, which runs only its FIRST argument. A plain
    # string gets split into words, so every flag after "whisper-server" is silently
    # dropped and it starts on the image's bundled model, bound to 127.0.0.1.
    # A one-element list keeps the whole command line as that first argument.
    command: ["whisper-server --host 0.0.0.0 --port 8080 --model /models/ggml-${WHISPER_MODEL:-tiny.en}.bin --threads 4"]
    volumes:
      - ./models:/models:ro
    ports:
      - "127.0.0.1:8080:8080"

  piper:
    build: ./piper
    ports:
      - "127.0.0.1:5000:5000"

Piper has no official image with its HTTP server, so it gets a five-line piper/Dockerfile. The voice is downloaded at build time, so the container starts without the network:

FROM python:3.12-slim
RUN pip install --no-cache-dir "piper-tts[http]==1.8.0"
WORKDIR /voices
# Bake the voice into the image so the container starts offline.
RUN python -m piper.download_voices en_US-lessac-medium --data-dir /voices
EXPOSE 5000
CMD ["python", "-m", "piper.http_server", "--host", "0.0.0.0", "--port", "5000", "--data-dir", "/voices", "-m", "en_US-lessac-medium"]

The Whisper model is a file you download once into models/:

curl -L -o models/ggml-tiny.en.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin
docker compose up -d

The comment on command: is there because I lost an hour to it. whisper.cpp's image uses bash -c as its entrypoint, and bash -c only runs its first argument. Written as a plain string, Compose splits the command into words, so bash -c receives whisper-server as its script and every flag after it as an ignored positional parameter: the server starts on the model bundled inside the image, bound to 127.0.0.1, where nothing outside the container can reach it. Written as a folded multi-line string it fails more quietly. The line break ends the command, so the server listens where you asked but loads the bundled model. My first run asked for large-v3-turbo, got base.en, and I spent a while blaming my accent. The one-element list keeps the whole command line together.

Pin the image by commit, too. :main is rebuilt daily, and the tag above is the one built from the v1.9.5 release.

Capture: push-to-talk, at the format Whisper wants

The .NET side needs one native dependency for audio. PortAudioSharp2 ships PortAudio's binaries for Windows, Linux and macOS inside the package, so there is nothing to install:

dotnet new console -n VoiceDevAgent
dotnet add package Anthropic --version 12.57.0
dotnet add package PortAudioSharp2 --version 1.0.6

Recording happens at 16 kHz, mono, 16-bit, which is exactly what Whisper consumes, so nothing gets resampled on either side. PortAudio delivers audio on its own thread through a callback, and that callback has one job: copy the samples somewhere and return, without ever waiting on the rest of the program. A Channel<T> is the right somewhere, because writing to an unbounded one never blocks.

using System.Buffers.Binary;
using System.Runtime.InteropServices;
using System.Threading.Channels;
using PortAudioSharp;
using PaStream = PortAudioSharp.Stream;

/// <summary>Push-to-talk capture: 16 kHz, mono, 16-bit — the format Whisper wants, so nothing resamples.</summary>
sealed class Microphone
{
    public const int SampleRate = 16_000;

    public static async Task<byte[]> RecordAsync(Func<CancellationToken, Task> waitForStop, CancellationToken ct)
    {
        // The callback runs on PortAudio's audio thread. It must never wait on anything the
        // rest of the program does, so it only copies samples into an unbounded channel,
        // whose writes never block.
        Channel<short[]> chunks = Channel.CreateUnbounded<short[]>(new() { SingleReader = true, SingleWriter = true });

        PaStream.Callback callback = (IntPtr input, IntPtr output, uint frameCount,
            ref StreamCallbackTimeInfo timeInfo, StreamCallbackFlags flags, IntPtr userData) =>
        {
            short[] chunk = new short[frameCount];
            Marshal.Copy(input, chunk, 0, (int)frameCount);
            chunks.Writer.TryWrite(chunk);
            return StreamCallbackResult.Continue;
        };

        int device = PortAudio.DefaultInputDevice;
        if (device == PortAudio.NoDevice) throw new InvalidOperationException("No input device found.");

        StreamParameters parameters = new()
        {
            device = device,
            channelCount = 1,
            sampleFormat = SampleFormat.Int16,
            suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency,
            hostApiSpecificStreamInfo = IntPtr.Zero,
        };

        using (PaStream stream = new(parameters, null, SampleRate, 0, StreamFlags.ClipOff, callback, null!))
        {
            stream.Start();
            try
            {
                await waitForStop(ct);
            }
            finally
            {
                stream.Stop();          // blocks until the last callback has returned
                chunks.Writer.Complete();
            }
        }

        List<short> samples = [];
        await foreach (short[] chunk in chunks.Reader.ReadAllAsync(CancellationToken.None))
            samples.AddRange(chunk);

        return Wav.Encode([.. samples], SampleRate);
    }
}

/// <summary>Plays 16-bit mono PCM, and stops the moment the token is cancelled.</summary>
static class Speaker
{
    static readonly short[] Silence = new short[4096];

    public static async Task PlayAsync(byte[] wav, CancellationToken ct)
    {
        (short[] samples, int sampleRate) = Wav.Decode(wav);
        int position = 0;
        TaskCompletionSource finished = new(TaskCreationOptions.RunContinuationsAsynchronously);

        PaStream.Callback callback = (IntPtr input, IntPtr output, uint frameCount,
            ref StreamCallbackTimeInfo timeInfo, StreamCallbackFlags flags, IntPtr userData) =>
        {
            int remaining = samples.Length - position;
            int count = Math.Min(remaining, (int)frameCount);
            if (count > 0) Marshal.Copy(samples, position, output, count);

            // Pad the final buffer with silence instead of replaying stale memory,
            // without allocating on the audio thread.
            for (int padded = count; padded < frameCount; )
            {
                int n = Math.Min(Silence.Length, (int)frameCount - padded);
                Marshal.Copy(Silence, 0, output + padded * sizeof(short), n);
                padded += n;
            }

            position += count;
            return position >= samples.Length ? StreamCallbackResult.Complete : StreamCallbackResult.Continue;
        };

        int device = PortAudio.DefaultOutputDevice;
        if (device == PortAudio.NoDevice) throw new InvalidOperationException("No output device found.");

        StreamParameters parameters = new()
        {
            device = device,
            channelCount = 1,
            sampleFormat = SampleFormat.Int16,
            suggestedLatency = PortAudio.GetDeviceInfo(device).defaultHighOutputLatency,
            hostApiSpecificStreamInfo = IntPtr.Zero,
        };

        using PaStream stream = new(null, parameters, sampleRate, 0, StreamFlags.ClipOff, callback, null!);
        stream.SetFinishedCallback(_ => finished.TrySetResult());
        stream.Start();

        try
        {
            await finished.Task.WaitAsync(ct);
        }
        catch (OperationCanceledException)
        {
            // Barge-in: drop the buffered audio instead of draining it. The finished
            // callback still fires after this; completing an already-abandoned task is harmless.
            stream.Abort();
        }
    }
}

static class Wav
{
    public static byte[] Encode(short[] samples, int sampleRate)
    {
        byte[] wav = new byte[44 + samples.Length * 2];
        Span<byte> h = wav;
        "RIFF"u8.CopyTo(h);
        BinaryPrimitives.WriteInt32LittleEndian(h[4..], wav.Length - 8);
        "WAVEfmt "u8.CopyTo(h[8..]);
        BinaryPrimitives.WriteInt32LittleEndian(h[16..], 16);              // fmt chunk size
        BinaryPrimitives.WriteInt16LittleEndian(h[20..], 1);               // PCM
        BinaryPrimitives.WriteInt16LittleEndian(h[22..], 1);               // mono
        BinaryPrimitives.WriteInt32LittleEndian(h[24..], sampleRate);
        BinaryPrimitives.WriteInt32LittleEndian(h[28..], sampleRate * 2);  // byte rate
        BinaryPrimitives.WriteInt16LittleEndian(h[32..], 2);               // block align
        BinaryPrimitives.WriteInt16LittleEndian(h[34..], 16);              // bits per sample
        "data"u8.CopyTo(h[36..]);
        BinaryPrimitives.WriteInt32LittleEndian(h[40..], samples.Length * 2);
        MemoryMarshal.AsBytes(samples.AsSpan()).CopyTo(h[44..]);
        return wav;
    }

    /// <summary>Walks the chunks instead of assuming a 44-byte header: Piper's is, other writers' aren't.</summary>
    public static (short[] Samples, int SampleRate) Decode(byte[] wav)
    {
        ReadOnlySpan<byte> s = wav;
        if (!s[..4].SequenceEqual("RIFF"u8) || !s[8..12].SequenceEqual("WAVE"u8))
            throw new InvalidDataException("Not a RIFF/WAVE file.");

        int sampleRate = 0, offset = 12;
        while (offset + 8 <= s.Length)
        {
            ReadOnlySpan<byte> id = s.Slice(offset, 4);
            // Streaming writers put 0xFFFFFFFF here ("unknown"), which reads as -1: take the rest.
            uint declared = BinaryPrimitives.ReadUInt32LittleEndian(s[(offset + 4)..]);
            int size = (int)Math.Min(declared, (uint)(s.Length - offset - 8));
            ReadOnlySpan<byte> body = s.Slice(offset + 8, size);

            if (id.SequenceEqual("fmt "u8))
            {
                if (body.Length < 16) throw new InvalidDataException("fmt chunk too short.");
                short channels = BinaryPrimitives.ReadInt16LittleEndian(body[2..]);
                short bits = BinaryPrimitives.ReadInt16LittleEndian(body[14..]);
                if (channels != 1 || bits != 16) throw new InvalidDataException($"Expected 16-bit mono, got {bits}-bit x{channels}.");
                sampleRate = BinaryPrimitives.ReadInt32LittleEndian(body[4..]);
            }
            else if (id.SequenceEqual("data"u8))
            {
                if (sampleRate == 0) throw new InvalidDataException("data chunk before fmt chunk.");
                return (MemoryMarshal.Cast<byte, short>(body).ToArray(), sampleRate);
            }
            offset += 8 + size + (size & 1);    // chunks are word-aligned
        }
        throw new InvalidDataException("No data chunk.");
    }

    public static TimeSpan Duration(byte[] wav)
    {
        (short[] samples, int rate) = Decode(wav);
        return TimeSpan.FromSeconds((double)samples.Length / rate);
    }
}

Three details in there earn a sentence. stream.Stop() blocks until the last callback has returned, so the channel can be completed after it without losing the final buffer. The capture callback still allocates a small array per buffer. .NET makes no real-time guarantees either way; if you need them, record into a preallocated ring buffer instead. Playback pads its last buffer from a preallocated block of silence. And there is no GC.KeepAlive on the callbacks because PortAudioSharp2 pins each delegate with a GCHandle for as long as the stream lives. If you bind PortAudio yourself, that pinning is your job, or the next callback jumps into freed memory.

Speaker.PlayAsync takes a cancellation token for barge-in: when you have heard enough, Enter stops the audio with Abort(), which drops what is buffered instead of draining it.

Why push-to-talk, and not always listening

An always-on microphone looks friendlier and is worse in three measurable ways for this tool:

• It hears the agent. On laptop speakers, the reply goes straight back into the microphone and arrives as a new request.
• Whisper is unreliable at the edges of speech. On my eight recorded sentences, tiny.en returned [BLANK_AUDIO] for two that were perfectly audible, base.en turned one of them, a question about CS1061, into "Peace. See you in the next video. Bye." To be fair to Whisper, it was trained on a lot of YouTube. An always-on microphone feeds it far more edges than a key press does.
• You lose the edges. With push-to-talk, the press says now I'm talking to you, and the release says I'm done. With always-on you need voice-activity detection to guess both, and every wrong guess is a turn.

So the harness filters Whisper's silence markers before the model ever sees them, which you will see in VoiceAgent.IsBlank below. An accidental key press costs nothing and runs nothing.

Speech-to-text: wrong roughly every other word

The HTTP clients are short:

using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text.Json.Serialization;

/// <summary>whisper.cpp's whisper-server: POST /inference, multipart, returns {"text": "..."}.</summary>
sealed class WhisperClient(HttpClient http)
{
    public async Task<string> TranscribeAsync(byte[] wav, string? vocabulary, string? language, CancellationToken ct)
    {
        using MultipartFormDataContent form = new();
        ByteArrayContent file = new(wav);
        file.Headers.ContentType = new MediaTypeHeaderValue("audio/wav");
        form.Add(file, "file", "speech.wav");
        form.Add(new StringContent("json"), "response_format");
        form.Add(new StringContent("0.0"), "temperature");
        if (vocabulary is not null) form.Add(new StringContent(vocabulary), "prompt");
        if (language is not null) form.Add(new StringContent(language), "language");

        using HttpResponseMessage response = await http.PostAsync("/inference", form, ct);
        response.EnsureSuccessStatusCode();
        WhisperResult result = await response.Content.ReadFromJsonAsync<WhisperResult>(ct)
            ?? throw new InvalidDataException("Empty response from whisper-server.");
        return result.Text.Trim();
    }

    sealed record WhisperResult([property: JsonPropertyName("text")] string Text);
}

/// <summary>Piper's HTTP server: POST /synthesize with {"text": "..."}, returns a WAV.</summary>
sealed class PiperClient(HttpClient http)
{
    public async Task<byte[]> SynthesizeAsync(string text, CancellationToken ct)
    {
        using HttpResponseMessage response = await http.PostAsJsonAsync("/synthesize", new { text }, ct);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsByteArrayAsync(ct);
    }
}

prompt is Whisper's initial prompt: text it treats as if it came just before the audio. Passing the project's vocabulary there is the cheapest accuracy fix available, and it works on exactly the words that break.

To see how often they break, I recorded the eight requests I actually make (Why is the build red?, Is CS1061 coming from the rename I did in ChangedFile?, Don't run the tests yet, just build.) in my own voice and accent, on a USB headset, at normal speed. Then I ran each clip through four Whisper models, with and without the vocabulary prompt. This is one speaker and eight sentences, so read it as a measured anecdote, not a benchmark.

The machine is a laptop with a 4-core i7-1165G7 running at a 1.69 GHz power limit, Docker on Windows, 4 threads. Latency is the median per sentence; the encoder column is whisper.cpp's own whisper-bench:

Model	Word error rate (no prompt / with prompt)	Encoder, one window	Median per sentence
tiny.en	0.76 / 0.68	1.4 s	1.8 s
base.en	0.66 / 0.59	3.8 s	5.6 s
small.en	0.64 / 0.48	12.3 s	13.4 s
large-v3-turbo (q5_0)	0.48 / 0.41	77.6 s	not measured, over a minute

By word error rate, not one model transcribed a single one of the eight sentences perfectly, though some misses were as small as "test" for "tests". Why is the build red? came back as "What is the view right?" from three of the four. The words that broke were the project's own: ClaudeReviewBot became "cloud review bot" and "Cloud Review", ChangedFile became "change flight" and "Change. Fight.", origin main became "originally". The prompt fixed several of those, and it cost something: on one clip, small.en returned fragments of the prompt, "CS1061, nullable, origin main, IDP, ChangedFile.", instead of what I said.

The encoder column explains most of the latency. Whisper's encoder always processes a full 30-second window, so a four-second sentence costs the encoder as much as a thirty-second one. The rest is decoding, and that is where the tail comes from: when the decoder hallucinates, it keeps going. That one prompted clip took tiny.en 31.2 seconds and small.en 38.6. On this hardware only tiny.en and base.en answer at a conversational pace, and those are the two least accurate.

One more result, unplanned. During the live test I spoke my first request in Spanish without thinking about it. tiny.en is English-only, and it did not fail: "¿Por qué está rojo el build?" came back as "Okay, let's start from a little bit." This one did not tell me it could not understand me. It handed me a fluent sentence in its own language.

Design for the transcript you will get

If the speech-to-text cannot be made accurate on a laptop, the agent has to work with what arrives. So I gave every distinct transcript from that table to the agent and recorded which tool it reached for first, without running anything: 64 transcripts, three runs each, on Claude Haiku 5.5.

Transcripts from	Right tool	Asked or did nothing	Wrong tool	Silence, filtered
The exact sentences	8	0	0	0
large-v3-turbo + prompt	8	0	0	0
large-v3-turbo, no prompt	7	1	0	0
small.en + prompt	7	0	1	0
small.en, no prompt	6	2	0	0
base.en + prompt	6	0	1	1
base.en, no prompt	5	1	1	1
tiny.en + prompt	6	1	0	1
tiny.en, no prompt	5	1	0	2

Each cell counts sentences, by the verdict most of the three runs agreed on.

"What is the view right?" ran dotnet_build every time. "Bravo to test, until which ones failed" ran dotnet_test. "We did a changed size minus minus moment" ran git_status, which is as close to "what did I change since my last commit" as anyone could hope for. When the transcript could mean two things, it asked: "Did you want me to run the build, or are you just letting me know what you're working on?"

The negated request, Don't run the tests yet, just build, never ran dotnet_test in any variant. That was the one I was most worried about.

The failures were real failures, not a strict scorer. "Did any nullable warnings show up" lost the word warnings in six transcripts (five of them heard "nullable coordinates"), and without it, the agent sometimes read the question as being about changes or tests. "What is the new thread?" ran git_status three times out of three.

None of this works because the model is clever about audio. It works because the system prompt says the transcript is bad, gives two examples of how, and tells it what to do when two readings lead to different actions. And because the harness prints heard: "..." on every turn, so when one wrong word flips the question, you see it before you believe the answer.

The first thing that breaks: everything it says

Here is the answer from the opening, in full. This is Part 1's system prompt, unchanged, with the reply sent to the speaker:

4 of 91 tests failed, all in `PatchParserTests` (`tests/ClaudeReviewBot.Tests/PatchParserTests.cs`):

1. **`AddedLines_reports_new_file_numbers_across_hunks`** (line 29): expected `[15, 19, 20, 21]`, got `[15, 20, 21, 22]`.
2. **`ParseUnifiedDiff_splits_per_file_and_strips_the_b_prefix`** (line 109): expected first element `1`, got `2`.
3. **`AddedLines_removed_lines_do_not_advance_the_counter`** (line 43): expected first element `2`, got `3`.
4. **`AddedLines_ignores_no_newline_marker_and_crlf`** (line 62): expected first element `2`, got `3`.

The failures are all in line-number tracking, and the actual values are consistently one higher or shifted by one. That suggests an off-by-one in how the parser counts new-file line numbers, but I haven't read the parser code to confirm. I can look at the parser in `src/ClaudeReviewBot.Core` next if you want.

That is a good written answer. Piper reads it in 67.9 seconds. The diagnosis, off by one, starts after the fourth test name. Reading Part 1's raw dotnet_test output aloud instead, the 1,600 characters with stack frames, takes 109.4 seconds.

The fix has two halves, and the second is the one that matters.

Give the agent somewhere else to put things. A fourth tool, show_on_screen, does nothing but print. It turns "don't read out file paths" from a prohibition into an instruction with somewhere to go.

Rewrite the system prompt for an ear. Two sentences maximum. Never say a path, a frame, a hash or a full identifier. Lead with the answer. Here is the same question, same repository, same model, with the new prompt:

Four tests failed, all in PatchParserTests, and they all show line numbers off by one. Details are on screen.

The four test names, lines and arrays went to show_on_screen. I ran both prompts twice against two broken states of the repository:

Scenario	Part 1's prompt, spoken	Voice prompt, spoken
Four failing tests	88–105 words, 57.2–67.9 s	19–21 words, 6.5–7.2 s
One compile error	103–120 words, 46.7–53.2 s	20 words, 7.1–7.2 s

And the diagnosis moved from the last paragraph to the first sentence, because "lead with the answer" also means find the answer before speaking. It is not free. Each turn makes one more API call, for show_on_screen, and sends two to three thousand more input tokens, which on Haiku is under a tenth of a cent. It is also slower before it starts talking: the voice-prompt turns took 8.3–19.9 seconds against 6.1–12.2 for Part 1's prompt, tools included. You wait a few seconds longer and then listen for seven seconds instead of a minute.

The agent

Here is the whole agent. The loop is Part 1's, the three tools are Part 1's, and the new parts are the prompt, show_on_screen, the silence filter and the length guard at the bottom:

using System.ComponentModel;
using System.Text.Json;
using System.Text.RegularExpressions;
using Anthropic;
using Anthropic.Models.Messages;

/// <summary>
/// Part 1's tool-use loop with two output channels: the final text is spoken, and anything
/// that can't be listened to goes to the screen through the show_on_screen tool.
/// </summary>
sealed class VoiceAgent(AnthropicClient client, string repo)
{
    const string Model = "claude-haiku-5-5";

    // About ten seconds of listening at Piper's pace. Longer replies go to the screen.
    const int MaxSpokenWords = 30;

    const string SystemPrompt = """
        You are a development assistant for the ClaudeReviewBot repository, and the
        developer is TALKING to you. Their words arrive through speech-to-text, and your
        reply is read aloud by a text-to-speech voice.

        Three projects. src/ClaudeReviewBot.Core holds the logic and is the one that
        matters. src/ClaudeReviewBot is a thin CLI wrapper. tests/ is xUnit.
        TreatWarningsAsErrors is on. CS1591 is noise; nullable warnings are real defects.

        The transcript is often wrong. Speech-to-text mangles identifiers and jargon:
        "ChangedFile" may arrive as "changed file" or "unchanged file", "build red" as
        "view right". Read for intent. If you can tell what was meant, act on it. If two
        readings would lead to different actions, ask one short question instead of guessing.

        Your reply is heard, not read. So:
        - Speak at most two short sentences. No lists, no markdown, no code.
        - Never say a file path, stack frame, hash, or full identifier aloud. Put them on
          the screen with show_on_screen, then say what matters and "details on screen".
        - Lead with the answer: "Build's red, three errors, all from your rename."

        Check state before answering. If the question is about the build, build. If it is
        about what changed, call git_status. Do not speculate.
        """;

    static readonly List<ToolUnion> Tools =
    [
        new Tool
        {
            Name = "git_status",
            Description = "Branch, ahead/behind counts, and changed and untracked files in the working copy.",
            InputSchema = new() { Properties = new Dictionary<string, JsonElement>() },
        },
        new Tool
        {
            Name = "dotnet_build",
            Description = "Builds the solution, or one project, and returns deduplicated errors and warnings.",
            InputSchema = new()
            {
                Properties = new Dictionary<string, JsonElement>
                {
                    ["project"] = JsonSerializer.SerializeToElement(new
                    {
                        type = "string",
                        description = "Optional project path relative to the repository root.",
                    }),
                },
            },
        },
        new Tool
        {
            Name = "dotnet_test",
            Description = "Runs the test suite and returns pass/fail counts plus each failure's message and own-code frames.",
            InputSchema = new() { Properties = new Dictionary<string, JsonElement>() },
        },
        new Tool
        {
            Name = "show_on_screen",
            Description = "Shows text on the developer's screen without reading it aloud. Use for paths, " +
                          "diagnostics, stack frames, lists: anything too long or too precise to listen to.",
            InputSchema = new()
            {
                Properties = new Dictionary<string, JsonElement>
                {
                    ["text"] = JsonSerializer.SerializeToElement(new { type = "string" }),
                },
                Required = ["text"],
            },
        },
    ];

    readonly List<MessageParam> messages = [];

    /// <summary>
    /// Whisper marks silence and noise with bracketed tags: [BLANK_AUDIO], [inaudible],
    /// (silence), [Music]. A transcript that is nothing but tags is not a request: an
    /// accidental push-to-talk press must not cost an API call or run a tool.
    /// </summary>
    public static bool IsBlank(string transcript) =>
        Regex.Replace(transcript, @"\[[^\]]*\]|\([^)]*\)", "").Trim().Trim('.', ' ', '>').Length == 0;

    /// <summary>One spoken request in, one spoken reply out. Screen output goes to <paramref name="screen"/>.</summary>
    public async Task<string> AskAsync(string transcript, Action<string> screen, CancellationToken ct)
    {
        if (IsBlank(transcript)) return "";

        int turnStart = messages.Count;
        messages.Add(new() { Role = Role.User, Content = transcript });

        try
        {
            string? reply = await RunTurnAsync(screen, ct);
            if (reply is not null) return reply;
        }
        catch
        {
            // A failed request can leave a tool_use with no tool_result at the end of the
            // history, and every later request would be rejected. Drop the whole turn.
            messages.RemoveRange(turnStart, messages.Count - turnStart);
            throw;
        }

        messages.RemoveRange(turnStart, messages.Count - turnStart);
        return "I went in circles on that one. Try asking differently.";
    }

    /// <summary>Returns the spoken reply, or null if the step limit ran out.</summary>
    async Task<string?> RunTurnAsync(Action<string> screen, CancellationToken ct)
    {
        for (int step = 0; step < 8; step++)
        {
            Message response = await client.Messages.Create(new MessageCreateParams
            {
                Model = Model,
                MaxTokens = 4096,
                System = new List<TextBlockParam> { new() { Text = SystemPrompt, CacheControl = new CacheControlEphemeral() } },
                Tools = Tools,
                OutputConfig = new OutputConfig { Effort = Effort.Low },
                Messages = messages,
            }, ct);

            if (response.StopReason == "refusal")
            {
                // Haiku 5.5 has no server-side fallback, so the request stays declined.
                throw new RefusalException();
            }

            // Echo every block back unchanged - thinking blocks included, even when their
            // text is empty - or the next request fails the history check.
            List<ContentBlockParam> echo = [];
            List<ContentBlockParam> results = [];
            string spoken = "";

            foreach (ContentBlock block in response.Content)
            {
                if (block.TryPickThinking(out ThinkingBlock? thinking))
                {
                    echo.Add(new ThinkingBlockParam { Thinking = thinking.Thinking, Signature = thinking.Signature });
                }
                else if (block.TryPickRedactedThinking(out RedactedThinkingBlock? redacted))
                {
                    echo.Add(new RedactedThinkingBlockParam { Data = redacted.Data });
                }
                else if (block.TryPickText(out TextBlock? text))
                {
                    echo.Add(new TextBlockParam { Text = text.Text });
                    spoken += text.Text;
                }
                else if (block.TryPickToolUse(out ToolUseBlock? use))
                {
                    echo.Add(new ToolUseBlockParam { ID = use.ID, Name = use.Name, Input = use.Input });
                    (string content, bool isError) = await RunToolAsync(use, screen, ct);
                    results.Add(new ToolResultBlockParam { ToolUseID = use.ID, Content = content, IsError = isError });
                }
            }

            messages.Add(new() { Role = Role.Assistant, Content = echo });

            if (results.Count == 0) return GuardSpoken(spoken.Trim(), screen);

            // All results for one turn go back in ONE user message.
            messages.Add(new() { Role = Role.User, Content = results });
        }

        return null;
    }

    async Task<(string Content, bool IsError)> RunToolAsync(ToolUseBlock use, Action<string> screen, CancellationToken ct)
    {
        try
        {
            switch (use.Name)
            {
                case "git_status":
                    return (await DevTools.GitStatusAsync(repo, ct), false);
                case "dotnet_build":
                    string? project = use.Input.TryGetValue("project", out JsonElement p) && p.ValueKind == JsonValueKind.String
                        ? p.GetString()
                        : null;
                    return (await DevTools.DotnetBuildAsync(repo, project, ct), false);
                case "dotnet_test":
                    return (await DevTools.DotnetTestAsync(repo, ct), false);
                case "show_on_screen":
                    screen(use.Input["text"].GetString() ?? "");
                    return ("Shown.", false);
                default:
                    return ($"Unknown tool {use.Name}.", true);
            }
        }
        // A tool that can't run is an answer for the model ("I couldn't build"), not a crash.
        // Win32Exception is what Process.Start throws for a missing executable or a bad
        // working directory.
        catch (Exception ex) when (ex is TimeoutException or InvalidOperationException or KeyNotFoundException
                                       or Win32Exception or IOException)
        {
            return (ex.Message, true);
        }
    }

    /// <summary>
    /// The prompt asks for short replies; this makes sure of it. A reply too long to listen
    /// to goes to the screen, and only its first sentence is spoken.
    /// </summary>
    static string GuardSpoken(string spoken, Action<string> screen)
    {
        string[] words = spoken.Split(' ', StringSplitOptions.RemoveEmptyEntries);
        if (words.Length <= MaxSpokenWords) return spoken;

        screen(spoken);
        // A sentence ends at . ? ! followed by a space or the end - not inside
        // "ChangedFile.cs" or "1.8 seconds".
        Match end = Regex.Match(spoken, @"[.?!](\s|$)");
        string first = end.Success ? spoken[..(end.Index + 1)] : spoken;
        string[] firstWords = first.Split(' ', StringSplitOptions.RemoveEmptyEntries);
        if (firstWords.Length > MaxSpokenWords) first = string.Join(' ', firstWords.Take(MaxSpokenWords)) + "...";
        return $"{first} The rest is on screen.";
    }
}

sealed class RefusalException() : Exception("The model declined this request.");

DevTools is Part 1's three tool functions and its RunAsync helper, wrapped in static partial class DevTools and made public, along with Part 1's BuildOutput regex class and its Diagnostic and ProcessResult records.

The prompt is a request; GuardSpoken is a guarantee. In the four scripted voice turns above, the prompt alone kept every reply under 30 words and the guard never fired. Then in the live session I asked an open question, "What do I need to do to fix it?", and the reply ran long. The guard moved it to the screen and spoke one sentence and "The rest is on screen." A rule that only lives in a prompt holds until the first question nobody tested.

Tool failures go back to the model. If dotnet cannot start, because of a missing SDK or a bad path, Process.Start throws a Win32Exception. My first version did not catch it, and the first time it happened, the whole agent died mid-sentence. Now the exception becomes a tool_result with IsError = true, and the agent says so. Pointed at a directory that does not exist, it answered: "I couldn't tell whether it's red. The build tool failed twice before compiling, because it's pointed at a working directory that doesn't exist. Details on screen."

A failed request must not poison the history. If a request fails after a tool step (a rate limit, an overloaded API, a dropped connection), messages ends with a tool_use that has no tool_result, and every later request is rejected. So AskAsync drops the whole turn on any exception, and also when it runs out of steps, and the loop below catches the error, says so, and keeps the session. With an invalid key, two turns in a row failed, were announced, and the session carried on.

The loop

using System.Diagnostics;
using Anthropic;
using Anthropic.Exceptions;
using PortAudioSharp;

// usage: dotnet run -- <repo path> [--text]
//   --text  type instead of talking (the reply is still spoken)
if (args.Length == 0)
{
    Console.Error.WriteLine("usage: dotnet run -- <repo path> [--text]");
    return 2;
}

string repo = args[0];
bool typed = args.Contains("--text");

// Whisper's initial prompt: spelling hints for the words it gets wrong most.
const string Vocabulary =
    "dotnet build, dotnet test, git status, xUnit, ClaudeReviewBot, ChangedFile, CS1061, nullable, origin main.";

// Fail before opening the microphone, not after the first recording.
if (string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable("ANTHROPIC_API_KEY")))
{
    Console.Error.WriteLine("ANTHROPIC_API_KEY is not set.");
    return 1;
}

using HttpClient whisperHttp = new() { BaseAddress = new Uri("http://localhost:8080"), Timeout = TimeSpan.FromMinutes(2) };
using HttpClient piperHttp = new() { BaseAddress = new Uri("http://localhost:5000"), Timeout = TimeSpan.FromMinutes(2) };
WhisperClient whisper = new(whisperHttp);
PiperClient piper = new(piperHttp);

try
{
    await piper.SynthesizeAsync("ready", CancellationToken.None);
    if (!typed)
        await whisper.TranscribeAsync(Wav.Encode(new short[Microphone.SampleRate / 2], Microphone.SampleRate), null, "en", CancellationToken.None);
}
catch (HttpRequestException ex)
{
    Console.Error.WriteLine($"Start the containers first (docker compose up -d): {ex.Message}");
    return 1;
}

PortAudio.Initialize();
try
{
    VoiceAgent agent = new(new AnthropicClient(), repo);
    Console.WriteLine(typed ? "Type a request, empty line to quit." : "Enter to talk, Enter again to stop. Ctrl+C to quit.");
    bool alreadyPressed = false;

    while (true)
    {
        string heard;
        if (typed)
        {
            Console.Write("\n> ");
            heard = Console.ReadLine() ?? "";
            if (heard.Length == 0) break;
        }
        else
        {
            if (!alreadyPressed)
            {
                Console.Write("\n[Enter to talk] ");
                Console.ReadLine();
            }
            alreadyPressed = false;
            Console.Write("listening... ");
            byte[] wav = await Microphone.RecordAsync(ct => Task.Run(() => Console.ReadLine(), ct), CancellationToken.None);

            Stopwatch stt = Stopwatch.StartNew();
            try
            {
                heard = await whisper.TranscribeAsync(wav, Vocabulary, "en", CancellationToken.None);
            }
            catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
            {
                Console.WriteLine($"\n  speech-to-text failed: {ex.Message}");
                continue;
            }
            // Always show what was heard: one wrong word can flip the question.
            Console.WriteLine($"\n  heard: \"{heard}\"  ({stt.ElapsedMilliseconds} ms)");
        }

        Stopwatch thinking = Stopwatch.StartNew();
        string reply;
        try
        {
            reply = await agent.AskAsync(heard, text => Console.WriteLine($"  ┌ screen\n  │ {text.Replace("\n", "\n  │ ")}\n  └"), CancellationToken.None);
        }
        catch (RefusalException)
        {
            reply = "I can't help with that one.";
        }
        catch (Exception ex) when (ex is AnthropicException or HttpRequestException or TaskCanceledException)
        {
            // Rate limits, overload, a dropped connection: say so and keep the session.
            Console.WriteLine($"  request failed: {ex.Message}");
            reply = "I couldn't reach the model. Try again in a moment.";
        }
        Console.WriteLine($"  ({thinking.ElapsedMilliseconds} ms)");
        if (reply.Length == 0) continue;

        Console.WriteLine($"  says: \"{reply}\"");
        byte[] speech;
        try
        {
            speech = await piper.SynthesizeAsync(reply, CancellationToken.None);
        }
        catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
        {
            Console.WriteLine($"  text-to-speech failed: {ex.Message}");
            continue;
        }

        if (typed)
        {
            await Speaker.PlayAsync(speech, CancellationToken.None);
            continue;
        }

        // Enter while it speaks cuts it off (barge-in). Enter after it finishes starts the
        // next recording, so either way one key press moves the conversation on.
        using CancellationTokenSource stop = new();
        Task<string?> press = Task.Run(Console.ReadLine);
        Task playback = Speaker.PlayAsync(speech, stop.Token);
        if (await Task.WhenAny(press, playback) == press) stop.Cancel();
        await playback;
        if (!press.IsCompleted) Console.Write("  [Enter to talk] ");
        await press;
        alreadyPressed = true;
    }
}
finally
{
    PortAudio.Terminate();
}
return 0;

One key drives everything. Enter while it speaks cuts it off; Enter after it finishes starts the next recording. --text swaps the microphone for the keyboard and keeps the voice, which is useful in an open-plan office and for testing the loop without talking to it.

Why Haiku, not the model from Part 1

Part 1 used Claude Opus 5. This post uses Claude Haiku 5.5, and the reason is that the reply now sits behind two other slow things: speech-to-text before it, and synthesis after it.

I gave Claude Opus 5.5, the current Opus, the twelve transcripts where Haiku had asked, failed or changed its answer between runs. Opus ran once on each, against Haiku's three, so this is a small sample. Opus did recover more of them: all four "nullable coordinates" questions became builds. But it also acted where Haiku had asked. "What is the field print?" made Haiku ask three times out of three; Opus, on its one run, ran the build and the tests. For an agent you talk to, a question when the transcript is nonsense is the behavior you want.

On those same twelve transcripts, an Opus call cost about thirty times what a Haiku call did, with a median of 2.5 seconds against 1.8. A turn takes one to three calls, so that is up to two more seconds on a loop that already takes seven to twelve. The design above (show what was heard, ask when unsure) covers most of the accuracy gap. The cost and the seconds stay.

Haiku 5.5 brings three API differences that shape the code. It rejects any non-default temperature, so you cannot pin its output, and the same transcript can get a different tool on a different run: 8 of my 64 transcripts did. Its thinking is adaptive and on by default, so thinking blocks must be echoed back unchanged even when their text is empty. And effort is the latency lever: on the transcripts of the three easiest requests (build, tests, status), low and medium picked the same tool in 21 of 22, so the agent runs at low. I did not compare them on the hard cases.

A real session

Four turns, spoken, against a repository where I had renamed ChangedFile.Patch to Diff and forgotten one call site. Transcripts are as Whisper heard them, with tiny.en and the vocabulary prompt:

Heard	Tools	Said
"What is the build, great?"	dotnet_build, screen	"The build's red with one error: ReviewOrchestrator is calling a Patch property that ChangedFile doesn't have. Details on screen."
"Why did I change?"	git_status, screen	"I can't see your reasons, only the edits. Three files changed, and ChangedFile.cs is the likely cause of the Patch error. The list is on screen."
"Run the test."	dotnet_test	"The tests didn't run because the build is still failing, so there are no results yet. Fixing the Patch error in ReviewOrchestrator should unblock them."
"What I need to do to fix it."	none	"I can't read the code from here, so I can't say the exact fix yet. The rest is on screen."

The second row is the transcript problem in one line. I said what; Whisper heard why; the agent answered the question it was given, philosophically, before giving me the file list anyway. The fourth row is a missing tool, and Part 3's material.

The measured stages between releasing Enter and the first word of the reply added up to 7 to 12 seconds per turn: 2.6–3.6 s of speech-to-text, 2.1–7.4 s of agent (one to three API calls, plus the build itself), and 2.3–2.9 s for Piper to synthesize the whole reply before any of it plays. Decoding the WAV and the audio device's own latency come on top, and I did not time them. That last number is the cheapest to cut: synthesize the first sentence, start playing it, synthesize the rest behind it.

The bug Part 1 shipped

Halfway through this post, an update installed the .NET 11 RC SDK next to .NET 10. From then on, the agent reported every build as broken with MSB4216, a task-host error from deep inside MSBuild, even on code that compiled fine from a terminal.

dotnet run on a project exports MSBuildSDKsPath and MSBuildExtensionsPath, pointing at the SDK that built the agent. Every dotnet build the agent starts inherits them. The repository's global.json selects .NET 10, the inherited variables load .NET 11's targets, and the build dies before it reaches a line of code. Run against a build with one real CS1061, Part 1's original helper returned six MSB4216 and six MSB4027 errors and no CS1061 at all. Interestingly, dotnet run app.cs on a file-based app does not export those two variables, which is why the bug hides in some setups and not others.

The fix is three lines in RunAsync, which Part 1 now includes. It drops every MSBuild* variable, including any you set on purpose, such as MSBUILDDEBUGPATH; if you rely on one, remove MSBuildSDKsPath and MSBuildExtensionsPath by name instead.

foreach (string key in psi.Environment.Keys
             .Where(k => k.StartsWith("MSBuild", StringComparison.OrdinalIgnoreCase)).ToList())
    psi.Environment.Remove(key);

When not to give your agent a voice

If you are at your desk with your hands on the keyboard, typing is faster and more precise than talking. Every number above says so: 7 to 12 seconds per turn, and a transcript that gets more than half the words wrong.

Voice earns its place when your hands or eyes are busy elsewhere: walking a test run while you read a diff on the other monitor, or asking "is it green yet?" from across the room while a slow build finishes. It also suits short questions with short answers, which is most of what you ask a build agent.

It is the wrong tool for anything that writes, deletes or pushes. Part 1's tools are read-only on purpose, and that matters more now: a misheard request can at worst run a build. Before adding a tool with side effects to a voice agent, make it ask for confirmation in code, not in the prompt.

If you would rather not run models locally, hosted speech-to-text and text-to-speech services run on GPUs and skip the CPU latency above, at the price of sending your voice to someone else. I have not measured them here. Everything in the design still applies to them; only the latency table would change.

---

Key Takeaways

• Your agent's answers are written to be read. Read aloud, the same answer took 57–68 seconds. A voice prompt with an on-screen channel brought it to 7, with the diagnosis first.
• Design for the transcript you will get. On a laptop CPU, the two Whisper models fast enough to talk to got 59 to 76 percent of my words wrong. Tell the model the transcript is bad, have it ask when two readings diverge, and always show what was heard.
• A prompt rule needs a code guarantee. The 30-word guard never fired in testing and fired on the first open question in real use. The same goes for silence: filter it before it costs a call.
• Pick the model for the whole loop, not the reply. On my twelve hardest transcripts, Opus recovered more garbled requests, but cost about thirty times as much per call, added time to a loop that is already slow, and acted where Haiku asked. For this agent, Haiku plus good design was the better choice.
• Measure in the configuration your reader runs. The SDK leak only appears under dotnet run on a project, and the Compose flags only fail silently. Both looked fine until measured the right way.

The complete code is on GitHub, ready to clone and run: Juxn89/dotnet-voice-dev-agent. It is the code from this post, with Part 1's tools and both of their fixes.

---

Next in the series: Building a .NET Dev Agent, Part 3: Make It Remember You, session memory, so the agent stops asking which project matters every morning, and so "what was I doing yesterday?" becomes a question it can answer.