A few weeks ago I wrote about the learning system I built inside my AI coding workflow: notes in Obsidian for every project, and an exam the agent offers when I reach a milestone. That post mentioned an oral mode almost in passing. This is the story of that oral mode, which grew into its own project: Socra.
Writing an answer lets me edit until it sounds right. Saying it out loud doesn't. If I can't explain a decision in my own project while someone listens, I don't understand it yet. So I wanted the agent to ask me questions with its voice and listen to my answers.
What Socra is
Socra is a local voice layer for coding agents. The agent speaks (Piper TTS), I answer out loud (faster-whisper STT), and a small web page shows only what the conversation needs: a code snippet, a diagram, the question. Typing is always one gesture away.
Today it runs with Claude Code and Pi, which use the same Socra tools. The tools are plain MCP, so the goal is any agent that speaks MCP (Codex, OpenCode and others), but I haven't tested those yet.
Running locally gives me three things:
-
Privacy. The server binds to
127.0.0.1only. Nothing leaves the machine except what the agent itself sends. - No cost for the voice. Piper and faster-whisper are free, open models. There is no speech API key and no per-minute bill; the only paid part is the agent's model, which I was already using.
- No network round trip for speech.
It started on October 20 as one thing: an oral exam. Claude Code called three MCP tools (say, ask, get_answer) and the page listened. It now has seven, including present, which shows a whole scene, and it became a session with two modes I can switch between, by voice or from the prompt bar:
- Mentor: the agent explains my own project to me and checks understanding without grading.
- Exam: spoken questions, spoken answers, spoken feedback.
What I learned
Most of what I learned had nothing to do with the model. It was about what changes when the interface is a voice.
Silence sounds like a crash
On screen, a spinner tells you something is happening. In a voice conversation, five seconds of silence feels like the app froze. The first versions waited for the whole reply before speaking.
Now Socra speaks replies sentence by sentence while they are being written: the first words play as soon as the first sentence is complete. The next step, still open, is to have the agent say short lines before and between long steps, so I know what it is doing while it works.
The first words kept disappearing
After a pause, replies started a few words late. The audio file was fine; the browser's audio output was going to sleep while idle, and waking it up ate the beginning of the clip. The fix keeps the output awake with an inaudible signal. I measured the lead-in silence of each voice, because one voice's slower start had been hiding the problem.
One language first
I'm a Spanish speaker and I started in Spanish. The Spanish voice mispronounced English technical terms, so I added a glossary, and then another fix, and then another. Eventually I decided to make Socra work well in one language first: English. Every language-dependent value (Whisper language, the agent's reply-language instruction, the default voice, the phrases Whisper hallucinates on silence) now comes from one setting, and a test fails if any other file hardcodes a language. Spanish comes back later, on purpose, instead of half working now.
The voice itself I picked with blind listening rounds: same sentences, voices labelled A to G, rated before I knew which was which. The winner (LibriTTS-R speaker 0) was also faster to synthesize than the previous default.
The prompt between sessions is part of the state machine
When I switch modes, Socra starts a new agent session and seeds it with a handoff: what happened, the last turns, and how to search the transcript. A bug report said that switching from mentor to exam left the session stuck. I never reproduced the hang itself, but I found something else: after the switch, the exam agent said it was waiting for me to press Switch again.
The handoff never told the new session that I had already confirmed the switch, and it ended with "Do not start any work yet". The model did exactly what the text said. The fix was in the text, not in the code: the handoff now states that the switch was confirmed, and each mode defines how it opens (the exam starts with its first question). I also added logging for every step of a switch, so if the original hang shows up again, the log will say where it stopped.
Scenes are state, not animations
For the mentor, I wanted the agent to show things while it talks: focus lines of code, a headline, a shape. Instead of letting the model drive the page step by step, it sends a scene: a versioned JSON document with steps to say, objects to show, and cues that bind visual actions to the start of a step. The server validates it against a schema and resolves the full visual state after each step. The page only renders that state; it never replays actions. That makes back, repeat and reconnecting the page exact, and when the model writes an invalid scene, the error points to the exact path (cues[0].do[1].params.lines) so it can fix it.
Where it stands
- Open source (MIT): https://github.com/teofurlan/socra
- Early and Windows-first. I use it myself; expect rough edges and setup steps that still need work.
- English only for now. Spanish voice and macOS/Linux support are open issues.
- Not tested with other users yet.
If you have tried voice interfaces for coding agents, I'd like to know what broke first for you.
AI disclaimer: Socra was built with AI coding agents (Claude Code and Pi) under my direction and review, and AI helped me draft this post from my project notes. I edited it and every claim comes from the repository and its issues.
Top comments (0)