Originally published at ictinnovations.com
If you're wiring an LLM into an Asterisk call, the hard part isn't the model. It's the 20 ms frame clock, the five separate things a barge-in has to cancel, and a voice engine that isn't thread safe. We open sourced asterisk-ai-voice-agent four weeks ago, shipped six releases since, and got most of the corrections from strangers on Reddit. This is the engineering log.
What the Asterisk AI voice agent is, in one paragraph
It's a Python sidecar. Asterisk bridges a call into it over AudioSocket, or since 0.1.4 over chan_websocket, and from there it runs the loop you'd expect: voice activity detection on every 20 ms frame, speech to text through Whisper or ElevenLabs Scribe, a streaming turn with Claude that can call tools on your own webhook, and speech out through Piper locally or ElevenLabs in the cloud. It came out of the AI voice agent that ships inside ICTContact, stripped back to the parts that don't depend on our product. MIT, on PyPI and Docker Hub. We wrote up the release itself back in August, so this post skips the pitch and goes straight to what went wrong.
Lesson one: decide who owns the playout clock before you write a line
The bug that cost us the most time looked like nothing. Logs were clean, no errors, and the caller heard the last word of every sentence and nothing before it. The cause is that app_audiosocket forwards each frame to the channel the moment it arrives. Hand it 2.4 seconds of speech in one write and all 120 frames hit the far end in a few milliseconds. The jitter buffer keeps a handful and throws the rest away.
The fix everyone knows is to write one 320 byte frame and sleep until the next 20 ms deadline. The fix fewer people know is to re-clamp that deadline on every frame. When synthesis stalls for half a second, a pacer that computes the next deadline from where it thinks it should be will burst to catch up, and you're dropping audio again, just later. Snap the deadline to now instead. Lateness you can't recover; burst loss you can avoid.
Then a commenter on our first thread pointed out something we'd never checked. Python's asyncio leaves Nagle's algorithm on by default, and one small write every 20 ms is exactly the pattern Nagle exists to coalesce. Our pacer was correct and the kernel was re-bunching the frames underneath it. Setting TCP_NODELAY on the accepted socket fixed it, and we added a test that reads the option back off a real accepted socket, then ran it once with the fix removed to make sure the test wasn't lying. It came back zero. Nagle had been on the whole time.
Here's my honest recommendation after all that. If your Asterisk is 20.18 or 22.8 or newer, use chan_websocket and let Asterisk own the clock. You hand it audio ahead of real time, it re-frames and re-times, and FLUSH_MEDIA takes queued audio back when the caller interrupts. We measured that last claim rather than trusting it: queued 18 seconds of audio on 22.10.1, flushed after two, and not one flushed byte reached the caller. Replacement audio came back about 10 ms later. Six runs, ulaw and slin, same result every time. AudioSocket is still the right choice for the 18 and 20 boxes most sites actually run, and the protocol is small enough to read in one sitting. Just know that under AudioSocket, queue depth is your barge-in floor, because audio you've handed over is gone.
Lesson two: barge-in is five cuts, not one
Barge-in "worked" from the first release, in the sense that playback stopped when the caller spoke. It took an engineer from Exotel, who works on the streaming side of a voice API, to point out what was still running after playback stopped. Once text to speech streams into a pacer, a flush on its own leaves the synthesizer rendering a tail nobody will hear. With API priced voices that's real money. With a local voice it's worse, and I'll get to why.
So we went looking, and found that our stop flag was only polled between frames. Synthesis sat one await earlier. A caller who interrupted while we were still rendering had no effect at all: the whole sentence was built, then playback stopped at frame zero. Release 0.1.4 passed the stop check into synthesis itself. On the ElevenLabs path that closes the HTTP stream, so an interrupted sentence stops being billed.
The Piper path is where it gets interesting. Piper phonemizes through espeak-ng, and that C API is not thread safe, so every render on the box goes through one process wide lock. Our first instinct in 0.1.5 was to cancel the synthesis task outright on barge-in. That cancel lands on the await inside the lock's context, which releases the lock while espeak-ng is still running in its worker thread. The next render, from this call or any other, walks straight in. We proved it in the shipped Docker image with the real voice: barge in five frames into a turn while sentence two is rendering, close the turn, then render from a second call. The 0.1.5 code showed two concurrent Piper renders and the second caller waited 2.14 seconds. Letting the in-flight render finish and discarding it, which is what 0.1.6 does, showed one render and 1.32 seconds. The wait is just the remainder of a render that was already happening, and nothing about the interrupt itself changes.
The fifth cut is the one I'd never have found without reading our own code against the thread. Breaking out of the LLM stream skipped the step that records the assistant's message, so the model never learned it had said anything. And if the model had already asked for a tool before the caller cut in, the tool still ran and its result was filed against a tool call that was never recorded. The API rejects that request, and every request after it, so one badly timed interruption left the caller hearing only the apology line for the rest of the call. Now the record holds the sentences that actually reached the transport, tool calls from an interrupted turn are dropped, and the abandoned stream is closed immediately rather than whenever the garbage collector gets to it.
Lesson three: measure before you optimise, especially the obvious fix
Someone opened an issue saying the process wide Piper lock caps concurrent calls, and I agreed with them. Obviously a pool of worker processes, each with its own voice, would let four callers render at once. We built it. Then we measured it in the shipped image with a real voice, four concurrent calls, five trials each, because single runs had disagreed with each other at 0.82x, 0.93x and 1.31x, which is noise.
The pool was worse on every axis. Total time 2.30 seconds against 1.99 for the lock. The first caller waited 1.89 seconds instead of 0.48. ONNX Runtime already parallelises a single render across every core, so the lock was serialising work that was already saturating the CPU, and the pool only added contention, inter-process copying, and the loss of first in, first out ordering. Capping each worker to one thread was worse again, 5.41 seconds. The ceiling was CPU, not the lock. We reverted it, closed the issue with the numbers, and I'd argue the numbers on that issue are worth more than the feature would have been.
The measurement that did pay off was overlapping synthesis with playback. Speaking one sentence at a time meant sentence N+1 didn't start rendering until the caller had finished hearing N, so every sentence boundary carried a gap the length of the next render. With a real Asterisk echoing our audio back so we could time arrival rather than departure, a three sentence turn went from 2.65 seconds with two 500 ms holes to 1.88 seconds with a 120 ms worst gap. When synthesis is faster than playback, which it usually is, the gap closes completely. Overlap can't create throughput though. Where a sentence costs more to render than to play, the residual gap is the deficit and no queue fixes that.
Lesson four: the small text handling bugs are the ones callers notice
Two of them. First, the splitter that hands sentences to the voice while the model is still streaming fired on every comma, semicolon and colon, and on any full stop before whitespace. So "Dr. Smith" was two utterances, "1. Restart the phone" became a spoken "one." followed by a fragment, and every clause played as its own choppy piece with the voice losing the run of the sentence at each cut. It now splits at sentence ends only, knows about abbreviations, initials and list numbers, and falls back to a clause break only once a sentence runs past 120 characters.
Second, the first word after a barge-in kept getting clipped. The voice detector only calls a frame voiced once the talk spurt has enough energy, so a quiet onset was already gone by the time the utterance opened. A commenter suggested a lookback buffer and I initially said we'd gone another way. He was right and I was wrong. Release 0.1.3 holds a 300 ms rolling window of pre-onset frames and prepends them when an utterance opens, one ring buffer per call so it stays at 4.8 kB however long the call runs. The part I didn't expect: that prepended audio had to stay out of the voiced duration count, because that number feeds a word density check for Whisper hallucinations, and padding it made short answers look like garbage.
Lesson five: post the war story, not the announcement
Almost every fix above came from two threads on r/Asterisk. The first, why writing a whole TTS sentence to AudioSocket means the caller only hears the end of it, brought in an Asterisk maintainer who corrected me on chan_websocket availability (it landed in 20, not just 22), pointed at the mark mechanism for playout confirmation, and confirmed how the drain feedback works. It brought the Exotel engineer with the chunk size floor argument that reframed barge-in latency for me as a question of what unit you can't cancel once you've handed it over. It brought the Nagle catch, an external media comparison from someone running it in production, and a correction on format naming: in Asterisk terms slin is 8 kHz and slin16 is 16 kHz, so 320 bytes per 20 ms is plain slin. Our docs said slin16 for a month. The second thread, fixing dead air and clipped words when driving Piper into AudioSocket, produced the lookback buffer.
What made those threads work, as far as I can tell, was structure. Five paragraphs of the actual bug told as a story, the GitHub links only in paragraph six, and a closing question that gave people somewhere to go. Every correction got a reply within a day that either shipped the fix with a commit link or explained honestly why not. Nothing about that is clever. It's just the opposite of a launch post.
What's still open in the voice agent
The listening side is the biggest remaining latency. Speech to text is cloud only and not streaming: the agent waits for 550 ms of silence, wraps the utterance as a WAV, uploads it, and waits for one final result before the model can even start. A streaming engine with partial words and its own endpointing would cut the fixed wait and the round trip together. Deepgram is the least code for the cloud path and sherpa-onnx is the local answer; neither is implemented today, so don't read the config as if they were. On the voice side, Kokoro is the obvious upgrade from Piper for anyone who wants a more natural voice without a GPU. And chan_websocket play currently returns on a wall clock estimate rather than Asterisk's own drain notification, which is exactly the feedback the maintainer told us to use. All three are on the list, and the lab that measures them, real Asterisk 22 with an echo dialplan and the real Piper image, already exists.
The rest of the open source set this came from
The voice agent is one of a family, and the pieces underneath it are published on their own so you can take just the part you need.
asterisk-audiosocket: the AudioSocket protocol for Node.js and TypeScript, zero dependencies, both server and client sides.
piper-tts-server: the voice cache, synthesis lock and paced writer from this project as a standalone library and server.
pbx-mcp: an MCP server that lets an AI assistant inspect channels, registrations, trunks and dialplan on Asterisk or FreeSWITCH, read only unless you flip a flag.
Seven product MCP servers, for ICTFax, ICTPBX, ICTContact, ICTBroadcast, ICTDialer, ICTCRM and ICTExam. The overview of all eight explains what each one exposes.
asterisk-ami-node and freeswitch-esl-node: zero dependency clients for the two control protocols.
ICTPBX Community Edition: a multi-tenant IP PBX on FreeSWITCH, FusionPBX and ICTCore under MPL 2.0, with an Ansible role that installs it in one play.
LangChain toolkits and n8n community nodes for the same products, all indexed on our open source projects page.
If you only take one thing from this post, take the lab. A real Asterisk in a container with an Echo() dialplan returns every byte you play, so the arrival times of the echoed stream are the caller's actual experience. Three of the bugs above were invisible to unit tests with fake peers and obvious within minutes against that echo. Build it before you build the agent.
Frequently asked questions
Does an Asterisk AI voice agent need chan_websocket, or does AudioSocket work?
Both work. AudioSocket runs on every supported Asterisk and the protocol is tiny, but your application owns the playout clock and has to pace one 320 byte frame every 20 ms with a re-clamped deadline and TCP_NODELAY set. chan_websocket, on 20.18, 22.8 and newer, lets Asterisk pace and gives you FLUSH_MEDIA for barge-in. If your version allows it, chan_websocket is less to get wrong.
Why did the caller only hear the last word of each sentence?
Because the whole sentence was written to AudioSocket at once. Asterisk forwards frames as they arrive, so the burst overran the far end's jitter buffer and everything but the tail was discarded. Pacing fixes it, and so does leaving pacing to chan_websocket.
Can Piper handle several concurrent calls on one server?
Yes, behind one process wide lock, and that lock isn't the bottleneck. We measured a worker pool against it with four concurrent calls and the pool was slower in total and made the first caller wait four times longer. ONNX Runtime already uses every core for a single render, so the CPU is the ceiling. Add cores or add boxes; don't add processes.
What has to happen on barge-in besides stopping playback?
Drop the frames in your own queue, stop or discard the sentence still being synthesised without breaking the voice engine's lock, close the LLM stream, record only what the caller actually heard, and drop any tool calls from the interrupted turn so the conversation history stays valid for the next request.
Is the voice agent tied to ICTContact or any other product?
No. It came out of the AI voice agent inside ICTContact, but the open source sidecar runs against a plain Asterisk with your own API keys for the model and voices. The product specific pieces live in the separate MCP servers and LangChain packages, which do need the matching product.
Which speech to text engine should I use with it?
Today it supports OpenAI Whisper and ElevenLabs Scribe, both cloud and both non-streaming. That's the largest remaining latency item. A streaming engine with partial transcripts and endpointing is the next change we'd make, and we'd measure it in the same lab before recommending one.
The code, the tests that time frames rather than count them, and the changelog with every measurement above are in the asterisk-ai-voice-agent repository. Issues and corrections are welcome; the best ones so far came from people who'd never seen the code.
Top comments (0)