Last weekend I got mad at voicemail. Not at any individual voicemail — at the entire concept. Someone calls, leaves a 30-second message, and then I have to remember to check it. By the time I do, the urgent ones are old news and the spam ones have wasted my attention. So I built an AI that listens for me.
This post is about the build — what I chose, what I got wrong, and the two specific decisions that turned a flaky demo into something I'm actually comfortable shipping.
The canonical code example is in the Telnyx code examples repo:
Picking the example
Telnyx has a public repo of around 480 code examples. The vast majority are too complex for a six-minute demo. I needed something that was small enough to teach the pattern, real enough to be useful, and visual enough to work on camera.
I picked ai-voicemail-transcription-forwarding-python because:
- It uses a recognizable, painful workflow. Everyone hates voicemail. No one has to be sold on why this matters.
- It's small. Around 280 lines of real code after fixes. That's two screens of scrolling.
- It has a clear "wow" moment. A phone call becomes an SMS with a priority emoji. That's the kind of thing you can show in five seconds and people get it.
- It hits three Telnyx products in one app — voice, SMS, and AI Inference.
The first surprise: it didn't actually work
I cloned the example, filled in my API key, and the AI part worked great. Then I tested the SMS delivery. It failed. Every time.
The bug: the send_sms function was passing the caller's phone number as the from field. The owner's number was set as to. So the API was saying "send an SMS from this random number I don't own, to this number." Telnyx (correctly) rejects that.
This was in the upstream team-telnyx/telnyx-code-examples repo. It's listed as "production-ready." It is not production-ready. It would never have sent a single SMS in production.
I dug through the rest of the code and found twelve issues total. The takeaway was the same: read example code skeptically, even from authoritative sources.
The SDK method names were wrong
This one took me a while. The upstream example used client.calls.actions.record_start() and client.calls.actions.transcription_start(). Neither of those methods exist in the current Telnyx SDK. The actual names are start_recording() and start_transcription().
Even worse: start_recording() has a built-in transcription parameter. You can do recording + transcription in one call instead of two. The upstream example was doing it the hard way AND using wrong method names.
Then there was the language_code parameter on the speak() call. The SDK uses language, not language_code. The speak returned 200 but then a call.speak.failed webhook arrived with a 400 "not well-formed" error. The greeting never played.
The transcription arrives after hangup
This was the sneakiest bug. I assumed the transcription would stream in chunks via call.transcription webhooks while the caller was speaking. That's how the upstream example was written.
In reality, the transcription arrives as a single call.recording.transcription.saved webhook AFTER the caller hangs up, with the full text in a transcription_text field. Different event name, different payload structure, different timing.
The fix: the app now waits for the transcription on hangup instead of processing immediately. If the transcript arrives after the session is popped, it recreates the session and processes it.
The real production gotcha: Kimi reasoning eats your token budget
The first version used max_tokens=300 for the LLM call. It worked in my unit tests. It failed in production. The AI Inference call returned, but the JSON got truncated mid-word, like {"priority":"urgent","summary":"Sarah reports. Why?
Because the model — moonshotai/Kimi-K2.6 — has reasoning tokens. When you ask it to think step-by-step, the reasoning comes out of the same token budget as the answer. With max_tokens=300, my answer was getting cut off after the first ~50 tokens of actual JSON because Kimi had burned 250 tokens reasoning about whether it should return JSON.
The fix was bumping max_tokens to 1500. Rule of thumb: when using reasoning models, budget 5-10x more tokens than the visible output needs.
The voice quality problem
The upstream example used voice="female" which is the basic TTS tier. It sounded robotic. I upgraded to AWS.Polly.Joanna-Neural with service_level="premium" — a natural, human-like voice. One parameter change, dramatic quality difference.
What I'd do differently
A few things I'd change if I was starting over:
- Skip Flask, use a single-file approach with stdlib. The app is small enough that Flask is overkill.
- Use a real prompt framework. I hardcoded the prompt as a string. For a real product I'd want a prompt file with version control, A/B testing, eval set.
-
Add a recording-storage step. The carrier records the voicemail to an MP3. I don't subscribe to the
call.recording.savedwebhook, so the audio is just sitting in Telnyx's storage with no way to retrieve it from my app.
The thing I'm still figuring out
The hardest part wasn't the code. It was figuring out what the AI should actually decide. "Urgent vs normal vs spam" feels obvious, but the edge cases are where the value is:
- A voicemail from your spouse saying "don't forget to pick up milk" — normal or spam?
- A sales call that says "I just wanted to introduce myself" — spam or normal?
- A "happy birthday" message with no other content — what category?
I haven't solved this. The next iteration would be a small eval set — twenty voicemails with my labels — and a prompt I tune against it.
What's next
This is the first of three apps I'm building for a YouTube series. The next one is an AI language tutor — call a phone number, practice a foreign language with an AI that adapts to your level. Same skeleton. By the third video, the audience will have seen the pattern three times and should be able to build anything on top of it.
If you want to build the voicemail yourself, the repo has the full code. The state machine itself fits in one screen — the rest is error handling, logging, and JSON parsing defenses. Should take you 30 minutes including the inevitable debugging.
Top comments (0)