Yes, if your customers actually send them. In many markets, especially across the Gulf, North Africa and Latin America, a large share of inbound WhatsApp sales messages arrive as voice notes, and an agent that only reads text is silently ignoring those leads or dumping them on a human. Adding speech-to-text to an existing WhatsApp agent is a bounded project: expect one to three weeks of engineering on top of a working text agent, with recurring cost driven mostly by how many minutes of audio you process and which languages you need to understand well.
Why voice notes break text-only agents
A WhatsApp AI sales agent is usually built around a simple loop: receive a message via webhook, reason about it with an LLM, call a few tools (catalogue, CRM, booking), reply. Voice notes break the first step. The webhook does not deliver words; it delivers a media ID pointing to an audio file.
According to the WhatsApp Cloud API media documentation, audio arrives in formats such as OGG with the Opus codec (the native voice note format), AAC, MP4, MPEG and AMR. Your system has to download that file with a short-lived URL, convert or pass it to a speech model, and only then does the rest of your pipeline have anything to work with.
If you built your agent without this step, one of three things is happening today:
- Voice notes are dropped. The agent replies with something generic or nothing at all, and the lead goes cold.
- Voice notes are routed to a human. This works until volume grows, and it quietly destroys the response-time advantage that justified the agent.
- Customers adapt by typing. Some do. The ones who prefer to talk simply go to a competitor who answers.
Before committing budget, measure it. Count inbound messages by type for two or three weeks. If voice notes are under five percent of volume, a graceful fallback ("Could you type that out for me?") may be enough. If they are a quarter of your inbound, you are leaving real revenue in an audio file nobody listens to.
What you are actually building
The architecture that works best is boring on purpose: transcribe first, then feed the transcript into the exact same text agent you already have.
Step 1: Media ingestion
A small service receives the webhook, detects an audio message, fetches the media through the short-lived URL before it expires, and stores the raw file. Store it, not just the transcript. Keep the original OGG/Opus file alongside the sender, timestamp and media ID, and later attach the transcript and the agent's reply to the same record. That one record does three jobs. When a customer complains that the agent quoted the wrong item, you can replay the audio against the transcript and see whether the speech model misheard the product name or the agent misread a correct transcript. When a compliance team asks what was actually said, you have the source. And when you reach the point of comparing speech models, those stored notes are the fifty-to-hundred-recording evaluation set you would otherwise have to go back and collect. Without the audio you can only guess which of the two layers broke.
Step 2: Speech-to-text
This is the real decision point. You have three broad options:
- A cloud speech API. Fastest to integrate, priced per minute of audio, good for English and major languages. Quality on dialects and noisy recordings varies widely between providers.
- A self-hosted open model. Whisper and its faster derivatives run well on a modest GPU and cost you infrastructure rather than per-minute fees. Worth it when volume is high or when the audio cannot leave your region. We compared the two paths in detail in self-hosted speech-to-text vs cloud API.
- A specialised or fine-tuned model. Necessary when the general models fail on your audience. Gulf Arabic, medical vocabulary, product names and heavy code-switching are the typical triggers. See what Arabic speech recognition really costs for the specific challenges there.
Step 3: The existing agent, unchanged
The transcript becomes the user message. Everything downstream, your prompts, tool calls, guardrails, escalation rules and evals, keeps working. This is the main reason to resist the temptation of a model that "listens and answers" in one go. It feels elegant, but it hides the transcript, couples your speech quality to your reasoning quality, and makes every bug twice as hard to find.
One useful addition: pass a short note into the prompt that the message was a voice note and may contain transcription errors. The LLM then handles a misheard product name more gracefully, asking a clarifying question instead of confidently quoting the wrong item.
Step 4: Replying, in text first
The Cloud API allows you to send audio back, so a voice reply is possible. Start with text anyway. Text replies are cheaper, can be skimmed, are easier for a human supervisor to review, and avoid the extra latency of text-to-speech. Add voice output later only if customer feedback demands it, and then only for short confirmations rather than full product explanations.
What drives the cost
The budget splits into a one-off build and a recurring run cost. The build is comparatively predictable. The run cost depends on your volume and language.
Build cost drivers
- Whether a text agent already exists. If it does, voice is an increment. If not, you are really buying a full agent, and our breakdown of WhatsApp sales agent cost is the right starting point.
- Language and audio quality. English voice notes in quiet rooms are a solved problem. Dialectal Arabic recorded on a motorbike is not. The second case needs an evaluation set of real recordings, time to compare models, and possibly fine-tuning.
- Compliance requirements. Healthcare and government buyers often cannot send customer audio to a foreign cloud. That pushes you toward self-hosting, which adds infrastructure setup and a few more days of work.
- Review tooling. You will want a simple screen where a human can play the audio, see the transcript, and see the agent's reply. Skip it and your team will debug from log files for months.
Recurring cost drivers
- Minutes of audio per month. This is the only number that matters for the speech line item, and you should derive it from your own two-to-three-week count rather than guess. If voice notes are a quarter of your inbound and a typical one runs about 40 seconds, every thousand voice notes is roughly 670 minutes of audio. A cloud provider bills those minutes directly, so the cost grows in a straight line with volume. A self-hosted GPU bills by the hour whether it is transcribing or sitting idle, so its cost is flat and only wins once there are enough minutes to keep it busy.
- Slightly longer prompts. A rambling 40-second voice note transcribes into several sentences where the typed version would be one line asking whether the blue one comes in medium. LLM token spend rises a little per conversation as a result. Unless your customers record very long notes, it is a rounding error next to the speech minutes.
- Model upkeep. If you fine-tuned for a dialect, someone has to re-run your fifty-to-hundred-note evaluation set and re-score it with a native speaker whenever the base model or vendor changes. That is a recurring cost, not a one-off, and it is not zero.
A practical rule: take your measured monthly minutes, get a per-minute price from one cloud API and the hourly price of the smallest GPU instance that runs Whisper comfortably, and compare the two against that single number. At low volume the API wins easily, because a GPU idle for most of the day is pure waste. At high volume, or when a healthcare or government buyer forbids audio leaving the region, self-hosting wins regardless of the arithmetic.
Keep the pipeline simple, it pays twice
There is a strong pull, once you have audio, to build an elaborate chain: transcribe, then summarise, then classify intent, then extract entities, then draft a reply. Each extra step adds latency, cost and a new place for errors to compound.
We learned this on our own outreach engine, which scrapes each prospect's website and drafts a tailored email with a local model. Our first version chained several specialised calls together. The version that won on both cost and quality simply asked one model to pull out the relevant facts and write the draft in a single pass. The same lesson applies here: once you have a transcript, one well-structured call that understands the request and decides the next action usually beats a multi-step chain. We wrote more about this trade-off in single call vs agent chains.
The one step you should not collapse is transcription itself. Keep it separate and keep the transcript. Everything else can be a single call until you have evidence it should not be.
When it is not worth it
Voice note support is a poor investment when:
- Voice notes are rare in your actual inbound traffic and a polite typed fallback is accepted by customers.
- Your product requires precise identifiers such as serial numbers or dosages, and your audience's language is poorly served by current speech models. A misheard digit is worse than asking for text.
- You have not fixed the text agent yet. If the text agent escalates half of its conversations, voice will escalate even more. Fix the foundation first.
A sensible rollout
- Measure the share of voice notes for two to three weeks.
- Collect fifty to a hundred real voice notes (with consent and your privacy policy in order) as an evaluation set.
- Run two or three speech models over that set and have a native speaker score the transcripts. Pick the winner on your data, not on a benchmark.
- Ship transcription into the existing text agent, with the "this was a voice note" hint in the prompt.
- Review a sample of audio, transcript and reply every week for the first month.
- Only then decide on voice replies, fine-tuning, or self-hosting.
If your WhatsApp inbox is full of audio your agent cannot hear, that is a measurable leak in your sales funnel and a fairly contained project to fix. If you want a second opinion on your volume, language and the cheapest path to a working pipeline, let's talk.
Originally published on the Pykero blog.
Top comments (0)