DEV Community

Cover image for What Is ASR? Automatic Speech Recognition for Voice AI Calls
Piyush Sahoo
Piyush Sahoo

Posted on

What Is ASR? Automatic Speech Recognition for Voice AI Calls

ASR transcription is the automatic conversion of speech into written text by a machine-learning model. ASR stands for automatic speech recognition, also called speech-to-text. On a phone call, the model receives digitised call audio, either live or from a recording, and returns words that software can search, analyse, or act on.

This guide follows one phone call from microphone to transcript. It then shows how to read accuracy claims, what published research says about 8 kHz audio, why multilingual calls need their own test, and which Vobiz path suits each job.

Note: Key takeaways

  • Treat any accuracy figure as a test result. One model in a 2023 Hindi study scored 7.6% on one test set and 26.8% on telephone-quality audio.
  • 8 kHz does not doom a transcript. Heavy compression, noise, and the wrong model can matter as much as the missing high frequencies.
  • Upsampling does not restore detail. Send the format your engine recommends, and avoid processing that the engine advises against.
  • Score the words you act on. A transcript can have a respectable overall error rate and still get the amount wrong.
  • Test each language mix separately. Code-switched speech, such as Hinglish or Spanglish, and single-language speech behave differently, and so do the engines.

What do ASR, STT, and transcription each mean?

ASR and speech-to-text name the same technology. The other terms around them describe different jobs, and mixing them up leads to the wrong product choice.

Term The question it answers Output
ASR or STT What words were said? Text from a model
Transcription What is the written record? Text from a model or a person
Diarization Which speaker said each segment? Speaker labels
Speaker recognition Whose voice is this? An identity match
Natural language understanding What does the speaker want? An intent or action

On a live AI call, ASR is the listening step. Its text goes to a language model, and a text-to-speech service speaks the reply. See how a voice API connects these parts for the wider call flow.

What happens between a phone call and a transcript?

A transcript is the last step of a chain. Each stage can remove information or add errors before the next stage sees the audio.

Five stages from a phone call to a transcript: caller and handset, network and codec, telephony media layer, ASR engine, and post-processing, each listing what decides quality at that stage.

The audio becomes features

A model does not read raw samples directly. It usually reads a compact picture of how energy moves across frequencies over time. OpenAI's Whisper offers a concrete example: it resamples all audio to 16,000 Hz and computes an 80-channel log-Mel spectrogram over 25 ms windows, moving 10 ms at a time.

The model maps features to text

Older systems combined separate parts for sounds, pronunciation, and word sequences. Connectionist Temporal Classification (CTC) helped change that, because it trains a neural network on unsegmented audio without a pre-aligned label for each moment.

Whisper goes further with an encoder-decoder transformer trained on 680,000 hours of labelled audio. Model architecture matters, but training data matters too. A model cannot be expected to handle speech it rarely saw in training, as the Indian results below show.

The decoder picks the likely words

Sound alone is often ambiguous. "Fifteen" and "fifty" can sit close together on a poor line, and so can "by" and "buy". The decoder weighs acoustic evidence against what is likely in context, so a model tuned to banking language can make a better choice on banking calls.

Post-processing makes the text usable

Raw output still needs work. Later steps can add punctuation, turn spoken numbers into digits, label speakers, and redact personal data. These steps also change how accuracy gets scored, as the WER section below explains.

Should you transcribe during the call or after it?

Choose by when your software needs the words. The accuracy cost of live transcription comes from a basic limit: a streaming model cannot wait for audio that has not arrived.

Research puts a figure on that limit under clean conditions. One 2023 paper compares models on LibriSpeech, a set of read speech. Its streaming model scored 3.90% and 8.21% WER, while its non-streaming counterpart scored 2.42% and 4.35% on the two test sets. Those numbers show the direction, not what you will see on phone calls.

Live engines work around the limit by revising themselves. Deepgram's documentation shows the pattern: interim transcripts are guesses that improve as more audio arrives, and endpointing finalises text when a voice activity detector finds a long enough pause. Your application must then decide when to act on words that may still change.

Question Live (streaming) After the call (batch)
When does text arrive? While the caller speaks Once the recording is processed
What context can the model use? Mostly what it has already heard The audio before and after each word
What must your code handle? Partial text, revisions, turn endings Files, queues, and result delivery
What suits it best? AI agents, agent assist, live captions Quality review, analytics, search, and records

You do not have to choose only one. One workable design streams audio for the live conversation, then runs a batch engine over the recording to produce the record that people review.

How do you measure ASR transcription accuracy?

Word Error Rate (WER) is the standard measure. Align the machine's text with a correct reference, count the substitutions, deletions, and insertions, then divide by the number of reference words. Lower is better.

An eight-word reference sentence aligned with a machine transcript. One substitution, one insertion, and one deletion give a Word Error Rate of 37.5 percent, and only the substitution changes the amount owed.

WER is useful for comparison, but it can mislead in three ways:

WER can even pass 100%, because insertions count against the same denominator. A model that invents words during silence can score worse than one that writes nothing.

Why the digits deserve their own score

Small error rates grow when a business depends on a long exact value. As an illustrative calculation, suppose an engine gets each digit right 98% of the time, independently. A 10-digit account number would then come through fully correct only about 82% of the time, because 0.98 raised to the power of 10 is 0.817.

That means roughly one in five account numbers needs a fallback, even though a 2% digit error rate sounds excellent. For payment, identity, or consent steps, confirm the value with the caller or collect it another way, such as keypad input.

What accuracy can you expect on phone audio?

Expect a range, not a number. Quote no percentage without its test conditions, because the audio and the test method can move the result more than the headline suggests.

Test Audio Reported WER How to read it
Microsoft, 2017 Switchboard telephone speech 5.1% A research system on a standard benchmark, nine years old
Szymański et al., 2020 50 call-centre conversations at 8 kHz 16.5-19.2% Three unnamed commercial engines on real work audio, six years old
Vistaar, 2023 Telephone-quality Hindi (Gramvaani) 26.8-59.9% Six systems on regional speech, three years old

The authors of the 2020 call-centre study help explain the gap. Their engines scored 10.2-11.6% on the Switchboard evaluation subset, about twice the best published figures at the time. They attribute this partly to testing on whole audio chunks with speech detection, and to general language models, where research systems often get prepared segments and models built for the data.

The chart below shows how much the test set alone can matter. It uses one model, one language, and seven benchmarks.

Bar chart of IndicWhisper Word Error Rate on seven Hindi test sets, from 7.6 percent on IndicTTS to 26.8 percent on telephone-quality Gramvaani audio, from the 2023 Vistaar study.

The Vistaar authors found that results on one benchmark can be a poor guide to another. Two of their test sets showed a negative correlation of -0.26 in how they ranked 14 systems. So a leaderboard tells you little until its audio resembles yours.

These published figures are also dated, and engines have moved on. Use them to understand the size of the spread, not to rank any vendor today.

Does 8 kHz audio limit ASR accuracy?

It limits what the audio can contain, but it does not set a fixed error rate. An 8 kHz sample rate can only represent frequencies up to 4 kHz, half the sample rate. Fricatives such as "s" and "f" show significant energy between 5 and 10 kHz, so that part of the signal never reaches the model.

Frequency map from 0 to 10 kHz. The telephone band ends near 3.4 kHz, 8 kHz sampling stops at 4 kHz, 16 kHz sampling stops at 8 kHz, and fricative energy sits between 5 and 10 kHz. A narrowband call in a 16 kHz stream is empty above 4 kHz.

That sounds fatal, yet the evidence is more mixed. The strongest recent data for Indian speech comes from a June 2026 preprint from ARTPARK and the Indian Institute of Science. It degraded Hindi audio in controlled ways before running current models on it:

Condition (simulated) What the authors observed
Filtered to a 3.4 kHz band, with no codec artifacts Stayed close to the 16 kHz original
GSM (2G) encode and decode Consistently worse, and the authors call narrowband limits like this the most critical audio factor
Opus (5G) encode and decode Only marginal degradation
Classical upsampling after downsampling Moderate loss
Neural restoration such as VoiceFixer and AudioSR Worse WER, even though the audio sounded better
Low amplitude precision Stable at 10-12 bits, with sharp degradation at 8 bits and below

Read the 8-bit row with care. The study reduced amplitude precision directly, while G.711 stores eight bits per sample after logarithmic scaling. The result does not transfer to phone calls one for one.

What should you actually do with the audio?

Follow your engine first, because engines differ. Whisper resamples everything to 16 kHz, while Google Cloud's guidance gives this advice for its own service:

  • Keep the native rate. Google prefers capture at 16,000 Hz or higher where you control it. It also says to avoid resampling, and to send telephony audio at 8,000 Hz if that is its native rate.
  • Prefer lossless encoding. FLAC or LINEAR16 is recommended, because mu-law and other lossy codecs may reduce accuracy.
  • Leave the audio alone. Google advises against automatic gain control, and says noise reduction before the service typically reduces accuracy.

Prevent mixed voices as well. When two people talk at once on one channel, Google warns that the recogniser may treat them as noise. Stereo recording avoids this, because a multichannel request transcribes each channel independently and each party has a track of their own.

Why do multilingual calls need their own test?

Calls in multilingual markets often combine three difficulties: mixed language, regional speech, and telephone audio. India is one of the best-studied examples, yet even there systematic analysis is still limited, so a global benchmark tells you little about your own calls.

Code-switching breaks one-language assumptions

Code-switching is common wherever people speak two languages, from Spanish and English in the US to Hindi and English in India. Researchers estimate that more than 250 million people in India use code-switched speech, especially blending English with Hindi. The hard form is a switch inside a sentence, where no pause marks the change. A speaker might say "aapka payment fail ho gaya, please retry karo", and an engine that fixed on one language at the start must now handle both.

Model choice has a large effect here. On a 2025 Hinglish corpus recorded at 16 kHz, three baseline models scored 16%, 31%, and 38% WER on adult speech from the same test audio. The same paper's audio was clean by telephone standards, so expect a phone line to add difficulty.

Language coverage is uneven

Published results vary widely by language and model. India's Vistaar benchmark spans 59 test sets across 12 Indian languages, and its authors fine-tuned Whisper on 10,700 hours of Indian speech to get the lowest WER on 39 of those sets. For you, the lesson is simple: a good Hindi result says little about Tamil or Bengali, and each language pair needs its own evaluation.

Domain words carry the business value

Every industry has terms that general training sets rarely contain enough of, such as EMI, UPI, and NEFT on Indian fintech calls. Some engines, including Deepgram's, let you boost expected terms at request time. Add your product names, branch names, and local place names to your test.

Transcripts carry privacy duties

A call transcript can hold names, account numbers, and health details in searchable form, so privacy laws such as the EU's GDPR treat it as personal data. If you're in India, most of the Digital Personal Data Protection Rules, 2025 start in May 2027, and they call for itemised notices, safeguards such as encryption or masking, and at least a year's retention of processing logs.

Call-recording consent and retention depend on your sector and role, so take legal advice for your case. For calling rules, see the India calling regulations guide.

How do you choose and test an ASR engine?

Run a small test on your own calls before you commit.

  1. Collect real audio. Use de-identified calls with the correct consent, spread across languages, regions, handsets, and busy hours.
  2. Write the reference by hand. Set one rule for numbers, punctuation, and script before you score anything.
  3. Send it the way production will. Use the same codec, sample rate, and channels, and do not test with a cleaned-up copy.
  4. Score in layers. Report overall WER, then accuracy on amounts, dates, names, and consent phrases.
  5. Split the results. Compare single-language speech, sentence-level switching, and mid-sentence switching separately.
  6. Measure the live path. For streaming, log time to first text, time to final text, and how often interim words change.
If your priority is Weigh most heavily
A live AI agent Time to final text, turn detection, and stability of interim results
Compliance records Critical-word accuracy, timestamps, and speaker separation
Analytics at volume Batch accuracy, cost per minute, and vocabulary support
Multi-language calls Mid-sentence switching accuracy for each language pair

Vobiz docs for this topic

The setup details for this are in the Vobiz docs:

Anything that places calls or sends messages is billed and reaches real people, so run it only with your own credentials after confirming the details.

Frequently asked questions

What does ASR stand for in transcription?

ASR stands for automatic speech recognition. In transcription, it means a machine-learning model converts spoken audio into written text without a human typist. You will also see it called speech-to-text or STT.

Are ASR and speech-to-text the same thing?

In practice, yes. Both names describe software that turns speech into text, and vendors use them interchangeably. Transcription is the broader word, because a person can also produce a transcript. Speaker recognition is different, because it identifies who spoke and not what they said.

What is a good Word Error Rate for phone calls?

No single figure holds. Published phone-audio results range from 5.1% on a research benchmark to 16.5-19.2% on real call-centre recordings. Hindi telephone audio scored 26.8-59.9% across six systems in a 2023 study. Measure on your own calls, and separately score amounts, names, and dates.

Is 8 kHz audio good enough for ASR?

Often, yes. Research systems have reached low error rates on telephone speech, and a 2026 Hindi study found a simulated 3.4 kHz band stayed close to the 16 kHz original. Compression, noise, and model fit can matter as much. Send the format your engine recommends, then test it.

Does upsampling 8 kHz audio to 16 kHz improve ASR?

It cannot restore detail that was never captured. A 2026 Hindi study measured moderate loss from classical upsampling, and neural restoration made error rates worse. Google advises sending telephony audio at its native 8,000 Hz. Some engines resample input themselves, so check your engine's documentation.

How do I get transcripts from phone calls?

You can stream live call audio to an ASR engine you choose, or send stored recordings to a batch engine. The Vobiz docs cover the streaming and recording options.

Further reading on Vobiz

Sources


If your speech pipeline is part of a voice AI agent on the phone, Vobiz provides the telephony that carries the call audio to it.

Top comments (0)