Speech recognition turns a one-dimensional pressure signal into a sequence of words, and the whole design problem is that nobody tells the model where one word stops. Everything below follows from that.
The shape of the problem
A microphone produces numbers: at 16 kHz, sixteen thousand of them per second per channel, typically 16-bit signed. Thirty seconds of mono audio is 480,000 samples, which is 960 kB raw. The text it corresponds to is about seventy-five words. So the job is a reduction of roughly five thousand to one, from a signal with no boundaries in it to a sequence of discrete symbols.
Two properties of the input make it hard. It is enormously redundant — almost all of those samples carry no linguistic information at all, only the pitch of the speaker’s vocal folds and the acoustics of the room. And it is unsegmented: there is no reliable silence between spoken words, which is why an unfamiliar language sounds like one continuous stream. The front end deals with the first problem and the decoder deals with the second.
From samples to a spectrogram
Every modern system starts the same way, and this part has barely changed in thirty years. The waveform is cut into overlapping frames, each frame is turned into a short spectrum, and the spectrum is squeezed onto a perceptual frequency scale.
Input: 16,000 Hz mono, 16-bit PCM, 30 seconds
= 480,000 samples
Framing (the near-universal settings):
window length 25 ms = 400 samples
hop length 10 ms = 160 samples
frames 30 s / 10 ms = 3,000 frames
(each frame overlaps its neighbour by 15 ms, so the
signal is covered 2.5 times over)
Per frame:
Hann window, then a 512-point FFT -> 257 magnitude bins
project onto a mel filterbank -> 80 bins
take the log -> 80 values
Feature tensor for the clip:
80 mel bins x 3,000 frames = 240,000 floats
at 4 bytes each = 960 kB
So the "compression" so far is exactly none. What changed is that
the 240,000 numbers are now arranged so that a convolution over
time and frequency is a sensible thing to do, which it was not on
the raw waveform.
The mel scale is the only perceptually motivated step. It spaces filters roughly logarithmically, so the difference between 200 Hz and 300 Hz gets many more bins than the difference between 6,000 Hz and 6,100 Hz — which matches how human hearing resolves pitch, and matches where the information in speech actually lives. The log is there because loudness is perceived multiplicatively; it also stops a shout and a whisper of the same sentence from looking like different inputs.
The 25 ms window is not arbitrary either. Speech is quasi-stationary over about that long — short enough that a vowel does not change within the window, long enough to resolve the formants that identify it. Shorten it and frequency resolution collapses; lengthen it and the transitions that distinguish consonants smear.
The encoder, and why it is the expensive half
Three thousand frames is far too many positions for a transformer to attend over comfortably, and it is also far more than the output needs. So every architecture subsamples first, almost always with strided convolutions.
3,000 frames
-> two conv layers, stride 2 each -> 750 positions (typical of RNN-T stacks)
-> one conv layer, stride 2 -> 1,500 positions (Whisper's arrangement)
Then a transformer encoder over those positions.
Now compare the two sides of the model for the same 30 seconds:
encoder positions ~1,500
text tokens produced ~100 (75 words at ~1.3 tokens/word)
ratio ~15 : 1
That ratio is the single most useful number on this page. The encoder is doing self-attention over fifteen times as many positions as the decoder ever emits, and attention cost grows with the square of the sequence, so for a large ASR model the encoder is typically the majority of the compute for a full-length window. It is why transcription is priced per minute of audio rather than per word, why a two-second clip padded to a fixed window costs the same as a full one, and why trimming silence before you send audio is a real saving rather than a tidiness measure. That last point is worked through in the preprocessing harness.
The alignment problem
The encoder gives 1,500 vectors. The answer is 100 tokens. Nothing in the training data says which of those 1,500 positions produced which token — transcripts are not time-aligned, and hand-aligning them at phoneme level is exactly the manual work that made pre-neural ASR expensive.
Connectionist Temporal Classification is the trick that removed the need. It lets the model emit one label per encoder position, from an alphabet extended with a special blank symbol, and then defines the probability of a transcript as the sum over every frame-level path that collapses to it. Collapsing means: delete repeated labels, then delete blanks. So h h _ e _ l l _ l o collapses to hello, and the double L survives only because a blank separates the two runs. The model is trained on the sum over all such paths, computed efficiently by dynamic programming, so it never has to be told the alignment — it learns one.
Understanding blank explains a behaviour you will see in real output. CTC models are silent by construction on silence: with no speech, every position is overwhelmingly blank and the collapsed output is the empty string. That is not true of the third family below, and it is the root of the most-reported failure in speech AI.
Three decoder families
Almost every production system is one of these, and which one it is predicts more about its behaviour than its size does.
| Decoder | Description |
|---|---|
| CTC | One label per encoder frame, conditionally independent given the audio. Frame-synchronous, so it streams naturally and its timestamps are honest. Because outputs are independent it has no internal language model, so it produces phonetically plausible nonsense on hard audio and is usually paired with an external n-gram or neural LM at decoding time. Fast, cheap, boringly reliable. |
| RNN-T | Transducer. Adds a prediction network over the tokens emitted so far and a joint network that combines it with the encoder frame, so it has an internal language model while remaining frame-synchronous. This is the standard architecture for on-device and low-latency streaming recognition, because it can emit a token the moment the evidence arrives without waiting for the end of the utterance. |
| Attention encoder-decoder | An ordinary autoregressive decoder cross-attending to the encoder output. Best offline accuracy, because it is free to reorder, insert punctuation and use unlimited left context. It is also the only one of the three with no structural link between output position and time, which is why it can keep generating fluent text when the audio contains nothing — see the failure modes of Whisper. |
Hybrids are common: CTC and attention losses trained jointly on one encoder, with the CTC head used for alignment and streaming and the attention head used for the final text. If a system gives you accurate word timestamps and punctuated text, it is almost certainly doing something of that kind, or running a separate aligner afterwards.
Which one you are actually using
You can usually tell from behaviour alone, without documentation:
- It returns partial text that changes as you speak. Streaming, so RNN-T or a streaming CTC. Handling that revision correctly in a UI is its own problem, covered in stable partial rendering.
- It only answers after the audio ends, with punctuation and capitals. Attention encoder-decoder, almost certainly, processing a fixed-length window.
- Word timestamps are exact to the frame. There is a CTC or transducer alignment underneath. If they are vague and occasionally impossible, they were inferred from cross-attention, which is a heuristic.
- Silence produces text. Attention decoder with no voice-activity gate in front of it. Put one there; voice activity detection is a millisecond-scale model and it prevents an entire class of embarrassment.
Everything downstream — diarisation, timestamps, punctuation restoration, cost — attaches to one of these three shapes. It is worth knowing which one you bought.
Top comments (0)