I've been running Whisper locally (via whisper.cpp) on my MacBook for offline voice-to-text, and at some point using a tool stops being enough — you want to know what's actually happening inside. So I pulled up the architecture diagrams and went down the rabbit hole.
Whisper is a standard encoder-decoder Transformer, the same family of architecture used in machine translation. The encoder processes the audio; the decoder autoregressively generates text conditioned on what the encoder produced.
The encoder: turning audio into meaning
The encoder never touches raw audio waveforms. Audio first gets converted into a log-mel spectrogram — a 2D representation of frequency content over time. That's what actually enters the network as "audio features."
From there:
Two 1D convolution layers (conv1, conv2) downsample and locally process the spectrogram, compressing it into a more manageable sequence length before it hits the transformer blocks. The second conv typically has a stride of 2, halving the time resolution.
Positional embeddings get added to the conv output. Transformers have no inherent sense of sequence order — self-attention treats input as a set, not a sequence — so position has to be injected explicitly.
The result feeds into a stack of WhisperEncoderLayer blocks, repeated 32 times in the model I looked at (large-v* scale). Each layer is:
LayerNorm → self-attention (20 heads, 64-dim each) → residual add
LayerNorm → feed-forward (Linear → GELU → Linear) → residual add
Self-attention here lets every audio frame attend to every other frame, weighting which parts of the audio are relevant to understanding any given moment — useful for things like resolving ambiguous phonemes using surrounding context. Residual connections (the Add nodes) are what make training a 32-layer-deep stack tractable at all; without them, gradients would vanish long before reaching the early layers.
After the stack, a final encoder layernorm cleans up the output. This is the encoder's final product: contextual hidden states representing the audio.
The decoder: generating text **autoregressively
**
The decoder is structurally similar but with one key addition, and it's much shallower — only 4 layers in this model, versus the encoder's 32. That asymmetry isn't arbitrary: most of the heavy lifting (extracting structure from a messy, high-dimensional signal) happens in the encoder. The decoder's job is comparatively lighter — condition on that representation and produce fluent text.
Each WhisperDecoderLayer:
Masked self-attention over previously generated tokens — masked because at generation step t, the model can only see tokens 1 through t-1. It can't peek at words it hasn't generated yet.
Cross-attention into the encoder's output. This is the actual bridge between audio and text — every decoder token attends over the full encoder hidden states here, pulling in whatever audio context is relevant to predicting the next word.
Feed-forward (Linear → GELU → Linear), same pattern as the encoder.
Each of these three sub-blocks is wrapped in its own LayerNorm + residual add, identical pattern to the encoder.
The decoder's input side mirrors the encoder: token embeddings + positional embeddings, summed before entering the layer stack. The output side runs through a final layer norm, then an output projection into vocabulary space — a softmax over every possible next token, from which one gets picked (greedy, beam search, or sampling depending on config).
This whole decoder pass repeats, one token at a time, feeding each generated token back in as input for the next step, until an end-of-sequence token is produced.
Why this design works
A few things stood out to me on closer inspection:
Depth asymmetry (32 vs 4) reflects task asymmetry. Audio is messy and high-bandwidth; language, once you have good audio context, is comparatively constrained.
Cross-attention is the only place audio and text actually meet. Everything before it in the decoder is pure language modeling (self-attention + FFN); everything in the encoder is pure audio modeling. Cross-attention is the single conditioning point.
Residuals aren't optional at this depth. 32 stacked layers without skip connections would be nearly untrainable — this is the same lesson from ResNets showing up in a completely different domain.
What I like about this architecture is how unglamorous the individual pieces are — convolutions, layer norm, linear layers, GELU — and how the interesting behavior emerges entirely from how they're composed and repeated, not from any single exotic component. It's a good reminder that a lot of what looks like "magic" in these models is really just careful, repeated application of a few well-understood ideas.
Top comments (0)