Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
In 2017, eight researchers published a paper with an almost provocative title: ""
Attention Is All You Need.
They were not proposing another small improvement to recurrent neural networks. They were removing recurrence itself.
That decision eventually became the architectural foundation for BERT, GPT-style models, T5, and most modern large language models.
As developers, it is tempting to treat “Transformer” as one thing. It is not.
An encoder-only Transformer, a decoder-only Transformer, and an encoder-decoder Transformer use closely related building blocks, but they impose very different information-flow constraints.
Understanding those constraints makes a lot of current LLM behavior much easier to reason about.
This article builds from intuition to the actual attention equations, then connects the mathematics to the three major Transformer architectures.
1. What attention actually does
Consider:
The server crashed because it ran out of memory.
Suppose the model is processing the word memory.
A useful representation of memory depends on other tokens:
-
ran out oftells us this is probably a resource. -
servertells us which kind of memory we mean. -
crashedtells us the event associated with it.
The basic idea of attention is therefore:
For each token, dynamically decide which other tokens are useful, and combine information from them.
This is different from a traditional feed-forward network, where each position can be processed more independently.
It is also fundamentally different from an RNN. An RNN processes a sequence step by step:
x1 -> h1 -> h2 -> h3 -> h4 -> ...
The Transformer instead lets every token interact with other tokens in parallel:
x1 <-> x2 <-> x3 <-> x4
At least, that is the basic encoder-style version. Later we will see that decoder-only models intentionally restrict those connections.
The key engineering consequence is parallelism.
The original Transformer paper demonstrated that a purely attention-based architecture could outperform recurrent models on machine translation while being substantially more parallelizable during training. That paper came from Ashish Vaswani and colleagues at Google and the University of Toronto.
The core operation responsible for this is scaled dot-product attention.
2. The mathematics of attention
Let the input representations be:
X
From X, we produce three different matrices:
Q = X W_Q
K = X W_K
V = X W_V
They are called:
Q = Queries
K = Keys
V = Values
The attention operation is:
Attention(Q, K, V)
= softmax((Q K^T) / sqrt(d_k)) V
That single equation contains most of the important idea.
Step 1: Compare queries with keys
We compute:
Q K^T
This produces a matrix of scores.
For token i attending to token j, the corresponding value is essentially:
score(i, j) = q_i . k_j
That is just a dot product.
If the query and key point in similar directions in representation space, the score is high.
If they are poorly aligned, the score is low.
So you can think of:
Query = "What information am I looking for?"
Key = "What kind of information do I contain?"
Value = "Here is the information."
The model learns W_Q, W_K, and W_V during training.
Step 2: Scale the scores
We divide by:
sqrt(d_k)
Why?
As vector dimensionality increases, the magnitude of dot products tends to increase as well. Without scaling, the softmax can become excessively sharp, pushing probabilities toward 0 and 1 and making optimization harder.
So:
scores = (Q K^T) / sqrt(d_k)
Step 3: Convert scores into weights
Apply softmax:
weights = softmax(scores)
Now each row contains something resembling:
[0.05, 0.10, 0.70, 0.15]
Meaning:
For this token, pay 70% of the attention to position 3, 15% to position 4, and so on.
Step 4: Mix the values
Finally:
output = weights V
The output for a token is therefore a weighted combination of information from other tokens.
That is the important conceptual leap.
Attention is not merely "looking at nearby words."
It is a learned, content-dependent routing mechanism.
The model can decide that one token should strongly interact with another even when they are far apart in the sequence.
A tiny example
Suppose we have:
"The programmer fixed the bug because it was obvious."
When processing it, the representation may need to determine whether it refers to bug, programmer, or something else.
Attention provides a mechanism for assigning different weights to those positions.
A simplified attention row might look like:
The programmer fixed the bug because it was obvious
0 0.02 0 0 0.70 0.03 0.25
The actual model does not contain a symbolic rule saying "it refers to bug."
It learns internal representations in which useful relationships produce useful attention patterns.
And importantly, one attention layer contains many such mechanisms simultaneously.
3. Why there are multiple attention heads
The Transformer does not normally run one attention operation.
It runs multiple attention heads.
Conceptually:
head_1 = Attention(Q_1, K_1, V_1)
head_2 = Attention(Q_2, K_2, V_2)
...
head_h = Attention(Q_h, K_h, V_h)
Then:
MHA(Q,K,V)
= Concat(head_1, ..., head_h) W_O
This is multi-head attention.
The point is not simply "more attention."
Different heads can learn different relationships.
One head might become useful for syntactic dependencies.
Another might track entity relationships.
Another may pay attention to delimiter structure or positional patterns.
We should be careful here: interpreting individual attention heads as clean human-defined linguistic concepts is often unreliable. A head is simply one learned projection and information-routing mechanism. Its behavior can overlap with other heads and change substantially across layers.
There is also an important computational fact hiding in the equation.
Suppose:
sequence length n = 4096
model dimension d = 4096
The attention score matrix has:
n^2 = 4096^2
= 16,777,216
entries.
So attention creates a matrix with roughly 16.8 million pairwise token interactions.
For the matrix multiplication Q K^T, the work scales approximately as:
O(n^2 d)
and the subsequent multiplication by V has the same order.
Using the numbers above:
n^2 d
= 4096^2 * 4096
≈ 68.7 billion multiply-accumulates
for one of those matrix multiplications.
Both together are roughly:
137 billion MACs
or around:
274 GFLOPs
if one multiply-accumulate is counted as two floating-point operations.
That is for one attention layer, one sequence, ignoring other work such as projections, normalization, and the feed-forward network.
This is the central scaling problem of vanilla attention:
double sequence length
-> roughly 4x attention interaction work
The model dimension matters too, but the quadratic dependence on sequence length is what makes long-context attention expensive.
This cost also explains a major economic fact about LLM systems:
Long context is not merely a product feature. It is a compute and memory budget.
4. The mask changes everything
At this point, you might think every Transformer token simply attends to every other token.
That is true for an ordinary encoder.
It is not true for an autoregressive decoder.
Consider generating:
"The cat sat on the ..."
When predicting the next token, the model must not be allowed to inspect the answer.
During training, however, it is convenient to present the entire target sequence at once.
The solution is a causal mask.
For positions:
1 2 3 4 5
the allowed attention pattern looks roughly like:
X . . . .
X X . . .
X X X . .
X X X X .
X X X X X
A token can attend to itself and earlier positions, but not future positions.
Mathematically, we add a mask M:
Attention(Q,K,V)
= softmax((Q K^T + M) / sqrt(d_k)) V
where forbidden positions receive something equivalent to:
-inf
before softmax.
After softmax, those positions effectively have probability zero.
This tiny-looking masking decision is one of the reasons GPT-style models behave differently from BERT-style models.
The underlying attention mathematics is almost the same.
The permitted information flow is not.
That distinction gives us the three major architectural families.
5. Encoder-only: understand the whole input
BERT is the canonical example.
Its name literally expands to:
Bidirectional Encoder Representations from Transformers
An encoder processes the sequence with unrestricted self-attention.
For:
"The database server is slow"
the representation of database can attend to:
The
server
is
slow
and slow can attend back to database.
There is no autoregressive "future-token" restriction.
This makes the architecture naturally suited to representation and understanding tasks.
Examples include:
classification
semantic similarity
entity extraction
token classification
retrieval embeddings
reranking
BERT, introduced by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova in 2018, became an important demonstration of what bidirectional Transformer representations could do. Its pretraining objective was mainly masked language modeling: hide some tokens and train the model to reconstruct them from surrounding context.
For example:
"The capital of France is [MASK]."
The model sees both:
The capital of France is
...
and the surrounding context when constructing the representation needed to predict the missing token.
That creates a useful kind of contextual representation.
But there is a catch.
A standard encoder does not naturally operate as a left-to-right text generator.
The model is trained to understand a complete sequence with corruption, rather than repeatedly predicting:
x1 -> x2 -> x3 -> x4 -> ...
So if the job is:
Generate 500 new tokens one token at a time.
an encoder-only design is usually not the natural fit.
This leads directly to decoder-only models.
6. Decoder-only: predict what comes next
GPT-style models use a decoder-only Transformer.
The architecture keeps self-attention but applies a causal mask.
For token position t:
token_t can attend to:
token_1 ... token_t
but not:
token_(t+1) ... token_n
The training objective becomes next-token prediction.
For a sequence:
"The engineer opened the"
the model learns:
P(token_5 | token_1 ... token_4)
and more generally:
P(x_1, ..., x_n)
= product over t of P(x_t | x_1, ..., x_(t-1))
That factorization is extraordinarily important.
It means the same machinery used during training can be used during generation.
At inference time:
prompt
-> predict token
-> append token
-> predict next token
-> append token
-> ...
The model is therefore naturally a text generator.
This architecture has another engineering advantage.
During training, the entire sequence can still be processed in parallel because the causal mask enforces the dependency structure mathematically.
You do not have to run a separate neural-network invocation for each token during training.
At inference, however, generation is sequential.
That creates a different bottleneck.
Suppose an application generates 1,000 tokens.
Even if each Transformer forward pass is highly optimized, the application still has to execute the autoregressive process 1,000 times.
This is one reason inference optimization focuses heavily on things such as:
KV caching
batching
quantization
speculative decoding
continuous batching
The decoder-only architecture is therefore conceptually simple:
read everything so far
-> predict one more token
and operationally expensive in a very particular way:
generation length -> number of sequential decoding steps
This is one reason current LLM infrastructure looks much more like a systems-engineering problem than merely a machine-learning problem.
You are paying for memory bandwidth, matrix multiplication, synchronization, batching efficiency, and latency at every decoding step.
7. Encoder-decoder: understand an input, then generate another sequence
The original Transformer was actually neither BERT-style nor GPT-style.
It was an encoder-decoder Transformer designed for sequence-to-sequence tasks such as machine translation.
The architecture looks conceptually like this:
Input sequence
|
v
+----------+
| Encoder |
+----------+
|
| contextual representations
v
+----------+
| Decoder |
+----------+
|
v
Output sequence
The encoder uses ordinary bidirectional self-attention.
The decoder uses causal self-attention.
But there is a third operation in the decoder:
cross-attention.
The decoder creates:
Q = decoder representation
while taking keys and values from the encoder:
K = encoder representation
V = encoder representation
So:
CrossAttention(Q_decoder, K_encoder, V_encoder)
This gives the decoder access to the encoded source sequence while it generates the target sequence.
Consider translation:
English:
"The cat is sleeping."
French:
"Le chat dort."
The encoder reads the complete English sentence.
The decoder then generates the French sentence one token at a time.
Conceptually:
English sentence
|
v
Encoder
|
v
contextual representation
|
+---------------------+
| |
v |
Decoder <--- cross-attention
|
v
"Le"
|
v
"chat"
|
v
"dort"
This is a very useful architecture when there are explicitly two sequences:
input -> output
Examples include:
translation
summarization
structured text generation
some speech and multimodal pipelines
T5 took this idea and pushed it into a particularly elegant abstraction:
Treat essentially every NLP task as text-to-text.
For example:
translate English to German:
"The house is small."
becomes a text generation problem.
Classification can also be represented as:
sentiment: "This movie is terrible."
->
negative
The T5 work by Colin Raffel and colleagues systematically explored this text-to-text framework and compared different transfer-learning choices across many NLP tasks.
This reveals something important about Transformer architecture.
The three families are not three completely different technologies.
They are mostly different ways of constraining information flow.
8. The developer's mental model
A useful way to remember the architectures is:
| Architecture | Self-attention | Future tokens visible? | Natural strength |
|---|---|---|---|
| Encoder-only | Bidirectional | Yes | Understanding representations |
| Decoder-only | Causal | No | Generation |
| Encoder-decoder | Bidirectional encoder + causal decoder | No in decoder | Input-to-output transformation |
Another way to visualize it:
ENCODER-ONLY
x1 <-> x2 <-> x3 <-> x4
\ | | /
everyone can interact
DECODER-ONLY
x1
|
v
x2
^ \
| \
x1 x2
|
v
x3
Each position sees only the past.
ENCODER-DECODER
x1 <-> x2 <-> x3 <-> x4
|
| encoded information
v
y1 -> y2 -> y3 -> y4
The underlying attention primitive is almost unchanged.
What changes is:
Who can attend to whom?
That question determines a lot of the model's behavior.
And this is a useful lesson when reading papers or evaluating LLM architectures:
Architecture is largely information-flow policy implemented with matrix operations.
Once you see it this way, several confusing facts become easier to reconcile.
Why can BERT understand both sides of a sentence?
Because the attention graph is bidirectional.
Why can GPT generate text?
Because the attention graph is causal and the training objective factorizes the probability of a sequence into next-token predictions.
Why can T5 translate?
Because one network constructs a representation of the source while another autoregressively generates the target using cross-attention.
And why does context length become expensive?
Because ordinary self-attention creates interactions that scale quadratically with sequence length.
The equations are compact.
The systems implications are not.
For an engineer, the most useful abstraction is therefore not:
"Transformers are neural networks that use attention."
It is:
A Transformer is a stack of learned information-routing operations, and the mask and attention structure determine the communication graph between tokens.
That viewpoint connects the mathematics directly to implementation.
It also explains why changing seemingly small architectural details can change inference costs, memory requirements, latency, and what kinds of tasks a model is naturally good at.
Conclusion: one equation, three architectures
The central equation is still:
Attention(Q,K,V)
= softmax((Q K^T) / sqrt(d_k)) V
From that primitive, we can construct:
Encoder-only
-> bidirectional self-attention
-> strong contextual representations
Decoder-only
-> causal self-attention
-> autoregressive generation
Encoder-decoder
-> bidirectional encoder
-> causal decoder
-> cross-attention between them
-> sequence-to-sequence transformation
The historical progression is also worth remembering.
The original 2017 Transformer replaced recurrence with attention for translation. BERT then showed how a bidirectional encoder could produce powerful contextual representations. T5 demonstrated how encoder-decoder Transformers could unify a wide range of language tasks under a text-to-text interface.
What looks today like one giant category called "LLMs" is really a collection of architectural decisions built around a relatively small mathematical core.
And once you understand the attention equation, the next interesting question is no longer:
"What is a Transformer?"
It becomes:
"What information is this model allowed to move between which tokens, and what does that imply for computation, memory, and behavior?"
That is the question that tends to matter when you move from reading LLM papers to actually building systems with them.
What architectural choice do you think matters most in practice for an LLM system today: attention pattern, context length, model size, or something else?
References
- Vaswani, Ashish, et al. Attention Is All You Need. 2017.
- Devlin, Jacob, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019.
- Raffel, Colin, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. 2019.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)