Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
If you open the implementation of a modern LLM, the interesting part is often hidden in a handful of lines.
There is the attention mechanism, which gets most of the attention. Then there are two components that look almost embarrassingly simple:
x = x * rsqrt(mean(x^2) + eps)
and
y = W2 * (Swish(W1 * x) * W3 * x)
The first is RMSNorm.
The second is SwiGLU.
Together, they are part of the architecture that became characteristic of models such as LLaMA and its descendants. Meta's Llama 2 technical report describes a decoder-only Transformer using pre-normalization with RMSNorm and SwiGLU feed-forward layers.
Neither idea is particularly complicated.
The interesting engineering story is why these simple operations work so well, why they displaced older choices such as LayerNorm + ReLU/GELU, and what their arithmetic looks like when you scale them to billions of parameters and trillions of tokens.
This article builds the intuition first, then gets into the equations, parameter economics, and implementation details.
1. First, where do RMSNorm and SwiGLU actually live?
A Transformer block roughly looks like this:
┌──────────────────────┐
│ Attention │
└──────────────────────┘
│
v
RMSNorm
│
v
┌──────────────────────┐
│ SwiGLU │
│ FFN │
└──────────────────────┘
│
v
output
A modern pre-norm Transformer is more precisely something like:
h1 = x + Attention(RMSNorm(x))
h2 = h1 + SwiGLU(RMSNorm(h1))
The two components solve very different problems.
RMSNorm controls the scale of representations.
As information passes through dozens or hundreds of Transformer blocks, activations can acquire wildly different magnitudes. RMSNorm keeps their overall scale under control.
SwiGLU provides nonlinear computation.
Attention lets tokens communicate with one another. The FFN processes each token's representation independently and gives the network substantial computational capacity for transforming that representation.
This distinction is useful:
Attention -> "Which information should I retrieve?"
SwiGLU -> "How should I transform this information?"
RMSNorm -> "Keep the representation numerically well behaved."
The modern LLM stack therefore isn't just "attention."
A considerable amount of the model's computation happens in the FFN.
And that makes the design of the FFN extremely important.
2. RMSNorm: LayerNorm with one operation removed
Let's start with the older and more familiar idea.
Suppose a hidden vector is:
x = [2, 4, 6, 8]
LayerNorm computes the mean:
mean(x) = 5
and the variance:
variance(x)
= ((2-5)^2 + (4-5)^2 + (6-5)^2 + (8-5)^2) / 4
= 5
Then it transforms each value roughly as:
x_normalized = (x - mean) / sqrt(variance + eps)
So LayerNorm does two things:
1. subtract the mean
2. divide by the standard deviation
RMSNorm asks a surprisingly provocative question:
Do we actually need the first operation?
Biao Zhang and Rico Sennrich asked essentially this question in their 2019 paper, Root Mean Square Layer Normalization. Their argument was that re-centering the activations around zero may be unnecessary.
Instead of calculating variance around the mean, RMSNorm calculates the root mean square:
RMS(x) = sqrt( (x1^2 + x2^2 + ... + xd^2) / d )
and normalizes:
RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps)
where gamma is a learned scale vector.
That's it.
There is no:
mean = sum(x) / d
x - mean
operation.
This matters at scale because normalization is performed constantly.
The original RMSNorm paper reported comparable performance to LayerNorm while reducing running time by 7% to 64% across the models they tested.
The exact speedup depends heavily on the surrounding architecture and implementation, but the basic engineering attraction is obvious:
LayerNorm:
mean
↓
subtract
↓
square
↓
variance
↓
sqrt
↓
scale
RMSNorm:
square
↓
mean
↓
sqrt
↓
scale
One statistical operation disappeared.
3. The intuition behind RMSNorm
Consider two vectors:
A = [1, 2, 3, 4]
B = [101, 102, 103, 104]
They have the same shape but very different offsets.
LayerNorm treats the offset as something to remove.
RMSNorm doesn't.
Instead, it asks:
How large is this vector overall?
For A:
mean(square(A))
= (1 + 4 + 9 + 16) / 4
= 7.5
RMS(A) = sqrt(7.5)
~= 2.74
So:
A / RMS(A)
~= [0.36, 0.73, 1.09, 1.46]
The operation preserves the direction of the vector while controlling its magnitude.
This gives a useful geometric interpretation.
Imagine a high-dimensional vector as an arrow.
RMSNorm approximately says:
x
|
| x
| /
| /
| /
----------+----------------
Keep the direction.
Normalize the length.
The learned gamma then allows the network to decide how much scale each feature should ultimately receive.
This is particularly attractive in a deep residual network.
A Transformer repeatedly performs operations like:
x -> x + f(x)
If the magnitude of f(x) continually changes relative to x, optimization can become difficult.
RMSNorm provides a recurring scale-control mechanism.
There is also an interesting theoretical interpretation in the original paper: RMSNorm has an implicit adaptive-learning-rate-like effect because its normalization depends on the magnitude of the current activation.
4. Why RMSNorm became particularly interesting for LLMs
The original RMSNorm paper predates the current LLM explosion.
The idea was proposed in 2019, when large Transformer language models were already emerging but the modern LLM ecosystem was still several years away.
Then the architectural pieces started converging.
The Transformer appeared in 2017.
T5 explored large-scale Transformer language modeling and heavily influenced later architectures.
Noam Shazeer investigated gated FFNs.
Zhang and Sennrich investigated RMSNorm.
Meta's LLaMA architecture subsequently combined several of these ideas into a compact decoder-only design.
Llama 2, for example, uses RMSNorm and SwiGLU in its Transformer blocks.
This is one of the interesting patterns in deep learning architecture:
Research paper
↓
small architectural modification
↓
empirical validation
↓
adoption by a major model
↓
becomes "obvious" architecture
↓
eventually becomes boilerplate code
By the time you encounter:
self.norm = RMSNorm(dim)
in a LLaMA implementation, you're looking at the endpoint of several years of experimentation.
5. SwiGLU: the feed-forward network gets a gate
Now consider the other half.
The original Transformer FFN was conceptually simple:
FFN(x) = W2 * ReLU(W1 * x)
There are two linear projections.
The first expands the representation.
The activation introduces nonlinearity.
The second projects it back down.
For example:
hidden size = 4096
FFN size = 16384
The representation gets expanded:
4096 -> 16384 -> 4096
Why expand it?
Because the intermediate layer provides a large computational workspace.
A useful analogy is a temporary scratchpad.
The model takes a 4096-dimensional representation, expands it into a much larger space, performs nonlinear computation there, and compresses it back.
But there is a problem with a conventional activation such as ReLU.
ReLU is:
ReLU(x) = max(0, x)
It makes a hard decision:
negative -> 0
positive -> x
Shazeer's 2020 paper GLU Variants Improve Transformer explored a different idea: gating.
Instead of simply activating a projection, construct two projections:
a = W1 * x
b = W3 * x
Apply a nonlinear function to one:
g = activation(a)
Then multiply:
h = g * b
The second projection acts like a learned gate.
This creates a powerful mechanism:
input
|
+----> projection A ---> activation ---+
| |
+----> projection B --------------------*----> output projection
Each dimension can effectively modulate another dimension.
That's the core GLU idea.
6. Why SwiGLU?
Shazeer tested several GLU variants, including:
GLU
ReGLU
GEGLU
SwiGLU
The activation in SwiGLU is the SiLU/Swish function:
SiLU(x) = x * sigmoid(x)
where:
sigmoid(x) = 1 / (1 + exp(-x))
Therefore:
SwiGLU(x)
= SiLU(W1 * x) * (W3 * x)
= [W1*x * sigmoid(W1*x)] * [W3*x]
followed by the output projection:
FFN(x) = W2 * (SiLU(W1*x) * (W3*x))
Notice something important.
A conventional FFN has two matrices:
W1
W2
SwiGLU has three:
W1
W2
W3
That sounds like a 50% increase in parameters.
And it would be, if we kept exactly the same intermediate dimension.
Shazeer's experiments instead reduced the intermediate dimension so that the total parameter count stayed approximately constant.
This is one of the most important implementation details.
7. The 2/3 rule and the economics of SwiGLU
Suppose a conventional FFN has:
d_model = 4096
d_ff = 16384
Ignoring biases, its parameter count is approximately:
W1: 4096 * 16384
W2: 16384 * 4096
total = 2 * 4096 * 16384
~= 134.2 million parameters
Now suppose we naïvely introduce SwiGLU with the same d_ff.
We have three matrices:
W1: 4096 * 16384
W2: 16384 * 4096
W3: 4096 * 16384
Total:
3 * 4096 * 16384
~= 201.3 million parameters
That's roughly 50% more.
Instead, choose:
d_ff_swiglu ~= (2/3) * d_ff
So:
d_ff_swiglu ~= 10923
Then:
3 * 4096 * 10923
~= 134.2 million
Approximately the same parameter budget.
This is why you will often see modern architectures with apparently strange FFN dimensions.
For example, instead of thinking:
4096 -> 16384 -> 4096
you might encounter something closer to:
4096 -> 11008 -> 4096
The number is not arbitrary.
It is also frequently rounded to hardware-friendly values such as multiples of 256.
This is where machine-learning architecture meets systems engineering.
A theoretically perfect dimension is irrelevant if it produces ugly GPU kernels.
You care about:
parameter count
FLOPs
memory traffic
tensor-core utilization
kernel shapes
alignment
batch size
A 10,922-wide matrix may have nearly the same mathematical cost as an 11,008-wide matrix, while the latter may map much more conveniently onto the hardware.
8. A concrete LLM block
Let's put the pieces together.
A simplified modern Transformer block can look like:
def block(x):
h = x + attention(rmsnorm(x))
h = h + swiglu(rmsnorm(h))
return h
The RMSNorm is roughly:
def rmsnorm(x, weight, eps=1e-6):
rms = sqrt(mean(x * x) + eps)
return weight * x / rms
And SwiGLU:
def swiglu(x):
a = W1 @ x
b = W3 @ x
return W2 @ (silu(a) * b)
There are several implementation details worth noticing.
First, there are no biases in the canonical LLaMA-style FFN.
Second, the two projections into the intermediate dimension happen in parallel.
Third, the elementwise multiplication is extremely cheap relative to the matrix multiplications.
For:
d_model = 4096
d_ff = 11008
one token requires approximately:
W1: 4096 * 11008
W3: 4096 * 11008
W2: 11008 * 4096
multiply-accumulate operations.
That is roughly:
3 * 45.1M
~= 135M MACs
or approximately:
270M FLOPs
if one multiply-add is counted as two floating-point operations.
And that's per token, per layer, just for the FFN.
For a 32-layer model:
270M * 32
~= 8.6 billion FLOPs/token
The attention mechanism gets the conceptual spotlight, but the FFN is doing enormous amounts of numerical work.
9. What are these components actually buying us?
There is a useful way to think about the two operations.
RMSNorm controls the numerical regime
Without normalization, successive transformations can alter the scale of representations.
RMSNorm gives the network a stable reference:
"Whatever magnitude this representation currently has,
scale it into a predictable range before the next transformation."
SwiGLU increases conditional computation
A standard FFN essentially says:
transform -> activate -> transform
SwiGLU says:
transform A -> nonlinear gate
\
multiply -> transform
/
transform B -------
So the model gets an input-dependent mechanism for controlling information flow through the FFN.
This is subtle.
The gate isn't a separate high-level routing system like a Mixture-of-Experts model.
It is a cheap elementwise interaction inside every token's FFN.
You can think of it as giving every intermediate feature a learned, input-dependent volume knob.
10. Why this matters economically
LLM architecture is ultimately constrained by arithmetic.
Suppose you train a model for:
1 trillion tokens
and your architecture performs an additional:
100 million FLOPs/token
because of an architectural choice.
That's:
100M * 1T
= 1e20 FLOPs
An apparently tiny per-token change becomes enormous at training scale.
The same applies to inference.
If a model generates:
1 billion tokens/day
then an extra:
100M FLOPs/token
means:
1e17 FLOPs/day
So the reason architectural details matter is economic as much as mathematical.
A one-line normalization change can affect:
training stability
kernel complexity
memory bandwidth
latency
power consumption
GPU-hours
And a change to the FFN affects one of the largest computational components of every Transformer layer.
This is why Shazeer's paper is interesting from an engineering perspective.
He wasn't merely inventing another activation function. He was exploring whether a different FFN structure could improve model quality under approximately the same computational and parameter budget.
That constraint is much closer to how production ML actually works.
11. The bigger lesson: modern LLM architecture is evolutionary
There is a funny historical irony here.
When you first learn Transformers, the architecture looks almost inevitable:
Attention
+
Feed Forward
+
Normalization
Modern LLMs make the historical process visible.
The architecture wasn't designed in one shot.
Researchers kept asking small questions:
Do we really need mean-centering?
Can the FFN use a better nonlinearity?
Can a gate improve the representation?
Can we maintain the same parameter budget?
Can we make the operation cheaper?
Does the change survive large-scale experiments?
Zhang and Sennrich proposed RMSNorm in 2019.
Noam Shazeer explored GLU variants in 2020 and found that variants such as SwiGLU and GEGLU could improve Transformer results relative to conventional ReLU/GELU FFNs in his experiments.
A few years later, these ideas appeared together in LLaMA-style architectures. Llama 2, released by Meta in 2023 at 7B through 70B parameter scales, documents this architectural family explicitly.
That is a useful pattern to remember when reading new architecture papers.
The important innovation is frequently hidden inside something that looks too small to deserve its own section.
A new normalization rule.
A different activation.
A changed projection.
A different tensor layout.
A different way of spending the same FLOPs.
At billion-parameter scale, these details compound.
Conclusion: the two lines worth remembering
If you strip away the enormous matrices and billions of parameters, two ideas are remarkably compact.
RMSNorm:
RMS(x) = sqrt(mean(x^2) + eps)
y = gamma * x / RMS(x)
Its job is to control representation scale while avoiding the explicit mean-centering of LayerNorm.
SwiGLU:
SwiGLU(x)
= W2 * (SiLU(W1*x) * (W3*x))
Its job is to give the FFN an input-dependent gating mechanism while retaining roughly the same parameter/FLOP budget by shrinking the intermediate dimension.
Together they illustrate something important about LLM engineering:
small mathematical change
+
large number of layers
+
trillions of tokens
=
major systems consequence
And perhaps the most interesting part is that neither RMSNorm nor SwiGLU is conceptually exotic.
The difficult engineering question is often:
Which simple operation should a 70-billion-parameter machine perform trillions of times?
If you were designing a Transformer from scratch today, which "boring" component would you investigate first: normalization, activation functions, attention, or the FFN itself?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)