Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Most people learn neural networks by staring at the model.
Weights. Attention. MLPs. LayerNorm. Tokenizers. Context windows.
But when you actually train an LLM, there is another piece of machinery making billions of decisions every second:
the optimizer.
A 70-billion-parameter model does not "learn" because gradient descent tells it which direction is better. It learns because an optimizer turns an enormous, noisy stream of gradients into parameter updates that are small enough not to explode, large enough to make progress, and adaptive enough that different parameters can move at radically different effective rates.
For the last decade, the dominant answer has largely been some form of Adam, and increasingly AdamW.
The interesting part is that Adam is not some mysterious LLM-specific invention. The original Adam paper was submitted in December 2014 by Diederik Kingma and Jimmy Ba, before the Transformer, before GPT, and before the modern LLM era. Kingma was working on scalable machine learning and generative models; Ba was then a PhD student working with Geoffrey Hinton at Toronto.
Three years later, the Transformer paper used Adam directly in its training recipe.
Then came AdamW, which fixed a subtle but important problem in how regularization interacted with adaptive optimization.
By 2025, Adam was sufficiently influential to receive an ICLR Test of Time award.
So what exactly is Adam doing?
And why is AdamW usually what you actually want when training a Transformer?
1. First, forget Adam: what problem is the optimizer solving?
Suppose your neural network has parameters
theta = [theta_1, theta_2, ..., theta_N]
and your training batch produces a loss L.
Backpropagation gives you
g = dL/dtheta
The simplest possible optimizer is gradient descent:
theta <- theta - alpha * g
where alpha is the learning rate.
That looks almost embarrassingly simple.
And that is indeed roughly what people did before adaptive optimizers became dominant.
The problem is that the gradients of a neural network are not nicely behaved.
Imagine two parameters:
g_1 = 0.001
g_2 = 10
A single learning rate has to deal with both.
If you choose alpha = 0.001, parameter 1 barely moves:
delta_1 = -0.001 * 0.001 = -0.000001
while parameter 2 gets:
delta_2 = -0.001 * 10 = -0.01
And this situation is not exotic.
Different parameters can have wildly different gradient scales. Some receive dense gradients every step. Others receive sparse or intermittent signals. Some directions in parameter space are noisy. Others are remarkably consistent.
So the fundamental problem is:
How do we turn a raw gradient into a sensible update for each individual parameter?
Momentum gives one answer.
Adaptive methods give another.
Adam essentially combines both.
2. Adam's key idea: keep a memory of the gradient
Adam stands for Adaptive Moment Estimation.
The easiest way to understand it is to imagine that every parameter maintains two small pieces of memory.
The first remembers:
"What direction have gradients generally been pointing?"
The second remembers:
"How large have those gradients generally been?"
For every parameter, Adam maintains:
m = moving average of gradients
v = moving average of squared gradients
More precisely:
m_t = beta1 * m_(t-1) + (1 - beta1) * g_t
v_t = beta2 * v_(t-1) + (1 - beta2) * g_t^2
Typically:
beta1 = 0.9
beta2 = 0.999
The interpretation is surprisingly intuitive.
m: momentum
Suppose gradients over five steps are:
+1
+1
+1
+1
+1
Then the moving average also points strongly positive.
Now imagine:
+1
-1
+1
-1
+1
The signs keep cancelling.
Adam therefore distinguishes:
consistent signal
from
noisy oscillation
This is momentum.
v: gradient scale
Now suppose a parameter frequently gets gradients around 10, while another gets gradients around 0.01.
Their squared gradients differ by a factor of:
10^2 / 0.01^2 = 100 / 0.0001 = 1,000,000
Adam remembers this.
That allows it to normalize the effective update.
Ignoring some details for a moment, the update looks like:
delta_theta ~= -alpha * m / sqrt(v)
So if a parameter has persistently large gradients, its denominator is large.
If its gradients are consistently tiny, its denominator is small.
Adam is therefore doing something qualitatively like:
move in the direction supported by recent gradients, but normalize the step according to how volatile/large those gradients have been.
That is the core idea.
It is not just "gradient descent with momentum."
It is per-parameter adaptive step sizing.
3. The weird-looking bias correction is actually necessary
There is an immediately obvious problem with the equations above.
At initialization:
m_0 = 0
v_0 = 0
Suppose the very first gradient is:
g_1 = 1
Then:
m_1 = 0.1
because:
m_1 = 0.9 * 0 + 0.1 * 1
But the actual observed gradient was 1, not 0.1.
The exponential moving average starts biased toward zero because its history is artificially filled with zeros.
Adam therefore uses bias correction:
m_hat_t = m_t / (1 - beta1^t)
v_hat_t = v_t / (1 - beta2^t)
and the actual update becomes:
theta <- theta - alpha * m_hat / (sqrt(v_hat) + epsilon)
That little correction matters most early in training.
For example, with:
beta1 = 0.9
t = 1
we have:
1 - beta1^t = 1 - 0.9 = 0.1
so:
m_hat_1 = 0.1 / 0.1 = 1
Exactly what we wanted.
The full Adam algorithm therefore has only a handful of moving pieces:
m_t = beta1 * m_(t-1) + (1-beta1) * g_t
v_t = beta2 * v_(t-1) + (1-beta2) * g_t^2
m_hat = m_t / (1-beta1^t)
v_hat = v_t / (1-beta2^t)
theta <- theta - alpha * m_hat / (sqrt(v_hat) + epsilon)
That's basically it.
A remarkable amount of modern deep learning sits on top of those few equations.
4. Why Adam was such a big deal for Transformers
The timing here is worth appreciating.
Kingma and Ba submitted the Adam paper in December 2014.
At that point, the dominant deep-learning world looked very different. Recurrent networks, convolutional networks, and SGD-style training were central. The Transformer did not yet exist.
Then, in 2017, Vaswani and colleagues published Attention Is All You Need.
The Transformer paper didn't invent some new optimizer specially designed for attention. It simply used Adam:
beta1 = 0.9
beta2 = 0.98
epsilon = 1e-9
with a warmup-and-decay learning-rate schedule.
That is historically significant because the Transformer went on to become the basic architecture underneath the modern LLM ecosystem.
In other words, one of the most consequential architecture papers in modern AI essentially plugged an existing adaptive optimizer into a radically different neural architecture.
And it worked spectacularly well.
There is a useful practical lesson here:
The optimizer does not have to understand the semantics of the architecture.
Adam has no idea whether a parameter belongs to:
Q projection
K projection
V projection
MLP
embedding table
layer normalization
It simply sees gradients and maintains statistics about them.
That abstraction is part of its power.
A tiny numerical example
Suppose two parameters receive:
Parameter A:
gradients ≈ [0.1, 0.2, 0.15, 0.1]
Parameter B:
gradients ≈ [10, 20, 15, 10]
Parameter B has gradients roughly 100x larger.
With vanilla SGD:
delta_B ≈ 100 * delta_A
Adam partially cancels that scale difference because its denominator tracks gradient magnitude.
You can think of Adam as making the optimizer less sensitive to the arbitrary units in which different parts of the network happen to express their gradients.
That is especially attractive in giant heterogeneous models.
5. But Adam has an enormous hidden cost: memory
There is a catch.
Adam needs to store two additional tensors:
m
v
for every parameter.
So if your model has N parameters, Adam needs roughly:
2N extra values
If those optimizer states are stored in FP32:
4 bytes/value
then optimizer state alone costs:
2 * 4 * N = 8N bytes
Consider a 7B parameter model:
7,000,000,000 * 8 bytes
= 56,000,000,000 bytes
≈ 56 GB
Just for the two Adam moment tensors.
Not model weights.
Not activations.
Not gradients.
Not KV cache.
Just:
m + v
For a 70B model:
70B * 8 bytes ≈ 560 GB
This is one reason optimizer engineering becomes a systems problem at LLM scale.
You can easily have a situation where the matrix multiplications themselves are perfectly GPU-friendly, but your optimizer state is forcing enormous distributed-memory and communication overhead.
There are several ways modern systems deal with this:
FSDP / ZeRO-style sharding
optimizer-state partitioning
CPU/NVMe offload
8-bit optimizer states
fused optimizer kernels
mixed precision
But Adam's conceptual simplicity hides a surprisingly expensive implementation reality.
For example, suppose your parameters are BF16:
2 bytes / parameter
but your Adam moments are FP32:
8 bytes / parameter total for m and v
The "small" optimizer logic now consumes roughly four times as much memory as the model parameters themselves.
That is why optimizer state can become a first-class architectural concern in large training systems.
6. Adam versus AdamW: the subtle problem with weight decay
This is probably the most important distinction to understand in practice.
People often use the terms:
L2 regularization
weight decay
as though they are interchangeable.
For ordinary SGD, they can effectively be equivalent.
For Adam, they are not.
Suppose we add an L2 penalty to the loss:
L' = L + (lambda / 2) * ||theta||^2
The gradient becomes:
g' = g + lambda * theta
Now notice what Adam does to g'.
It doesn't simply subtract:
alpha * lambda * theta
from the weights.
The regularization term goes into the adaptive machinery:
g' -> m -> v -> normalization
So the shrinkage of a parameter becomes entangled with Adam's gradient statistics.
That produces a surprising effect.
Two parameters with the same weight magnitude can receive different effective regularization depending on their gradient history.
Suppose:
theta_1 = 1
theta_2 = 1
and the only difference is that:
sqrt(v_1) = 0.1
sqrt(v_2) = 10
The same regularization contribution gets normalized very differently.
So the thing you thought was:
"shrink every weight by some amount"
has turned into something closer to:
"shrink weights according to how the optimizer's adaptive statistics happen to scale their gradients."
That is not the same operation.
Enter AdamW
In 2017, Ilya Loshchilov and Frank Hutter proposed a simple fix.
Don't put weight decay inside the gradient.
Do it separately.
Instead of conceptually doing:
g <- g + lambda * theta
Adam(g)
AdamW does:
Adam(g)
theta <- theta - alpha * lambda * theta
or, equivalently:
theta <- (1 - alpha * lambda) * theta
Now the optimization step and the shrinkage step are decoupled.
That is the entire conceptual breakthrough.
It sounds tiny.
It isn't.
This means the optimizer controls:
How should the model move to reduce the loss?
while weight decay controls:
How strongly should parameters be pulled toward zero?
Those are different jobs.
AdamW keeps them separate.
A concrete comparison
Suppose:
theta = 2
alpha = 0.001
lambda = 0.1
Then AdamW's direct decay contribution is:
alpha * lambda * theta
= 0.001 * 0.1 * 2
= 0.0002
So the weight gets multiplied by:
1 - 0.0001
= 0.9999
per optimization step, ignoring the gradient update for illustration.
After 10,000 steps, that multiplicative factor becomes approximately:
0.9999^10000 ≈ e^(-1) ≈ 0.368
So repeated tiny decay can become very substantial.
This is a useful way to think about weight decay:
It is not a tiny penalty applied occasionally. It is a multiplicative force acting at every optimization step.
And that is why seemingly boring hyperparameters like weight_decay=0.1 can have a large effect over a long training run.
7. What Adam/AdamW actually means when training an LLM
At this point, the practical picture looks something like this:
forward pass
|
v
compute loss
|
v
backprop
|
v
gradient g_t
|
+----> Adam exponential moving averages
| |
| v
| m_t, v_t
| |
| v
| adaptive update
|
+----> AdamW weight decay
|
v
parameters
There are several consequences worth keeping in your head.
Learning rate is still incredibly important
Adam does not eliminate the need to tune the learning rate.
The optimizer normalizes gradients, but alpha still determines the global scale of movement.
A useful mental model is:
Adam decides:
"How large should this parameter's step be relative to its gradient history?"
Learning rate decides:
"How aggressive should the entire optimizer be?"
That is why learning-rate schedules remain central in LLM training.
The Transformer paper, for example, used a warmup followed by inverse-square-root decay rather than holding the learning rate constant.
beta1 controls gradient-memory timescale
The moving average
m_t = beta1*m_(t-1) + (1-beta1)*g_t
has an effective memory on the order of roughly:
1 / (1 - beta1)
steps.
So:
beta1 = 0.9
means roughly a ten-step memory scale.
That is not an exact cutoff; it is an intuition for the EMA timescale.
Likewise:
beta2 = 0.999
corresponds to a much longer memory:
~1000 steps
for the second-moment estimate.
This is why changing beta values is not just changing some arbitrary constants.
You're changing the temporal horizon over which the optimizer interprets gradient behavior.
epsilon is mostly a numerical stabilizer
The denominator is:
sqrt(v_hat) + epsilon
The epsilon prevents division by something vanishingly small.
In many practical regimes, it is not the dominant behavioral hyperparameter.
But in low-gradient or low-precision regimes, its interaction with numerical scale can matter.
Adam is not Newton's method
A common misunderstanding is:
"Adam uses second-order information."
Not really.
It tracks a second moment of gradients:
E[g^2]
but it does not construct the Hessian:
H = d^2L/dtheta^2
and does not estimate the full curvature matrix.
Adam is still a first-order optimizer.
Its sophistication comes from using historical statistics of first-order information.
8. Why developers should care beyond knowing the equations
If you are debugging LLM training, AdamW is not an implementation detail.
It can directly influence:
training stability
loss curves
sample efficiency
generalization
memory footprint
distributed-training architecture
hyperparameter sensitivity
A few practical examples:
Training is unstable
You might immediately suspect:
bad initialization
bad normalization
bad data
exploding gradients
But the optimizer configuration is also part of the system.
A learning rate that is perfectly reasonable under one optimizer can behave differently under another.
Loss decreases but validation quality stagnates
Weight decay becomes interesting.
Because AdamW separates optimization from regularization, you can reason about:
learning rate
and
weight decay
as two separate control knobs.
That conceptual separation is much cleaner than treating "L2 regularization" as something buried inside the gradient.
GPU memory is unexpectedly full
Check the optimizer state.
For a 7B model:
Adam moments ≈ 56 GB in FP32
That number alone can explain a lot of apparently mysterious infrastructure decisions.
You scale the model and everything changes
This is an increasingly interesting research question.
The optimal AdamW weight decay is not necessarily a universal constant that you can blindly copy from a smaller model.
Recent work has explicitly studied how the optimal weight decay changes with model size, dataset size, and training dynamics.
In other words, once you're operating at serious scale, "just set AdamW to 0.1" is more cargo cult than theory.
9. The big picture: Adam is really a control system for noisy learning
The cleanest mental model I know is this:
A neural network is trying to optimize an absurdly high-dimensional function using noisy measurements.
The raw gradient says:
"Here is what today's minibatch thinks you should do."
Adam says:
"Fine. But I also remember what the gradients have been doing lately."
It keeps track of:
direction -> m
scale -> v
and uses those statistics to construct an adaptive update.
AdamW then says:
"And separately, I want the parameters to decay."
That separation turns out to matter.
So the evolution is roughly:
SGD
|
+-- momentum
|
+-- adaptive scaling
|
v
Adam
|
+-- decoupled weight decay
|
v
AdamW
And the reason this matters for LLMs is not that Adam is mathematically glamorous.
It is that training billion-parameter models is fundamentally an optimization-and-systems problem.
The model might contain 70 billion parameters, but every one of those parameters is being updated by a tiny piece of state maintained over the entire training trajectory.
That makes the optimizer part of the model's computational machinery.
The next time you see:
optimizer = AdamW(...)
you are not looking at five lines of boilerplate.
You are looking at a compact algorithm that is simultaneously doing:
momentum
adaptive normalization
bias correction
parameter updates
regularization
for billions of variables, potentially millions of times.
That is a rather extraordinary amount of machinery hiding behind one constructor.
One question for you
When you train or fine-tune an LLM, how much attention do you actually pay to the optimizer compared with the model architecture and data?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (0)