DEV Community: Shrijith Venkatramana

Scaled Dot-Product Attention: The 4-Line Algorithm That Powers Modern LLMs

Shrijith Venkatramana — Wed, 01 Jul 2026 17:49:09 +0000

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

In 2017, a small group of Google researchers removed recurrence, removed convolution, and bet everything on one deceptively simple idea: every word should decide for itself what deserves attention.

That idea—Scaled Dot-Product Attention—became the computational primitive behind GPT, Claude, Gemini, Llama, DeepSeek, and nearly every modern Large Language Model.

The remarkable part isn't just that it works.

It's that the core algorithm fits into a single equation.

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

In this article we'll build intuition first, then gradually unpack the mathematics, engineering, and economics behind the mechanism that made modern LLMs possible.

(Include the screenshot from the "Attention Is All You Need" paper here.)

Before Transformers: The Memory Problem

Imagine asking someone to finish this sentence:

"The trophy didn't fit in the suitcase because it was too small."

What does it refer to?

The trophy?

Or the suitcase?

Humans answer almost instantly because our brains naturally connect related concepts across a sentence.

Earlier neural networks struggled.

RNNs

Recurrent Neural Networks processed words one at a time.

The -> trophy -> didn't -> fit -> ...

Each word updated a hidden state.

The problem was that information had to travel through dozens or hundreds of sequential steps before reaching later words.

By the time the network reached the end of a paragraph, early information had often faded away.

LSTMs improved the situation with gating mechanisms, but they still fundamentally processed sequences sequentially.

That became an enormous bottleneck.

Both computationally.

And conceptually.

The Insight: Let Every Word Look Everywhere

One of the authors of the Transformer paper, Ashish Vaswani, later described the goal simply:

Instead of carrying memory forward step-by-step, why not allow every word to directly inspect every other word?

Suppose we have:

The cat sat on the mat.

When processing sat, perhaps the model mostly cares about:

It doesn't need to care very much about The.

Instead of forcing information through intermediate states, attention allows direct communication.

        sat
      /  |  \
     /   |   \
   cat   on  mat

Every token asks:

"Which other tokens are relevant to me?"

That's attention.

Queries, Keys and Values: Think Like a Search Engine

The names sound intimidating.

They're actually borrowed from information retrieval.

Imagine Google Search.

When you search:

best pizza near me

You issue a query.

Every webpage has characteristics that determine whether it matches.

Those are analogous to keys.

The content you finally read is the value.

Exactly the same thing happens inside attention.

Every word generates three vectors:

Query (Q) - What am I looking for?
Key (K) - What information do I offer?
Value (V) - What information should I contribute if selected?

Suppose we have:

The animal didn't cross the road because it was tired.

For the token it:

Its Query might strongly match:

animal

instead of

road

because their semantic representations are more compatible.

Attention is therefore a sophisticated matching process.

The Famous Equation (That Looks Scarier Than It Is)

The Transformer paper defines attention as:

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

Let's decode it piece by piece.

Step 1: Compare Queries Against Every Key

Each Query is compared with every Key using a dot product.

A dot product is simply a similarity score.

Large positive value:

Very relevant.

Near zero:

Mostly unrelated.

Negative:

Probably irrelevant.

Suppose our similarities are:

Word	Score
cat	12
dog	2
road	-1

Higher means stronger relevance.

Notice that we don't compare a Query against just one Key.

Every Query is compared against every Key simultaneously.

If there are 100 tokens in the sentence, each Query computes 100 similarity scores.

Step 2: Why Divide by sqrt(d_k)?

This is the "Scaled" part.

Without scaling, dot products become enormous.

Imagine vectors of length 512.

Even if each component averages only around 1, adding hundreds of multiplications quickly produces very large numbers.

A useful back-of-the-envelope calculation is that the variance of a dot product grows roughly in proportion to d_k.

If:

d_k = 512

then

sqrt(512) is approximately 22.6

Those values are fed into the softmax function.

Softmax contains exponentials.

For example:

exp(22) is roughly 3.5 billion
exp(10) is roughly 22 thousand

A small increase in the input suddenly creates a huge difference in the output.

One score completely dominates.

Everything else effectively becomes zero.

That creates two problems:

unstable gradients
slower learning

Dividing every score by sqrt(d_k) keeps the values in a healthy numerical range.

It's essentially variance normalization.

A remarkably small trick with enormous practical consequences.

Step 3: Softmax Creates Probabilities

Suppose the scaled scores become:

3
2
0

Softmax converts them into something approximately like:

0.71
0.26
0.03

These become attention weights.

Now the model knows:

spend about 71% of attention here
spend about 26% here
mostly ignore the rest

The probabilities always sum to 1.

Step 4: Weighted Sum of Values

Finally those probabilities weight the Value vectors.

Think of it like averaging expert opinions.

Expert A : 70%
Expert B : 25%
Expert C : 5%

The final representation becomes:

0.70 * A + 0.25 * B + 0.05 * C

That weighted average becomes the new representation for the current token.

Instead of copying information from a single location, attention intelligently blends information from multiple relevant tokens.

Why Matrix Multiplication Changed Everything

The equation often looks abstract because it's written with matrices.

That choice was an engineering breakthrough.

Instead of processing one word at a time:

word 1
word 2
word 3
...

the Transformer processes every token simultaneously.

If a sentence contains 128 words,

it computes attention for all 128 together using large matrix multiplications.

Modern GPUs are extraordinarily efficient at matrix multiplication.

This wasn't merely mathematically elegant.

It matched the hardware.

Google's TPUs were designed around massive matrix operations.

NVIDIA GPUs excel at them too.

The algorithm and the hardware reinforced one another.

This is one reason Transformers scaled so dramatically.

Sometimes the biggest breakthrough isn't inventing a new algorithm.

It's inventing one that perfectly matches the hardware already available.

The Hidden Cost: Attention Isn't Free

Attention is powerful.

It is also expensive.

Suppose a sequence contains n tokens.

Every token compares itself with every other token.

That means roughly:

n * n

or simply:

O(n^2)

comparisons.

Double the sequence length:

1,000 tokens
      ->
2,000 tokens

and you perform about four times as much work.

Approximate example:

Tokens	Pairwise Comparisons
1,000	1 million
10,000	100 million
100,000	10 billion

This single operation dominates both memory usage and inference cost.

Much of today's LLM research—including FlashAttention, sparse attention, sliding-window attention, grouped-query attention, and linear attention—is fundamentally about making this computation cheaper without sacrificing quality.

In many ways, modern AI engineering has become an optimization problem built around this one equation.

A Historical Moment Few Papers Ever Achieve

When Ashish Vaswani and seven colleagues published "Attention Is All You Need" in 2017, they were solving a machine translation problem.

They were not trying to build ChatGPT.

Yet within a few years:

OpenAI built GPT on the Transformer architecture.
Google introduced BERT using the same core attention mechanism.
Nearly every frontier LLM adopted Scaled Dot-Product Attention as its computational primitive.

Some research papers introduce new techniques.

Very few redefine an entire field.

This was one of them.

Today, when billions of people interact with ChatGPT, Claude, Gemini, or Llama, they're ultimately benefiting from an idea that occupies only a few lines in a research paper.

Final Thoughts

The beauty of Scaled Dot-Product Attention lies in its simplicity.

Every token asks a question.

Every other token advertises what it knows.

Similarity determines relevance.

Softmax decides how much to trust each source.

The answers are blended into a richer representation.

From those four operations emerged language models capable of writing code, translating languages, solving mathematical problems, generating images, and powering AI assistants used by hundreds of millions of people.

Sometimes revolutions begin not with thousands of lines of code, but with a single elegant equation.

What surprised you most about Scaled Dot-Product Attention? Was it the simplicity of the mathematics, the engineering insight of matching GPUs with matrix operations, or the fact that dividing by sqrt(d_k) turned out to be one of the key ingredients that made today's LLMs train reliably at scale?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

What Actually Happens When You Train an LLM? Following the First 12 Hours of the Original Transformer

Shrijith Venkatramana — Tue, 30 Jun 2026 18:03:36 +0000

In 2017, eight NVIDIA P100 GPUs sat in a Google data center for about twelve hours.

During those twelve hours, they repeatedly did something almost embarrassingly simple.

They picked up a batch of sentences.

Made predictions.

Measured how wrong those predictions were.

Adjusted a few million numbers.

Then did it again.

Exactly 100,000 times.

Those twelve hours produced the base Transformer model described in Attention Is All You Need. The larger version trained for 300,000 optimization steps, taking roughly 3.5 days on the same hardware.

Today, frontier LLMs train on tens of thousands of GPUs for weeks, but if you inspected the training logs, the core loop would still look remarkably familiar.

This article follows that loop.

We'll watch one training run unfold—from raw text on disk to a model that can translate languages—and along the way learn a bit about every intimidating term that deal with training a transformer model.

8:00 AM — Nothing Has Been Learned Yet

Imagine switching on the machine.

The Transformer knows absolutely no language. It doesn't know English, or German. And it doesn't know grammar. It doesn't even have conception of what a "word" is.

Internally it contains millions of parameters—ordinary floating-point numbers initialized almost randomly.

The training data, however, already contains knowledge.

For the English-German task, the authors used the WMT 2014 dataset containing roughly 4.5 million sentence pairs.

A tiny sample might look like this:

English:
The meeting begins tomorrow.

German:
Das Treffen beginnt morgen.

English:
The cat sat on the mat.

German:
Die Katze saß auf der Matte.

Notice what's missing.

Nobody wrote rules like

"Adjectives come before nouns."

"German verbs often appear at the end."

The only supervision is example after example after example.

The model's job is to discover those rules itself.

8:00:01 — The Computer Doesn't See Words

Before training starts, the text is transformed into something GPUs understand.

Integers.

The paper uses Byte Pair Encoding (BPE), introduced a year earlier by Rico Sennrich and colleagues.

Instead of storing every possible English word, BPE builds a vocabulary of common subword pieces.

For example,

unbelievable

might become

un
believ
able

Those pieces become IDs.

un      → 517
believ  → 10328
able    → 294

Why go through this trouble?

Imagine giving every English word its own entry.

You'd need hundreds of thousands of entries, and every new word—"ChatGPT", "Kubernetes", "DeepSeek"—would be unknown.

Subwords solve that elegantly.

Once the model understands "micro", "service" and "architecture", it already has much of what it needs to interpret "microservice architecture", even if it has never encountered the exact phrase before.

Modern tokenizers have evolved, but this basic idea remains.

8:00:02 — The First Batch Arrives

One beginner misconception is that the GPU trains on one sentence at a time.

That would waste almost all of its computational power.

GPUs are throughput machines.

They become efficient only when thousands of arithmetic units work simultaneously.

Instead, the Transformer paper groups examples into batches containing approximately

25,000 source-language tokens
25,000 target-language tokens

or about 50,000 tokens in total.

Think of a factory.

Running one car down an assembly line would be absurd.

Factories move hundreds of products simultaneously because keeping machines idle is expensive.

GPU training works the same way.

Batching is not a machine-learning trick.

It's operations optimization.

8:00:02.4 — The Model Makes Its First Mistake

The first forward pass takes roughly 0.4 seconds.

The model receives

The cat sat on the mat.

and produces...

garbage.

Maybe something equivalent to

House.

Tomorrow.

Blue.

Water.

That isn't failure.

It's exactly what we expect.

Every parameter was random only moments ago.

Now comes the crucial question.

How wrong was the prediction?

The answer is summarized by a single number called the loss.

Everything that follows exists solely to reduce that number.

8:00:02.5 — Which of the 65 Million Parameters Was Responsible?

Suppose I asked you to tune an old radio using sixty-five million knobs.

After hearing static, which knob would you turn?

You wouldn't know.

Yet that's essentially the problem.

The Transformer base model contains about 65 million trainable parameters.

The larger model contains around 213 million.

Backpropagation solves this enormous credit-assignment problem.

Rather than saying

"Parameter #18,423 is wrong,"

it computes

"If this parameter increased slightly, would the loss increase or decrease?"

for every single parameter.

The result is a gigantic map of tiny suggested adjustments called gradients.

Now another algorithm enters the story.

Adam: The Engineer Who Turns the Knobs

The paper uses the Adam optimizer with

β₁ = 0.9
β₂ = 0.98
ε = 10⁻⁹

These aren't arbitrary constants copied from Stack Overflow.

Adam remembers recent gradients, rather like giving the optimization process momentum.

Imagine descending a foggy mountain.

If every step depended only on the slope beneath your feet, you'd zigzag constantly.

Adam remembers where you've been heading over the past several steps, smoothing the journey downhill.

Interestingly, the paper chose β₂ = 0.98 rather than the more familiar 0.999 found in many deep-learning libraries today. That makes Adam respond more quickly to changing gradients—a small but deliberate engineering decision.

Millions of parameters are nudged.

Tiny, incremental changes.

Often by less than one thousandth.

Then the next batch arrives.

9:00 AM — The Strange Equation Everyone Hates

One of the paper's most intimidating equations defines the learning rate.

It looks frightening.

Its purpose is not.

Early in training, every parameter is effectively random.

Large updates can make optimization unstable.

So the authors warm up the learning rate over the first 4,000 optimization steps, gradually increasing it rather than starting at full speed.

After warmup, the learning rate begins shrinking.

Imagine sanding a table.

At first you remove material aggressively.

Near the end you make tiny finishing passes.

Training behaves similarly.

The equation also contains the term (d_{\text{model}}^{-1/2}).

This compensates for model size.

As hidden representations become larger, gradients naturally change scale. Dividing by the square root of the model dimension helps keep parameter updates numerically well behaved as architectures grow.

The equation is just common sensical engineering.

Noon — Preventing the Model From Memorizing

If optimization only chased lower loss, the network could simply memorize the training data.

The paper deliberately makes learning harder.

First comes dropout.

Ten percent of activations are randomly disabled during training.

Every batch therefore sees a slightly different network.

No neuron can become indispensable.

Second comes label smoothing with a value of 0.1.

Instead of pretending the correct next token has probability exactly one, the target distribution is softened slightly.

That sounds counterintuitive.

Yet translation quality improved.

Real language is messy.

There are often several acceptable translations.

Slight uncertainty produces a less overconfident model.

8:00 PM — Twelve Hours Later

After roughly 100,000 optimization steps, the base model has finished training.

The larger model continues until 300,000 steps, taking approximately 3.5 days.

Each step processed about 50,000 tokens.

Back-of-the-envelope, that's around five billion token presentations during the base run—not unique tokens, but training exposures. The same examples are revisited across multiple passes through the dataset.

The paper doesn't stop by reporting translation accuracy.

It also reports FLOPs.

That's significant.

Even in 2017, the authors understood that machine learning was becoming an engineering discipline constrained not only by accuracy, but also by computation.

A model that is 1% better but requires ten times more compute is often a poor engineering trade-off.

That thinking has only become more relevant.

Today, training an LLM is as much about distributed systems, networking, storage bandwidth, GPU utilization, checkpointing and failure recovery as it is about neural networks.

Closing Thoughts

People often remember Attention Is All You Need for introducing self-attention.

Equally important was something less glamorous: it demonstrated a training recipe that scaled.

Large batches kept GPUs busy.

Carefully designed learning-rate schedules stabilized optimization.

Adam made billions of tiny updates practical.

Regularization techniques prevented memorization.

None of these ideas are individually magical. Together, repeated hundreds of thousands of times, they turned random numbers into a model that could translate language.

Nearly a decade later, today's frontier LLMs still follow the same rhythm.

The numbers have changed by orders of magnitude.

The loop has not.

Load a batch. Predict. Measure the loss. Update the weights. Repeat.

That's the heartbeat of every modern language model.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

The Transformer Architecture Behind Modern LLMs: A Developer's Guide to the Diagram That Changed AI

Shrijith Venkatramana — Mon, 29 Jun 2026 17:09:22 +0000

If you've used ChatGPT, Claude, Gemini, or any modern LLM, you've already benefited from one of the most influential transformer architecture in modern AI.

In 2017, a team of researchers at Google published a paper titled "Attention Is All You Need." Hidden inside it was the now-famous architecture diagram shown below—the blueprint for the Transformer.

At first glance, it looks intimidating: boxes, arrows, loops, encoder stacks, decoder stacks, attention blocks...

But underneath the complexity lies a surprisingly elegant idea.

By the end of this article, you'll understand every single component in this diagram, why it exists, how information flows through it, and how all these pieces collaborate to generate human-like language.

Let's build our understanding layer by layer.

The Big Picture: Two Factories Working Together

Before diving into the individual blocks, ignore the details and look at the overall shape.

There are two major halves:

Encoder (left)
Decoder (right)

Think of them as two specialized factories.

Input sentence
      │
      ▼
 ┌───────────────┐
 │    Encoder    │
 └───────────────┘
      │
 Learned representation
      │
      ▼
 ┌───────────────┐
 │    Decoder    │
 └───────────────┘
      │
      ▼
 Generated text

Originally, this architecture was designed for machine translation.

Example:

English:
"The cat sat on the mat."

Encoder understands it.

↓

Decoder generates:

"Le chat était assis sur le tapis."

The encoder's only responsibility is understanding.

The decoder's responsibility is generation.

Today's GPT models actually use only the decoder, while models like BERT use only the encoder. The original Transformer paper contained both because translation requires understanding one language before producing another.

Step 1 — Input Embeddings: Converting Words into Numbers

Computers don't understand words.

They understand vectors.

When the sentence

The cat sat

enters the Transformer, each word is converted into a dense numerical vector.

"The"
↓

[0.17, -0.42, 1.33, ...]

These vectors are called embeddings.

Words with similar meanings naturally end up close together in this high-dimensional space.

For example,

king
queen
prince
princess

occupy nearby regions.

Instead of manually designing these vectors, the Transformer learns them during training.

In the architecture diagram, this is the very first pink box:

Inputs
   │
   ▼
Input Embedding

At this stage, every token is simply represented as a learned numerical vector.

Step 2 — Positional Encoding: Giving Words an Order

Here's an interesting problem.

Attention doesn't inherently know word order.

Consider:

Dog bites man

Man bites dog

Same words.

Completely different meaning.

Since attention processes every word simultaneously, we must explicitly tell the model where each word appears.

That's the purpose of Positional Encoding.

The positional vector is added directly to the embedding.

Embedding

+

Position

=

Final input vector

This explains the small ⊕ symbol in the diagram.

Embedding
      │
      ▼
      ⊕
     / \
Embedding Position

Rather than learning grammar from sequence alone, the model receives positional information immediately.

You can think of it as giving every token both:

what it is
where it occurs

Step 3 — The Encoder Stack: Understanding the Entire Sentence

Now we reach the large box labelled N×.

┌─────────────────────┐
│ Multi-Head Attention│
│ Add & Norm          │
│ Feed Forward        │
│ Add & Norm          │
└─────────────────────┘

Repeated N times

The paper used N = 6.

Modern LLMs often use dozens or even hundreds of layers.

Each encoder layer gradually refines the representation.

Think of reading a paragraph.

Your first reading identifies words.

The second discovers phrases.

The third understands relationships.

The fourth extracts meaning.

Each encoder layer performs another refinement pass.

Step 4 — Multi-Head Attention: The Heart of the Transformer

This is the innovation that changed deep learning.

Suppose the sentence is:

The animal didn't cross the street because it was tired.

What does it refer to?

Attention allows every word to examine every other word before updating its representation.

animal  ←──────────┐
                   │
cross ─────────────┤
                   │
street ────────────┤
                   │
it  ◄──────────────┘

Instead of only looking at nearby words like RNNs, every token has access to the entire sentence.

This allows long-distance dependencies to be captured naturally.

Why Multiple Heads?

One attention mechanism isn't enough.

Different relationships matter.

One head may learn:

grammatical structure

Another:

pronoun resolution

Another:

verb-object relationships

Another:

semantic similarity

Imagine several experts reading the same sentence simultaneously.

Head 1:
Grammar

Head 2:
Meaning

Head 3:
Syntax

Head 4:
Long-range context

Their outputs are combined into a richer representation.

This is why it's called Multi-Head Attention.

Step 5 — Add & Norm: Keeping Training Stable

Notice that after every major block we see

Add & Norm

This performs two operations.

Residual Connection (Add)

Instead of replacing information, we preserve the original.

Output

=

Attention(x)

+

x

This shortcut helps gradients flow through deep networks.

Without it, training hundreds of layers becomes extremely difficult.

Layer Normalization (Norm)

Different layers naturally produce values on different scales.

Layer normalization keeps activations well-behaved.

Think of it as recalibrating measurements after every processing stage.

Without normalization:

Layer 1

0.3

Layer 2

500

Layer 3

0.0004

Training quickly becomes unstable.

Normalization keeps everything numerically manageable.

Step 6 — Feed Forward Networks: Thinking Independently

Attention allows tokens to exchange information.

The Feed Forward layer allows each token to process what it has learned.

For every token independently:

Vector

↓

Linear

↓

Activation

↓

Linear

No interaction happens here.

Instead, this stage performs deeper feature extraction.

An analogy:

Attention is a group discussion.

Feed Forward is everyone quietly thinking afterward.

This alternating pattern—

Discuss

↓

Think

↓

Discuss

↓

Think

is repeated across every Transformer layer.

Step 7 — The Decoder: Generating One Token at a Time

Now we move to the right half of the diagram.

The decoder is responsible for producing text.

Its input isn't the original sentence.

Instead it receives:

<START>

↓

The

↓

The cat

↓

The cat sat

↓

...

Notice the label:

Outputs
(shifted right)

This means that during training, the decoder receives the correct previous token as input while learning to predict the next one.

If the target sentence is:

The cat sat

the decoder sees:

<START>

The

The cat

and learns to predict:

The

cat

sat

This "teacher forcing" strategy makes training much more efficient because the model always conditions on the correct history rather than its own mistakes.

Step 8 — Masked Multi-Head Attention: Preventing Cheating

Imagine predicting the next word in:

The cat sat on the __

If the model could already see

mat

there would be nothing to learn.

So the decoder applies Masked Multi-Head Attention.

Current token

↓

Can see:

Previous words ✔

Future words ✘

During generation:

I love

cannot attend to

pizza

until pizza has actually been generated.

The mask preserves causality.

This is precisely why GPT models generate text one token at a time.

Step 9 — Cross-Attention: Looking Back at the Encoder

The second attention block inside the decoder is different.

Here, the decoder attends to the encoder output.

Encoder

↓

Sentence meaning

↓

Decoder consults it

Suppose we're translating:

"The red car."

While generating

voiture

the decoder continually asks:

Which parts of the original sentence are relevant right now?

This interaction between encoder and decoder is called cross-attention (shown simply as "Multi-Head Attention" in the original diagram, with arrows coming from the encoder stack).

It lets the decoder ground each generated token in the encoded meaning of the source sentence instead of relying only on previously generated words.

Decoder-only models like GPT omit this block because there is no separate encoder to consult.

Step 10 — Linear Layer + Softmax: Choosing the Next Word

After the decoder finishes processing, we finally reach the top of the diagram.

Decoder Output

↓

Linear

↓

Softmax

↓

Output Probabilities

The Linear layer converts the decoder's hidden representation into one score for every token in the vocabulary.

Imagine a vocabulary containing 50,000 words.

The output might look like:

cat      5.8
dog      2.1
apple   -0.4
car      1.7
...

These raw scores (often called logits) aren't probabilities yet.

The Softmax layer transforms them into a probability distribution:

cat      0.81
dog      0.09
car      0.04
apple    0.01
...

The model can then choose the next token—either the most probable one or a sampled alternative depending on the decoding strategy.

That chosen token is fed back into the decoder, and the entire process repeats until an end-of-sequence token is produced.

Putting It All Together: Following the Data Through the Diagram

Now the entire figure becomes much easier to read.

Input text

↓

Input Embedding

↓

Positional Encoding

↓

Encoder Stack (N layers)
    ├─ Multi-Head Attention
    ├─ Add & Norm
    ├─ Feed Forward
    └─ Add & Norm

↓

Context-rich representation

↓

Decoder receives previous outputs
(shifted right)

↓

Output Embedding

↓

Positional Encoding

↓

Masked Multi-Head Attention
(looks only at earlier generated tokens)

↓

Cross-Attention
(consults the encoder output)

↓

Feed Forward

↓

Repeat N layers

↓

Linear

↓

Softmax

↓

Next token probability

↓

Repeat until complete sentence

Every block has a specific role:

Component	Purpose
Input Embedding	Convert tokens into dense vectors
Positional Encoding	Encode word order
Multi-Head Attention	Let tokens exchange information globally
Masked Multi-Head Attention	Prevent access to future tokens during generation
Cross-Attention	Allow the decoder to consult the encoder's understanding
Feed Forward	Transform each token's representation independently
Add & Norm	Stabilize optimization and preserve information via residual connections
Encoder Stack (N×)	Build increasingly rich contextual representations
Decoder Stack (N×)	Generate the output sequence one token at a time
Linear	Produce a score for every vocabulary token
Softmax	Convert scores into probabilities for selecting the next token

A Crisp Summary To Help You Remember This

The Transformer succeeded because it replaced sequential processing with parallel attention, enabling models to reason over entire sequences at once while still generating coherent text token by token.

Almost every major language model today—from GPT and Claude to Llama, Mistral, and Gemini—can trace its lineage back to this deceptively simple diagram. While modern architectures introduce refinements such as rotary positional embeddings, grouped-query attention, mixture-of-experts layers, and optimized decoding strategies, the core ideas remain strikingly similar to those introduced in 2017.

The next time someone says an LLM is "just predicting the next token," remember what's happening under the hood: embeddings capture meaning, positional encodings preserve order, attention weaves relationships across the sequence, feed-forward networks refine those representations, residual connections keep deep networks trainable, and the decoder repeatedly transforms all of that into one probability distribution after another until a coherent response emerges.

What part of the Transformer architecture surprised you the most—the fact that every token can attend to every other token, the masking that enables autoregressive generation, or how such a simple stack of repeated blocks scales to models with hundreds of billions of parameters?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Self-Attention: The Brilliant Idea That Made Large Language Models Possible

Shrijith Venkatramana — Sun, 28 Jun 2026 17:18:46 +0000

How a seemingly simple mathematical trick replaced decades of sequential neural networks and unlocked the age of GPT.

Imagine asking ten software engineers to summarize a pull request.

One engineer reads every line from top to bottom. Another immediately jumps to the files that seem most relevant. A senior engineer skims most of the code but pays close attention to the parts that affect authentication, concurrency, or performance.

The senior engineer isn't processing every line equally.

They're paying attention.

That simple observation eventually became one of the most important ideas in modern machine learning. In 2017, a group of researchers at Google published a paper with an almost understated title: "\"Attention Is All You Need.\" The paper introduced the Transformer, a new neural network architecture that abandoned recurrent networks entirely in favor of one central mechanism: self-attention."

Today, nearly every major Large Language Model—GPT, Claude, Gemini, Llama, DeepSeek, Mistral—builds upon this idea.

Let's understand why.

Before Transformers: Language Was Processed Like a Conveyor Belt

For nearly two decades, sequence models were dominated by Recurrent Neural Networks (RNNs) and later LSTMs and GRUs.

Suppose we have the sentence:

The animal didn't cross the road because it was tired.

An RNN processes it like this:

The
 ↓
animal
 ↓
didn't
 ↓
cross
 ↓
...
 ↓
tired

Every new word updates a hidden state.

If the model wants to understand "it", information about "animal" has already travelled through six or seven intermediate computations.

It's a little like the children's game of telephone. Every time information is passed forward, a little noise is introduced.

The longer the sentence becomes, the harder it is to preserve distant information.

This caused several practical problems:

Long-range dependencies became difficult.
Training was inherently sequential.
GPUs—which thrive on parallel computation—were underutilized.

Even clever improvements like LSTMs only partially solved these issues.

Researchers began asking a different question:

What if every word could simply look at every other word directly?

That question became self-attention.

The Core Idea: Every Word Gets to Read the Entire Sentence

Instead of processing words one after another, self-attention lets every token consult every other token before deciding what it should represent.

Consider:

The trophy didn't fit into the suitcase because it was too small.

When humans read "it", we naturally ask:

trophy?
suitcase?

Our brain briefly looks backward.

Transformers perform the same operation mathematically.

When computing the representation for it, the model assigns attention weights:

Word	Importance
trophy	0.08
suitcase	0.67
small	0.17
fit	0.05
others	0.03

These numbers are not programmed.

They are learned from enormous amounts of text.

The new representation becomes approximately:

representation(it)

=
0.67 × suitcase
+
0.17 × small
+
0.08 × trophy
...

Notice something subtle.

The word it itself never changes.

Instead, its vector representation becomes richer because it incorporates contextual information from the rest of the sentence.

This is why the mechanism is called self-attention.

The sentence is attending to itself.

Why This Was Revolutionary

The Google paper's title—Attention Is All You Need—was intentionally provocative.

At the time, attention mechanisms already existed.

Bahdanau and colleagues had introduced attention in neural machine translation in 2014. However, attention was only an add-on to recurrent networks.

The Transformer asked a far bolder question:

What happens if we remove recurrence completely?

Instead of:

Input
 ↓
LSTM
 ↓
LSTM
 ↓
LSTM

the Transformer became:

Input
 ↓
Self Attention
 ↓
Feed Forward
 ↓
Self Attention
 ↓
Feed Forward

No recurrence.

No convolutions.

Just attention layers stacked dozens—or eventually hundreds—of times.

Many researchers initially viewed this as risky.

Within a year, it became obvious the idea worked astonishingly well.

The Math Is Simple and Elegant

The mathematics often intimidates newcomers, but the underlying idea is straightforward.

Each token produces three vectors:

Query (Q) → What information am I looking for?
Key (K) → What information do I contain?
Value (V) → What information should I contribute?

Think of attending a technical conference.

Every attendee carries:

a list of questions they're interested in (Query),
a badge describing their expertise (Key),
the knowledge they can share (Value).

Conversation happens when someone's questions match another person's expertise.

Mathematically:

score = Query · Key

The dot product measures compatibility.

Large dot product?

Pay attention.

Small dot product?

Ignore.

The scores are normalized using the Softmax function:

weights = softmax(QKᵀ / √d)

The division by √d prevents very large vector dimensions from producing excessively large dot products that would make Softmax saturate. Without this scaling, gradients become small and training becomes unstable.

Finally,

Output = weights × V

Each token becomes a weighted combination of information from every other token.

That's the entire mechanism.

The famous equation occupies only a single line in the original paper.

Yet it changed AI forever.

A Back-of-the-Envelope Calculation: Why Attention Is Expensive

Self-attention's biggest strength is also its biggest weakness.

Suppose a context contains 4,096 tokens.

Every token compares itself against every other token.

Total comparisons:

4096 × 4096

≈ 16.8 million

Now consider modern models.

8,192 tokens
32 attention heads
dozens of Transformer layers
billions of parameters

The number of operations quickly reaches the trillions during training.

This explains why training frontier models requires thousands of GPUs running continuously for weeks or months.

The economics become equally striking.

A single GPU might cost tens of thousands of dollars. Training clusters contain thousands of them.

Electricity, cooling, networking, storage, engineering time, and failed experiments all contribute to training costs that can reach tens or even hundreds of millions of dollars for the largest models.

This computational expense has also motivated an entire research field devoted to making attention cheaper.

Techniques such as FlashAttention, grouped-query attention, sparse attention, and linear attention all attempt to preserve the quality of self-attention while reducing memory usage or computational complexity.

Ironically, many innovations in modern LLM engineering are really innovations in making self-attention practical at scale.

Why Self-Attention Became the Foundation of LLMs

Language isn't fundamentally sequential.

Relationships often span entire documents.

A variable declared hundreds of lines earlier influences the current line of code.

A pronoun refers to a noun introduced several paragraphs ago.

An API call depends on documentation presented earlier in the conversation.

Self-attention naturally models these relationships.

It also parallelizes beautifully.

Unlike RNNs, every token in a sequence can be processed simultaneously on modern GPUs.

That single architectural decision dramatically increased hardware utilization and enabled models to scale from millions of parameters to today's trillion-parameter frontier.

Perhaps the greatest lesson is that breakthroughs are not always about making systems more complicated.

Sometimes they're about removing assumptions.

The Transformer removed the assumption that language must be processed one word at a time.

Everything that followed—from GPT-2 to ChatGPT—was built on that realization.

Final Thoughts

It's easy to think of GPT as an impossibly complex black box.

But underneath the billions of parameters lies a surprisingly elegant principle.

Every word asks:

Which other words should I pay attention to?

That single question replaced decades of recurrent architectures and reshaped artificial intelligence.

Sometimes, the most revolutionary ideas aren't new ways of computing.

They're new ways of deciding what deserves attention.

What surprised you most about self-attention?

Was it that the core algorithm fits into a single equation, or that one architectural decision replaced decades of recurrent neural networks? I'd love to hear your thoughts—or any clever analogies you've found useful when explaining Transformers to other developers.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Sequence Transduction: The Forgotten Problem That Led to Modern LLMs

Shrijith Venkatramana — Sat, 27 Jun 2026 17:15:41 +0000

Most developers think large language models were built to predict the next word. They weren't—not at first.

If you travel back to the early 2010s, the hardest problems in AI weren't writing poems or generating code. They were translating English into French, converting speech into text, and summarizing documents. These were all instances of the same challenge: sequence transduction.

The term appears almost casually in the opening paragraph of the Transformer paper:

"...sequence modeling and transduction problems such as language modeling and machine translation."

Today, almost everyone knows the Transformer. Very few remember the problem it was invented to solve.

Ironically, solving sequence transduction turned out to create the foundation upon which modern LLMs would later emerge.

Let's explore why.

What Exactly Is Sequence Transduction?

Imagine you own a factory.

A sequence modeling problem asks:

"Given everything that has happened so far, what comes next?"

Like predicting the next product coming off the conveyor belt.

The cat sat on the _____
                ↓
               mat

This is language modeling.

A sequence transduction problem is larger idea.

Instead of predicting one missing piece, you transform an entire sequence into another.

English
↓

"The weather is nice."

↓

French

"Il fait beau."

Audio
↓

Waveform

↓

Text

"Welcome everyone."

Buggy code

↓

Correct code

Different input.

Different output.

Often different lengths.

The model must understand the entire source or at least large parts of it before generating the target.

In hindsight, modern AI assistants spend almost all of their time doing sequence transduction:

translating languages
summarizing reports
generating SQL
converting Python to Rust
explaining stack traces
producing commit messages
writing documentation

They all reduce to:

Input sequence → Output sequence

Why Was This Such a Hard Problem?

Humans underestimate how much memory translation requires.

Consider translating:

"The committee, after reviewing several proposals over three months, finally approved the budget."

Suppose you're translating into German.

The verb may not appear until the end.

To translate correctly, the model must remember:

who performed the action
tense
plurality
grammatical structure
dependencies dozens of words apart

Early neural networks processed text one word at a time.

Word₁ → hidden state
              ↓
Word₂ → hidden state
              ↓
Word₃ → hidden state

Everything had to be compressed into one hidden vector.

It was like asking someone to summarize an entire novel using only one sticky note.

Eventually information disappears.

This became known as the long-range dependency problem.

The Rise—and Limits—of Recurrent Neural Networks

During the late 1980s and 1990s, researchers developed Recurrent Neural Networks (RNNs) to process sequential data.

Unlike ordinary neural networks, RNNs reused the same parameters at every time step.

Instead of building a different network for every word, one network repeatedly updated an internal memory.

Mathematically:

hidden_state = f(previous_hidden_state, current_input)

The same computation runs repeatedly.

This parameter sharing was elegant.

Suppose an RNN contains one million parameters.

A thousand-word paragraph still uses one million parameters—not a billion.

The network simply reuses them.

Economically, this was attractive. But computationally, it was painful.

Everything had to happen sequentially.

Word 500 could not begin until word 499 finished.

No parallelism. No GPUs in picture. Training was slow.

LSTMs: Teaching Neural Networks to Remember

In 1997, Sepp Hochreiter and Jürgen Schmidhuber introduced one of the most influential ideas in deep learning:

Long Short-Term Memory (LSTM).

Instead of blindly overwriting memory every step, the network learned gates.

Think of memory like a whiteboard.

Each word asks three questions:

Should I erase something?
Should I remember this?
Should I reveal this later?

Those questions became three learned gates.

Forget gate.

Input gate.

Output gate.

Instead of forcing every piece of information through the same bottleneck, the model learned what deserved long-term storage.

A surprisingly intuitive analogy is human note-taking.

Most conversations are forgotten.

A few facts are written into your notebook.

LSTMs learned which facts deserved the notebook.

For over a decade, LSTMs dominated speech recognition, handwriting recognition, language translation, and time-series forecasting.

Google, Apple, Microsoft, and Baidu all deployed enormous production systems powered by them.

The Encoder–Decoder Revolution

Around 2014, another breakthrough appeared.

Instead of using one RNN/LSTMs for everything, researchers separated the task into two parts.

Input sentence
      ↓
Encoder
      ↓
Meaning vector
      ↓
Decoder
      ↓
Output sentence

This architecture became known as the sequence-to-sequence (Seq2Seq) model.

For the first time, neural networks learned translation end-to-end.

No phrase tables or handcrafted grammar or brittle rules. It was just millions of examples.

One famous anecdote came from Google.

Traditional statistical machine translation systems consisted of dozens of independently engineered components accumulated over years.

Neural machine translation replaced much of that complexity with a single differentiable model trained from data. In 2016, Google reported that its neural system substantially reduced translation errors across multiple language pairs while simplifying the overall pipeline.

This represented an engineering improvement for sure, but more importantly it was a philosophical shift.

Instead of programming language knowledge -- we trained it.

Attention Changed Everything

The Seq2Seq model still had one weakness.

Everything had to fit inside one vector.

Information gets lost.

In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed attention.

Instead of remembering everything, just look back whenever necessary.

While generating each output word, the decoder asks:

Which input words matter right now?

Not every word.

Only the relevant ones.

Translation suddenly became much easier.

Long sentences improved dramatically.

From Translation Engine to ChatGPT

The Transformer paper in 2017 -- instead of improving recurrent networks, removed recurrence entirely.

Every word could attend directly to every other word.

Parallel computation became possible.

Training speed increased enormously.

GPUs became dramatically more efficient because every token in a sequence could be processed simultaneously rather than one after another.

Even more interesting was the economics.

Suppose translating a sentence of 100 words with an RNN requires roughly 100 sequential computation steps.

A Transformer still performs similar amounts of arithmetic overall, but many of those operations can execute in parallel on modern accelerators.

The wall-clock training time drops dramatically because GPUs are optimized for large batches of matrix multiplications rather than long chains of sequential dependencies.

That operational advantage—not merely higher accuracy—made scaling practical.

The remarkable twist is that the architecture built to solve translation generalized astonishingly well.

Replace:

English → French

with

Question → Answer

Code → Documentation

Prompt → Python program

The underlying problem barely changes.

It remains sequence transduction.

Modern LLMs still perform next-token prediction during training.

But from a developer's perspective, they are universal transduction engines.

Every prompt is transformed into another sequence.

The interface changed.

The underlying abstraction survived.

Final Thoughts

The history of AI is often told as a story about predicting the next word.

That story is incomplete.

For decades, researchers wrestled with a harder question:

How do we transform one complex sequence into another while preserving meaning?

That single question drove the invention of encoder–decoder architectures, LSTMs, attention mechanisms, and ultimately the Transformer itself.

The next time you ask an LLM to refactor code, summarize a meeting, or generate a SQL query, remember what it's really doing.

Not merely predicting words.

Performing sequence transduction at an extraordinary scale.

What surprised you most about this history? Did you always think LLMs grew out of language modeling, or is it more useful to think of them as the latest—and perhaps most powerful—generation of sequence transduction systems?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

How We Actually Measure Whether an LLM's Output Is Good - BLEU, COMET and BLEURT

Shrijith Venkatramana — Fri, 26 Jun 2026 18:36:27 +0000

An AI model writes a paragraph. It sounds fluent. It looks convincing. But how do you know whether it's actually good?

This deceptively simple question has occupied researchers for more than two decades.

Long before ChatGPT, machine translation researchers faced exactly the same problem. Human evaluation was expensive, inconsistent, and painfully slow. If every new model required thousands of humans to compare translations, research would crawl.

That necessity gave rise to BLEU, one of the most influential evaluation metrics in AI history. Years later, as language models became better at paraphrasing and reasoning, BLEU started to show its age. Researchers responded with learned metrics like BLEURT and COMET, which use neural networks to judge language much more like humans do.

Interestingly, this mirrors software engineering itself. We first wrote simple unit tests, then integration tests, and today we increasingly rely on sophisticated observability systems. Evaluation metrics for LLMs have undergone a similar evolution.

Let's see why.

Before BLEU: The Evaluation Bottleneck

Imagine you're building Google Translate in 2001.

Every time your team improves the model, someone has to read thousands of translated sentences and score them.

Suppose a single sentence pair takes only 20 seconds to judge.

Evaluating 50,000 sentences would require nearly 280 human-hours.

Now imagine dozens of experiments every week.

Evaluation—not training—quickly becomes the bottleneck.

Researchers at IBM, led by Kishore Papineni, introduced BLEU (Bilingual Evaluation Understudy) in 2002 to automate this process.

Their idea was surprisingly simple:

If a machine translation resembles what professional translators write, it's probably good.

This became one of the most cited papers in natural language processing.

BLEU: Counting Shared Phrases

BLEU compares a model's output against one or more human reference translations.

Suppose the reference is:

The cat is sitting on the mat.

The model produces:

The cat sat on the mat.

Many words and short phrases overlap.

Now consider:

A feline rested indoors.

A human recognizes this as a perfectly reasonable translation.

BLEU mostly doesn't.

Why?

Because BLEU isn't measuring meaning.

It measures shared n-grams—contiguous sequences of words.

The score combines:

word matches (1-grams)
two-word phrases
three-word phrases
four-word phrases

It also penalizes outputs that are suspiciously short.

High-level intuition:

More matching phrases → higher score
Longer matching phrases → even better
Too short → penalty

This simple idea turned out to correlate surprisingly well with human judgments across large datasets.

When the famous Transformer paper Attention Is All You Need reported 28.4 BLEU on the WMT English-German benchmark, that represented roughly a 2 BLEU improvement over previous systems—a significant jump that helped establish the Transformer as the new state of the art.

Why BLEU Eventually Broke Down

BLEU assumes that good translations look similar.

Modern LLMs don't.

Consider these summaries.

Reference:

The meeting was postponed because the client requested additional documentation.

Output A:

The meeting was delayed after the client asked for more documents.

Output B:

Client requested more paperwork, so the meeting moved.

Humans would probably give both excellent scores.

BLEU prefers whichever shares more exact phrases.

Now imagine asking ChatGPT:

Explain recursion like I'm five.

There are hundreds of excellent answers.

BLEU expects one.

This becomes even worse for:

summarization
question answering
code explanations
dialogue
reasoning

As models became more creative, exact word overlap became a poor proxy for quality.

Researchers needed evaluation metrics that understood meaning rather than wording.

BLEURT: Teaching AI to Judge AI

Google Research introduced BLEURT in 2020.

Instead of counting words, BLEURT fine-tunes a pretrained Transformer to predict human evaluation scores.

Think of it as hiring a reviewer instead of using a spell checker.

During training, BLEURT sees:

candidate answer
reference answer
human quality score

After millions of examples, it learns patterns humans value:

preserved meaning
factual consistency
grammatical quality
fluency

An interesting engineering trick made BLEURT particularly effective.

Human-scored datasets are relatively small.

The researchers first generated large amounts of synthetically corrupted text—introducing deletions, substitutions, and paraphrases—to pretrain the evaluator before fine-tuning on expensive human judgments.

This significantly reduced the amount of labeled data needed.

COMET: Learning from Human Preferences

Around the same time, researchers at Unbabel developed COMET.

Like BLEURT, COMET uses a neural network.

But it has access to something BLEURT often doesn't:

original source sentence
reference translation
candidate translation

That additional context matters.

Suppose the French sentence is:

Il fait froid.

One candidate says:

It is cold.

Another says:

It is freezing.

Without seeing the original sentence, both might seem acceptable.

With the source available, COMET can better judge whether meaning has shifted.

Modern COMET models consistently show stronger correlation with professional human evaluators than BLEU across many translation benchmarks.

Today, COMET is frequently reported alongside BLEU in machine translation research.

The Economics of Evaluation

Training frontier models can cost millions of dollars.

Evaluation, surprisingly, can become expensive too.

Imagine comparing three model versions.

Each produces answers for:

10 benchmark datasets
5,000 prompts each

That's 150,000 outputs.

If humans spend only 15 seconds per evaluation:

150,000 × 15 seconds ≈ 625 hours

At $40/hour for expert annotators, that's $25,000 for a single evaluation round.

And that's before measuring agreement between multiple reviewers.

Automatic metrics dramatically reduce this cost.

A common workflow today looks like:

Evaluate every experiment automatically.
Keep only the best-performing candidates.
Send those few models to human reviewers.

The automatic metric acts as a high-quality filter rather than a replacement for humans.

Where We Are Today

No single metric captures quality perfectly.

BLEU remains valuable because:

it's simple
reproducible
historically comparable

BLEURT improves semantic understanding by learning from human judgments.

COMET goes even further by incorporating the original source sentence and demonstrating stronger agreement with professional evaluators.

For frontier LLMs, evaluation has become even broader.

Researchers increasingly combine:

learned metrics like COMET
human preference studies
benchmark suites
domain-specific tests
LLM-as-a-judge evaluations

The lesson is larger than machine translation.

As AI systems become more capable, evaluating them becomes an AI problem itself.

The future of language models may depend just as much on better judges as on better generators.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Synthetic Data: The Hidden Ingredient That Made Modern LLMs Scale

Shrijith Venkatramana — Thu, 25 Jun 2026 18:07:49 +0000

Everyone knows modern AI runs on data. Fewer people realize that today's most capable AI systems increasingly learn from data they created themselves.

For years, the common belief was simple: collect more human-written text, train bigger models, and intelligence would emerge.

That worked—until it didn't.

By 2022, frontier AI labs had consumed a significant fraction of the publicly available high-quality text on the internet. The next leap wasn't going to come from scraping another few billion web pages.

Instead, researchers turned to something different:

They started letting AI generate its own training data.

Today, synthetic data powers reasoning models, coding assistants, math solvers, autonomous agents, and many of the capabilities developers now take for granted. In many cases, the most valuable dataset isn't written by humans anymore—it's produced by another AI.

Let's explore how that works out.

The Factory Analogy

Imagine you're building a car factory.

Initially, every part has to be handcrafted by skilled workers. Production is slow and expensive.

Eventually, you build machines that manufacture car parts automatically.

Now the factory spends less time making parts manually and more time building better machines that manufacture even better parts.

Synthetic data works similarly.

Initially:

Humans write examples
Humans solve problems
Humans explain reasoning

Eventually:

AI generates new examples
AI writes explanations
AI creates harder problems
AI critiques and improves its own answers

The model becomes both the student and one of the teachers.

This Idea Was Proven Long Before ChatGPT

Perhaps the most famous example isn't from language models at all.

In 2017, DeepMind introduced AlphaGo Zero.

Earlier versions of AlphaGo learned partly from millions of expert human games.

AlphaGo Zero didn't.

It started knowing only the rules of Go.

Then it played against itself.

Millions of games later, it became stronger than every previous version—and stronger than every human on Earth.

No additional human demonstrations.

Only synthetic experience generated through self-play.

Researchers realized something profound:

Once a system becomes good enough, it can manufacture experiences that are more useful than collecting additional human ones.

That idea quietly became one of the foundations of modern AI.

What Does Synthetic Data Look Like for LLMs?

Those less acquainted often imagine synthetic data as "ChatGPT writing more paragraphs."

That's only a tiny piece.

Modern models generate many different kinds of training data:

Question Generation

Instead of waiting for humans to ask questions, models invent thousands of new ones.

Example:

Human:
How do binary search trees work?

Synthetic examples:

Explain AVL trees.
Compare B-trees with BSTs.
Design a filesystem index.
Implement interval trees.

One human question becomes hundreds of training examples.

Reasoning Traces

Instead of merely generating answers:

The model produces detailed reasoning explaining how it reached the answer.

Those reasoning traces become valuable training material for future models.

Code

A coding model can generate:

buggy code
corrected versions
optimized implementations
documentation
test cases

One programming problem becomes an entire software engineering dataset.

Harder Problems

Models can intentionally create problems just beyond their current capability.

Just like a good teacher gradually increases difficulty, the dataset evolves as the model improves.

The Economics Are Incredible

Suppose hiring experts costs roughly:

$40 per hour
each example takes 3 minutes

That's approximately $2 per example.

A million examples?

Around $2 million.

Now imagine a frontier model generates one million candidate examples overnight.

Even after filtering aggressively, perhaps only 20% are worth keeping.

That's still 200,000 useful examples produced in hours instead of months.

Of course, GPUs aren't free.

But frontier labs already own massive compute clusters.

Once the infrastructure exists, generating another million examples is often dramatically cheaper—and much faster—than coordinating thousands of human annotators around the world.

Synthetic data changes the limiting resource from human labor to compute.

That is a profound shift.

Why Doesn't the Model Eventually Teach Itself Nonsense?

A natural concern is:

"If AI keeps learning from AI, won't errors compound forever?"

Absolutely—if done carelessly.

Modern pipelines therefore look more like factories with quality control than simple generators.

A typical loop is:

Generate
      ↓
Verify
      ↓
Filter
      ↓
Rank
      ↓
Train

Only a fraction of generated examples survive.

For coding tasks:

Does the code compile?
Do unit tests pass?

For mathematics:

Is the answer verifiably correct?

For reasoning:

Do multiple models agree?
Does a stronger model approve?

Synthetic data is valuable precisely because most of it gets thrown away.

Quality matters far more than quantity.

How You can Put This Into Practice in Your Apps

Many developers think synthetic data is something only OpenAI or DeepMind can use.

Not anymore.

Suppose you're building an AI assistant for SQL.

Instead of manually writing 5,000 examples, you might:

Write 100 excellent examples.
Ask a strong model to generate variations.
Automatically execute the SQL.
Keep only queries that execute correctly.
Fine-tune on the verified dataset.

Or imagine building an AI tutor.

Generate:

beginner questions
intermediate questions
trick questions
incorrect student answers
ideal explanations

One carefully designed seed dataset can grow into thousands of high-quality training examples.

The bottleneck becomes designing good verification systems—not endlessly producing data by hand.

That's a very different engineering problem.

The Bigger Picture

Scaling laws taught us that more compute and more data generally produce better models.

Synthetic data adds a fascinating twist:

The model itself becomes part of the data-generation pipeline.

Instead of relying solely on humanity's existing knowledge, AI systems increasingly create new training experiences for future AI systems.

In some domains—coding, mathematics, games, and formal reasoning—that approach is already proving remarkably effective because correctness can often be verified automatically.

It is one of the reasons today's reasoning models feel dramatically more capable than models from just a few years ago.

Ironically, one of the biggest breakthroughs in machine learning wasn't finding more human data.

It was discovering that, with the right safeguards, machines can help create the next generation of training data themselves.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

The Scaling Laws That Made LLMs Work

Shrijith Venkatramana — Wed, 24 Jun 2026 18:58:52 +0000

How the AI industry accidentally discovered that "just make it bigger" was one of the most important scientific findings of the decade

In 2018, many researchers believed language models were clever toys.

They could autocomplete text, generate amusing sentences, and occasionally fool people for a paragraph or two. But few expected them to become software engineers, researchers, tutors, designers, and writing assistants.

Then something strange happened.

Teams at OpenAI, Google, DeepMind, Anthropic and elsewhere kept increasing three things:

Model size
Training data
Compute

And performance kept improving.

Not linearly.

Not randomly.

Predictably.

The shocking discovery was that intelligence-like capabilities emerged from scale itself.

Today, ChatGPT, Claude, Gemini, and other frontier models exist largely because researchers discovered scaling laws—empirical mathematical relationships that revealed how performance improves as models become larger and are trained on more data.

This is the story of that discovery, why it mattered, and why it changed the economics of software forever.

Before Scaling Laws: The Era of Clever Tricks

For decades, AI progress often came from clever architecture changes.

Researchers would invent:

Better feature engineering
Better optimization algorithms
Better linguistic rules
Better neural network structures

Progress was often irregular.

A breakthrough would appear.

Then improvements would stall.

Many people assumed future progress would continue this way.

Then deep learning arrived.

Researchers began noticing something unusual.

A bigger neural network often outperformed a smaller one.

A lot.

Even when nobody fully understood why.

One famous observation came from researchers at Google working on machine translation.

Instead of hand-crafting linguistic rules, larger neural networks trained on larger datasets simply worked better.

The trend kept repeating.

The AlexNet Lesson: Compute Can Beat Cleverness

A key moment occurred in 2012.

At the ImageNet competition, a neural network called AlexNet built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton dramatically outperformed competitors.

The architecture was important.

But equally important was something less glamorous:

They used GPUs.

Lots of compute.

The lesson was subtle but profound:

More computation could unlock capabilities that smaller systems never exhibited.

This idea would later become the foundation of modern LLM development.

The OpenAI Scaling Laws Paper That Changed Everything

In 2020, researchers at OpenAI published a landmark paper:

"Scaling Laws for Neural Language Models"

Authored by:

Jared Kaplan
Sam McCandlish
and colleagues

The paper reported a surprising result.

Language model loss followed a smooth power-law relationship with:

Parameter count
Dataset size
Training compute

Instead of hitting obvious plateaus, performance improved according to remarkably predictable mathematical curves.

The researchers found relationships resembling:

L(N) is proportional to N^(-alpha)

where:

L = loss
N = number of parameters
α = scaling exponent

The exact constants differed across experiments, but the important insight was this:

Every additional order of magnitude in scale delivered measurable gains.

No magic tricks required.

No fundamentally new algorithms required.

Just scale.

This result was shocking because many researchers expected diminishing returns to arrive much sooner.

Instead, the curves kept going.

The Back-of-the-Envelope Economics

Let's build intuition.

Imagine a model with:

1 billion parameters

Suppose increasing it to:

10 billion parameters

reduces error by a meaningful amount.

Then increasing to:

100 billion parameters

reduces error again.

Each improvement costs vastly more compute.

But here's the key:

For large organizations, even small quality improvements are worth enormous amounts of money.

Consider search engines.

If improving answer quality by 1% generates hundreds of millions of dollars in user value, spending tens of millions on training becomes rational.

The economics start resembling semiconductor manufacturing.

The biggest players can afford massive upfront investment because performance gains compound downstream.

This is one reason frontier AI rapidly became a contest among organizations with access to:

Massive capital
GPU clusters
Infrastructure expertise

Scaling laws transformed AI from a pure research problem into an industrial production problem.

Chinchilla: The Industry Discovers It Was Scaling Wrong

Then another surprise arrived.

In 2022, researchers at DeepMind published the famous Chinchilla paper "Training Compute-Optimal Large Language Models," led by Jordan Hoffmann.

The team discovered something important.

Many models were too large relative to the amount of training data they consumed.

The industry had been spending enormous compute training gigantic models that were under-trained.

Chinchilla showed that for a fixed compute budget, better performance often comes from:

Smaller models
More tokens

rather than:

Larger models
Fewer tokens

The result fundamentally changed training strategies across the industry.

Many later frontier models incorporated lessons from Chinchilla-style compute-optimal training.

Why Emergent Abilities Appeared

One of the most fascinating observations came from large language models exhibiting capabilities not visible in smaller versions.

Examples included:

Multi-step reasoning
Code generation
Translation
Few-shot learning
Tool usage

A small model might completely fail a task.

A larger version suddenly succeeds.

Researchers called these behaviors emergent abilities.

The exact mechanisms remain debated.

However, scaling laws provided an important clue.

If performance improves smoothly on underlying representations, task-level capabilities may appear abrupt only because evaluation thresholds are discrete.

For example:

45% accuracy feels useless
55% accuracy feels useful

A small continuous improvement underneath can create a seemingly sudden jump in usefulness.

This observation continues to influence modern research into reasoning models.

The Operations Story Matters As Well

The public often imagines AI breakthroughs occurring through genius insights alone.

The reality is much messier.

Scaling laws forced organizations to become experts in:

Distributed systems
Networking
Data pipelines
Storage systems
Cluster scheduling
GPU utilization
Fault tolerance

Training frontier models became one of the largest computing operations ever attempted.

Modern training runs can involve tens of thousands of GPUs operating simultaneously.

At that scale, hardware failures become routine.

Engineers must design systems assuming components will constantly fail.

Ironically, many advances enabling modern AI came not from machine learning itself but from classical systems engineering.

The people building the training infrastructure often look more like distributed systems engineers than traditional AI researchers.

Why Developers Should Care

Scaling laws explain why capabilities keep arriving that seem surprising.

Many developers ask:

"How did models suddenly become good at coding?"

The answer is often less mysterious than it appears.

A large portion of progress comes from moving further along predictable scaling curves.

More compute.

More data.

More parameters.

Better optimization.

The resulting improvements accumulate until tasks become economically useful.

This perspective is valuable because it reframes AI progress.

Instead of viewing each new model as a miracle, we can see many advances as the expected outcome of operating larger and more efficient training systems.

The future may contain architectural breakthroughs.

But one lesson from the past decade is difficult to ignore:

Scale itself turned out to be one of the most important algorithms.

Conclusion

One of the great scientific surprises of modern AI is that intelligence-like capabilities did not emerge primarily from increasingly clever hand-designed systems.

They emerged from discovering a predictable relationship between computation and capability.

The researchers who uncovered scaling laws effectively found a map.

That map allowed organizations to forecast future performance before spending billions of dollars building larger systems.

Few discoveries have reshaped an industry so quickly.

The next time a new language model appears with capabilities that seem impossibly better than its predecessor, it is worth remembering:

The improvement may not be magic.

It may simply be another point on a scaling curve that researchers have been following for years.

If scaling laws continue holding for another decade, do you think future breakthroughs will come primarily from more compute, better architectures, or entirely new paradigms beyond today's transformers?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Reinforcement Learning with Verifiable Rewards: Why AI is Learning to Grade Its Own Homework

Shrijith Venkatramana — Tue, 23 Jun 2026 19:56:18 +0000

Large Language Models have gotten remarkably good at generating text.

But there has always been a fundamental problem:

How do you tell an AI whether its answer is actually correct?

For creative writing, opinions, brainstorming, and conversations, correctness is fuzzy. Human feedback is usually required.

But what about problems where correctness can be objectively verified?

Is the code passing the tests?
Did the SQL query return the expected result?
Does the mathematical proof produce the correct answer?
Does the generated webpage match the specification?

This simple observation is driving one of the most interesting developments in modern AI training:

Reinforcement Learning with Verifiable Rewards (RLVR).

Instead of asking humans to score every answer, we let reality score it.

And that changes everything.

The Traditional RLHF Approach

Most developers have heard of Reinforcement Learning from Human Feedback (RLHF).

The basic process looks like this:

Model generates answers.
Humans rank the answers.
A reward model learns human preferences.
Reinforcement learning optimizes against that reward model.

Conceptually:

Question
    ↓
Model Output
    ↓
Human Evaluation
    ↓
Reward Signal
    ↓
Model Improvement

This worked well for making models more helpful, harmless, and conversational.

But it has a major limitation:

Humans are expensive.

You need thousands or millions of human judgments.

Even worse, humans often disagree.

Ask ten programmers whether a piece of code is elegant and you'll get eleven opinions.

The Key Insight: Some Tasks Are Self-Verifying

Now imagine a different task:

Write a Python function that reverses a linked list.

You don't necessarily need a human reviewer.

You can simply run:

pytest

If all tests pass:

Reward = 1

If tests fail:

Reward = 0

The reward becomes objective.

This is the central idea behind RLVR.

Instead of asking:

"Does a human like this answer?"

we ask:

"Can we verify this answer automatically?"

Whenever verification is possible, reward generation becomes dramatically cheaper and more scalable.

Why This Works So Well for Coding

Coding is one of the most natural domains for RLVR.

Consider a coding benchmark:

Input:
Implement binary search.

Output:
Generated code

Verification is straightforward:

run_tests()

If:

assert binary_search([1,2,3],2) == 1

passes for all test cases, the model receives a high reward.

Otherwise it receives a low reward.

The model gradually learns patterns that lead to successful execution.

Over millions of examples, it begins discovering:

Better debugging strategies
Better decomposition strategies
Better reasoning chains
Better code structures

without needing humans to manually inspect every solution.

This is one reason coding models have improved so rapidly in recent years.

Beyond Coding: Mathematics

Mathematics is another ideal RLVR environment.

Suppose the task is:

Solve:
127 × 348

The final answer can be checked automatically.

Even more interesting:

Find x:
2x + 5 = 17

Verification is trivial:

Substitute x
Check equation

Correct answer?

Reward = 1.

Incorrect answer?

Reward = 0.

This allows models to practice enormous numbers of mathematical problems without requiring armies of human annotators.

Many recent reasoning-focused models have benefited heavily from this kind of training.

What Is Actually Being Optimized?

Under the hood, RLVR still looks like reinforcement learning.

The model generates a solution:

State → Action → Outcome

The difference is the source of the reward.

Traditional RLHF:

Reward = Human Preference

RLVR:

Reward = Verifiable Correctness

A simplified objective looks like:

maximize E[reward]

where reward comes from an automated verifier.

The verifier might be:

Unit tests
Mathematical checking
Compilation success
Benchmark execution
Formal proof validation
Simulation outcomes

The model is effectively searching for behaviors that maximize success rates.

An Example: Training a Coding Model

Imagine training an AI on algorithmic problems.

For each problem:

Problem
 ↓
Model generates solution
 ↓
Compile
 ↓
Run tests
 ↓
Assign reward

Example:

def factorial(n):
    return n

Tests:

assert factorial(5) == 120

Fails.

Reward:

The model tries another approach:

def factorial(n):
    if n <= 1:
        return 1
    return n * factorial(n - 1)

Tests pass.

Reward:

Over time, reinforcement learning shifts probability mass toward successful behaviors.

The model isn't memorizing solutions.

It's learning patterns associated with success.

The Hidden Superpower: Scaling Rewards

The most important consequence of RLVR is not accuracy.

It's scalability.

Suppose you want:

10 million training examples
100 million training examples
1 billion training examples

Human evaluation becomes impossible.

Automated verification remains feasible.

Once a verifier exists, reward generation can scale almost indefinitely.

This transforms the economics of model training.

Instead of hiring more evaluators, you simply generate more problems and run more verifications.

Many researchers believe this is one of the major reasons reasoning and coding models have improved so quickly over the last few years.

Limitations and Open Problems

RLVR is powerful, but it is not universal.

Many important tasks lack objective verification.

Examples include:

Writing a compelling novel
Designing a great product strategy
Creating a persuasive marketing campaign
Conducting a nuanced negotiation

For these domains, correctness is subjective.

Human judgment remains necessary.

Another challenge is reward hacking.

If a model discovers shortcuts that exploit the verifier rather than solving the underlying problem, training can become misleading.

The verifier itself must be robust.

In practice, designing good reward functions is often harder than training the model.

Final Thoughts

For years, the AI community focused on teaching models through human preferences.

RLVR introduces a different idea:

Whenever reality can verify an answer, let reality provide the reward.

For coding, mathematics, theorem proving, scientific reasoning, and other objective domains, this approach dramatically reduces the need for human supervision while enabling massive training scale.

The result is a new generation of models that aren't just learning from people.

They're learning from whether their outputs actually work.

And that may be one of the most important shifts in modern AI training.

If you were training an AI for your domain, what would serve as the verifier? Unit tests, simulations, customer metrics, formal proofs, or something else entirely?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

TPUs vs GPUs: How Google's Tensor Processing Units Actually Work

Shrijith Venkatramana — Sun, 21 Jun 2026 16:44:01 +0000

Machine learning engineers spend countless hours optimizing models, tweaking architectures, and squeezing performance out of hardware.

Yet many developers who train large models today have only a vague understanding of the machines doing the actual work.

Ask a developer how a GPU works, and you'll usually hear something about "lots of parallel cores."

Ask how a TPU works, and the answer is often, "Google made a chip for AI."

But the design differences are much more interesting than that.

TPUs weren't built as faster GPUs. They were built around a different assumption: that neural networks spend most of their time performing enormous matrix multiplications. Once you accept that premise, the entire chip architecture changes.

Let's explore how TPUs work, why Google built them, and where they outperform GPUs.

Why Deep Learning Is Mostly Matrix Multiplication

At a high level, modern neural networks are giant collections of matrix operations.

Consider a simple transformer layer:

output = X @ W

Where:

X is the input activation matrix
W is the weight matrix

Under the hood, this becomes millions or billions of multiply-and-add operations.

For example:

A (4096 x 4096)
×
B (4096 x 4096)
=
C (4096 x 4096)

This single operation contains over 68 billion multiply-accumulate computations.

Training and inference repeatedly execute these operations.

The hardware question becomes:

What is the fastest possible machine for multiplying giant matrices?

GPUs and TPUs answer this question differently.

How GPUs Became the First AI Accelerators

GPUs were never originally designed for machine learning.

They were built to render graphics.

Rendering a video game requires performing similar operations on millions of pixels simultaneously.

This naturally led GPU manufacturers to create architectures containing thousands of lightweight processing cores.

A simplified GPU architecture looks like this:

CPU
 |
 | launches kernels
 |
GPU
 ├── Thousands of parallel cores
 ├── Shared memory
 ├── Global memory
 └── Scheduling logic

The key idea:

Many threads run simultaneously
Each thread performs small calculations
Hardware schedules work dynamically

This approach works extremely well for deep learning because matrix multiplication can be broken into many independent tasks.

The result was almost accidental:

The hardware built for gaming turned out to be excellent for neural networks.

Google's Observation: GPUs Are Doing Too Much

Around 2013–2015, Google's infrastructure was serving billions of machine learning predictions every day.

Engineers noticed something important.

Many GPU features were rarely used during inference:

Complex scheduling
Branch prediction
General-purpose execution logic
Graphics-related circuitry

These features are valuable for a broad range of workloads.

But neural networks are highly predictable.

Most of the work boils down to:

Multiply
Add
Multiply
Add
Multiply
Add

Over and over.

Google asked a radical question:

What if we remove everything that isn't useful for matrix multiplication?

The answer became the TPU.

The Heart of a TPU: The Systolic Array

The most important component inside a TPU is the systolic array.

A systolic array is a grid of processing elements that pass data rhythmically through the chip.

Imagine a matrix multiplication:

A × B = C

Instead of sending data back and forth to memory repeatedly, the TPU streams values through a grid.

A simplified example:

A →
[PE][PE][PE]
[PE][PE][PE]
[PE][PE][PE]
      ↓
      B

Each Processing Element (PE):

Receives values from neighbors
Multiplies them
Accumulates partial results
Passes data onward

The data "flows" through the chip like blood through arteries.

That's where the name systolic comes from.

This architecture dramatically reduces memory movement, which is often the true bottleneck in modern computing.

Moving data frequently costs more energy and time than performing arithmetic.

TPUs are designed around minimizing that movement.

Why Memory Bandwidth Matters More Than Compute

Many developers assume AI workloads are limited by compute.

In reality, large models are often limited by memory.

Consider two scenarios.

Scenario 1

The processor performs:

2 × 3

This operation is extremely cheap.

Scenario 2

The processor fetches:

2
3

from distant memory before performing the multiplication.

The memory access can cost far more than the arithmetic.

As models scale, this becomes increasingly important.

TPUs address this problem using:

Large on-chip buffers
High-bandwidth memory
Data reuse strategies
Systolic execution

The goal is simple:

Move data as little as possible.

This is one reason TPUs achieve impressive performance-per-watt.

TPU Training Pods: Scaling Beyond a Single Chip

One TPU is powerful.

A TPU Pod is where things become interesting.

Google connects thousands of TPUs using specialized high-speed interconnects.

Conceptually:

TPU  TPU  TPU  TPU
 |    |    |    |
TPU  TPU  TPU  TPU
 |    |    |    |
TPU  TPU  TPU  TPU

These chips behave almost like one giant distributed accelerator.

Large language models frequently require:

Model parallelism
Data parallelism
Pipeline parallelism

TPU Pods were designed with these workloads in mind.

This is one reason many frontier-scale models have historically been trained on TPU infrastructure.

The networking architecture becomes nearly as important as the chips themselves.

TPU vs GPU: Which Is Better?

The answer depends on the workload.

GPUs excel when:

Running diverse workloads
Supporting many frameworks
Performing graphics and AI together
Requiring maximum ecosystem support

Advantages:

Mature tooling
Massive developer community
Broad software compatibility
Flexible execution

TPUs excel when:

Running large-scale neural networks
Using TensorFlow or JAX ecosystems
Maximizing throughput
Optimizing energy efficiency

Advantages:

Specialized matrix hardware
Excellent scaling characteristics
High utilization for AI workloads
Lower overhead for tensor operations

The tradeoff is flexibility.

A GPU is a powerful general-purpose parallel computer.

A TPU is a highly specialized neural network machine.

Think of it like:

GPU = Swiss Army knife
TPU = Industrial assembly line

The assembly line wins if your workload matches its design.

Why This Matters for Modern AI Engineers

As models continue growing, hardware architecture is becoming a first-class concern.

Ten years ago, most developers could treat hardware as a black box.

Today:

Training costs millions of dollars
Model efficiency directly impacts profitability
Inference latency affects user experience
Hardware choices influence architecture decisions

Understanding TPUs isn't just about learning another chip.

It's about understanding a broader trend:

The era of general-purpose computing is giving way to increasingly specialized hardware.

TPUs are one example.

AI accelerators from NVIDIA, AMD, Amazon, Microsoft, Cerebras, Groq, and many others are pushing the same idea further.

The future of AI may not belong to the fastest processor.

It may belong to the processor whose architecture most closely matches the mathematics of machine learning.

Conclusion

GPUs helped ignite the deep learning revolution because they offered massive parallelism at scale. TPUs took the next step by asking a narrower question: if neural networks mostly perform matrix multiplication, why not build hardware specifically for that task?

The result was a radically different architecture centered around systolic arrays, data movement efficiency, and large-scale distributed training.

As AI systems continue growing, understanding these architectural choices becomes increasingly valuable—not just for hardware engineers, but for every developer building machine learning systems.

If you were training a large model today, would you prioritize the flexibility of GPUs or the specialization of TPUs—and why?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Multi-Step Learning Rate Schedulers in LLM Training: Why Some Teams Are Moving Beyond Cosine Decay

Shrijith Venkatramana — Sat, 20 Jun 2026 11:42:42 +0000

Training modern Large Language Models is expensive.

When a single training run can consume millions of GPU hours, even small optimization decisions become important. Most developers focus on model architecture, dataset quality, and scaling laws. Yet one of the most influential knobs in training is surprisingly simple:

How should the learning rate change over time?

For years, cosine decay has been the default answer. But many recent LLM projects have quietly adopted an alternative: the multi-step learning rate scheduler.

What's interesting is that the reason isn't necessarily better final accuracy.

It's because multi-step schedules make something else much easier: continuing training later without wasting previous compute.

Let's explore why.

The Learning Rate Is the Model's Step Size

A neural network learns by repeatedly adjusting its parameters.

The learning rate controls the size of those adjustments.

Imagine hiking toward a destination in dense fog:

Too large a step → you overshoot repeatedly.
Too small a step → progress becomes painfully slow.
A well-chosen step size gets you there efficiently.

During LLM training, we rarely keep the learning rate constant.

Instead, we start with a relatively large learning rate to make rapid progress and gradually reduce it as training converges.

The scheduler determines how that reduction happens.

The Popular Choice: Cosine Decay

The most common scheduler in modern LLM training is cosine decay.

The learning rate follows a smooth curve:

At the beginning, the learning rate is high.

As training progresses, it smoothly decreases until reaching a very small value near the end.

Visually, it looks like a gently descending hill.

Why is cosine decay popular?

Simple
Stable
Works across many model sizes
Requires little tuning

For years, it became the default choice for transformer training.

However, it has an important limitation.

The Problem with Cosine Decay in Continual Training

Suppose your original plan was:

Train for 1 trillion tokens
Stop
Evaluate results

A month later you decide:

Let's continue training for another trillion tokens.

With cosine decay, things become awkward.

The scheduler assumed training would end at a specific point.

By the time you reach that endpoint, the learning rate has already decayed close to zero.

Extending training now requires redesigning the schedule.

You must decide:

Restart the scheduler?
Stretch the curve?
Create a new decay function?

Each choice changes optimization behavior.

This becomes increasingly inconvenient for organizations that frequently expand training runs as new compute becomes available.

Enter Multi-Step Learning Rate Scheduling

A multi-step scheduler divides training into distinct phases.

Instead of continuously decreasing the learning rate, it stays constant for long periods and then drops abruptly.

For example:

Training Progress	Learning Rate
0% - 80%	1.0×
80% - 90%	0.1×
90% - 100%	0.01×

Rather than a smooth curve, the graph resembles a staircase.

The learning rate remains fixed during a stage and changes only at predefined milestones.

Conceptually:

Stage 1: High LR
───────────────

Stage 2: Lower LR
        ─────────

Stage 3: Very Low LR
                 ───

Many recent LLM efforts use schedules resembling an 80% / 10% / 10% distribution.

Most computation happens in the first phase, while later phases act as refinement stages.

Why Multi-Step Schedulers Work Surprisingly Well

At first glance, abrupt drops seem less elegant than smooth cosine decay.

Yet empirical results often show something surprising:

Final model quality is frequently very similar.

Why?

Because optimization is dominated by the large early stage.

Most useful learning occurs when:

The learning rate is relatively high
The model is far from convergence
Large parameter updates are still beneficial

The later stages mainly fine-tune the model.

Whether the transition between stages is smooth or abrupt often matters less than developers expect.

In practice, many teams observe nearly identical downstream performance between:

Cosine schedules
Carefully tuned multi-step schedules

This makes the operational advantages of multi-step scheduling very attractive.

The Hidden Advantage: Reusing Training Compute

This is where multi-step scheduling becomes especially interesting for large-scale training.

Imagine training proceeds like this:

Phase	Tokens Trained
Stage 1	800B
Stage 2	100B
Stage 3	100B

Now suppose additional funding or GPU capacity appears.

Instead of redesigning the entire learning-rate trajectory, you can simply:

Extend Stage 1
Continue training at the same learning rate
Delay later stage transitions

The optimization process remains consistent.

The expensive computation already completed during Stage 1 remains fully reusable.

This flexibility becomes valuable when:

Scaling budgets change
New datasets arrive
Training targets evolve
Additional compute becomes available unexpectedly

For frontier model teams, this operational convenience can outweigh theoretical elegance.

Implementing a Multi-Step Scheduler

A simplified PyTorch example looks like this:

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=1e-4
)

scheduler = torch.optim.lr_scheduler.MultiStepLR(
    optimizer,
    milestones=[800000, 900000],
    gamma=0.1
)

This configuration means:

Learning rate = 1e-4 initially
At step 800,000 → drop by 10×
At step 900,000 → drop by another 10×

Result:

Phase	LR
0 - 800k	1e-4
800k - 900k	1e-5
900k+	1e-6

Real LLM training systems usually include:

Warmup stages
More sophisticated milestone selection
Token-based rather than step-based scheduling
Distributed optimizer considerations

But the core idea remains the same.

When Should You Use Multi-Step Scheduling?

For smaller projects, cosine decay remains an excellent default.

However, multi-step scheduling becomes compelling when:

Training runs may be extended later
Compute availability is uncertain
Continual pretraining is expected
Multiple training phases are planned
Reusing partially completed training is important

In these environments, optimization quality may remain nearly unchanged while operational flexibility improves significantly.

Sometimes the best engineering decision isn't the theoretically cleanest one.

It's the one that makes future decisions easier.

Conclusion

Learning-rate schedulers are often treated as a minor implementation detail.

Yet at LLM scale, they influence not only optimization but also the economics of training.

Cosine decay offers smooth and reliable convergence. Multi-step schedules often achieve similar final performance while making continual training far easier to manage.

That tradeoff explains why several modern LLM training efforts have adopted multi-step schedulers as their default strategy.

As model training increasingly becomes a long-running, iterative process rather than a single fixed experiment, flexibility may matter just as much as raw optimization performance.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

The Three Phases of Post-Training: How LLMs Learn to Provide Sensible Responses

Shrijith Venkatramana — Fri, 19 Jun 2026 19:02:11 +0000

Most developers have heard the phrase:

"LLMs are trained on massive amounts of internet data."

While technically true, it leaves out the most interesting part.

Pretraining teaches a model how language works. But it doesn't teach the model how to be helpful, harmless, honest, or aligned with human expectations.

If pretraining creates a brilliant but socially awkward intern, post-training turns that intern into a productive teammate.

Modern AI systems such as ChatGPT, Gemini, Claude, and others rely heavily on a multi-stage post-training pipeline. While implementations differ, the overall pattern is surprisingly consistent:

Supervised Fine-Tuning (SFT)
Reward Modeling (RM)
Reinforcement Learning (RL)

Let's explore what each stage does, why it exists, and how they work together.

Why Pretraining Isn't Enough

Imagine we train a model on the entire internet and ask:

"How do I become a better software engineer?"

The model has seen thousands of answers:

Good advice
Bad advice
Contradictory advice
Sarcasm
Reddit arguments
Technical blog posts
Motivational speeches

The model learns patterns in text, but it doesn't inherently know which response humans would prefer.

It only knows what tends to come next.

This is the core limitation of pretraining.

The model learns:

"What people write."

But not:

"What humans want."

Post-training bridges this gap.

Stage 1: Supervised Fine-Tuning (SFT)

The first step is teaching the model what good behavior looks like.

Researchers create high-quality examples consisting of:

User: Explain TCP vs UDP.

Assistant:
TCP provides reliable ordered delivery...

Or:

User: Write a Python function that reverses a linked list.

Assistant:
def reverse(head):
    ...

Thousands or millions of these examples are collected.

The model is then trained to imitate the desired responses.

Conceptually:

Question → Ideal Answer

becomes

Model → Learn to reproduce ideal answer

The objective is still next-token prediction, but now on carefully curated data instead of arbitrary internet text.

Software Engineering Analogy

Think of SFT as onboarding a new engineer.

Instead of letting them learn exclusively from random GitHub repositories, you provide:

Coding standards
Architecture guidelines
Example pull requests
Internal best practices

The engineer begins to imitate the patterns you want.

What SFT Solves

SFT dramatically improves:

Instruction following
Formatting
Tool usage
Coding style
Conversational behavior

However, it still has a limitation.

For many prompts, there isn't one correct answer.

There may be multiple reasonable responses with varying quality levels.

That's where Reward Modeling enters.

Stage 2: Reward Modeling (RM)

Suppose a user asks:

"How should I learn distributed systems?"

Three responses might all be technically correct.

Response A:

Read a textbook.

Response B:

Read a textbook and build projects.

Response C:

Study networking, databases, consensus algorithms,
then implement a small Raft cluster.

Most humans would likely prefer C.

But how does a model learn that preference?

The answer is Reward Modeling.

Collecting Human Preferences

Human evaluators compare multiple outputs:

Prompt

Answer A
Answer B

They choose the better response.

Thousands or millions of comparisons are collected.

Example:

Prompt:
How do I learn Go?

Preferred:
Build projects and read effective Go.

Rejected:
Just read documentation.

A separate model is trained to predict these preferences.

This becomes the Reward Model.

Conceptually:

Response → Quality Score

The reward model acts like an automated judge.

Why This Matters

SFT teaches:

"Produce answers similar to examples."

Reward Modeling teaches:

"Recognize which answers humans prefer."

This distinction is subtle but important.

One is imitation.

The other is evaluation.

Stage 3: Reinforcement Learning (RL)

Now we have:

A policy model (the assistant)
A reward model (the judge)

The final stage uses Reinforcement Learning to optimize the assistant.

The process looks like:

Prompt
   ↓
Model generates answer
   ↓
Reward model scores answer
   ↓
Update model to increase reward

Repeated millions of times.

Over time, the assistant learns to generate responses that maximize the reward signal.

PPO and Modern Variants

Historically, many systems used:

PPO (Proximal Policy Optimization)

More recently, newer approaches such as:

DPO (Direct Preference Optimization)
RLAIF (Reinforcement Learning from AI Feedback)
GRPO and related techniques

have gained popularity.

The exact algorithm matters less than the goal:

Move the model toward outputs that humans consistently prefer.

Software Engineering Analogy

Imagine code review automation.

SFT teaches an engineer using examples of good pull requests.

Reward Modeling creates a senior reviewer that scores submissions.

RL repeatedly updates the engineer based on reviewer feedback.

Eventually the engineer starts producing code that receives better review scores.

The New Trend: Models Training Models

One interesting detail from recent Gemini research is the growing use of models themselves in post-training workflows.

Instead of relying exclusively on humans, powerful models help:

Generate candidate responses
Identify low-quality data
Detect inconsistencies
Perform ranking tasks
Assist evaluation pipelines

This creates a feedback loop:

Model
  ↓
Generates data
  ↓
Humans verify
  ↓
Improved model
  ↓
Generates better data

The result is dramatically improved scalability.

The future of post-training may involve humans increasingly acting as supervisors while models handle much of the operational workload.

Why Data Quality Beats Model Size

A common assumption is that better AI comes primarily from larger models.

The industry increasingly suggests otherwise.

Many recent gains come not from:

More parameters

but from:

Better post-training data

A smaller model trained on excellent preference data can often outperform a larger model trained on mediocre data.

This explains why modern research papers frequently emphasize:

Preference datasets
Evaluation quality
Synthetic data generation
Data filtering
Alignment pipelines

The quality of feedback often matters more than the quantity of compute.

Pretraining is about language, SFT is about sensible responses

Pretraining teaches a model how language works.

Supervised Fine-Tuning teaches it how to respond.

Reward Modeling teaches it what humans prefer.

Reinforcement Learning teaches it to consistently optimize for those preferences.

Together, these stages transform a statistical text predictor into something that feels surprisingly useful.

As foundation models become increasingly capable, the competitive advantage may shift away from raw model size and toward the sophistication of post-training systems and data quality pipelines.

The next major breakthrough in AI might not come from a bigger model.

It might come from a better teacher.

If you were designing a reward model for a coding assistant, what signals would you optimize for—correctness, readability, maintainability, performance, or something else entirely?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub