DEV Community: Shrijith Venkatramana

A Practical Tour of SVG Tools Every Developer Should Know

Shrijith Venkatramana — Fri, 17 Jul 2026 19:58:13 +0000

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

If you're building web applications in 2026, you're probably using SVGs whether you realize it or not.

Every icon in your navigation, every logo, every loading spinner, every architecture diagram exported from Figma—there's a good chance it's an SVG.

Most developers treat SVGs like images. They download them, drop them into public/, and move on.

That's a missed opportunity.

SVGs are code. They can be optimized, themed, animated, generated, transformed into components, and even manipulated programmatically.

Let's look at the SVG ecosystem from the perspective of a developer.

1. SVG is Code, Not an Image

Unlike PNG or JPEG, an SVG is simply XML describing vector shapes.

A tiny circle looks like this:

<svg width="100" height="100">
  <circle
    cx="50"
    cy="50"
    r="40"
    fill="#3b82f6"
  />
</svg>

Because it's text:

it compresses well
it scales perfectly
Git can diff it
CSS can style it
JavaScript can manipulate it
build tools can optimize it

Once you start thinking of SVG as source code rather than an asset, an entirely new tooling ecosystem opens up.

2. First Step: Optimize Everything

Design tools (Figma, Illustrator, Sketch, Inkscape) produce SVGs for editing—not necessarily for production.

A simple icon might contain:

editor metadata
hidden layers
unnecessary groups
comments
redundant transforms
verbose path definitions

That's exactly why SVGO exists.

SVGO removes unnecessary data while preserving the rendered output, often reducing SVG size significantly. It works as both a CLI and a Node.js library, and integrates with popular frontend build pipelines.

Example:

npx svgo logo.svg

If you don't want to install anything, SVGOMG provides a browser interface built on top of SVGO where you can experiment with optimization options visually.

3. Converting SVGs into Framework Components

Most React developers eventually write something like:

<img src="/logo.svg" />

But that's often leaving flexibility on the table.

Tools like SVGR convert SVGs into React components.

Instead of:

<img src="/logo.svg" />

you get:

<Logo className="w-6 h-6 text-blue-500" />

Now your SVG becomes:

themeable
type-safe
tree-shakable
easy to compose
configurable via props

Many modern build systems already integrate SVGR alongside SVGO.

4. Picking Better Icon Libraries

One of the biggest mistakes teams make is mixing icon packs.

Different stroke widths.

Different corner radii.

Different visual language.

The UI quietly starts feeling inconsistent.

A few excellent SVG icon libraries include:

Lucide — lightweight, consistent, actively maintained, and tree-shakable. Great default choice for developer tools and SaaS dashboards.
Heroicons — designed alongside Tailwind CSS, available in outline and solid variants.
Tabler Icons
Phosphor Icons
Bootstrap Icons

The important thing isn't choosing the "best" library.

It's choosing one and staying consistent.

5. Working with Large SVG Collections

Eventually you'll need:

company logos
cloud provider icons
technology logos
UI illustrations
architecture diagrams

Searching Google Images usually ends badly.

A better approach is using curated SVG repositories.

Some useful resources include:

SVG Repo
Simple Icons
Lucide
Heroicons

SVG Repo also provides browser-based editing tools for recoloring, resizing, and modifying vectors before downloading.

6. SVG Can Be Generated Programmatically

Because SVG is text, generating graphics becomes surprisingly straightforward.

For example:

function circle(color) {
  return `
  <svg viewBox="0 0 100 100">
    <circle
      cx="50"
      cy="50"
      r="45"
      fill="${color}"
    />
  </svg>`;
}

Libraries like D3 have used this idea for years.

Today we're also seeing LLMs generate SVG directly from prompts, enabling workflows such as icon generation, diagram editing, and vector illustration. Recent research benchmarks focus specifically on SVG generation and editing because it combines programming with visual reasoning.

Unlike raster images, generated SVGs remain editable after they're created.

7. A Simple Production Workflow

A workflow that scales well for many frontend projects looks like this:

Designer
      ↓
Export SVG
      ↓
SVGO optimization
      ↓
SVGR conversion (optional)
      ↓
Commit to repository
      ↓
Import as React/Vue/Svelte component
      ↓
Theme with CSS variables

It keeps assets:

small
consistent
reviewable
version-controlled
easy to maintain

Once this pipeline is in place, SVGs stop being static assets and become part of your application's source code.

Final Thoughts

SVG is one of those technologies that quietly powers almost every modern frontend application.

Yet many developers only scratch the surface.

Learning a handful of tools—SVGO for optimization, SVGR for component generation, Lucide or Heroicons for icons, and repositories like SVG Repo—can noticeably improve performance, maintainability, and developer experience without adding much complexity.

What's your go-to SVG workflow?

Do you inline SVGs, use React components, rely on icon libraries, or generate them programmatically? I'd love to hear what has worked well for your projects.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Stop Writing Layout Code Twice: Using React Yoga from Go

Shrijith Venkatramana — Mon, 13 Jul 2026 18:42:15 +0000

Every UI framework eventually runs into the same problem: how do you arrange things on the screen?

Buttons need to line up. Sidebars need to stretch. Cards should wrap. Mobile layouts should adapt. Suddenly you're writing layout algorithms instead of building your application.

The web largely solved this with Flexbox. But what if you're building a desktop app, a game UI, a PDF generator, a terminal application, or even an image renderer in Go?

Do you reinvent layout from scratch?

Thankfully, no.

Yoga, the layout engine originally developed at Meta, lets you reuse the same Flexbox model almost anywhere—including Go.

Let's see how.

What Exactly Is Yoga?

Yoga is not a UI toolkit.

It doesn't draw buttons.
It doesn't render text.
It doesn't know anything about HTML.

Instead, Yoga answers one question:

Given a tree of elements and a set of Flexbox rules, where should every element be positioned and how large should it be?

That's it.

Think of it as a geometry engine.

Your widgets
      │
      ▼
    Yoga
      │
      ▼
Calculated positions and sizes
      │
      ▼
Your renderer (OpenGL, terminal, PDF, etc.)

This separation makes Yoga incredibly portable. The same engine powers layouts across many different rendering systems.

Why Would a Go Developer Care?

If you're writing Go, chances are you're building something outside the browser.

Examples include:

desktop applications
terminal user interfaces
PDF generators
dashboards
game interfaces
diagram generators
SVG renderers
custom visualization tools

All of these need layout.

Without Yoga, developers often end up writing code like:

button.X = sidebarWidth + 20
button.Y = headerHeight + 15
button.Width = windowWidth - 40

This quickly becomes fragile.

Instead, you describe relationships.

Container
├── Sidebar (fixed width)
└── Content (grow to fill)

Yoga computes the actual numbers for you.

Flexbox Without a Browser

If you've ever written CSS like this:

.container {
    display: flex;
    flex-direction: row;
}

.sidebar {
    width: 240px;
}

.content {
    flex-grow: 1;
}

You've already learned most of Yoga.

The same concepts exist:

Flex Direction
Justify Content
Align Items
Flex Grow
Flex Shrink
Gap
Padding
Margin

Instead of manipulating DOM elements, you're configuring layout nodes.

A tree might look like this:

Root
├── Header
├── Body
│   ├── Sidebar
│   └── Main Content
└── Footer

Yoga walks this tree and calculates every rectangle.

Building Your First Layout in Go

Suppose we want a horizontal layout with two panels.

left panel: fixed width
right panel: fills remaining space

The structure looks like this:

Root (800px)
├── Left (200px)
└── Right (grow)

Using a Go binding, the code is surprisingly small:

root := yoga.NewNode()
root.StyleSetWidth(800)
root.StyleSetHeight(600)
root.StyleSetFlexDirection(yoga.FlexDirectionRow)

left := yoga.NewNode()
left.StyleSetWidth(200)

right := yoga.NewNode()
right.StyleSetFlexGrow(1)

root.InsertChild(left, 0)
root.InsertChild(right, 1)

root.CalculateLayout(
    yoga.Undefined,
    yoga.Undefined,
    yoga.DirectionLTR,
)

After layout is calculated:

Left:
    x = 0
    width = 200

Right:
    x = 200
    width = 600

Notice there's no coordinate math.

You describe intent.

Yoga computes geometry.

Measuring Dynamic Content

Real interfaces aren't just boxes.

Text changes.

Images resize.

Widgets have intrinsic sizes.

Yoga handles this using measure functions.

Instead of giving a node a fixed width and height, you provide a callback that tells Yoga how much space the content actually needs.

Conceptually:

textNode.SetMeasureFunc(func(...) Size {
    return MeasureText(...)
})

During layout, Yoga asks:

"If I give you this much width, how tall will you become?"

This is exactly how browsers lay out paragraphs.

The important point is that Yoga doesn't measure text itself.

Your application remains responsible for fonts, images, and rendering.

Yoga only performs layout.

How Yoga Fits Into a Rendering Pipeline

A common architecture looks like this:

Application State
        │
        ▼
Widget Tree
        │
        ▼
Yoga Node Tree
        │
 CalculateLayout()
        │
        ▼
Computed Rectangles
        │
        ▼
Renderer
(OpenGL / Gio / Ebiten / PDF / SVG / Terminal)

This separation has several advantages:

rendering and layout stay independent
swapping renderers becomes easier
layout becomes deterministic
widgets don't need to know screen coordinates
Flexbox knowledge transfers directly from web development

Many modern UI systems follow this architecture because it scales well as applications grow.

Final Thoughts

One of Yoga's biggest strengths is that it brings an already-familiar mental model outside the browser.

Instead of inventing another layout system for every project, you get a mature Flexbox implementation that has been battle-tested across countless interfaces.

For Go developers building desktop applications, renderers, editors, dashboards, PDFs, games, or custom graphics pipelines, Yoga removes an entire class of layout problems. You can focus on describing what the interface should look like rather than calculating every coordinate yourself.

As UI complexity grows, that distinction becomes increasingly valuable.

Have you used Yoga—or another layout engine—in a Go project? I'd be interested to hear what you're building and whether you prefer declarative layout systems over manual coordinate calculations.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark

Shrijith Venkatramana — Sun, 12 Jul 2026 15:42:34 +0000

Benchmarks are interesting. They instantaneously give an air of something authoritative, objective, and precise, and for good reason.

For most people, benchmarks look like something done by knowledgeable people on the topic—by those "who know better than me."

However, until we go into the details of their exact workings, especially for those who are true students of a particular subject, relying on them blindly is a bad personal policy.

If you actually know about a particular subject and care about it, I recommend you go deep into the reasoning of why particular metrics are worth your time and consideration, and try to make sense of things from first principles.

In this case, I will go through the detailed Code Review Bench methodology and explain why I think tool builders in this category must look beyond the confines of the benchmark and think in a totally different way about the whole problem of AI code review.

First and foremost, the AI Code Review Bench starts with seemingly good qualities:

Looks objective
Looks precise
Looks authoritative
Looks detailed

But for anyone who has gone into the depths of the problem and wrestled with the nitty-gritty of larger parts of the problem space, many shortcomings become apparent.

Side Note: My Background

I'm Shrijith Venkatramana, founder of Hexmos, presently building LiveReview, an org-wide harness for enforcing your engineering standards. My team and I have been building software systems together for the past 5 years in various capacities. I worked at Amazon before that, and studied Software Engineering under Prof. Crista Lopes at the University of California, Irvine. I've built engineering teams and systems for almost a decade.

As someone with the background mentioned above, and since we do a lot of AI code reviews, issue detection, and so on, there were frequent requests from people asking, "How do you score on benchmark X or Y?" I'd casually answer that I am not entirely convinced by most benchmarks around, but felt like I ought to provide a deeper and detailed justification for my position. Hence, this post.

In particular, the benchmark perfectly hits the wrong targets for what AI code review benchmarks should do.

However, I don't mean to discourage or criticize the authors personally or anything of the sort: I believe they have put great effort into setting this system up, and I respect their efforts. I do believe they have made a truly honest effort in finding various potential issues with their work and tried to get as good a result as possible.

Despite all that, I'd like to put my views on this benchmark, and benchmarks in general, on the record.

First, the methodology makes an ambitious attempt to answer an important question:

How should we benchmark AI code review?

Unfortunately, I don't think the paper answers that question.

In fact, I came away thinking it demonstrates why we're not yet ready for a benchmark.

I argue that the methodology is excellent and detailed, but jumps to a solution before the problem of AI code review itself is defined from first principles.

The paper openly acknowledges many major challenges in great detail:

Goodhart's Law
Incomplete gold sets
Benchmark contamination
LLM judges
Context
Severity
OSS vs. enterprise repositories
Evolving datasets
Autonomous agents

These are all highly valuable components in understanding what the "AI code review problem" is about.

However, I still consider them lower priority issues. Some bigger, more important things have been totally sidestepped.

To be specific, I believe that it misses the mark at the fundamental problem analysis stage.

1. The Core Dictum: Work Backwards from the Customer; To Win, Start From the End

In the business world, Jeff Bezos articulated the "working backwards" process to build up Amazon. Although he was the first to institutionalize it, we find Steve Jobs working backwards from the customer experience before him.

But then, why stop there? We can go back to George Polya, and in his How to Solve It, he specifies that step one is defining the goal, specifically the unknown, the x factor which needs to be unearthed. The problem solver must work backwards from the x to what they have. So, analysis is another word for the working backwards process. Once we have a good analysis, we can use what we have and turn it into what we want. The process of construction or moving forward based on the analysis is called synthesis.

By the way, it was not Polya who came up with the idea of analysis and synthesis, but Pappus of Alexandria, an ancient mathematician of great mathematical insight and capability. Pappus was a big influence on Newton, who studied the analysis and synthesis methodology to get good at his scientific work.

So, we go through all these references to Bezos, Jobs, Polya, and Pappus to mention the importance of defining the problem well. Interestingly, this methodology completely misses an attempt to characterize the problem of code review in explicit terms. That is, there is no organized effort to systematize:

Who are the customers of (AI) code review?
What is the problem of each customer class?
What is the most important benefit of doing code reviews?
What data is available to know customers care about code reviews?
What does the customer experience look like?

One can ask more questions of this sort to actually nail down the problem space to begin with. It looks like such a systematic inquiry was not performed by the benchmark authors. It just jumps into the solution space, which largely robs us of an opportunity to learn how the authors think about the problem in question.

Despite them not clearly defining x, or the unknown, we still get a sense of how they look at the problem by piecing together the various statements made across the document.

The first problem is that the benchmark largely treats AI code review as one problem.

I think it has already become two.

2. The Two Distinct Problems of AI Code Review

Problem #1: Human Comprehension

Humans have limited attention.

The problem here is helping engineers understand what matters.

Showing every possible issue is rarely optimal. Instead, information should be prioritized differently depending on:

The engineer
The team
The organization
Business priorities
Operational context
Stage of development

The problem is fundamentally one of recommendation rather than search.

Success is measured by helping humans make better engineering decisions.

Problem #2: Machine Verification and Prod Stabilization

The second problem has an entirely different consumer: another AI system.

An agent does not become overwhelmed by hundreds or thousands of findings. In fact, exhaustive analysis is often desirable.

The objective shifts toward reducing:

Production outages
Security vulnerabilities
Correctness issues
Technical debt

Many findings never need to be shown to humans at all. They're simply passed to repair agents.

The agent can handle volume; it doesn't get tired or overwhelmed.

Volume and detail translate to higher capability for the repair agents.

The paper acknowledges that autonomous agents will eventually require different evaluation methods.

Where I disagree is that this is presented as a future transition. I think that future is already here.

Many developers are no longer reading every review comment themselves. They're handing comments directly to coding agents, reviewing the resulting changes, and supervising the overall process.

The workflow has already changed. That fundamentally changes what should be measured.

These are different optimization problems for different consumers

One process optimizes for scarce human attention.

The other optimizes for agent characteristics in the service of software quality.

Once this distinction is made, many familiar benchmark concepts become much less obvious.

Questions such as precision, recall, comment volume, and "noise" primarily belong to the human comprehension problem.

Questions such as exhaustive verification, defensive hardening, and automated repair belong to the machine verification problem.

The paper evaluates these under a single framework. I think separating them first would lead us toward very different benchmark designs.

Metrics like comment volume, "noise", and precision have a very different meaning when another AI system—not a human—is consuming the output.

3. Human Review Is a Baseline, Not the Destination

To be fair, the document explicitly recognizes that human reviewers are imperfect and discusses multiple ways of eventually moving beyond human-generated gold sets. That is one of its strengths.

However, the overall framing still begins with evaluating review performance against real-world human performance.

I am not convinced that this is the right destination.

Most code reviews happen under deadline pressure, reviewer fatigue, and incomplete context.

Machines have obvious advantages:

They never tire.
They never lose concentration.
They inspect everything consistently.
They can apply thousands of checks without complaint.

Humans contribute something different rather than simply being "better."

Experienced engineers recognize architectural problems, product trade-offs, operational concerns, and business realities that exist outside the code itself.

Machines and humans, therefore, have complementary strengths rather than existing on a single performance scale.

4. The Benchmark Measures Proxies, Not Software Outcomes

The document rightly warns about Goodhart's Law.

Ironically, I think it also demonstrates how difficult Goodhart's Law really is.

The benchmark measures increasingly sophisticated proxies:

Review comments
Developer actions
Human gold sets
Production bug traces
Online behavioral signals

Each of these is individually reasonable. None of them is actually the objective.

The objective is producing better software.

The further we optimize increasingly elaborate proxies, the easier it becomes to mistake them for the goal itself.

Agreement with historical reviewers is not necessarily the same thing as reducing operational failures.

5. The Paper Acknowledges Many Open Questions—But Still Produces a Benchmark

One thing I genuinely appreciated about the paper is how candid it is.

It openly discusses uncertainty around:

Bug definitions
Severity
Enterprise versus OSS repositories
Incomplete context
Benchmark contamination
LLM judge quality
Developer behavior
Future autonomous agents
Evolving datasets

What surprised me was that after acknowledging so many unresolved assumptions, the paper still culminates in a benchmark that inevitably appears objective.

My concern is not that the benchmark is wrong.

My concern is that the apparent precision of the benchmark exceeds our current understanding of the problem.

6. Several Methodological Choices Become Questionable

Production outcomes feel under-emphasized

Production bugs do appear in the methodology, but mostly as one signal among many.

Personally, I would place operational outcomes much closer to the center.

Many code review activities ultimately exist to reduce production failures, improve reliability, and avoid embarrassing incidents.

Those outcomes deserve greater prominence than they currently receive.

Severity matters more than distribution

The paper spends considerable effort matching the observed distribution of historical bugs.

I suspect severity is considerably more important.

Different organizations legitimately tolerate different classes of issues.

A regulated financial institution and an early-stage startup do not optimize for the same level of operational risk.

Without organizational context, a universal benchmark inevitably averages together very different notions of software quality.

Code review is about more than code

Many valuable review comments concern things such as:

Architecture
Deployment strategy
Operational cost
Organizational coordination
Business priorities
Long-term maintainability

These are difficult to benchmark because they extend beyond the code itself.

Yet they are often exactly what distinguishes experienced reviewers.

7. The Online Benchmark Should Be Interpreted Carefully

I also remain somewhat skeptical about using developer actions as evidence of comment quality.

Increasingly, developers delegate review comments directly to coding agents.

A comment being "acted upon" may therefore measure workflow automation as much as genuine human agreement.

The paper already acknowledges several limitations of this proxy.

My concern is simply that these behavioral signals may become progressively noisier as AI-assisted development becomes commonplace.

8. I Think the Future Looks Different

If the distinction between human comprehension and machine verification is correct, I suspect future code review systems will evolve very differently.

For humans, the problem becomes one of interface design.

The important questions become:

What should this person see?
In what order should issues appear?
What information should be hidden?
How should recommendations adapt to the team and organization?

This looks much closer to a recommendation system than a traditional static analyzer.

For AI agents, the optimization is almost the opposite:

Find everything. Generate exhaustive evidence. Prioritize defensive hardening. Feed repair agents. Optimize for robustness rather than attention.

Trying to evaluate both of these worlds using a single benchmark seems increasingly difficult.

9. What the Paper Gets Right

Despite my criticisms, I think this paper advances the discussion significantly.

Among the ideas I found particularly valuable are:

Recognizing that human gold sets cap evaluation
Adversarial expansion of benchmark data
Continuous benchmark evolution
Combining online and offline evaluation
Methodological transparency
Openly documenting assumptions instead of hiding them

These are meaningful contributions. Even where I disagree with the conclusions, I found the discussion itself valuable.

10. Conclusion

I came away thinking that the paper asks many of the right questions, but begins benchmarking before the underlying objective has fully stabilized.

My disagreement is therefore less about the methodology than about the framing.

I think AI code review is currently splitting into two different disciplines.

One is about helping humans understand software.

The other is about helping machines systematically harden software.

Those are different problems, with different consumers, different interfaces, different optimization criteria, and ultimately different definitions of success.

Until we separate them, I suspect any benchmark will necessarily measure a blended objective rather than a clearly defined one.

For that reason, I see this paper not as the definitive benchmark for AI code review, but as an excellent exploration of why benchmarking AI code review remains an open research problem.

11. Summary

The Code Review Bench is one of the more thoughtful AI code review benchmark proposals I've seen. It openly discusses many methodological challenges that existing benchmarks ignore.
My disagreement is not with the engineering of the benchmark, but with the problem definition. I think the benchmark is being built before the underlying problem has fully stabilized.
The paper treats AI code review as one problem, whereas I think AI is splitting it into two fundamentally different problems.
The first problem is human comprehension: helping engineers understand what matters through prioritization, personalization, and good information presentation.
The second problem is machine verification: exhaustively finding and fixing issues to reduce outages, security vulnerabilities, and technical debt, often with another AI agent as the consumer.
These two problems have different users and different objectives, and should likely have different evaluation criteria. Mixing them together makes metrics like precision, recall, and "noise" difficult to interpret.
The ultimate objective should be better software—not matching historical human reviewers. Human review is a baseline, not the end destination.
The benchmark measures increasingly sophisticated proxies (comments, developer actions, gold sets, online behavior), but these remain proxies for review quality rather than software quality itself.
The paper honestly acknowledges many unresolved questions—bug definitions, context, severity, judge quality, OSS vs. enterprise, and future agents—but still culminates in a benchmark that appears more objective than the underlying problem currently justifies.
Several methodological choices follow from this framing, including an emphasis on bug distributions over severity, code over broader engineering context, and developer actions as a proxy for usefulness.
I believe future human-facing code review will look increasingly like a recommendation system, while agent-facing code review will optimize for exhaustive verification and automated repair.
Overall, I see this paper as an important contribution—not because it has solved AI code review benchmarking, but because it makes clear how much of the underlying problem remains open.

AI-Assistance Disclosure: I spent in total 3 days to come up with this piece. I read through original materials myself, formulated all the key objections and conclusions myself and stand by the opinions expressed therein. At the same time, I have used AI to polish my writing at certain places, such as fixing punctuation, grammar, awkward phrases and faulty heading levels and so on. These are merely presentation and stylistic changes but nothing fundamental.

Adam: The Optimization Algorithm That Made LLMs Practical

Shrijith Venkatramana — Sat, 04 Jul 2026 18:53:06 +0000

Without Adam, there's a good chance ChatGPT, Claude, Gemini, Llama, and many of today's large language models would have taken much longer to become reality.

When people talk about the breakthroughs behind modern AI, they usually mention the Transformer, attention, GPUs, or massive datasets.

Rarely does anyone mention the optimizer.

Yet every single gradient update during the training of a modern LLM depends on an optimization algorithm deciding how much every parameter should change. With billions of parameters and trillions of training tokens, that decision becomes one of the most important engineering problems in machine learning.

One optimizer, proposed in 2014, ended up becoming the default choice across much of deep learning:

Adam (Adaptive Moment Estimation).

Let's explore why.

Before Adam: Why Training Neural Networks Was So Difficult

Imagine you're hiking down a mountain in dense fog.

You can only see the slope directly beneath your feet.

The obvious strategy is simple:

Take one step downhill.

This is essentially what gradient descent does.

For a neural network, the gradient tells us which direction reduces the loss.

The update rule is simply:

[
\theta = \theta - \eta \nabla L
]

where

θ = model parameters
η = learning rate
∇L = gradient

Simple.

Unfortunately, real neural networks don't resemble smooth mountains.

Instead, they're more like:

steep cliffs
long narrow valleys
flat plateaus
noisy terrain
millions—or billions—of dimensions

The gradient also changes every mini-batch because we only compute it on a small sample of data.

This leads to stochastic gradient descent (SGD).

Instead of walking smoothly downhill, it's like trying to descend while someone randomly shakes the mountain beneath you.

Momentum: Giving Optimization Some Inertia

Researchers realized that humans don't instantly stop and change direction every step.

Instead, we build momentum.

Optimization borrowed the same idea.

Instead of only following today's gradient, momentum also remembers previous gradients.

Imagine pushing a heavy shopping cart.

If you push in roughly the same direction repeatedly, it builds speed.

Small bumps don't immediately change its path.

Mathematically,

[
v_t=\beta v_{t-1}+(1-\beta)g_t
]

where

(g_t) is today's gradient
(v_t) is the accumulated velocity

Now updates become

[
\theta=\theta-\eta v_t
]

This smooths noisy updates and helps escape shallow local irregularities.

Momentum was already a huge improvement.

But another problem remained.

Different Parameters Learn at Different Speeds

Suppose you're training a neural network with 500 million parameters.

Some parameters receive gradients like

0.00002

Others receive gradients like

Using one global learning rate becomes problematic.

If the learning rate is large enough for tiny gradients...

...the large gradients explode.

If it's safe for the large gradients...

...the tiny gradients barely move.

It's like paying every employee in a company exactly the same bonus regardless of performance, seniority, or role.

Some people are overpaid.

Others barely notice the reward.

Optimization needs to adapt individually.

This insight led to algorithms like AdaGrad and RMSProp.

Adam combined the best ideas from both.

Adam: Combining Momentum and Adaptive Learning Rates

In 2014, Diederik P. Kingma and Jimmy Ba introduced Adam in the paper:

Adam: A Method for Stochastic Optimization

The idea is beautifully elegant.

Adam maintains two running statistics for every parameter.

First moment

The average gradient.

Think of this as momentum.

[
m_t
]

Second moment

The average squared gradient.

Think of this as measuring how "volatile" or "uncertain" this parameter's updates have been.

[
v_t
]

The update becomes approximately

[

\theta

\eta
\frac{m_t}
{\sqrt{v_t}+\epsilon}
]

This produces a fascinating behavior.

If a parameter consistently receives huge gradients,

its denominator becomes larger,

making future updates smaller.

If another parameter rarely changes,

its denominator stays small,

allowing relatively larger updates.

Every parameter effectively receives its own personalized learning rate.

A Back-of-the-Envelope Example

Suppose two parameters have identical momentum:

Parameter A

Average gradient = 2
Average squared gradient = 100

Parameter B

Average gradient = 2
Average squared gradient = 4

Ignoring ε,

Parameter A updates by

2 / √100 = 0.2

Parameter B updates by

2 / √4 = 1

Even though both gradients are identical today,

Adam trusts Parameter B much more because its historical variance is lower.

This automatic scaling is one reason Adam trains deep networks so effectively.

Why Bias Correction Exists

There's one subtle issue.

At the beginning of training,

both moving averages start at zero.

That means early estimates are biased toward zero.

Kingma and Ba introduced bias correction:

[

\hat m_t

\frac{m_t}
{1-\beta_1^t}
]

and

[

\hat v_t

\frac{v_t}
{1-\beta_2^t}
]

These corrections rapidly remove the startup bias.

It's a small mathematical trick that has a surprisingly large practical impact during the first optimization steps.

Why Adam Became So Important for Deep Learning

Consider training GPT-style models.

Modern LLMs easily contain

7 billion parameters
70 billion parameters
over a trillion parameters in some research systems

Every optimization step updates every trainable parameter.

Even a modest training run might execute hundreds of thousands of optimization steps.

That means Adam performs on the order of

billions of parameters
×

hundreds of thousands of updates

resulting in quadrillions of parameter update decisions over the course of training.

Without stable optimization,

training would often diverge.

Learning would become painfully slow.

GPU time—costing thousands or even millions of dollars—would be wasted.

Optimization isn't merely a mathematical curiosity.

It's an operations and economics problem.

A 10% improvement in convergence speed on a multi-million-dollar training run can translate into hundreds of thousands of dollars in savings, shorter experimentation cycles, and faster scientific progress. Faster convergence also means researchers can iterate on model architectures more quickly, reducing the opportunity cost of long training jobs.

Adam became popular because it usually works well with relatively little hyperparameter tuning.

Researchers could spend less time adjusting learning rates and more time exploring new model architectures.

That practicality accelerated progress across computer vision, speech recognition, recommendation systems, and eventually large language models.

Adam Isn't Perfect

As influential as Adam has been, researchers have also identified limitations.

Some studies found that vanilla stochastic gradient descent can produce models that generalize better on certain vision tasks.

Others observed convergence issues under specific theoretical settings.

As LLMs grew larger, practitioners developed variants such as:

AdamW (decoupled weight decay)
AdaFactor (reduced memory footprint)
Lion (sign-based optimization)

In fact, many modern Transformer implementations train with AdamW, which separates weight decay from Adam's adaptive updates and often improves regularization.

Engineering rarely ends with one perfect algorithm.

Instead, progress comes through continual refinement.

The Bigger Lesson

When the Transformer paper appeared in 2017, attention deservedly captured the headlines.

But Transformers alone weren't enough.

Modern deep learning stands on layers of innovations:

better hardware
larger datasets
improved initialization
normalization methods
residual connections
efficient optimizers

Adam is one of those foundational technologies.

It's rarely discussed outside machine learning circles, yet it quietly powers the optimization of billions of parameters every day.

Sometimes the biggest breakthroughs aren't new model architectures.

Sometimes they're simply better ways of taking the next step downhill.

Did Adam fundamentally change deep learning, or was it simply the optimizer that happened to arrive at the right time?

I'd love to hear your thoughts—and if you've trained neural networks yourself, have you ever switched away from Adam and seen better results?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

The Small Mathematical Trick That Helped Make LLMs Possible: Understanding Layer Normalization

Shrijith Venkatramana — Fri, 03 Jul 2026 19:34:34 +0000

Layer Normalization is one of those ideas that seems almost disappointingly simple.

It has no billion-parameter architecture.
No attention mechanism.
No clever prompting strategy.

Just a few lines of mathematics that normalize numbers.

Yet without it, training today's Transformer models—from GPT to Llama to Claude—would be dramatically more difficult, slower, and often unstable.

Like many important engineering ideas, its brilliance lies in making everything else work.

Let's look at why.

Before Transformers: Why Deep Networks Were Difficult to Train

Imagine building a neural network with 100 layers.

Each layer receives activations from the previous one and transforms them.

Now suppose one layer begins producing values that are twice as large as yesterday.

Every subsequent layer suddenly receives inputs from a completely different distribution.

The next layer must constantly adapt.

Then the next one.

Then the next.

Training becomes like trying to walk on an escalator whose speed changes every second.

This phenomenon became widely known as internal covariate shift, a term popularized by Sergey Ioffe and Christian Szegedy in their 2015 paper introducing Batch Normalization.

Batch Normalization was enormously successful for convolutional networks.

But it came with an important limitation.

It depends on statistics computed across a batch of examples.

That works well for image classification.

It is much less convenient for recurrent networks and later for language models, where sequence lengths vary and batches are often small or irregular.

Researchers needed something different.

Something that normalized each individual example independently.

The Elegant Idea: Normalize Inside Every Layer

In 2016, Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton introduced Layer Normalization.

The insight was surprisingly simple.

Instead of asking:

"How does this neuron compare across different training examples?"

Layer Normalization asks:

"Within this single example, are the activations well-scaled?"

Suppose a hidden representation contains:

[12, 15, 18, 21]

These numbers are relatively large.

Another example might produce:

[-0.8, 0.2, 1.1, -0.5]

Instead of allowing every layer to operate over wildly different numerical ranges, Layer Normalization rescales each representation so that its activations have roughly:

mean = 0
variance = 1

Each layer therefore receives inputs with predictable numerical properties.

That consistency dramatically improves optimization.

Think of it as automatically adjusting the zoom level before every calculation.

The underlying information stays the same.

Only the scale changes.

The Mathematics Isn't Complicated

Suppose one token's hidden representation is

[4, 6, 8]

The average is

mu = (4 + 6 + 8) / 3 = 6

The variance becomes

((4 - 6)^2 + (6 - 6)^2 + (8 - 6)^2) / 3
= (4 + 0 + 4) / 3
~ 2.67

Standard deviation:

sqrt(2.67) ~ 1.63

Now normalize:

(4 - 6) / 1.63 = -1.22

(6 - 6) / 1.63 = 0

(8 - 6) / 1.63 = 1.22

Instead of arbitrary numbers,

[4,6,8]

we obtain

[-1.22, 0, 1.22]

The representation now has a stable scale regardless of how large or small the original values were.

In practice, an epsilon is added to avoid division by zero:

x_hat = (x - mu) / sqrt(sigma^2 + epsilon)

Then the model immediately learns two trainable parameters:

y = gamma * x_hat + beta

Why?

Because sometimes the optimal distribution isn't exactly mean zero and variance one.

The model learns the best scale (gamma) and offset (beta) automatically.

Normalization gives stability.

The learned parameters preserve flexibility.

Why This Became Essential for Transformers

When Ashish Vaswani and colleagues introduced the Transformer in 2017 with the famous paper Attention Is All You Need, every Transformer block contained Layer Normalization.

Each block performs operations like:

Multi-head attention
Residual connections
Feed-forward networks

Without normalization, residual additions can gradually amplify activations as depth increases.

Imagine adding numbers repeatedly:

10
+12
+15
+18
...

Soon the values become much larger than earlier layers expected.

Layer Normalization continuously recenters and rescales these representations before computation proceeds.

This keeps optimization well-conditioned.

Modern LLMs stack dozens—or even hundreds—of Transformer layers.

Small numerical instabilities accumulate rapidly.

Layer Normalization prevents those instabilities from snowballing.

Over time researchers also discovered that moving Layer Normalization before each sub-layer (Pre-LN Transformers) significantly improved gradient flow for extremely deep models.

Today, nearly every large language model adopts some variation of this design.

A Quick Back-of-the-Envelope Calculation

Suppose a Transformer has

hidden dimension = 4096
sequence length = 2048

Each token requires computing:

one mean
one variance
one normalization

That's roughly proportional to 4096 floating-point operations per token.

For the entire sequence:

4096 x 2048
~ 8.4 million values

That sounds large.

Until you compare it with attention.

Self-attention scales approximately as

O(sequence^2)

For 2048 tokens:

2048^2
~ 4.2 million pairwise interactions

And each interaction itself involves vector operations.

In practice, Layer Normalization contributes only a tiny fraction of the total computational cost.

The economics are excellent.

A relatively inexpensive computation dramatically improves optimization stability, allowing larger learning rates, deeper models, and more reliable convergence.

It's one of those rare engineering trade-offs that is overwhelmingly favorable.

Beyond LayerNorm: RMSNorm and the Next Generation

As models became larger, researchers began asking:

Do we really need to subtract the mean?

One popular alternative is RMSNorm, introduced by Biao Zhang and Rico Sennrich.

Instead of computing both mean and variance, RMSNorm normalizes only using the root-mean-square magnitude.

This removes some computation while preserving much of the optimization benefit.

Many modern open-source LLMs—including several recent Llama-family models—use RMSNorm instead of classic LayerNorm.

This illustrates an important engineering pattern.

Once researchers understood why normalization worked, they could simplify it without sacrificing performance.

The original idea remained.

Only the implementation evolved.

The Bigger Lesson

History often celebrates attention as the invention that created modern language models.

Attention certainly deserves the spotlight.

But attention alone was never enough.

Deep learning progresses because many seemingly "small" ideas accumulate:

residual connections
better optimizers
positional encodings
normalization
improved initialization

Layer Normalization is a perfect example.

It rarely appears in product announcements.

Few conference talks focus exclusively on it.

Yet every token processed by today's LLMs quietly passes through it again and again.

Sometimes the biggest breakthroughs aren't entirely new capabilities.

They're mathematical refinements that make ambitious ideas practical.

And engineering history is full of exactly these kinds of invisible innovations.

What do you think is the most underrated idea in deep learning?

Is it Layer Normalization, residual connections, Adam, positional encodings—or something else that enables modern AI without getting much of the credit?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

The English Parsing Problem That Led to Modern LLM Transformers

Shrijith Venkatramana — Thu, 02 Jul 2026 17:44:47 +0000

When people explain Large Language Models, they usually start with ChatGPT, attention, or transformers.

A much better place to start is a problem linguists had been struggling with for decades:

How do you teach a computer to understand the grammatical structure of a sentence?

This wasn't just an academic curiosity.

Search engines, machine translation, question answering, speech recognition, document understanding, and even programming languages all depend on extracting structure from sequences of symbols.

One of the benchmark problems became English Constituency Parsing—given a sentence, recover the tree representing its grammatical structure.

It turns out that this seemingly narrow problem became one of the best demonstrations that the Transformer architecture had fundamentally changed NLP.

Let's see why.

Before Deep Learning: Parsing Was Mostly Hand-Crafted Rules

Imagine the sentence:

The little girl saw the dog with the telescope.

Humans immediately recognize that this sentence is ambiguous.

Did the girl use the telescope?

Or did the dog have the telescope?

Those are two completely different parse trees.

Computers, however, simply receive a sequence of words.

The | little | girl | saw | the | dog | with | the | telescope

The challenge is recovering hidden grammatical structure.

A simplified parse tree might look like:

Sentence
├── Noun Phrase
│   ├── Determiner
│   ├── Adjective
│   └── Noun
└── Verb Phrase
    ├── Verb
    ├── Noun Phrase
    └── Prepositional Phrase

This problem became known as constituency parsing.

Beginning in the 1970s and 1980s, researchers built enormous collections of grammar rules inspired by linguistic theories such as those developed by Noam Chomsky.

These systems worked...

...until real language got messy.

Every exception required another rule.

Every new language required another grammar.

Maintenance costs exploded.

Then Statistics Entered the Picture

During the 1990s, NLP underwent what many call the "statistical revolution."

Instead of manually writing thousands of grammar rules, researchers asked:

What if we simply learn grammar from data?

The creation of the Penn Treebank was a turning point.

Thousands of English newspaper sentences were manually annotated with parse trees.

For example:

(S
   (NP The cat)
   (VP sat
       (PP on
           (NP the mat))))

Instead of writing rules, researchers could now estimate probabilities.

Rather than saying

NP -> Determiner Noun

always occurs,

they estimated

P(NP -> Determiner Noun)

from millions of examples.

Suddenly parsing became a machine learning problem.

This dramatically improved accuracy.

But another limitation remained.

The models still relied heavily on manually designed features.

Neural Networks Changed the Game

Around 2013-2016, neural networks began replacing handcrafted features across NLP.

Instead of engineers inventing hundreds of linguistic features, models learned useful representations directly from text.

One breakthrough came from recurrent neural networks (RNNs) and later LSTMs.

These models could process words sequentially.

The -> little -> girl -> saw -> ...

Each word updated an internal hidden state.

This worked surprisingly well.

But there was a problem.

Suppose the sentence contains 40 words.

The subject might appear near the beginning.

The verb might appear much later.

Information had to travel through dozens of recurrent steps.

Even LSTMs struggled with long-range dependencies.

Training also became difficult because computations were inherently sequential.

GPUs like parallel work.

RNNs do not.

The Transformer Arrived

In 2017, researchers at Google published the landmark paper:

"Attention Is All You Need."

Instead of processing words one after another, the Transformer lets every word directly inspect every other word.

Imagine each word asking:

Which other words matter for understanding me?

For example:

The programmer who fixed the parser yesterday deployed it.

The word

deployed

can directly connect to

programmer

instead of waiting for information to propagate through every intermediate word.

This is called self-attention.

The result was two enormous advantages.

First, long-distance grammatical relationships became much easier to model.

Second, every word could be processed simultaneously.

Modern GPUs love this.

Training speed increased dramatically.

English Constituency Parsing Became an Early Proof

Many people associate Transformers with machine translation.

Less widely remembered is how quickly they dominated constituency parsing.

One particularly influential paper was:

"Constituency Parsing with a Self-Attentive Encoder"

published by Nikita Kitaev and Dan Klein in 2018.

Instead of recurrent networks, they built the parser entirely around self-attention.

The results were striking.

On the Penn Treebank benchmark, the model achieved state-of-the-art accuracy while being conceptually simpler than many previous systems.

Even more interesting was why it worked.

The authors found that separating content information from position information inside attention significantly improved parsing performance.

In other words, knowing what a word is and where it occurs are distinct signals, and the model benefits from treating them differently.

This observation influenced later Transformer research far beyond parsing.

Why Self-Attention Fits Parsing So Well

Parsing is fundamentally about relationships.

Take the sentence:

The book on the table near the window belongs to Alice.

The subject

book

must eventually connect with

belongs

even though several phrases intervene.

A recurrent model passes information through many intermediate states.

A Transformer simply creates a direct interaction.

Mathematically, each word produces three vectors:

Query
Key
Value

The attention score between two words is proportional to

Query · Key

The dot product measures compatibility.

If the vectors point in similar directions, the score becomes large.

A softmax converts these scores into probabilities.

Finally, each word becomes a weighted average of the Value vectors from every other word.

Intuitively:

Queries ask questions.
Keys advertise available information.
Values carry the information itself.

Every word gets to decide whom it should listen to.

This is exactly the kind of computation grammatical analysis requires.

The Economics of Parallelism

Why was this architecture such a big deal operationally?

Suppose we have a sentence of 100 words.

An RNN performs roughly 100 sequential computation steps.

Even if each step is small, the next cannot begin until the previous finishes.

A Transformer computes attention between all pairs of words.

That is roughly:

100 × 100 = 10,000

pairwise interactions.

That sounds much more expensive.

And in raw arithmetic, it is.

Self-attention has quadratic complexity with sentence length.

So why did it win?

Because GPUs can compute thousands of matrix operations simultaneously.

Instead of performing 100 serialized operations,

they perform one enormous parallel matrix multiplication.

Hardware utilization skyrockets.

The result is a classic engineering tradeoff:

More arithmetic
Far less waiting

Modern accelerators strongly favor the second option.

This shift—from minimizing floating-point operations to maximizing hardware throughput—is one reason Transformers displaced RNNs so quickly.

Looking Back

English constituency parsing might sound like a niche benchmark today.

In reality, it helped demonstrate something profound.

Language understanding isn't primarily about processing words one after another.

It's about modeling relationships between them.

The Transformer architecture embraced that idea directly.

The same self-attention mechanism that learned grammatical trees now powers systems capable of writing software, translating dozens of languages, summarizing books, answering scientific questions, and helping developers every day.

Sometimes the technologies that change the world first prove themselves on problems most people have never heard of.

English constituency parsing was one of those problems.

What surprised you most about the history of Transformers?

Was it that they first proved themselves on tasks like machine translation and parsing, or did you expect conversational AI to be the original breakthrough? I'd love to hear your thoughts.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Scaled Dot-Product Attention: The 4-Line Algorithm That Powers Modern LLMs

Shrijith Venkatramana — Wed, 01 Jul 2026 17:49:09 +0000

In 2017, a small group of Google researchers removed recurrence, removed convolution, and bet everything on one deceptively simple idea: every word should decide for itself what deserves attention.

That idea—Scaled Dot-Product Attention—became the computational primitive behind GPT, Claude, Gemini, Llama, DeepSeek, and nearly every modern Large Language Model.

The remarkable part isn't just that it works.

It's that the core algorithm fits into a single equation.

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

In this article we'll build intuition first, then gradually unpack the mathematics, engineering, and economics behind the mechanism that made modern LLMs possible.

(Include the screenshot from the "Attention Is All You Need" paper here.)

Before Transformers: The Memory Problem

Imagine asking someone to finish this sentence:

"The trophy didn't fit in the suitcase because it was too small."

What does it refer to?

The trophy?

Or the suitcase?

Humans answer almost instantly because our brains naturally connect related concepts across a sentence.

Earlier neural networks struggled.

RNNs

Recurrent Neural Networks processed words one at a time.

The -> trophy -> didn't -> fit -> ...

Each word updated a hidden state.

The problem was that information had to travel through dozens or hundreds of sequential steps before reaching later words.

By the time the network reached the end of a paragraph, early information had often faded away.

LSTMs improved the situation with gating mechanisms, but they still fundamentally processed sequences sequentially.

That became an enormous bottleneck.

Both computationally.

And conceptually.

The Insight: Let Every Word Look Everywhere

One of the authors of the Transformer paper, Ashish Vaswani, later described the goal simply:

Instead of carrying memory forward step-by-step, why not allow every word to directly inspect every other word?

Suppose we have:

The cat sat on the mat.

When processing sat, perhaps the model mostly cares about:

It doesn't need to care very much about The.

Instead of forcing information through intermediate states, attention allows direct communication.

        sat
      /  |  \
     /   |   \
   cat   on  mat

Every token asks:

"Which other tokens are relevant to me?"

That's attention.

Queries, Keys and Values: Think Like a Search Engine

The names sound intimidating.

They're actually borrowed from information retrieval.

Imagine Google Search.

When you search:

best pizza near me

You issue a query.

Every webpage has characteristics that determine whether it matches.

Those are analogous to keys.

The content you finally read is the value.

Exactly the same thing happens inside attention.

Every word generates three vectors:

Query (Q) - What am I looking for?
Key (K) - What information do I offer?
Value (V) - What information should I contribute if selected?

Suppose we have:

The animal didn't cross the road because it was tired.

For the token it:

Its Query might strongly match:

animal

instead of

road

because their semantic representations are more compatible.

Attention is therefore a sophisticated matching process.

The Famous Equation (That Looks Scarier Than It Is)

The Transformer paper defines attention as:

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

Let's decode it piece by piece.

Step 1: Compare Queries Against Every Key

Each Query is compared with every Key using a dot product.

A dot product is simply a similarity score.

Large positive value:

Very relevant.

Near zero:

Mostly unrelated.

Negative:

Probably irrelevant.

Suppose our similarities are:

Word	Score
cat	12
dog	2
road	-1

Higher means stronger relevance.

Notice that we don't compare a Query against just one Key.

Every Query is compared against every Key simultaneously.

If there are 100 tokens in the sentence, each Query computes 100 similarity scores.

Step 2: Why Divide by sqrt(d_k)?

This is the "Scaled" part.

Without scaling, dot products become enormous.

Imagine vectors of length 512.

Even if each component averages only around 1, adding hundreds of multiplications quickly produces very large numbers.

A useful back-of-the-envelope calculation is that the variance of a dot product grows roughly in proportion to d_k.

If:

d_k = 512

then

sqrt(512) is approximately 22.6

Those values are fed into the softmax function.

Softmax contains exponentials.

For example:

exp(22) is roughly 3.5 billion
exp(10) is roughly 22 thousand

A small increase in the input suddenly creates a huge difference in the output.

One score completely dominates.

Everything else effectively becomes zero.

That creates two problems:

unstable gradients
slower learning

Dividing every score by sqrt(d_k) keeps the values in a healthy numerical range.

It's essentially variance normalization.

A remarkably small trick with enormous practical consequences.

Step 3: Softmax Creates Probabilities

Suppose the scaled scores become:

3
2
0

Softmax converts them into something approximately like:

0.71
0.26
0.03

These become attention weights.

Now the model knows:

spend about 71% of attention here
spend about 26% here
mostly ignore the rest

The probabilities always sum to 1.

Step 4: Weighted Sum of Values

Finally those probabilities weight the Value vectors.

Think of it like averaging expert opinions.

Expert A : 70%
Expert B : 25%
Expert C : 5%

The final representation becomes:

0.70 * A + 0.25 * B + 0.05 * C

That weighted average becomes the new representation for the current token.

Instead of copying information from a single location, attention intelligently blends information from multiple relevant tokens.

Why Matrix Multiplication Changed Everything

The equation often looks abstract because it's written with matrices.

That choice was an engineering breakthrough.

Instead of processing one word at a time:

word 1
word 2
word 3
...

the Transformer processes every token simultaneously.

If a sentence contains 128 words,

it computes attention for all 128 together using large matrix multiplications.

Modern GPUs are extraordinarily efficient at matrix multiplication.

This wasn't merely mathematically elegant.

It matched the hardware.

Google's TPUs were designed around massive matrix operations.

NVIDIA GPUs excel at them too.

The algorithm and the hardware reinforced one another.

This is one reason Transformers scaled so dramatically.

Sometimes the biggest breakthrough isn't inventing a new algorithm.

It's inventing one that perfectly matches the hardware already available.

The Hidden Cost: Attention Isn't Free

Attention is powerful.

It is also expensive.

Suppose a sequence contains n tokens.

Every token compares itself with every other token.

That means roughly:

n * n

or simply:

O(n^2)

comparisons.

Double the sequence length:

1,000 tokens
      ->
2,000 tokens

and you perform about four times as much work.

Approximate example:

Tokens	Pairwise Comparisons
1,000	1 million
10,000	100 million
100,000	10 billion

This single operation dominates both memory usage and inference cost.

Much of today's LLM research—including FlashAttention, sparse attention, sliding-window attention, grouped-query attention, and linear attention—is fundamentally about making this computation cheaper without sacrificing quality.

In many ways, modern AI engineering has become an optimization problem built around this one equation.

A Historical Moment Few Papers Ever Achieve

When Ashish Vaswani and seven colleagues published "Attention Is All You Need" in 2017, they were solving a machine translation problem.

They were not trying to build ChatGPT.

Yet within a few years:

OpenAI built GPT on the Transformer architecture.
Google introduced BERT using the same core attention mechanism.
Nearly every frontier LLM adopted Scaled Dot-Product Attention as its computational primitive.

Some research papers introduce new techniques.

Very few redefine an entire field.

This was one of them.

Today, when billions of people interact with ChatGPT, Claude, Gemini, or Llama, they're ultimately benefiting from an idea that occupies only a few lines in a research paper.

Final Thoughts

The beauty of Scaled Dot-Product Attention lies in its simplicity.

Every token asks a question.

Every other token advertises what it knows.

Similarity determines relevance.

Softmax decides how much to trust each source.

The answers are blended into a richer representation.

From those four operations emerged language models capable of writing code, translating languages, solving mathematical problems, generating images, and powering AI assistants used by hundreds of millions of people.

Sometimes revolutions begin not with thousands of lines of code, but with a single elegant equation.

What surprised you most about Scaled Dot-Product Attention? Was it the simplicity of the mathematics, the engineering insight of matching GPUs with matrix operations, or the fact that dividing by sqrt(d_k) turned out to be one of the key ingredients that made today's LLMs train reliably at scale?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

What Actually Happens When You Train an LLM? Following the First 12 Hours of the Original Transformer

Shrijith Venkatramana — Tue, 30 Jun 2026 18:03:36 +0000

In 2017, eight NVIDIA P100 GPUs sat in a Google data center for about twelve hours.

During those twelve hours, they repeatedly did something almost embarrassingly simple.

They picked up a batch of sentences.

Made predictions.

Measured how wrong those predictions were.

Adjusted a few million numbers.

Then did it again.

Exactly 100,000 times.

Those twelve hours produced the base Transformer model described in Attention Is All You Need. The larger version trained for 300,000 optimization steps, taking roughly 3.5 days on the same hardware.

Today, frontier LLMs train on tens of thousands of GPUs for weeks, but if you inspected the training logs, the core loop would still look remarkably familiar.

This article follows that loop.

We'll watch one training run unfold—from raw text on disk to a model that can translate languages—and along the way learn a bit about every intimidating term that deal with training a transformer model.

8:00 AM — Nothing Has Been Learned Yet

Imagine switching on the machine.

The Transformer knows absolutely no language. It doesn't know English, or German. And it doesn't know grammar. It doesn't even have conception of what a "word" is.

Internally it contains millions of parameters—ordinary floating-point numbers initialized almost randomly.

The training data, however, already contains knowledge.

For the English-German task, the authors used the WMT 2014 dataset containing roughly 4.5 million sentence pairs.

A tiny sample might look like this:

English:
The meeting begins tomorrow.

German:
Das Treffen beginnt morgen.

English:
The cat sat on the mat.

German:
Die Katze saß auf der Matte.

Notice what's missing.

Nobody wrote rules like

"Adjectives come before nouns."

"German verbs often appear at the end."

The only supervision is example after example after example.

The model's job is to discover those rules itself.

8:00:01 — The Computer Doesn't See Words

Before training starts, the text is transformed into something GPUs understand.

Integers.

The paper uses Byte Pair Encoding (BPE), introduced a year earlier by Rico Sennrich and colleagues.

Instead of storing every possible English word, BPE builds a vocabulary of common subword pieces.

For example,

unbelievable

might become

un
believ
able

Those pieces become IDs.

un      → 517
believ  → 10328
able    → 294

Why go through this trouble?

Imagine giving every English word its own entry.

You'd need hundreds of thousands of entries, and every new word—"ChatGPT", "Kubernetes", "DeepSeek"—would be unknown.

Subwords solve that elegantly.

Once the model understands "micro", "service" and "architecture", it already has much of what it needs to interpret "microservice architecture", even if it has never encountered the exact phrase before.

Modern tokenizers have evolved, but this basic idea remains.

8:00:02 — The First Batch Arrives

One beginner misconception is that the GPU trains on one sentence at a time.

That would waste almost all of its computational power.

GPUs are throughput machines.

They become efficient only when thousands of arithmetic units work simultaneously.

Instead, the Transformer paper groups examples into batches containing approximately

25,000 source-language tokens
25,000 target-language tokens

or about 50,000 tokens in total.

Think of a factory.

Running one car down an assembly line would be absurd.

Factories move hundreds of products simultaneously because keeping machines idle is expensive.

GPU training works the same way.

Batching is not a machine-learning trick.

It's operations optimization.

8:00:02.4 — The Model Makes Its First Mistake

The first forward pass takes roughly 0.4 seconds.

The model receives

The cat sat on the mat.

and produces...

garbage.

Maybe something equivalent to

House.

Tomorrow.

Blue.

Water.

That isn't failure.

It's exactly what we expect.

Every parameter was random only moments ago.

Now comes the crucial question.

How wrong was the prediction?

The answer is summarized by a single number called the loss.

Everything that follows exists solely to reduce that number.

8:00:02.5 — Which of the 65 Million Parameters Was Responsible?

Suppose I asked you to tune an old radio using sixty-five million knobs.

After hearing static, which knob would you turn?

You wouldn't know.

Yet that's essentially the problem.

The Transformer base model contains about 65 million trainable parameters.

The larger model contains around 213 million.

Backpropagation solves this enormous credit-assignment problem.

Rather than saying

"Parameter #18,423 is wrong,"

it computes

"If this parameter increased slightly, would the loss increase or decrease?"

for every single parameter.

The result is a gigantic map of tiny suggested adjustments called gradients.

Now another algorithm enters the story.

Adam: The Engineer Who Turns the Knobs

The paper uses the Adam optimizer with

β₁ = 0.9
β₂ = 0.98
ε = 10⁻⁹

These aren't arbitrary constants copied from Stack Overflow.

Adam remembers recent gradients, rather like giving the optimization process momentum.

Imagine descending a foggy mountain.

If every step depended only on the slope beneath your feet, you'd zigzag constantly.

Adam remembers where you've been heading over the past several steps, smoothing the journey downhill.

Interestingly, the paper chose β₂ = 0.98 rather than the more familiar 0.999 found in many deep-learning libraries today. That makes Adam respond more quickly to changing gradients—a small but deliberate engineering decision.

Millions of parameters are nudged.

Tiny, incremental changes.

Often by less than one thousandth.

Then the next batch arrives.

9:00 AM — The Strange Equation Everyone Hates

One of the paper's most intimidating equations defines the learning rate.

It looks frightening.

Its purpose is not.

Early in training, every parameter is effectively random.

Large updates can make optimization unstable.

So the authors warm up the learning rate over the first 4,000 optimization steps, gradually increasing it rather than starting at full speed.

After warmup, the learning rate begins shrinking.

Imagine sanding a table.

At first you remove material aggressively.

Near the end you make tiny finishing passes.

Training behaves similarly.

The equation also contains the term (d_{\text{model}}^{-1/2}).

This compensates for model size.

As hidden representations become larger, gradients naturally change scale. Dividing by the square root of the model dimension helps keep parameter updates numerically well behaved as architectures grow.

The equation is just common sensical engineering.

Noon — Preventing the Model From Memorizing

If optimization only chased lower loss, the network could simply memorize the training data.

The paper deliberately makes learning harder.

First comes dropout.

Ten percent of activations are randomly disabled during training.

Every batch therefore sees a slightly different network.

No neuron can become indispensable.

Second comes label smoothing with a value of 0.1.

Instead of pretending the correct next token has probability exactly one, the target distribution is softened slightly.

That sounds counterintuitive.

Yet translation quality improved.

Real language is messy.

There are often several acceptable translations.

Slight uncertainty produces a less overconfident model.

8:00 PM — Twelve Hours Later

After roughly 100,000 optimization steps, the base model has finished training.

The larger model continues until 300,000 steps, taking approximately 3.5 days.

Each step processed about 50,000 tokens.

Back-of-the-envelope, that's around five billion token presentations during the base run—not unique tokens, but training exposures. The same examples are revisited across multiple passes through the dataset.

The paper doesn't stop by reporting translation accuracy.

It also reports FLOPs.

That's significant.

Even in 2017, the authors understood that machine learning was becoming an engineering discipline constrained not only by accuracy, but also by computation.

A model that is 1% better but requires ten times more compute is often a poor engineering trade-off.

That thinking has only become more relevant.

Today, training an LLM is as much about distributed systems, networking, storage bandwidth, GPU utilization, checkpointing and failure recovery as it is about neural networks.

Closing Thoughts

People often remember Attention Is All You Need for introducing self-attention.

Equally important was something less glamorous: it demonstrated a training recipe that scaled.

Large batches kept GPUs busy.

Carefully designed learning-rate schedules stabilized optimization.

Adam made billions of tiny updates practical.

Regularization techniques prevented memorization.

None of these ideas are individually magical. Together, repeated hundreds of thousands of times, they turned random numbers into a model that could translate language.

Nearly a decade later, today's frontier LLMs still follow the same rhythm.

The numbers have changed by orders of magnitude.

The loop has not.

Load a batch. Predict. Measure the loss. Update the weights. Repeat.

That's the heartbeat of every modern language model.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

The Transformer Architecture Behind Modern LLMs: A Developer's Guide to the Diagram That Changed AI

Shrijith Venkatramana — Mon, 29 Jun 2026 17:09:22 +0000

If you've used ChatGPT, Claude, Gemini, or any modern LLM, you've already benefited from one of the most influential transformer architecture in modern AI.

In 2017, a team of researchers at Google published a paper titled "Attention Is All You Need." Hidden inside it was the now-famous architecture diagram shown below—the blueprint for the Transformer.

At first glance, it looks intimidating: boxes, arrows, loops, encoder stacks, decoder stacks, attention blocks...

But underneath the complexity lies a surprisingly elegant idea.

By the end of this article, you'll understand every single component in this diagram, why it exists, how information flows through it, and how all these pieces collaborate to generate human-like language.

Let's build our understanding layer by layer.

The Big Picture: Two Factories Working Together

Before diving into the individual blocks, ignore the details and look at the overall shape.

There are two major halves:

Encoder (left)
Decoder (right)

Think of them as two specialized factories.

Input sentence
      │
      ▼
 ┌───────────────┐
 │    Encoder    │
 └───────────────┘
      │
 Learned representation
      │
      ▼
 ┌───────────────┐
 │    Decoder    │
 └───────────────┘
      │
      ▼
 Generated text

Originally, this architecture was designed for machine translation.

Example:

English:
"The cat sat on the mat."

Encoder understands it.

↓

Decoder generates:

"Le chat était assis sur le tapis."

The encoder's only responsibility is understanding.

The decoder's responsibility is generation.

Today's GPT models actually use only the decoder, while models like BERT use only the encoder. The original Transformer paper contained both because translation requires understanding one language before producing another.

Step 1 — Input Embeddings: Converting Words into Numbers

Computers don't understand words.

They understand vectors.

When the sentence

The cat sat

enters the Transformer, each word is converted into a dense numerical vector.

"The"
↓

[0.17, -0.42, 1.33, ...]

These vectors are called embeddings.

Words with similar meanings naturally end up close together in this high-dimensional space.

For example,

king
queen
prince
princess

occupy nearby regions.

Instead of manually designing these vectors, the Transformer learns them during training.

In the architecture diagram, this is the very first pink box:

Inputs
   │
   ▼
Input Embedding

At this stage, every token is simply represented as a learned numerical vector.

Step 2 — Positional Encoding: Giving Words an Order

Here's an interesting problem.

Attention doesn't inherently know word order.

Consider:

Dog bites man

Man bites dog

Same words.

Completely different meaning.

Since attention processes every word simultaneously, we must explicitly tell the model where each word appears.

That's the purpose of Positional Encoding.

The positional vector is added directly to the embedding.

Embedding

+

Position

=

Final input vector

This explains the small ⊕ symbol in the diagram.

Embedding
      │
      ▼
      ⊕
     / \
Embedding Position

Rather than learning grammar from sequence alone, the model receives positional information immediately.

You can think of it as giving every token both:

what it is
where it occurs

Step 3 — The Encoder Stack: Understanding the Entire Sentence

Now we reach the large box labelled N×.

┌─────────────────────┐
│ Multi-Head Attention│
│ Add & Norm          │
│ Feed Forward        │
│ Add & Norm          │
└─────────────────────┘

Repeated N times

The paper used N = 6.

Modern LLMs often use dozens or even hundreds of layers.

Each encoder layer gradually refines the representation.

Think of reading a paragraph.

Your first reading identifies words.

The second discovers phrases.

The third understands relationships.

The fourth extracts meaning.

Each encoder layer performs another refinement pass.

Step 4 — Multi-Head Attention: The Heart of the Transformer

This is the innovation that changed deep learning.

Suppose the sentence is:

The animal didn't cross the street because it was tired.

What does it refer to?

Attention allows every word to examine every other word before updating its representation.

animal  ←──────────┐
                   │
cross ─────────────┤
                   │
street ────────────┤
                   │
it  ◄──────────────┘

Instead of only looking at nearby words like RNNs, every token has access to the entire sentence.

This allows long-distance dependencies to be captured naturally.

Why Multiple Heads?

One attention mechanism isn't enough.

Different relationships matter.

One head may learn:

grammatical structure

Another:

pronoun resolution

Another:

verb-object relationships

Another:

semantic similarity

Imagine several experts reading the same sentence simultaneously.

Head 1:
Grammar

Head 2:
Meaning

Head 3:
Syntax

Head 4:
Long-range context

Their outputs are combined into a richer representation.

This is why it's called Multi-Head Attention.

Step 5 — Add & Norm: Keeping Training Stable

Notice that after every major block we see

Add & Norm

This performs two operations.

Residual Connection (Add)

Instead of replacing information, we preserve the original.

Output

=

Attention(x)

+

x

This shortcut helps gradients flow through deep networks.

Without it, training hundreds of layers becomes extremely difficult.

Layer Normalization (Norm)

Different layers naturally produce values on different scales.

Layer normalization keeps activations well-behaved.

Think of it as recalibrating measurements after every processing stage.

Without normalization:

Layer 1

0.3

Layer 2

500

Layer 3

0.0004

Training quickly becomes unstable.

Normalization keeps everything numerically manageable.

Step 6 — Feed Forward Networks: Thinking Independently

Attention allows tokens to exchange information.

The Feed Forward layer allows each token to process what it has learned.

For every token independently:

Vector

↓

Linear

↓

Activation

↓

Linear

No interaction happens here.

Instead, this stage performs deeper feature extraction.

An analogy:

Attention is a group discussion.

Feed Forward is everyone quietly thinking afterward.

This alternating pattern—

Discuss

↓

Think

↓

Discuss

↓

Think

is repeated across every Transformer layer.

Step 7 — The Decoder: Generating One Token at a Time

Now we move to the right half of the diagram.

The decoder is responsible for producing text.

Its input isn't the original sentence.

Instead it receives:

<START>

↓

The

↓

The cat

↓

The cat sat

↓

...

Notice the label:

Outputs
(shifted right)

This means that during training, the decoder receives the correct previous token as input while learning to predict the next one.

If the target sentence is:

The cat sat

the decoder sees:

<START>

The

The cat

and learns to predict:

The

cat

sat

This "teacher forcing" strategy makes training much more efficient because the model always conditions on the correct history rather than its own mistakes.

Step 8 — Masked Multi-Head Attention: Preventing Cheating

Imagine predicting the next word in:

The cat sat on the __

If the model could already see

mat

there would be nothing to learn.

So the decoder applies Masked Multi-Head Attention.

Current token

↓

Can see:

Previous words ✔

Future words ✘

During generation:

I love

cannot attend to

pizza

until pizza has actually been generated.

The mask preserves causality.

This is precisely why GPT models generate text one token at a time.

Step 9 — Cross-Attention: Looking Back at the Encoder

The second attention block inside the decoder is different.

Here, the decoder attends to the encoder output.

Encoder

↓

Sentence meaning

↓

Decoder consults it

Suppose we're translating:

"The red car."

While generating

voiture

the decoder continually asks:

Which parts of the original sentence are relevant right now?

This interaction between encoder and decoder is called cross-attention (shown simply as "Multi-Head Attention" in the original diagram, with arrows coming from the encoder stack).

It lets the decoder ground each generated token in the encoded meaning of the source sentence instead of relying only on previously generated words.

Decoder-only models like GPT omit this block because there is no separate encoder to consult.

Step 10 — Linear Layer + Softmax: Choosing the Next Word

After the decoder finishes processing, we finally reach the top of the diagram.

Decoder Output

↓

Linear

↓

Softmax

↓

Output Probabilities

The Linear layer converts the decoder's hidden representation into one score for every token in the vocabulary.

Imagine a vocabulary containing 50,000 words.

The output might look like:

cat      5.8
dog      2.1
apple   -0.4
car      1.7
...

These raw scores (often called logits) aren't probabilities yet.

The Softmax layer transforms them into a probability distribution:

cat      0.81
dog      0.09
car      0.04
apple    0.01
...

The model can then choose the next token—either the most probable one or a sampled alternative depending on the decoding strategy.

That chosen token is fed back into the decoder, and the entire process repeats until an end-of-sequence token is produced.

Putting It All Together: Following the Data Through the Diagram

Now the entire figure becomes much easier to read.

Input text

↓

Input Embedding

↓

Positional Encoding

↓

Encoder Stack (N layers)
    ├─ Multi-Head Attention
    ├─ Add & Norm
    ├─ Feed Forward
    └─ Add & Norm

↓

Context-rich representation

↓

Decoder receives previous outputs
(shifted right)

↓

Output Embedding

↓

Positional Encoding

↓

Masked Multi-Head Attention
(looks only at earlier generated tokens)

↓

Cross-Attention
(consults the encoder output)

↓

Feed Forward

↓

Repeat N layers

↓

Linear

↓

Softmax

↓

Next token probability

↓

Repeat until complete sentence

Every block has a specific role:

Component	Purpose
Input Embedding	Convert tokens into dense vectors
Positional Encoding	Encode word order
Multi-Head Attention	Let tokens exchange information globally
Masked Multi-Head Attention	Prevent access to future tokens during generation
Cross-Attention	Allow the decoder to consult the encoder's understanding
Feed Forward	Transform each token's representation independently
Add & Norm	Stabilize optimization and preserve information via residual connections
Encoder Stack (N×)	Build increasingly rich contextual representations
Decoder Stack (N×)	Generate the output sequence one token at a time
Linear	Produce a score for every vocabulary token
Softmax	Convert scores into probabilities for selecting the next token

A Crisp Summary To Help You Remember This

The Transformer succeeded because it replaced sequential processing with parallel attention, enabling models to reason over entire sequences at once while still generating coherent text token by token.

Almost every major language model today—from GPT and Claude to Llama, Mistral, and Gemini—can trace its lineage back to this deceptively simple diagram. While modern architectures introduce refinements such as rotary positional embeddings, grouped-query attention, mixture-of-experts layers, and optimized decoding strategies, the core ideas remain strikingly similar to those introduced in 2017.

The next time someone says an LLM is "just predicting the next token," remember what's happening under the hood: embeddings capture meaning, positional encodings preserve order, attention weaves relationships across the sequence, feed-forward networks refine those representations, residual connections keep deep networks trainable, and the decoder repeatedly transforms all of that into one probability distribution after another until a coherent response emerges.

What part of the Transformer architecture surprised you the most—the fact that every token can attend to every other token, the masking that enables autoregressive generation, or how such a simple stack of repeated blocks scales to models with hundreds of billions of parameters?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Self-Attention: The Brilliant Idea That Made Large Language Models Possible

Shrijith Venkatramana — Sun, 28 Jun 2026 17:18:46 +0000

How a seemingly simple mathematical trick replaced decades of sequential neural networks and unlocked the age of GPT.

Imagine asking ten software engineers to summarize a pull request.

One engineer reads every line from top to bottom. Another immediately jumps to the files that seem most relevant. A senior engineer skims most of the code but pays close attention to the parts that affect authentication, concurrency, or performance.

The senior engineer isn't processing every line equally.

They're paying attention.

That simple observation eventually became one of the most important ideas in modern machine learning. In 2017, a group of researchers at Google published a paper with an almost understated title: "\"Attention Is All You Need.\" The paper introduced the Transformer, a new neural network architecture that abandoned recurrent networks entirely in favor of one central mechanism: self-attention."

Today, nearly every major Large Language Model—GPT, Claude, Gemini, Llama, DeepSeek, Mistral—builds upon this idea.

Let's understand why.

Before Transformers: Language Was Processed Like a Conveyor Belt

For nearly two decades, sequence models were dominated by Recurrent Neural Networks (RNNs) and later LSTMs and GRUs.

Suppose we have the sentence:

The animal didn't cross the road because it was tired.

An RNN processes it like this:

The
 ↓
animal
 ↓
didn't
 ↓
cross
 ↓
...
 ↓
tired

Every new word updates a hidden state.

If the model wants to understand "it", information about "animal" has already travelled through six or seven intermediate computations.

It's a little like the children's game of telephone. Every time information is passed forward, a little noise is introduced.

The longer the sentence becomes, the harder it is to preserve distant information.

This caused several practical problems:

Long-range dependencies became difficult.
Training was inherently sequential.
GPUs—which thrive on parallel computation—were underutilized.

Even clever improvements like LSTMs only partially solved these issues.

Researchers began asking a different question:

What if every word could simply look at every other word directly?

That question became self-attention.

The Core Idea: Every Word Gets to Read the Entire Sentence

Instead of processing words one after another, self-attention lets every token consult every other token before deciding what it should represent.

Consider:

The trophy didn't fit into the suitcase because it was too small.

When humans read "it", we naturally ask:

trophy?
suitcase?

Our brain briefly looks backward.

Transformers perform the same operation mathematically.

When computing the representation for it, the model assigns attention weights:

Word	Importance
trophy	0.08
suitcase	0.67
small	0.17
fit	0.05
others	0.03

These numbers are not programmed.

They are learned from enormous amounts of text.

The new representation becomes approximately:

representation(it)

=
0.67 × suitcase
+
0.17 × small
+
0.08 × trophy
...

Notice something subtle.

The word it itself never changes.

Instead, its vector representation becomes richer because it incorporates contextual information from the rest of the sentence.

This is why the mechanism is called self-attention.

The sentence is attending to itself.

Why This Was Revolutionary

The Google paper's title—Attention Is All You Need—was intentionally provocative.

At the time, attention mechanisms already existed.

Bahdanau and colleagues had introduced attention in neural machine translation in 2014. However, attention was only an add-on to recurrent networks.

The Transformer asked a far bolder question:

What happens if we remove recurrence completely?

Instead of:

Input
 ↓
LSTM
 ↓
LSTM
 ↓
LSTM

the Transformer became:

Input
 ↓
Self Attention
 ↓
Feed Forward
 ↓
Self Attention
 ↓
Feed Forward

No recurrence.

No convolutions.

Just attention layers stacked dozens—or eventually hundreds—of times.

Many researchers initially viewed this as risky.

Within a year, it became obvious the idea worked astonishingly well.

The Math Is Simple and Elegant

The mathematics often intimidates newcomers, but the underlying idea is straightforward.

Each token produces three vectors:

Query (Q) → What information am I looking for?
Key (K) → What information do I contain?
Value (V) → What information should I contribute?

Think of attending a technical conference.

Every attendee carries:

a list of questions they're interested in (Query),
a badge describing their expertise (Key),
the knowledge they can share (Value).

Conversation happens when someone's questions match another person's expertise.

Mathematically:

score = Query · Key

The dot product measures compatibility.

Large dot product?

Pay attention.

Small dot product?

Ignore.

The scores are normalized using the Softmax function:

weights = softmax(QKᵀ / √d)

The division by √d prevents very large vector dimensions from producing excessively large dot products that would make Softmax saturate. Without this scaling, gradients become small and training becomes unstable.

Finally,

Output = weights × V

Each token becomes a weighted combination of information from every other token.

That's the entire mechanism.

The famous equation occupies only a single line in the original paper.

Yet it changed AI forever.

A Back-of-the-Envelope Calculation: Why Attention Is Expensive

Self-attention's biggest strength is also its biggest weakness.

Suppose a context contains 4,096 tokens.

Every token compares itself against every other token.

Total comparisons:

4096 × 4096

≈ 16.8 million

Now consider modern models.

8,192 tokens
32 attention heads
dozens of Transformer layers
billions of parameters

The number of operations quickly reaches the trillions during training.

This explains why training frontier models requires thousands of GPUs running continuously for weeks or months.

The economics become equally striking.

A single GPU might cost tens of thousands of dollars. Training clusters contain thousands of them.

Electricity, cooling, networking, storage, engineering time, and failed experiments all contribute to training costs that can reach tens or even hundreds of millions of dollars for the largest models.

This computational expense has also motivated an entire research field devoted to making attention cheaper.

Techniques such as FlashAttention, grouped-query attention, sparse attention, and linear attention all attempt to preserve the quality of self-attention while reducing memory usage or computational complexity.

Ironically, many innovations in modern LLM engineering are really innovations in making self-attention practical at scale.

Why Self-Attention Became the Foundation of LLMs

Language isn't fundamentally sequential.

Relationships often span entire documents.

A variable declared hundreds of lines earlier influences the current line of code.

A pronoun refers to a noun introduced several paragraphs ago.

An API call depends on documentation presented earlier in the conversation.

Self-attention naturally models these relationships.

It also parallelizes beautifully.

Unlike RNNs, every token in a sequence can be processed simultaneously on modern GPUs.

That single architectural decision dramatically increased hardware utilization and enabled models to scale from millions of parameters to today's trillion-parameter frontier.

Perhaps the greatest lesson is that breakthroughs are not always about making systems more complicated.

Sometimes they're about removing assumptions.

The Transformer removed the assumption that language must be processed one word at a time.

Everything that followed—from GPT-2 to ChatGPT—was built on that realization.

Final Thoughts

It's easy to think of GPT as an impossibly complex black box.

But underneath the billions of parameters lies a surprisingly elegant principle.

Every word asks:

Which other words should I pay attention to?

That single question replaced decades of recurrent architectures and reshaped artificial intelligence.

Sometimes, the most revolutionary ideas aren't new ways of computing.

They're new ways of deciding what deserves attention.

What surprised you most about self-attention?

Was it that the core algorithm fits into a single equation, or that one architectural decision replaced decades of recurrent neural networks? I'd love to hear your thoughts—or any clever analogies you've found useful when explaining Transformers to other developers.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Sequence Transduction: The Forgotten Problem That Led to Modern LLMs

Shrijith Venkatramana — Sat, 27 Jun 2026 17:15:41 +0000

Most developers think large language models were built to predict the next word. They weren't—not at first.

If you travel back to the early 2010s, the hardest problems in AI weren't writing poems or generating code. They were translating English into French, converting speech into text, and summarizing documents. These were all instances of the same challenge: sequence transduction.

The term appears almost casually in the opening paragraph of the Transformer paper:

"...sequence modeling and transduction problems such as language modeling and machine translation."

Today, almost everyone knows the Transformer. Very few remember the problem it was invented to solve.

Ironically, solving sequence transduction turned out to create the foundation upon which modern LLMs would later emerge.

Let's explore why.

What Exactly Is Sequence Transduction?

Imagine you own a factory.

A sequence modeling problem asks:

"Given everything that has happened so far, what comes next?"

Like predicting the next product coming off the conveyor belt.

The cat sat on the _____
                ↓
               mat

This is language modeling.

A sequence transduction problem is larger idea.

Instead of predicting one missing piece, you transform an entire sequence into another.

English
↓

"The weather is nice."

↓

French

"Il fait beau."

Audio
↓

Waveform

↓

Text

"Welcome everyone."

Buggy code

↓

Correct code

Different input.

Different output.

Often different lengths.

The model must understand the entire source or at least large parts of it before generating the target.

In hindsight, modern AI assistants spend almost all of their time doing sequence transduction:

translating languages
summarizing reports
generating SQL
converting Python to Rust
explaining stack traces
producing commit messages
writing documentation

They all reduce to:

Input sequence → Output sequence

Why Was This Such a Hard Problem?

Humans underestimate how much memory translation requires.

Consider translating:

"The committee, after reviewing several proposals over three months, finally approved the budget."

Suppose you're translating into German.

The verb may not appear until the end.

To translate correctly, the model must remember:

who performed the action
tense
plurality
grammatical structure
dependencies dozens of words apart

Early neural networks processed text one word at a time.

Word₁ → hidden state
              ↓
Word₂ → hidden state
              ↓
Word₃ → hidden state

Everything had to be compressed into one hidden vector.

It was like asking someone to summarize an entire novel using only one sticky note.

Eventually information disappears.

This became known as the long-range dependency problem.

The Rise—and Limits—of Recurrent Neural Networks

During the late 1980s and 1990s, researchers developed Recurrent Neural Networks (RNNs) to process sequential data.

Unlike ordinary neural networks, RNNs reused the same parameters at every time step.

Instead of building a different network for every word, one network repeatedly updated an internal memory.

Mathematically:

hidden_state = f(previous_hidden_state, current_input)

The same computation runs repeatedly.

This parameter sharing was elegant.

Suppose an RNN contains one million parameters.

A thousand-word paragraph still uses one million parameters—not a billion.

The network simply reuses them.

Economically, this was attractive. But computationally, it was painful.

Everything had to happen sequentially.

Word 500 could not begin until word 499 finished.

No parallelism. No GPUs in picture. Training was slow.

LSTMs: Teaching Neural Networks to Remember

In 1997, Sepp Hochreiter and Jürgen Schmidhuber introduced one of the most influential ideas in deep learning:

Long Short-Term Memory (LSTM).

Instead of blindly overwriting memory every step, the network learned gates.

Think of memory like a whiteboard.

Each word asks three questions:

Should I erase something?
Should I remember this?
Should I reveal this later?

Those questions became three learned gates.

Forget gate.

Input gate.

Output gate.

Instead of forcing every piece of information through the same bottleneck, the model learned what deserved long-term storage.

A surprisingly intuitive analogy is human note-taking.

Most conversations are forgotten.

A few facts are written into your notebook.

LSTMs learned which facts deserved the notebook.

For over a decade, LSTMs dominated speech recognition, handwriting recognition, language translation, and time-series forecasting.

Google, Apple, Microsoft, and Baidu all deployed enormous production systems powered by them.

The Encoder–Decoder Revolution

Around 2014, another breakthrough appeared.

Instead of using one RNN/LSTMs for everything, researchers separated the task into two parts.

Input sentence
      ↓
Encoder
      ↓
Meaning vector
      ↓
Decoder
      ↓
Output sentence

This architecture became known as the sequence-to-sequence (Seq2Seq) model.

For the first time, neural networks learned translation end-to-end.

No phrase tables or handcrafted grammar or brittle rules. It was just millions of examples.

One famous anecdote came from Google.

Traditional statistical machine translation systems consisted of dozens of independently engineered components accumulated over years.

Neural machine translation replaced much of that complexity with a single differentiable model trained from data. In 2016, Google reported that its neural system substantially reduced translation errors across multiple language pairs while simplifying the overall pipeline.

This represented an engineering improvement for sure, but more importantly it was a philosophical shift.

Instead of programming language knowledge -- we trained it.

Attention Changed Everything

The Seq2Seq model still had one weakness.

Everything had to fit inside one vector.

Information gets lost.

In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed attention.

Instead of remembering everything, just look back whenever necessary.

While generating each output word, the decoder asks:

Which input words matter right now?

Not every word.

Only the relevant ones.

Translation suddenly became much easier.

Long sentences improved dramatically.

From Translation Engine to ChatGPT

The Transformer paper in 2017 -- instead of improving recurrent networks, removed recurrence entirely.

Every word could attend directly to every other word.

Parallel computation became possible.

Training speed increased enormously.

GPUs became dramatically more efficient because every token in a sequence could be processed simultaneously rather than one after another.

Even more interesting was the economics.

Suppose translating a sentence of 100 words with an RNN requires roughly 100 sequential computation steps.

A Transformer still performs similar amounts of arithmetic overall, but many of those operations can execute in parallel on modern accelerators.

The wall-clock training time drops dramatically because GPUs are optimized for large batches of matrix multiplications rather than long chains of sequential dependencies.

That operational advantage—not merely higher accuracy—made scaling practical.

The remarkable twist is that the architecture built to solve translation generalized astonishingly well.

Replace:

English → French

with

Question → Answer

Code → Documentation

Prompt → Python program

The underlying problem barely changes.

It remains sequence transduction.

Modern LLMs still perform next-token prediction during training.

But from a developer's perspective, they are universal transduction engines.

Every prompt is transformed into another sequence.

The interface changed.

The underlying abstraction survived.

Final Thoughts

The history of AI is often told as a story about predicting the next word.

That story is incomplete.

For decades, researchers wrestled with a harder question:

How do we transform one complex sequence into another while preserving meaning?

That single question drove the invention of encoder–decoder architectures, LSTMs, attention mechanisms, and ultimately the Transformer itself.

The next time you ask an LLM to refactor code, summarize a meeting, or generate a SQL query, remember what it's really doing.

Not merely predicting words.

Performing sequence transduction at an extraordinary scale.

What surprised you most about this history? Did you always think LLMs grew out of language modeling, or is it more useful to think of them as the latest—and perhaps most powerful—generation of sequence transduction systems?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

How We Actually Measure Whether an LLM's Output Is Good - BLEU, COMET and BLEURT

Shrijith Venkatramana — Fri, 26 Jun 2026 18:36:27 +0000

An AI model writes a paragraph. It sounds fluent. It looks convincing. But how do you know whether it's actually good?

This deceptively simple question has occupied researchers for more than two decades.

Long before ChatGPT, machine translation researchers faced exactly the same problem. Human evaluation was expensive, inconsistent, and painfully slow. If every new model required thousands of humans to compare translations, research would crawl.

That necessity gave rise to BLEU, one of the most influential evaluation metrics in AI history. Years later, as language models became better at paraphrasing and reasoning, BLEU started to show its age. Researchers responded with learned metrics like BLEURT and COMET, which use neural networks to judge language much more like humans do.

Interestingly, this mirrors software engineering itself. We first wrote simple unit tests, then integration tests, and today we increasingly rely on sophisticated observability systems. Evaluation metrics for LLMs have undergone a similar evolution.

Let's see why.

Before BLEU: The Evaluation Bottleneck

Imagine you're building Google Translate in 2001.

Every time your team improves the model, someone has to read thousands of translated sentences and score them.

Suppose a single sentence pair takes only 20 seconds to judge.

Evaluating 50,000 sentences would require nearly 280 human-hours.

Now imagine dozens of experiments every week.

Evaluation—not training—quickly becomes the bottleneck.

Researchers at IBM, led by Kishore Papineni, introduced BLEU (Bilingual Evaluation Understudy) in 2002 to automate this process.

Their idea was surprisingly simple:

If a machine translation resembles what professional translators write, it's probably good.

This became one of the most cited papers in natural language processing.

BLEU: Counting Shared Phrases

BLEU compares a model's output against one or more human reference translations.

Suppose the reference is:

The cat is sitting on the mat.

The model produces:

The cat sat on the mat.

Many words and short phrases overlap.

Now consider:

A feline rested indoors.

A human recognizes this as a perfectly reasonable translation.

BLEU mostly doesn't.

Why?

Because BLEU isn't measuring meaning.

It measures shared n-grams—contiguous sequences of words.

The score combines:

word matches (1-grams)
two-word phrases
three-word phrases
four-word phrases

It also penalizes outputs that are suspiciously short.

High-level intuition:

More matching phrases → higher score
Longer matching phrases → even better
Too short → penalty

This simple idea turned out to correlate surprisingly well with human judgments across large datasets.

When the famous Transformer paper Attention Is All You Need reported 28.4 BLEU on the WMT English-German benchmark, that represented roughly a 2 BLEU improvement over previous systems—a significant jump that helped establish the Transformer as the new state of the art.

Why BLEU Eventually Broke Down

BLEU assumes that good translations look similar.

Modern LLMs don't.

Consider these summaries.

Reference:

The meeting was postponed because the client requested additional documentation.

Output A:

The meeting was delayed after the client asked for more documents.

Output B:

Client requested more paperwork, so the meeting moved.

Humans would probably give both excellent scores.

BLEU prefers whichever shares more exact phrases.

Now imagine asking ChatGPT:

Explain recursion like I'm five.

There are hundreds of excellent answers.

BLEU expects one.

This becomes even worse for:

summarization
question answering
code explanations
dialogue
reasoning

As models became more creative, exact word overlap became a poor proxy for quality.

Researchers needed evaluation metrics that understood meaning rather than wording.

BLEURT: Teaching AI to Judge AI

Google Research introduced BLEURT in 2020.

Instead of counting words, BLEURT fine-tunes a pretrained Transformer to predict human evaluation scores.

Think of it as hiring a reviewer instead of using a spell checker.

During training, BLEURT sees:

candidate answer
reference answer
human quality score

After millions of examples, it learns patterns humans value:

preserved meaning
factual consistency
grammatical quality
fluency

An interesting engineering trick made BLEURT particularly effective.

Human-scored datasets are relatively small.

The researchers first generated large amounts of synthetically corrupted text—introducing deletions, substitutions, and paraphrases—to pretrain the evaluator before fine-tuning on expensive human judgments.

This significantly reduced the amount of labeled data needed.

COMET: Learning from Human Preferences

Around the same time, researchers at Unbabel developed COMET.

Like BLEURT, COMET uses a neural network.

But it has access to something BLEURT often doesn't:

original source sentence
reference translation
candidate translation

That additional context matters.

Suppose the French sentence is:

Il fait froid.

One candidate says:

It is cold.

Another says:

It is freezing.

Without seeing the original sentence, both might seem acceptable.

With the source available, COMET can better judge whether meaning has shifted.

Modern COMET models consistently show stronger correlation with professional human evaluators than BLEU across many translation benchmarks.

Today, COMET is frequently reported alongside BLEU in machine translation research.

The Economics of Evaluation

Training frontier models can cost millions of dollars.

Evaluation, surprisingly, can become expensive too.

Imagine comparing three model versions.

Each produces answers for:

10 benchmark datasets
5,000 prompts each

That's 150,000 outputs.

If humans spend only 15 seconds per evaluation:

150,000 × 15 seconds ≈ 625 hours

At $40/hour for expert annotators, that's $25,000 for a single evaluation round.

And that's before measuring agreement between multiple reviewers.

Automatic metrics dramatically reduce this cost.

A common workflow today looks like:

Evaluate every experiment automatically.
Keep only the best-performing candidates.
Send those few models to human reviewers.

The automatic metric acts as a high-quality filter rather than a replacement for humans.

Where We Are Today

No single metric captures quality perfectly.

BLEU remains valuable because:

it's simple
reproducible
historically comparable

BLEURT improves semantic understanding by learning from human judgments.

COMET goes even further by incorporating the original source sentence and demonstrating stronger agreement with professional evaluators.

For frontier LLMs, evaluation has become even broader.

Researchers increasingly combine:

learned metrics like COMET
human preference studies
benchmark suites
domain-specific tests
LLM-as-a-judge evaluations

The lesson is larger than machine translation.

As AI systems become more capable, evaluating them becomes an AI problem itself.

The future of language models may depend just as much on better judges as on better generators.

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub