How Transformers Actually Work (Attention, Explained With Real Code)

Written by Syed Muhammad Ali Raza

New arc starting here. The last series was entirely about building things on top of LLMs, RAG, agents, evals, production, all of it treating the model itself as a black box you call through an API. Fair approach for building products, genuinely the right level of abstraction for most of that work. But I kept getting the same question from people reading along, okay but what's actually happening inside that black box.

This arc is the answer. We're going under the hood, starting with the single idea that made modern LLMs possible in the first place, attention, the core mechanism inside the transformer architecture that every model in this entire series has been running on the whole time.

A real life example before any math

Read this sentence, "the trophy didn't fit in the suitcase because it was too big." What does "it" refer to, the trophy or the suitcase?

You answered that instantly, without even noticing you did any work. Your brain looked at the word "it," then scanned back across the sentence, weighing every other word by how relevant it was to figuring out what "it" meant, and landed on "trophy" because "big" makes sense paired with a trophy not fitting, while barely even considering words like "didn't" or "the," which are grammatically necessary but carry zero relevance to resolving what "it" means.

That's genuinely the entire intuition behind attention. For every word in a sentence, the model asks "which other words in this sentence actually matter for understanding this one," and assigns each other word a relevance score, paying more attention to the ones that matter and less to the ones that don't. Change one word, "the trophy didn't fit in the suitcase because it was too small," and suddenly "it" points to the suitcase instead, same sentence structure, completely different answer, because the actual meaning shifted. A model that couldn't do this, that just processed words in isolation without weighing how they relate to each other, would have no way to get this right.

Why this was such a big deal when it showed up

Before attention, the dominant way of processing sequences of text was reading word by word, in order, carrying forward a running summary as you went, similar to reading a sentence left to right and trying to remember everything important as a single running mental note. That approach genuinely struggles with long sentences, by the time you reach word thirty, the running summary has been squeezed and overwritten so many times that details from word two are often just gone.

The actual breakthrough, from a 2017 paper literally titled "Attention Is All You Need," was realizing you don't need that sequential running summary at all. Every word can directly look at every other word in the sentence at once, regardless of distance, and decide for itself how relevant each one is. Word two hundred can directly attend to word one, no information bottleneck in between. That direct, all at once connectivity is genuinely why transformers scaled so well and became the architecture behind essentially every major LLM since.

The actual mechanism, queries, keys, and values

Here's where it gets concrete. For every word, sorry, every token, since we covered back in an earlier article that text gets broken into tokens, not always whole words, the model creates three different vectors from it, a query, a key, and a value.

Think of it like a library search. Your query is what you're looking for. Every book on the shelf has a key, a short label describing what that book is about. You compare your query against every book's key to figure out how relevant each book is to what you want, and the books with the best matching keys are the ones whose actual content, the value, you pay the most attention to.

In a transformer, every single token does this simultaneously, for every other token in the sequence. Token five generates a query asking "what's relevant to me," compares that query against the keys of every other token including itself, gets a relevance score for each one, and then combines the values of all tokens, weighted by those relevance scores, into a new, context aware representation of itself.

Let's actually build this with code

I'll implement genuine, simplified self attention from scratch using nothing but numpy, so you can see the actual math instead of just the concept. This is deliberately the bare mechanism, real transformers add more machinery around this, but this is the actual core computation running inside every one of them.

Step 1, represent a tiny sequence as vectors

import numpy as np

# pretend these are embeddings for the words "the", "cat", "sat"
# in reality these come from a learned embedding table, here
# we're just making up small numbers to see the mechanism clearly
np.random.seed(42)
embeddings = np.random.randn(3, 4)  # 3 tokens, 4 dimensional embeddings each

print("Token embeddings:")
print(embeddings)

Step 2, generate query, key, and value vectors

In a real model these come from learned weight matrices, trained through the whole process we'll get to later in this arc. Here, we'll use small random matrices just to see the actual mechanics work.

d_model = 4  # embedding dimension

W_query = np.random.randn(d_model, d_model) * 0.5
W_key = np.random.randn(d_model, d_model) * 0.5
W_value = np.random.randn(d_model, d_model) * 0.5

queries = embeddings @ W_query
keys = embeddings @ W_key
values = embeddings @ W_value

print("Queries shape:", queries.shape)  # 3 tokens, each with a query vector

Step 3, compute attention scores, how relevant is every token to every other token

# dot product between every query and every key gives a relevance score
attention_scores = queries @ keys.T
print("Raw attention scores:")
print(attention_scores)

# scale down, this keeps the numbers from getting too large as
# dimensions grow, a detail the original paper found mattered
attention_scores = attention_scores / np.sqrt(d_model)

# turn scores into proper weights that sum to 1 per row, using softmax
def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

attention_weights = softmax(attention_scores)
print("\nAttention weights, each row sums to 1:")
print(attention_weights)

That attention_weights matrix is genuinely the heart of the entire mechanism. Row one tells you how much token one, "the," attends to itself, to "cat," and to "sat," row two tells you the same for "cat," and so on. These aren't fixed, they're computed fresh for every single input, based on the actual content of that specific sequence.

Step 4, combine values using those weights

# each token's new representation is a weighted blend of every
# token's value vector, weighted by how much attention it paid
output = attention_weights @ values

print("\nNew context aware representations:")
print(output)

That output is the actual result of self attention, each token's original embedding has now been blended with information from every other relevant token, weighted by how much it actually mattered. This is genuinely what happens, at a massive scale with far more dimensions and far more tokens, inside every layer of every transformer based model you've used in this entire series.

Why "multi-head" attention, and why bother

Real transformers don't do this once, they do it several times in parallel, called multiple attention heads, each with its own separate query, key, and value weight matrices, each learning to focus on a different kind of relationship.

def single_attention_head(embeddings, d_model):
    W_q = np.random.randn(d_model, d_model) * 0.5
    W_k = np.random.randn(d_model, d_model) * 0.5
    W_v = np.random.randn(d_model, d_model) * 0.5

    q = embeddings @ W_q
    k = embeddings @ W_k
    v = embeddings @ W_v

    scores = (q @ k.T) / np.sqrt(d_model)
    weights = softmax(scores)
    return weights @ v


def multi_head_attention(embeddings, num_heads=3):
    d_model = embeddings.shape[1]
    head_outputs = [single_attention_head(embeddings, d_model) for _ in range(num_heads)]
    # in a real model these get concatenated and projected back down,
    # here we're just showing that each head produces its own view
    return head_outputs


heads = multi_head_attention(embeddings, num_heads=3)
for i, head_output in enumerate(heads):
    print(f"Head {i} output:\n{head_output}\n")

The intuition, one head might genuinely learn to focus on grammatical relationships, like matching pronouns to the nouns they refer to. Another might focus on relationships between adjacent words. Another might specialize in something else the training process finds useful. Nobody explicitly programs what each head focuses on, it emerges from training, which is exactly what the next article in this arc is going to dig into.

Where this fits into the full transformer

Attention is the standout idea, but a real transformer layer wraps a few more things around it. After self attention, each token's new representation gets passed through a small feedforward neural network, applied identically to every token, which adds additional processing capacity beyond just mixing information between tokens. Layer normalization keeps the numbers in a stable, well behaved range as they flow through many stacked layers. And residual connections, adding a layer's input back onto its output, help information and gradients flow cleanly through very deep stacks of these layers without vanishing.

Stack a few dozen of these layers, attention, then feedforward, repeated, each one refining the token representations a bit further based on what came before, and you've got the actual architecture behind GPT, Claude, and essentially every major LLM built since 2017.

The honest caveat

Real production transformers have a lot more engineering detail than this simplified version, positional encoding so the model knows token order since attention itself has no inherent sense of sequence, causal masking so a token generating text can't peek at future tokens it hasn't generated yet, and various optimizations for actually running this efficiently at scale. But the core computation, query, key, value, weighted combination, is genuinely this, at massive scale, not some fundamentally different secret mechanism hiding underneath.

What's next in this arc

This article covered the mechanism that lets a model relate tokens to each other within a single forward pass. It didn't cover how a model actually learns the weight matrices we just randomly initialized here, or what's actually happening during training that turns random numbers into something that produces coherent language. That's genuinely the next piece, and it's where this arc goes next.

---

If you run this code yourself and print out the attention weights for a sentence you actually care about, I'd genuinely like to hear what you notice, that's usually the fastest way to make this concept stop feeling abstract.