DEV Community: Rijul Rajesh

Understanding Multi-Head Attention in Transformers

Rijul Rajesh — Sun, 03 May 2026 20:08:43 +0000

Self-attention already helps a transformer understand relationships between words using Query, Key, and Value. But there’s a problem.

One attention mechanism usually ends up focusing on a limited kind of relationship at a time.

Language doesn’t work like that. A sentence can have structure, meaning, and long-range links all at once.

That’s why transformers use multi-head attention.

What happens in multi-head attention

Instead of doing attention once, the model does it multiple times in parallel.

Each run is called a head, and each head has its own learned weights for Query, Key, and Value.

So every head looks at the same sentence, but in its own way.

How it flows

The input embeddings are first prepared
They are split into multiple heads using linear projections
Each head runs its own self-attention
Each head produces its own output
All outputs are joined back together
A final layer mixes them into one result

Why this works better compared to previous approach

Different heads naturally pick up different things:

word order and grammar
nearby word relationships
long-distance links
meaning-based connections

So instead of forcing one attention mechanism to do everything, the model spreads the job across multiple perspectives.

One head is like reading a sentence with one focus.

Multiple heads is like reading it several times, each time noticing something different, then combining those notes.

Multi-head attention doesn’t change the idea of self-attention. It just runs it multiple times in parallel so the model can understand language from different angles at once.

Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 17: Generating the Output Word

Rijul Rajesh — Fri, 01 May 2026 20:53:08 +0000

In the previous article, we set up the residual connections to get the final output values from the decoder.

In this article, we begin by passing these two output values through a fully connected layer.

This layer has:

One input for each value representing the current token (in this case, 2 inputs)
One output for each word in the output vocabulary

Since our vocabulary has 4 tokens, this gives us 4 output values.

Selecting the Output Word

Next, we pass these 4 output values through a softmax function.

This allows us to select the most likely output word, which in this case is “vamos”.

So far, the translation is correct. However, the process does not stop here.

Continuing the Decoding Process

The decoder continues generating words until it produces an token, which indicates the end of the sentence.

To generate the next word, we feed the predicted word back into the decoder.

We will explore this step in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers – Part 16: Preparing for Output Prediction with Residual Connections

Rijul Rajesh — Wed, 29 Apr 2026 21:26:20 +0000

In the previous article, we handled values in encoder-decoder attention, now we will simplify the diagram a bit add another set of residual connections.

This allows the encoder–decoder attention to focus on the relationships between the output words and the input words, without needing to preserve the self-attention and positional encoding from earlier.

Lastly, we need a way to take these two values that represent the token in the decoder and select one of the four output tokens: ir, vamos, y, or <EOS>.

To do this, we pass these two values through a fully connected layer.

We will explore this further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 15: Scaling and Combining Values in Encoder–Decoder Attention

Rijul Rajesh — Tue, 28 Apr 2026 19:23:58 +0000

In the previous article, we gained an understanding how much each input word contributes, in this article we will start to compute the value vectors for each input word and combine them accordingly.

We scale those values using the Softmax percentages, and add the scaled values together to obtain the encoder–decoder attention values.

The sets of weights used to calculate the queries, keys, and values for encoder–decoder attention are different from the sets of weights used in self-attention.

Just like in self-attention, these sets of weights are copied and reused for each word, which allows the model to be flexible with different input and output lengths.

We can also stack encoder–decoder attention layers, just like we do with self-attention, to better handle more complex phrases.

We will continue with more details in the next article

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 14: Calculating Encoder–Decoder Attention

Rijul Rajesh — Sun, 26 Apr 2026 19:34:55 +0000

In the previous article, we just began introducing the concept of encoder-decoder attention.

Now lets start digging into the details.

Encoder–Decoder Attention in Action

Just like in self-attention, we start by creating query values.

In this case, we create two values to represent the query for the <EOS> token in the decoder.

Next, we create key values for each word in the encoder output.

Calculating Similarity

Now, we calculate the similarity between the <EOS> token in the decoder and each word in the encoder.

This is done using the dot product.

Applying Softmax

We then pass these similarity scores through a softmax function:

This gives us weights that determine how much attention the decoder should pay to each input word.

In this example:

The first input word gets 100% attention
The second word gets 0% attention

This means the decoder will focus entirely on the first input word when deciding the first translated word.

What’s Next?

Now that we know how much each input word contributes, the next step is to compute the value vectors for each input word and combine them accordingly.

We will explore this in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 13: Introducing Encoder–Decoder Attention

Rijul Rajesh — Sat, 25 Apr 2026 19:40:51 +0000

In the previous article, we built up the decoder layers and stopped at the relationship between input and output sentence.

So this brings us to the concept of Encoder-Decoder Attention

Why Encoder–Decoder Attention Matters

Consider the input sentence:

“Don’t eat the delicious looking and smelling pizza.”

When translating this sentence, it is very important to keep track of the word “Don’t”.

If the translation ignores this word, we might end up with:

“Eat the delicious looking and smelling pizza.”

These two sentences have completely opposite meanings.

Key Idea

Because of this, the decoder must pay close attention to the important words in the input.

This is where encoder–decoder attention comes in.

It allows the decoder to focus on the most relevant parts of the input sentence while generating the output.

In simple terms, encoder–decoder attention helps the decoder keep track of significant words in the input.

Updated Transformer Structure

With this idea, our current encoder–decoder structure looks like this:

We will build on this and explore the details in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 12: Building the Decoder Layers

Rijul Rajesh — Thu, 23 Apr 2026 19:23:30 +0000

In the previous article, we just began with the concept of decoders in a transformer.

Now we will start adding the positional encoding.

Adding Positional Encoding in the Decoder

Now, for the decoder, let’s add positional encoding.

Just like before, we use the same sine and cosine curves to get positional values based on the embedding positions.

These are the same curves that were used earlier when encoding the input.

Applying Positional Values

Since the <EOS> token is in the first position and has two embedding values, we take the corresponding positional values from the curves.

For the first embedding, the value is 0
For the second embedding, the value is 1

Now, we add these positional values to the embedding:

As a result, we get 2.70 and -0.34, which represent the <EOS> token after adding positional encoding.

Adding Self-Attention

Next, we add the self-attention layer so the decoder can keep track of relationships between output words.

The self-attention values for the <EOS> token are -2.8 and -2.3.

Note that the weights used in the decoder’s self-attention (for queries, keys, and values) are different from those used in the encoder.

Adding Residual Connections

Now, we add residual connections, just like we did in the encoder.

What’s Next?

So far, we have seen how self-attention helps the transformer understand relationships within the output sentence.

However, for tasks like translation, the model also needs to understand relationships between the input sentence and the output sentence.

We will explore this in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 11: How Decoding Begins

Rijul Rajesh — Wed, 22 Apr 2026 19:31:56 +0000

In the previous article we wrapped up the encoder part, In this article, we will start building the second part of the transformer: the decoder.

Just like the encoder, the decoder also begins with word embeddings.

However, this time the embeddings are created for the output vocabulary, which consists of Spanish words such as:

ir
vamos
y
<EOS> token

Starting the Decoding Process

To begin decoding, we use the <EOS> token as the input.

This is a common way to initialize the decoding process for an encoded sentence.

In some cases, people use a <SOS> (Start of Sentence) token instead.

Creating the Initial Input

We represent the <EOS> token as a vector by assigning:

1 to <EOS>
0 to all other words in the vocabulary

From this we can see that 2.70 and -1.34 are the numbers that represent the value for the EOS token.

Now that we have the initial input for the decoder, the next step is to add positional encoding.

We will explore this in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 10: Final Step in Encoding

Rijul Rajesh — Tue, 21 Apr 2026 19:36:28 +0000

In the previous article, we explored the use of self-attention layers, now we will dive into the final step of encoding and start moving into decoders

As the final step, we take the positional encoded values and add them to the self-attention values.

These connections are called residual connections. They make it easier to train complex neural networks by allowing the self-attention layer to focus on learning relationships between words, without needing to preserve the original word embedding and positional information.

At this point, we have everything needed to encode the input for this simple transformer.

These four components work together to convert words into meaningful numerical representations:

Word embedding
Positional encoding
Self-attention
Residual connections

Now that we have encoded the English input phrase “Let’s go”, the next step is to decode it into Spanish.

To do this, we need to build a decoder, which we will explore in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 9: Stacking Self-Attention Layers

Rijul Rajesh — Fri, 17 Apr 2026 20:50:25 +0000

In the previous article, we explored how the weights are shared in self-attention.

Now we will see why we have these self-attention values instead of the initial positional encoding values.

Using Self-Attention Values

We now use the self-attention values instead of the original positional encoded values.

This is because the self-attention values for each word include information from all the other words in the sentence. This helps give each word context.

It also helps establish how each word in the input is related to the others.

If we think of this unit, along with its weights for calculating queries, keys, and values, as a self-attention cell, then we can extend this idea further.

To correctly capture relationships in more complex sentences and paragraphs, we can stack multiple self-attention cells, each with its own set of weights. These layers are applied to the position-encoded values of each word, allowing the model to learn different types of relationships.

Going back to our example, there is one more step required to fully encode the input. We will explore that in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 8: Shared Weights in Self-Attention

Rijul Rajesh — Thu, 16 Apr 2026 21:08:46 +0000

In the previous article, we started calculating the self-attention values.

Let’s now calculate the self-attention values for the word “go”.

We do not need to recalculate the keys and values.

Instead, we only need to create the query that represents the word “go”, and then perform the same calculations as before.

After completing the calculations, we get the self-attention values for “go” as:

2.5 and -2.1

Key Observations About Self-Attention

The weights used to calculate queries are the same for both “Let’s” and “go”.
- This means that regardless of the number of words, we use one shared set of weights.
- Similarly, the same sets of weights are reused to calculate keys and values for every input word.
- No matter how many words are given as input, the transformer reuses the same weights for queries, keys, and values.
We do not need to compute queries, keys, and values sequentially.
- All of them can be computed at the same time.
- This allows transformers to take advantage of parallel computation, making them very efficient.

We will continue building our transformer step by step in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Transformers Part 7: From Similarity Scores to Self-Attention

Rijul Rajesh — Wed, 15 Apr 2026 02:24:37 +0000

In the previous article, we calculated the similarities between Queries and Keys.

We can use the output of the softmax function to determine how much each input word should contribute when encoding the word “Let’s”.

Interpreting the Weights

In this case, “Let’s” is much more similar to itself than to “go”.

So after applying softmax:

“Let’s” gets a weight close to 1 (100%)
“go” gets a weight close to 0 (0%)

This means:

“Let’s” contributes almost entirely to its own encoding
“go” contributes very little

Creating Value Representations

To apply these weights, we create another set of values for each word.

First, we create two values to represent “Let’s”

Then, we scale these values by 1 (since its weight is 100%)
Next, we create two values to represent “go”

These values are scaled by 0 (since its weight is 0%)

Combining the Values

Finally, we add the scaled values together:

The result is a new set of values that represent the word “Let’s”, now enriched by its relationship with all input words.

These final values are called the self-attention values for “Let’s”.

They combine information from all words in the sentence, weighted by how relevant each word is to “Let’s”.

We can now repeat the same process for the word “go”, which we will explore in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here