DEV Community: Rijul Rajesh

Understanding Attention Mechanisms – Part 6: Final Step in Decoding

Rijul Rajesh — Sat, 04 Apr 2026 20:50:14 +0000

In the previous article, we obtained the initial output, but we didn’t receive the EOS token yet.

To get that, we need to unroll the embedding layer and the LSTMs in the decoder, and then feed the translated word “vamos” into the decoder’s unrolled embedding layer.

After that, we follow the same process as before.
But this time, we use the encoded values for “vamos”.

The second output from the decoder is EOS, which means we are done decoding.

When we add attention to an encoder-decoder model, the encoder mostly stays the same.

However, during each step of decoding, the model has access to the individual encodings for each input word.

We use similarity scores and the softmax function to determine what percentage of each encoded input word should be used to predict the next output word.

Now that we have added attention to the model, we won’t strictly need LSTMs in the same way.

We’ll explore this further when we move on to transformers.

Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Attention Mechanisms – Part 5: How Attention Produces the First Output

Rijul Rajesh — Wed, 01 Apr 2026 21:16:46 +0000

In the previous article, we stopped at using the softmax function to scale the scores.

When we scale the values for the first encoded word “Let’s” by 0.4:

And we scale the values for the second encoded word “go” by 0.6:

Finally, we add the scaled values together:

These sums combine the separate encodings for both input words, “Let’s” and “go”, based on their similarity to EOS.
These are the attention values for EOS.

Now, to determine our first output word, we need to:

Feed the attention values into a fully connected layer
Also include the encoding for EOS
Then pass everything through a softmax function

This allows the model to select the first output word, “vamos”.

But we haven’t reached EOS yet.
We will explore how to move further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Attention Mechanisms – Part 4: Turning Similarity Scores into Attention Weights

Rijul Rajesh — Tue, 31 Mar 2026 19:10:00 +0000

In the previous article, we just explored the benefits of using dot product instead of cosine similarity for attention.

Let's dig in further and see how our diagram looks after using the dot product.

We simply multiply each pair of output values and add them together:

(-0.76 × 0.91) + (0.75 × 0.38)

This gives us -0.41.

Likewise, we can compute a similarity score using the dot product between the second input word “go” and the EOS token.

This gives us 0.01.

Now that we have the scores, let’s see how to use them.

The score between “go” and EOS (0.01) is higher than the score between “let’s” and EOS (-0.41).

Since the score for “go” is higher, we want the encoding for “go” to have more influence on the first word that comes out of the decoder.

We can achieve this by passing the scores through the softmax function.

The softmax function gives us values between 0 and 1, and they all add up to 1.

So, we can think of the softmax function as a way to determine what percentage of each encoded input word we should use when decoding.

In this case:

We use 40% of the first encoded word (“let’s”)
We use 60% of the second encoded word (“go”)

This helps the decoder decide what the first translated word should be.

We will continue with the scaling part in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Cosine Similarity vs Dot Product in Attention Mechanisms

Rijul Rajesh — Mon, 30 Mar 2026 21:35:34 +0000

For comparing the hidden states between the encoder and decoder, we need a similarity score.

Two common approaches to calculate this are:

Cosine similarity
Dot product

Cosine Similarity

It performs a dot product on the vectors and then normalizes the result.

Example

Encoder output:

[-0.76, 0.75]

Decoder output:

[0.91, 0.38]

Cosine similarity ≈ -0.39

Close to 1 → very similar → strong attention
Close to 0 → not related
Negative → opposite → low attention

This is useful when:

Values can vary a lot in size
You want a consistent scale (-1 to 1)

The problem is that it’s a bit expensive. It requires extra calculations (division, square roots), and in attention we don’t always need that.

Dot Product

Dot product is much simpler. It does the following:

Multiply corresponding values
Add them up

Example

(-0.76 × 0.91) + (0.75 × 0.38) = -0.41

Dot product is preferred in attention because:

It’s fast
It’s simple
It gives good relative scores

Even if the numbers are not normalized, the model can still figure out:

Which words are more important
Which words to ignore

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Attention Mechanisms – Part 3: From Cosine Similarity to Dot Product

Rijul Rajesh — Sat, 28 Mar 2026 21:55:13 +0000

In the previous article, we explored the comparison between encoder and decoder outputs. In this article, we will be checking the math on how the calculation is done, and how it can be further simplified.

The output values for the two LSTM cells in the encoder for the word "Let’s" are -0.76 and 0.75.

The output values from the two LSTM cells in the decoder for the <EOS> token are 0.91 and 0.38.

We can represent this as:

A = Encoder
B = Decoder

Cell #1     Cell #2
-0.76       0.75
 0.91       0.38

Now, we plug these values into the cosine similarity equation.

This gives us a result of -0.39.

To simplify this further, a common approach is to compute only the numerator.

The denominator mainly scales the value between -1 and 1, so in some cases, we can ignore it for simplicity.

Since we are dealing with a fixed number of cells, this simplification works well. This is also known as the dot product.

When we calculate only the dot product, we get:

(-0.76 × 0.91) + (0.75 × 0.38) = -0.41

We will explore this further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Attention Mechanisms – Part 2: Comparing Encoder and Decoder Outputs

Rijul Rajesh — Fri, 27 Mar 2026 19:16:17 +0000

In the previous article, we explored the main idea of attention and the modifications it requires in an encoder–decoder model. Now, we will explore that idea further.

An encoder–decoder model can be as simple as an embedding layer attached to a single LSTM. If we want a more advanced encoder, we can add additional LSTM cells.

Now, we initialize the long-term and short-term memory in the LSTMs of the encoder with zeros.

If our input sentence, which we want to translate into Spanish, is "Let's go", we can feed a 1 for "Let's" into the embedding layer, unroll the network, and then feed a 1 for "go" into the embedding layer.

This process creates the context vector, which we use to initialize a separate set of LSTM cells in the decoder.

All of the input is compressed into the context vector.

But the idea of attention is that each step in the decoder should have direct access to the inputs.

So, let’s understand how attention connects the inputs to each step of the decoder.

In this example, the first thing attention does is determine how similar the outputs from the encoder LSTMs are to the outputs from the decoder LSTMs at each step.

In other words, we compute a similarity score between the LSTM outputs (the short-term memory or hidden states) from the encoder and the decoder.

For instance, we calculate a similarity score between:

The LSTM output from the first step in the encoder, and
The LSTM output from the first step in the decoder

We also calculate a similarity score between:

The LSTM output from the second step in the encoder, and
The LSTM output from the first step in the decoder

There are various ways to calculate this similarity.

One simple method is cosine similarity, which measures how similar two sequences of numbers (representing words) are.

We will explore this further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Attention Mechanisms – Part 1: Why Long Sentences Break Encoder–Decoders

Rijul Rajesh — Thu, 26 Mar 2026 19:48:40 +0000

In the previous articles, we understood Seq2Seq models. Now, on the path toward transformers, we need to understand one more concept before reaching there: Attention.

The encoder in a basic encoder–decoder, by unrolling the LSTMs, compresses the entire input sentence into a single context vector.

This works fine for short phrases like "Let's go".

But if we had a bigger input vocabulary with thousands of words, then we could input longer and more complicated sentences, like "Don't eat the delicious-looking and smelling pasta".

For longer phrases, even with LSTMs, words that are input early on can be forgotten.

In this case, if we forget the first word "Don't", then it becomes:

"eat the delicious-looking and smelling pasta"

So, sometimes it is important to remember the first word.

Basic RNNs had problems with long-term memory because they ran both long- and short-term information through a single path.

The main idea of Long Short-Term Memory (LSTM) units is that they solve this problem by providing separate paths for long- and short-term memory.

Even with separate paths, if we have a lot of data, both paths still have to carry a large amount of information.

So, a word at the start of a long phrase, like "Don't", can still get lost.

So, the main idea of attention is to add multiple new paths from the encoder to the decoder.

There is one path per input value, so each step of the decoder can directly access the relevant input values.

We will explore more about attention in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Seq2Seq Neural Networks – Part 8: When Does the Decoder Stop?

Rijul Rajesh — Wed, 25 Mar 2026 22:23:53 +0000

In the previous article, we saw the translation being done.

But there is an issue.

The decoder does not stop until it outputs an EOS token.

So, we plug the word "Vamos" into the decoder’s unrolled embedding layer and unroll the two LSTM cells in each layer.

Then, we run the output values (short-term memory or hidden states) into the same fully connected layer.

The next predicted token is EOS.

How the Decoder Works

So now, this means we translated the English sentence "let’s go" into the correct Spanish sentence.

For the decoder, the context vector, which is created by both layers of encoder unrolled LSTM cells, is used to initialize the LSTMs in the decoder.

The input to the LSTMs comes from the output word embedding layer, which starts with EOS. After that, it uses whatever word was predicted by the output layer.

In practice, the decoder keeps predicting words until it predicts the EOS token or reaches some maximum output length.

All these weights and biases are trained using backpropagation.

When training an encoder-decoder model, instead of using the predicted token as input to the decoder LSTMs, we use the known correct token. This is known as teacher forcing (explained in this article).

What’s Next

That’s it for sequence-to-sequence neural networks.

In the next article, we will continue with the attention mechanism for neural networks.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Teacher Forcing in Seq2Seq Models

Rijul Rajesh — Mon, 23 Mar 2026 19:35:35 +0000

When we learn about seq2seq neural networks, there is a term we should know called Teacher Forcing.

When we train a seq2seq model, the decoder generates one token at a time, building the output sequence step by step.

At each step, it needs a previous token as input to predict the next one.

So in this case, we should think about what to provide as the previous token, since this choice directly affects how well the model learns.

Without teacher forcing, the model uses its own previous prediction as input.

Suppose the target is "I am learning"

Predict "I" ✅
Uses "I" and predicts "Is" ❌
Uses "is", and everything goes off track

Here, one small mistake early causes all the following predictions to be wrong.

So, if there is one mistake early, the mistakes keep compounding step by step.

This makes training slow, unstable, and harder for the model to learn correct sequences.

With teacher forcing, instead of using the model’s prediction, we feed the correct token from the dataset at every step.

So even if the model makes a mistake at one step, we replace it with the correct token before moving forward.

This ensures that the model always sees the right context while learning.

Even if the model makes a mistake, we do not let that mistake affect future steps during training.

This makes training faster, more stable, and easier for the model to converge.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Seq2Seq Neural Networks – Part 7: Generating the Output with Softmax

Rijul Rajesh — Sat, 21 Mar 2026 20:26:23 +0000

In the previous article, we were transforming the outputs to the fully connected layer.

A fully connected layer is just another name for a basic neural network.

This fully connected layer has two inputs, corresponding to the two values that come from the top layer of LSTM cells.

It has four outputs, one for each token in the Spanish vocabulary.

In between, we have connections between each input and output, each with their own weights and biases.

Then, we run the output of the fully connected layer through a Softmax function to select the output word.

Now, going back to the full Encoder–Decoder model:

We can see that the output from the Softmax function is “Vamos,” which is the Spanish translation for “Let’s go.”

We will continue further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Seq2Seq Neural Networks – Part 6: Decoder Outputs and the Fully Connected Layer

Rijul Rajesh — Fri, 20 Mar 2026 19:45:53 +0000

In the previous article, we were looking at the embedding values in the encoder and the decoder.

As you can see, they have different input words and symbols (tokens) and different weights, which result in different embedding values for each token.

Because we have just finished encoding the English sentence “Let’s go,” the decoder starts with the embedding values for the token.

The decoder then performs computations using two layers of LSTMs, each with two LSTM cells.

The output values (the short-term memories, or hidden states) from the top layer of LSTM cells are then transformed using additional weights and biases in what is called a fully connected layer.

We will explore this further in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here

Understanding Seq2Seq Neural Networks – Part 5: Decoding the Context Vector

Rijul Rajesh — Thu, 19 Mar 2026 03:12:22 +0000

In the previous article, we stopped at the concept of the context vector.

In this article, we will start by decoding the context vector.

Connecting the Decoder

The first thing we need to do is connect the long-term and short-term memories (the cell states and hidden states) that form the context vector to a new set of LSTMs.

Just like the encoder, the decoder will also have two layers, and each layer will have two LSTM cells.

The LSTMs in the decoder are different from the ones in the encoder and have their own separate weights and biases.

Using the Context Vector

The context vector is used to initialize the long-term and short-term memories (the cell states and hidden states) in the LSTMs of the decoder.

This is important because it allows the decoder to start with the information learned from the input sentence.

Goal of the Decoder

The ultimate goal of the decoder is to convert the context vector into the output sentence.

In simple terms, the encoder understands the input, and the decoder generates the output based on that understanding.

Decoder Inputs

Just like in the encoder, the input to the LSTM cells in the first layer comes from an embedding layer.

However, in this case, the embedding layer creates embedding values for Spanish words, such as:

ir
vamos
y
(End of Sentence symbol)

Each of these words is treated as a token, and the embedding layer converts them into numbers that the neural network can process.

We will explore the details of how the decoder generates the output sentence in the next article.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here