Understanding Transformers Part 6: Calculating Similarity Between Queries and Keys

#machinelearning #ai

In the previous article, we explored the concepts of Queries and Keys. Now we will see how to calculate the similiarities

Calculating Similarity Using Dot Product

One way to calculate the similarity between a query and the keys is by using the dot product.

Query vs Key for “Let’s”

Let’s first compute the dot product between the query and key for the word “Let’s”.

We multiply each pair of values and then add the results. This gives us a similarity score of 11.7.

Query for “Let’s” vs Key for “go”

Now, let’s compute the dot product between the query for “Let’s” and the key for “go”.

This gives us a similarity score of -2.6.

Understanding the Result

The similarity score for “Let’s” with itself (11.7) is much higher than its similarity with “go” (-2.6).

This tells us that “Let’s” is much more similar to itself than it is to “go”.

As a result, when encoding the word “Let’s”, it should be influenced more by itself and less by “go”.

What’s Next?

To turn these similarity scores into meaningful weights, we pass them through a softmax function.

We will explore how softmax works in the next article.

Looking for an easier way to install tools, libraries, or entire repositories?
Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run: