Priyesh Dave

Posted on Sep 26

Day 26: "Attention Is All You Need": The Paper That Changed Everything

#ai #nlp #genai #100daysofcode

Setting the Stage: Why Sequence Matters in Language Tasks

Understanding sequence is essential for understanding language. The sentence “The cat sat on the mat” has a very different meaning from “The mat sat on the cat.” Meaning comes from both the words themselves and the order in which they appear.

Older natural language processing (NLP) methods, like bag-of-words models, ignored order. They treated text as a pile of words and lost all sense of sequence. These methods worked for tasks like spam detection, where the presence of certain words is enough. But they failed completely when order mattered—such as in translation, conversation, or even basic grammar.

Teaching machines to process sequences of words like humans has always been a challenge. Attention mechanisms changed the field by offering a new, more flexible way to tackle this problem.

The Old Guard: RNNs and LSTMs

Before Transformers, neural networks for sequences mainly used Recurrent Neural Networks (RNNs) or a special variant called Long Short-Term Memory (LSTM) networks.

An RNN looks at one word at a time. At each step, it considers both the current word and its hidden state—a memory of what it has seen so far. It updates this hidden state as it moves through the sentence word by word.

LSTM networks were designed to help RNNs handle long sequences. An LSTM cell contains small internal gates that decide, at every step, what information to remember or forget. This helps LSTMs hold on to important details over longer stretches of a sentence.

With RNNs and LSTMs, computers could generate sentences, translate languages, and write captions—tasks where context matters. But these models had serious limitations.

The Bottlenecks: Problems with RNNs and LSTMs

RNNs and LSTMs have three fundamental weaknesses.

1. No Parallelism:

They process words one at a time, updating their memory at each step. Imagine reading a book aloud, slowly, word by word. You can’t speed this up by looking at multiple words at once. Training on long texts is slow.

2. Vanishing Gradients:

During training, information from the start of a sentence should influence predictions at the end. But in practice, as the network updates through each word, the useful signal fades (“vanishes”). This is like trying to follow a message whispered through a crowd—by the last person, most of it is lost.

3. Trouble with Long-Range Dependencies:

Even LSTMs, despite their design, struggle to connect information from the beginning of a long sentence to the end. If a 40-word sentence needs the first word to affect the last, the model rarely succeeds.

These issues limited the power and efficiency of sequence models.

Enter Attention: The Breakthrough Idea

The idea of attention transformed NLP in 2017.

Attention allows a model to focus on the most relevant words in a sequence. Instead of processing left-to-right and hoping important information survives, attention computes—at every step—how significant every other word is to the current prediction.

Picture reading a sentence, and for each word, glancing back at earlier words to decide which ones matter most. Attention mechanisms automate this process.

Practically, the network calculates an attention score between a word and every other word in the input. These scores become weights. Using the weights, the model creates a context vector—a blend of all word vectors, giving more emphasis to the words that are most important for this prediction.

"Attention Is All You Need": The Transformer's Core Idea

In 2017, Vaswani et al. published "Attention Is All You Need". Their claim: you don’t need RNNs (step-by-step memory) or CNNs (windowed filters) to process sequences. Attention alone is enough.

The Transformer architecture uses only attention layers plus some simple feed-forward neural networks. There’s no stepwise memory, and no convolutions over word windows.

Instead of processing each word sequentially, the Transformer computes attention across the entire sentence at once—or at least the parts that matter. It’s like taking in a whole sentence at a glance and immediately knowing which words depend on each other.

This new approach was simpler, easier to parallelize, and produced better results.

What Changed: Speed, Scale, Results

Transformers took over NLP for three main reasons:

1. Parallelization = Speed:

Because the Transformer doesn’t process words one at a time, it can work on every word in a sentence simultaneously. This makes training much faster, especially on GPUs. Tasks that used to take days could finish in hours.

2. Direct Access to Context:

With attention, the model can connect any two words, no matter how far apart. The network isn’t limited by how many steps separate words.

3. Dramatically Better Outcomes:

Attention made translation, summarization, and many other NLP tasks much better. Transformers are the basis for state-of-the-art models like BERT (for text understanding) and GPT (for text generation).

Practically, you might see a speedup from days to hours when moving from an RNN to a Transformer—because everything is processed in parallel instead of word-by-word.

Anatomy of a Transformer: The Key Parts

A Transformer network has a few critical parts:

Input Embeddings:

Each word is represented as a vector of numbers, called an embedding. This vector encodes the word’s meaning in a way the model can use.
Positional Encoding:

Attention itself doesn’t know word order. To fix that, we add a vector to each word’s embedding that marks its position in the sentence—like giving each word a street address.
Self-Attention:

For every word, the network calculates attention scores for every other word, then blends their embeddings together to get a context-aware vector for each position. It’s like listening to everyone in a meeting before deciding what to say.
Feed-Forward Layers:

After self-attention, each word’s vector is processed by a small neural network. This adds depth and complexity to the representation.

These steps are stacked multiple times, allowing the model to build up an understanding layer by layer. But the core remains the same.

A Minimal Example: Computing Attention Scores

Let’s walk through the basic steps of attention in a sequence of three words. Imagine each word is already represented as a 2-D vector. We’ll compute scaled dot-product attention: first, we calculate attention weights, then use them to blend the vectors.

import numpy as np

# Example 'word' vectors (embeddings): 3 words, each with 2 features
word_vectors = np.array([
    [1.0, 0.0],  # word 1
    [0.0, 1.0],  # word 2
    [1.0, 1.0]   # word 3
])

# In scaled dot-product attention, you have 'queries', 'keys', and 'values'
# For simplicity, we'll just use the word_vectors for all three
Q = word_vectors
K = word_vectors
V = word_vectors

# Compute raw attention scores: Q @ K.T
attention_scores = np.dot(Q, K.T)  # shape (3, 3)

# Scale scores (normally by sqrt of vector size)
d_k = Q.shape[1]
scaled_scores = attention_scores / np.sqrt(d_k)

# Softmax to get attention weights for each word
def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

attention_weights = softmax(scaled_scores)  # shape (3, 3)

# Weighted sum: each row is the new context vector for a word
context_vectors = np.dot(attention_weights, V)

print("Attention Weights:\n", attention_weights)
print("Context Vectors:\n", context_vectors)

This code demonstrates how to calculate attention scores between words, convert those scores into weights, and use the weights to combine the word vectors. The result is a new vector for each word that emphasizes the most important connections—no matter where those words are in the sentence.

That mechanism is the core idea that powered the leap from older sequence models to Transformers.

Key Takeaways

Word order is crucial for language meaning; older models often ignored this.
RNNs and LSTMs process text sequentially, causing slow training and context loss.
Attention mechanisms allow models to focus on relevant words, regardless of position.
Transformers replace RNNs/LSTMs by using self-attention and parallel processing.
Attention enables faster training and better performance in language tasks.

Try It Yourself

Take three simple word vectors of your choice (for example, [1,0], [0,1], and [1,1]), and calculate the scaled dot-product attention scores for one word acting as a query. Walk through the steps: compute the raw scores, scale by the square root of the dimensionality, apply softmax to get attention weights, and use them to find the context vector. You can do this calculation either by hand or with a few lines of code, as shown in the lesson.

Further Resources

Coming up on Day 27: Self-Attention Explained From Scratch

DEV Community