Positional Encoding: Adding Sequence Awareness to Transformers

Introduction

As part of my 75-day learning journey into deep learning and NLP, I’m exploring one of the fundamental components that make Transformers so effective: Positional Encoding. In the Transformer architecture, input data is processed in parallel, unlike traditional Recurrent Neural Networks (RNNs) that process sequentially. This parallelism is one of the reasons Transformers achieve state-of-the-art performance, but it introduces a challenge: how to give the model a sense of order or position in the sequence? The answer lies in Positional Encoding.

In this article, we’ll explore what positional encoding is, why it is essential, and how to implement it in Python.

What is Positional Encoding?

Positional Encoding is a technique used in the Transformer architecture to inject information about the position of a token (word) in a sequence. Since Transformers process input sequences in parallel, they need a way to maintain the order of the tokens. The positional encoding assigns each token a unique vector representing its position.

These encodings are added to the input embeddings so that the model can differentiate between tokens based not only on their content but also on their position in the sequence.

Why is Positional Encoding Important?

Without positional encoding, the Transformer would have no way to understand the order of tokens. For example, consider the sentences:

• "The cat chased the mouse."
• "The mouse chased the cat."

Even though both sentences contain the same words, the meaning changes entirely based on the order of the words. Without positional encoding, a Transformer would treat the word "cat" identically in both sentences, even though in one sentence the cat is the subject and in the other, it is the object. Positional encodings allow the model to understand this difference in word order and thus capture the correct meaning of each sentence.

Mathematical Representation of Positional Encoding

The most common positional encoding used in Transformers involves using sine and cosine functions of different frequencies. This representation ensures that the positional encodings for different positions are unique and easily differentiable by the model.

For a given position pos and dimension i, the encoding is calculated as:

• For even dimensions (2i): PE(pos, 2i) = sin(pos / 10000^(2i / d))
• For odd dimensions (2i+1): PE(pos, 2i+1) = cos(pos / 10000^(2i / d))

Where:

• pos is the position in the sequence.
• i is the dimension.
• d is the dimensionality of the model (e.g., the size of the embedding vector).

This ensures that positions that are close to each other have similar encodings, but as the positions diverge, their encodings become more distinct.

How to Implement Positional Encoding in Python

Now, let's implement the positional encoding in Python. We'll use NumPy to compute the sine and cosine functions for the positional encodings.

import numpy as np
import matplotlib.pyplot as plt

def positional_encoding(max_position, d_model):
    """
    Compute the positional encoding for a sequence.

    Parameters:
    max_position (int): Maximum length of the sequence.
    d_model (int): Dimensionality of the embedding vector.

    Returns:
    numpy.ndarray: Positional encoding matrix of shape (max_position, d_model)
    """
    pos_enc = np.zeros((max_position, d_model))
    for pos in range(max_position):
        for i in range(0, d_model, 2):
            pos_enc[pos, i] = np.sin(pos / (10000 ** (2 * i / d_model)))
            if i + 1 < d_model:
                pos_enc[pos, i + 1] = np.cos(pos / (10000 ** (2 * i / d_model)))

    return pos_enc

# Example usage
max_position = 50  # Length of sequence
d_model = 128      # Dimensionality of embeddings
pos_enc_matrix = positional_encoding(max_position, d_model)

# Plotting the first two dimensions of positional encoding
plt.figure(figsize=(10, 6))
plt.plot(pos_enc_matrix[:, 0], label="Dimension 0 (sin)")
plt.plot(pos_enc_matrix[:, 1], label="Dimension 1 (cos)")
plt.xlabel("Position in Sequence")
plt.ylabel("Positional Encoding Value")
plt.title("Positional Encoding for the First Two Dimensions")
plt.legend()
plt.show()

Explanation:

• max_position: This is the maximum length of the input sequence, representing how many positions we want to encode.
• d_model: This is the size of the embedding vector. Each position is represented by a vector of size d_model, where even indices are sine values and odd indices are cosine values.
• pos_enc_matrix: This matrix holds the positional encoding for every position in the sequence.

The resulting matrix can be added to the word embeddings before feeding them into the Transformer layers.

Using Positional Encoding in Transformers

In practice, positional encodings are added to word embeddings before they are passed into the encoder layers of a Transformer model. The positional encoding matrix is added element-wise to the input embeddings so that the resulting vectors contain information about both the content of the tokens and their positions in the sequence.

Here’s how it’s done:

# Assuming word_embeddings is a matrix of shape (batch_size, seq_len, d_model)
word_embeddings = np.random.randn(32, max_position, d_model)  # Random embeddings for example

# Add positional encoding to the embeddings
input_with_positional_encoding = word_embeddings + pos_enc_matrix[:word_embeddings.shape[1], :]

print("Shape of input with positional encoding:", input_with_positional_encoding.shape)

This step ensures that the Transformer model is aware of the sequence’s structure, allowing it to better understand and model the data.

Conclusion

Positional encoding is a critical component of the Transformer architecture, allowing the model to handle sequences while still processing them in parallel. By using sine and cosine functions of varying frequencies, the Transformer can capture both relative and absolute position information in the input sequence.

In this article, we explored the importance of positional encoding, its mathematical foundation, and implemented it in Python. Positional encoding enables the Transformer to outperform traditional RNNs and LSTMs, particularly in tasks like language modeling, translation, and text generation.