Motivation
This is the beginning of a blog series where I try to understand large language models by reading Build a Large Language Model (From Scratch) by Sebastian Raschka and writing down what I learn along the way.
Table of Contents
- Introduction to LLMs
- Architectures behind LLMs
- Recurrent and convolutional architecture
- Transformer architecture
- Self-attention mechanism
- BERT vs GPT
- Understanding GPT architecture
What's an LLM?
A large language model is a neural network that is designed to understand, generate, and respond to text. The fundamental understanding here is that large language models are trained on huge datasets, from web-scraped data to Wikipedia.
They utilize the transformer architecture, but note that there are some LLMs that are based upon alternative architectures like recurrent and convolutional architectures.
Recurrent Architecture
A recurrent neural network reads the text sequentially.
The model processes a sentence like "I grew up in Nepal, so I speak ..." in this order:
I -> update memory
grew -> update memory
up -> update memory
in -> update memory
Nepal -> update memory
...
speak -> use memory to predict next token
Languages are mostly sequential. Therefore, prediction is possible using RNNs, but the main problem here lies in efficiency. After reading each token, or word, the model requires a compressed memory of everything so far. Using this memory, the model predicts the next token.
This works for small sentences, but for larger sentences this architecture can lead to context loss if the memory was compressed badly.
Convolutional Architecture
A convolutional language model does not process the sequence recurrently one token at a time like an RNN. Instead, it applies filters over local token windows. But if it is used as a language model, it can still be trained to predict the next token.
For example:
The food I ate at the restaurant last week was really good compared to the restaurant I ate at yesterday.
Here we take filter size = 4 and stride = 2.
Filter size, or kernel size, determines how many neighboring tokens the model looks at once. Stride determines how many tokens the window shifts each time.
The food I ate
I ate at the
at the restaurant last
restaurant last week was
week was really good
...
I ate at yesterday
It learns local patterns. Words alone can have a different, or even reverse, meaning when nearby words are accounted for. But this still does not directly solve it entirely, as the association between the food being great can be mistaken for "week was really good" when accounted for locally. A convolutional model needs stacked layers to gradually expand its view.
Transformer Architecture
A transformer is a deep neural network architecture that consists of two submodules: an encoder and a decoder. This architecture was originally designed for machine translation.
In simple words, the encoder processes the input text and encodes it into a series of numerical representations: vectors that capture the contextual information of the input. Then, the decoder module takes these encoded vectors and generates the output text.
Let's take an example. "My name is Pragalva" is an input text that will be preprocessed before it enters the encoder. Preprocessing means converting raw input text into numerical input embeddings that a transformer can process. These embeddings are then converted into vectors, and the decoder decodes those vectors to generate text in the target language.
These explanations are surface level and will be explored in depth in the coming blogs as I continue my reading.
Self-Attention Mechanism
Self-attention is a key component of transformer architecture, and it sets transformers apart from the architectures we explored previously. This mechanism allows the model to weigh the importance of different tokens in a sequence relative to each other.
This also helps solve the problem we discussed with CNNs. With a self-attention mechanism, the model is able to capture long-range dependencies and contextual relationships within the input data, resulting in more contextually relevant output.
BERT vs GPT
BERT and GPT are two variants that originated from the transformer architecture. BERT, short for Bidirectional Encoder Representations from Transformers, focuses on the encoder side of the transformer architecture. GPT, short for Generative Pre-trained Transformer, was built on the decoder submodule.
Understanding GPT Architecture
Compared to the original transformer architecture we covered earlier, the general GPT architecture is relatively simple. Essentially, it is just the decoder part.
Since decoder-style models like GPT generate text by predicting text one word at a time, they are considered a type of autoregressive model. Autoregressive models incorporate their previous outputs as inputs for future predictions. Consequently, in GPT, each new word is chosen based on the sequence that precedes it, which improves the coherence of the resulting text. GPT architecture is the foundation for the majority of the models that we currently use in our daily workflows.
As seen above, the GPT architecture only uses the decoder portion of the transformer. GPT is also designed for unidirectional processing, left to right in this case, and is well suited for next-token prediction and text generation.
While simpler decoder-only models like GPT are aimed at next-token prediction, they are also capable of performing tasks that they were not initially trained for. This phenomenon is called emergent behavior, meaning the capability was not explicitly taught during training but emerges naturally. For example, GPT models trained for next-token prediction can also perform translations as a natural consequence of the model's exposure to vast quantities of multilingual data in diverse contexts.
Summary
In this blog, we mainly discussed LLMs and the architecture behind them. While we might have just scratched the surface, these foundational topics are required as we dive further into these concepts.
Originally published at pragalva.me.
Top comments (0)