Finding a Better Model for My Next-Word Prediction Project

Introduction

Have you ever typed a sentence on your phone and noticed that it suggests the next word? For example, when you type, “I am going to the…”, your phone might suggest “school,” “market,” or “hospital.” This is an example of next-word prediction, an important task in Natural Language Processing (NLP).

While we were learning about Large Language Models (LLMs) and Transformers, I worked on a next-word prediction project using TinyGPT and GPT-2. Through this project, I learned how language models use the words that come before a prediction to estimate what might come next.

After working with these models, I wanted to explore other language models that could help improve my project. My goal was to understand the available options, compare their advantages and limitations, and choose a model worth testing.

In this blog, I will discuss GPT-2, DistilGPT-2, GPT-Neo 125M, and SmolLM2. I will explain their differences and why I chose SmolLM2 as a candidate for further experimentation.

Understanding Next-Word Prediction

Next-word prediction is the process of predicting the next word or token based on the text that has already been provided.

For example, if the input is:

“I like drinking a cup of…”

A language model might predict “tea” or “coffee.”

Language models learn patterns from large amounts of text during training. When they receive a sentence, they use these learned patterns to estimate which token is likely to come next. A token can be a complete word, part of a word, or punctuation.

Many modern LLMs use this approach to generate text. GPT-2, for example, was trained to predict the next token using the tokens that came before it. This makes it directly relevant to my project.

Reference: GPT-2 model documentation

Models I Researched

1. GPT-2

GPT-2 is a Transformer-based language model that was pretrained on a large collection of text. It learns language patterns by predicting the next token in a sequence.

One of its main strengths is that it can generate text based on a sentence or prompt without requiring me to train a language model from the beginning. It is also a useful starting point for understanding how LLMs work.

I already used GPT-2 in my project, so I can use it as a baseline when evaluating other models.

Limitation: GPT-2 is not the newest available language model, and its predictions depend on its training data, model size, and the text being provided. However, being older does not automatically mean it is unsuitable for next-word prediction.

Reference: GPT-2 on Hugging Face

2. DistilGPT-2

DistilGPT-2 is a smaller version of GPT-2 developed using knowledge distillation. Knowledge distillation is a technique in which a smaller model learns to reproduce important behaviours of a larger model.

Its main advantage is that it is designed to be lighter than the original GPT-2, making it an option to consider when computing resources are limited.

For my project, DistilGPT-2 could help me investigate whether a smaller model can provide useful next-word predictions while requiring fewer resources.

Limitation: A smaller model may not capture every language pattern as well as a larger model. Its actual prediction quality would need to be tested.

Reference: DistilGPT-2 model

3. GPT-Neo 125M

GPT-Neo 125M is a pretrained Transformer language model developed by EleutherAI. Like GPT-2, it can generate text by predicting the next token based on the previous context.

I included GPT-Neo because I wanted to investigate another pretrained Transformer model rather than relying only on GPT-2. Comparing the two could help me understand how different training processes and model designs affect their predictions.

Limitation: GPT-Neo 125M has a similar general architecture to GPT-2. Therefore, it offers a comparison between language models but does not introduce a completely different architecture.

Reference: GPT-Neo 125M model

4. SmolLM2

SmolLM2 is a family of compact language models developed by Hugging Face. The family includes models of different sizes, such as 135 million, 360 million, and 1.7 billion parameters.

I found SmolLM2 interesting because it provides smaller language-model options that can be explored for text generation. A smaller model may be more practical when computing resources are limited, although the actual requirements depend on the model size and how it is run.

I want to investigate whether SmolLM2 can produce useful next-token predictions for my project and how its results compare with GPT-2.

Limitation: SmolLM2 is not guaranteed to perform better than GPT-2 on my particular dataset. The result will depend on the selected version, the input text, and the evaluation method.

Reference: SmolLM2 models

Comparing the Models

After researching these models, I compared them according to their main advantages and limitations.

Model	Main advantage	Main limitation
GPT-2	Already used in my project; designed for next-token prediction	May not be the most efficient option for every task
DistilGPT-2	Smaller alternative to GPT-2	May sacrifice some prediction quality
GPT-Neo 125M	Another pretrained Transformer to compare	Similar general architecture to GPT-2
SmolLM2	Offers several compact language-model options	Performance depends on the chosen size and task

This comparison shows that there is no model that is automatically the best for every next-word prediction project. GPT-2 is already a suitable model for this task, while the alternatives give me opportunities to investigate prediction quality and resource requirements.

The Model I Choose to Test: SmolLM2

After comparing the options, I chose SmolLM2 as the next model I want to test in my project.

My first reason is that it gives me an opportunity to explore another family of language models after working with TinyGPT and GPT-2.

My second reason is that SmolLM2 offers models in different sizes. I can begin by investigating a smaller version, which may be more practical for experimentation when computing resources are limited.

My third reason is that I want to compare its next-token predictions with those of GPT-2. I want to find out whether SmolLM2 can provide a useful balance between prediction quality and computational requirements.

However, I am not claiming that SmolLM2 is already better than GPT-2. I have selected it as a candidate because its design and available sizes make it worth investigating. The experiments will help me decide whether it is a better choice for my particular project.

How I Plan to Test the Model

To evaluate SmolLM2 fairly, I plan to follow these steps:

1. Prepare a test dataset containing sentences that were not used to train or fine-tune the models.
2. Use GPT-2 and SmolLM2 to predict the next token from the same input text.
3. Compare their predictions with the expected next tokens.
4. Measure prediction quality using appropriate metrics, such as top-1 accuracy and perplexity.
5. Compare their response times and computational requirements.
6. Record the results and decide which model is more suitable for my project.

I will also consider tokenization because different models may split the same word into different tokens. For a fair comparison, I must use appropriate evaluation methods and be clear about what each metric measures.

If SmolLM2 performs better on my test data and uses acceptable resources, I will consider using it in the improved version of my project. If GPT-2 performs better, I will keep GPT-2 and use the experiment to understand its advantages.

Conclusion

Working on my next-word prediction project while learning about LLMs and Transformers helped me understand how models such as TinyGPT and GPT-2 can generate text.

To explore possible improvements, I researched GPT-2, DistilGPT-2, GPT-Neo 125M, and SmolLM2. Each has different strengths and limitations, and none should be considered the best without considering the task and evaluating its performance.

I chose SmolLM2 as the next model I want to test because it offers compact language-model options and provides an opportunity to compare its predictions with GPT-2.

My goal is not simply to replace an existing model with a different one. I want to understand the alternatives, compare their performance fairly, and choose the model that best meets the needs of my next-word prediction project.

Through this process, I hope to strengthen my understanding of LLMs and learn how to make better decisions when selecting models for NLP tasks.