Large Language Model- Models worth considering to improve our project.

The previous TinyGPT that my team worked on was a small character or word-level Transformer trained from scratch on a small text collection dataset.

Where TinyGPT hit its limits

A TinyGPT trained from scratch on a small dataset can learn the shape of language, but it hits predictable walls:

• It has seen very little text, so it repeats itself and loses the thread after a few words.
• Its vocabulary is small or character-based, so rare words and spelling get messy.
• Its context window is short, so it forgets the start of a sentence by the end.

Models to check if they are suitable for improving our project;

1. LSTM / GRU (the classic baseline). These recurrent networks were the standard for next-word prediction before Transformers. They are small, fast to train, and make a good baseline, but they struggle with long-range dependencies. They’d be a step sideways rather than forward.

2. DistilGPT-2 / GPT-2 Small. These are the same architecture as TinyGPT, but pretrained on a huge amount of web text. DistilGPT-2 is a compressed version of GPT-2 (about 82M parameters) that runs on a laptop or free Colab GPU. Because it has already learned grammar, facts and style, fine-tuning on my own data gives a big jump in quality.

3. Modern small language models (SmolLM, Qwen small variants, Phi-style models). Newer models in the roughly 100M to 3B range are trained on carefully curated data and are far stronger than GPT-2 at the same size. The trade-off is that they need more memory, and I’d need to check which versions are currently available and what their licenses allow.

4. Mamba / RWKV (state-space and linear-attention models). These handle long sequences more cheaply than standard attention. They are exciting and educational, but the tooling is less mature and less beginner-friendly.

5. Retrieval-augmented approaches. Here the model looks up relevant text before predicting. That helps with factual accuracy, but it’s more than a next-word predictor needs.

My Choice: DistilGPT2

After comparing these models, I would choose DistilGPT2 to improve my TinyGPT project.

My main reason is that both models are designed for autoregressive text generation, but DistilGPT2 already benefits from pretraining on a large text dataset. My original TinyGPT was built to predict the next word using a small Transformer architecture, with token embeddings, positional information, attention layers, and residual connections. Although this approach helped me understand how language models work internally, a model trained from scratch may not learn enough language patterns from a limited dataset.

DistilGPT2 would give me an opportunity to benefit from previously learned language patterns while keeping the experiment more manageable than using a much larger model. I could compare its predictions with those of my original TinyGPT and evaluate both models using the same test sentences.

Although DistilGPT2 would be the best option to use to improve our project there are also some models that can also be worth trying. They include;

• Qwen2.5-0.5B

This model has approximately 500 million parameters and supports a much larger context window than your original TinyGPT configuration.
Why choose it: It is an interesting option if you want to explore a more capable small language model, particularly for longer text prompts and multilingual experiments.

• _ SmolLM2_

SmolLM2 comes in different sizes, including 135M and 360M parameters. It is designed for relatively lightweight language-model applications and offers pretrained text-generation capabilities.
Why choose it: You can compare a pretrained model against your smaller, custom-built TinyGPT without needing to train a large model from scratch.