How Open-Source LLMs Can Improve My Tiny GPT-2 Next-Word Predictor

1. Project Background: From a Tiny Model to a Better Predictor

For our Natural Language Processing (NLP) group project, we built a Tiny GPT-2 from scratch: a decoder-only Transformer with a causal attention mask, trained to predict the next token in English text. The model runs, and given a prompt such as “She went to the”, it can suggest possible continuations such as “market”, “school” or “park”.

However, getting a model to generate text does not necessarily mean it generates good text. Our small model has limited capacity, learns from a relatively small training corpus and sometimes produces repetitive or grammatically awkward sentences.

This raised an important question: How can open-weight Large Language Models (LLMs) help improve our Tiny GPT-2 without replacing the model we built?

After researching several models, I concluded that they can serve different purposes: generating supplementary training data, providing a reference for knowledge distillation, helping evaluate generated text and offering insights into training techniques. My main choice for data generation and evaluation is Qwen3-4B-Instruct-2507, while GPT-2 Small is a potential teacher for distillation.

2. The Open-Weight Models I Researched

Qwen3-4B-Instruct-2507 — My Primary Choice

Developer: Qwen team, Alibaba Cloud

Parameters: 4 billion, including approximately 3.6 billion non-embedding parameters

Context window: 262,144 tokens

Licence: Apache 2.0

Qwen3-4B-Instruct-2507 is an instruction-tuned language model designed for tasks including text generation, instruction following, logical reasoning and multilingual understanding. Unlike a basic next-token predictor, it can respond to instructions and generate text in a requested format.

The model supports non-thinking mode and does not generate separate <think> blocks. This can simplify its use in automated data-generation pipelines, although the requested output format still needs to be checked.

I selected Qwen3 for three potential uses:

• Data augmentation: Generate additional English sentences and continuations to supplement our original training corpus.
• Semantic evaluation: Assess whether generated continuations appear relevant and plausible in context, alongside objective metrics such as perplexity.
• Training examples: Produce examples for experiments involving specific continuation styles.

Qwen3's public model weights can be downloaded for local use without a paid inference API. Its Apache 2.0 licence permits broad use subject to its terms. At 16-bit precision, its weights require approximately 8 GB of memory. This makes it a plausible candidate for a Google Colab T4 with 16 GB of GPU memory, but the complete model may require additional memory for inference, temporary computations and the attention cache. Shorter contexts or quantisation may be necessary.

Reference: Qwen3-4B-Instruct-2507 model card.

GPT-2 Small — A Potential Distillation Teacher

Developer: OpenAI

Parameters: Approximately 124 million

Context window: 1,024 tokens

Licence: MIT

GPT-2 Small is a pretrained, decoder-only Transformer that predicts the next token using causal language modelling. Its architecture and training objective make it a useful reference for our Tiny GPT-2.

Knowledge distillation is a technique in which a smaller student model learns from the outputs of a larger or more capable teacher. Instead of training only against the correct next token, the student can learn from the teacher's probability distribution over possible next tokens.

However, successful distillation requires compatible tokenisation and output vocabularies, or an appropriate method for aligning them. Using the same GPT-2 tokenizer and vocabulary would make the experiment more straightforward. The training loop would also need to support a distillation loss; this is not achieved automatically without code changes.

GPT-2 Small is therefore a promising teacher to investigate, particularly if our Tiny GPT-2 uses the same tokenizer. Its smaller size also makes it practical for experiments on limited hardware.

Reference: GPT-2 model card.

SmolLM3-3B — A Training and Reproducibility Reference

Developer: Hugging Face

Parameters: Approximately 3 billion

Context window: 65,536 tokens by default; extension to 131,072 tokens is possible with YaRN configuration

Licence: Apache 2.0

SmolLM3-3B is an open-weight language model whose documentation provides useful information about its architecture, training and supported configurations.

For our project, its main value is not necessarily to become another component in the final system. Instead, we can study its published training approach and compare its techniques with our own implementation.

For example, we can investigate how data preparation, tokenisation, training duration and model configuration affect language modelling performance. Its approximately 6 GB of weights at 16-bit precision make it a plausible candidate for a T4, although additional runtime memory is still required.

SmolLM3 is particularly relevant to a learning project because its documented training recipe offers a reference for understanding how modern open models are developed.

Reference: SmolLM3-3B model card.

Other Models Worth Considering

Several additional models provide useful points of comparison:

• Llama 3.2-3B-Instruct — Meta: A 3.21-billion-parameter instruction-tuned model with a 128K context window. It is useful for text generation and multilingual dialogue, but its custom community licence and access conditions require review.
• Gemma 3 4B IT — Google: A 4-billion-parameter instruction-tuned model with a 128K context window. It supports text and image input, making it interesting for multimodal research, although image processing is not necessary for our current text-only project. It uses Google's Gemma terms of use.
• Phi-4-mini-instruct — Microsoft: A 3.8-billion-parameter model with a 128K context window, designed for instruction following and strong reasoning performance. It is another candidate for generating and assessing text.
• Mistral 7B Instruct v0.3 — Mistral AI: A 7-billion-parameter instruction-tuned model with a 32K context window and Apache 2.0 licence. Its 16-bit weights alone require approximately 14 GB, making it a less comfortable choice for a 16 GB T4.

These models demonstrate that parameter count, context length, licensing and intended use all matter when selecting a model. A larger model is not automatically the best choice for a student project.

3. Comparison Table

Model	Developer	Parameters	Context window	Licence	T4 suitability at 16-bit precision	Main value for my project
Qwen3-4B-Instruct-2507	Alibaba Cloud's Qwen team	4B	262K	Apache 2.0	Plausible; ~8 GB weights	Data augmentation and semantic evaluation
GPT-2 Small	OpenAI	124M	1,024	MIT	Very comfortable; ~0.25 GB weights	Potential distillation teacher
SmolLM3-3B	Hugging Face	~3B	65,536 native	Apache 2.0	Plausible; ~6 GB weights	Training reference and reproducibility
Llama 3.2-3B-Instruct	Meta	3.21B	128K	Llama 3.2 Community Licence	Plausible; ~6.4 GB weights	Alternative data generator
Gemma 3 4B IT	Google	4B	128K	Gemma terms of use	Plausible; ~8 GB weights, with additional multimodal overhead where applicable	Alternative instruction-following model
Phi-4-mini-instruct	Microsoft	3.8B	128K	MIT	Plausible; ~7.6 GB weights	Alternative for text generation and reasoning
Mistral 7B Instruct v0.3	Mistral AI	7B	32K	Apache 2.0	Tight; ~14 GB weights	Larger instruction-following comparison

Memory figures are approximate weight sizes, not measured peak GPU memory. Actual requirements depend on precision, context length, batch size and implementation. Public weights can generally be used locally without a paid API, but account requirements and licence acceptance vary by model repository.

4. Why I Chose Qwen3-4B-Instruct-2507

My choice is based on three main reasons.

First, it can help expand our training data. A small training corpus limits the range of language patterns our model encounters. Qwen3 can generate varied English sentences and continuations that supplement the original corpus. However, synthetic examples must be reviewed for quality and diversity. Adding more generated text does not automatically improve performance and may introduce errors or repetitive patterns.

Second, it offers an additional evaluation method. Perplexity measures how well a language model predicts a reference sequence, but it does not directly measure every aspect of grammatical quality or semantic usefulness. Qwen3 can act as an auxiliary evaluator by scoring continuations against a consistent rubric. Because an LLM judge can also make mistakes or show preferences for particular writing styles, its scores should be compared with human assessments and objective metrics.

Third, it is practical for experimentation. The model has a permissive Apache 2.0 licence and can be used locally through tools such as Hugging Face Transformers. Its weight size makes a T4 experiment plausible, although actual memory usage and generation speed must be measured rather than assumed.

Why not use Qwen3 for every task, including distillation? GPT-2 Small may be a more convenient teacher when our student uses the original GPT-2 tokenizer and vocabulary. A teacher with a compatible vocabulary makes it easier to compare next-token distributions. Qwen3 has a different tokenizer, so direct distribution matching requires additional alignment or a different distillation strategy.

My proposed division of work is therefore: Qwen3 for supplementary data and evaluation, and GPT-2 Small for a carefully designed distillation experiment.

5. What I Plan to Build and Measure

To determine whether these techniques genuinely improve Tiny GPT-2, I would conduct the following experiments:

1. Baseline: Train Tiny GPT-2 on the original corpus and record validation loss, perplexity and generated samples.
2. Distillation: Train a separate student using GPT-2 Small's probability distributions, provided the tokenizer and vocabulary are compatible. Compare it with the baseline.
3. Data augmentation: Generate supplementary English text with Qwen3, retrain the model under comparable conditions and measure changes in validation performance.
4. Semantic evaluation: Ask Qwen3 to assess a fixed sample of continuations using a consistent rubric. Compare its ratings with human judgements and perplexity.
5. Error analysis: Examine repetition, grammatical errors, irrelevant continuations and differences between training and validation performance.

All experiments should use a held-out validation set that is not included in the training data. This helps reveal whether an improvement generalises beyond the examples used to train the model.

Conclusion

Building Tiny GPT-2 helped us understand the practical foundations of language modelling, from tokenisation and causal attention to next-token prediction. Researching open-weight LLMs showed me that we can improve a small model without simply replacing it with a larger one.

Qwen3-4B-Instruct-2507 is my preferred model for generating supplementary training examples and providing an additional perspective on output quality. GPT-2 Small offers a promising teacher for distillation, while SmolLM3 provides a useful reference for studying modern training methods.

The key is to test these ideas systematically. If our experiments show lower validation perplexity, more useful continuations and fewer repetitive errors, we will have evidence that the improvements worked. If not, the results will still teach us something important about the limitations of synthetic data, distillation and automated evaluation.

My main takeaway is that open-weight LLMs can be more than ready-made applications: they can also be tools for learning, experimentation and improving the models we build ourselves.

References

1. Qwen Team. Qwen3-4B-Instruct-2507 model card.
2. OpenAI Community. GPT-2 model card.
3. Hugging Face. SmolLM3-3B model card.
4. Meta. Llama 3.2 model documentation.
5. Google DeepMind. Gemma 3 model card.
6. Microsoft. Phi-4-mini-instruct model information.
7. Mistral AI. Mistral 7B v0.3 documentation.
8. Hinton, G., Vinyals, O., and Dean, J. Distilling the Knowledge in a Neural Network.