DEV Community

Grace UMUTONIWASE
Grace UMUTONIWASE

Posted on

Open-Source Large Language Models: Finding the Best Model for Our Project

#webdev #ai #programming #beginners

Which Open-Source LLM Should We Use to Predict the Next Word?

Our project is about next-word prediction: the model reads the start of a sentence and guesses which token comes next. In our first experiments we opened up GPT-2 (124 million parameters) and also trained a tiny GPT of our own on Kinyarwanda news. One result stood out. GPT-2 found a Kinyarwanda sentence much harder than the same sentence in English, and it needed about 1.7 times as many tokens to write it, because it was trained almost only on English.

That raised a question for us: is there a newer open-source model that does better, costs nothing, and still runs on a free Google Colab T4 GPU (16 GB)? We looked at five and picked one.

1. The models we looked at

Qwen2.5-1.5B (Alibaba Cloud, Qwen team). It has 1.54 billion parameters and a 32,768-token context window. The licence is Apache 2.0 and the model is not gated on Hugging Face, so nobody needs an account or a licence click. It was trained on text in more than 29 languages and is known for following instructions well. There is a base version, which simply continues text (exactly our task), and an Instruct version for chat.

Llama 3.2 3B (Meta). It has 3.21 billion parameters and a 128K-token context window. It uses Meta's own Llama 3.2 Community License, and the official files on Hugging Face are gated, so you must log in and accept the licence first. It is a strong all-round small model for chat and summaries. Meta officially supports eight languages, and Kinyarwanda is not one of them.

Phi-3.5-mini (Microsoft). It has 3.8 billion parameters and a 128K-token context window, under the very permissive MIT licence. It was trained on carefully chosen, reasoning-heavy data, so it is good at reasoning and long documents for its size. It is the biggest model on our list and focuses mainly on English.

Gemma 3 1B (Google DeepMind). It has 1 billion parameters and a 32K-token context window, and it was trained on 2 trillion tokens in over 140 languages with a large 262K-entry vocabulary. It is released under the Gemma Terms of Use, which is not a standard open-source licence, and you must accept those terms on Hugging Face before downloading. It is very light, but at this size it has the least capacity.

SmolLM2-1.7B (Hugging Face). It has 1.7 billion parameters, was trained on 11 trillion tokens, and uses Apache 2.0 with no gating. Its context window is only about 8K tokens, the shortest here, and it is mostly English, so we expect it to struggle with Kinyarwanda just as GPT-2 did.

2. Comparison table

Model Made by Parameters Context window Licence Account or licence click needed? Good at
Qwen2.5-1.5B Alibaba (Qwen) 1.54B 32K Apache 2.0 No Instructions, many languages, small size
Llama 3.2 3B Meta 3.21B 128K Llama 3.2 Community Yes (gated) Chat, summaries, tool calling
Phi-3.5-mini Microsoft 3.8B 128K MIT No Reasoning, long documents
Gemma 3 1B Google DeepMind 1.0B 32K Gemma Terms of Use Yes (gated) Light multilingual text
SmolLM2-1.7B Hugging Face 1.7B ~8K Apache 2.0 No Short English text

3. Our choice: Qwen2.5-1.5B

We chose Qwen2.5-1.5B. Here is how it answers the questions we set ourselves.

Does it run on a free Colab T4? Yes, with plenty of room. A model needs roughly 2 bytes per parameter in 16-bit precision, so 1.54 billion parameters is about 3 GB. By the same rule, Phi-3.5-mini needs about 7.6 GB and Llama 3.2 3B about 6.5 GB. Both would fit, but with less space left and slower predictions. These are our own estimates, and we will check the real numbers in Colab with torch.cuda.max_memory_allocated().

Is it free, with no API key or account? Yes. It is Apache 2.0 and not gated, so any team member can load it with one line of code. Llama 3.2 and Gemma both need a Hugging Face login and an accepted licence, which is annoying for a group project and for anyone who wants to re-run our notebook.

Does it do what next-word prediction needs? Yes. Like GPT-2, it is a decoder-only transformer. We feed in tokens, read the probabilities for the next token and compute the loss, so we can reuse the compute_loss and continue_text functions we already wrote. Its context window of 32K tokens is also far bigger than GPT-2's 1,024 positions or our tiny GPT's 64.

How fast is it and how much memory does it use? At about 3 GB it is the second-lightest model on our list, after Gemma 3 1B, and it should predict faster than the 3B and 3.8B models on the same GPU. It is still about ten times bigger than GPT-2 small, but comfortable on a free T4.

How might it improve our project? First, it is a newer model trained on far more data than GPT-2, so we expect better predictions on ordinary English text. Second, and more interesting for us, it was trained on more languages and has a vocabulary of about 150,000 tokens, compared with GPT-2's 50,000. That might mean fewer pieces per Kinyarwanda word. We have not tested this yet, and none of the five models officially lists Kinyarwanda, so we expect imperfect results.

How we will check our choice

We will run the same two example sentences from our GPT-2 notebook through Qwen2.5-1.5B and compare three things: tokens per word in English and Kinyarwanda, the loss on each sentence, and the memory and time used on the T4. If Qwen2.5 does not beat GPT-2 on Kinyarwanda, we will say so and keep our tiny GPT, which learned from Kinyarwanda news, for that language.

Why not the others?

Phi-3.5-mini is our runner-up for its reasoning, but it is more than twice the size, mostly English, and we have no use for 128K of context. Llama 3.2 3B is gated and has a custom licence, which fails our "no account" rule. Gemma 3 1B is gated too and has the least capacity, although its multilingual vocabulary makes it the next model we would test. SmolLM2-1.7B is free and light, but its short context and English focus make it a weaker fit.

Limits

A 1.5B model is still small and will make mistakes, especially in Kinyarwanda, which is thin in its training data. Predicting the next word also means choosing a likely word, not a true one.

4. References

Top comments (0)