ASSGNIMENT

Greetings!
I have learned about Large language models are AI systems used to model and process human language. They are called “large” because these types of models are normally made of hundreds of millions or even billions of parameters that define the model's behavior, which are pre-trained using a massive corpus of text data.

In Three Models, the Project: Which Is Best for Kinyarwanda?
Gemma is the model I would choose to improve our project, because it is the one most likely to understand Kinyarwanda. This blog explains the three models I compared, in simple words.
Our project in one minute
In our NLP class, we looked inside GPT-2, a small language model. We also built our own tiny GPT and trained it on Kinyarwanda news.
We found three problems:

*•GPT-2 * splits Kinyarwanda words into many small pieces. It needs about 1.7 times more pieces than English for the same sentence.
•GPT-2 learned almost only from English, so it cannot write good Kinyarwanda.
•Our tiny GPT is very small, so it writes text that looks like Kinyarwanda but makes no sense.
So we need a model that is better with Kinyarwanda. We also need a model we can open and look inside, because that is what our project is about.

What is a model, and what is a token?
A language model is a program that reads text and guesses the next word. A token is a small piece of text that the model reads. English words are often one token. Words from languages the model did not study are cut into many tokens, and that makes reading harder.

1.Phi-3.5-mini
FeatureSc
•Creator: Microsoft
•Size: 3.8 billion parameters
•Context Window: Up to 128,000 tokens
•License: MIT License (Highly permissive)
•What it is good at: Despite a compact footprint, Phi-3.5 was trained on a heavily curated, high-quality dataset containing vast amounts of multilingual text and code data. It punches well above its weight class for reasoning, instruction following, and structured formatting.

Model 2: Qwen2.5-1.5B
Qwen is a newer model with 1.5 billion parameters. It learned from text in many languages, not only English.
•Good: it is much bigger than GPT-2, it knows more languages, and it needs no login to download.
•Bad: it is smaller than Gemma, and I have not tested how well it handles Kinyarwanda.
•Role in our project: it is our backup choice.
Feature
•Creator: Alibaba Group
•Size: 1.5 billion parameters
•Context Window: Up to 32,768 tokens
•License: Apache 2.0 / Qwen License (Permissive commercial and research use)
• What it is good at: Trained on a massive multilingual dataset covering over 29 languages. It features strong instruction-following capabilities, structured data generation, and an exceptionally generous context window for its size class.

Model 3: Gemma-2-2b
Gemma is a model from Google with 2 billion parameters. It also learned from many languages. From what I know of the Gemma family, its list of tokens is very large, about 256,000 pieces, and it was made to work across many languages.
•Good: *it is the biggest of the three, and its large token list should cut Kinyarwanda words into fewer pieces.
*•Bad: you must accept a license and log in to Hugging Face to download it, and I have not tested it on Kinyarwanda yet.
•Role in our project: it is our main choice.
Feature
•Creator: Google
•Size: 2 billion parameters
•Context Window: *8,192 tokens
*•License: Gemma Terms of Use (Requires a free Hugging Face account and license acceptance)
•What it is good at: Built from research and technology similar to Google's larger production models, Gemma features a massive vocabulary size (roughly 256,000 tokens) designed explicitly to handle multilingual corpora and tokenization efficiency across diverse languages.

The best one for our project: Gemma
I choose Gemma for four reasons.
1.It should read Kinyarwanda better. Our biggest finding was that GPT-2 cuts Kinyarwanda into too many pieces. Gemma has a bigger token list made for many languages, so it should need fewer pieces.
2.It already knows how language works. Our tiny GPT had to learn everything from a small amount of news text. Gemma already knows a lot, so we only need to teach it Kinyarwanda. We can do this with a small training step called LoRA, which changes only a few of the model's numbers.
3.It is bigger. More parameters means more room to learn a new language.
4.We can still look inside it. Gemma is an open model, so our attention plots and layer experiments still work. Closed models like ChatGPT or Gemini only give answers, so we could not study them the same way.
Qwen is a close second. If Gemma's login is a problem,
Qwen is a good choice.

References
•Google DeepMind. (2024). Gemma 2: Improving Open Language Models at Size. Google AI Technical Report.
•Alibaba Group. (2024). Qwen2.5 Technical Report. Qwen Documentation and GitHub Repository.
•Microsoft. (2024). Phi-3.5 Technical Report: Highly Capable Open Models on Your Device. Microsoft Research.
•Hugging Face Model Hub: Gemma-2-2b Model Card, Qwen2.5-1.5B Model Card, and Phi-3.5-mini Model Card.