MasryGPT-Chat-1.5B: Teaching Qwen2.5 to Speak Egyptian Arabic (Masry)

Try it: huggingface.co/ISLAM-PO/MasryGPT-Chat-1.5B · Gradio demo

Most Arabic LLMs speak Modern Standard Arabic (MSA) — the language of news and textbooks. But 105 million Egyptians speak Masry (العامية المصرية) every day: "ازيك يا باشا؟" not "كيف حالك؟". That gap is cultural, not just linguistic, and it is the reason MasryGPT exists.

MasryGPT-Chat-1.5B is an open-source, instruction-tuned chat model built on Qwen2.5-1.5B-Instruct, fine-tuned with QLoRA + Unsloth on 80,000 Egyptian-Arabic examples. It is small enough to run on a single T4 GPU (~1.2 GB in 4-bit) or even edge devices, and it is released under Apache 2.0.

This article is a technical deep-dive: motivation, model details, dataset, training pipeline, loss analysis, evaluation, usage, limitations, and roadmap.

---

1. Motivation

• MSA vs. Dialect gap. Existing Arabic LLMs (Qwen, Jais, AceGPT) optimize for MSA. Egyptian users get stiff, translated-sounding replies. "ازيك" vs "كيف حالك" is the difference between a friend and a textbook.
• Lightweight deployment. 7B models need 14 GB+ VRAM. A 1.5B model runs on mobile, T4, or edge — perfect for Egyptian startups with limited GPU budgets.
• Unsloth efficiency. 2× faster fine-tuning on a free Kaggle T4×2, no A100 required.

2. Model Details

Attribute	Value
Model ID	ISLAM-PO/MasryGPT-Chat-1.5B
Base model	Qwen/Qwen2.5-1.5B-Instruct
Architecture	Qwen2ForCausalLM (28 layers, 12 attn heads, 2 KV heads)
Hidden / intermediate size	1536 / 8960
Vocab size	151,936
Max position embeddings	32,768 (trained on 2,048)
Total parameters	1,562,179,072
Trainable (LoRA)	18,464,768 (1.18%)
LoRA config	r=16, alpha=16, dropout=0, bias=none, targets=[q,k,v,o,gate,up,down]_proj
Precision	Trained 4-bit (NF4), released 16-bit (bfloat16), ~2.9 GB
Tokenizer	Qwen2 tokenizer + chat_template.jinja (qwen-2.5)
License	Apache 2.0 (Qwen license applies)

3. Dataset

qwen_egyptian_80k_terms.jsonl — 80,000 examples.

• Source & curation: synthetic generation from an Egyptian lexicon + manual vulgarity filtering.
• Format: {"messages": [{"role":"user","content":"..."},{"role":"assistant","content":"مصري..."}]}
• Preprocessing: get_chat_template(tokenizer, chat_template="qwen-2.5") + tokenizer.apply_chat_template(..., tokenize=False) → packed text field.

Distribution:

Category	Share
Greetings & daily chat	35%
Customer service & instructions	20%
Humor & jokes	15%
Expressions & idioms	15%
Culture & food	15%

Example:

{"messages": [{"role": "user", "content": "يعني ايه عيش وملح؟"},
              {"role": "assistant", "content": "عيش وملح يعني العشرة والعهد بين الصحاب، اللي كل معاك عيش وملح ميخونكش"}]}

4. Training Pipeline & Deep Loss Analysis

Pipeline: HF Hub → FastLanguageModel (4-bit) → get_peft_model (QLoRA) → SFTTrainer (TRL) → save_pretrained_merged (16-bit) → HF Hub

Hyperparameters (final fast config, Kaggle T4×2):

Param	Value	Rationale
max_seq_length	2048	Captures longer Egyptian rants without OOM
per_device_train_batch_size	32	Maximized for T4 15 GB
gradient_accumulation_steps	1	Effective batch 32 → 80k/32 = 2,500 steps
learning_rate	2e-4	QLoRA standard for 1.5B, cosine decay
optim	adamw_8bit	GPU-resident (vs paged → RAM)
weight_decay	0.01	Regularization against overfitting on 80k
warmup_steps	20	0.8% of steps
lr_scheduler	cosine	Best for 1 epoch
packing	True	10× sample/sec on short sentences
num_train_epochs	1	2nd epoch risks memorization

Loss curve (2,500 steps, 2h 13m):

Step 10:    0.091025
Step 100:   0.098729  (initial spike — warmup)
Step 500:   0.082923
Step 1000:  0.081538
Step 1500:  0.077019
Step 2000:  0.074811
Step 2500:  0.074394
Avg: 0.07995   Perplexity exp(0.079) ≈ 1.083

• Convergence: smooth 18.7% drop, no divergence. Loss plateaus at 0.074–0.077 after step 1,500 → dataset saturated. More epochs would overfit.
• Throughput: 10.002 samples/sec, 0.313 steps/sec (Unsloth 2× vs vanilla).
• GPU: 2.4 GB / 15 GB (GPU0, 93%) + 4.1 GB / 15 GB (GPU1, 51%). Low memory is a feature of 4-bit + Unsloth, not a bug. CPU at 100% was the tokenization bottleneck.
• FLOPs: 1.05e17 total ≈ 13.1 TFLOPs/s on T4.

Why not 10 GB VRAM? A 1.5B 4-bit model weighs ~0.8 GB; even batch 32 adds ~1.5 GB activations. To reach 10 GB you must disable 4-bit (full 16-bit = 3 GB base + 6 GB optimizer) — 3× slower and unnecessary. The current 2–4 GB is optimal.

5. Evaluation & Benchmarks

Status note: these figures are an internal pilot, not an independently reproduced benchmark. See EVALUATION.md in the repo for protocol and limitations.

Quantitative:

Benchmark	MasryGPT Chat	Qwen2.5-1.5B Base	Gain
Train loss	0.074	0.115 (zero-shot)	−35%
Perplexity (5% test split)	1.08	1.22	−11%
Egyptian fluency (human, 100 prompts, 1–5)	4.6	3.1	+48%
MSA fluency	4.2	4.7	−10% (tradeoff)
Idiom correctness ("عيش وملح")	40%	20%	+100%, still weak

ARC-Challenge (via lm_eval, vLLM on H100, greedy, temp 0): 47.7% acc_norm (43.69% acc) on 1,172 tasks.

Qualitative:

• ✅ User: ازيك؟ → ازيك يا باشا عامل ايه النهاردة؟ (natural)
• ❌ User: يعني ايه عيش وملح؟ → عيش وملس = خبز (FAIL — needs more idiom data)
• ✅ User: احكيلي نكتة → short joke (needs max_new_tokens=256, temperature=0.8)

Base Qwen answers in MSA ("كيف حالك؟"); MasryGPT answers "ازيك يا باشا؟" — judged 48% more Egyptian by 3 native speakers.

6. Usage

Unsloth (recommended, fast):

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="ISLAM-PO/MasryGPT-Chat-1.5B",
    max_seq_length=2048, dtype=None, load_in_4bit=True,
)
FastLanguageModel.for_inference(model)
messages = [{"role": "user", "content": "ازيك يا باشا؟"}]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_tensors="pt", return_dict=True).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.8,
                        top_p=0.95, do_sample=True, repetition_penalty=1.1)
print(tokenizer.batch_decode(outputs[:, inputs["input_ids"].shape[1]:],
      skip_special_tokens=True)[0])

Transformers (standard):

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "ISLAM-PO/MasryGPT-Chat-1.5B", device_map="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("ISLAM-PO/MasryGPT-Chat-1.5B")

Generation tips: temperature 0.7–0.8 for creativity, 0.3 for facts; always return_dict=True to fix the attention_mask warning; add fix_mistral_regex=True if you see that warning.

7. Limitations & Error Analysis

1. Idioms still weak (40%) — dataset only 15% idioms; needs ~5k real idiom pairs.
2. Hallucination — may invent Egyptian facts; no RAG.
3. Short responses — trained on packed short text; raise max_new_tokens and use repetition_penalty.
4. MSA degradation (−10%) — catastrophic forgetting from dialect fine-tuning.
5. No safety filter beyond base Qwen.

Failure-mode analysis: loss plateau at step 1,500 → dataset saturated. Fix: add 40k diverse real conversations + 1 more epoch at LR 1e-4.

8. Roadmap

• [ ] v1.1: +5k idioms + 10k real chat logs → target 90% idiom accuracy
• [ ] v2.0: DPO/RLHF for Egyptian humor alignment
• [ ] v2.5: 3B version for better reasoning, keep 1.5B "lite"
• [ ] Deployment: GGUF + Ollama + ONNX

9. Citation

@misc{masrygpt_chat2026,
  author = {ISLAM-PO},
  title  = {MasryGPT Chat: Egyptian Dialect Qwen2.5-1.5B-Instruct via QLoRA & Unsloth},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/ISLAM-PO/MasryGPT-Chat-1.5B},
  note   = {80k terms, 2500 steps, loss 0.079}
}

Contribute: found a wrong Masry word? Open an issue with prompt + expected answer. Have anonymized Egyptian chat data (with documented permission)? Propose a separate dataset repo after privacy/provenance review.

---

Author: Islam Al-Nashar — Full-Stack Developer & AI builder. GitHub · Hugging Face · Portfolio