How I Fine-Tuned a 1.5B LLM for Lightning-Fast Offline Q&A on 1GB VRAM

Running large language models locally often demands expensive hardware with high VRAM. However, for specialized tasks like offline Q&A and knowledge retrieval, a lightweight, highly optimized small language model (SLM) can deliver incredible speed and efficiency without breaking the bank.

In this post, I will share how I fine-tuned moeinGTS 1.5B, quantized it to GGUF, and deployed it for fast offline inference.

---

🚀 Model Overview: moeinGTS 1.5B

moeinGTS is a compact 1.5B parameter language model fine-tuned specifically for fast, accurate question-and-answering based on Wikipedia and structured datasets.

Key Highlights:

• Parameter Size: 1.5 Billion
• Quantization: Q4_K_M & F16 GGUF
• Download Size: ~1 GB
• VRAM Requirements: Only ~1.5 GB
• Primary Use Case: Offline local Q&A, low-latency edge AI applications

---

🛠️ The Pipeline: Fine-Tuning & Deployment

1. Dataset & Training: Fine-tuned on high-quality Q&A and knowledge retrieval datasets to improve concise factual responses.
2. LoRA Fine-Tuning: Applied Low-Rank Adaptation (LoRA) via PEFT to optimize training efficiency and keep resource usage minimal.
3. Quantization: Converted the model weights to GGUF format for seamless cross-platform execution on low-end hardware, CPUs, and edge devices.

---

💻 Quickstart: Run It Locally

You can run moeinGTS instantly in your terminal using Ollama directly from Hugging Face:

bash
# Run GGUF version directly via Ollama
ollama run hf.co/arshiysohrevardi/moeinGTS1.5-1.5b-F16-GGUF