DEV Community: Isha_17Bhardwaj

Building Llama.cpp-Based Local AI Chat Assistant

Isha_17Bhardwaj — Thu, 10 Jul 2025 10:16:14 +0000

“I was learning LangChain basics, and then just in short period of time … I built my own AI assistant running entirely offline on my WSL. Let’s talk about how it happened.”

The Inspiration

I came across this awesome Docker blog about building a smart, local AI chat assistant using Goose CLI and Docker Model Runner. It was neat, powerful, and looked so plug-and-play.
But… then came the reality check: I’m on a Windows laptop with WSL2 and limited storage Pulling multi-GB Docker images wasn’t just risky—it was destructive for my system.

So, instead of giving up, I pivoted. I asked myself:

“Can I build something similar using lightweight tools, run models offline, and still impress myself?”

And the answer was a resounding yes, thanks to:
llama.cpp and Hugging Face's GGUF models with A bit of determination

What Will You Learn From This Blog?
In this blog, you’ll learn:

🔹 How to build an AI chat assistant that runs completely offline using llama.cpp.
🔹 How to download and run quantized GGUF models from Hugging Face without using GPUs or Docker.
🔹 How to set up a full project with WSL2, fix common issues with dependencies, tokens, and model errors.
🔹 And most importantly, how to ship a working AI app even with limited storage and system resources.

Tools Used

llama.cpp- for model inference
Qwen1.5-0.5B-Chat-GGUF - (Q4_K_M) quantized model from Hugging Face
WSL- (Ubuntu 22.04 on Windows)
Git- CMake, GCC (g++) for compilation

Before even jumping directly into the building stuff few Initial concepts I need you to look into .

 if you already know this feel free to skip

Initial Concepts

What exactly is LLaMA and Llama.cpp?

Let me break down this for you in simple manner

LLaMA (Large Language Model Meta AI) is a family of open-source language models developed by Meta. It offers powerful NLP capabilities with smaller computational requirements, making it perfect for offline and local inference tasks.

Llama.cpp is a C++ implementation of the LLaMA inference engine that can run these models efficiently on a wide range of hardware including CPUs, especially in resource-constrained environments like laptops.

In short

Llama is a family of large language models (LLMs) developed by Meta (formerly Facebook). These models are designed to understand and generate human-like text based on the input they receive.
Llama.cpp, on the other hand, is a high-performance, lightweight inference engine that allows these models to run efficiently on consumer-grade hardware (even without a powerful GPU)

For more easy understanding you can check out the glossary part of this article

1. What is Qwen1.5-0.5B-Chat?

Qwen1.5-0.5B-Chat is a small but efficient open-source AI language model developed by Alibaba Group. It is part of the Qwen (Tongyi Qianwen) family of models, designed for chat-based applications while being lightweight enough to run on consumer hardware.

Features

Model Type: A 0.5 billion parameter (0.5B) chat-optimized language model.
Developed by: Alibaba’s AI research team (Tongyi Qianwen).
Open-weight: Unlike closed models like GPT-4, its weights are publicly available.
Efficient: Designed to run on low-resource devices (laptops, edge devices, etc.)

2. Why Choose Qwen1.5-0.5B-Chat?

(A) Advantages Over Larger Models
🔹 Runs on weak hardware (even a Raspberry Pi 5 can handle it with optimizations).
🔹 Lower latency – Faster response times due to smaller size.
🔹 Privacy-friendly – No need to send data to cloud APIs.

(B) Limitations
⚠ Less knowledge depth than 7B+ models.
⚠ Shorter memory in conversations.
⚠ May struggle with complex reasoning (compared to GPT-4 or Llama 70B).

3. How Qwen1.5-0.5B-Chat Works (Simplified)

Step-by-Step Inference Process

Input Prompt → User sends a message (e.g., "Explain quantum computing
Tokenization → Text is split into smaller units (tokens).
Model Processing → Neural network predicts the next words.
Response Generation → Outputs a coherent answer.

User Input  
    ↓  
[Tokenization] → Converts text to numbers  
    ↓  
[Qwen1.5-0.5B Model] → Processes input & generates predictions  
    ↓  
[Detokenization] → Converts numbers back to text  
    ↓  
AI Response → "Quantum computing uses qubits instead of bits..."

For more information you can refer to the official Qwen documentation here

qwen.readthedocs.io

What is GGUF?

GGUF (GPT-Generated Unified Format) is a file format designed to store and run large language models (LLMs) efficiently on consumer hardware (like your laptop or even a Raspberry Pi). It supports quantization (shrinking model size without losing too much performance) and is optimized for CPU-first inference (but can also use GPUs).

In short its
PURPOSE

GGUF is a binary file format for storing LLMs (like Llama 2, Qwen, Mistral, etc.)
Designed for fast loading, efficient memory usage, and hardware compatibility (CPU/GPU).
Supports quantization (reducing model size while maintaining performance).

REAL LIFE EXAMPLE
Example: Running a Chatbot on a Laptop
Let’s say you want to run a Llama 2 7B model on your laptop (which doesn’t have a powerful GPU).

Step 1: Original Model (Before GGUF)
Format: PyTorch (.bin or .safetensors).
Size: ~13GB (FP16 precision).
Problem: Too big for most laptops, slow on CPU.

Step 2: Convert to GGUF + Quantization
Process:
The model is converted to GGUF format.
Quantized to 4-bit precision (Q4_K_M).
Result: Size drops from 13GB → ~3.8GB.
Runs smoothly on a laptop CPU.

What is Hugging Face & Why I Used It

Hugging Face is the GitHub of AI models. It hosts thousands of pre-trained models, including GGUF-formatted, quantized models like Qwen, TinyLlama, and Phi.

Using Hugging Face

I could pick smaller, CPU-friendly models (like Qwen-0.5B)

I could download models securely with access tokens

And yes, I faced errors like:

401 Unauthorized
RepositoryNotFoundError
Token config issues

But I eventually fixed them with:
Creating a Hugging Face access token
Using the right filenames and huggingface-cli download command

Step-by-Step: How I Built the AI Chat Assistant

1. Cloning & Building llama.cpp

cmake ..
What It Does:
Generates build files (like Makefile) from CMakeLists.txt (a configuration file).means it looks for CMakeLists.txt in the parent directory (since you typically run this inside a build/ folder).
Why Use It?
Converts high-level project definitions into platform-specific build instructions (for Linux, macOS, Windows, etc.).
Handles dependencies, compiler flags, and system-specific settings automatically.

make -j
What It Does:
Compiles the source code into executable binaries using the generated Makefile.
-j = Parallel compilation: Uses all CPU cores to speed up the build.
Why Use -j?
Without -j: Compiles files one at a time (slow).
With -j: Compiles multiple files simultaneously (much faster)

2. Downloading the Model from Hugging Face
I used Qwen1.5-0.5B (chat-tuned, quantized to Q4_K_M for low resource)

3. Running the Chat Assistant

Here is the final output of how the AI assistant actually looks -

and the Demo video you can see here - Video

GitHub Repository
I organized all the code, scripts, model config, and docs in a neat public repo.
Repo:

IshaSri-17Speed / llama-ai-assistant

Build A lightweight, interactive AI Assistant powered by llama.cpp and the Qwen1.5 GGUF model. Fully runs offline in WSL on Windows 10, optimized for low-resource hardware (~4GB RAM usage). Ideal for developers and students who want to learn how to run large language models (LLMs) locally, without relying on cloud APIs or GPUs.

🧠 Llama.cpp-Based AI Chat Assistant

This project is a lightweight, fully local AI assistant built using llama.cpp and a quantized Qwen1.5 0.5B GGUF model. It runs completely offline on my local machine using WSL (Ubuntu on Windows 10) — no internet or cloud required.

✨ Features

🧠 Uses Qwen1.5-0.5B-Chat model in GGUF format
⚡ Runs on CPU with no GPU required
💻 Built using llama.cpp with full CMake build system
🪶 Lightweight: ~4GB RAM usage with quantized Q4_K_M model
🌐 Works entirely offline after download
💬 Interactive CLI with conversation-style responses
🧰 Beginner-friendly — no prior ML experience needed

✨ What It Does

Interactively chats like a personal assistant using a local LLM (Qwen1.5-0.5B GGUF)
Processes user prompts in real-time via command line
Runs efficiently on low-end hardware (8GB RAM / no GPU)
Uses llama.cpp, a C++ inference engine optimized for speed and low memory

🔧 Tech Stack

💻…

View on GitHub

Project Structure:

llama-ai-assistant/
├── README.md
├── llama.cpp/              # Submodule (excluded from .git)
├── models/
│   └── qwen1_5-0_5b-chat-q4_k_m.gguf
├── screenshots/
│   └── ai-screenshot.png
├── demo/
│   └── AI-assistant-demo.mp4
├── LICENSE

Glossary

(A) Large Language Models (LLMs)
AI models trained on massive amounts of text data.
Predict the next word in a sequence (autocomplete on steroids).
Examples: GPT-4, Llama 2, Claude, Gemini.

(B) Model Weights
The "knowledge" of the model stored as numerical values.
Bigger models (e.g., 70B parameters) are smarter but require more computing power.

(C) Inference
The process of generating text from a trained model.
Requires significant computational power if not optimized.

(D) Quantization
Reduces model size by lowering precision (e.g., from 32-bit to 4-bit numbers).
Makes models run faster on weaker hardware but slightly reduces accuracy.
Example: A 7B model can go from 13GB (FP16) to ~4GB (4-bit quantized).

Hardware Requirements
Full models (no quantization): Need powerful GPUs (e.g., NVIDIA A100).
Quantized models (llama.cpp): Can run on a laptop CPU (e.g., Intel i5/i7, Apple M1/M2).

This AI assistant runs completely offline, is lightweight, and built with free tools. It’s not just a project—it’s a reminder that constraints are often the best source of creativity.

So if you're out of memory or disk but high on motivation—go build your own AI assistant.

Enjoyed this article? Buy me a coffee! ☕

Building Fun games with Amazon Q Cli

Isha_17Bhardwaj — Mon, 16 Jun 2025 14:19:52 +0000

INTRODUCTION

As a developer, I’m always looking for tools that can streamline my workflow, automate repetitive tasks, and enhance productivity. When I first discovered Amazon Q CLI, I was intrigued by its promise of simplifying cloud-based development and AI-assisted coding. Little did I know that it would become an indispensable part of my journey in building Quantum Heist, a unique puzzle-strategy game that blends quantum mechanics with a heist adventure.

In this blog, I’ll cover:
✅ How I discovered Amazon Q CLI and its benefits
✅ Step-by-step development of Quantum Heist using Q CLI
✅ Challenges faced and how I overcame them
✅ Why Amazon Q CLI is a game-changer for modern developers

Discovering Amazon Q CLI

Let's start with the very first
What is Amazon Q CLI?
Amazon Q CLI is a command-line interface tool powered by AWS’s AI assistant, Amazon Q. It helps developers:

Generate code using AI prompts Automate cloud deployments Debug and optimize scripts Manage AWS resources efficiently

🔗 Official Installation Guide: [https://docs.aws.amazon.com/amazonq/]

Now the big question arises Why I Started Using It??
And found I can use it , for more than just giving commands and doing my work

I was working on Quantum Heist, a game that required:
Quantum physics simulations (superposition, entanglement)
Complex Python scripting
GitHub repository management

Manually setting up the project structure, debugging, and pushing to GitHub was time-consuming. That’s when I decided to try Amazon Q CLI—and it transformed my workflow.

🚀 **

Building Quantum Heist with Amazon Q CLI

**
Step 1: Project Initialization
Instead of manually creating folders, I used:

✅ What it did:

Created src/, assets/, docs/, and tests/ folders
Generated starter Python files (main.py, game.py, etc.)

Step 2: Writing the Quantum Mechanics Engine
I needed a QuantumSimulator class to handle:

Superposition (multiple states at once) Entanglement (linked objects) Observation (collapsing quantum states)

Instead of coding from scratch, I prompted Q CLI:

✅ Outcome:

Generated a working quantum simulation system
Saved me hours of debugging

Enhancements Added Later:

Entanglement Logic:
Error Handling:

Step 3: Pygame Integration
Q CLI-Generated Boilerplate:

Manual Improvements:
Added Event Handling:

Boom!!!! and the game is ready within few minutes :)

quantum_heist/
├── src/
│   ├── main.py          # Entry point
│   ├── quantum_simulator.py  # Core mechanic
│   └── game.py          # Pygame logic
├── assets/
│   ├── images/          # Sprite placeholder
│   ├── sounds/          # Audio placeholder
│   └── fonts/           # Typography
└── requirements.txt     # Dependencies

Final video of how it works actually https://go.screenpal.com/watch/cT1DIxnXuKb

🚨 Critical Problems Solved

Missing Dependencies: Q CLI detected unresolved imports and auto- generated requirements.txt withnumpy and pygame.
Path Conflicts: Translated WSL paths (/home/@username) to Windows (C:\Users\hp) using q cli translate-path.
Pygame Freezes: Added pygame.event.pump() after Q CLI identified event-loop bottlenecks.
State Bugs: Enhanced error handling in QuantumSimulator when Q CLI flagged uncaught KeyError cases.

thanks to Amazon Q CLI for:

Being my 2am coding buddy haha
Fixing bugs before I even noticed them
Making me look way smarter than I am

For
🔗 Full code and setup of this game I have made a dedicated repo https://github.com/IshaSri-17Speed/quantum-heist
🔗 Q CLI Docs: aws.amazon.com/q

May your quantum states always collapse in your favor! Until next time, keep entangling those bits!

Enjoyed this article? Buy me a coffee! ☕