Introduction to Open-Weights AI Model

" Your journey is yours to define, your dreams are yours to pursue, and your story is yours to write. " -- with this let me start my first post

This is a new topic I am exploring currently, so decided to share what I understood so far, and will be grateful if you can add or correct me if I am wrong anywhere , so that it will help me as well as any beginner reading this

What is Open-Weight AI Model??

An open-weight AI model is an AI model whose trained parameters — called weights — are made available for people to download, run, and sometimes fine-tune for their own applications.
Some Popular open-weights AI model include -- Gemma (By Google DeepMind) , gpt-oss (By OpenAI), Qwen, Llama (By Meta) and many more.

• SO we can download the weights but how much RAM do we need to run ? The Formula is :

WEIGHT Memory (in GB) = (Parameters * Bits per Parameter) / 8

Parameters - No. of Parameters in billions(B) [For example - 4B,8B,70B]
Bits per Parameters - Bits used to store each parameter [For Example - BF16/FP16(16 bits), INT8(8 bits) ]
8 - number of bytes

Also you need additional memory for KV Cache and additional Runtime Overhead

TOTAL MEMORY = WEIGHT MEMORY + KV CACHE MEMORY + RUNTIME OVERHEAD 

So for example, a 7B model in 4-bit quantization has roughly 3.5 GB of theoretical weight storage. But it may require additional space for KV Cache and other overhaed so it may approx to 7-8GB.

• What is a PARAMETER?

You might be familiar with relationship like this in simple ML:

Price = w1(Area) + w2(Bedrooms) + b
Here w1,w2 are nothing but parameters

A parameter is a numerical value inside an AI model that it learns during training. These values help the model recognize patterns, understand language, write code, and generate answers.

• Does more parameters mean a smarter AI?

Not necessarily. A larger model often has greater capacity to learn complex patterns, but actual performance also depends on:

• Training data quality and quantity.
• Model architecture.
• Training methods.
• Fine-tuning and reasoning capabilities.
• The task you're asking it to perform.

How to run and use a Open-Weight AI model ?

The simplest way is :

• Download a pre-trained model
• Load it in the memory
• Using the loaded model use an interface to interact

The easiest way to do these is by using ollama , vLLM or any other tools

• What are these tools ?

They are the engine and the pipeline that allow you to download open-source models (like Llama 3, Qwen, or Mistral) and run/deploy them yourself on your own hardware.

• What they actually do?

When you install them on a computer:

1. They download the raw brain (the model weights) from open repositories like Hugging Face or the Ollama Library.
2. They load that model into your computer's RAM or GPU memory.
3. They spin up a local network address (like http://localhost:11434 or http://localhost:8000) on your machine.
4. Any app you write can now send text to that address, and your computer will generate the response—100% locally, for free, and completely offline.

Ollama vs vLLM

• Ollama prioritizes simplicity: It acts like Docker for AI models. It manages the downloading, memory cleanup, and execution behind a very clean interface. It uses llama.cpp under the hood to squeeze models onto consumer hardware.

• vLLM prioritizes raw performance: It uses an advanced memory management algorithm called PagedAttention and Continuous Batching. Instead of processing user requests sequentially (like Ollama mostly does under load), vLLM overlaps incoming queries dynamically, achieving up to 20x–24x higher throughput in multi-user environments

HOPE YOU LIKE IT - THIS IS SOMETHING NEW FOR ME TOO - SO WILL BE GRATEFUL IF YOU CAN SHARE YOUR THOUGHTS AND REVIEWS