shashank ms

Posted on Jul 12

Introduction to LLM Model Quantization

#learnai #oxlo #ai

What we are building

Today we are building a quantization impact analyzer. It estimates GPU memory for any open-source LLM at different precision levels, then uses Oxlo.ai to explain the trade-offs and recommend a strategy. If you are deploying models on limited hardware, this saves you from guessing.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai. See https://oxlo.ai/pricing for current plans.
The OpenAI SDK: pip install openai

Step 1: Bootstrap the Oxlo.ai client

We start by importing the SDK and pointing it at Oxlo.ai. Because Oxlo.ai is fully OpenAI-compatible, this is a drop-in replacement.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Define precision constants and the memory calculator

Quantization reduces bit width per parameter. We map common precisions to their bits-per-parameter, then compute raw VRAM plus a 15 percent overhead for activations and system buffers.

PRECISION_MAP = {
    "FP16": 16,
    "INT8": 8,
    "INT4": 4,
    "Q4_K_M": 4.5,
    "Q8_0": 8,
}

def estimate_vram(params_billion, precision):
    bytes_per_param = PRECISION_MAP[precision] / 8
    model_gb = params_billion * bytes_per_param
    return round(model_gb * 1.15, 2)

Step 3: Write the advisor system prompt

The system prompt tells the model how to reason about quantization formats like GGML, GPTQ, and AWQ. We will use Llama 3.3 70B on Oxlo.ai because it follows long instructions reliably.

SYSTEM_PROMPT = """You are a quantization engineer. Given a model and its memory footprint at different precisions, do the following:

1. State the estimated VRAM for each precision level.
2. Explain the trade-off between the top two candidates for a single consumer GPU (e.g., RTX 4090 24GB).
3. Recommend one precision level and justify it based on quality preservation vs. hardware constraints.
4. Mention whether the user should consider multi-GPU sharding instead.

Be concise. Use bullet points."""

Step 4: Build the advisor function

We assemble the user message with the model specs and memory table, then call Oxlo.ai. I use streaming so the response prints as it arrives.

def analyze_quantization(model_name, params_billion, context_k):
    rows = []
    for prec in PRECISION_MAP:
        gb = estimate_vram(params_billion, prec)
        rows.append(f"{prec}: {gb} GB")
    table = "\n".join(rows)

    user_msg = f"""Model: {model_name}
Parameters: {params_billion}B
Desired context: {context_k}k tokens

Estimated VRAM (including 15% activation overhead):
{table}

Which precision should I use for a single-GPU setup?"""

    stream = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_msg},
        ],
        stream=True,
    )

    print("Advisor output:")
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            print(content, end="")
    print()

Step 5: Add a CLI loop

We wrap the advisor in a small CLI so we can test multiple models without restarting the script.

if __name__ == "__main__":
    print("Quantization Advisor")
    print("Enter a model name, param count (B), and context length (k).")
    print("Example: llama-3.3-70b 70 4")
    print("Type 'quit' to exit.\n")

    while True:
        line = input("> ").strip()
        if line.lower() == "quit":
            break
        parts = line.split()
        if len(parts) != 3:
            print("Expected: <model> <params_b> <context_k>")
            continue
        model, params, ctx = parts
        try:
            analyze_quantization(model, float(params), int(ctx))
        except Exception as e:
            print(f"Error: {e}")

Run it

Save the file as quant_advisor.py, export your key, and run it. Here is an example session analyzing a 70B model at 4k context.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python quant_advisor.py
> llama-3.3-70b 70 4

Example output:

Advisor output:
* FP16: 161.0 GB - Requires 2-3 A100s. Best quality, impractical for single GPU.
* Q8_0: 80.5 GB - Fits on A100 80GB with zero headroom.
* Q4_K_M: 45.3 GB - Fits comfortably on a single RTX 4090 24GB with room for KV cache.
* INT4: 40.25 GB - Even smaller, but may degrade reasoning.

Recommendation: Choose Q4_K_M. It preserves most of the model's reasoning capability while fitting inside 24GB VRAM. The 4k context keeps KV cache manageable. If you need maximum accuracy and have multiple GPUs, shard the FP16 version instead.

Wrap-up and next steps

You now have a working quantization advisor backed by Oxlo.ai. Because Oxlo.ai uses flat per-request pricing, running this tool dozens of times with long model cards costs the same regardless of prompt length. For long-context comparisons, this request-based model can be 10-100x cheaper than token-based providers. Two concrete next steps: integrate this into a CI pipeline that checks whether new releases fit your GPU cluster, or extend the calculator to include throughput estimates by adding layer counts and batch size logic.

DEV Community