I built a small CLI agent that asks four questions about your target model and hardware, then returns an exact quantization strategy and the corresponding shell or Python command. It removes the guesswork from choosing between GGUF, AWQ, GPTQ, and MLX bit widths, so you can shrink billion-parameter models for edge or consumer GPU deployments without hunting through documentation.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Scaffold the advisor script
Create a file named quantize_advisor.py. I initialize the Oxlo.ai client first. Because Oxlo.ai uses flat per-request pricing (see https://oxlo.ai/pricing), iterating on long system prompts that include framework documentation does not inflate cost the way token-based billing would.
from openai import OpenAI
import json
import sys
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def advise(model_name, hardware_gb, latency_ms, framework):
# Populated in the steps below
pass
if __name__ == "__main__":
if len(sys.argv) != 5:
print("Usage: python quantize_advisor.py ")
sys.exit(1)
_, model, hw, lat, fw = sys.argv
plan = advise(model, float(hw), float(lat), fw)
print(json.dumps(plan, indent=2))
Step 2: Define the system prompt
The system prompt constrains the model to emit only JSON and grounds it in real quantization practices. I treat this as the single source of truth for the agent's behavior.
SYSTEM_PROMPT = """You are a quantization engineer. The user provides a Hugging Face model name, available memory in GB, a target latency in milliseconds per token, and a target framework.
Respond ONLY with a JSON object containing:
- recommended_method: string (e.g., Q4_K_M, AWQ, GPTQ, MLX_4BIT, INT8)
- bits: number
- group_size: number or null
- reasoning: string, max 2 sentences, no fluff
- generated_code: string, a shell command or Python script that performs the quantization
Rules:
- For llama.cpp, recommend GGUF methods (Q4_K_M, Q5_K_M, Q6_K, Q8_0).
- For transformers, prefer BitsAndBytesConfig 4-bit or pre-quantized AWQ/GPTQ.
- For mlx, use mlx_lm 4-bit or 8-bit.
- Keep reasoning concrete. Do not explain JSON syntax."""
Step 3: Collect constraints and build the user message
I format the CLI arguments into a structured message so the model does not have to guess which value maps to which constraint.
def advise(model_name, hardware_gb, latency_ms, framework):
user_message = (
f"Model: {model_name}\n"
f"Available memory: {hardware_gb} GB\n"
f"Target latency: {latency_ms} ms/token\n"
f"Framework: {framework}"
)
# Next step adds the Oxlo.ai API call
return user_message
Step 4: Call Oxlo.ai and parse the recommendation
I use deepseek-v3.2 because it is strong at coding and reasoning, and Oxlo.ai exposes it through a standard OpenAI compatible endpoint with no cold starts. I also enable JSON mode so the output is predictable.
def advise(model_name, hardware_gb, latency_ms, framework):
user_message = (
f"Model: {model_name}\n"
f"Available memory: {hardware_gb} GB\n"
f"Target latency: {latency_ms} ms/token\n"
f"Framework: {framework}"
)
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
Step 5: Write the generated command to disk
Finally, I persist the generated code and print a human readable summary. Saving the command makes it easy to review before you run it on a multi-gigabyte checkpoint.
if __name__ == "__main__":
if len(sys.argv) != 5:
print("Usage: python quantize_advisor.py ")
sys.exit(1)
_, model, hw, lat, fw = sys.argv
plan = advise(model, float(hw), float(lat), fw)
out_file = "quantize.sh" if fw == "llama.cpp" else "quantize.py"
with open(out_file, "w") as f:
f.write(plan["generated_code"])
print(f"Method: {plan['recommended_method']} ({plan['bits']}-bit)")
print(f"Reasoning: {plan['reasoning']}")
print(f"Saved command to {out_file}")
Run it
Here is a real invocation for quantizing a 70B parameter model on a 24 GB consumer GPU using llama.cpp.
$ python quantize_advisor.py "meta-llama/Llama-3.3-70B-Instruct" 24 500 "llama.cpp"
Method: Q4_K_M (4-bit)
Reasoning: Q4_K_M fits a 70B parameter model into roughly 40 GB of disk and less than 24 GB of RAM at inference time. It preserves most of the model's reasoning quality while hitting the 500 ms/token target on modern CPUs.
Saved command to quantize.sh
Inspect quantize.sh before executing it. In my run it contained the exact convert_hf_to_gguf.py invocation with the correct --outtype q4_k_m flag.
Next steps
Wire this advisor into a CI check that fails the build if the recommended quantization method drops below a specific bit width, or add an evaluation pass that calls an Oxlo.ai full-precision model as a reference and compares perplexity against your newly quantized local weights. If you need a different model for the eval step, Oxlo.ai hosts options like llama-3.3-70b and qwen-3-32b with the same flat per-request pricing and OpenAI compatible endpoints.
Top comments (1)
I appreciate how the
SYSTEM_PROMPTis designed to constrain the model's output to a specific JSON format, ensuring consistency in the recommendations provided by the quantization advisor. The use ofdeepseek-v3.2with JSON mode enabled also seems like a good choice for generating predictable and coding-oriented responses. One potential improvement could be to explore the addition of more fine-grained constraints or preferences in theuser_message, such as specific hardware accelerator support or power consumption targets, to further tailor the quantization strategy to the user's needs. Have you considered incorporating any such additional constraints or parameters into the advisor's decision-making process?