I built a small model router that treats Oxlo.ai's catalog as quantization tiers, automatically picking the smallest model that can handle each request. This gives you the speed and cost benefits of weight quantization without the DevOps overhead of hosting your own GGUF files. In this tutorial we will build that router from scratch.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK and tabulate:
pip install openai tabulate - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Define the benchmark suite
We need a small set of prompts that cover the three workload types we care about: factual lookup, code generation, and multi-step reasoning. Each prompt will be sent to every model in our tier list.
BENCHMARKS = [
{
"id": "facts",
"prompt": "What is the capital of France and what river runs through it?",
},
{
"id": "coding",
"prompt": "Write a Python function that checks if a string is a palindrome. Include docstrings.",
},
{
"id": "reasoning",
"prompt": "A farmer has 17 sheep and all but 9 die. How many are left? Explain your answer.",
},
]Step 2: Set up the Oxlo.ai client
We initialize the OpenAI-compatible client pointing at Oxlo.ai. I keep the model tiers in a list ordered from smallest to largest so we can stop at the first one that passes quality checks.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
MODEL_TIERS = [
"deepseek-v3.2", # smallest, efficient for coding and simple tasks
"qwen-3-32b", # mid-size, strong multilingual reasoning
"llama-3.3-70b", # general-purpose flagship
"kimi-k2.6", # largest, advanced reasoning and agentic coding
]Step 3: Run the multi-model evaluation
This function loops over every model and prompt, records the response, and tracks latency. We use a low temperature to keep outputs deterministic during testing.
import time
def evaluate_prompt(model, user_message):
start = time.time()
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a helpful assistant. Answer concisely."},
{"role": "user", "content": user_message},
],
temperature=0.2,
)
latency = time.time() - start
return {
"model": model,
"output": response.choices[0].message.content,
"latency": round(latency, 2),
}Step 4: Grade outputs with a judge model
We need an objective score for each response. I use kimi-k2.6 as a judge because it handles complex evaluation well. The judge receives only the original prompt and the candidate response, then returns an integer score from 1 to 5.
JUDGE_PROMPT = """
Question: {question}
Candidate Answer: {answer}
Rate the candidate answer for accuracy and completeness on a scale of 1 to 5.
Respond with only the integer score.
"""
def score_response(question, answer):
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": "You are an expert evaluator. Be strict."},
{"role": "user", "content": JUDGE_PROMPT.format(question=question, answer=answer)},
],
temperature=0.0,
)
text = response.choices[0].message.content.strip()
try:
return int(text[0])
except (ValueError, IndexError):
return 1With the grader in place, we can run the full benchmark and build a lookup table that maps each workload category to the smallest model that scored 4 or higher.
from tabulate import tabulate
results = []
for bench in BENCHMARKS:
for model in MODEL_TIERS:
result = evaluate_prompt(model, bench["prompt"])
score = score_response(bench["prompt"], result["output"])
results.append({
"category": bench["id"],
"model": model,
"score": score,
"latency": result["latency"],
})
print(tabulate(results, headers="keys"))Step 5: Build the production router
Now we harden the results into a router. I use a lightweight classifier prompt to pick the workload category at runtime, then select the corresponding model from our benchmark results. This is the system prompt for the classifier agent.
CLASSIFIER_SYSTEM_PROMPT = """
You are a query classifier. Analyze the user request and choose exactly one category: facts, coding, or reasoning.
Output only the category name, lowercase, with no punctuation.
"""The router class below first classifies the request, then forwards it to the appropriate Oxlo.ai model. Because Oxlo.ai uses flat per-request pricing, the extra classification call adds a single predictable cost, not a token-scaled surcharge.
class QuantizationRouter:
def __init__(self):
self.client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
self.tier_map = {
"facts": "deepseek-v3.2",
"coding": "qwen-3-32b",
"reasoning": "llama-3.3-70b",
}
def classify(self, user_message):
response = self.client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": CLASSIFIER_SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
temperature=0.0,
)
category = response.choices[0].message.content.strip().lower()
return self.tier_map.get(category, "llama-3.3-70b")
def chat(self, user_message):
model = self.classify(user_message)
response = self.client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
],
)
return {
"model_used": model,
"content": response.choices[0].message.content,
}Run it
Here is how to call the finished agent. I use a coding question because that is where the router saves the most latency by choosing qwen-3-32b instead of always defaulting to the largest model.
router = QuantizationRouter()
result = router.chat("Write a Python function that flattens a nested list of arbitrary depth.")
print(f"Model selected: {result['model_used']}")
print(f"Response:\n{result['content']}")Expected output:
Model selected: qwen-3-32b
Response:
```python
def flatten(nested):
"""Flatten a nested list of arbitrary depth."""
result = []
for item in nested:
if isinstance(item, list):
result.extend(flatten(item))
else:
result.append(item)
return result
```
Next steps
Wire this router into a FastAPI middleware or an async worker so every incoming request is automatically tiered. You can also add a fallback loop that retries with kimi-k2.6 whenever the user provides negative feedback. Because Oxlo.ai charges per request rather than per token, running this classify-then-generate pipeline costs exactly two requests, which makes the economics easy to forecast for high-volume agentic workloads. For exact plan details, see https://oxlo.ai/pricing.
Top comments (0)