I built a model router that automatically picks the best Oxlo.ai model for any incoming task. If you run mixed workloads, this saves you from maintaining separate provider integrations or guessing which checkpoint fits a prompt. The router is itself an LLM call on Oxlo.ai, and because pricing is request-based rather than token-based, adding this classification stage does not inflate costs on long inputs.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Define the model registry
Start by defining the model registry. This maps each Oxlo.ai model to its strengths so the router has a structured menu to choose from.
MODEL_REGISTRY = {
"deepseek-v3.2": {
"strengths": ["coding", "debugging", "technical reasoning"],
"description": "DeepSeek V3.2, strong for coding and reasoning"
},
"qwen-3-32b": {
"strengths": ["multilingual text", "agent workflows", "long-context reasoning"],
"description": "Qwen 3 32B, ideal for multilingual and agentic tasks"
},
"kimi-k2.6": {
"strengths": ["advanced reasoning", "agentic coding", "vision", "long context"],
"description": "Kimi K2.6, handles complex reasoning and 131K context"
},
"llama-3.3-70b": {
"strengths": ["general purpose", "chat", "fast inference"],
"description": "Llama 3.3 70B, general-purpose flagship with no cold starts"
},
}
if __name__ == "__main__":
print(f"Loaded {len(MODEL_REGISTRY)} models from Oxlo.ai")Step 2: Write the router system prompt
The router is just another LLM call with a strict system prompt. I force JSON mode so the output is predictable and easy to parse.
SYSTEM_PROMPT = """You are a model router. Select exactly one model ID from the registry that best fits the user's task.
Registry:
- deepseek-v3.2: coding, debugging, technical reasoning
- qwen-3-32b: multilingual text, agent workflows, long-context reasoning
- kimi-k2.6: advanced reasoning, agentic coding, vision, 131K context
- llama-3.3-70b: general-purpose chat, fast inference, broad knowledge
Respond with valid JSON in this exact format:
{"model_id": "", "reason": ""}
Rules:
- If the task involves writing or analyzing code, choose deepseek-v3.2.
- If the task is in a non-English language or requires an agent loop, choose qwen-3-32b.
- If the task requires deep reasoning, complex math, or image understanding, choose kimi-k2.6.
- For everything else, default to llama-3.3-70b.
"""Step 3: Build the classifier function
Next, wire up the classifier function. I use Llama 3.3 70B as the judge because it has no cold starts and handles structured output reliably.
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
MODEL_REGISTRY = {
"deepseek-v3.2": {
"strengths": ["coding", "debugging", "technical reasoning"],
"description": "DeepSeek V3.2, strong for coding and reasoning"
},
"qwen-3-32b": {
"strengths": ["multilingual text", "agent workflows", "long-context reasoning"],
"description": "Qwen 3 32B, ideal for multilingual and agentic tasks"
},
"kimi-k2.6": {
"strengths": ["advanced reasoning", "agentic coding", "vision", "long context"],
"description": "Kimi K2.6, handles complex reasoning and 131K context"
},
"llama-3.3-70b": {
"strengths": ["general purpose", "chat", "fast inference"],
"description": "Llama 3.3 70B, general-purpose flagship with no cold starts"
},
}
SYSTEM_PROMPT = """You are a model router. Select exactly one model ID from the registry that best fits the user's task.
Registry:
- deepseek-v3.2: coding, debugging, technical reasoning
- qwen-3-32b: multilingual text, agent workflows, long-context reasoning
- kimi-k2.6: advanced reasoning, agentic coding, vision, 131K context
- llama-3.3-70b: general-purpose chat, fast inference, broad knowledge
Respond with valid JSON in this exact format:
{"model_id": "", "reason": ""}
Rules:
- If the task involves writing or analyzing code, choose deepseek-v3.2.
- If the task is in a non-English language or requires an agent loop, choose qwen-3-32b.
- If the task requires deep reasoning, complex math, or image understanding, choose kimi-k2.6.
- For everything else, default to llama-3.3-70b.
"""
def select_model(user_message: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
max_tokens=256,
)
return json.loads(response.choices[0].message.content)
if __name__ == "__main__":
test = select_model("Write a Python decorator that measures execution time.")
print(json.dumps(test, indent=2))Step 4: Implement the execution layer
Once the router returns a model ID, forward the original user message to that model. Oxlo.ai is fully OpenAI SDK compatible, so the only change is the model string.
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
MODEL_REGISTRY = {
"deepseek-v3.2": {
"strengths": ["coding", "debugging", "technical reasoning"],
"description": "DeepSeek V3.2, strong for coding and reasoning"
},
"qwen-3-32b": {
"strengths": ["multilingual text", "agent workflows", "long-context reasoning"],
"description": "Qwen 3 32B, ideal for multilingual and agentic tasks"
},
"kimi-k2.6": {
"strengths": ["advanced reasoning", "agentic coding", "vision", "long context"],
"description": "Kimi K2.6, handles complex reasoning and 131K context"
},
"llama-3.3-70b": {
"strengths": ["general purpose", "chat", "fast inference"],
"description": "Llama 3.3 70B, general-purpose flagship with no cold starts"
},
}
SYSTEM_PROMPT = """You are a model router. Select exactly one model ID from the registry that best fits the user's task.
Registry:
- deepseek-v3.2: coding, debugging, technical reasoning
- qwen-3-32b: multilingual text, agent workflows, long-context reasoning
- kimi-k2.6: advanced reasoning, agentic coding, vision, 131K context
- llama-3.3-70b: general-purpose chat, fast inference, broad knowledge
Respond with valid JSON in this exact format:
{"model_id": "", "reason": ""}
Rules:
- If the task involves writing or analyzing code, choose deepseek-v3.2.
- If the task is in a non-English language or requires an agent loop, choose qwen-3-32b.
- If the task requires deep reasoning, complex math, or image understanding, choose kimi-k2.6.
- For everything else, default to llama-3.3-70b.
"""
def select_model(user_message: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
max_tokens=256,
)
return json.loads(response.choices[0].message.content)
def run_task(user_message: str) -> dict:
router_result = select_model(user_message)
model_id = router_result["model_id"]
if model_id not in MODEL_REGISTRY:
model_id = "llama-3.3-70b"
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
],
temperature=0.7,
)
return {
"model_used": model_id,
"router_reason": router_result["reason"],
"content": response.choices[0].message.content,
}
if __name__ == "__main__":
result = run_task("Explain how MoE architectures reduce inference cost.")
print(f"Router: {result['model_used']}")
print(f"Why: {result['router_reason']}")
print(result["content"][:400])Step 5: Add a head-to-head benchmark
Finally, add a benchmark loop. Running the same prompt through every model makes the tradeoffs concrete and validates your routing rules.
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
MODEL_REGISTRY = {
"deepseek-v3.2": {
"strengths": ["coding", "debugging", "technical reasoning"],
"description": "DeepSeek V3.2, strong for coding and reasoning"
},
"qwen-3-32b": {
"strengths": ["multilingual text", "agent workflows", "long-context reasoning"],
"description": "Qwen 3 32B, ideal for multilingual and agentic tasks"
},
"kimi-k2.6": {
"strengths": ["advanced reasoning", "agentic coding", "vision", "long context"],
"description": "Kimi K2.6, handles complex reasoning and 131K context"
},
"llama-3.3-70b": {
"strengths": ["general purpose", "chat", "fast inference"],
"description": "Llama 3.3 70B, general-purpose flagship with no cold starts"
},
}
SYSTEM_PROMPT = """You are a model router. Select exactly one model ID from the registry that best fits the user's task.
Registry:
- deepseek-v3.2: coding, debugging, technical reasoning
- qwen-3-32b: multilingual text, agent workflows, long-context reasoning
- kimi-k2.6: advanced reasoning, agentic coding, vision, 131K context
- llama-3.3-70b: general-purpose chat, fast inference, broad knowledge
Respond with valid JSON in this exact format:
{"model_id": "", "reason": ""}
Rules:
- If the task involves writing or analyzing code, choose deepseek-v3.2.
- If the task is in a non-English language or requires an agent loop, choose qwen-3-32b.
- If the task requires deep reasoning, complex math, or image understanding, choose kimi-k2.6.
- For everything else, default to llama-3.3-70b.
"""
def select_model(user_message: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
max_tokens=256,
)
return json.loads(response.choices[0].message.content)
def run_task(user_message: str) -> dict:
router_result = select_model(user_message)
model_id = router_result["model_id"]
if model_id not in MODEL_REGISTRY:
model_id = "llama-3.3-70b"
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
],
temperature=0.7,
)
return {
"model_used": model_id,
"router_reason": router_result["reason"],
"content": response.choices[0].message.content,
}
def benchmark(user_message: str):
print(f"Prompt: {user_message}\n")
for model_id in MODEL_REGISTRY.keys():
resp = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_message},
],
temperature=0.7,
)
print(f"--- {model_id} ---")
print(resp.choices[0].message.content[:500])
print()
if __name__ == "__main__":
# Example 1: Route and run
result = run_task("Write a Python function that parses a nested JSON log and returns unique error codes.")
print(f"Router picked: {result['model_used']}")
print(f"Why: {result['router_reason']}")
print(f"Output:\n{result['content']}\n")
# Example 2: Head-to-head benchmark
benchmark("Explain the tradeoffs between request-based and token-based pricing for LLM APIs.")Run it
Save the complete script as router.py, export your key, and run it.
export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python router.pyYou should see output similar to this:
Router picked: deepseek-v3.2
Why: The task explicitly asks for writing Python code to parse JSON.
Output:
```python
import json
def extract_unique_errors(log_data):
errors = set()
...
```
Prompt: Explain the tradeoffs between request-based and token-based pricing for LLM APIs.
--- deepseek-v3.2 ---
Request-based pricing charges a flat fee per API call, while token-based pricing bills per input and output token. For long-context workloads, request-based models like those on Oxlo.ai can be significantly cheaper because the cost does not scale with prompt length.
--- qwen-3-32b ---
Under a request-based model, each API call incurs one fixed cost regardless of token count. Token-based providers bill proportionally to usage, which makes long inputs expensive.
--- kimi-k2.6 ---
The primary distinction is cost predictability. Request-based pricing flattens expenses for large prompts, whereas token-based pricing linearly increases cost with context size.
--- llama-3.3-70b ---
Request-based pricing means one price per call. Token-based means you pay for what you use. Long prompts favor request-based structures.Wrap-up
You now have a single integration point that dynamically selects the best Oxlo.ai model for each task. Because Oxlo.ai uses flat per-request pricing, you can route freely without watching token counters tick up on long prompts. See https://oxlo.ai/pricing for plan details.
Two concrete next steps:
- Add vision support by detecting image paths in the user message and routing those to
kimi-k2.6with a vision-compatible payload. - Cache the router decision in Redis for recurring prompt patterns so you skip the classification call on repeated workload types.
Top comments (0)