DEV Community

shashank ms
shashank ms

Posted on

Choosing the Best LLM Model for Your Use Case with Oxlo

#learnai #oxlo #ai

I built a small model router that reads a task description, picks the most appropriate Oxlo.ai model for the job, and runs it. Because Oxlo.ai uses flat per-request pricing instead of token-based metering, switching models between calls does not create cost surprises, so you can route freely. In this tutorial I will walk through the Python script so you can adapt it to your own stack.

What you'll need

Step 1: Define the Model Catalog

First I map task types to the Oxlo.ai models I actually want to consider. I keep the descriptions short so the router has clear signal.

MODEL_CATALOG = {
    "coding": {
        "id": "deepseek-v3.2",
        "name": "DeepSeek V3.2",
        "context": "Coding, debugging, and structured reasoning. Free-tier eligible."
    },
    "reasoning": {
        "id": "qwen-3-32b",
        "name": "Qwen 3 32B",
        "context": "Multilingual reasoning, agent workflows, and tool use."
    },
    "general": {
        "id": "llama-3.3-70b",
        "name": "Llama 3.3 70B",
        "context": "General-purpose chat, summarization, and creative writing."
    },
    "vision": {
        "id": "kimi-k2.6",
        "name": "Kimi K2.6",
        "context": "Vision inputs, long-context analysis up to 131K tokens, and agentic coding."
    }
}

Step 2: Initialize the Oxlo.ai Client

The client is a drop-in replacement for the OpenAI SDK. We only need to swap the base URL and plug in the Oxlo.ai API key.

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # from https://portal.oxlo.ai
)

Step 3: Write the Router System Prompt

This prompt is the core logic of the agent. It tells the model to classify the user request and return JSON containing the chosen category and a one-sentence justification.

ROUTER_SYSTEM_PROMPT = """
You are a model router. Your job is to analyze the user's request and pick the single best model category from the catalog below. Return ONLY a JSON object with two keys: "category" and "reason".

Catalog:
- coding: deepseek-v3.2 - Coding, debugging, and structured reasoning. Free-tier eligible.
- reasoning: qwen-3-32b - Multilingual reasoning, agent workflows, and tool use.
- general: llama-3.3-70b - General-purpose chat, summarization, and creative writing.
- vision: kimi-k2.6 - Vision inputs, long-context analysis up to 131K tokens, and agentic coding.

Rules:
1. If the user mentions an image, screenshot, or URL to an image, pick "vision".
2. If the user asks for code, a script, or debugging, pick "coding".
3. If the user asks for logic puzzles, math, or step-by-step analysis, pick "reasoning".
4. Otherwise pick "general".
"""

Step 4: Classify the Task

I use Qwen 3 32B for the router itself because it handles agent workflows well. I set the temperature low and request JSON mode so the output is predictable.

def classify_task(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": ROUTER_SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    content = response.choices[0].message.content
    return json.loads(content)

Step 5: Execute on the Selected Model

Once the classifier returns a category, I look up the Oxlo.ai model ID and stream the final response from that model. Oxlo.ai has no cold starts on these popular models, so the handoff feels instant.

def run_agent(user_message: str):
    selection = classify_task(user_message)
    category = selection.get("category", "general")
    reason = selection.get("reason", "No reason provided.")

    model_info = MODEL_CATALOG.get(category, MODEL_CATALOG["general"])
    model_id = model_info["id"]

    print(f"Router chose: {model_id} ({category})")
    print(f"Reason: {reason}\n")
    print("--- Response ---")

    stream = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": user_message},
        ],
        stream=True,
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
    print()

Run it

Here is how I test the finished agent with two different requests. The first is a coding task, and the second is a general question.

if __name__ == "__main__":
    print("=== Test 1: Coding task ===")
    run_agent("Write a Python function that flattens a nested list of arbitrary depth.")

    print("\n=== Test 2: General task ===")
    run_agent("Summarize the key benefits of flat per-request pricing for LLM APIs.")

Example output:

=== Test 1: Coding task ===
Router chose: deepseek-v3.2 (coding)
Reason: The user asked for a Python function, which is a coding task.

--- Response ---
Here is a concise Python function that flattens a nested list using recursion...

=== Test 2: General task ===
Router chose: llama-3.3-70b (general)
Reason: The request is a high-level summary question with no code or images involved.

--- Response ---
Flat per-request pricing simplifies cost forecasting because the price does not scale with prompt length...

Next steps

Try adding a fallback to Llama 3.3 70B when the classifier confidence is low, or extend the catalog with GLM 5 for long-horizon agentic tasks. You could also cache the category choice in Redis for repeated query patterns so you skip the router call entirely.

Top comments (0)