LLM Model Selection for Text Generation: Best Practices and Considerations

#learnai #oxlo #ai

Today I am building a task-aware model router that automatically selects the best Oxlo.ai LLM for any text generation job. Instead of hardcoding model names, the system profiles each request and matches it to the cheapest capable model. This saves money and cuts latency for production workloads that mix simple chat, deep reasoning, and code generation.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Define the model registry

I start by mapping Oxlo.ai models to their strengths. I keep this as plain Python config so I can add new models without touching the router logic.

MODEL_REGISTRY = {
    "deepseek-v3.2": {
        "strengths": {"coding", "reasoning", "tool_use"},
        "context": 128000,
        "cost_tier": "free",
        "latency_profile": "fast"
    },
    "llama-3.3-70b": {
        "strengths": {"general_chat", "instruction_following", "summarization"},
        "context": 128000,
        "cost_tier": "pro",
        "latency_profile": "balanced"
    },
    "qwen-3-32b": {
        "strengths": {"multilingual", "agent_workflows", "reasoning"},
        "context": 128000,
        "cost_tier": "pro",
        "latency_profile": "balanced"
    },
    "kimi-k2.6": {
        "strengths": {"advanced_reasoning", "coding", "vision", "long_context"},
        "context": 131000,
        "cost_tier": "premium",
        "latency_profile": "slower"
    }
}

Step 2: Classify incoming requests

I use a fast, cheap model to tag every incoming prompt with required capabilities. Because Oxlo.ai uses flat per-request pricing, this classifier call costs the same whether the user message is ten tokens or ten thousand.

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

CLASSIFIER_PROMPT = """You are a capability classifier. Analyze the user request and output a JSON object with exactly these keys:
- "task_type": one of ["general_chat", "coding", "reasoning", "multilingual", "vision", "summarization", "agent_workflow"]
- "complexity": one of ["simple", "standard", "advanced"]
- "needs_vision": boolean
- "preferred_latency": one of ["fast", "balanced", "no_preference"]

Output ONLY the JSON object, no markdown fences."""

def classify_task(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256,
    )
    return json.loads(response.choices[0].message.content)

Step 3: Score and select a model

I score every model in the registry against the classified requirements, then return the best match. This keeps selection logic transparent and easy to tune.

def select_model(classification: dict) -> str:
    task_type = classification["task_type"]
    complexity = classification["complexity"]
    needs_vision = classification["needs_vision"]

    scores = {}
    for model_id, meta in MODEL_REGISTRY.items():
        score = 0
        if task_type in meta["strengths"]:
            score += 10
        if needs_vision and "vision" in meta["strengths"]:
            score += 20
        if complexity == "advanced" and meta["cost_tier"] == "premium":
            score += 5
        elif complexity == "simple" and meta["latency_profile"] == "fast":
            score += 5
        if meta["cost_tier"] == "free":
            score += 2
        scores[model_id] = score

    return max(scores, key=scores.get)

Step 4: Generate with the selected model

I keep one system prompt that works across all models. Here it is in isolation so you can tune it without touching the client code.

SYSTEM_PROMPT = """You are a helpful research assistant. Answer the user's question accurately and concisely. If the question involves code, provide working examples. If reasoning is required, show your chain of thought briefly."""

Then I wire the selected model into the chat completion call.

def generate(user_message: str) -> dict:
    classification = classify_task(user_message)
    model = select_model(classification)

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.3,
    )

    return {
        "model_used": model,
        "classification": classification,
        "content": response.choices[0].message.content,
    }

Step 5: Batch evaluation

To verify the router, I run a diverse batch of prompts and log the decisions. This is how I catch misclassifications before shipping to production.

TEST_PROMPTS = [
    "Explain Python list comprehensions with a short example.",
    "Write a React component that fetches data from an API and handles loading states.",
    "A farmer has 17 sheep and all but 9 die. How many are left? Explain your reasoning.",
    "Summarize the benefits of request-based pricing for LLM APIs in two sentences.",
    "Translate the following English text to Japanese: 'Hello, how are you?'",
]

def run_evaluation():
    for prompt in TEST_PROMPTS:
        result = generate(prompt)
        print(f"Task: {result['classification']['task_type']} | "
              f"Complexity: {result['classification']['complexity']} | "
              f"Model: {result['model_used']}")
        print(f"Preview: {result['content'][:120]}...\n")

if __name__ == "__main__":
    run_evaluation()

Run it

Save the script as model_router.py, export your key, and run it.

export OXLO_API_KEY="sk-oxlo.ai-..."
python model_router.py

Example output:

Task: coding | Complexity: simple | Model: deepseek-v3.2
Preview: Here is a concise example: squares = [x**2 for x in range(10)]. This creates a list of squares from 0 to 9...

Task: coding | Complexity: advanced | Model: kimi-k2.6
Preview: import React, { useState, useEffect } from 'react'; function DataFetcher() { const [data, setData] = useState(null); ...

Task: reasoning | Complexity: advanced | Model: kimi-k2.6
Preview: Let me think through this carefully. The phrase "all but 9 die" means that 9 sheep survive. Therefore, the answer is 9...

Task: general_chat | Complexity: simple | Model: deepseek-v3.2
Preview: Request-based pricing charges a flat fee per API call, which makes costs predictable and often cheaper for long inputs...

Task: multilingual | Complexity: standard | Model: qwen-3-32b
Preview: こんにちは、お元気ですか？...

Wrap-up

That is the core of a production model router on Oxlo.ai. Two concrete next steps: add a feedback loop that logs thumbs-up or thumbs-down per model choice and retrains the scoring weights, and wrap the script in an async FastAPI service so classification and generation can run in parallel for high-throughput workloads.

DEV Community