Introduction to LLM Inference Engines for Developers

#learnai #oxlo #ai

We are building a lightweight model router that picks the best Oxlo.ai inference engine for any developer question. If you are deciding which model powers your next feature, this script lets you see how different engines handle the same workload through one flat-priced API.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Verify Connectivity to Oxlo.ai

First, I check that the SDK can reach Oxlo.ai and that my key is active. I will use Llama 3.3 70B, the general-purpose flagship, for a quick smoke test.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "user", "content": "Say 'Oxlo.ai is up' and nothing else."},
    ],
)

print(response.choices[0].message.content)

Step 2: Define the Router System Prompt

The agent needs to classify the user request into one of four buckets so it can choose the right engine. Here is the prompt I use.

SYSTEM_PROMPT = """You are a routing layer.
Classify the user's request into exactly one category:
code, reasoning, multilingual, or general.
Respond with only the category name, lowercase, no punctuation."""

Step 3: Classify the Request

I send the user question to the classifier. I use Qwen 3 32B because it handles multilingual reasoning and agent workflows well, which makes it a fast and accurate tagger.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def classify_request(user_message: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.0,
        max_tokens=10,
    )
    category = response.choices[0].message.content.strip().lower()
    return category

# Test
user = "Write a Python function that merges two sorted lists."
print(classify_request(user))

Step 4: Route to the Target Engine

Now I map the category to the model ID that fits it best. DeepSeek V3.2 for code, Kimi K2.6 for reasoning, Qwen 3 32B for multilingual, and Llama 3.3 70B as the fallback. Oxlo.ai uses the same endpoint for all of them, so I only change the model string.

MODEL_MAP = {
    "code": "deepseek-v3.2",
    "reasoning": "kimi-k2.6",
    "multilingual": "qwen-3-32b",
    "general": "llama-3.3-70b",
}

def run_agent(user_message: str) -> str:
    category = classify_request(user_message)
    model = MODEL_MAP.get(category, "llama-3.3-70b")

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": user_message},
        ],
    )

    answer = response.choices[0].message.content
    return f"[Engine: {model} | Category: {category}]\n\n{answer}"

# Test
print(run_agent("Explain the trade-offs between breadth-first and depth-first search."))

Step 5: Add an Interactive Loop

I wrap the router in a small CLI so I can keep testing prompts without restarting the script.

if __name__ == "__main__":
    print("Multi-engine router ready. Type 'exit' to quit.")
    while True:
        user_input = input("\nQuestion: ").strip()
        if user_input.lower() == "exit":
            break
        print(run_agent(user_input))

Run It

Save the complete script as router.py and run it. Because Oxlo.ai uses flat per-request pricing, you can experiment with multiple models without scaling costs by token count. See the exact rates at https://oxlo.ai/pricing.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a routing layer.
Classify the user's request into exactly one category:
code, reasoning, multilingual, or general.
Respond with only the category name, lowercase, no punctuation."""

MODEL_MAP = {
    "code": "deepseek-v3.2",
    "reasoning": "kimi-k2.6",
    "multilingual": "qwen-3-32b",
    "general": "llama-3.3-70b",
}

def classify_request(user_message: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.0,
        max_tokens=10,
    )
    return response.choices[0].message.content.strip().lower()

def run_agent(user_message: str) -> str:
    category = classify_request(user_message)
    model = MODEL_MAP.get(category, "llama-3.3-70b")
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": user_message},
        ],
    )
    answer = response.choices[0].message.content
    return f"[Engine: {model} | Category: {category}]\n\n{answer}"

if __name__ == "__main__":
    print("Multi-engine router ready. Type 'exit' to quit.")
    while True:
        user_input = input("\nQuestion: ").strip()
        if user_input.lower() == "exit":
            break
        print(run_agent(user_input))

python router.py

Example session:

Multi-engine router ready. Type 'exit' to quit.

Question: Write a Python function that merges two sorted lists.
[Engine: deepseek-v3.2 | Category: code]

def merge_sorted(a, b):
    merged = []
    i = j = 0
    while i < len(a) and j < len(b):
        if a[i] < b[j]:
            merged.append(a[i])
            i += 1
        else:
            merged.append(b[j])
            j += 1
    merged.extend(a[i:])
    merged.extend(b[j:])
    return merged

Question: Explain the trade-offs between breadth-first and depth-first search.
[Engine: kimi-k2.6 | Category: reasoning]

BFS guarantees the shortest path in unweighted graphs because it explores all neighbors at the present depth before moving deeper. DFS uses less memory because it only stores the current path, but it can get stuck exploring deep branches first. Choose BFS when you need the optimal path and the branching factor is manageable. Choose DFS when memory is constrained or you want to detect cycles.

Question: exit

Next Steps

Swap in DeepSeek R1 671B for the reasoning category if you need deep chain-of-thought before the final answer. You could also add response streaming by setting stream=True in the chat completion call and iterating over chunks. If you want to run this as a service, the Oxlo.ai endpoint is fully compatible with the OpenAI Python SDK, so you can drop it into an existing FastAPI or Flask app without vendor lock-in.