Choosing the right text generation model for a workload usually means running the same prompt through several candidates and judging the results. In this tutorial, we will build a lightweight model comparison agent that routes one user task through four different Oxlo.ai models and uses a fifth evaluator run to pick the best output for the job. Everything stays inside the Oxlo.ai API, so you only need one key and one client.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai. The free tier includes 60 requests per day, which is enough to test this script several times.
Step 1: Set up the client and candidate registry
We start with a single OpenAI client pointed at Oxlo.ai. Because Oxlo.ai is fully OpenAI SDK compatible, the only difference is the base URL. We will compare four models that cover distinct strengths: Llama 3.3 70B for general instruction following, Qwen 3 32B for multilingual reasoning, Kimi K2.6 for advanced reasoning and coding, and DeepSeek V3.2 for coding and concise outputs.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)
CANDIDATES = [
{"id": "llama-3.3-70b", "name": "Llama 3.3 70B", "role": "General-purpose"},
{"id": "qwen-3-32b", "name": "Qwen 3 32B", "role": "Multilingual reasoning"},
{"id": "kimi-k2.6", "name": "Kimi K2.6", "role": "Advanced reasoning & coding"},
{"id": "deepseek-v3.2", "name": "DeepSeek V3.2", "role": "Coding & concise reasoning"},
]Step 2: Define the evaluator system prompt
The agent needs a consistent rubric so its judgments are repeatable. This system prompt tells the evaluator to score each candidate on accuracy, clarity, and style, then recommend one winner and explain why.
SYSTEM_PROMPT = """You are an expert ML engineer evaluating LLM outputs.
A user task and several candidate responses are provided below.
Score each candidate on a scale of 1 to 5 for accuracy, clarity, and style.
Then recommend the single best model for this specific task and explain why.
Keep the total analysis under 200 words."""Step 3: Fetch candidate responses in parallel
To keep latency low, we call all four models concurrently. Oxlo.ai has no cold starts on popular models, so the first request warms up just as fast as the tenth.
import concurrent.futures
def fetch_candidate(model_id, user_task):
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": user_task},
],
temperature=0.7,
max_tokens=512,
)
return {
"model_id": model_id,
"content": response.choices[0].message.content.strip()
}
def gather_candidates(user_task):
outputs = []
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
futures = {
executor.submit(fetch_candidate, c["id"], user_task): c
for c in CANDIDATES
}
for future in concurrent.futures.as_completed(futures):
try:
outputs.append(future.result())
except Exception as e:
outputs.append({
"model_id": futures[future]["id"],
"content": f"Error: {e}"
})
return outputsStep 4: Build the evaluator payload and run the judge
We format the candidate outputs into a single user message and send it to Llama 3.3 70B with the system prompt from Step 2. I keep the temperature low so the analysis stays deterministic.
def build_evaluator_message(user_task, candidate_outputs):
lines = [f"Task: {user_task}\n", "Candidates:\n"]
for out in candidate_outputs:
lines.append(f"--- {out['model_id']} ---\n{out['content']}\n")
return "\n".join(lines)
def evaluate(user_task, candidate_outputs):
evaluator_message = build_evaluator_message(user_task, candidate_outputs)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": evaluator_message},
],
temperature=0.3,
max_tokens=800,
)
return response.choices[0].message.content.strip()Step 5: Wire everything into a simple CLI
A short main block ties the pipeline together. You can swap the user task string for any prompt you are researching.
if __name__ == "__main__":
user_task = (
"Explain recursion to a junior developer in one paragraph. "
"Include a simple Python example."
)
print("Fetching candidates from Oxlo.ai...\n")
candidates = gather_candidates(user_task)
for c in candidates:
print(f"Model: {c['model_id']}")
print(f"Output: {c['content'][:200]}...\n")
print("Running evaluator...\n")
verdict = evaluate(user_task, candidates)
print(verdict)Run it
Save the file as compare_models.py, export your key, and run python compare_models.py. Because Oxlo.ai uses flat request-based pricing, running this comparison costs one request per model call regardless of prompt length. For long evaluation prompts, that can be significantly cheaper than token-based providers. You can view exact plan details at https://oxlo.ai/pricing.
Example output:
Fetching candidates from Oxlo.ai...
Model: qwen-3-32b
Output: Recursion is when a function calls itself to solve a smaller piece of the same problem. Here is a simple Python example...
Model: deepseek-v3.2
Output: Recursion is a programming pattern where a function invokes itself with modified arguments until reaching a base case...
Running evaluator...
Accuracy scores:
- llama-3.3-70b: 5/5
- qwen-3-32b: 5/5
- kimi-k2.6: 4/5
- deepseek-v3.2: 5/5
Clarity scores:
- llama-3.3-70b: 4/5
- qwen-3-32b: 5/5
- kimi-k2.6: 5/5
- deepseek-v3.2: 4/5
Recommendation: Qwen 3 32B wins for this educational task because its explanation is the most accessible while still being technically correct.Wrap-up
Two concrete next steps. First, add a timing wrapper around each fetch_candidate call so you can compare wall-clock latency across models on Oxlo.ai. Second, store the evaluator verdicts in a SQLite database and track which model wins for each task category over time. That turns the script into a living model-selection benchmark you can query before deploying new features.
Top comments (0)