We are building a small benchmark runner that sends hard reasoning, coding, and multilingual prompts to several frontier models hosted on Oxlo.ai, then scores the answers with a local judge. If you are trying to decide which flagship model actually fits your product, this gives you a reproducible lab instead of a leaderboard screenshot.
What you'll need
- Python 3.10 or newer
pip install openai- An Oxlo.ai API key from https://portal.oxlo.ai
Oxlo.ai uses request-based pricing, so running a 20-request test batch costs the same whether your prompts are 500 tokens or 50,000 tokens. For a benchmark that often involves long system prompts and context, that makes experimentation predictable.
Step 1: Configure the Oxlo.ai client and model lineup
I picked four models from Oxlo.ai that represent different frontier specializations. Llama 3.3 70B is the general-purpose workhorse, DeepSeek R1 671B handles deep reasoning, Kimi K2.6 targets agentic coding, and Qwen 3 32B covers multilingual workflows.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
FRONTIER_MODELS = [
"llama-3.3-70b",
"deepseek-r1-671b",
"kimi-k2.6",
"qwen-3-32b",
]Step 2: Define the evaluation tasks
Frontier performance is not one number. I use three prompts that stress different capabilities: chain-of-thought reasoning, a Python scripting task, and a Chinese-to-English translation with technical jargon.
BENCHMARK_TASKS = [
{
"name": "logic_puzzle",
"prompt": (
"Three friends, Alma, Ben, and Chris, live in different cities: Paris, Tokyo, or New York. "
"Alma does not live in Paris. Ben does not live in Tokyo. Chris does not live in New York. "
"The person in Paris is not Alma or Chris. Where does each person live? Think step by step."
),
},
{
"name": "code_generation",
"prompt": (
"Write a Python function that takes a list of file paths and returns a dict mapping "
"file extension to total size in bytes, efficiently handling directories and broken symlinks."
),
},
{
"name": "multilingual_translation",
"prompt": (
"Translate the following sentence from Chinese to English, preserving technical precision: "
"'我们使用744B参数的MoE模型进行长上下文推理，上下文窗口可达1M tokens。'"
),
},
]Step 3: Collect answers from each frontier model
We loop over tasks and models, calling Oxlo.ai for each combination. Because Oxlo.ai keeps popular models warm, we do not need to handle cold-start delays.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = "You are an expert assistant. Answer concisely and accurately."
def get_answer(model_id: str, user_prompt: str) -> str:
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
],
)
return response.choices[0].message.content
results = []
for task in BENCHMARK_TASKS:
for model in FRONTIER_MODELS:
answer = get_answer(model, task["prompt"])
results.append({
"task": task["name"],
"model": model,
"answer": answer,
})Step 4: Define the judge system prompt
To compare outputs without reading every line, we use Llama 3.3 70B as a judge. The system prompt below forces a structured 1-to-5 score and a one-sentence justification.
JUDGE_SYSTEM_PROMPT = (
"You are a strict evaluator. Rate the candidate answer on a scale of 1 to 5, "
"where 5 is fully correct, well-reasoned, and clearly written. "
"Respond in exactly this format: Score: X/5. Reason: one sentence."
)Step 5: Score answers with the judge
We pass each candidate answer back to Oxlo.ai alongside the original task so the judge can assess correctness.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
def score_answer(task_name: str, task_prompt: str, candidate: str) -> str:
user_message = (
f"Task ({task_name}): {task_prompt}\n\n"
f"Candidate answer: {candidate}\n\n"
"Provide your rating."
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.content
for r in results:
task_obj = next(t for t in BENCHMARK_TASKS if t["name"] == r["task"])
r["score"] = score_answer(r["task"], task_obj["prompt"], r["answer"])Step 6: Print the report
Finally, we render a simple text table. This makes it easy to spot which model wins on which task shape.
print(f"{'Task':<25} {'Model':<20} {'Score'}")
print("-" * 60)
for r in results:
print(f"{r['task']:<25} {r['model']:<20} {r['score']}")Run it
Save the script as benchmark.py, replace YOUR_OXLO_API_KEY, and run:
python benchmark.pyExample output from my last run:
Task Model Score
------------------------------------------------------------
logic_puzzle llama-3.3-70b Score: 5/5. Reason: Correctly deduced all three cities with valid logic.
logic_puzzle deepseek-r1-671b Score: 5/5. Reason: Step-by-step elimination is accurate and complete.
logic_puzzle kimi-k2.6 Score: 5/5. Reason: Correct final mapping with clear reasoning.
logic_puzzle qwen-3-32b Score: 5/5. Reason: Accurate deduction in logical order.
code_generation llama-3.3-70b Score: 4/5. Reason: Solid function but omits broken-symlink handling.
code_generation deepseek-r1-671b Score: 4/5. Reason: Good structure, slightly verbose error handling.
code_generation kimi-k2.6 Score: 5/5. Reason: Handles symlinks and directories correctly with os.walk.
code_generation qwen-3-32b Score: 4/5. Reason: Correct logic, missing docstring and type hints.
multilingual_translation llama-3.3-70b Score: 3/5. Reason: Missed the exact meaning of 1M context window.
multilingual_translation deepseek-r1-671b Score: 4/5. Reason: Accurate but slightly awkward phrasing on MoE.
multilingual_translation kimi-k2.6 Score: 4/5. Reason: Good technical terms, could be more natural.
multilingual_translation qwen-3-32b Score: 5/5. Reason: Precise translation preserving all numeric and technical details.
Wrap-up
With this runner, you can swap in new models as Oxlo.ai adds them, or add vision tasks using Kimi VL A3B and Gemma 3 27B. A natural next step is to turn the results into a routing layer that picks the best model per task type so your application always uses the cheapest frontier performer for the job.
Top comments (0)