I needed a fast way to compare open-source models for a long-context agent project. Instead of trusting public leaderboards, I built a lightweight evaluation harness that calls multiple models through Oxlo.ai and scores their outputs with a dedicated judge. Because Oxlo.ai charges a flat rate per request, running thirty long-context evals costs the same as thirty short pings, which keeps the budget predictable.
What you'll need
Prerequisites are minimal. You need Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key from https://portal.oxlo.ai. Install the SDK with pip.
pip install openaiStep 1: Define the test suite
I started with five prompts that stress the skills I actually care about: reasoning, coding, and concise instruction following. Each entry stores the user prompt and the expected behavior.
TEST_SUITE = [
{
"id": "reasoning-1",
"prompt": "A farmer has 17 sheep and all but 9 die. How many are left? Explain your reasoning.",
"rubric": "Answer must state 9 sheep remain with clear, correct logic."
},
{
"id": "code-1",
"prompt": "Write a Python function that returns the n-th Fibonacci number using O(log n) time.",
"rubric": "Must provide matrix exponentiation or fast doubling method with correct code."
},
{
"id": "instruction-1",
"prompt": "Summarize the following text in exactly 10 words: The quick brown fox jumps over the lazy dog while the sun sets.",
"rubric": "Summary must be exactly ten words."
},
{
"id": "reasoning-2",
"prompt": "If it takes 5 machines 5 minutes to make 5 widgets, how long does it take 100 machines to make 100 widgets? Explain.",
"rubric": "Answer must be 5 minutes with correct explanation of rates."
},
{
"id": "code-2",
"prompt": "Write a regex that validates an IPv4 address. Return only the regex pattern, no explanation.",
"rubric": "Must be a valid regex matching standard IPv4 dotted-decimal format."
}
]Step 2: Initialize the Oxlo.ai client
The client is a drop-in replacement for the OpenAI SDK. I point base_url to Oxlo.ai and load the key from the environment.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
CANDIDATE_MODELS = [
"llama-3.3-70b",
"qwen-3-32b",
"deepseek-v3.2"
]Step 3: Generate candidate responses
I wrote a small helper that calls a candidate model and returns the raw text. I keep temperature low so the results are stable across runs.
def generate(model_id: str, prompt: str) -> str:
resp = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": prompt}],
temperature=0.2,
max_tokens=1024
)
return resp.choices[0].message.contentStep 4: Build the judge prompt
I use a stronger model on Oxlo.ai, kimi-k2.6, as the judge. It scores each response 1 to 5 against the rubric. The system prompt below is the only prompt engineering in the whole pipeline.
JUDGE_SYSTEM_PROMPT = """You are an expert evaluator. You will be given a user prompt, a scoring rubric, and a candidate response.
Your task:
1. Compare the candidate response against the rubric.
2. Assign an integer score from 1 to 5, where 5 means fully satisfies the rubric.
3. Respond with ONLY a JSON object in this exact format:
{"score": 3, "reason": "brief justification"}
Do not include markdown fences, extra text, or explanations outside the JSON."""Step 5: Score a single response
The judge helper sends the rubric and candidate answer to kimi-k2.6. I parse the JSON it returns.
import json
def judge_score(prompt: str, rubric: str, candidate_answer: str) -> dict:
user_msg = f"User Prompt: {prompt}\n\nRubric: {rubric}\n\nCandidate Response:\n{candidate_answer}"
resp = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": user_msg}
],
temperature=0.1,
max_tokens=512
)
raw = resp.choices[0].message.content.strip()
# Handle possible markdown fences
if raw.startswith("
```"):
raw = raw.split("```
")[1]
if raw.startswith("json"):
raw = raw[4:]
return json.loads(raw.strip())Step 6: Run the full benchmark
Now I loop over every model and every test case, collecting scores. Because Oxlo.ai uses request-based pricing, I know the cost upfront even when the judge prompt is long. There are no cold starts, so the loop finishes quickly.
from statistics import mean
def run_benchmark():
results = {m: [] for m in CANDIDATE_MODELS}
for model in CANDIDATE_MODELS:
print(f"Evaluating {model}...")
for test in TEST_SUITE:
answer = generate(model, test["prompt"])
verdict = judge_score(test["prompt"], test["rubric"], answer)
results[model].append({
"test_id": test["id"],
"score": verdict.get("score", 1),
"reason": verdict.get("reason", "")
})
print("\n=== Leaderboard ===")
for model in CANDIDATE_MODELS:
scores = [r["score"] for r in results[model]]
avg = mean(scores)
print(f"{model}: avg score {avg:.2f} / 5.0")
return results
if __name__ == "__main__":
results = run_benchmark()Run it
Export your key and execute the script.
export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python eval_harness.pyOn my last run the output looked like this. Your exact scores will vary slightly with temperature and model updates.
Evaluating llama-3.3-70b...
Evaluating qwen-3-32b...
Evaluating deepseek-v3.2...
=== Leaderboard ===
llama-3.3-70b: avg score 4.20 / 5.0
qwen-3-32b: avg score 4.40 / 5.0
deepseek-v3.2: avg score 4.60 / 5.0Wrap-up and next steps
This harness is easy to extend. You can add latency tracking by timing each request, or swap in additional Oxlo.ai models such as DeepSeek R1 671B MoE for reasoning-heavy tasks. If you move to long-context evals, the flat per-request pricing at Oxlo.ai becomes a significant advantage over token-based providers. See https://oxlo.ai/pricing for current plan details.
Top comments (0)