I recently needed to compare a few open models for an internal reasoning tool. Instead of eyeballing outputs, I built a small LLM-as-a-judge harness that scores answers against a reference set. In this tutorial, we will build the same thing: a Python evaluator that generates candidate responses on Oxlo.ai, then judges them with a separate model call, producing a reproducible scorecard.
What you'll need
Prerequisites are minimal. You will need Python 3.10 or newer, an Oxlo.ai API key from https://portal.oxlo.ai, and the OpenAI SDK installed with pip install openai. You will also want a small credit balance on your Oxlo.ai account. Because Oxlo.ai charges a flat rate per request, you can predict the exact cost of this evaluation before you run it. See https://oxlo.ai/pricing for current rates.
Step 1: Create an evaluation dataset
We need questions with gold-standard answers. I use a tiny in-memory list for this demo, but in production I read from a JSONL file.
EVAL_DATASET = [
{
"question": "A train travels 120 km in 2 hours. How far will it travel in 5 hours at the same speed?",
"reference": "The train travels at 60 km/h. In 5 hours it will travel 300 km."
},
{
"question": "If a basket has 8 apples and you remove 3, how many do you have?",
"reference": "You have 3 apples, the ones you removed."
},
{
"question": "A farmer has 17 sheep and all but 9 die. How many are left?",
"reference": "9 sheep are left."
}
]Step 2: Build the candidate generator
This function calls an Oxlo.ai model to produce an answer. We will test three models, so the model name is a parameter.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = "You are a helpful assistant. Answer the question briefly and clearly."
def generate_answer(model_id: str, question: str) -> str:
user_message = question
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.content.strip()Step 3: Write the judge prompt
The judge needs to compare a candidate answer against the reference and return structured scores. I use a 1-to-10 scale for correctness, clarity, and completeness.
JUDGE_SYSTEM_PROMPT = """You are an expert evaluator. Compare the candidate answer below to the reference answer.
Respond ONLY with a JSON object in this exact format:
{
"correctness": 1-10,
"clarity": 1-10,
"completeness": 1-10,
"reasoning": "one sentence explaining the score"
}
Be strict. If the candidate is wrong, say so."""Step 4: Implement the judge client
Now we send the candidate and reference to a strong reasoning model. I use kimi-k2.6 for the judge because it handles instruction following well.
import json
def judge_answer(candidate: str, reference: str) -> dict:
user_message = f"""Reference answer: {reference}
Candidate answer: {candidate}
Evaluate the candidate answer and return JSON only."""
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
raw = response.choices[0].message.content.strip()
# Handle possible markdown fences
if raw.startswith("
```"):
raw = raw.split("```
")[1].replace("json", "").strip()
return json.loads(raw)Step 5: Run the batch evaluation
We loop over the dataset, generate answers with three different models, and judge each one. I collect scores in a simple dictionary.
def evaluate_model(model_id: str) -> list[dict]:
results = []
for item in EVAL_DATASET:
candidate = generate_answer(model_id, item["question"])
scores = judge_answer(candidate, item["reference"])
results.append({
"question": item["question"],
"candidate": candidate,
"scores": scores,
})
return results
def print_report(model_id: str, results: list[dict]):
print(f"\n=== {model_id} ===")
for r in results:
sc = r["scores"]
print(f"Q: {r['question'][:50]}...")
print(f" correctness={sc['correctness']}, clarity={sc['clarity']}, completeness={sc['completeness']}")
print(f" reasoning: {sc['reasoning']}")
avg_corr = sum(r["scores"]["correctness"] for r in results) / len(results)
avg_clar = sum(r["scores"]["clarity"] for r in results) / len(results)
avg_comp = sum(r["scores"]["completeness"] for r in results) / len(results)
print(f"Averages: correctness={avg_corr:.1f}, clarity={avg_clar:.1f}, completeness={avg_comp:.1f}")Step 6: Track costs and report
Because Oxlo.ai uses flat per-request pricing, the cost of this entire run is easy to calculate. Each generate_answer is one request, and each judge_answer is one request. We do not need to count tokens.
def estimate_cost(num_requests: int):
# Flat per-request pricing means cost scales linearly with API calls.
# See the current rate at https://oxlo.ai/pricing
print(f"\nTotal requests: {num_requests}")
print(f"Cost = {num_requests} * (flat per-request rate)")
print("Check https://oxlo.ai/pricing for the exact amount.")
def main():
models = ["llama-3.3-70b", "qwen-3-32b", "deepseek-v3.2"]
total_requests = 0
for model in models:
results = evaluate_model(model)
print_report(model, results)
total_requests += len(EVAL_DATASET) * 2 # one gen + one judge per question
estimate_cost(total_requests)
if __name__ == "__main__":
main()Run it
Running the script produces a side-by-side scorecard. Here is example output from my last run.
=== llama-3.3-70b ===
Q: A train travels 120 km in 2 hours. How far will it...
correctness=10, clarity=10, completeness=9
reasoning: Correct calculation and clearly explained.
Q: If a basket has 8 apples and you remove 3, how man...
correctness=8, clarity=9, completeness=8
reasoning: Correct but missed the nuance about having the removed apples.
Q: A farmer has 17 sheep and all but 9 die. How many...
correctness=10, clarity=10, completeness=10
reasoning: Perfect answer.
Averages: correctness=9.3, clarity=9.7, completeness=9.0
=== qwen-3-32b ===
Q: A train travels 120 km in 2 hours. How far will it...
correctness=10, clarity=9, completeness=10
reasoning: Accurate and thorough.
Q: If a basket has 8 apples and you remove 3, how man...
correctness=9, clarity=9, completeness=9
reasoning: Good, but slightly verbose.
Q: A farmer has 17 sheep and all but 9 die. How many...
correctness=10, clarity=10, completeness=10
reasoning: Correct.
Averages: correctness=9.7, clarity=9.3, completeness=9.7
=== deepseek-v3.2 ===
Q: A train travels 120 km in 2 hours. How far will it...
correctness=10, clarity=10, completeness=10
reasoning: Correct and concise.
Q: If a basket has 8 apples and you remove 3, how man...
correctness=9, clarity=10, completeness=9
reasoning: Accurate with clear logic.
Q: A farmer has 17 sheep and all but 9 die. How many...
correctness=10, clarity=10, completeness=10
reasoning: Correct.
Averages: correctness=9.7, clarity=10.0, completeness=9.7
Total requests: 18
Cost = 18 * (flat per-request rate)
Check https://oxlo.ai/pricing for the exact amount.Wrap-up and next steps
This harness is already useful for regression testing before deploys. Two concrete next steps: swap the static dataset for a continuously updated evaluation set loaded from S3 or Git, and add a pairwise comparison mode where the judge picks a winner between two model outputs instead of scoring in isolation. Both are straightforward extensions of the same Oxlo.ai client code.
Top comments (0)