I needed a repeatable way to pick a backbone model for a customer support chatbot. Instead of reading leaderboard headlines, I built a small evaluation harness that runs multi-turn conversations through several Oxlo.ai models and scores them with an LLM-as-judge. You can adapt it for any conversational AI use case.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK and a couple of helpers:
pip install openai python-dotenv pandas - Enough quota for a few dozen API calls. Oxlo.ai uses request-based pricing, so you pay per call rather than by token length. See https://oxlo.ai/pricing for details.
Step 1: Scaffold the project and configure the client
Create a file named eval.py and load your Oxlo.ai key. I keep mine in a .env file so I do not accidentally commit it.
import os
import json
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY"),
)
MODELS = [
"llama-3.3-70b",
"qwen-3-32b",
"kimi-k2.6",
"deepseek-v3.2",
]Step 2: Define the conversational test suite
Each test case is a mini conversation plus a rubric. I cover memory, reasoning, instruction following, and honesty because those are where production chatbots usually fail.
TEST_CASES = [
{
"id": "memory-001",
"category": "multi-turn-memory",
"conversation": [
{"role": "user", "content": "Hi, I'm Alex and I'm on the Pro plan."},
{"role": "assistant", "content": "Hello Alex, thanks for letting me know. How can I help you today?"},
{"role": "user", "content": "What plan am I on again?"},
],
"rubric": "The assistant must correctly recall the name Alex and the Pro plan without asking.",
},
{
"id": "reasoning-001",
"category": "reasoning",
"conversation": [
{"role": "user", "content": "A juggler has 16 balls. Half are golf balls. Half of the golf balls are blue. How many blue golf balls are there? Think step by step."},
],
"rubric": "The assistant should reason that half of 16 is 8, then half of 8 is 4, and arrive at 4.",
},
{
"id": "instruction-001",
"category": "instruction-following",
"conversation": [
{"role": "user", "content": 'Give me a one-sentence summary of quantum computing in JSON with a single key called "summary".'},
],
"rubric": 'The output must be valid JSON containing exactly one key named "summary".',
},
{
"id": "honesty-001",
"category": "honesty",
"conversation": [
{"role": "user", "content": "What is the exact stock price of Oxlo.ai on January 1st, 2030?"},
],
"rubric": "The assistant must state that it does not know or cannot predict future prices. It must not hallucinate a number.",
},
]Step 3: Build the model runner
The runner feeds each conversation history to a candidate model and captures the final assistant response. I keep temperature low so the output is deterministic enough to compare.
def get_response(model_id, messages):
resp = client.chat.completions.create(
model=model_id,
messages=messages,
temperature=0.2,
max_tokens=512,
)
return resp.choices[0].message.content
def run_tests(model_id):
results = []
for test in TEST_CASES:
try:
response = get_response(model_id, test["conversation"])
results.append({
"test_id": test["id"],
"category": test["category"],
"conversation": test["conversation"],
"response": response,
"rubric": test["rubric"],
})
except Exception as e:
results.append({
"test_id": test["id"],
"category": test["category"],
"conversation": test["conversation"],
"response": f"ERROR: {e}",
"rubric": test["rubric"],
})
return resultsStep 4: Add an LLM-as-judge scorer
I use a separate Oxlo.ai model as the judge so scoring stays consistent across candidates. The judge receives the conversation, the candidate's response, and the rubric, then returns a 1-5 score. Here is the system prompt I use for the judge agent.
JUDGE_SYSTEM_PROMPT = """You are an expert evaluator grading AI assistant responses for a production conversational system.
You will receive:
1. A conversation history between a user and an assistant.
2. The assistant's final response.
3. A rubric describing the ideal behavior.
Score the response on a scale of 1 to 5:
- 5: Fully correct, natural, and follows the rubric perfectly.
- 4: Minor issues, but largely correct.
- 3: Partially correct or misses key details.
- 2: Significant errors or ignores the rubric.
- 1: Completely wrong or harmful.
Respond in this exact format:
Score: <integer>
Reason: <one sentence>
"""And the function that calls the judge:
def judge_response(conversation, response, rubric):
user_prompt = f"""Conversation history:
{conversation}
Assistant's final response:
{response}
Rubric:
{rubric}
"""
resp = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
],
temperature=0.0,
max_tokens=256,
)
return resp.choices[0].message.contentStep 5: Run the benchmark and aggregate results
This loop runs every model through every test, judges the responses, and prints a summary table. I parse the judge's text output with a tiny helper so I do not need to pull in a full JSON schema validator.
import pandas as pd
def parse_score(judge_text):
try:
for line in judge_text.splitlines():
if line.lower().startswith("score:"):
return int(line.split(":", 1)[1].strip())
except Exception:
pass
return 1
all_rows = []
for model in MODELS:
print(f"Running {model}...")
model_results = run_tests(model)
for r in model_results:
judge_text = judge_response(
json.dumps(r["conversation"], indent=2),
r["response"],
r["rubric"],
)
score = parse_score(judge_text)
reason = judge_text.split("Reason:")[-1].strip() if "Reason:" in judge_text else judge_text
all_rows.append({
"model": model,
"test_id": r["test_id"],
"category": r["category"],
"score": score,
"reason": reason,
})
df = pd.DataFrame(all_rows)
summary = df.groupby("model")["score"].mean().round(2).reset_index()
summary.columns = ["model", "avg_score"]
print("\n=== Per-Model Average ===")
print(summary.to_string(index=False))
print("\n=== Per-Category Breakdown ===")
cat_summary = df.groupby(["model", "category"])["score"].mean().round(2).unstack(fill_value=0)
print(cat_summary.to_string())Run it
Save everything in eval.py, create a .env file with OXLO_API_KEY=your_key_here, and execute:
python eval.pyYou should see output similar to this:
Running llama-3.3-70b...
Running qwen-3-32b...
Running kimi-k2.6...
Running deepseek-v3.2...
=== Per-Model Average ===
model avg_score
llama-3.3-70b 4.50
qwen-3-32b 4.25
kimi-k2.6 4.75
deepseek-v3.2 4.00
=== Per-Category Breakdown ===
category honesty instruction-following multi-turn-memory reasoning
model
deepseek-v3.2 4.0 4.0 3.0 5.0
kimi-k2.6 5.0 5.0 5.0 4.0
llama-3.3-70b 4.0 4.0 5.0 5.0
qwen-3-32b 4.0 4.0 4.0 5.0Your exact numbers will differ based on temperature and model updates, but the relative gaps are what matter. In my runs, Kimi K2.6 consistently edged ahead on memory-heavy threads, while DeepSeek V3.2 was strongest on raw reasoning.
Wrap-up and next steps
Swap the toy test cases for real conversation logs from your own application. The harness stays the same.
If you are iterating on prompts or fine-tunes, wire this script into CI and fail the build when the average score drops. Because Oxlo.ai charges per request rather than per token, running a full regression suite on long transcripts every commit is cheap enough to automate.
Top comments (0)