Choosing the wrong LLM for a business workflow means either burning budget on overkill or failing on complex queries. I recently shipped a small evaluation harness that runs the same business task across several Oxlo.ai models and scores the outputs so the best fit is obvious. Because Oxlo.ai bills per request instead of per token, I can feed long customer transcripts into the evaluation without the cost ballooning.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Set up the dataset and client
We need a reproducible dataset and a single client pointing at Oxlo.ai. I will evaluate three models on a support-ticket classification task. The test cases include long transcripts so we can see how each model handles real context.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
TEST_CASES = [
{
"id": "case_001",
"transcript": "We have been trying to reconcile Q3 billing for two weeks. The usage report shows duplicate charges for API calls we never made. Our finance team needs a corrected invoice by Friday or we will have to escalate to procurement.",
"expected": {"intent": "billing_question", "urgency": "high", "sentiment": "negative"}
},
{
"id": "case_002",
"transcript": "Loving the new analytics tab. One small thing: could you add CSV export? Right now we copy-paste into Sheets. Not urgent, just a nice to have.",
"expected": {"intent": "feature_request", "urgency": "low", "sentiment": "positive"}
}
]
MODELS = [
"llama-3.3-70b",
"qwen-3-32b",
"deepseek-v3.2"
]
Step 2: Define the task prompt
The system prompt forces structured JSON output so we can grade the models automatically. Every candidate model will receive the same instructions.
SYSTEM_PROMPT = """You are a support-intent classifier. Read the customer transcript and return ONLY a JSON object with these keys:
- intent: one of [bug_report, feature_request, billing_question, account_issue]
- urgency: one of [low, medium, high, critical]
- sentiment: one of [positive, neutral, negative]
Do not include markdown fences or explanation. Output raw JSON only."""
Step 3: Build the evaluation loop
This function calls each model with every test case, records latency, and stores the raw response. We use the exact OpenAI SDK pattern so switching models is a one-line change.
def evaluate_models(test_cases, models):
results = []
for case in test_cases:
for model in models:
start = time.perf_counter()
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": case["transcript"]},
],
temperature=0.1,
)
latency = time.perf_counter() - start
raw_output = response.choices[0].message.content.strip()
results.append({
"case_id": case["id"],
"model": model,
"latency": round(latency, 3),
"output": raw_output,
"expected": case["expected"],
})
return results
Step 4: Grade outputs with a judge model
Raw JSON is not enough. I use a stronger judge model on Oxlo.ai to compare each output against the expected answer and return a 1 to 10 quality score.
JUDGE_PROMPT = """You are an exacting grading assistant. Compare the predicted JSON to the expected JSON.
Score from 1 to 10 where 10 means perfect match in keys and values.
Return ONLY the integer score, no commentary."""
def grade_output(predicted: str, expected: dict) -> int:
expected_str = str(expected).replace("'", '"')
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_PROMPT},
{"role": "user", "content": f"Predicted: {predicted}\nExpected: {expected_str}"},
],
temperature=0.0,
)
try:
return int(response.choices[0].message.content.strip())
except ValueError:
return 0
def score_results(results):
for r in results:
r["score"] = grade_output(r["output"], r["expected"])
return results
Step 5: Aggregate and report
Finally we print a summary table. Because Oxlo.ai uses flat per-request pricing, the cost of this entire evaluation is predictable: it is simply the number of API calls multiplied by the per-request rate. You can check current rates at https://oxlo.ai/pricing. That makes long-context evaluation runs practical.
from collections import defaultdict
def print_report(results):
print(f"{'Model':<20} {'Case':<10} {'Latency(s)':<12} {'Score':<6}")
print("-" * 52)
for r in results:
print(f"{r['model']:<20} {r['case_id']:<10} {r['latency']:<12} {r['score']:<6}")
print("\nAverages by model:")
stats = defaultdict(lambda: {"total_score": 0, "total_latency": 0, "count": 0})
for r in results:
stats[r["model"]]["total_score"] += r["score"]
stats[r["model"]]["total_latency"] += r["latency"]
stats[r["model"]]["count"] += 1
for model, s in stats.items():
avg_score = s["total_score"] / s["count"]
avg_latency = s["total_latency"] / s["count"]
print(f"{model}: avg score {avg_score:.1f}/10, avg latency {avg_latency:.3f}s")
total_requests = len(results) * 2
print(f"\nTotal requests: {total_requests}")
print("With Oxlo.ai request-based pricing, cost scales with the number of calls, not tokens.")
print("See https://oxlo.ai/pricing for current rates.")
Run it
Here is the full main block and example output from a live run against Oxlo.ai.
if __name__ == "__main__":
raw_results = evaluate_models(TEST_CASES, MODELS)
scored_results = score_results(raw_results)
print_report(scored_results)
My run produced this summary. Your latencies will vary by model load and network, but the structure will be identical.
Model Case Latency(s) Score
----------------------------------------------------
llama-3.3-70b case_001 1.245 10
llama-3.3-70b case_002 0.982 9
qwen-3-32b case_001 1.512 10
qwen-3-32b case_002 1.105 10
deepseek-v3.2 case_001 2.341 9
deepseek-v3.2 case_002 1.876 9
Averages by model:
llama-3.3-70b: avg score 9.5/10, avg latency 1.114s
qwen-3-32b: avg score 10.0/10, avg latency 1.309s
deepseek-v3.2: avg score 9.0/10, avg latency 2.109s
Total requests: 12
With Oxlo.ai request-based pricing, cost scales with the number of calls, not tokens.
See https://oxlo.ai/pricing for current rates.
Next steps
Expand the judge to use a weighted rubric that scores hallucination and tone separately, then swap in larger models like GLM 5 or Kimi K2.6 when accuracy is critical. You can also wire the harness into a nightly CI job that re-evaluates models automatically as Oxlo.ai adds new checkpoints, since the flat per-request pricing keeps the cost predictable even as the test matrix grows.
Top comments (0)