Today we are building a lightweight coding evaluator that runs the same programming task through four different Oxlo.ai models and scores the outputs. If you are tired of generic leaderboards and want to know which model actually works for your specific codebase, this tool gives you a concrete, side-by-side answer.
What you will need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
I also recommend a virtual environment. Oxlo.ai is fully OpenAI SDK compatible, so the client setup is a drop-in replacement.
Step 1: Configure the client and model list
First, I initialize the OpenAI client pointing at Oxlo.ai and define the four models we want to test. I picked these because they cover different strengths: Llama 3.3 70B for general fluency, Qwen 3 32B for agentic reasoning, Kimi K2.6 for advanced coding, and DeepSeek V3.2 because it offers strong reasoning on the free tier.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY")
)
MODELS = {
"llama-3.3-70b": "General-purpose flagship",
"qwen-3-32b": "Multilingual reasoning and agents",
"kimi-k2.6": "Advanced reasoning and agentic coding",
"deepseek-v3.2": "Coding and reasoning",
}
TASK_PROMPT = """
Write a Python function `parse_log_errors(filepath)` that reads a server log file, counts errors per hour, and returns a dictionary like `{"14": 3, "15": 1}`. The log lines look like:
2024-01-15 14:23:01 ERROR Connection timeout
2024-01-15 14:45:22 ERROR Disk full
2024-01-15 15:10:00 INFO Service restarted
Requirements:
1. Handle missing or unreadable files with a clear ValueError.
2. Use only the standard library.
3. Include a docstring and type hints.
"""Step 2: Define the system prompt
I keep the system prompt identical across all models so the only variable is the weights. This prompt emphasizes correctness and brevity over explanation.
SYSTEM_PROMPT = """You are an expert software engineer.
Write clean, correct Python code that follows PEP 8.
Return only the code block inside triple backticks.
Do not include explanations outside the code block."""Step 3: Build the evaluation runner
Next, I write a helper that sends the task to each model, strips the markdown fences, and saves the raw Python to a temporary file. I use a ten-second timeout because fast iteration matters more than exhaustive generation for this benchmark.
import tempfile
import time
def fetch_code(model_id: str, prompt: str) -> tuple[str, float]:
start = time.time()
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
],
temperature=0.2,
max_tokens=1024,
)
elapsed = time.time() - start
raw = response.choices[0].message.content or ""
# Strip markdown fences
lines = raw.splitlines()
if lines and lines[0].strip().startswith("
```"):
lines = lines[1:]
if lines and lines[-1].strip().startswith("```
"):
lines = lines[:-1]
code = "\n".join(lines).strip()
return code, elapsed
results = {}
for model_id in MODELS:
print(f"Running {model_id}...")
code, elapsed = fetch_code(model_id, TASK_PROMPT)
results[model_id] = {"code": code, "time": elapsed}Step 4: Score the outputs automatically
Automated scoring is blunt, but it catches the basics. I check for syntax errors with Python's built-in compiler, verify the required function name exists, and look for a docstring and type hints. Each check awards one point.
import py_compile
import ast
import tempfile
import os
def score_snippet(code: str) -> dict:
points = {"syntax": 0, "has_function": 0, "has_docstring": 0, "has_type_hints": 0}
with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
f.write(code)
tmp_path = f.name
try:
py_compile.compile(tmp_path, doraise=True)
points["syntax"] = 1
except Exception:
pass
finally:
os.unlink(tmp_path)
try:
tree = ast.parse(code)
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name == "parse_log_errors":
points["has_function"] = 1
if ast.get_docstring(node):
points["has_docstring"] = 1
if node.returns or any(a.annotation for a in node.args.args):
points["has_type_hints"] = 1
break
except Exception:
pass
points["total"] = sum(v for k, v in points.items() if k != "total")
return points
for model_id, data in results.items():
data["score"] = score_snippet(data["code"])Step 5: Render the leaderboard
Finally, I print a Markdown table sorted by total score, then by generation time. This gives an at-a-glance view of which Oxlo.ai model is the best fit for this specific coding task.
print("\n| Model | Description | Syntax | Function | Docstring | Types | Total | Time (s) |")
print("|-------|-------------|--------|----------|-----------|-------|-------|----------|")
sorted_results = sorted(
results.items(),
key=lambda x: (-x[1]["score"]["total"], x[1]["time"])
)
for model_id, data in sorted_results:
s = data["score"]
print(f"| {model_id} | {MODELS[model_id]} | {s['syntax']} | {s['has_function']} | {s['has_docstring']} | {s['has_type_hints']} | {s['total']} | {data['time']:.2f} |")Run it
Save everything into eval.py, export your key, and run it. On my last execution against Oxlo.ai, the output looked like this.
$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python eval.py
Running llama-3.3-70b...
Running qwen-3-32b...
Running kimi-k2.6...
Running deepseek-v3.2...
| Model | Description | Syntax | Function | Docstring | Types | Total | Time (s) |
|-------|-------------|--------|----------|-----------|-------|-------|----------|
| deepseek-v3.2 | Coding and reasoning | 1 | 1 | 1 | 1 | 4 | 2.34 |
| kimi-k2.6 | Advanced reasoning and agentic coding | 1 | 1 | 1 | 1 | 4 | 3.12 |
| qwen-3-32b | Multilingual reasoning and agents | 1 | 1 | 1 | 0 | 3 | 2.89 |
| llama-3.3-70b | General-purpose flagship | 1 | 1 | 0 | 0 | 2 | 2.56 |Your exact scores will vary depending on temperature and the current version weights, but the relative pattern usually holds. DeepSeek V3.2 and Kimi K2.6 tend to nail the structure, while general-purpose models sometimes skip the docstring or type hints on short prompts.
Wrap-up
You now have a reproducible script for pitting Oxlo.ai models against real tasks from your own backlog. Two concrete next steps: wire the winning model into a GitHub Action that reviews pull requests, or extend the scorer to actually execute the generated code against unit tests instead of just parsing it.
Because Oxlo.ai uses flat per-request pricing, running this benchmark with long prompts or massive diffs does not scale in cost the way token-based billing would. If you are evaluating models for an agentic coding pipeline, that pricing difference adds up quickly. See https://oxlo.ai/pricing for details.
Top comments (0)