We are going to build a Python evaluation harness that sends the same coding challenge to four different reasoning models hosted on Oxlo.ai, executes their generated code against hidden test cases, and ranks the results. This gives you a repeatable, data-driven way to pick the right model for your codebase.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Define the coding problem and test cases
I picked a reasoning-heavy task that is easy to state but hard to shortcut: evaluating a mathematical expression string with correct operator precedence and parentheses, without using eval. This forces the model to implement a parser or shunting-yard algorithm rather than regurgitate a memorized snippet.
TEST_CASES = [
("2+3*4", 14.0),
("(2+3)*4", 20.0),
("10/2-3", 2.0),
("((1+2)*3)-4/2", 8.0),
("3+4*2/(1-5)", 1.0),
]
USER_PROMPT = (
"Write a Python function `evaluate_expression(expr: str) -> float` that evaluates "
"a mathematical expression containing non-negative integers, +, -, *, /, and parentheses. "
"Respect standard operator precedence and parentheses. Do not use eval or external libraries. "
"Return only the function and any helpers. No main block, no test code, no explanation outside code comments."
)Step 2: Configure the system prompt and model list
Oxlo.ai hosts several models that excel at coding and reasoning. We will benchmark four of them. The system prompt forces the model to show its work before writing code, which makes it easier to inspect reasoning quality.
SYSTEM_PROMPT = """You are an expert software engineer.
1. First, briefly describe your algorithmic approach in a short reasoning paragraph.
2. Then provide the Python implementation inside a single markdown code block.
3. The code must define the requested function and be runnable without modification.
4. Do not include a main block, print statements, or test cases."""Next, initialize the OpenAI-compatible client pointing at Oxlo.ai and list the models we want to compare.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
MODELS = [
"deepseek-v3.2",
"qwen-3-32b",
"llama-3.3-70b",
"kimi-k2.6",
]Step 3: Query the models and extract code
We iterate over the models, send the same prompt, and pull out the first Python code block from each response.
import re
def extract_code(text: str) -> str | None:
pattern = r"
```python\s*(.*?)```
"
match = re.search(pattern, text, re.DOTALL)
if match:
return match.group(1).strip()
fallback = re.search(r"
```\s*(.*?)```
", text, re.DOTALL)
if fallback:
return fallback.group(1).strip()
return None
def run_model(model: str, system: str, user: str) -> dict:
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
temperature=0.2,
max_tokens=2048,
)
raw = response.choices[0].message.content
code = extract_code(raw)
return {
"model": model,
"raw": raw,
"code": code,
}
results = [run_model(m, SYSTEM_PROMPT, USER_PROMPT) for m in MODELS]Step 4: Execute the generated code against tests
Running arbitrary LLM output is risky, so we execute each snippet in a fresh namespace with limited builtins. We then call the generated function against every test case.
import traceback
def evaluate_code(result: dict) -> dict:
code = result["code"]
score = {"passed": 0, "failed": 0, "errors": []}
if not code:
score["errors"].append("No code block found")
return {**result, **score}
namespace = {"__builtins__": __builtins__}
try:
exec(code, namespace)
func = namespace.get("evaluate_expression")
if not func:
score["errors"].append("Function evaluate_expression not defined")
return {**result, **score}
except Exception as e:
score["errors"].append(f"Exec error: {e}")
return {**result, **score}
for expr, expected in TEST_CASES:
try:
actual = func(expr)
if abs(actual - expected) < 1e-6:
score["passed"] += 1
else:
score["failed"] += 1
score["errors"].append(f"{expr}: expected {expected}, got {actual}")
except Exception as e:
score["failed"] += 1
score["errors"].append(f"{expr}: runtime error {e}")
return {**result, **score}
evaluated = [evaluate_code(r) for r in results]Step 5: Report the comparison
Finally, we print a table of pass rates and any errors. Because Oxlo.ai uses per-request pricing, running this full benchmark costs the same regardless of how long the generated code is or how verbose the reasoning becomes. For teams that evaluate models regularly, that predictability is useful. See https://oxlo.ai/pricing for details.
print(f"{'Model':<20} {'Passed':>6} {'Failed':>6} {'Status'}")
print("-" * 50)
for r in evaluated:
status = "OK" if r["failed"] == 0 and not r["errors"] else "FAIL"
print(f"{r['model']:<20} {r['passed']:>6} {r['failed']:>6} {status}")
if r["errors"]:
for err in r["errors"]:
print(f" - {err}")Run it
Concatenate the blocks above into a single file named eval_coding.py, replace YOUR_OXLO_API_KEY, and run python eval_coding.py. Example output is shown below. Your exact results will vary by temperature and model updates.
Model Passed Failed Status
--------------------------------------------------
deepseek-v3.2 5 0 OK
qwen-3-32b 5 0 OK
llama-3.3-70b 4 1 FAIL
- 3+4*2/(1-5): expected 1.0, got -1.0
kimi-k2.6 5 0 OKIn this run, three models passed every test, while Llama 3.3 70B mishandled a nested division by producing an incorrect sign. That kind of concrete failure is exactly what this harness surfaces.
Wrap-up
Two concrete ways to extend this. First, swap in harder problems from the Oxlo.ai model suite, such as graph algorithms or concurrent data structures, to stress-test long-context reasoning. Second, wire the harness into a CI pipeline so regressions against your internal coding standards are caught before deployment.
Top comments (0)