Best LLM Models for Coding Tasks: A Comparative Analysis

#learnai #oxlo #ai

I built a small Python harness that sends the same coding task to four different Oxlo.ai models and automatically checks which ones pass a set of unit tests. It helps me pick the right model for internal code-generation tools without guessing.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the Oxlo.ai client and model roster

First I instantiate a single OpenAI-compatible client pointed at Oxlo.ai. Because Oxlo.ai uses flat request-based pricing, running the same long prompt against four models costs the same per call regardless of input size. I then list the model slugs I want to compare.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

MODELS = [
    "deepseek-v3.2",
    "qwen-3-32b",
    "llama-3.3-70b",
    "kimi-k2.6",
]

Step 2: Define the coding task and system prompt

I keep the system prompt strict so the models return code in a predictable format. The user prompt is a real normalization function with edge cases that are easy to test automatically.

SYSTEM_PROMPT = """You are an expert Python engineer.
Write clean, production-ready code with type hints and docstrings.
Include error handling for edge cases.
Output only the Python code inside a markdown code block, followed by a brief explanation."""

USER_PROMPT = """Write a Python function normalize_phone_number(phone: str) -> str that converts
various US phone formats into E.164 (+1XXXXXXXXXX).
Raise ValueError for empty strings, non-digits (except an optional leading +1), or invalid lengths.
Do not write a main block or unit tests."""

Step 3: Build the evaluation runner

This helper fires the request and records how long the round trip takes. I set temperature low because I want deterministic code, not creative variation.

import time

def run_coding_task(model_id: str, system: str, user: str):
    start = time.time()
    response = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
        temperature=0.2,
    )
    latency = time.time() - start
    content = response.choices[0].message.content
    return content, latency

Step 4: Extract code from markdown fences

Models usually wrap code in markdown fences. This helper pulls out the raw Python so I can exec it safely in a clean namespace.

import re

def extract_python_code(text: str) -> str:
    pattern = r"

```python\s*(.*?)```

"
    match = re.search(pattern, text, re.DOTALL)
    if match:
        return match.group(1).strip()
    return text.strip()

Step 5: Run unit tests against each model

I define six test cases covering happy paths and expected failures. The evaluator imports nothing on its own, it simply runs the generated function and counts pass or fail results.

TEST_CASES = [
    ("(555) 123-4567", "+15551234567"),
    ("555.123.4567", "+15551234567"),
    ("+1 555 123 4567", "+15551234567"),
    ("5551234567", "+15551234567"),
    ("", None),
    ("123", None),
]

def evaluate_model(model_id: str, code: str):
    namespace = {}
    try:
        exec(code, namespace)
        func = namespace.get("normalize_phone_number")
        if not func:
            return "FAIL: function not found in output"
    except Exception as e:
        return f"FAIL: exec error - {e}"

    passed = 0
    failed = 0
    for inp, expected in TEST_CASES:
        try:
            result = func(inp)
            if expected is None:
                failed += 1
            elif result == expected:
                passed += 1
            else:
                failed += 1
        except ValueError:
            if expected is None:
                passed += 1
            else:
                failed += 1
        except Exception:
            failed += 1

    return f"PASS={passed} FAIL={failed}"

Run it

Save everything in compare_models.py, replace YOUR_OXLO_API_KEY, and run it. You should see each model generate a solution and the test runner report the counts. Here is what the output looks like on my machine.

$ python compare_models.py
Running deepseek-v3.2 ...
  PASS=6 FAIL=0

Running qwen-3-32b ...
  PASS=6 FAIL=0

Running llama-3.3-70b ...
  PASS=6 FAIL=0

Running kimi-k2.6 ...
  PASS=6 FAIL=0

Your results will vary depending on model updates and temperature settings. If a model fails, inspect the raw output to see whether it ignored the system prompt format or missed an edge case.

Wrap-up

Two concrete next steps. First, wire this harness into a GitHub Action to regression-test model behavior on your own codebase every time Oxlo.ai adds a new model to the catalog. Second, swap in specialized models like Qwen 3 Coder or DeepSeek R1 to see how reasoning-focused weights handle the same task. You can explore flat request-based pricing for long prompts at https://oxlo.ai/pricing.

DEV Community