I Ran the Same 40 Prompts Through Qwen2.5 and Qwen3. Here's the Script and Results.

Why I Didn't Just Trust the Benchmarks

Qwen3's published benchmarks look like a clean win over Qwen2.5 — real gains on MMLU-Pro, MATH, and coding tasks, plus a much larger training set (roughly 36 trillion tokens versus Qwen2.5's 18 trillion) and support for far more languages. On paper, swapping in Qwen3 should have been an easy call for my project.

I still wanted my own numbers, because a general benchmark tells you what a model can do on average, not what it does on the specific, sometimes-weird prompts your actual application receives. So I wrote a small script to run the same prompt set through both models and log the results side by side.

The Eval Script

Nothing fancy — this loops through a list of test prompts, calls both models with each one, and writes the outputs to a CSV for manual review.

import csv
import time
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"  # Qwen's own endpoint
)

test_prompts = [
    "Classify this support message: 'My payment failed twice this week.'",
    "A user says their invoice total doesn't match what they were quoted. What category does this fall under, and what follow-up question would you ask?",
    # ... add your own prompts here
]

models_to_compare = ["qwen2.5-72b-instruct", "qwen3-235b-a22b", "qwen3-235b-a22b-instruct"]

def run_eval(prompts, models):
    results = []
    for model in models:
        for prompt in prompts:
            response = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                temperature=0,
            )
            results.append({
                "model": model,
                "prompt": prompt,
                "output": response.choices[0].message.content,
            })
            time.sleep(0.5)  # basic pacing to avoid rate limits
    return results

results = run_eval(test_prompts, models_to_compare)

with open("qwen_comparison.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["model", "prompt", "output"])
    writer.writeheader()
    writer.writerows(results)

I deliberately kept temperature=0 to reduce randomness between runs — you want the comparison to reflect model differences, not sampling noise.

What I Actually Found

Out of 40 prompts specific to my ticket-classification use case, plain Qwen2.5-72B-Instruct and Qwen3-235B-A22B (the default, hybrid-thinking variant) landed close to each other overall — Qwen3 won clearly on the more ambiguous, multi-step prompts, but on short, unambiguous ones, Qwen2.5's answers came back faster and were occasionally more directly usable without extra parsing.

The bigger difference showed up when I added qwen3-235b-a22b-instruct — the non-thinking Instruct variant Qwen released separately — as a third comparison point. It matched Qwen2.5's speed on simple prompts while still outperforming it on the harder ones, which lines up with something I found afterward while reading about the initial Qwen3 release: early evaluations of the original hybrid-thinking Qwen3 models showed them underperforming Qwen2.5 on some agentic and instruction-following benchmarks, which is part of why the dedicated Instruct and Thinking variants exist as a separate release.

That's a specific, checkable claim for your own use case, not a general one — the point isn't "Qwen3-Instruct is better," it's that "Qwen3" isn't one model, and comparing against the wrong variant will give you a misleading result.

A Few Things Worth Watching in Your Own Comparison
Pin temperature=0 unless your task actually benefits from sampling variance — otherwise you're comparing noise, not models
Test the actual variant you'd deploy, not just whichever one your first pip install example happens to reference — Qwen3's hybrid, Instruct, and Thinking variants behave differently enough that lumping them together will skew your conclusion
Weight your prompt set toward your real traffic distribution — if 90% of your inputs are simple, a benchmark heavy on hard reasoning tasks won't tell you much about your actual cost/latency tradeoff
Where I Took This Next

Once I had this script working against Qwen's own endpoint, I wanted to run the same comparison against a couple of other providers, mostly out of curiosity about whether my results were Qwen-specific or held more generally. I ended up routing the same script through RouteAI instead of maintaining separate client configs per provider — the eval loop above didn't change at all, just the base_url and the model names in the list. That's a convenience thing, not a result — the comparison numbers above came from testing directly against Qwen's models.

TL;DR: Qwen3's benchmark gains are real, but "Qwen3" isn't a single model — the hybrid-thinking default, Instruct, and Thinking variants perform differently enough that comparing the wrong one against Qwen2.5 will mislead you. Full eval script above; test on your own prompts before deciding.

Here's the tool I referenced in this post: www.fastrouteai.com