I Benchmarked 4 Qwen Models on My Own Task. Here's the Script and What I'd Recommend.

"Best" Depends on a Question Nobody Asks First

Every "best Qwen model" article I found gave me a ranking with no context about what I was building. I'm working on a tool that extracts themes from customer feedback — mostly simple text, occasionally ambiguous, needs to run cheap and fast at volume. That's a specific enough task that a general ranking wasn't going to answer my actual question, so I built a small eval instead.

The Eval Setup

I tested four models against the same 30 feedback snippets: Qwen3-4B, Qwen3-32B, Qwen3-235B-A22B, and Qwen2.5-72B (included mainly out of curiosity about the older lineup). Same prompt, same scoring, temperature=0 to keep the comparison fair.

import csv
import time
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)

feedback_snippets = [
    "Shipping was way slower than the estimate said.",
    "It's fine I guess, just not what I expected.",
    # ... add your own real examples here
]

models_to_test = ["qwen3-4b", "qwen3-32b", "qwen3-235b-a22b", "qwen2.5-72b-instruct"]

PROMPT_TEMPLATE = """Extract the main theme from this customer feedback in one short phrase.
Feedback: {feedback}"""

def run_eval(snippets, models):
    results = []
    for model in models:
        for snippet in snippets:
            response = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(feedback=snippet)}],
                temperature=0,
            )
            results.append({
                "model": model,
                "input": snippet,
                "output": response.choices[0].message.content,
            })
            time.sleep(0.5)
    return results

results = run_eval(feedback_snippets, models_to_test)

with open("qwen_model_eval.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["model", "input", "output"])
    writer.writeheader()
    writer.writerows(results)

I scored outputs manually against what I, as a human reviewer, would have extracted — not against a public benchmark, since that wouldn't tell me anything about my specific prompt and data.

What I Actually Found

On straightforward feedback ("shipping was slow," "loved the packaging"), Qwen3-4B performed close to identically to the much larger models — noticeably faster and cheaper per request, with no meaningful drop in accuracy for my task. This lines up with something Qwen's team has said about the Qwen3 lineup: due to architecture and training improvements, several smaller Qwen3 dense models reportedly perform comparably to much larger Qwen2.5 models on general benchmarks. I hadn't taken that at face value until it held up on my own data.

The larger models — Qwen3-32B and Qwen3-235B-A22B — only pulled ahead on genuinely ambiguous inputs, roughly 20% of my dataset. Qwen2.5-72B was competitive on the easy cases and, for my specific prompt phrasing, occasionally produced more consistently formatted output than Qwen3-4B, which mattered for how much post-processing I had to write around it.

None of that adds up to one model being "best." It adds up to: for mostly-simple workloads, a smaller model likely handles it fine at lower cost. For workloads with a meaningful share of ambiguous inputs, you're paying for reasoning capacity whether every request needs it or not — unless you route by difficulty.

What I Built Because of This

Since only ~20% of my inputs actually needed the larger model's reasoning, I split the pipeline: run everything through Qwen3-4B first, and escalate to Qwen3-32B only when the output looked uncertain (short output, hedge words like "maybe" or "unclear" in the theme extracted).

def extract_theme(feedback, client):
    response = client.chat.completions.create(
        model="qwen3-4b",
        messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(feedback=feedback)}],
        temperature=0,
    )
    output = response.choices[0].message.content

    uncertain_markers = ["maybe", "unclear", "possibly", "not sure"]
    if len(output.split()) < 3 or any(marker in output.lower() for marker in uncertain_markers):
        response = client.chat.completions.create(
            model="qwen3-32b",
            messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(feedback=feedback)}],
            temperature=0,
        )
        output = response.choices[0].message.content

    return output

This is a simple heuristic, not a robust confidence-scoring system — good enough for a side project, probably not for production without more testing.

The One Thing I'd Change Next Time

Running this eval against Qwen's own endpoint directly worked fine, but I ended up routing the same script through RouteAI afterward, mainly because the same eval structure let me sanity-check whether my results held on a couple of other providers too, without maintaining separate client setups per provider. The eval code above didn't change — just the base_url and model names. That's a convenience note, not a finding; the numbers above came from testing Qwen's models directly.

If You're Trying to Answer This for Your Own Project
Skip the ranking articles for your final decision — use them to pick 2-3 candidates worth testing, nothing more
Build a small eval on 20-30 real examples from your actual data, not a public benchmark
Check whether your workload is mostly simple or mostly ambiguous — that ratio matters more than any single benchmark score
Consider routing by difficulty instead of picking one model for everything, if your workload is mixed

TL;DR: There's no single "best Qwen model" — it depends on how much of your workload actually needs reasoning versus straightforward extraction. Full eval script above; I ended up routing simple cases to a small model and escalating only when needed, which cut cost without a meaningful accuracy hit on my data.

Worth exploring if this is relevant to your stack: www.fastrouteai.com