I Benchmarked Qwen3 Thinking Mode vs Non-Thinking on My Own Task. Here's the Script.

The Default I Never Questioned

My support-ticket classifier had been running fine on Qwen3 for weeks. Then I noticed response times had roughly tripled, and a handful of cases that used to get clean, confident classifications were coming back hedged. I traced it to one thing: I'd left the model on its default thinking-enabled configuration without checking what that actually cost me on my specific workload.

Qwen3 can generate an internal step-by-step reasoning trace before answering (thinking mode), or respond more directly without it (non-thinking). I assumed thinking mode was a strict improvement. It isn't, not universally — so I built a small eval to find out where the line actually was for my task.

The Eval Script

import csv
import time
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)

test_messages = [
    "My card was declined for the third time this week.",
    "I'm not sure if this is a billing thing or a bug — my invoice looks wrong AND the app crashed.",
    # ... add your own real examples here
]

configs = [
    {"model": "qwen3-235b-a22b", "enable_thinking": True, "label": "thinking"},
    {"model": "qwen3-235b-a22b", "enable_thinking": False, "label": "non-thinking"},
]

def run_eval(messages, configs):
    results = []
    for config in configs:
        for msg in messages:
            start = time.time()
            response = client.chat.completions.create(
                model=config["model"],
                messages=[{"role": "user", "content": f"Classify this support message as billing, technical, account, or general: {msg}"}],
                extra_body={"enable_thinking": config["enable_thinking"]},
                temperature=0,
            )
            elapsed = time.time() - start
            results.append({
                "config": config["label"],
                "input": msg,
                "output": response.choices[0].message.content,
                "latency_seconds": round(elapsed, 2),
            })
    return results

results = run_eval(test_messages, configs)

with open("thinking_mode_eval.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["config", "input", "output", "latency_seconds"])
    writer.writeheader()
    writer.writerows(results)

I logged latency alongside the output specifically because that's the cost thinking mode adds — the accuracy question and the speed question need to be looked at together, not separately.

What I Actually Found

Running this against 50 real messages from my own logs, split roughly into "unambiguous" and "genuinely ambiguous" by manual review beforehand:

On unambiguous messages (~75% of my traffic), non-thinking mode matched my old baseline's speed almost exactly, with no accuracy loss. Thinking mode on the same messages took roughly 2-3x longer per request and didn't improve accuracy — occasionally it added unnecessary hedging to what should have been a clean categorical answer.

On genuinely ambiguous messages, the result flipped: thinking mode caught distinctions non-thinking mode missed, meaningfully more often than chance would explain. This wasn't marginal — on the ambiguous subset specifically, thinking mode's accuracy advantage was large enough to matter for a real application.

The takeaway wasn't "thinking mode is better" or "worse." It's that applying it uniformly to a workload that's mostly simple means paying a latency cost on most of your traffic to get an accuracy gain on a minority of it.

What I Built From This

A lightweight router: run everything through non-thinking mode first, and only escalate to thinking mode when a simple heuristic flags the input as ambiguous.

def classify_message(message, client):
    response = client.chat.completions.create(
        model="qwen3-235b-a22b",
        messages=[{"role": "user", "content": f"Classify this support message as billing, technical, account, or general: {message}"}],
        extra_body={"enable_thinking": False},
        temperature=0,
    )
    output = response.choices[0].message.content

    ambiguous_signals = ["and", "or", "not sure", "maybe"]
    if any(signal in message.lower() for signal in ambiguous_signals):
        response = client.chat.completions.create(
            model="qwen3-235b-a22b",
            messages=[{"role": "user", "content": f"Classify this support message as billing, technical, account, or general: {message}"}],
            extra_body={"enable_thinking": True},
            temperature=0,
        )
        output = response.choices[0].message.content

    return output

This heuristic is intentionally crude — checking for conflicting-signal keywords, not a real ambiguity classifier. It's good enough for a side project; I wouldn't trust it in production without more testing.

Where I Took This Next

Once the routing logic worked, I ran the same eval through RouteAI instead of Qwen's endpoint directly, mainly because switching between thinking and non-thinking configurations, and testing them against other models for comparison, meant changing arguments instead of maintaining separate client setups. The eval numbers above came from testing directly against Qwen's API — the gateway is a convenience layer on top, not part of the finding.

If You're Deciding Whether to Use Thinking Mode
Don't leave it on the default without checking what it costs on your actual workload — latency compounds fast at scale
Split your test set into "obviously simple" and "genuinely ambiguous" before running your eval; a single aggregate accuracy number will hide the real pattern
If your workload is mixed, consider routing by a cheap heuristic rather than picking one mode for everything

TL;DR: Qwen3's thinking mode isn't a universal upgrade — it helps on genuinely ambiguous inputs and mostly adds latency on simple ones with no accuracy gain. Full eval script above; a lightweight router based on the results cut my average latency without hurting accuracy on the harder cases.

Linking the tool mentioned above: www.fastrouteai.com